Knowledge enhancement based chest x-ray image classification method and imaging method

Abstract

The application discloses a chest X-ray image classification method based on knowledge enhancement, comprising the following steps: acquiring a benchmark data set and constructing a knowledge graph; performing node type differentiation and node relationship modeling processing on the constructed knowledge graph; constructing a local classifier based on the acquired data set, and performing local classification processing on the processed knowledge graph; constructing a global classifier based on the acquired data set and the local classifier, and performing global classification processing on the processed knowledge graph; and using the trained global classifier to complete classification processing on actual input X-ray images; and the application further discloses an imaging method comprising the chest X-ray image classification method based on knowledge enhancement. The method creates a KG knowledge graph based on a chest X-ray data set, integrates existing knowledge into the established model, and effectively improves the classification effect of chest X-ray local classification and global classification; meanwhile, the method has high effectiveness, improved precision and enhanced generalization.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a chest X-ray image classification method based on knowledge enhancement and an imaging method. BACKGROUND

[0002] The automatic diagnosis method based on chest X-ray is one of the key steps for developing a computer-aided diagnosis (CAD) system. In recent years, the chest X-ray image dataset has the advantages of large scale and good annotation, which provides researchers with more resources to explore automatic diagnosis. However, due to the complexity of medical images, the diversity of diseases and the unbalanced distribution of labels, automatic diagnosis is still challenging for researchers. Although the automatic diagnosis of chest X-ray can effectively reduce the burden of radiologists, many existing methods rely on visual features or labels, which may lack robustness for small or rare diseases.

[0003] At present, the automatic diagnosis method for chest X-ray mainly includes two types: (1) a large-scale natural image dataset is used to pre-train a deep convolutional model, and based on the transfer learning algorithm, the trained deep convolutional model is transferred through the chest X-ray image dataset, and the global image features are used to determine the lesion condition. However, due to the huge difference between natural images and medical images, the domain knowledge transfer may have serious domain shift problem, and the local information is not concerned, so the position of the lesion cannot be accurately located, resulting in lack of explainability; (2) the co-occurrence frequency of multiple labels in the training set is used to construct a label dependency graph, and the graph is used to model the dependency between lesions. However, this method is limited by the long-tail distribution of labels in the training set, and overfitting problem may occur.

[0004] In summary, the current automatic diagnosis method has certain problems when classifying chest X-ray, and the classification accuracy is lacking, the effect is not ideal, and the classification result lacks generalization. SUMMARY

[0005] One of the purposes of the present application is to provide a chest X-ray image classification method based on knowledge enhancement with high effectiveness, improved accuracy and enhanced generalization.

[0006] The second purpose of the present application is to provide an imaging method comprising the chest X-ray image classification method based on knowledge enhancement.

[0007] The chest X-ray image classification method based on knowledge enhancement provided by the present application comprises the following steps:

[0008] S1. Obtain a reference dataset and construct a knowledge graph;

[0009] S2. Node type distinguishing and node relationship modeling processing are performed on the knowledge graph constructed in step S1;

[0010] S3. A local classifier is constructed based on the data set obtained in step S1, and local classification processing is performed on the knowledge graph processed in step S2;

[0011] S4. A global classifier is constructed based on the data set obtained in step S1 and the local classifier constructed in step S3, and global classification processing is performed on the knowledge graph processed in step S2;

[0012] S5. The global classifier trained in step S4 is used to complete the classification processing of the actual input X-ray image.

[0013] The step S1 described obtaining the reference data set, constructing the knowledge graph, specifically includes:

[0014] The obtained reference data set includes Chest ImaGenome data set, ChestX-ray14 data set, VinBigData data set; the knowledge graph KG is constructed based on the Chest ImaGenome data set, specifically including:

[0015] In each chest X-ray image in the Chest ImaGenome data set, the anatomical position with a bounding box is labeled, the "position" is selected, and the local finding found in the "position" is labeled;

[0016] The chest X-ray images in the data set are processed, the positions without any local finding are deleted, the set local finding is retained, the "anatomical position" is referred to as "position", and the "local finding" is referred to as "attribute", which constitutes the nodes in the knowledge graph KG;

[0017] The Chest ImaGenome data set provides the relationship between the position and the attribute; each node is connected through the "relationship", which constitutes the "edge" in the knowledge graph KG; the constructed knowledge graph KG can be used to assist the lesion detection on multiple data sets.

[0018] The step S2 described node type distinguishing and node relationship modeling processing are performed on the knowledge graph constructed in step S1, specifically including:

[0019] 1) Node type distinguishing:

[0020] The types of nodes are distinguished by different initialization methods:

[0021] ① Position node:

[0022] The initialization is performed by using the ROI features of the anatomical position, so that the features of the nodes are based on the anatomical position, thereby distinguishing the position nodes, specifically including:

[0023] The front X-ray image of the Chest ImaGenome dataset is taken as the input of the CNN, and the image features f are extracted by the Faster R-CNN target detection algorithm I ∈R d×H×W , and the bounding box or the ROI of the anatomical position in the input X-ray is detected Wherein, d represents the number of channels of the visual features, H represents the height of the visual features, W represents the width of the visual features, n o is the number of positions, and R represents the dimension of the feature vector; by applying ROI pooling on f I and B, the ROI features of each position are obtained used to represent each position node in the knowledge graph;

[0024] ② Attribute node:

[0025] The attribute nodes are distributed in the entire region of the X-ray image, and the initialization is realized by applying spatial attention to f I , and the distinction of the attribute nodes is completed;

[0026] By applying a convolution layer with a specified filter size and a SoftMax function in the spatial dimension, an attention map is generated for each attribute, and the initialization of the attribute nodes is completed by weighting f I through the attention map, and the formula is as follows:

[0027]

[0028]

[0029] Wherein, is the attention map, Conv2d(·) is a two-dimensional convolution operation with a filter size of 1x1, the input channel number of Conv2d(·) is d, and the output channel number is n a , n a is the number of attribute nodes, is the attribute feature, is the visual feature mentioned above;

[0030] Each attribute node embedded is classified by using a binary classifier, and the existence of the corresponding attribute (local discovery) in the entire image is predicted;

[0031] 2) Node relationship modeling:

[0032] R-GCN is applied for feature aggregation processing, and the relationship between the position and the attribute is modeled, specifically including:

[0033] R-GCN distinguishes the importance of different adjacent nodes by constructing independent message propagation parameters for each relationship type, and the message propagation process of R-GCN is described by the following formula:

[0034]

[0035] wherein, is the hidden state of node i at the l+1 layer, is a learnable parameter, is the hidden state of node i at the l layer, R is the relationship set, and N r (i) is the neighbor of node i under the relationship r∈R, c r is a normalization parameter, is a learnable parameter based on the relationship r∈R, is the hidden state of node j at the l layer;

[0036] The final embedding of the node is obtained by inputting the initialized feature of the node into two layers of R-GCN connected in series and and respectively, the hidden representation of the position node and the attribute node fused with the knowledge structure; the finally obtained embedding is used to calculate the score of each attribute node at each position to determine whether the local discovery corresponding to the attribute node appears at this position.

[0037] The local classifier is constructed based on the data set obtained in step S1, and the local classification processing is performed on the knowledge graph processed in step S2, and specifically includes:

[0038] The local classifier is constructed by using the Faster R-CNN target detection network and the R-GCN graph neural network described in step S2, and given the input chest X-ray, the normalized score of each attribute-position pair is predicted by the local classifier, and the relationship between the attribute and the position is determined by the score; the normalized score represents the possibility of the occurrence of a certain attribute at a certain position, and the specific result is determined by setting the threshold value of the score; the threshold value of the multi-label classification is obtained by maximizing the AUC index of the lesion detection, and the multi-target multi-label classification on the image is realized;

[0039] The score of each attribute-position pair is calculated by the dot product between the two node embeddings, and the formula is as follows:

[0040]

[0041] wherein, p l is the detection score, and are the hidden representations of the position node and attribute node fused with the knowledge structure respectively, and is the dot product operation;

[0042] The loss function is binary cross-entropy, which is used to optimize the parameters in the model during the model training process, and the loss function is expressed by the following formula:

[0043]

[0044] Wherein, L local is the loss value of local lesion classification, y i,j is the true label, represents the probability of the jth attribute appearing at the ith position;

[0045] Two auxiliary tasks are introduced to improve the performance of the local classifier.

[0046] The two auxiliary tasks included in step S3 specifically include:

[0047] Auxiliary task one:

[0048] Through auxiliary task one, the position detection is performed, the region proposal network RPN of Faster R-CNN is used to detect each position, and the real boundary box is used as the training target of the network, so that the model can accurately detect each anatomical position, and L rcnn represents the loss of auxiliary task one;

[0049] Auxiliary task two:

[0050] Through auxiliary task two, the attribute classification is performed, and the full connection layer and the Sigmoid function constitute a classifier, and auxiliary task two enables the attribute node features generated by the model to focus on the features of the corresponding discovery area on the image, and L attr represents the loss of auxiliary task two;

[0051] The total loss term of local classification training is expressed by the following formula:

[0052] L total = L rcnn + L attr + L local

[0053] All parameters of the local classifier are updated through the loss function L total , including the parameters of Faster R-CNN, AttributeAttention and R-CNN, so as to optimize the accuracy of anatomical position detection and the accuracy of local classification.

[0054] The global classifier is constructed based on the data set obtained in step S1 and the local classifier constructed in step S3, and global classification processing is performed on the knowledge graph processed in step S2, and specifically includes:

[0055] Multi-label classification is realized through the global classifier, and different label sets are corresponded according to the differences of the data sets; the local classifier described in step S3 is adopted, the knowledge-enhanced local features are extracted based on the trained local classifier, and the global-local attention is applied to fuse the global and local features;

[0056] The Chest ImaGenome data set is used to train the local classifier, the parameters of the trained classifier are fixed as the knowledge enhancement module in the global classifier, and the data sets ChestX-ray14 and VinBigData are trained;

[0057] The set position features are used attribute features and detection scores The sigmoid function is applied on p l to obtain the probability of each position-attribute pair, the attribute features are weighted according to the probability, and are fused with the position features, and the fusion process is described by the following formula:

[0058]

[0059] wherein, is the knowledge-enhanced local feature or KE feature of each position, f o,i is the feature of the i-th position node, f a,j is the feature of the j-th attribute node, is the score between the i-th position node and the j-th attribute node, and sigma(·) represents the sigmoid function.

[0060] The global-local attention is designed to fuse multi-granularity features, the CNN is applied to extract the global feature f g ∈R 1×d , f g is used as a query, f ke is used as a key and a value, the KE feature is processed through the attention mechanism, and then is fused with the global feature, according to the score of the position-attribute pair, the information of each attribute (local discovery) that may appear on each position is fused into the feature of the position node with different importance, and the following formula is used for description:

[0061]

[0062] wherein, f final ∈R 1×dis the final feature of multi-label classification, d is the feature dimension, is the transpose of knowledge enhanced local feature;

[0063] An application of a fully connected layer is used to classify the final feature, and the formula used is as follows:

[0064] p g =f final ·W g +b

[0065] Where p g is the possibility of each classification result appearing, as the result of global classification; W g is the weight term of the classifier, and b is the bias term of the classifier;

[0066] The loss function is binary cross entropy, as described in the following formula:

[0067]

[0068] Where L is the binary cross entropy loss, c is the number of classes, y i is the label of the i-th class, is the predicted probability of the i-th lesion.

[0069] The application also provides an imaging method comprising the knowledge-enhanced chest X-ray image classification method, comprising the following steps:

[0070] S6. Based on the classification result obtained in step S5, secondary imaging processing is performed on the actual X-ray image to obtain an X-ray image with a classification result mark.

[0071] The knowledge-enhanced chest X-ray image classification method provided by the application effectively improves the classification effect of chest X-ray local classification and global classification by creating a KG knowledge graph based on a chest X-ray dataset and integrating existing knowledge into the established model. At the same time, the application has high effectiveness, improved precision and enhanced generalization. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 is a method flowchart of the method of the application.

[0073] Figure 2 is a schematic diagram of the knowledge graph and model framework proposed by the method of the application.

[0074] Figure 3 is a process visualization schematic diagram of the implementation of the method of the application on a test case.

[0075] Figure 4 is a schematic diagram of the imaging method flowchart comprising the method of the application. DETAILED DESCRIPTION

[0076] As Figure 1 The method flow diagram of the method of the application is shown: the chest X-ray image classification method based on knowledge enhancement provided by the application comprises the following steps:

[0077] S1. Obtain a reference dataset and construct a knowledge graph; specifically comprising:

[0078] The obtained reference dataset comprises a Chest ImaGenome dataset, a ChestX-ray14 dataset, and a VinBigData dataset; a knowledge graph (KG) is constructed based on the Chest ImaGenome dataset, specifically comprising:

[0079] As Figure 2 shown, in each chest X-ray image in the Chest ImaGenome dataset, an anatomical position is labeled with a bounding box, a "position" is selected, and a local finding found in the "position" is labeled; the anatomical position comprises a left low lung area, a cardiac silhouette, the local finding comprises lung opacity, effusion, and the finding type comprises "anatomical finding", "technical assessment", and "disease";

[0080] The chest X-ray images in the dataset are processed, positions without any local finding are deleted, local findings of the type "anatomical finding" or "disease" are retained, the "anatomical position" is referred to as "position", and the "local finding" is referred to as "attribute", thereby constituting nodes in the KG;

[0081] In the dataset used in the method of the application, 21 anatomical positions and 46 local findings are retained;

[0082] The Chest ImaGenome dataset provides the relationship between positions and attributes; various nodes are connected through "relationships" to constitute "edges" in the KG graph; the constructed knowledge graph (KG) can be used to assist lesion detection on multiple datasets;

[0083] S2. Node type differentiation and node relationship modeling processing are performed on the knowledge graph constructed in step S1; specifically comprising:

[0084] 1) Node type differentiation:

[0085] Different initialization methods are used to differentiate the types of nodes:

[0086] ① Position node:

[0087] The initialization is performed by using ROI features of an anatomical position, so that the features of the nodes are based on the anatomical position, thereby distinguishing the position nodes, specifically including:

[0088] The front X-ray image of the Chest ImaGenome dataset is taken as the input of the CNN, and the image features f are extracted by the Faster R-CNN target detection algorithm I ∈R d×H×W , and the bounding box or the ROI of the anatomical position in the input X-ray is detected wherein d represents the number of channels of the visual features, H represents the height of the visual features, W represents the width of the visual features, n o is the number of positions, and R represents the dimension of the feature vector; by applying ROI pooling on f I and B, the ROI features of each position are obtained used to represent each position node in the knowledge graph;

[0089] ② attribute node:

[0090] The attribute nodes are distributed in the entire region of the X-ray image, and the initialization is performed by applying spatial attention on f I , and the distinguishing of the attribute nodes is completed;

[0091] By applying a convolution layer with a specified filter size and a SoftMax function in the spatial dimension, an attention map is generated for each attribute, and the initialization of the attribute nodes is completed by weighting f I through the attention map, and the formula is as follows:

[0092]

[0093]

[0094] wherein, is the attention map, Conv2d(·) is a two-dimensional convolution operation with a filter size of 1x1, the input channel number of Conv2d(·) is d, and the output channel number is n a , n a is the number of attribute nodes, is the attribute feature, is the visual feature described above;

[0095] Each embedded attribute node is classified by using a binary classifier, and the existence of the corresponding attribute (local discovery) in the entire image is predicted;

[0096] 2) node relationship modeling:

[0097] R-GCN is applied for feature aggregation processing, and the relationship between the positions and the attributes is modeled, specifically including:

[0098] R-GCN distinguishes the importance of different adjacent nodes by constructing independent message propagation parameters for each relationship type, and the message propagation process of R-GCN is described by the following formula:

[0099]

[0100] wherein, is the hidden state of node i at the l+1 layer, is a learnable parameter, is the hidden state of node i at the l layer, R is the relationship set, N r (i) is the neighbor of node i under the relationship r∈R, c r is a normalization parameter, is a learnable parameter based on the relationship r∈R, is the hidden state of node j at the l layer;

[0101] The final embedding of the node is obtained by inputting the initialized feature of the node into two layers of R-GCN connected in series and and are the hidden representations of the position node and the attribute node fused with the knowledge structure, respectively. The final obtained embedding is used to calculate the score of each attribute node at each position to determine whether the local discovery corresponding to the attribute node appears at this position;

[0102] S3. Construct a local classifier based on the data set obtained in step S1, and perform local classification processing on the knowledge graph processed in step S2; specifically including:

[0103] The Faster R-CNN target detection network and the R-GCN graph neural network described in step S2 are used to construct a local classifier. Given an input chest X-ray, the normalized score of each attribute-position pair is predicted by the local classifier, and the relationship between the attribute and the position is determined by the score. The normalized score represents the possibility of the occurrence of a certain attribute at a certain position, and the specific result is determined by setting the threshold value of the score. The threshold value of the multi-label classification is obtained by maximizing the AUC index of the lesion detection, so as to realize the classification of multiple targets and multiple labels on the image;

[0104] The score of each attribute-position pair is calculated by the dot product between the two node embeddings, and the formula is as follows:

[0105]

[0106] wherein, p l is the detection score, and are the hidden representations of the position node and attribute node fused with the knowledge structure respectively, and is the dot product operation;

[0107] The loss function is binary cross-entropy, which is used to optimize the parameters in the model during the model training process, and the loss function is expressed by the following formula:

[0108]

[0109] wherein, L local is the loss value of local lesion classification, y i,j is the true label, represents the probability of the jth attribute appearing at the ith position;

[0110] Two auxiliary tasks are introduced to improve the performance of the local classifier, which specifically includes:

[0111] Auxiliary task one:

[0112] Through auxiliary task one, the position detection is performed, the region proposal network (RPN) of Faster R-CNN is used to detect each position, and the real boundary box is used as the training target of the network, so that the model can accurately detect each anatomical position, and L rcnn represents the loss of auxiliary task one;

[0113] Auxiliary task two:

[0114] Through auxiliary task two, the attribute classification is performed, and the full connection layer and the Sigmoid function constitute a classifier, auxiliary task two enables the attribute node features generated by the model to focus on the features of the corresponding discovery area on the image, and L attr represents the loss of auxiliary task two;

[0115] The total loss term of local classification training is expressed by the following formula:

[0116] L total = L rcnn + L attr + L local

[0117] All parameters of the local classifier are updated through the loss function L total , including the parameters of Faster R-CNN, AttributeAttention and R-CNN, so as to optimize the detection accuracy of anatomical position and the classification accuracy of local position;

[0118] The ROI features are generated by the real boundary box during training, and are generated by the model during testing;

[0119] S4. Construct a global classifier based on the dataset obtained in step S1 and the local classifier constructed in step S3, and perform global classification processing on the knowledge graph processed in step S2; specifically including:

[0120] Multi-label classification is realized through the global classifier, and different label sets are corresponded according to the differences of the dataset; the local classifier described in step S3 is adopted, the knowledge-enhanced local features are extracted based on the trained local classifier, and the global-local attention is applied to fuse the global and local features;

[0121] The Chest ImaGenome dataset is used to train the local classifier, the parameters of the trained classifier are fixed as the knowledge enhancement module in the global classifier, and the ChestX-ray14 and VinBigData datasets are used for training;

[0122] The set position features are used attribute features and detection scores The sigmoid function is applied on p l to obtain the probability of each position-attribute pair, the attribute features are weighted according to the probability, and are fused with the position features, and the fusion process is described by the following formula:

[0123]

[0124] wherein, is the knowledge-enhanced local feature or KE feature of each position, f o,i is the feature of the i-th position node, f a,j is the feature of the j-th attribute node, is the score between the i-th position node and the j-th attribute node, and σ(·) represents the sigmoid function.

[0125] Global-local attention is designed to fuse multi-granularity features, CNN is applied to extract global features f g ∈R 1×d , f g is used as a query, f ke is used as a key and a value, the KE feature is processed through the attention mechanism, and then is fused with the global feature, this operation fuses the information of each attribute (local discovery) that may appear on each position into the feature of the position node with different importance according to the score of the position-attribute pair, and is described by the following formula:

[0126]

[0127] wherein, f final ∈R 1×dis the final feature of multi-label classification, d is the feature dimension, is the transpose of knowledge enhanced local feature;

[0128] An application of a fully connected layer is used to classify the final feature, and the formula used is as follows:

[0129] p g =f final ·W g +b

[0130] Where p g is the possibility of each classification result appearing, as the result of global classification; W g is the weight term of the classifier, and b is the bias term of the classifier;

[0131] The loss function is binary cross entropy, as described by the following formula:

[0132]

[0133] Where L is the binary cross entropy loss, c is the number of classes, y i is the label of the i-th class, is the predicted probability of the i-th lesion.

[0134] The experimental results of local classification obtained by training the Chest ImaGenome dataset are shown in Table 1;

[0135] The results show that, under the condition of similar anatomical position detection accuracy AP, the method significantly improves the AUC of local classification, and the improvement of AUC compared with two baselines is 11.2% and 8.9%, respectively;

[0136] Table 1

[0137]

[0138] The experimental results of global classification obtained by training the ChestX-ray14 dataset and the VinBigData dataset are shown in Table 2;

[0139] Table 2

[0140]

[0141]

[0142] The results show that the method achieves the best results on both datasets with different sizes and label sets, showing excellent performance. In particular, our method shows significant improvement in detecting small area lesions such as Nodule (nodule) and Nodule-Mass (nodule-mass) and rare lesions such as Fibrosis (fibrosis), Hernia (hernia) and Calcification (calcification). In addition, KEXNet shows competitive or even better performance in detecting some common lesions compared to the state-of-the-art methods, such as Effusion (effusion), Pneumonia (pneumonia) and Lung Opacity (lung opacity). Consistent and stable improvement demonstrates the generalization ability and robustness of the method.

[0143] An ablation study was conducted for the above two tasks (chest X-ray local lesion classification and global lesion classification) to verify the effectiveness of each module (see description below) included in the method of the present application, and the comparative results are shown in Table 3;

[0144] Table 3

[0145]

[0146] For local classification, ablation was performed on knowledge graph initialization and feature aggregation. In the -ATT-INIT setting, the proposed attribute attention was deleted and the attribute nodes were randomly initialized; in the -R-GCN setting, the multi-relation feature aggregation was deleted; in the -Relation setting, the original GCN was used to avoid the message propagation between nodes being distinguished by different relations; it can be observed that each setting has an impact on the attributes; attribute attention plays a role in prompting the model to generate embeddings related to the classification results, and R-GCN and multi-relation can model different constraints between nodes, which collectively guide the model to make correct classification predictions at specific locations;

[0147] For global classification, ablation was performed on the knowledge enhancement module and global-local attention. In the -KEFeature setting, KE features were replaced by ROI features; in the -Fusion setting, KE features were averaged pooled and directly concatenated with global features; in the -KE Feature&Fusion setting, ROI features were averaged pooled and directly concatenated with global features; it can be observed that both modules contribute to improving the performance of the model; KE features contain rich anatomical information, while global-local attention enables the model to fuse multi-granularity features to achieve complementary benefits;

[0148] Figure 3Two test cases predicted using the global classification detector are shown, as well as the visualization of the location and attribute nodes corresponding to the local classification results; in the first case, the local lesion detector identifies Atelectasis lung atelectasis in the right lower lung area, and the corresponding visualization allows the localization of the lesion area, providing the basis for the existence of Atelectasis lung atelectasis on the entire image; in the second case, the local classification detector identifies Enlarged Cardiac Silhouette enlarged cardiac silhouette in the Cardiac Silhouette heart contour, which can be used as evidence of the existence of Cardiomegaly cardiac enlargement;

[0149] The above analysis shows that the local classifier of the method of the present application can locate the abnormal findings in each region, providing the basis for the final diagnosis of the entire X-ray, ensuring the explainability of the diagnosis result, which is very important in medical imaging analysis.

[0150] Figure 4 The imaging method provided by the present application includes the chest X-ray image classification method based on knowledge enhancement, and comprises the following steps:

[0151] S1. Obtain a reference data set and construct a knowledge graph;

[0152] S2. Perform node type differentiation and node relationship modeling processing on the knowledge graph constructed in step S1;

[0153] S3. Construct a local classifier based on the data set obtained in step S1, and perform local classification processing on the knowledge graph processed in step S2;

[0154] S4. Construct a global classifier based on the data set obtained in step S1 and the local classifier constructed in step S3, and perform global classification processing on the knowledge graph processed in step S2;

[0155] S5. Use the global classifier trained in step S4 to complete the classification processing of the actual input X-ray image;

[0156] S6. Based on the classification result obtained in step S5, perform secondary imaging processing on the actual X-ray image by the CT machine to obtain an X-ray image with classification result marking.

[0157] The imaging method provided by the present application is suitable for existing chest X-ray image acquisition devices, such as existing X-ray machine devices. In a specific application, the imaging method of the present application is applied to the X-ray machine device. In operation, the X-ray machine device first acquires an actual chest X-ray image in a normal working mode, and then performs secondary imaging on the acquired chest X-ray image by using the imaging method provided by the present application and applied to the device, to obtain an X-ray image corresponding to the chest X-ray image and having a classification result mark. Then, the X-ray machine device can output the chest X-ray image and the X-ray image having the classification result mark at one time, thereby greatly facilitating the staff and the detected person.

Claims

1. A knowledge enhancement-based chest X-ray image classification method, comprising the following steps: S1. Obtain a reference dataset and construct a knowledge graph; S2. Perform node type differentiation and node relationship modeling processing on the knowledge graph constructed in step S1; S3. Construct a local classifier based on the dataset obtained in step S1, and perform local classification processing on the knowledge graph processed in step S2; S4. Based on the dataset obtained in step S1 and the local classifier constructed in step S3, construct a global classifier, and perform global classification processing on the knowledge graph processed in step S2; Specifically comprising: Implementing multi-label classification through the global classifier, while corresponding to different label sets according to the differences in the dataset; using the local classifier described in step S3, extracting knowledge-enhanced local features based on the trained local classifier, and applying global-local attention to fuse global and local features; Using the Chest ImaGenome dataset to train the local classifier, fixing the parameters of the trained classifier as the knowledge enhancement module in the global classifier, and training on the ChestX-ray14 and VinBigData datasets; Adopting the set position feature Attribute feature And detection score On the p l The sigmoid function is applied to obtain the probability of each position-attribute pair, the attribute feature is weighted according to the probability, and the position feature is fused, and the fusion process is described by the following formula; wherein, is the knowledge-enhanced local feature or KE feature for each location, f o,i is the feature for the i-th location node, f a,j is the feature for the j-th attribute node, is the score between the i-th location node and the j-th attribute node, σ(·) denotes the sigmoid function; n o is the number of locations; n a is the number of attribute nodes; A global-local attention fusion multi-granularity feature is designed, and a CNN is applied to extract a global feature f g ∈R 1×d , f g is used as a query, f ke is used as a key and a value, and KE features are processed through an attention mechanism, and then fused with the global features, and the following formula is used to describe that according to the score of the position-attribute pair, the information of each attribute that may appear on each position is fused into the feature of the position node with different importance: wherein f final ∈R 1×d is the final feature of multi-label classification, d is the feature dimension, is the transpose of knowledge enhanced local features; Applying a fully connected layer to classify the final features, using the following formula: p g = f final • W g + b where p g is the likelihood of each classification outcome as a result of the global classification; W g is the weight term of the classifier, and b is the bias term of the classifier. The loss function is binary cross-entropy, described as follows: where L is binary cross-entropy loss, c is the number of classes, y i is the label of the i-th class, is the predicted probability of the i-th lesion. S5. Using the global classifier trained in step S4, complete the classification processing of the actual input X-ray image. 2.The knowledge augmentation based chest X-ray image classification method of claim 1, wherein The reference dataset obtained in step S1 and the construction of the knowledge graph are specifically as follows: The obtained reference dataset includes the Chest ImaGenome dataset, the ChestX-ray14 dataset, and the VinBigData dataset; the knowledge graph KG is constructed based on the Chest ImaGenome dataset, specifically including: In each chest X-ray image in the Chest ImaGenome dataset, the anatomical position with a bounding box is labeled, the "position" is selected, and the local findings found in the "position" are labeled; Processing the chest X-ray images in the dataset, deleting the positions that do not have any local findings, retaining the set local findings, referring to the "anatomical position" as "position" and the "local findings" as "attribute", to form the nodes in the knowledge graph KG; The Chest ImaGenome dataset provides the relationship between the position and the attribute; each node is connected through the "relationship" to form the "edge" in the knowledge graph KG. 3.The knowledge augmentation based chest X-ray image classification method of claim 2, wherein The node type differentiation and node relationship modeling processing on the knowledge graph constructed in step S1 in step S2 are specifically as follows: 1) Node type differentiation: Different initialization methods are used to differentiate the types of nodes: ① Position node: Using the ROI features of the anatomical position for initialization, making the features of the nodes based on the anatomical position, and then differentiating the position nodes, specifically including: The frontal X-ray image of the Chest ImaGenome dataset is taken as the input of the CNN, and the image features f are extracted by the Faster R-CNN target detection algorithm I ∈R d×H×W , a bounding box or a region of interest of the anatomical position in the input X-ray is detected wherein d represents the number of channels of the visual features, H represents the height of the visual features, W represents the width of the visual features, n o is the number of positions, and R represents the dimension of the feature vector; by applying ROI pooling on f I and B, the ROI features of each position are obtained for representing each position node in the knowledge graph ② Attribute node: The attribute nodes are distributed in the whole region of the X-ray image, and the f I The application space attention is applied to realize initialization and complete the differentiation of the attribute nodes; By applying a convolution layer with a set filter size and a SoftMax function in the spatial dimension, an attention map is generated for each attribute, and the f I The initialization of the weighted completion attribute node is completed, and the formula is as follows: wherein, is an attention map, Conv2d(·) is a two-dimensional convolution operation with a filter size of 1x1, the input channel number of Conv2d(·) is d, and the output channel number is n a , n a is the number of attribute nodes, is an attribute feature, is the visual feature described above; Using a binary classifier to classify each attribute node embedded, to predict the existence of the corresponding attribute in the entire image; 2) Node relationship modeling: The R-GCN is applied to feature aggregation processing, and is used for modeling the relationship between the positions and the attributes, and specifically includes the following steps. The R-GCN distinguishes the importance of different adjacent nodes by constructing independent message propagation parameters for each relationship type, and uses the following formula to describe the message propagation process of the R-GCN: wherein, is the hidden state of node i at layer l + 1, are learnable parameters, is the hidden state of node i at layer l, R is a set of relations, N r (i) is a neighbor of node i under relation r e R, c r is a normalization parameter, are learnable parameters based on relation r e R, is the hidden state of node j at layer l; The final embedding is obtained by inputting the initialized features of the nodes into two layers of R-GCN connected in series and and respectively, are the hidden representations of the position nodes and attribute nodes fused with the knowledge structure; the final obtained embedding is used to calculate the score of each attribute node at each position to determine whether the local discovery corresponding to the attribute node appears at this position.

4. The knowledge augmentation based chest X-ray image classification method of claim 3, wherein The local classifier is constructed based on the data set obtained in step S1, and the local classification processing is performed on the knowledge graph processed in step S2, and specifically includes the following steps. The local classifier is constructed by using the Faster R-CNN target detection network and the R-GCN graph neural network described in step S2, a given input chest X-ray is predicted by the local classifier, a normalized score of each attribute-position pair is predicted, and the relationship between the attribute and the position is determined by the score; the normalized score represents the possibility of the occurrence of a certain attribute in a certain position, and the specific result is determined by setting a threshold value of the score; the threshold value of the multi-label classification is obtained by maximizing the AUC index of the lesion detection, and the multi-target multi-label classification on the image is realized; The score of each attribute-position pair is calculated by the dot product between the two node embeddings, and the formula is as follows: where p l is the detection score, and are the hidden representations of the position node and attribute node fused with the knowledge structure, respectively, and is the dot product operation. The loss function is a binary cross entropy, which is used for optimizing the parameters in the model during the model training process, and the loss function is represented by the following formula: where L local is the loss value for local lesion classification, y i,j is the true label, denotes the probability of the jth attribute occurring at the ith position; Two auxiliary tasks are introduced to improve the performance of the local classifier.

5. The knowledge augmentation based chest X-ray image classification method according to claim 4, characterized in that The two auxiliary tasks included in step S3 specifically include the following steps. Auxiliary task one: Through auxiliary task one, the position detection, the region proposal network RPN of Faster R-CNN is used to detect each position, and the real boundary box is taken as the training target of the network, so that the model can accurately detect each anatomical position, L rcnn represents the loss of auxiliary task one; Auxiliary task two: By performing attribute classification through the second auxiliary task, the full-connection layer and the Sigmoid function constitute a classifier, the second auxiliary task enables the attribute node features generated by the model to focus on the features of the corresponding discovery area on the image, and L attr represents the loss of the second auxiliary task; The total loss term of the local classification training is represented by the following formula: L total = L rcnn + L attr + L local by a loss function L total All parameters of the local classifier are updated, including the parameters of Faster R-CNN, AttributeAttention, and R-CNN, while optimizing the accuracy of anatomical location detection and local classification accuracy.

6. An imaging method comprising the knowledge-enhanced based chest X-ray image classification method according to one of claims 1 to 5, characterized in that The method further includes the following steps: S6. Based on the classification result obtained in step S5, secondary imaging processing is performed on the actual X-ray image to obtain an X-ray image with a classification result mark.

Images