Method for realizing classification recognition of x-ray film image based on double-path decoder

Abstract

The application relates to a method for classifying and recognizing X-ray film images based on a Transformer two-way decoder, wherein the method comprises the following steps: (1) collecting a chest X-ray film data set and performing a pretreatment operation on the X-ray film images; (2) constructing a feature encoder module and a label encoder module to extract relevant information of the collected data; (3) constructing a two-way decoder module to decode and process the extracted relevant information; (4) constructing a classification attention module to perform classification and recognition processing; and (5) using an asymmetric loss function to train the network for balanced optimization. The application also relates to a corresponding system, device, processor and storage medium thereof. The method, system, device, processor and storage medium for classifying and recognizing X-ray film images based on the Transformer two-way decoder have the advantages that common features are extracted from image features and label embedding graphs through a novel cross-attention mechanism, and the X-ray classification and recognition effect is better than that of a baseline model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of digital images, in particular to the technical field of computer vision, and more particularly to a method, system and device for classifying and recognizing X-ray images based on a Transformer dual-decoder, a processor and a computer readable storage medium thereof. BACKGROUND

[0002] At present, chest X-ray (CXR) is a kind of efficient and inexpensive imaging examination technology, which is often used for target recognition and screening. With the continuous development of deep convolutional networks, researchers can mine useful information from large-scale medical data, and various CNN-based methods are applied to the image recognition of chest X-ray. However, most of the existing CNN-based methods still have some defects. For example, the patent application with the application number CN202210171528.X discloses adding FCAnet multi-spectrum channel attention mechanism on the basis of backbone to enhance the feature extraction ability of backbone, and the design focus is on the FCAnet multi-spectrum channel attention mechanism; the patent application with the application number CN202111510584.3 uses X-ray information and personal experience data for joint training and recognition, and focuses on data preprocessing, and does not have an advantage in network model design; the patent application with the application number CN202010157966.1 designs a dual-channel separation network for chest X-ray, two channels are used to extract shallow and deep information respectively, and finally the extracted features are fused for final classification. The defects of the above disclosed applications are that they directly use the feature maps extracted from the CXR images to recognize and classify the chest images, but do not consider the correlation between the local features and the corresponding labels, which should be the defect problem to be solved urgently. SUMMARY

[0003] The present application aims to overcome the above-mentioned defects of the prior art, and provides a method, system, device, processor and computer readable storage medium thereof for classifying and recognizing X-ray images based on a Transformer dual-decoder, which can effectively consider the correlation between local features and corresponding labels.

[0004] In order to achieve the above-mentioned purpose, the method, system, device, processor and computer readable storage medium thereof for classifying and recognizing X-ray images based on a Transformer dual-decoder are as follows:

[0005] The method for classifying and recognizing X-ray image based on the two-way decoder of the Transformer mainly comprises the following steps:

[0006] (1) Collecting chest X-ray data set and pre-processing X-ray image;

[0007] (2) Constructing feature encoder module and label encoder module to extract relevant information from the collected data;

[0008] (3) Constructing a two-way decoder module to decode the extracted relevant information;

[0009] (4) Constructing a classification attention module to classify and recognize the decoding information output by the two-way decoder module to obtain the recognition result of the target object;

[0010] (5) Using an asymmetric loss function to train the network to balance and optimize the relevant data with unbalanced category distribution in the collected data set.

[0011] Preferably, the step (1) specifically comprises the following steps:

[0012] (1.1) Downloading CC-CXRI data set from the official website of the data set to obtain original chest X-ray image with size of 1024x1024pt;

[0013] (1.2) Scaling the original chest X-ray image to 384x384pt and processing the scaled image by using data enhancement method to obtain the final training and testing image.

[0014] Preferably, the data enhancement method comprises:

[0015] The scaled image is sequentially rotated randomly [-15°, 15°], then randomly horizontally flipped, and normalized by using mean value [0.485, 0.456, 0.406] and variance [0.229, 0.224, 0.225] to obtain the final training and testing image.

[0016] Especially, the step (2) specifically comprises the following steps:

[0017] (2.1) The feature encoder module is constructed in the following way:

[0018] Using CNN model or Transformer-based model as chest X-ray feature extraction network, and then converting the extracted 3D feature map dimension to 2D feature map where h and w represent the length and width of the feature map respectively, and d represents the dimension of the feature map;

[0019] (2.2) The label encoder module is constructed in the following way:

[0020] A learnable label embedding graph is constructed as a label encoder module, where N represents the number of classes, and d represents the dimension, and the label encoder module is consistent with the dimension of the feature encoder module;

[0021] (2.3) Based on the constructed feature encoder module and label encoder module, relevant features and labels of the collected chest X-ray are extracted.

[0022] Especially, the two-way decoder module specifically includes a label decoder module and a feature decoder module, and the step (3) includes the following steps:

[0023] (3.1) The label decoder module is constructed: using multi-head self-attention mechanism MHSA, multi-head cross-attention mechanism MHCA and a multi-layer MLP to construct the label decoder, and after each operation is performed, layer normalization and residual connection are used to further adjust the output, to obtain a label decoder output result;

[0024] (3.2) The feature decoder module is constructed: using multi-head self-attention mechanism MHSA, multi-head cross-attention mechanism MHCA and a multi-layer MLP to construct the feature decoder, and after each operation is performed, normalization and residual connection are used to further adjust the output, to obtain a feature decoder output result.

[0025] More preferably, the step (3.1) specifically includes the following steps:

[0026] (3.1.1) Label embedding graph First, a linear mapping layer is passed through to generate a query matrix a Key matrix and a Value matrix Then the three matrices are input into the multi-head self-attention mechanism MHSA, as shown in the following formula:

[0027]

[0028] Where T is the matrix transpose operation.

[0029] (3.1.2) Then the 2D feature map After position encoding, a linear mapping layer is passed through to generate a query matrix a Key matrix and a Value matrix the output value M of the multi-head self-attention mechanism MHSA and the matrix into the multi-head cross-attention mechanism MHCA, as shown in the following formula:

[0030]

[0031] (3.1.3) Finally, the output of the label decoder is obtained by processing through a multi-layer MLP

[0032] More preferably, the step (3.2) specifically comprises the following steps:

[0033] (3.2.1) Label embedding graph First, a linear mapping layer is passed through to generate a query matrix a Key matrix and a Value matrix Then, the three matrices are input into the multi-head self-attention mechanism MHSA, as shown in the following formula:

[0034]

[0035] Wherein, T is the matrix transposition operation.

[0036] (3.2.2) Then, the 2D feature map After the position encoding, a linear mapping layer is passed through to generate a query matrix a Key matrix and a Value matrix the output value M of the multi-head self-attention mechanism MHSA The Key matrix K and the Value matrix V of the label embedding graph are input into the multi-head cross-attention mechanism MHCA, as shown in the following formula:

[0037]

[0038] (3.2.3) Finally, the output of the feature decoder is obtained by processing through a multi-layer MLP

[0039] More preferably, the step (4) specifically comprises the following steps:

[0040] (4.1) The output of the label decoder is processed in the following manner

[0041] The output of the label decoder Taking the average of the rows, we get Then use a fully connected network to Dimension d is mapped to the number of categories c, resulting in

[0042] in, For L o The result of averaging by row for The result after mapping.

[0043] (4.2) The output of the feature decoder is processed in the following manner.

[0044] The output of the feature decoder Take the maximum value from each row to get Then, using a fully connected network, we map dimension d to the number of categories c, and obtain...

[0045] in, For F o The result of averaging by row for The result after mapping.

[0046] (4.3) Introduce a balancing parameter la to adjust the output of the label decoder. and the output of the feature decoder A balancing process is performed to obtain the final X-ray image recognition result S. B Specifically, it is expressed using the following formula:

[0047]

[0048] More preferably, step (5) specifically includes:

[0049] Each collected sample is labeled as Y = {y1, y2, ... y}. N When the collected data contains the labeled data, then y will be... N Set it to 1 otherwise, and set it to 0; and set the predicted value of the recognition model to P = {p1, p2, ... p...} N}, γ + and γ - The hyperparameters of the model are set to 0 and 4 respectively, and the following formula is used for balancing optimization:

[0050]

[0051] wherein ASL is an asymmetric loss function Asymmetric Loss.

[0052] The system for classifying and recognizing X-ray image based on the above-mentioned method of the two-path decoder of the Transformer mainly comprises:

[0053] a feature and label encoder module for extracting feature data and label data from the relevant data information in the collected chest X-ray dataset;

[0054] a two-path decoder module connected to the feature and label encoder module, for constructing a label decoder and a feature decoder respectively by using a multi-head self-attention mechanism MHSA, a multi-head cross-attention mechanism MHCA and a multi-layer MLP, and decoding the extracted data by using the label decoder and the feature decoder; and

[0055] a classification attention module connected to the two-path decoder module, for balancing the output values of the label decoder and the feature decoder by introducing a balance parameter, to obtain the final X-ray image recognition result.

[0056] The device for classifying and recognizing X-ray image based on the two-path decoder of the Transformer mainly comprises:

[0057] a processor configured to execute computer executable instructions;

[0058] a memory storing one or more computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned method for classifying and recognizing X-ray image based on the two-path decoder of the Transformer.

[0059] The processor for classifying and recognizing X-ray image based on the two-path decoder of the Transformer mainly comprises: the processor is configured to execute computer executable instructions, and the computer executable instructions, when executed by the processor, implement the steps of the above-mentioned method for classifying and recognizing X-ray image based on the two-path decoder of the Transformer.

[0060] The computer readable storage medium mainly comprises: a computer program is stored thereon, and the computer program can be executed by the processor to implement the steps of the above-mentioned method for classifying and recognizing X-ray image based on the two-path decoder of the Transformer.

[0061] The method, system, device, processor and computer readable storage medium of the application based on the Transformer dual-path decoder for classifying and identifying X-ray images use a classic CNN model (such as ResNet101 and DenseNet121) or the latest Transformer-based model (such as Swin-B and PVT) as a feature encoder module of a chest X-ray film. In order to strengthen the connection between the target area and the identification label, the application also innovatively introduces a learnable label embedding graph as a label encoder. The dual-path decoding module is composed of a label decoder and a feature decoder, which extracts common features from image features and label embedding graphs through a novel cross-attention mechanism. The dual-path can effectively optimize the visual features and match the corresponding labels to improve the performance of accurate identification. The classification attention module is used to balance the output of the feature decoder and the label decoder to achieve better identification effect. The technical solution is verified by experiments on the CC-CXRI dataset, and has more outstanding classification and identification effect compared with the baseline model. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 It is a schematic diagram of the overall structure of the application.

[0063] Figure 2 It is a schematic diagram of the classification attention module structure of the application.

[0064] Figure 3 It is a visualization result diagram of the lesion area positioning of the TransDD-PVT model and its baseline model PVTv2-B4 in a specific embodiment of the application. DETAILED DESCRIPTION

[0065] In order to more clearly describe the technical content of the application, the following further describes in combination with specific embodiments.

[0066] Before explaining the embodiments of the application in detail, it should be noted that in the following, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes these elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device.

[0067] Please refer to Figure 1 The method for classifying and identifying X-ray images based on the Transformer dual-path decoder shown in the figure, wherein the method comprises the following steps:

[0068] (1) Collecting chest X-ray film dataset and performing pretreatment operation on X-ray film image;

[0069] (2) constructing a feature encoder module and a label encoder module to extract relevant information from the collected data;

[0070] (3) constructing a two-way decoder module to decode the extracted relevant information;

[0071] (4) constructing a classification attention module to perform classification and recognition processing on the decoding information output by the two-way decoder module, to obtain the recognition result of the target object;

[0072] (5) using an asymmetric loss function to train the network to balance and optimize the relevant data with uneven category distribution in the collected data set.

[0073] As a preferred embodiment of the present application, the step (1) specifically comprises the following steps:

[0074] (1.1) downloading the CC-CXRI data set from the official website of the data set to obtain the original chest X-ray image with a size of 1024x1024pt;

[0075] (1.2) scaling the original chest X-ray image to 384x384pt, and processing the scaled image using data enhancement to obtain the final training and test image.

[0076] In practical application, the above step (1) is specifically:

[0077] Download the CC-CXRI data set from the official website of the data set to obtain the original chest X-ray image with a size of 1024x1024pt. Then, scale the original X-ray image size to 384x384pt, and use data enhancement (including image rotation, image flipping, histogram transformation) to obtain the final training and test image.

[0078] As a preferred embodiment of the present application, the data enhancement method comprises:

[0079] The scaled image is sequentially randomly rotated [-15°, 15°], then randomly horizontally flipped, and normalized with a mean of [0.485, 0.456, 0.406] and a variance of [0.229, 0.224, 0.225] to obtain the final training and test image.

[0080] As a preferred embodiment of the present application, the step (2) specifically comprises the following steps:

[0081] (2.1) constructing the feature encoder module in the following manner:

[0082] using a CNN model or a Transformer-based model as the feature extraction network of the chest X-ray, and then converting the extracted 3D feature map into a 2D feature map into a 2D feature map wherein h and w represent the length and width of the feature map respectively, and d represents the dimension of the feature map;

[0083] (2.2) The label encoder module is constructed in the following manner:

[0084] A learnable label embedding graph is constructed as the label encoder module, wherein N represents the number of categories, d represents the dimension, and the dimension of the label encoder module is consistent with that of the feature encoder module;

[0085] (2.3) Based on the constructed feature encoder module and label encoder module, the relevant features and labels of the collected chest X-ray are extracted.

[0086] In actual application, the above step (2) is specifically:

[0087] Step 2.1: Constructing the feature encoder module:

[0088] The present application can use a classic CNN model (such as ResNet101 and DenseNet121) or the latest Transformer-based model (such as Swin-B and PVT) as the feature extraction network of the chest X-ray, and then convert the extracted 3D feature map into a 2D feature map into a 2D feature map wherein h and w represent the length and width of the feature map respectively, and d represents the dimension of the feature map, which are determined by the original feature extraction network;

[0089] Step 2.2: Constructing the label encoder module:

[0090] In order to better establish the connection between the local lesion features and the corresponding labels, the present application constructs a learnable label embedding graph as the label encoder module, wherein N represents the number of categories, d represents the dimension, and the dimension is consistent with that of the feature encoder.

[0091] As a preferred embodiment of the present application, the dual-path decoder module specifically includes a label decoder module and a feature decoder module, and the step (3) includes the following steps:

[0092] (3.1) constructing the label decoder module: using multi-head self-attention mechanism MHSA, multi-head cross-attention mechanism MHCA and a multi-layer MLP to construct the label decoder, and after each operation is performed, using layer normalization and residual connection to further adjust the output, obtaining a label decoder output result;

[0093] (3.2) constructing the feature decoder module: using multi-head self-attention mechanism MHSA, multi-head cross-attention mechanism MHCA and a multi-layer MLP to construct the feature decoder, and after each operation is performed, using normalization and residual connection to further adjust the output, obtaining a feature decoder output result.

[0094] As a preferred embodiment of the present application, the step (3.1) specifically comprises the following steps:

[0095] (3.1.1) label embedding graph First, a linear mapping layer is used to generate a query matrix A Key matrix And a Value matrix Then the three matrices are input into the multi-head self-attention mechanism MHSA, as shown in the following formula:

[0096]

[0097] Where T is the matrix transpose operation.

[0098] (3.1.2) then 2D feature map After position encoding, a linear mapping layer is used to generate a query matrix A Key matrix And a Value matrix The output value M of the multi-head self-attention mechanism MHSA is input into the multi-head cross-attention mechanism MHCA, as shown in the following formula:

[0099]

[0100] (3.1.3) finally, a multi-layer MLP is used to process and obtain the output of the label decoder

[0101] As a preferred embodiment of the present application, the step (3.2) specifically comprises the following steps:

[0102] (3.2.1) label embedding graph ​​Firstly, a linear mapping layer is used to generate a query matrix a Key matrix and a Value matrix Then, the three matrices are input into the multi-head self-attention mechanism MHSA, as shown in the following formula:

[0103]

[0104] where T is the matrix transposition operation.

[0105] (3.2.2) Then, the 2D feature map Firstly, a linear mapping layer is used to generate a query matrix a Key matrix and a Value matrix Then, the output value of the multi-head self-attention mechanism MHSA The Key matrix K and the Value matrix V of the label embedding graph are input into the multi-head cross-attention mechanism MHCA, as shown in the following formula:

[0106]

[0107] (3.2.3) Finally, a multi-layer MLP is used to process the output of the feature decoder

[0108] In practical applications, the above step (3) is specifically:

[0109] Step 3.1: Label decoder design:

[0110] The label decoder mainly consists of a multi-head self-attention mechanism (MHSA), a multi-head cross-attention mechanism (MHCA), and a multi-layer MLP, and after each operation is performed, layer normalization and residual connection are used to further adjust the output. Label embedding graph Firstly, a linear mapping layer is used to generate a query matrix a Key matrix and a Value matrix Then, the three matrices are input into the multi-head MHSA, as shown in the following formula:

[0111]

[0112] Then, the feature map After position encoding, a linear mapping layer is used to generate a query matrix A Key matrix And a Value matrix The output M matrix of MHSA and the matrix are input into MHCA, and the calculation formula is as follows:

[0113]

[0114] Finally, an output of the label decoder is obtained through a multi-layer MLP

[0115] Step 3.2: Feature decoder design

[0116] The design of the feature decoder is similar to that of the label decoder. It is also composed of three parts of MHSA, MHCA and multi-layer MLP, and the same, after each part is executed, normalization and residual connection operations are followed. The formula of MHSA is as follows:

[0117]

[0118] Then the output of MHSA and K and V generated by the label embedding graph are input into MHCA.

[0119]

[0120] Finally, an output of the feature decoder is obtained through a multi-layer MLP

[0121] As a preferred embodiment of the present application, the step (4) specifically comprises the following steps:

[0122] (4.1) The output of the label decoder is processed in the following manner

[0123] The output of the label decoder is averaged by row to obtain Then a fully connected network is used to map dimension d to the number of categories c to obtain

[0124] wherein, is L o The result of averaging by row, is the result after mapping.

[0125] (4.2) The output of the feature decoder is processed in the following way

[0126] The output of the feature decoder is processed in the following way The maximum value is taken in each row to obtain Then a fully connected network is used to map the dimension d to the number of categories c to obtain

[0127] wherein, is F o The average value is taken in each row to obtain is The result after mapping.

[0128] (4.3) A balance parameter la is introduced to balance the output of the label decoder and the output of the feature decoder to obtain the final X-ray image recognition result S B which is specifically expressed by the following formula:

[0129]

[0130] In actual application, the above step (4) is specifically:

[0131] Step 4.1: Process the output of the label decoder

[0132] After the dual-channel decoder, the output of the label decoder is obtained and the output of the feature decoder The role of the classification attention module is to use the two outputs to obtain the score of each category, thereby completing the classification recognition task. Take The average value is taken in each row to obtain Then a fully connected network is used to map the dimension d to the number of categories c to obtain

[0133] Step 4.2: Process the output of the feature decoder

[0134] Take The maximum value is taken in each row to obtain Similarly, a fully connected network is used to map the dimension d to the number of categories c to obtain

[0135] Step 4.3: Balance the two outputs:

[0136] A balance parameter la is introduced to obtain the final target recognition prediction score through formula 5.

[0137]

[0138] As a preferred embodiment of the present application, the step (5) is specifically:

[0139] In practical application, due to the unbalanced distribution of various categories in the data set, an asymmetric loss function is introduced. Specifically, the label of each collected sample is marked as Y={y1, y2, … y N}, when the collected data contains the marked label, then y N is set to 1, otherwise it is set to 0; and the prediction value of the recognition model is set to P={p1, p2, … p N}, γ + and γ - are the hyperparameters of the model, which are set to 0 and 4 respectively, and the following formula is used for balance optimization:

[0140]

[0141] Where, ASL is the asymmetric loss function.

[0142] The system for classifying and recognizing X-ray images based on the above method of the two-path decoder of the Transformer, wherein the system comprises:

[0143] A feature and label encoder module is used to extract feature data and label data from the relevant data information in the collected chest X-ray data set;

[0144] A two-path decoder module is connected to the feature and label encoder module, which is used to construct a label decoder and a feature decoder by using a multi-head self-attention mechanism MHSA, a multi-head cross-attention mechanism MHCA and a multi-layer MLP, and to decode the extracted data using the label decoder and the feature decoder; and

[0145] A classification attention module is connected to the two-path decoder module, which is used to balance the output values of the label decoder and the feature decoder by introducing a balance parameter, so as to obtain the final X-ray image recognition result.

[0146] In a specific embodiment of the present application, the classification and recognition method of the present technical solution is tested as follows:

[0147] (1) Experimental data set

[0148] The present application uses the chest X-ray dataset published by the Chinese Chest X-ray Image Consortium (CC-CXRI) for experimental verification. The dataset consists of two sub-datasets SYSU and CC-CXRI-P. Among them, SYSU contains 120,012 chest X-ray images, a total of 14 types of chest diseases (consolidation, fibrosis, nodule, hernia, atelectasis, pneumothorax, edema, pneumonia, emphysema, effusion, infiltration, pleural thickening, mass, cardiac enlargement), each picture contains one or more chest diseases. The CC-CXRI-P dataset contains 7,921 chest X-ray images, which are divided into normal, viral pneumonia (COVID-19 coronavirus pneumonia, non-COVID-19 coronavirus pneumonia), and other types of pneumonia. Each picture contains only one type of disease. Both sub-datasets are divided into 80% training set, 10% validation set, and 10% test set.

[0149] (2) Training process

[0150] The training picture is scaled to 384x384pt, and data enhancement methods such as random rotation, flipping, and histogram equalization are used. The initial learning rate is set to le-4, the cosine annealing algorithm is used to decay the learning rate, the batch is set to 32, and the training is 20 rounds.

[0151] (3) Test results

[0152] As shown in Figure 3 , is the visualization result diagram of the lesion area positioning of the TransDD-PVT model and its baseline model PVTv2-B4 proposed in the present application. The first row represents the original X-ray image, the position marked by the black box in the second row is the lesion area, and the third and fourth rows are the results of PVTv2-B4 and TransDD-PVT lesion positioning respectively. The area with significant color change in the diagram is the positioning result of the model. By comparison, it can be seen that the TransDD-PVT model proposed in the present application can better locate the lesion area compared with the baseline model, thereby improving the classification result of the chest X-ray.

[0153] The device for classifying and identifying X-ray images based on the Transformer dual decoding device, wherein the device comprises:

[0154] a processor configured to execute computer executable instructions;

[0155] a memory storing one or more computer executable instructions, which, when executed by the processor, implement the steps of the method for classifying and identifying X-ray images based on the Transformer dual decoding device described above.

[0156] The processor based on the Transformer dual decoding implement classifies and identifies the X-ray image, wherein the processor is configured to execute computer executable instructions, and the computer executable instructions are executed by the processor to implement the steps of the method for classifying and identifying the X-ray image based on the Transformer dual decoding.

[0157] The computer readable storage medium, wherein a computer program is stored thereon, and the computer program can be executed by the processor to implement the steps of the method for classifying and identifying the X-ray image based on the Transformer dual decoding.

[0158] Any process or method descriptions or any other descriptions in flow charts or otherwise described herein can be understood as representing one or more steps of a method or process, and the steps can be performed in the order described, in a different order, or concurrently. The various embodiments of the present application can be implemented in hardware, software, firmware, or any combination thereof.

[0159] It should be understood that various parts of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.

[0160] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiments can be instructed by a program to complete the relevant hardware, and the program can be stored in a computer readable storage medium, and the program includes one or a combination of steps of the method embodiments when executed.

[0161] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0162] In the description of the present specification, the description of the terms "an embodiment", "some embodiments", "an example", "a specific example", or "embodiments" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily mean the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0163] Although the embodiments of the present application have been shown and described above, it is to be understood that the above-described embodiments are exemplary only, and are not to be construed as limiting the present application, and that changes, modifications, substitutions and variations can be made thereto by those skilled in the art without departing from the scope of the present application.

[0164] The method, system, device, processor and computer readable storage medium of the application based on the Transformer dual-path decoder are used for classifying and identifying X-ray images. A classic CNN model (such as ResNet101 and DenseNet121) or the latest Transformer-based model (such as Swin-B and PVT) is used as a feature encoder module of the chest X-ray image. In order to strengthen the connection between the target area and the identification label, the application also innovatively introduces a learnable label embedding graph as a label encoder. The dual-path decoding module is composed of a label decoder and a feature decoder, which extracts common features from image features and label embedding graphs through a novel cross-attention mechanism. The dual-path can effectively optimize the visual features and match the corresponding labels to improve the performance of accurate identification. The classification attention module is used to balance the output of the feature decoder and the label decoder to achieve better identification effect. The technical solution is verified by experiments on the CC-CXRI dataset, and has more outstanding classification and identification effect compared with the baseline model.

[0165] In this specification, the present application has been described with reference to its particular embodiments. It is to be understood that variations and modifications can be affected without departing from the spirit and scope of the present application. Accordingly, the specification and figures are to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A method for classifying and recognizing X-ray images based on a Transformer dual-channel decoder, characterized in that, The method includes the following steps: (1) Collect chest X-ray images and perform preprocessing operations on the X-ray images; (2) Construct a feature encoder module and a label encoder module to extract relevant information from the collected data; (3) Construct a dual-channel decoder module to decode and process the extracted relevant information; (4) Construct a classification attention module to classify and identify the decoded information output by the dual-channel decoder module in order to obtain the identification result of the target object; (5) Use an asymmetric loss function to train the network to balance and optimize the related data with imbalanced class distribution in the collected dataset; Step (1) specifically includes the following steps: (1.1) Download the CC-CXRI dataset from the official dataset website to obtain the original chest X-ray image with a size of 1024×1024pt; (1.2) The original chest X-ray image is scaled up to 384×384pt, and data augmentation is used to process the scaled image to obtain the final training and testing images; Step (2) specifically includes the following steps: (2.1) The feature encoder module is constructed in the following manner: Use a CNN model or a Transformer-based model as the feature extraction network for chest X-rays, and then extract the 3D feature maps. Convert to 2D feature map Where h and w represent the length and width of the feature map, respectively, and d represents the dimension of the feature map; (2.2) The tag encoder module is constructed in the following manner: A learnable tag embedding graph was constructed. As a label encoder module, N represents the number of categories, d represents the dimension, and the dimensions of the label encoder module are consistent with those of the feature encoder module; (2.3) Based on the constructed feature encoder module and tag encoder module, relevant features and tags are extracted from the acquired chest X-ray images.

2. The method for classifying and recognizing X-ray images based on a Transformer dual-channel decoder according to claim 1, characterized in that, The data augmentation methods mentioned include: The scaled images are then randomly rotated [-15°, 15°] sequentially, followed by random horizontal flipping. Data normalization is performed using a mean of [0.485, 0.456, 0.406] and a variance of [0.229, 0.224, 0.225] to obtain the final training and testing images.

3. The method for classifying and recognizing X-ray images based on a Transformer dual-channel decoder according to claim 1, characterized in that, The dual-channel decoder module specifically includes a tag decoder module and a feature decoder module, and step (3) includes the following steps: (3.1) Construct the label decoder module: Use multi-head self-attention mechanism MHSA, multi-head cross-attention mechanism MHCA and a multi-layer MLP to construct the label decoder. After each operation is performed, layer normalization and residual connection are used to further adjust the output to obtain the output result of a label decoder. (3.2) Construct the feature decoder module: Use multi-head self-attention mechanism MHSA, multi-head cross-attention mechanism MHCA and a multi-layer MLP to construct the feature decoder. After each operation is performed, normalization and residual connection are used to further adjust the output to obtain the output result of the feature decoder.

4. The method for classifying and recognizing X-ray images based on a Transformer dual-channel decoder according to claim 3, characterized in that, Step (3.1) specifically includes the following steps: (3.1.1) Tag Embedding Graph First, a linear mapping layer is used to generate a query matrix. A key matrix and a Value matrix Next, the three matrices are fed into the Multi-Head Self-Attention (MHSA) mechanism, as shown in the following formula: Where T represents the matrix transpose operation; (3.1.2) Next, the 2D feature map First, the query matrix is ​​generated by position encoding and then by a linear mapping layer. A key matrix and a Value matrix Then, the output value M of the multi-head self-attention mechanism MHSA, matrix sum matrix The input to the Multi-Head Cross-Attention (MHCA) mechanism is as shown in the following formula: (3.1.3) Finally, the output of the tag decoder is obtained through a multi-layer MLP process.

5. The method for classifying and recognizing X-ray images based on a Transformer dual-channel decoder according to claim 4, characterized in that, Step (3.2) specifically includes the following steps: (3.2.1) Tag Embedding Graph First, a linear mapping layer is used to generate a query matrix. A key matrix and a Value matrix Next, the three matrices are fed into the Multi-Head Self-Attention (MHSA) mechanism, as shown in the following formula: Where T represents the matrix transpose operation; (3.2.2) Next, the 2D feature map First, the query matrix is ​​generated by position encoding and then by a linear mapping layer. A key matrix and a Value matrix Then the output value of the multi-head self-attention mechanism MHSA is... The Key matrix K and Value matrix V of the label embedding graph are passed into the Multi-Head Cross-Attention (MHCA) mechanism, as shown in the following formula: (3.2.3) Finally, the output of the feature decoder is obtained through a multi-layer MLP process.

6. The method for classifying and recognizing X-ray images based on a Transformer dual-channel decoder according to claim 5, characterized in that, Step (4) specifically includes the following steps: (4.1) The output of the tag decoder is processed in the following manner. The output of the label decoder Taking the average of the rows, we get Then use a fully connected network to Dimension d is mapped to the number of categories c, resulting in in, For L o The result of averaging by row for The result after mapping; (4.2) The output of the feature decoder is processed in the following manner. The output of the feature decoder Taking the maximum value from each row, we get... Then, using a fully connected network, we map dimension d to the number of categories c, and obtain... in, For F o The result of averaging by row for The result after mapping; (4.3) Introduce a balancing parameter la to adjust the output of the label decoder. and the output of the feature decoder A balancing process is performed to obtain the final X-ray image recognition result S. B Specifically, it is expressed using the following formula:

7. The method for classifying and recognizing X-ray images based on a Transformer dual-channel decoder according to claim 6, characterized in that, Step (5) specifically refers to: Each collected sample is labeled as Y = {y1, y2, ... y}. B When the collected data contains the labeled data, then y will be... N Set it to 1 otherwise, and set it to 0; and set the predicted value of the recognition model to P = {p1, p2, ... p...} N }, γ + and γ - The hyperparameters of the model are set to 0 and 4 respectively, and the following formula is used for balancing optimization: ASL is an asymmetric loss function.

8. A system for classifying and recognizing X-ray images based on a Transformer dual-channel decoder using the method of any one of claims 1 to 7, characterized in that, The system includes: The feature and label encoder module is used to extract feature data and label data from relevant data information in the acquired chest X-ray dataset; A dual-path decoder module, connected to the feature and label encoder module, is used to construct a label decoder and a feature decoder respectively using a multi-head self-attention mechanism (MHSA), a multi-head cross-attention mechanism (MHCA), and a multi-layer MLP, and to decode the extracted data using the label decoder and feature decoder; and The classification attention module, connected to the dual-path decoder module, is used to balance the output values ​​of the label decoder and the feature decoder by introducing a balancing parameter to obtain the final X-ray image recognition result.

9. A device for classifying and recognizing X-ray images based on a Transformer dual-channel decoder, characterized in that, The device includes: A processor is configured to execute computer-executable instructions; The memory stores one or more computer-executable instructions, which, when executed by the processor, implement the steps of the method for classifying and recognizing X-ray images based on a Transformer dual-channel decoder as described in any one of claims 1 to 7.

10. A processor for classifying and recognizing X-ray images based on a Transformer dual-channel decoder, characterized in that, The processor is configured to execute computer-executable instructions, which, when executed by the processor, implement the steps of the method for classifying and recognizing X-ray images based on a Transformer dual-channel decoder as described in any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that, It stores a computer program that can be executed by a processor to implement the steps of the method for classifying and recognizing X-ray images based on a Transformer dual-channel decoder as described in any one of claims 1 to 7.

Images