Chest radiograph multi-label classification method and system based on cross-modal memory network

Abstract

The invention provides a chest radiography multi-label classification method and system based on a cross-modal memory network, and the method comprises the steps: obtaining chest radiography data, and carrying out the preprocessing of the chest radiography data; the chest radiograph data comprises an original chest radiograph image and corresponding label information; escaping the label information to map the label information to a uniquely corresponding text identifier; training a chest radiograph classification model established based on a cross-modal memory network (CMN) by using the preprocessed chest radiograph image and the transferred label information; and realizing a multi-label classification task of the chest radiograph by utilizing the chest radiograph classification model. According to the chest radiograph multi-label classification method and system, key features in chest radiograph images can be effectively captured and fused, effective alignment and fusion between medical terms and labels are achieved at the same time, and therefore the accuracy and interpretability of a classification model are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a chest radiograph multi-label classification method and system based on a cross-modal memory network. BACKGROUND

[0002] In the field of medical image analysis, chest radiography is an important means for diagnosing chest diseases, and the accuracy of its interpretation and diagnosis is directly related to the patient's treatment plan and prognosis evaluation. However, the traditional manual reading method not only consumes time and effort, but also the diagnostic results are easily affected by the doctor's experience and subjective judgment, making it difficult to guarantee the accuracy and consistency of the diagnostic results.

[0003] In recent years, with the rapid development of artificial intelligence technology, especially the application of deep learning technology in the field of medical image analysis, it has provided new possibilities for the automatic classification and diagnosis of chest radiography. However, the current chest radiography classification model still faces some challenges. The most critical problem is how to effectively realize the alignment and fusion of multi-modal information. The chest radiography classification model needs to process a large number of medical terms and labels, which have complex correlations and dependencies with the information in the chest radiograph images. Existing classification models often have difficulty in effectively understanding and utilizing the correlations between such multi-modal information, resulting in insufficient accuracy and explainability of the classification results. Therefore, designing a deep learning model that can efficiently realize the alignment and fusion of multi-modal information has become the key to improving the performance and accuracy of the chest radiography classification model. SUMMARY

[0004] To solve the above problems, the purpose of the present application is to provide a chest radiograph multi-label classification method and system based on a cross-modal memory network, which solves two key problems in the current chest radiography classification model: 1) the incompleteness and insufficient accuracy of image feature extraction, especially the capture of key features of lesions; 2) the difficulty of alignment and fusion between multi-modal information (including image information and medical terms, labels). By constructing an efficient cross-modal memory mechanism, the model aims to realize the deep integration of image features and label information, improve the accuracy and explainability of classification, and provide more accurate and reliable decision support for medical image analysis, auxiliary diagnosis, and health monitoring.

[0005] The present application provides a chest radiograph multi-label classification method based on a cross-modal memory network, which comprises: Obtaining chest radiograph data and performing preprocessing operations on the chest radiograph data; the chest radiograph data includes original chest radiograph images and corresponding label information; the original chest radiograph images in the preprocessed chest radiograph data are used as input data for model training, and the corresponding label information is used for loss calculation in the supervised learning process; escaping the label information to map the label information to unique corresponding text identifiers; training a chest radiograph classification model established based on a cross-modal memory network CMN using the preprocessed chest radiograph images and the escaped label information; implementing a multi-label classification task on chest radiographs using the chest radiograph classification model.

[0006] Optionally, the escaping of the label information comprises: by an escaping function mapping original classification labels to a structured text semantic space S.

[0007] The core of the escaping function T is realized by a predefined text template The basic formal definition is:

[0008] wherein is a label corresponding standard English medical term, denotes the standard form of the first letter of the term in capital letters, denotes the all-lowercase form of the term.

[0009] Optionally, training a chest radiograph classification model established using the preprocessed chest radiograph images and the escaped label information comprises: extracting features of the preprocessed chest radiograph images and the escaped label information using ResNet101 and BERT networks respectively and storing them into a CMN module; extracting visual features of the chest radiograph images by a ResNet-101 network to capture key visual patterns; at the same time, the corresponding escaped labels are encoded into semantic-rich text feature vectors by a BERT model; the visual features and the text feature vectors are input into a cross-modal memory network CMN, which dynamically establishes deep associations between the visual features and the semantic labels through its memory units and attention mechanisms; the fusion features output by the cross-modal memory network CMN are used as inputs of a multi-label classifier on the one hand and as initial states of a Transformer decoder on the other hand to generate coherent diagnostic report texts in an autoregressive manner; all module parameters are simultaneously optimized through backpropagation, so that the model can not only accurately identify multiple pathologies, but also generate readable diagnostic explanations.

[0010] Optionally, after the training of the chest radiograph classification model is completed, a reverse escaping method corresponding to the escaping method is used to map the text identifiers output by the chest radiograph classification model back to the original label information.

[0011] This invention also provides a chest radiograph multi-label classification system based on a cross-modal memory network, the system comprising: The data acquisition module is used to acquire chest X-ray data and preprocess the chest X-ray data; the chest X-ray data includes the original chest X-ray image and the corresponding label information; the original chest X-ray image in the preprocessed chest X-ray data is used as the input data for model training, and the corresponding label information is used for loss calculation in the supervised learning process; An escaping module is used to escape the label information so as to map the label information to a unique corresponding text identifier; The training module is used to train a chest X-ray classification model based on a cross-modal memory network (CMN) using preprocessed chest X-ray images and escaped label information. The classification module is used to perform multi-label classification of chest X-rays using the chest X-ray classification model.

[0012] The present invention also provides a computing device, the system comprising a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the chest X-ray multi-label classification method based on cross-modal memory network according to the instructions in the program code.

[0013] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the aforementioned chest radiograph multi-label classification method based on a cross-modal memory network.

[0014] This invention proposes a multi-label classification method and system for chest X-rays based on cross-modal memory networks (CMNs), which can effectively capture and fuse key features in chest X-ray images, while achieving effective alignment and fusion between medical terms and labels, thereby improving the accuracy and interpretability of the classification model.

[0015] The above and other objects, advantages and features of the present invention will become more apparent to those skilled in the art from the following detailed description of specific embodiments of the invention in conjunction with the accompanying drawings. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of the process of a multi-label chest radiograph classification method based on a cross-modal memory network according to an embodiment of the present invention; Figure 2 This is a flowchart of the multi-label prediction model for chest radiographs according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the tag escaping method according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the chest X-ray classification model structure according to an embodiment of the present invention. Detailed Implementation

[0017] The embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the present invention and are not restrictive.

[0018] This invention provides a multi-label classification method for chest radiographs based on cross-modal memory networks, such as... Figure 1 As shown, the method of this embodiment of the invention includes: S1, acquire chest X-ray data and preprocess the chest X-ray data; the chest X-ray data includes the original chest X-ray image and the corresponding label information; the original chest X-ray image in the preprocessed chest X-ray data is used as the input data for model training, and the corresponding label information is used for loss calculation in the supervised learning process; S2, the label information is escaped to map the label information to a unique corresponding text identifier; S3, train a chest X-ray classification model based on a cross-modal memory network (CMN) using the preprocessed chest X-ray images and escaped label information; S4. The chest X-ray classification model is used to perform multi-label classification of chest X-rays.

[0019] The present invention discloses a multi-label classification method and system for chest radiographs based on cross-modal memory networks, which can effectively capture and fuse key features in chest radiograph images, and simultaneously achieve effective alignment and fusion between medical terms and labels, thereby improving the accuracy and interpretability of the classification model.

[0020] The following is combined Figure 2 Each step will be explained in detail.

[0021] I. Chest X-ray Data Collection and Preprocessing This embodiment first collects chest X-ray data from a medical imaging database. This data includes raw chest X-ray images and corresponding label information (such as cardiomegaly, pulmonary edema, pleural effusion, etc.). The collected raw chest X-ray images undergo a series of preprocessing steps, including image enhancement, denoising, normalization, and preliminary labeling of lesion areas, to ensure the quality, clarity, and consistency of the image data. The preprocessed chest X-ray images will be used as input data for model training, while the corresponding label information will be used for loss calculation during supervised learning.

[0022] II. Tag Escape Tag escaping: In order to effectively integrate tag information into the model training process, this embodiment uses an escaping method to convert the tags into simple text data.

[0023] This method uses a specific escape function. , original category labels Mapped to a structured text semantic space S. This refers to a specific medical sign (e.g., "enlarged heart", "pulmonary edema"). For a set of tags.

[0024] The core of this escape function T consists of a predefined text template. Implementation, its basic formal definition is:

[0025] in It is a tag The corresponding standard English medical terminology, The standard form that indicates the first letter of the term is capitalized. The term is represented in its entirety in lowercase. Each label (such as cardiomegaly, pulmonary edema, pleural effusion) is mapped to a unique text identifier. This escaping method simplifies the processing of label information and allows it to be aligned and fused with chest X-ray image data during model training.

[0026] like Figure 3 As shown, “cardiac enlargement” can be translated as “Cardiomegaly / No cardiomegaly”; “pulmonary edema” can be translated as “Edema / No edema”; and “pleural effusion” can be translated as “Pleural effusion / No pleural effusion”.

[0027] III. Model Training During the training process of this model, such as Figure 2As shown, the preprocessed chest X-ray image and the escaped label text are input into the model in parallel for end-to-end learning. Specifically, a chest X-ray containing signs of "cardiomegaly," "pulmonary edema," and "pleural effusion" is first processed by a ResNet-101 network to extract its visual features and capture key visual patterns. Simultaneously, the corresponding escaped labels, such as "Cardiomegaly / No cardiomegaly," are encoded into semantically rich text feature vectors using a BERT model. These features are then fed into a cross-modal memory network (CMN), which dynamically establishes deep associations between visual features and semantic labels through its memory units and attention mechanisms. The specific association process is as follows: the CMN uses the label semantic features as the query and the image region features as the key and value, calculating cross-attention weights to generate a visual attention heatmap for each label—for example, the query for "cardiomegaly" will focus on the visual features of the mediastinal region in the image. Meanwhile, the learnable memory units are continuously iterated and updated during network training, gradually solidifying this cross-modal mapping relationship. On the other hand, it serves as the initial context for the Transformer decoder, generating diagnostic reports in an autoregressive manner. An example of the generation process is as follows: the decoder, conditioned on fused features, starts with a start character and progressively predicts the probability distribution of the next word. For example, given fused features and having already generated "heart," the model might output "enlarged" with a high probability, then continue writing "patchy shadows visible in the lung fields," until an end character is generated, forming a complete report. During training, the overall loss function is a weighted average of the binary cross-entropy loss for multi-label classification and the word-level cross-entropy loss for report generation. Backpropagation synchronously optimizes the parameters of all modules, including the image encoder, text encoder, CMN, classifier, and decoder, enabling the model to accurately identify lesions while generating readable, structured diagnostic descriptions. Figure 4 This is a schematic diagram of the chest X-ray classification model structure according to an embodiment of the present invention.

[0028] IV. Generation of Classification Results After the model training is complete, in order to restore the classification results to the original label information, this embodiment uses an anti-escaping method, which corresponds to the escaping method. Specifically, the text identifiers output by the model are mapped back to the original label information (cardiac enlargement, pulmonary edema, pleural effusion). This anti-escaping method ensures the readability and usability of the classification results.

[0029] This invention also provides a chest radiograph multi-label classification system based on a cross-modal memory network, used to execute the chest radiograph multi-label classification method based on a cross-modal memory network described above. The system includes: The data acquisition module is used to acquire chest X-ray data and preprocess the chest X-ray data; the chest X-ray data includes the original chest X-ray image and the corresponding label information; the original chest X-ray image in the preprocessed chest X-ray data is used as the input data for model training, and the corresponding label information is used for loss calculation in the supervised learning process; An escaping module is used to escape the label information so as to map the label information to a unique corresponding text identifier; The training module is used to train a chest X-ray classification model based on a cross-modal memory network (CMN) using preprocessed chest X-ray images and escaped label information. The classification module is used to perform multi-label classification of chest X-rays using the chest X-ray classification model.

[0030] The specific functions of each module in the chest radiograph multi-label classification system based on cross-modal memory network in this invention embodiment can be found in the above method embodiment, and will not be repeated here.

[0031] This invention also provides a computing device, the system including a processor and a memory: the memory is used to store program code and transmit the program code to the processor; the processor is used to execute the chest radiograph multi-label classification method based on cross-modal memory network described in the above embodiments according to the instructions in the program code.

[0032] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the chest radiograph multi-label classification method based on cross-modal memory networks described in the above embodiments.

[0033] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multi-label classification method for chest radiographs based on cross-modal memory networks, characterized in that, The method includes: Acquire chest X-ray data and preprocess the chest X-ray data; the chest X-ray data includes the original chest X-ray image and the corresponding label information; the original chest X-ray image in the preprocessed chest X-ray data is used as the input data for model training, and the corresponding label information is used for loss calculation in the supervised learning process; The label information is escaped to map the label information to a unique corresponding text identifier; The preprocessed chest X-ray images and escaped label information were used to train a chest X-ray classification model based on a cross-modal memory network (CMN). The chest X-ray classification model described above is used to perform multi-label classification of chest X-rays.

2. The method according to claim 1, characterized in that, Escaping the tag information includes: Using escape functions Original category labels Mapped to a structured text semantic space S.

3. The method according to claim 2, characterized in that, The core of the escape function T consists of a predefined text template. Implementation, its basic formal definition is: ； in It is a tag The corresponding standard English medical terminology, The standard form that indicates the first letter of the term is capitalized. This indicates the full lowercase form of the term.

4. The method according to claim 1, characterized in that, The chest radiograph classification model trained using preprocessed chest radiograph images and escaped label information includes: The preprocessed chest X-ray images and the escaped label information were used to extract features from ResNet101 and BERT networks respectively and stored in the cross-modal memory network CMN; Visual features of chest X-ray images are extracted using a ResNet-101 network to capture key visual patterns; meanwhile, the corresponding escape tags are encoded into semantically rich text feature vectors using a BERT model. Visual features and text feature vectors are input into the cross-modal memory network (CMN). The CMN dynamically establishes deep associations between visual features and semantic labels through its memory units and attention mechanisms. The fusion features output by the cross-modal memory network (CMN) serve as both input to a multi-label classifier and initial state for the Transformer decoder, autoregressively generating coherent diagnostic report text. By simultaneously optimizing all module parameters through backpropagation, the model can not only accurately identify various lesions but also generate readable diagnostic descriptions.

5. The method according to claim 4, characterized in that, After the chest X-ray classification model is trained, an anti-escape method corresponding to the escaping method is used to map the text identifiers output by the chest X-ray classification model back to the original label information.

6. A multi-label chest radiograph classification system based on a cross-modal memory network, characterized in that, The system includes: The data acquisition module is used to acquire chest X-ray data and preprocess the chest X-ray data; the chest X-ray data includes the original chest X-ray image and the corresponding label information; the original chest X-ray image in the preprocessed chest X-ray data is used as the input data for model training, and the corresponding label information is used for loss calculation in the supervised learning process; An escaping module is used to escape the label information so as to map the label information to a unique corresponding text identifier; The training module is used to train a chest X-ray classification model based on a cross-modal memory network (CMN) using preprocessed chest X-ray images and escaped label information. The classification module is used to perform multi-label classification of chest X-rays using the chest X-ray classification model.

7. A computing device, characterized in that, The system includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the chest radiograph multi-label classification method based on cross-modal memory network as described in any one of claims 1-5 according to the instructions in the program code.

8. A computer-readable storage medium having a computer program stored thereon, the computer program, when executed by a processor, implementing the chest radiograph multi-label classification method based on a cross-modal memory network as described in any one of claims 1-5.

Images