Acute exacerbation prediction method and device, electronic device and storage medium

Abstract

The present disclosure relates to a method and device for predicting acute exacerbation, an electronic device and a storage medium. The method for predicting acute exacerbation includes: obtaining the number of acute exacerbations corresponding to a plurality of patients at a first time and a second time, clinical text data and / or lung images at the first time; determining training labels according to the number of acute exacerbations corresponding to the first time and the second time; training a preset classifier using the clinical text data and / or the lung images and the corresponding training labels; based on the trained preset classifier, using the clinical text data and / or lung images of the patient to be predicted at the first time, completing the prediction of the acute exacerbation of the patient to be predicted at the second time. The embodiment of the present disclosure can realize the prediction of acute exacerbation.

Description

Technical Field

[0001] The present disclosure relates to the technical field of prediction of acute exacerbation, and in particular to a method and device for predicting acute exacerbation, an electronic device, and a storage medium. Background Art

[0002] Acute exacerbations of chronic obstructive pulmonary disease (AECOPD) are a critical event in the clinical course of COPD and a major determinant of a COPD patient's health status and prognosis. Lung function in COPD patients declines over time, and each acute exacerbation accelerates this process, leading to further decline in lung function, poorer quality of life, and higher mortality.

[0003] Clinically, AECOPD is an acute-onset disease characterized by an acute exacerbation of respiratory symptoms in patients with COPD (typically manifested by worsening dyspnea, increased cough, increased sputum volume, and / or purulent sputum) that exceeds daily variability and requires a change in medication. AECOPD is a diagnosis of exclusion, defined as the absence of other specific explanatory conditions (e.g., pneumonia, congestive heart failure, pneumothorax, pleural effusion, pulmonary embolism, and arrhythmias) found in clinical and / or laboratory tests. Exacerbations of respiratory symptoms may, but may not, improve with treatment, typically resolving within days to weeks. Summary of the Invention

[0004] The present disclosure proposes a method and device for predicting acute exacerbation, an electronic device, and a storage medium technical solution.

[0005] According to one aspect of the present disclosure, a method for predicting acute exacerbation is provided, comprising:

[0006] Obtaining the number of acute exacerbations corresponding to a first time and a second time for a plurality of patients, clinical text data at the first time, and / or lung images;

[0007] determining a training label according to the number of acute exacerbations corresponding to the first time and the second time;

[0008] Training a preset classifier using the clinical text data and / or the lung image and the corresponding training labels;

[0009] Based on the trained preset classifier, the prediction of acute exacerbation of the patient to be predicted at the second time is completed using the clinical text data and / or lung images of the patient to be predicted at the first time.

[0010] Preferably, the method for determining a training label based on the number of acute exacerbations corresponding to the first time and the second time includes:

[0011] If the number of acute exacerbations corresponding to the second time is greater than the number of acute exacerbations corresponding to the first time, the training label is acute exacerbation; otherwise, the training label is non-acute exacerbation.

[0012] Preferably, the method of training a preset classifier using the clinical text data and / or the lung image and the corresponding training labels includes:

[0013] Obtaining corresponding first local features and first global features according to the clinical text data and / or the lung image respectively;

[0014] Training a preset classifier using the first local feature, the first global feature, and the corresponding training labels;

[0015] and / or,

[0016] The method for predicting an acute exacerbation of a patient to be predicted at a second time by using the trained preset classifier and clinical text data and / or lung images of the patient to be predicted at a first time includes:

[0017] Obtaining corresponding second local features and second global features according to the clinical text data and / or lung image of the patient to be predicted at the first time;

[0018] Based on the trained preset classifier, the second local feature and the second global feature are used to complete the prediction of the acute exacerbation of the patient to be predicted at the second time.

[0019] Preferably, the method of training a preset classifier using the first local feature, the first global feature and the corresponding training labels includes:

[0020] Performing a fusion operation on the first local feature and the first global feature to obtain a first fused feature;

[0021] Training a preset classifier using the first fusion feature and the corresponding training label;

[0022] and / or,

[0023] The method for predicting the acute exacerbation of the patient to be predicted at a second time by using the second local feature and the second global feature includes:

[0024] performing a fusion operation on the second local feature and the second global feature to obtain a second fused feature;

[0025] Based on the trained preset classifier, the second fusion feature is used to complete the prediction of the acute exacerbation of the patient to be predicted at the second time.

[0026] Preferably, the method for obtaining the corresponding second local feature and second global feature based on the clinical text data and / or lung image of the patient to be predicted at the first time includes:

[0027] Based on the first local feature, selecting a feature that is the same as the first local feature from features corresponding to the clinical text data and / or lung image of the patient to be predicted at the first time;

[0028] and obtaining a model of a first global feature corresponding to the clinical text data and / or the lung image;

[0029] Based on the model, the corresponding second global feature is obtained using the clinical text data and / or lung image of the patient to be predicted at the first time.

[0030] Preferably, the prediction method further comprises: after obtaining the clinical text data corresponding to the patient to be predicted and / or the image features corresponding to the lung image, identifying or grading the patient for chronic obstructive pulmonary disease;

[0031] If the patient suffers from the chronic obstructive pulmonary disease or the chronic obstructive pulmonary disease of the patient reaches a set grade, a prediction is performed on the patient to be predicted.

[0032] Preferably, the method for identifying or grading chronic obstructive pulmonary disease in the patient comprises:

[0033] Determining whether the clinical text data corresponding to the patient to be predicted includes lung function data;

[0034] If included, identifying or grading chronic obstructive pulmonary disease in the patient based on the lung function data;

[0035] Otherwise, a lung image of the patient is acquired, and COPD of the patient is identified or graded using the lung image.

[0036] According to one aspect of the present disclosure, there is provided a device for predicting acute exacerbation, comprising:

[0037] an acquisition unit, configured to acquire the number of acute exacerbations corresponding to a first time and a second time for a plurality of patients, clinical text data at the first time, and / or lung images;

[0038] a determining unit, configured to determine a training label according to the number of acute exacerbations corresponding to the first time and the second time;

[0039] a training unit, configured to train a preset classifier using the clinical text data and / or the lung image and the corresponding training labels;

[0040] The prediction unit is used to complete the prediction of acute exacerbation of the patient to be predicted at the second time based on the trained preset classifier and using the clinical text data and / or lung images of the patient to be predicted at the first time.

[0041] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0042] processor;

[0043] a memory for storing processor-executable instructions;

[0044] Wherein, the processor is configured to: execute the above-mentioned method for predicting acute exacerbation.

[0045] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned method for predicting acute exacerbation is implemented.

[0046] In the disclosed embodiment, a training label is determined based on the number of acute exacerbations corresponding to the first and second time periods; a preset classifier is trained using the clinical text data and / or the lung images and the corresponding training labels; and based on the trained preset classifier, the clinical text data and / or lung images of the patient to be predicted at the first time are used to complete the prediction of the acute exacerbation of the patient to be predicted at the second time period. This solves the current problem that acute exacerbations accelerate this process, leading to further decline in lung function, poorer quality of life, and higher mortality.

[0047] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0048] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to explain the technical solutions of the present disclosure.

[0050] Figure 1 A flowchart showing a method for predicting acute exacerbation according to an embodiment of the present disclosure;

[0051] Figure 2 A schematic diagram of a convolutional network structure according to an embodiment of the present disclosure is shown;

[0052] Figure 3 A schematic diagram of a graph convolutional network structure according to an embodiment of the present disclosure is shown;

[0053] Figure 4 A schematic diagram of a convolutional network structure for performing a convolution operation on a three-dimensional difference matrix according to an embodiment of the present disclosure is shown;

[0054] Figure 5 A classification model based on deep learning according to an embodiment of the present disclosure is shown;

[0055] Figure 6 A schematic diagram of a network structure corresponding to a preset classification model according to an embodiment of the present disclosure is shown;

[0056] Figure 7 is a block diagram of an electronic device 800 according to an exemplary embodiment;

[0057] Figure 8 is a block diagram of an electronic device 1900 according to an exemplary embodiment. DETAILED DESCRIPTION

[0058] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

[0059] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0060] The term "and / or" herein simply describes an association relationship between associated objects, indicating that three relationships can exist. For example, "A and / or B" can represent the existence of three situations: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0061] In addition, numerous specific details are provided in the following detailed description to better illustrate the present disclosure. Those skilled in the art will appreciate that the present disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the present disclosure.

[0062] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form combined embodiments without violating the principle logic. Due to space limitations, the present disclosure will not elaborate on them.

[0063] In addition, the present disclosure also provides an acute exacerbation prediction device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any acute exacerbation prediction method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method section and will not be repeated here.

[0064] Figure 1 A flowchart of a method for predicting acute exacerbation according to an embodiment of the present disclosure is shown. Figure 1 As shown, the method for predicting acute exacerbation includes: step S101: obtaining the number of acute exacerbations corresponding to a first time and a second time for multiple patients, clinical text data at the first time, and / or lung images; step S102: determining training labels based on the number of acute exacerbations corresponding to the first time and the second time; step S103: training a preset classifier using the clinical text data and / or the lung images and the corresponding training labels; step S104: based on the trained preset classifier, using the clinical text data and / or lung images of the patient to be predicted at the first time, completing the prediction of the acute exacerbation of the patient to be predicted at the second time. This solves the current problem that acute exacerbations accelerate this process, leading to further decline in lung function, poor quality of life, and increased mortality.

[0065] Step S101: Acquire the number of acute exacerbations corresponding to a first time and a second time, clinical text data and / or lung images of a plurality of patients at the first time.

[0066] In the embodiments of the present disclosure and other possible embodiments, clinical text data may include: one or more of the following: lung function data, symptoms of COPD, any related diseases other than chronic obstructive pulmonary disease, medical instructions issued by physicians during medical activities, dyspnea level in the modified Medical Research Council Dyspnea Scale, adverse events, medical history and surgical history, routine physical examination data, COPD rehabilitation treatment, medication status, and other physical examinations (for example, blood tests, blood pressure measurements, etc.), etc.

[0067] In the embodiments of the present disclosure and other possible embodiments, the imaging device may be one or more conventional imaging devices such as CT, PET, MR, ultrasound, DR, or a combination of the aforementioned conventional imaging devices, such as PET-CT. Simultaneously, the aforementioned imaging devices are used to perform chest imaging on the patient to obtain corresponding lung images, such as chest CT lung images, chest PET lung images, chest MR lung images, chest ultrasound lung images, chest PET-CT lung images, and chest DR lung images.

[0068] In the embodiments of the present disclosure and other possible embodiments, the first time and the second time have a set time interval. For example, if the first time is June 2022, the set time interval can be configured to be 1 month, 2 months, 6 months, 12 months, 16 months, or 24 months. For example, if the set time interval is 12 months, the second time is June 2023. Those skilled in the art can configure the set time interval according to actual needs.

[0069] Step S102: determining a training label according to the number of acute exacerbations corresponding to the first time and the second time.

[0070] In an embodiment of the present disclosure, the method for determining a training label based on the number of acute exacerbations corresponding to the first time and the second time includes: if the number of acute exacerbations corresponding to the second time is greater than the number of acute exacerbations corresponding to the first time, then the training label is acute exacerbation; otherwise, the training label is non-acute exacerbation.

[0071] For example, if the number of acute exacerbations corresponding to the second time is greater than the number of acute exacerbations corresponding to the first time, the training label is acute exacerbation and is marked as 1; otherwise, the training label is non-acute exacerbation and is marked as 0. For example, if the number of acute exacerbations corresponding to the second time is 2 times and the number of acute exacerbations corresponding to the first time is times, the training label is acute exacerbation and is marked as 1.

[0072] Step S103: training a preset classifier using the clinical text data and / or the lung image and the corresponding training labels.

[0073] In the embodiments of the present disclosure and other possible embodiments, the preset classifier may be one or more of support vector machine (SVM), multi-layer perceptron (MLP), random forest (RF), K-nearest neighbor (KNN), logistic regression (LR), decision tree (DT), gradient boosting (GB), linear discriminant analysis (LDA), graph network (GNN), deep learning classification network (for example, GoogleNet), etc.

[0074] In the embodiment of the present disclosure and other possible embodiments, the clinical text data and / or the lung image and the corresponding training labels are input into a preset classifier to complete the training of the preset classifier.

[0075] Step S104: Based on the trained preset classifier, the clinical text data and / or lung images of the patient to be predicted at the first time are used to complete the prediction of the acute exacerbation of the patient to be predicted at the second time.

[0076] In the embodiment of the present disclosure and other possible embodiments, the clinical text data and / or lung images of the patient to be predicted at the first time are input into the trained preset classifier to complete the prediction of the acute exacerbation of the patient to be predicted at the second time.

[0077] In an embodiment of the present disclosure, the method for training a preset classifier using the clinical text data and / or the lung image and the corresponding training labels includes: obtaining the corresponding first local feature and first global feature according to the clinical text data and / or the lung image respectively; and training the preset classifier using the first local feature and the first global feature and the corresponding training labels. Wherein, the method for training a preset classifier using the first local feature and the first global feature and the corresponding training labels includes: performing a fusion operation on the first local feature and the first global feature to obtain a first fusion feature; and training the preset classifier using the first fusion feature and the corresponding training labels. Wherein, the method for performing a fusion operation on the first local feature and the first global feature to obtain a first fusion feature includes: performing a splicing operation on the first local feature and the first global feature to obtain a first fusion feature.

[0078] In the embodiments of the present disclosure and other possible embodiments, when the acquired data includes clinical text data and the lung image, the clinical text data and the lung image respectively have their own first local features and first global features; when the acquired data only includes clinical text data, the clinical text data has corresponding first local features and first global features; when the acquired data includes lung images, the lung images respectively have corresponding first local features and first global features.

[0079] In an embodiment of the present disclosure, the method for predicting the acute exacerbation of the patient to be predicted at the second time by using the clinical text data and / or lung image of the patient to be predicted at the first time based on the training of the preset classifier includes: obtaining the corresponding second local feature and second global feature based on the clinical text data and / or lung image of the patient to be predicted at the first time; based on the trained preset classifier, using the second local feature and the second global feature to predict the acute exacerbation of the patient to be predicted at the second time. Wherein, the method for predicting the acute exacerbation of the patient to be predicted at the second time by using the second local feature and the second global feature includes: performing a fusion operation on the second local feature and the second global feature to obtain a second fusion feature; based on the trained preset classifier, using the second fusion feature to predict the acute exacerbation of the patient to be predicted at the second time. Wherein, the method for performing a fusion operation on the second local feature and the second global feature to obtain a second fusion feature includes: performing a splicing operation on the second local feature and the second global feature to obtain a second fusion feature.

[0080] In an embodiment of the present disclosure, the method for obtaining the corresponding second local feature and second global feature based on the clinical text data and / or lung image of the patient to be predicted at the first time includes: based on the first local feature, selecting a feature that is the same as the first local feature from the features corresponding to the clinical text data and / or lung image of the patient to be predicted at the first time; and obtaining a model for obtaining the corresponding first global feature based on the clinical text data and / or lung image; based on the model, obtaining the corresponding second global feature using the clinical text data and / or lung image of the patient to be predicted at the first time.

[0081] In the embodiments of the present disclosure and other possible embodiments, local features can be understood as preset local features, and their corresponding types (radiology and / or clinical text data) or locations (convolutional features, such as the columns where the convolutional features are located) can be obtained when training the model through the following Lasso model or set feature selection rules. When the corresponding second local features and second global features are obtained based on the clinical text data and / or lung images of the patient to be predicted at the first time, it is only necessary to select the same features as the first local features from the features corresponding to the clinical text data and / or lung images of the patient to be predicted at the first time based on the first local features.

[0082] In the embodiments of the present disclosure and other possible embodiments, before training the preset classifier or using the preset classifier for prediction, the lung image is segmented into lung regions to obtain a lung region (lung parenchyma) image; and feature extraction is performed on the lung region image to obtain the image features.

[0083] In the embodiments of the present disclosure and other possible embodiments, the method of performing lung region segmentation on the lung image to obtain a lung region image includes: obtaining a preset segmentation model; performing lung region segmentation on the lung image based on the preset segmentation model to obtain a lung region (lung parenchyma) image. Wherein, the preset segmentation model is a trained lung region (lung parenchyma) segmentation model. For example, the lobe segmentation method, device and storage medium disclosed in Application No.: 202010534722.0 can be used to obtain the lobes of the left lung or the right lung; all the lobes of the left lung are spliced ​​according to the anatomical structure of the left lung to obtain the lung parenchyma of the left lung; all the lobes of the right lung are spliced ​​according to the anatomical structure of the right lung to obtain the lung parenchyma of the right lung. For subjects who have not undergone lobectomy, there are 2 lobes of the left lung and 3 lobes of the right lung. The 2 lobes of the left lung are spliced ​​according to the anatomical structure of the left lung to obtain the lung parenchyma of the left lung; the 3 lobes of the right lung are spliced ​​according to the anatomical structure of the right lung to obtain the lung parenchyma of the right lung. In the embodiment of the present disclosure and other possible embodiments, the lung parenchyma includes peripheral airways and pulmonary blood vessels. Alternatively, the lung region (lung parenchyma) segmentation model can be directly used to obtain lung parenchyma (lung parenchyma of the left lung and the right lung) images.

[0084] In the embodiments of the present disclosure and other possible embodiments, subjects after lobectomy are also considered, wherein at least one of the left upper lobe, left lower lobe, right upper lobe, right middle lobe and right lower lobe is removed in the subjects after lobectomy.

[0085] Based on the above, in the embodiments of the present disclosure and other possible embodiments, lung segmentation may include: a left lung segmentation model, a right lung segmentation model, a left lung lobe loss segmentation model, and a right lung lobe loss segmentation model; therefore, in the present disclosure, a technical solution for segmenting the left lung and the right lung separately is proposed, and the left lung and the right lung are segmented separately. Among them, the left lung segmentation model, the right lung segmentation model, the left lung lobe loss segmentation model, and the right lung lobe loss segmentation model can be lung segmentation models based on traditional segmentation algorithms, or they can be lung segmentation models based on deep learning, such as lung segmentation models based on U-Net or U-ResNet. The training methods of the models are technical means commonly used by those skilled in the art, and the present disclosure will not describe them in detail. However, it is worth noting that the method for segmenting the left lung and the right lung separately is proposed for the case of lobe loss, and there is currently no method for segmenting the remaining lung after lobectomy. Therefore, the method for segmenting the left lung and the right lung separately is not a technical means that is useful to those skilled in the art, and requires corresponding creative work by those skilled in the art.

[0086] In the embodiments of the present disclosure and other possible embodiments, the method for segmenting the left lung and the right lung respectively includes: obtaining a lung image to be processed, determining the position of the main bronchus (first-order trachea) in the lung image, and dividing the lung image into a left lung image and a right lung image according to the position of the main bronchus; respectively determining whether there is a lobe loss in the left lung image and the right lung image; if there is a lobe loss, judging whether the lobe loss is in the left lung or the right lung; if the lobe loss is in the left lung, obtaining a left lung lobe loss segmentation model and a right lung segmentation model, and using the left lung lobe loss segmentation model and the right lung segmentation model to perform lung parenchyma segmentation on the left lung with the loss and the right lung without the loss respectively; if the lobe loss is in the right lung, obtaining a right lung lobe loss segmentation model and a left lung segmentation model, and using the right lung lobe loss segmentation model and the left lung segmentation model to perform lung parenchyma segmentation on the right lung with the loss and the left lung without the loss respectively; splicing the segmented left lung parenchyma and right lung parenchyma according to the anatomical structure to obtain the above-mentioned lung parenchyma. The main bronchus is the trachea from the larynx to the hilum of the lungs.

[0087] For example, if the lung image to be processed only contains the left upper lobe or the left lower lobe, a left lung lobe missing segmentation model and a right lung segmentation model are obtained, and the left lung lobe missing segmentation model and the right lung segmentation model are used to perform lung parenchyma segmentation on the missing left lung and the non-missing right lung respectively; finally, the segmented left lung parenchyma and right lung parenchyma are spliced ​​according to the anatomical structure to obtain the above-mentioned lung parenchyma.

[0088] In the embodiments of the present disclosure and other possible embodiments, the method of determining the position of the main bronchus (first-level trachea) in the lung image and dividing the lung image into a left lung image and a right lung image according to the position of the main bronchus includes: obtaining an airway segmentation model, performing airway segmentation on the lung image, and obtaining an airway tree; determining the main bronchus in the airway tree, and calculating the centerline of the main bronchus; dividing the lung image into a left lung image and a right lung image according to the centerline. At the same time, the airway segmentation model can select an existing airway segmentation model, and the airway segmentation can only segment the main bronchus, and there is no need to perform fine segmentation of the airway. For example, the airway segmentation model used in the method and device based on lung lobes and tracheal trees, electronic equipment, and storage medium disclosed in Application No.: 202010540322.0.

[0089] In an embodiment of the present disclosure, a method for extracting features from the lung region image to obtain the image features includes: obtaining a transferred convolutional neural network and / or a radiomics computational model; and extracting features from the lung region image using the transferred convolutional neural network and / or radiomics computational model to obtain the image features. The image features may be radiomics features and / or convolutional features.

[0090] In the embodiments of the present disclosure and other possible embodiments, the extraction of the omics features of the lung parenchyma image (lung area image) can be achieved through a preset imaging omics calculation model. The preset imaging omics calculation model is an existing imaging omics calculation model and can be obtained through the website https: / / pyradiomics.readthedocs.io / en / latest / index.html. The preset imaging omics feature extraction model will not be described in detail here.

[0091] In the embodiments of the present disclosure and other possible embodiments, the convolution features of the lung parenchyma image (lung region image) can be extracted by a pre-trained feature extraction model; for example, the feature extraction model includes multi-layer convolution, and the convolution features of the lung parenchyma image are extracted by multi-layer convolution. For another example, the convolution features of the lung parenchyma image can be extracted by a transfer learning method; the convolution neural network of the transfer learning can select the segmentation model proposed in the article Med3d: Transfer learning for 3d medical image analysis. (Chen, S., K. Ma and Y. Zheng), but only the encoding structure (downsampling) 3DResNet10 or 3D ResNet18 or 3DResNet34 of the segmentation model needs to be used to extract the convolution features of the lung parenchyma image (lung region image).

[0092] In the embodiment of the present disclosure and other possible embodiments, a preset classifier is trained using the clinical text data and / or the radiomics features and / or convolutional features of the lung image, and the corresponding training labels.

[0093] In the embodiments of the present disclosure and other possible embodiments, the method of training a preset classifier using the clinical text data and / or the imaging omics features and / or convolution features of the lung image, and the corresponding training labels includes: obtaining the corresponding first local feature and first global feature based on the clinical text data and / or the imaging omics features and / or convolution features of the lung image, respectively; and training the preset classifier using the first local feature and the first global feature and the corresponding training labels.

[0094] In the embodiments of the present disclosure and other possible embodiments, when the acquired data includes clinical text data and the lung image, the imaging omics features and / or convolution features of the clinical text data and the lung image respectively have their own first local features and first global features; when the acquired data only includes clinical text data, the clinical text data has corresponding first local features and first global features; when the acquired data includes lung images, the imaging omics features and / or convolution features of the lung images respectively have corresponding first local features and first global features.

[0095] Furthermore, in the embodiments of the present disclosure and other possible embodiments, the method for obtaining the corresponding second local feature and second global feature based on the imaging omics features and / or convolution features of the clinical text data and / or lung images of the patient to be predicted at the first time includes: based on the first local feature, selecting the same feature as the first local feature from the imaging omics features and / or convolution features corresponding to the clinical text data and / or lung images of the patient to be predicted at the first time; and obtaining a model for obtaining the corresponding first global feature based on the imaging omics features and / or convolution features of the clinical text data and / or lung images; based on the model, obtaining the corresponding second global feature using the clinical text data and / or lung images of the patient to be predicted at the first time.

[0096] In the embodiments of the present disclosure and other possible embodiments, the method of obtaining the corresponding first local feature based on the imaging omics features and / or convolution features of the clinical text data and / or the lung image, respectively, includes: obtaining a set feature selection model; based on the feature selection model, selecting the corresponding first local feature from the imaging omics features and / or convolution features of the clinical text data and / or the lung image; or, obtaining a set feature selection rule; based on the feature selection rule, selecting the corresponding first local feature from the imaging omics features and / or convolution features of the clinical text data and / or the lung image.

[0097] In an embodiment of the present disclosure, the method for determining global features based on the imaging omics features and / or convolutional features of the clinical text data and / or the lung image, respectively, includes: obtaining a set feature fusion model; based on the feature fusion model, globally fusing the imaging omics features and / or convolutional features of the clinical text data and / or the lung image to obtain corresponding global features.

[0098] For example, in the embodiments of the present disclosure and other possible embodiments, the set feature selection model can use a Lasso model, and the Lasso model is used to perform feature selection on the clinical text data and / or the omics features and / or the convolutional features based on training labels (the training label is acute exacerbation, marked as 1; the training label is non-acute exacerbation, marked as 0), to obtain selected clinical text data and / or imaging omics features and / or selected convolutional features (local features). The Lasso model can be implemented by calling the standard python package LassoCV (python 3.6).

[0099] In another embodiment and other possible embodiments of the present disclosure, the method of obtaining a set feature selection rule; and selecting corresponding local features from clinical text data and / or multiple image features and / or convolution features based on the feature selection rule, includes: obtaining a set significance and training labels corresponding to the clinical text data and / or the multiple image features (the training label is acute exacerbation, marked as 1; the training label is non-acute exacerbation, marked as 0); respectively calculating multiple significance values ​​corresponding to the clinical text data and / or the multiple image features under the training label (the training label is acute exacerbation, marked as 1; the training label is non-acute exacerbation, marked as 0); and determining the clinical text data and / or image features corresponding to the significance value less than or equal to the set significance as local features.

[0100] For example, the clinical text data and / or the multiple image features corresponding to the training labels (the training label is acute exacerbation, marked as 1; the training label is non-acute exacerbation, marked as 0) are calculated respectively. The clinical text data and / or the multiple image features corresponding to the training labels (the training label is acute exacerbation, marked as 1; the training label is non-acute exacerbation, marked as 0) are calculated. The clinical text data and / or image features corresponding to the significance value less than or equal to the set significance are determined as local features; wherein, the number of significance values ​​calculated is the number corresponding to the clinical text data and / or image features (the omics features and / or the convolution features). wherein, the set significance can be configured to 0.5, 0.1, 0.01, etc., and those skilled in the art can configure the set significance according to actual needs.

[0101] In an embodiment of the present disclosure, the method for determining global features based on the clinical text data and / or the image features, respectively, includes: obtaining a set feature fusion model; and based on the feature fusion model, globally fusing the clinical text data and / or the image features to obtain corresponding global features. In the embodiment of the present disclosure and other possible embodiments, the feature fusion model can be a principal component analysis model (PCA) or a neural network model.

[0102] In an embodiment of the present disclosure, the method for globally fusing the clinical text data and / or the image features to obtain corresponding global features includes: obtaining a set contribution rate; constructing a feature matrix based on the clinical text data and / or the image features, and calculating a plurality of eigenvalues ​​corresponding to the feature matrix; normalizing the plurality of eigenvalues; sorting the normalized plurality of eigenvalues, and calculating the cumulative contribution of the sorted and normalized plurality of eigenvalues; when the cumulative contribution is greater than or equal to the set contribution rate, determining the eigenvector of the eigenvalue corresponding to the cumulative contribution; constructing a conversion matrix based on the eigenvector; and obtaining corresponding global features based on the feature matrix and the conversion matrix. Among them, the set contribution rate can be configured to 95%, and those skilled in the art can configure the set contribution rate according to actual needs.

[0103] In the embodiment of the present disclosure and other possible embodiments, a set contribution rate is obtained; a feature matrix A is constructed based on the clinical text data and / or the image features. m×n =(a1,a,a3,…,a n ); where a1, a, a3, …, a n For the clinical text data and / or the image features, a singular value decomposition (SVD) algorithm can be used to calculate multiple eigenvalues ​​(λ1, λ2, λ3, ..., λn ); normalize the multiple eigenvalues; sort the normalized multiple eigenvalues, and calculate the cumulative contribution of the normalized multiple eigenvalues ​​after sorting; when the cumulative contribution is greater than or equal to the set contribution rate, determine the eigenvector (λ1→ξ1,λ2→ξ2,λ3→ξ3,…,λ k →ξ k );Construct the transformation matrix P according to the eigenvector k×n =(ξ1,ξ2,ξ3,…,ξ k ) k×n ; Obtain corresponding global features based on the feature matrix and the transformation matrix.

[0104] Specifically, the singular value decomposition (SVD) algorithm is used to calculate the multiple eigenvalues ​​(λ1, λ2, λ3, ..., λ n ) can be obtained from Formula 1 and Formula 2.

[0105] A T A=(UΣV T ) T UΣV T =VΣ T U T UΣV T =VΣ T ΣV T =VΣ 2 V T Formula 1

[0106]

[0107] Among them, U m×m and V n×n is an orthogonal matrix, ∑ m×n =(σ1,σ2,σ3,…,σ k ) is a diagonal matrix, σ i is the matrix A T A corresponds to the eigenvalue, m and n are the dimensions of the matrix. Specifically, m is the number of patients, and n is the number of clinical text data and / or the image features.

[0108] Specifically, the corresponding global feature B is obtained based on the feature matrix and the transformation matrix. m×n It can be obtained by formula 3.

[0109]

[0110] Among them, b1, b, b3, …, b k is the corresponding global feature.

[0111] In another embodiment of the present disclosure, the method for globally fusing the clinical text data and / or the image features to obtain corresponding global features includes: constructing a neural network model; training the neural network model using the clinical text data and / or the image features; and based on the trained neural network model, globally fusing the clinical text data and / or the image features to obtain corresponding global features. The neural network model includes at least: an input layer, a hidden layer, and an output layer; and the parameters corresponding to the input layer, hidden layer, and output layer of the neural network are trained using the clinical text data and / or the image features to obtain the trained neural network model.

[0112] Specifically, a model for obtaining a corresponding first global feature based on the clinical text data and / or the lung image is obtained; based on the model, the corresponding second global feature is obtained using the clinical text data and / or the lung image of the patient to be predicted at the first time: for example, the model for obtaining the corresponding first global feature based on the clinical text data and / or the lung image is a neural network model, and at this time, a trained neural network model is obtained, and based on the neural network model, the corresponding second global feature is obtained using the clinical text data and / or the lung image of the patient to be predicted at the first time.

[0113] In the embodiments of the present disclosure and other possible embodiments, the imaging omics features and / or convolution features of the lung image can be calculated, and then the corresponding local features or global features can be determined based on the imaging omics features and / or convolution features, so as to complete the prediction of acute exacerbation of the patient to be predicted at the second time based on the trained preset classifier and the clinical text data of the patient to be predicted at the first time and / or the imaging omics features and / or the local features or global features corresponding to the convolution features.

[0114] More specifically, in the embodiments of the present disclosure and other possible embodiments, the clinical text data and / or the image features correspond to their respective set local features respectively; similarly, the clinical text data and / or the image features correspond to their respective global features respectively. For example, the clinical text data corresponds to the first local feature, and the image feature corresponds to the second local feature; the clinical text data corresponds to the first global feature, and the image feature corresponds to the second global local feature. Furthermore, if the image feature includes: imaging genomics features and CNN (convolution) features, the imaging genomics features and CNN (convolution) features correspond to their respective local features and global features respectively. For example, the imaging genomics features and convolution features in the image feature correspond to the second local feature and the third local feature respectively; the imaging genomics features and convolution features of the image feature correspond to the second global feature and the third global feature respectively.

[0115] In an embodiment of the present disclosure, the method for fusing the set local features and global features to obtain fused features includes: splicing the set local features and global features to obtain the fused features. For example, the clinical text data or lung images corresponding to the patient to be predicted, the model constructed at this time is also obtained by training multiple clinical text data or lung images, and at this time only the local features and global features corresponding to the clinical text data or the local features and global features corresponding to the lung images are spliced. For another example, the clinical text data and lung images corresponding to the clinical text data, the model constructed at this time is also obtained by training multiple clinical text data and lung images, and at this time the local features and global features corresponding to the clinical text data and lung images are spliced.

[0116] In an embodiment of the present disclosure, the prediction method further includes: after obtaining the clinical text data corresponding to the patient to be predicted and / or the image features corresponding to the lung image, identifying or grading the patient's chronic obstructive pulmonary disease; if the patient suffers from the chronic obstructive pulmonary disease or the patient's chronic obstructive pulmonary disease reaches a set grade, then predicting the patient to be predicted. The method for identifying or grading the patient's chronic obstructive pulmonary disease includes: determining whether the clinical text data corresponding to the patient to be predicted includes lung function data; if so, identifying or grading the patient's chronic obstructive pulmonary disease based on the lung function data; otherwise, obtaining the patient's lung image and using the lung image to identify or grade the patient's chronic obstructive pulmonary disease.

[0117] In the embodiments of the present disclosure and other possible embodiments, a pulmonary function tester can be used to perform a pulmonary function test on the patient to obtain the pulmonary function data. A pulmonary function tester is a medical device used to measure the volume of air inhaled and exhaled by the lungs, which can perform pulmonary function tests and track lung health. The pulmonary function data may include one or more of the commonly used pulmonary function test parameters such as FVC, FEV1, FEV1 / FVC, etc. The patient's COPD can be identified or graded based on the pulmonary function data. The patient can be identified (suffering from COPD or not suffering from COPD) or graded (0-4) according to the Global Initiative for Chronic Obstructive Lung Disease (GOLD) standards accepted by the American Thoracic Society and the European Respiratory Society. Among them, grade 0 indicates not suffering from COPD, while grades 1-4 indicate suffering from COPD, and the higher the grade, the more severe the COPD.

[0118] In the embodiments of the present disclosure and other possible embodiments, the method of obtaining a lung image of the patient and using the lung image to identify or grade chronic obstructive pulmonary disease in the patient includes: obtaining a lung image to be processed and performing lung region segmentation on the lung image to obtain a lung parenchyma image; extracting omics features and convolution features of the lung parenchyma image respectively; and, based on the COPD identification or grading labels corresponding to the lung image, performing feature selection on the omics features and the convolution features to obtain selected imaging omics features and selected convolution features; performing a first fusion operation on the selected imaging omics features and the selected convolution features to obtain a first fusion feature; and identifying and / or grading COPD based on the first fusion feature and a preset classifier. The embodiment of the present disclosure extracts the omics features and convolution features of the lung parenchyma image respectively, and performs feature selection on the omics features and convolution features based on the COPD identification and / or grading labels corresponding to the lung image, thereby obtaining selected imaging omics features and selected convolution features; performs a first fusion operation on the selected imaging omics features and the selected convolution features to obtain a first fusion feature; and based on the first fusion feature and a preset classifier, realizes the identification and / or grading of COPD. Compared with the traditional method, the present disclosure adds the convolution features corresponding to the lung parenchyma image, and fuses the imaging omics features and the convolution features to improve the accuracy of COPD identification and / or grading and the classification performance of the classifier.

[0119] The specific method for obtaining a lung image to be processed and performing lung region segmentation on the lung image to obtain a lung parenchyma image can be found in the detailed description above. In the embodiments of the present disclosure and other possible embodiments, subjects who have undergone lobectomy are also considered, wherein at least one of the left upper lobe, left lower lobe, right upper lobe, right middle lobe, and right lower lobe has been removed.

[0120] The omics features and convolution features of the lung parenchyma image are extracted respectively; and, based on the COPD identification and / or classification labels corresponding to the lung image, the omics features and the convolution features are subjected to feature selection to obtain selected imaging omics features and selected convolution features.

[0121] In the embodiments of the present disclosure and other possible embodiments, the extraction of the omics features of the lung parenchyma image can be achieved through a preset imaging omics feature extraction model. The preset imaging omics feature extraction model is an existing imaging omics calculation model, and the preset imaging omics feature extraction model is not described in detail here.

[0122] In the embodiments of the present disclosure and other possible embodiments, the convolution features of the lung parenchyma image can be extracted by a pre-trained feature extraction model; for example, the feature extraction model includes multi-layer convolution, and the convolution features of the lung parenchyma image are extracted by multi-layer convolution. For another example, the convolution features of the lung parenchyma image can be extracted by a transfer learning method; the transfer learning model can select the segmentation model proposed in the article Med3d: Transfer learning for 3d medical image analysis. (Chen, S., K. Ma and Y. Zheng), but only the encoding structure 3D ResNet10 or 3D ResNet18 or 3D ResNet34 of the segmentation model needs to be used to extract the convolution features of the lung parenchyma image.

[0123] At the same time, in the present disclosure, the method of performing feature selection on the omics features and the convolution features based on the COPD identification and / or grading labels corresponding to the lung image to obtain selected imaging omics features and selected convolution features includes: obtaining an imaging omics selection model; based on the imaging omics selection model, performing feature selection on the omics features and the convolution features by establishing the relationship between the COPD identification and / or grading labels and the corresponding omics features and the convolution features to obtain selected imaging omics features and selected convolution features.

[0124] In the embodiments of the present disclosure and other possible embodiments, the feature selection model may be a Lasso model. Using the Lasso model, feature selection is performed on the omics features and the convolution features based on the COPD identification and / or grading labels corresponding to the lung image, respectively, to obtain selected imaging omics features and selected convolution features. For example, the number of omics features extracted from the lung parenchyma image is 1316. When feature selection is performed on the omics features and the convolution features based on the COPD identification and / or grading labels corresponding to the lung image, selected imaging omics features are obtained.

[0125] A first fusion operation is performed on the selected radiomics features and the selected convolutional features to obtain a first fused feature; and based on the first fused feature and a preset classifier, COPD is identified and / or graded. The grading labels range from stage 0 to stage IV, with stage 0 indicating no COPD.

[0126] In the present disclosure, the method of performing a first fusion operation on the selected imaging omics feature and the selected convolution feature to obtain a first fusion feature includes: performing vector splicing on the selected imaging omics feature and the selected convolution feature to obtain a first fusion feature.

[0127] For example, the selected radiomics features corresponding to a lung image to be processed are [1, 2, 4], where 1, 2, and 4 correspond to radiomics features with different names respectively. The selected convolutional features corresponding to this lung image to be processed are [0, 2, 1]. Then, vector splicing is performed on the selected radiomics features and the selected convolutional features to obtain the first fusion feature [1, 2, 4, 0, 2, 1] or [0, 2, 1, 1, 2, 4].

[0128] In the present disclosure, before performing a first fusion operation on the selected imaging omics features and the selected convolution features to obtain a first fusion feature, it also includes: constructing an imaging omics feature map based on the omics features of the lung parenchyma image; and performing convolution processing on the imaging omics feature map to obtain omics convolution features; performing a second fusion operation on the omics convolution features and the omics features of the lung parenchyma image to obtain a second fusion feature; and performing a first fusion operation on the second fusion feature and the selected convolution features to obtain a first fusion feature.

[0129] In the present disclosure, the method of performing a first fusion operation on the second fusion feature and the selected convolution feature to obtain a first fusion feature includes: performing vector splicing on the second fusion feature and the selected convolution feature to obtain a first fusion feature.

[0130] For example, if the second fusion feature is [1, 2, 4] and the selected convolution feature is [0, 2, 1], vector concatenation is performed on the second fusion feature and the selected convolution feature to obtain the first fusion feature [1, 2, 4, 0, 2, 1] or [0, 2, 1, 1, 2, 4].

[0131] In the present disclosure, the method for constructing an imaging omics feature map based on the omics features of the lung parenchyma image includes: taking one feature of the omics features as the basic feature, and subtracting the remaining features of the omics features from the basic feature to obtain an imaging omics feature vector; similarly, taking the remaining features of the omics features as the basic features to obtain the corresponding imaging omics feature vectors; and splicing all the imaging omics feature vectors to obtain an imaging omics feature map.

[0132] For example, a second number N of omics features of the lung parenchyma image is obtained, and an imaging omics feature map with a size of N×(N-1) is constructed. The construction method is specific, including: taking one feature of the omics features as a basic feature, and subtracting the remaining features of the omics features from the basic feature to obtain a 1×(N-1) imaging omics feature vector; similarly, taking N-1 features of the omics features as basic features to obtain an (N-1)×(N-1) imaging omics feature vector; and splicing the imaging omics feature vectors to obtain an imaging omics feature map with a size of N×(N-1).

[0133] For another example, a second number N of omics features of the lung parenchyma image is obtained, and an imaging omics feature map of size N×N is constructed. The construction method is specific, including: taking one feature of the omics features as a basic feature, and subtracting the remaining features of the omics features from the basic feature to obtain a 1×N imaging omics feature vector, wherein the first element in the imaging omics feature vector is 0 (the basic feature minus itself); similarly, taking N-1 features of the omics features as basic features to obtain an (N-1)×N imaging omics feature vector; and splicing the imaging omics feature vectors to obtain an imaging omics feature map of size N×N.

[0134] In the present disclosure, the method for identifying and / or grading COPD based on the first fusion feature and the preset classifier includes: obtaining a set ratio; dividing the first fusion corresponding to all the lung images based on the set ratio to obtain a training set and a verification set; using the training set to train the preset classifier; using the verification set to verify the trained preset classifier, and then identifying and / or grading COPD.

[0135] In the embodiments of the present disclosure and other possible embodiments, for example, the number of lung images to be processed is 1,000 sets, and the ratio is set to 7:3. Based on the set ratio, the first fusion corresponding to all the lung images is divided, and the number of lung images in the training set is 700 sets and the number of lung images in the validation set is 300 sets; the preset classifier is trained using the training set; the trained preset classifier is verified using the validation set, and COPD is identified and / or classified.

[0136] In the embodiments of the present disclosure and other possible embodiments, the method for determining the preset classifier includes: obtaining multiple classifiers to be determined; based on the multiple classifiers to be determined, using the omics features of the lung parenchyma image for classification, and obtaining multiple corresponding first groups of classification indicators respectively; according to the multiple corresponding first groups of classification indicators, determining the first classifier corresponding to the best classification indicator from multiple classifiers; based on the multiple classifiers to be determined, using the convolution features of the lung parenchyma image for classification, and obtaining multiple corresponding second groups of classification indicators respectively; according to the multiple corresponding second groups of classification indicators, determining the second classifier corresponding to the best classification indicator from multiple classifiers; if the first classifier and the second classifier are the same, the preset classifier is the first classifier or the second classifier.

[0137] In the embodiment of the present disclosure and other possible embodiments, the classifier to be determined may be one or more of a support vector machine (SVM), a multi-layer perceptron (MLP), a random forest (RF), a K-nearest neighbor (KNN), a logistic regression (LR), a decision tree (DT), a gradient boosting (GB), a linear discriminant analysis (LDA), etc. At the same time, in the embodiment of the present disclosure and other possible embodiments, the preset classifier is a multi-layer perceptron (MLP).

[0138] In the embodiments of the present disclosure and other possible embodiments, the selected radiomics features are local features, while the omics convolution features obtained by convolution processing the radiomics feature map are global features. Therefore, the present disclosure proposes a method for fusing local features and global features to obtain a second fused feature that includes both local and global features. A first fusion operation is performed on the second fused feature and the selected convolution feature to obtain a first fused feature. Furthermore, based on the first fused feature and a preset classifier, COPD is identified and / or classified.

[0139] In the present disclosure, the method of performing convolution processing on the imaging omics feature map to obtain omics convolution features; and performing a second fusion operation on the omics convolution features and the omics features of the lung parenchyma image to obtain a second fusion feature includes: obtaining a first number corresponding to the selected imaging omics features; performing convolution processing on the imaging omics feature map to obtain the first number of omics convolution features; and performing an addition operation on the omics convolution features and the omics features of the lung parenchyma image to obtain a second fusion feature.

[0140] In the embodiment of the present disclosure and other possible embodiments, the first number corresponding to the selected imaging omics features is 106, so the imaging omics feature map is convolved to obtain the first number of omics convolution features of 106; only when the number corresponding to the selected imaging omics features is the same as the number of omics convolution features, can the omics convolution features and the omics features of the lung parenchyma image be added to obtain the second fusion feature.

[0141] For example, the omics feature of the lung parenchyma image is [1, 2, 4], and the omics convolution feature is [1, 1, 4]. The omics convolution feature and the omics feature of the lung parenchyma image are added to obtain the second fusion feature [2, 3, 8].

[0142] In another embodiment of the present disclosure and other possible embodiments, the method of obtaining a lung image of the patient and using the lung image to identify or grade chronic obstructive pulmonary disease in the patient includes: obtaining imaging omics features corresponding to the lung parenchyma image; selecting the imaging omics features based on the identification and / or grading labels corresponding to the lung parenchyma image to obtain selected imaging omics features; and determining the number of the imaging omics features; generating an imaging omics feature graph using the imaging omics features, performing a convolution operation on the imaging omics feature graph to obtain the number of convolution features; performing a fusion operation on the imaging omics features and the convolution features to obtain a fusion feature; determining the risk factor features of a graph convolutional network based on the fusion feature; performing a splicing operation on the risk factor features and the fusion features to obtain a spliced ​​feature; and using the splicing features to realize the identification and / or grading of chronic obstructive pulmonary disease based on the graph convolutional network. Different from the existing technology, the present invention fuses local radiomics features (selected radiomics features) with global radiomics features (convolution features) corresponding to the radiomics feature graph to obtain fusion features, and then determines the risk factor features of the graph convolution network based on the fusion features; splices the risk factor features and the fusion features to obtain splicing features; based on the graph convolution network, uses the splicing features to realize the identification and / or classification of COPD. Fully explore the radiomics features, further use the radiomics features to identify and / or classify COPD, and then promote the clinical application of radiomics features in the identification and / or classification of COPD.

[0143] Obtain radiomic features corresponding to the lung parenchyma image; select the radiomic features based on the identification and / or classification labels corresponding to the lung parenchyma image to obtain selected radiomic features; and determine the number of the radiomic features.

[0144] In an embodiment of the present disclosure, the method of performing lung region segmentation on the lung image to be processed to obtain a lung parenchyma image corresponding to the lung region includes: obtaining a preset lung region segmentation model; performing lung region segmentation on the lung image to be processed based on the preset lung region segmentation model to obtain a lung parenchyma image. The lung parenchyma image includes: lung parenchyma corresponding to the left lung and the right lung. The preset lung region segmentation model can select a trained U-net neural network or a ResU-Net neural network. The lung parenchyma in the embodiment of the present disclosure and other possible embodiments includes peripheral airways and pulmonary blood vessels.

[0145] In the embodiments of the present disclosure and other possible embodiments, before obtaining the imaging omics features corresponding to the lung parenchyma image, the omics features of the lung parenchyma image are extracted. The extraction of the omics features of the lung parenchyma image can be achieved through a preset imaging omics feature extraction model. The preset imaging omics feature extraction model is an existing imaging omics calculation model, and the preset imaging omics feature extraction model is not described in detail here.

[0146] In the embodiments of the present disclosure and other possible embodiments, the radiomics features are selected based on the identification and / or grading labels corresponding to the lung parenchyma images to obtain the selected radiomics features. A Lasso model is used to select the radiomics features based on the identification and / or grading labels corresponding to the lung parenchyma images to obtain the selected radiomics features.

[0147] In the embodiment of the present disclosure and other possible embodiments, the number of radiomic features corresponding to the lung parenchyma image is 1316; based on the identification and / or grading labels corresponding to the lung parenchyma image, the radiomic features are selected, and the number of selected radiomic features is 106.

[0148] An imaging omics feature map is generated using the imaging omics feature, and a convolution operation is performed on the imaging omics feature map to obtain the number of convolution features.

[0149] In the present disclosure, as distinguished from the prior art, the method for generating an imaging omics feature map using the imaging omics features includes: determining the number M of the imaging omics features; arranging the imaging omics features in a set manner to generate an M×M imaging omics feature map. In the embodiments of the present disclosure and other possible embodiments, the method for arranging the imaging omics features in a set manner to generate an M×M imaging omics feature map includes: using the M feature elements of the imaging omics features as the imaging omics feature vectors of the first row; for the imaging omics feature vectors of the next row, the imaging omics feature vectors of the previous row are all shifted to the right, with the last feature element being the first feature element of this row; and finally, generating an M×M imaging omics feature map.

[0150] For example, the imaging omics feature is [1,2,3,4], the number of imaging omics features is 4, the imaging omics feature vector in the first row is [1,2,3,4], then the imaging omics feature vector in the second row is [4,1,2,3], the imaging omics feature vector in the third row is [3,4,1,2,], and the imaging omics feature vector in the third row is [2,3,4,1]; finally, a 4×4 imaging omics feature map is generated, and the first to fourth rows of the 4×4 imaging omics feature map are [1,2,3,4], [4,1,2,3], [3,4,1,2,] and [2,3,4,1] respectively.

[0151] In the embodiment of the present disclosure and other possible embodiments, through the above method, the size of the radiomics feature map corresponding to the number of radiomics features 1316 is 1316×1316.

[0152] In the present disclosure, a method for performing a convolution operation on the imaging omics feature map to obtain the number of convolution features includes: obtaining a migration convolution model; and using the migration convolution model to perform a convolution operation on the imaging omics feature map to obtain the number of convolution features.

[0153] In the embodiments of the present disclosure and other possible embodiments, the transfer convolution model can select the segmentation model proposed in the article Med3d: Transfer learning for 3d medical image analysis. (Chen, S., K. Ma and Y. Zheng). However, it is only necessary to use the encoding structure 3D ResNet10 or 3D ResNet18 or 3DResNet34 in the segmentation model to perform a convolution operation on the imaging omics feature map to obtain the convolution features of the number.

[0154] Figure 2 FIG. 1 shows a schematic diagram of a convolutional network structure according to an embodiment of the present disclosure. Figure 2 As shown, the constructed convolutional network includes: a first convolutional layer Cov1 (1, 64, 7, 3, 1), a second convolutional layer Cov2 (64, 32, 5, 2, 0), a third convolutional layer Cov3 (32, 16, 3, 1, 0), a fourth convolutional layer Cov4 (16, 8, 3, 1, 0), a fifth convolutional layer Cov5 (8, 1, 2, 1, 0), a pooling layer Pool (3, 2, 0), a first fully connected layer FC1 (11236, 106) and a second fully connected layer FC2 (106, 4).

[0155] Figure 2During training of the constructed convolutional network, the second fully connected layer FC2 (106, 4) is retained; when a convolution operation is performed on the imaging omics feature map based on the trained convolutional network to obtain the number of convolutional features, the second fully connected layer FC2 (106, 4) is deleted.

[0156] Taking the constructed convolutional network as an example during training, Figure 2 For detailed explanation, the radiomics feature map is 1 feature map, and the radiomics feature map is convolved with a 7×7 convolution kernel with a step size of 3 and padding processing to obtain 64 feature maps; 64 feature maps are convolved with a 5×5 convolution kernel with a step size of 2 and no padding processing to obtain 32 feature maps; 32 feature maps are convolved with a 3×3 convolution kernel with a step size of 1 and no padding processing to obtain 16 feature maps; 16 feature maps are convolved with a 3×3 convolution kernel with a step size of 1 and no padding processing to obtain 16 feature maps. After convolution with a kernel with a stride of 1 and no padding, 8 feature maps are obtained. After convolution with a 2×2 kernel with a stride of 1 and no padding, 8 feature maps are obtained. After pooling with a 3×3 kernel with a stride of 2 and a stride of 1 on the 1 feature map, a pooled feature vector with 11236 features is obtained. After passing the pooled feature vector with 11236 features through the first fully connected layer FC1, 106 feature vectors are obtained. After passing the 106 feature vectors through the second fully connected layer FC2, 4 feature vectors are obtained. The loss function used in the training process can be selected by the user according to actual needs.

[0157] A fusion operation is performed on the imaging omics feature and the convolution feature to obtain a fusion feature; based on the fusion feature, a risk factor feature of a graph convolutional network is determined; and a splicing operation is performed on the risk factor feature and the fusion feature to obtain a splicing feature.

[0158] The fusion operation of the imaging omics features and the convolution features to obtain the fusion features is the basis for determining the risk factor features of the graph convolutional network based on the fusion features. It is precisely because the fusion operation of the imaging omics features and the convolution features to obtain the fusion features obtains the fusion of local features (selected imaging omics features) and global features (unselected imaging omics features, or imaging omics features before selection), the fusion features can better reflect the imaging omics features. Therefore, the method of fusing the imaging omics features and the convolution features to obtain the fusion features is not a technical means commonly used by those skilled in the art, and it also requires those skilled in the art to pay corresponding creative labor.

[0159] In the embodiments of the present disclosure and other possible embodiments, different from the prior art, the method of fusing the imaging omics features and the convolution features to obtain fused features includes: performing element-wise addition of the imaging omics features and the convolution features to obtain fused features.

[0160] For example, the radiomics feature is [1, 1, 1, 3], and the convolution feature is [0, 1, 0, 3]. The radiomics feature and the convolution feature are added at the pixel level to obtain a fusion feature [1, 2, 1, 6].

[0161] In the embodiments of the present disclosure and other possible embodiments, unlike the prior art, the present disclosure proposes three methods for determining risk factor characteristics of graph convolutional networks based on the fusion features. How to determine the risk factor characteristics of a graph convolutional network directly affects the generation of the edge constraint matrix E, which in turn affects the adjacency matrix. Therefore, the above-mentioned methods for determining risk factor characteristics of a graph convolutional network based on the fusion features are not conventional technical means used by those skilled in the art and require corresponding creative effort from those skilled in the art.

[0162] In the embodiments of the present disclosure and other possible embodiments, the first method, the method for determining the risk factor characteristics of the graph convolutional network based on the fusion features, includes: obtaining a generalized linear model (GLM); using the generalized linear model (GLM), based on the fusion features and corresponding identification and / or classification labels, obtaining the R^2 value of each feature element in the fusion features; using the R^2 value to sort the feature elements in the fusion features from large to small; and intercepting the first set number E of features after sorting as the risk factor characteristics of the graph convolutional network. The value of the set number E is generally 2-6.

[0163] In the embodiments of the present disclosure and other possible embodiments, the second method, the method for determining the risk factor characteristics of the graph convolutional network based on the fusion features, further includes: obtaining a Lasso model; using the Lasso model, according to the fusion features and the corresponding identification and / or classification labels, obtaining the screening features of the fusion features; using the regression coefficients of the screening features to sort the screening features from large to small, and intercepting the first set number E of the sorted screening features as the risk factor characteristics of the graph convolutional network. Wherein, the value of the set number E is generally 2-6, and the number of the screening features is less than the number of the fusion features.

[0164] In the embodiments of the present disclosure and other possible embodiments, a third method, the method for determining the risk factor characteristics of a graph convolutional network based on the fused features, includes: obtaining an independent component analysis (PCA) model; utilizing the independent component analysis model to obtain a set number E of dimensionality reduction features of the fused features based on the fused features and corresponding identification and / or classification labels; and using the set number E of dimensionality reduction features as the risk factor characteristics of the graph convolutional network. The set number E is generally 2-6, and the number of dimensionality reduction features is less than the number of fused features.

[0165] In the embodiments of the present disclosure and other possible embodiments, a specific embodiment of the method for concatenating the risk factor features and the fused features to obtain a concatenated feature is as follows: If the value of E is 4, then the 4 risk factor features are concatenated with the 106 fused features to obtain a concatenated feature (vector) with a dimension of 110.

[0166] Based on the graph convolutional network, the splicing features are used to realize the identification and / or classification of COPD.

[0167] Figure 3 FIG. 1 shows a schematic diagram of a graph convolutional network structure according to an embodiment of the present disclosure. Figure 3 As shown, the number of risk factor features is K, the number of fusion features is d, and the number of nodes of the graph convolutional network is N. In the embodiment of the present disclosure and other possible embodiments, the spliced ​​features are input into the graph convolutional network (GCN) to achieve the recognition and / or classification of COPD. Among them, the graph convolutional network is a method that can perform deep learning on graph data. The graph convolutional neural network actually has the same function as the convolutional neural network CNN, which is a feature extractor, except that its object is graph data. The graph convolutional network has ingeniously designed a method for extracting features from graph data, so that we can use these features to perform node classification, graph classification, and link prediction on the graph data, and we can also obtain the graph embedding representation (graph embedding) by the way, which shows that it has a wide range of uses.

[0168] In the present disclosure, the method for identifying and / or grading COPD based on the graph convolutional network and using the splicing features includes: obtaining the number of nodes N of the graph convolutional network; based on the splicing features V (l)The risk factor features are obtained by obtaining a sub-edge constraint matrix corresponding to the number of risk factor features; and the sub-edge constraint matrices are fused to obtain an edge constraint matrix E; based on the splicing feature V corresponding to the number of nodes N (l) The fusion features are obtained to obtain a three-dimensional difference matrix; and the three-dimensional difference matrix is ​​convolved to obtain an edge weight matrix W; based on the edge constraint matrix E and the edge weight matrix W, an adjacency matrix is ​​obtained; using the adjacency matrix and the splicing feature V (l) To achieve identification and / or classification of COPD. In the embodiment of the present disclosure and other possible embodiments, the splicing feature V (l) , including: K-dimensional risk factor features (vectors) and d-dimensional fusion features (vectors).

[0169] In the present disclosure, the method for obtaining a sub-edge constraint matrix corresponding to the number of risk factor features based on the risk factor features of the spliced ​​features includes: arranging the risk factor features of each of the spliced ​​features in columns to obtain risk factor feature vectors of the number of risk factor features; taking one feature of each risk factor feature vector as a basic feature, performing an element-level difference operation on all features of each risk factor feature vector and the basic feature to obtain a sub-edge constraint matrix corresponding to the number of risk factor features.

[0170] In the embodiments of the present disclosure and other possible embodiments, for example, K=4, the number is 4, so the risk factor features of each spliced ​​feature are arranged in columns to obtain 4 risk factor feature vectors (scale N×1); one feature in the risk factor feature vector is used as the basic feature, and the remaining features in the risk factor feature vector are subtracted from the basic feature to obtain an N×1 risk factor feature vector, where the first element in the risk factor feature vector is 0 (the basic feature minus itself); similarly, N-1 features in the risk factor feature vector are used as the basic features to obtain an N×(N-1) risk factor feature vector; the risk factor feature vectors are spliced ​​to obtain a sub-edge constraint matrix of size N×N. Therefore, the risk factor features of the 4 spliced ​​features are subjected to the above operation to obtain K=4 sub-edge constraint matrices of size N×N.

[0171] In the present disclosure, the method of fusing the sub-edge constraint matrices to obtain an edge constraint matrix includes: performing an element-wise addition operation on the sub-edge constraint matrices to obtain an edge constraint matrix. In the embodiment of the present disclosure and other possible embodiments, for example, when K=4, the four sub-edge constraint matrices of size N×N are obtained, and the corresponding elements are added together and normalized to obtain an edge constraint matrix.

[0172] In the present disclosure, the method for obtaining a three-dimensional difference matrix based on the fusion features of the splicing features corresponding to the number of nodes includes: arranging the fusion features of each of the splicing features in columns to obtain a fusion feature vector; copying the number of fusion feature vectors and performing a splicing operation to obtain a three-dimensional feature; and transposing the three-dimensional feature to obtain a transposed three-dimensional feature; performing an element-level difference operation on the three-dimensional feature and the transposed three-dimensional feature to obtain a three-dimensional difference matrix. After copying the number of fusion feature vectors and performing a splicing operation to obtain a three-dimensional feature; and transposing the three-dimensional feature to obtain a transposed three-dimensional feature; and performing an element-level difference operation on the three-dimensional feature and the transposed three-dimensional feature to obtain a three-dimensional difference matrix.

[0173] In the embodiments of the present disclosure and other possible embodiments, the fused features of each of the spliced ​​features are arranged in columns to obtain a fused feature vector with a scale of N×106; the fused feature vectors are copied N-1 times to obtain N N×106 fused feature vectors; the N N×106 fused feature vectors are spliced ​​to obtain a three-dimensional feature with a scale of N×N×106; the three-dimensional feature with a scale of N×N×106 is transposed to obtain a transposed three-dimensional feature; the three-dimensional feature and the transposed three-dimensional feature are element-wise differencing and then an absolute value operation is performed to obtain a three-dimensional difference matrix with a scale of N×N×106.

[0174] After duplicating the number of fused feature vectors, a concatenation operation is performed to obtain a three-dimensional feature; the three-dimensional feature is transposed to obtain a transposed three-dimensional feature; and the three-dimensional feature and the transposed three-dimensional feature are element-wise interpolated to obtain a three-dimensional difference matrix. Specifically, the 106 fused features are copied N-1 times and combined with the original N×106 fused features to form an N×N×106 three-dimensional matrix. Then, the Z axis remains unchanged, and the matrix is ​​transposed in the X and Y axis directions. The resulting matrix is ​​still an N×N×106 three-dimensional matrix. The two matrices are subtracted to obtain an N×N×106 three-dimensional difference matrix. The subtraction operation is to subtract the 106 features of different nodes one by one to obtain a three-dimensional adjacency matrix.

[0175] In the present disclosure, a new method for performing a convolution operation on the three-dimensional difference matrix to obtain an edge weight matrix is ​​proposed. Specifically, the method for performing a convolution operation on the three-dimensional difference matrix to obtain an edge weight matrix includes: using a convolution kernel larger than 1×1 to upsample the three-dimensional difference matrix to obtain an upsampled feature map; and using a convolution kernel larger than 1×1 to downsample the upsampled feature map to obtain an edge weight matrix.

[0176] Figure 4FIG. 1 shows a schematic diagram of a convolution network structure for performing a convolution operation on a three-dimensional difference matrix according to an embodiment of the present disclosure. Figure 4 As shown in the figure, the proposed convolutional network for convolution operations on the three-dimensional difference matrix has a ∩ structure and can therefore be defined as a ∩Net. The ∩Net transforms the N×N×106 three-dimensional difference matrix into an N×N edge weight matrix. The N×N×106 three-dimensional difference matrix is ​​upsampled using a 3×3 convolution kernel to obtain an upsampled feature map. The upsampled feature map is then downsampled using a 3×3 convolution kernel, and finally convolved using a 1×1 convolution kernel to obtain an N×N×1 edge weight matrix.

[0177] Specifically, if a 1×1 convolution kernel is directly used to superimpose different channels, the elements will not affect each other, resulting in the output matrix having no connection between differences. This means that some effective information is not utilized. The differences between different nodes are also connected, and not utilizing these connections during training will reduce the effectiveness of the model. Therefore, a 3×3 convolution kernel is introduced to extract this information. This process is equivalent to reshaping an adjacency matrix. By using an upsampling followed by a downsampling operation, the matrix after the network not only contains the relationships between nodes, but also the connections between the differences between different groups of nodes. At the same time, residual connections are introduced into the network. Residual connections are very effective when deepening the network, allowing smaller amounts of data to be used in deeper networks to obtain better structures. The deeper the ∩Net network, the more effective the network is. Including residual connections can enhance the effectiveness of the network.

[0178] In the present disclosure, the method for obtaining an adjacency matrix based on the edge constraint matrix and the edge weight matrix includes: performing element-wise multiplication of the edge constraint matrix and the edge weight matrix to obtain an adjacency matrix.

[0179] In the present disclosure, the method for identifying and / or grading COPD based on the graph convolutional network using the adjacency matrix and the splicing features includes: performing a matrix multiplication operation on the adjacency matrix and the splicing features to obtain an adjacency feature matrix; the adjacency feature matrix passes through a fully connected layer (FC) to obtain an adjacency feature vector Gn(V (l) ); and the adjacent feature vector Gn(V (l) ) and the splicing feature V (l) Splice and get the classification vector (V (l+1) ); and using the classification vector to identify and / or grade COPD.

[0180] Specifically, the edge weight matrix W is multiplied element-wise by the edge constraint matrix E, and the resulting matrix is ​​the adjacency matrix used for the graph network; the concatenation feature V of the adjacency matrix and the input node (l) Perform matrix multiplication. This step updates the node matrix. After the update, the result Gn(V (l) ), and then concatenate the features V with this layer (l) Connect them in series and you get the result of this layer. In the last layer, Gn(V (last) ) is not connected in series with the input node, but the output of the unknown node is directly input into the softmax layer, the output is normalized, and the final prediction of the unknown node is obtained.

[0181] In another embodiment of the present disclosure and other possible embodiments, the method for identifying or grading COPD (chronic obstructive pulmonary disease) includes: obtaining a lung image to be processed and a preset recognition model; projecting the lung image to be processed according to a set plurality of angles to obtain a plurality of projection images; and based on the preset recognition model, using the plurality of projection images, completing the COPD identification or grading of the lung image to be processed.

[0182] Obtain the lung image to be processed and the preset recognition model.

[0183] In the embodiments of the present disclosure and other possible embodiments, the preset recognition model may be a classification model based on machine learning, such as one or more of support vector machine (SVM), multi-layer perceptron (MLP), random forest (RF), K-nearest neighbor (KNN), logistic regression (LR), decision tree (DT), gradient boosting (GB), linear discriminant analysis (LDA), etc. The preset recognition model may be a classification model of deep learning, such as ResNet network. For details, see Figure 5 Detailed description.

[0184] Figure 5 The classification model based on deep learning according to the embodiment of the present disclosure is shown. Figure 5As shown, conv represents convolution (layer), s represents stride, pool represents pooling layer, avg pool represents average pooling layer, fc represents fully connected layer, COPD represents COPD, and HC represents COPD (healthy person). Specifically, the classification model based on deep learning includes four modules. Each module is repeated twice, and the two modules are connected in a shortcut manner. The modules mainly contain convolution layers. The convolution kernels of the convolution layer conv are 3×3 and 1×1. The network ends with the average pooling layer avgpool and the fully connected layer fc. The difference between ResNet26 and ResNet50 is mainly the number of modules. The number of modules of ResNet26 is (2,2,2,2), while the number of modules of ResNet50 is (3,4,6,3). The difference between ResNet26 and ResNet26d is mainly in the downsampling operation of each module. The ResNet26 network has downsampling operations in each module, such as Figure 5 As shown in the figure, there are two paths. The left path is three convolutional layers with convolution kernels of 1×1, 3×3, and 1×1 respectively. The strides of the convolution kernels are 2, 1, and 1 respectively. The right path is a convolutional layer with a convolution kernel of 1×1 and a stride of 2. The main difference between ResNet26d and ResNet26 is that the strides of the convolution kernels in the left path are 1, 2, and 1 respectively. In this disclosure, the transfer learning strategy is an effective method for ResNet-26d to avoid overfitting. This study uses the ResNet convolution weights from the ImageNet pre-trained model, and on this basis uses the dataset of this study to train the fully connected layer and the output layer.

[0185] The lung image to be processed is projected according to a plurality of set angles to obtain a plurality of projection images.

[0186] In the embodiments of the present disclosure and other possible embodiments, the multiple set angles may be at least one set angle, for example, any angle corresponding to the transverse section, coronal plane, and sagittal plane corresponding to the lung image to be processed. That is to say, the lung image to be processed may be an image in the xyz plane, and may be projected according to three set angles of the xyz plane (the angle in the x direction, the angle in the y direction, and the angle in the z direction) to obtain multiple projection images (a projection image of the transverse section, a projection image of the coronal plane, and a projection image of the sagittal plane). At the same time, the present disclosure does not limit the multiple set angles. For example, the multiple set angles may also be angles of 30°, 45°, 60°, etc. in the xy plane, the yz plane, and the x direction, the y direction, and the z direction of the xz plane.

[0187] In the present disclosure, the method of projecting the lung image to be processed according to multiple set angles to obtain multiple projection images includes: obtaining multiple set angles and projection criteria; based on the multiple set angles, projecting the lung image to be processed according to the projection criteria respectively to obtain multiple projection images.

[0188] In the embodiments of the present disclosure and other possible embodiments, the projection criterion may be a maximum intensity projection criterion (MIP) or a minimum intensity projection criterion, or a density interval projection criterion. For example, according to the maximum intensity projection criterion, the lung image to be processed is projected at its maximum value along the z-axis to obtain a corresponding projection image. Alternatively, according to the maximum intensity projection criterion, the lung image to be processed may be projected at its maximum value along both the x-axis and y-axis directions to obtain two corresponding projection images.

[0189] For example, the lung image to be processed is a CT image, which has three layers of xy images, namely Take the maximum value in the z-axis direction for projection and get the corresponding projection image

[0190] For another example, the lung image to be processed is a CT image, and the CT image has three layers of xy images, namely Take the minimum value in the z-axis direction for projection and get the corresponding projection image

[0191] At the same time, the present disclosure proposes a density interval projection criterion to extract a set density interval for projection, obtaining a projection corresponding to the density of interest, so as to better identify COPD. The method of projecting the lung image to be processed according to the density interval projection criterion based on multiple set angles to obtain multiple projected images includes: selecting values ​​of the lung image to be processed within the set density interval according to the multiple set angles, and superimposing and projecting the values ​​within the set density interval to obtain multiple projected images.

[0192] For example, the lung image to be processed is a CT image, and the CT image has three layers of xy images, which are the first xy image Second xy image Third xy image Take the set density interval [1,3] for projection along the z-axis direction to obtain the corresponding projection image Specifically, the value 2 in the first row and first column of the projected image is obtained by adding the value 1 in the first row and first column of the first xy image within the set density interval [1,3] to the value 1 in the first row and first column of the second xy image. Since the value 5 in the first row and first column of the third xy image is not within the set density interval [1,3], the value 5 is not added to the other values.

[0193] Based on the preset recognition model, the multiple projection images are used to complete the COPD recognition of the lung image to be processed.

[0194] In the present disclosure, the method for completing COPD identification or grading of the lung image to be processed based on the preset recognition or grading model and using the multiple projection images includes: respectively inputting the multiple projection images into the preset recognition or grading model to obtain multiple first classification results; respectively performing statistics on the results of COPD and non-COPD in the multiple first classification results to obtain a first statistical result of COPD and a second statistical result of non-COPD; if the first statistical result is greater than the second statistical result, it is determined that the patient has COPD; otherwise, it is determined that the patient does not have COPD.

[0195] For example, in the embodiment of the present disclosure and other possible embodiments, three projection images (a transverse projection image, a coronal projection image, and a sagittal projection image) are input into the image processing system. Figure 5 In the preset recognition model, the preset recognition model inputs the corresponding 3 first classification results; the results of suffering from COPD and not suffering from COPD in the 3 first classification results are respectively counted to obtain the first statistical results (2) of suffering from COPD and the second statistical result (1) of not suffering from COPD; at this time, if the first statistical results (2) are greater than the second statistical result (1), it is determined that the person has COPD.

[0196] In the embodiments of the present disclosure and other possible embodiments, before projecting the lung image to be processed according to multiple set angles to obtain multiple projection images, the method further includes: performing a lung parenchyma segmentation operation on the lung image to be processed to obtain a lung parenchyma image; projecting the lung parenchyma image according to multiple set angles to obtain multiple projection images; and then, based on the preset recognition model, using the multiple projection images to complete the COPD recognition of the lung image to be processed.

[0197] In the embodiments of the present disclosure and other possible embodiments, a method for performing a lung parenchyma segmentation operation on the lung image to be processed to obtain a lung parenchyma image includes: obtaining a lung segmentation model; and performing a lung parenchyma segmentation operation on the lung image to be processed based on the lung segmentation model to obtain a lung parenchyma image. The segmentation model can be a lung segmentation model based on a traditional segmentation algorithm or a lung segmentation model based on deep learning, such as a lung segmentation model based on U-Net or U-ResNet. The training method of the model is a technical means commonly used by those skilled in the art and will not be described in detail in this disclosure.

[0198] In the present disclosure, before projecting the lung image to be processed according to multiple set angles to obtain multiple projection images, it also includes: performing a lung parenchyma segmentation operation on the lung image to be processed to obtain a lung parenchyma image; based on an imaging omics calculation model, calculating the preset imaging omics features corresponding to the lung parenchyma image; based on the preset recognition model, using the multiple projection images and the preset imaging omics features, completing the COPD recognition of the lung image to be processed.

[0199] In the embodiments of the present disclosure and other possible embodiments, a method for performing a lung parenchyma segmentation operation on a lung image to be processed to obtain a lung parenchyma image includes: obtaining a lung segmentation model; and performing a lung parenchyma segmentation operation on the lung image to be processed based on the lung segmentation model to obtain a lung parenchyma image. The segmentation model can be a lung segmentation model based on a traditional segmentation algorithm or a lung segmentation model based on deep learning, such as a lung segmentation model based on U-Net or U-ResNet.

[0200] In the present disclosure, based on the preset recognition model, the method for completing the COPD identification of the lung image to be processed using the multiple projection images and the preset imaging genomics features includes: obtaining a preset convolutional neural network; using the preset convolutional neural network to perform feature extraction on the multiple projection images respectively to obtain multiple groups of convolution features; fusing the multiple groups of convolution features and the preset imaging genomics features to obtain recognition features; based on the preset recognition model, using the recognition features, completing the COPD identification of the lung image to be processed.

[0201] In the embodiment of the present disclosure and other possible embodiments, the preset convolutional neural network can be selected Figure 5The disclosed model extracts features from the multiple projection images respectively to obtain multiple groups of convolution features. At the same time, the segmentation model proposed in the article Med3d: Transfer learning for 3d medical image analysis. (Chen, S., K. Ma and Y. Zheng) can also be selected. However, it is only necessary to use the encoding structure 3D ResNet10 or 3DResNet18 or 3D ResNet34 of the segmentation model to extract the convolution features of the lung parenchyma image. Then, the multiple groups of convolution features and the preset imaging genomics features are fused to obtain recognition features. Based on one or more of the support vector machine (SVM), multi-layer perceptron (MLP), random forest (RF), K-nearest neighbor (KNN), logistic regression (LR), decision tree (DT), gradient boosting (GB), linear discriminant analysis (LDA) and the like in the preset recognition model, the recognition features are used to complete the recognition of COPD in the lung image to be processed.

[0202] In this disclosure, a method for fusing convolutional features and radiomics features is proposed to improve the accuracy of COPD identification. Specifically, the method of fusing the multiple sets of convolutional features and the preset radiomics features to obtain identification features includes: fusing the radiomics features with each set of convolutional features of the multiple sets of convolutional features to obtain multiple sets of identification features; and based on the preset recognition model, using the multiple sets of identification features to complete COPD identification in the lung image to be processed.

[0203] In the embodiments of the present disclosure and other possible embodiments, the method of fusing the imaging omics features with each group of convolutional features of the multiple groups of convolutional features to obtain multiple groups of identification features includes: performing a splicing operation on the imaging omics features with each group of convolutional features of the multiple groups of convolutional features to obtain multiple groups of identification features.

[0204] In the embodiment of the present disclosure and other possible embodiments, the radiomics features and the multiple sets of convolutional features are both radiomics features and multiple sets of convolutional features corresponding to the segmented lung parenchyma image.

[0205] For example, the multiple projection images (transverse projection images, coronal projection images, and sagittal projection images) of the lung image to be processed or the multiple projection images (transverse projection images, coronal projection images, and sagittal projection images) of the lung parenchyma image after lung segmentation correspond to the multiple groups of convolution features, namely, the first group of convolution features is [1, 2, 3], the second group of convolution features is [4, 5, 6], the first group of convolution features is [7, 8, 9], and the imaging omics features are [1, 1, 1]. The imaging omics features are concatenated with each group of convolution features of the multiple groups of convolution features to obtain multiple groups of identification features, namely, [1, 2, 3, 1, 1, 1], [4, 5, 6, 1, 1, 1], and [7, 8, 9, 1, 1, 1]. Wherein, the imaging omics features and the multiple groups of convolution features are the imaging omics features and the multiple groups of convolution features corresponding to the same lung image to be processed.

[0206] In the embodiments of the present disclosure and other possible embodiments, whether based on the preset recognition model, using the multiple projection images to complete the COPD recognition of the lung image to be processed; or based on the preset recognition model, using the multiple sets of recognition features to complete the COPD recognition of the lung image to be processed; it is necessary to first train the preset recognition model, and then complete the COPD recognition of the lung image to be processed based on the trained preset recognition model. At the same time, the method for training the preset recognition model is a technical means commonly used by those skilled in the art and will not be described in detail in this disclosure.

[0207] Based on the above, in the present disclosure, the method for identifying COPD in the lung image to be processed based on the preset recognition model and using the multiple groups of recognition features respectively includes: inputting the multiple groups of recognition features into the preset recognition model respectively to obtain multiple second classification results, and performing statistics on the results of COPD and non-COPD in the multiple second classification results to obtain a third statistical result of COPD and a fourth statistical result of non-COPD; if the third statistical result is greater than the fourth statistical result, it is determined that the patient has COPD; otherwise, it is determined that the patient does not have COPD.

[0208] In the embodiments of the present disclosure and other possible embodiments, the multiple groups of convolution features and preset imaging genomics features corresponding to the three projection images (cross-sectional projection image, coronal projection image, and sagittal projection image) are spliced ​​to obtain three groups of identification features; the three groups of identification features are input into a preset recognition model, and the preset recognition model inputs the corresponding three second classification results; the results of COPD and non-COPD in the three second classification results are statistically analyzed to obtain third statistical results (2) of COPD and fourth statistical results (1) of non-COPD; at this time, if the third statistical results (2) are greater than the fourth statistical result (1), it is determined that the patient has COPD.

[0209] In an embodiment of the present disclosure, the lung image includes: an inspiratory phase lung image and an expiratory phase lung image; performing a registration operation on the inspiratory phase lung image and the expiratory phase lung image to obtain displacement parameters corresponding to inspiration to expiration; and fusing the set local features, global features, and displacement parameters to obtain fused features.

[0210] In an embodiment of the present disclosure, before performing the registration operation of the inspiratory phase lung image and the expiratory phase lung image, lung region segmentation is performed on the inspiratory phase lung image and the expiratory phase lung image, respectively, to obtain corresponding inspiratory phase lung region images and expiratory phase lung region images; and performing the registration operation of the inspiratory phase lung region and the expiratory phase lung region to obtain displacement parameters corresponding to inspiration to expiration.

[0211] In the embodiments of the present disclosure and other possible embodiments, the registration algorithm may use an elastic registration algorithm or utilize the VGG network (VGG-net) in deep learning for registration, such as the registration model introduced in the paper Deformable image registration using convolutional neural networks or the U network (U-net), such as the registration model introduced in the paper Pulmonary CT Registration through Supervised Learning with Convolutional Neural Networks. This disclosure does not limit the specific registration algorithm.

[0212] In the embodiment of the present disclosure and other possible embodiments, the displacement parameter corresponding to the transition from inhalation to exhalation includes at least a movement distance, which can be calculated using a Euclidean distance formula. In the embodiment of the present disclosure, the method of fusing the set local features, global features, and displacement parameters to obtain a fused feature includes concatenating the set local features, global features, and displacement parameters to obtain a fused feature.

[0213] In addition, the embodiment of the present disclosure also discloses a lung image processing method for identifying chronic obstructive pulmonary disease. The lung image processing method includes: obtaining a preset classification model, a lung image to be processed and its corresponding lung area image and airway image; based on the preset classification model, using the lung image to be processed, the lung area image and the airway image to obtain corresponding first classification results, second classification results and third classification results; based on the first classification result, the second classification result and the third classification result, identifying chronic obstructive pulmonary disease.

[0214] Obtain a preset classification model, a lung image to be processed, and its corresponding lung region image and airway image.

[0215] In the embodiments of the present disclosure and other possible embodiments, the lung image may be one or more of a CT image, an MR image, or a DR image. For another example, in the embodiments of the present disclosure and other possible embodiments, a lung image to be processed is first acquired. The lung image to be processed may be slice scan data obtained from an imaging device, such as a CT machine. The lung image to be processed may be a CT image. Furthermore, the lung image to be processed may also be an MRI lung image or a CT-PET lung image. Those skilled in the art may select an appropriate lung image as needed.

[0216] In the present disclosure, before obtaining the lung region image and airway image corresponding to the lung image to be processed, the method includes: obtaining the lung image to be processed; performing lung region segmentation and airway extraction on the lung image to be processed respectively to obtain the lung region image and airway image.

[0217] In the embodiments of the present disclosure and other possible embodiments, airway extraction (airway segmentation) of the lung image to be processed can be performed using the "Deep Airway Segmentation" module of Mimics software (Materialise, Belgium) to semi-automatically extract a three-dimensional airway tree from the lung image to be processed to obtain an airway image.

[0218] In the embodiments of the present disclosure and other possible embodiments, the lung region of the lung image to be processed is segmented, and the nnU-Net model can be used to segment the lung region of the lung image to be processed. Isensee et al. proposed the nnU-Net model for biomedical image segmentation, which provided the best performance in the segmentation task of the BraTS 2020 challenge. The model can be automatically configured, including preprocessing, network structure, training and post-processing. The present disclosure pre-trained the nnU-Net model on the dataset of the LOLA11 challenge, and fine-tuned it on the dataset of the present disclosure to complete the lung region segmentation task and obtain a lung region image. For those skilled in the art, fine-tuning the pre-trained model to complete the lung region segmentation task is a common technical means used by those skilled in the art, and there are no technical obstacles. It will not be described in detail in the embodiments of the present disclosure.

[0219] Based on the preset classification model, the lung image to be processed, the lung region image and the airway image are used to obtain corresponding first classification results, second classification results and third classification results.

[0220] In the embodiments of the present disclosure and other possible embodiments, the preset classification model may be a classification model based on deep learning, for example, one or more classification models such as VGG16, VGG19, InceptionV3, Xception, MobileNet, AlexNet, LeNet, ZF_Net, ResNet18, ResNet34, ResNet50, ResNet_101, and ResNet_152.

[0221] For example, the lung image to be processed can be input into the trained VGG16 to obtain the corresponding first classification result, the lung area image can be input into the trained ResNet18 to obtain the corresponding second classification result, and the airway image can be input into the trained AlexNet to obtain the corresponding third classification result.

[0222] At the same time, the present disclosure proposes an image-based classification method, which can fully fuse the features extracted based on the image and the corresponding weights to obtain excellent classification features. Specifically, in the present disclosure, the method of obtaining the corresponding first classification result, second classification result and third classification result based on the preset classification model, using the lung image to be processed, the lung region image and the airway image, includes: based on the preset classification model, extracting features from the lung image to be processed, the lung region image and the airway image, respectively, to obtain the corresponding first group of features, second group of features and third group of features; calculating the weight matrices corresponding to the first group of features, the second group of features and the third group of features respectively; fusing the first group of features, the second group of features and the third group of features and the corresponding weight matrices respectively to obtain the first classification feature, the second classification feature and the third classification feature; obtaining the corresponding first classification result, the second classification result and the third classification result respectively based on the first classification feature, the second classification feature and the third classification feature. Wherein, the first classification result, the second classification result and the third classification result are suffering from COPD or not suffering from COPD.

[0223] For example, the preset classification model selects VGG16, and based on VGG16, feature extraction is performed on the lung image to be processed, the lung region image, and the airway image to obtain the corresponding first set of features, second set of features, and third set of features; the first set of features, the second set of features, and the third set of features are input into the fully connected layer (FC) respectively to obtain the first weight matrix, the second weight matrix, and the third weight matrix corresponding to the first set of features, the second set of features, and the third set of features; the first set of features, the second set of features, and the third set of features are multiplied by the corresponding first weight matrix, the second weight matrix, and the third weight matrix respectively to obtain the first classification feature, the second classification feature, and the third classification feature. Further, based on the first classification feature, the second classification feature, and the third classification feature, the corresponding first classification result, the second classification result, and the third classification result are obtained.

[0224] In the embodiments of the present disclosure and other possible embodiments, the classification model used to generate the first classification result, the second classification result and the third classification result can be a classification model based on machine learning, such as one or more of support vector machine (SVM), multi-layer perceptron (MLP), random forest (RF), K-nearest neighbor (KNN), logistic regression (LR), decision tree (DT), gradient boosting (GB), linear discriminant analysis (LDA), etc.; it can be a deep learning classification model.

[0225] For example, the first classification feature, the second classification feature and the third classification feature are respectively input into a multi-layer perceptron (MLP) to obtain a first classification result, a second classification result and a third classification result corresponding to the first classification feature, the second classification feature and the third classification feature.

[0226] In the present disclosure, the method of fusing the first group of features, the second group of features, the third group of features and the corresponding weight matrices to obtain first classification features, second classification features and third classification features includes: performing matrix multiplication operations on the first group of features, the second group of features, the third group of features and the corresponding weight matrices to obtain first classification features, second classification features and third classification features.

[0227] For example, the dimensions of the first group of features, the second group of features, and the third group of features are k1×N1, k2×N2, and k3×N3, respectively; the dimensions of the first group of features, the second group of features, and the third group of features and the corresponding first weight matrix, second weight matrix, and third weight matrix are k1×1, k2×1, and k3×1, respectively; matrix multiplication operations are performed on the first group of features, the second group of features, and the third group of features and the corresponding first weight matrix, second weight matrix, and third weight matrix, respectively, to obtain first classification features, second classification features, and third classification features with dimensions of 1×N1, 1×N2, and 1×N3.

[0228] Figure 6 Schematic diagram of the network structure corresponding to the preset classification model according to the embodiment of the present disclosure is shown. Figure 6 As shown in Figure 2, the proposed preset classification model is essentially a model corresponding to the multiple instance (MIL) method of the attention mechanism. Figure 6 Only the first classification result corresponding to the lung image to be processed is displayed. By replacing the lung image to be processed with the lung region image and the airway image, the second classification result and the third classification result corresponding to the lung region image and the airway image can be obtained.

[0229] In the embodiments of the present disclosure and other possible embodiments, some features of multi-instance learning (MIL) are suitable for medical applications. As a weakly supervised learning method, MIL generally includes instance-level and individual-level methods. In the instance-level method, all instances are considered to have the same contribution to the prediction of the individual label, and the prediction of the instance is made into an individual prediction through aggregation and voting, while the individual method is designed to classify the individual directly. The individual method can reduce the workload of annotation because there is no need to label pixels and instances.

[0230] In the embodiments of the present disclosure and other possible embodiments, the present disclosure proposes an attention-based MIL method, for example, the lung image to be processed, such as a CT image, combines the intensity of the CT image with the morphology of the airway and lung area to identify COPD. The main contributions of this study are as follows. First, an attention mechanism MIL model is constructed to classify subjects (individuals) and weight the selected slices (instances) for each subject using the attention mechanism. Secondly, multi-perspective snapshots of the three-dimensional airway tree and lung area are used as morphological information to improve the recognition performance of COPD. Finally, a logistic regression (LR) model is used to integrate the lung image to be processed, the lung area image and the airway image pre-classification to produce the final output. As a weakly supervised learning method, the attention mechanism-guided MIL method has the potential to provide an effective tool for the early detection of COPD.

[0231] exist Figure 6 In the example, the feature extraction module is used to convert the lung image to be processed into a k-dimensional embedding vector (the first classification feature) Then, using the multilayer perceptron u T tanh( T ) and a softmax layer produces an attention weight matrix α from the embedding vector H with k dimensions. Finally, a joint envelope representation (first classification feature) z′ is generated by applying a function f(·) to the aggregated k instance-level feature vectors, which is defined as follows.

[0232] α=Softmax[u T tanh(WH T ) (1)

[0233]

[0234] in, includes k instance feature vectors, and is the learning parameter of the MIL module, and h is the dimension of the hidden layer.

[0235] Finally, the fully connected layer FC is the output layer, which is divided into two categories. Cross entropy loss is used as the loss function in the model.

[0236]

[0237] Among them, y i represents the label of sample i, p i Represents the probability of predicting a positive output (suffering from COPD).

[0238] At the same time, in the embodiments of the present disclosure and other possible embodiments, the lung region image or the airway image can also obtain the second classification feature and the third classification feature through the above method or model, which will not be described in detail in the present disclosure.

[0239] In the embodiments of the present disclosure and other possible embodiments, the method of performing feature extraction on the lung image to be processed, the lung region image, and the airway image, respectively, to obtain corresponding first, second, and third groups of features, includes: obtaining a set feature extraction model; and based on the set feature extraction model, performing feature extraction on the lung image to be processed, the lung region image, and the airway image, respectively, to obtain corresponding first, second, and third groups of features.

[0240] In the embodiments of the present disclosure and other possible embodiments, the set feature extraction model may be a deep learning-based feature extraction model. For example, one or more classification models such as VGG16, VGG19, InceptionV3, Xception, MobileNet, AlexNet, LeNet, ZF_Net, ResNet18, ResNet34, ResNet50, ResNet_101, and ResNet_152 may be used. Specifically, before classification, the feature extraction process saves the corresponding features as the first, second, and third sets of features.

[0241] For example, the lung image to be processed can be input into the trained VGG16 (delete the classification layer) to obtain the corresponding first set of features, the lung area image can be input into the trained ResNet18 (delete the classification layer) to obtain the corresponding second set of features, and the airway image can be input into the trained AlexNet (delete the classification layer) to obtain the corresponding third set of features.

[0242] For another example, the lung image to be processed, the lung region image, and the airway image can be respectively input into the trained VGG16 (delete the classification layer) to obtain the corresponding first set of features, second set of features, and third set of features.

[0243] In the embodiments of the present disclosure and other possible embodiments, the method of performing feature extraction on the lung image to be processed, the lung region image and the airway image respectively to obtain corresponding first group of features, second group of features and third group of features includes: obtaining a preset feature extraction model, training the preset feature extraction model to obtain a trained feature extraction model; based on the trained feature extraction model, performing feature extraction on the lung image to be processed, the lung region image and the airway image respectively to obtain corresponding first group of features, second group of features and third group of features.

[0244] In the embodiments of the present disclosure and other possible embodiments, the above-mentioned CNN model requires a large amount of labeled data to train its weights and biases. However, in medical image tasks, it is difficult to meet this condition. Transfer learning has been proven to be a better method than training from scratch. Here, the modified VGG-16 (classification layer deleted) feature extraction module was pre-trained on the ImageNet dataset (1.2M training data) and fine-tuned on our dataset. The present disclosure also adopts the same transfer learning strategy as VGG-16 for all other comparison networks.

[0245] For example, the feature extraction part uses a pre-trained VGG-16 model. In the embodiment of the present disclosure, only the convolution layer part of VGG-16 is retained, and the last three fully connected (FC) layers are replaced by the attention MIL pooling module in our model to obtain the modified retained VGG-16 structure. It includes 13 convolutional layers and 5 maximum pooling layers. The 13 convolutional layers constitute four convolution blocks. The conv1 block has two convolutional layers in sequence, and the size of the resulting feature map is the same as the input image, with a dimension of 64, and the number of convolution kernels used in each convolution layer is also 64. The conv2 block also has two convolutional layers arranged in sequence, and its output size is 128. The conv3, conv4 and conv5 blocks have three convolutional layers in each block in sequence, with 256, 512 and 512 convolution kernels respectively. Each conv block is max pooled, which reduces the size of the feature map by half.

[0246] In the present disclosure, before performing feature extraction on the lung image to be processed, the lung region image and the airway image respectively to obtain the corresponding first set of features, second set of features and third set of features, the first image and / or second image and / or third image corresponding to the lung image to be processed and / or the lung region image and / or the airway image for feature extraction are respectively determined, and the determination method includes: deleting non-lung images in the lung image to be processed to obtain an image containing the lungs; extracting the image containing the lungs according to the set number of acquired features to obtain the first image corresponding to the feature extraction; performing three-dimensional reconstruction on the lung region image to obtain a three-dimensional lung region image; photographing the three-dimensional lung region image according to multiple first set angles to obtain multiple second images corresponding to the first two-dimensional snapshots; performing three-dimensional reconstruction on the airway image to obtain a three-dimensional airway image; photographing the three-dimensional airway image according to multiple second set angles to obtain a third image corresponding to the second two-dimensional snapshots. Among them, the first set angle can be one or more of the front view, back view and 45-degree oblique view of the three-dimensional airway (tree) image or other possible views; the second set angle can be one or more of the front, back, left, right, top and bottom views or views at other angles.

[0247] In the embodiments of the present disclosure and other possible embodiments, a method for extracting an image containing lungs according to a set number of acquired images to obtain a first image corresponding to feature extraction includes: dividing the image containing lungs into the set number of sub-parts along the longitudinal direction; and randomly selecting (extracting) a slice from each sub-part to obtain the first image corresponding to feature extraction.

[0248] For example, the lung image to be processed may be a lung image to be processed, in which the extra-pulmonary part of the CT image sequence is deleted and only the part containing the lung area is retained. Then, the preprocessed CT image is further divided into k (set number) sub-parts along the longitudinal direction. A CT slice is randomly selected from each sub-part and defined as an instance. These k instances constitute a new individual, which is used as the input of the MIL model (multi-instance model of attention mechanism). Then, the k instance images are scaled to 224×224 pixels and converted into .npy format using Python (version 3.9), SimpleITK (version 2.1.1) and NumPy (version 1.22.1). Each .npy file corresponds to an individual.

[0249] Snapshots were taken based on the above segmentation. Snapshots of the front view, back view, and oblique 45-degree view (i.e., F, B, and I views) of the three-dimensional airway tree were obtained. In addition, two-dimensional snapshots of the front, back, left, right, upper, and lower views of the lung region (i.e., A, P, L, R, S, and D views, respectively) were obtained. Multi-view snapshots were generated using a Python script program using the three-dimensional display module of Slicer (https: / / www.slicer.org / ). Automatic batch processing was performed on all COPD patients and HC subjects. The two-dimensional snapshots used in this disclosure are grayscale images, the size of which was set to 224×224 in all snapshots.

[0250] Based on the first classification result, the second classification result and the third classification result, COPD is identified.

[0251] The method for identifying COPD based on the first classification result, the second classification result and the third classification result includes: respectively determining the first probability value, the second probability value and the third probability value corresponding to the first classification result, the second classification result and the third classification result; performing regression analysis on the first probability value, the second probability value and the third probability value to identify COPD.

[0252] The execution subject of the acute exacerbation prediction method may be an acute exacerbation prediction device, for example, the acute exacerbation prediction method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the acute exacerbation prediction method may be implemented by a processor calling computer-readable instructions stored in a memory.

[0253] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0254] The embodiment of the present invention also discloses a device for predicting acute exacerbation, which includes: an acquisition unit for acquiring the number of acute exacerbations corresponding to a first time and a second time, clinical text data and / or lung images of multiple patients at the first time; a determination unit for determining a training label based on the number of acute exacerbations corresponding to the first time and the second time; a training unit for training a preset classifier using the clinical text data and / or the lung images and the corresponding training labels; and a prediction unit for completing the prediction of the acute exacerbation of the patient to be predicted at the second time based on the trained preset classifier and using the clinical text data and / or lung images of the patient to be predicted at the first time.

[0255] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method for predicting acute exacerbation described in the above method embodiments. Its specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0256] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions, when executed by a processor, implement the above-mentioned method for predicting acute exacerbation. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0257] The present disclosure also provides an electronic device comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above-mentioned method for predicting acute exacerbation. The electronic device may be provided as a terminal, a server, or other device.

[0258] Figure 7 8 is a block diagram of an electronic device 800 according to an exemplary embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0259] Reference Figure 7 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0260] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0261] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0262] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0263] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0264] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0265] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0266] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and temperature changes of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0267] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0268] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0269] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions. The computer program instructions can be executed by the processor 820 of the electronic device 800 to perform the above method.

[0270] Figure 8 1 is a block diagram of an electronic device 1900 according to an exemplary embodiment. For example, the electronic device 1900 may be provided as a server. Figure 8 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described method.

[0271] The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0272] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0273] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0274] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0275] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0276] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0277] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0278] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0279] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0280] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0281] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technical improvements to existing technologies, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for predicting acute exacerbation, characterized in that: include: Obtaining the number of acute exacerbations corresponding to a first time and a second time for a plurality of patients, clinical text data and / or lung images at the first time; Determining whether the clinical text data corresponding to the patient to be predicted includes lung function data; If included, identifying or grading chronic obstructive pulmonary disease in the patient to be predicted based on the lung function data; Otherwise, obtaining a lung image of the patient to be predicted; Using the lung image to identify or grade chronic obstructive pulmonary disease in the patient to be predicted, the method includes: determining the number of radiomic features corresponding to the lung parenchyma image of the lung image; generating a radiomic feature map using the radiomic features, performing a convolution operation on the radiomic feature map to obtain convolution features corresponding to the number; fusing the radiomic features and the convolution features to obtain a fusion feature; determining a risk factor feature of a graph convolutional network based on the fusion feature; performing a splicing operation on the risk factor feature and the fusion feature to obtain a splicing feature; and using the splicing feature to identify or grade chronic obstructive pulmonary disease in the patient to be predicted based on the graph convolutional network. If the patient to be predicted suffers from the chronic obstructive pulmonary disease or the chronic obstructive pulmonary disease of the patient to be predicted reaches a set grade, the patient to be predicted is predicted; a training label is determined according to the number of acute exacerbations corresponding to the first time and the second time; wherein, the determining of the training label according to the number of acute exacerbations corresponding to the first time and the second time includes: if the number of acute exacerbations corresponding to the second time is greater than the number of acute exacerbations corresponding to the first time, the training label is acute exacerbation; otherwise, the training label is non-acute exacerbation; Training a preset classifier using the clinical text data at the first time and / or the lung image at the first time and the corresponding training labels; Based on the trained preset classifier, the prediction of acute exacerbation of the patient to be predicted at the second time is completed using the clinical text data and / or lung images of the patient to be predicted at the first time.

2. The prediction method according to claim 1, characterized in that The training of a preset classifier using the clinical text data and / or the lung image and the corresponding training labels includes: Obtaining corresponding first local features and first global features according to the clinical text data and / or the lung image; A preset classifier is trained using the first local feature, the first global feature and the corresponding training labels.

3. The prediction method according to claim 2, characterized in that The training of a preset classifier using the first local feature, the first global feature and the corresponding training labels includes: Performing a fusion operation on the first local feature and the first global feature to obtain a first fused feature; The preset classifier is trained using the first fusion feature and the corresponding training label.

4. The prediction method according to any one of claims 1 to 3, characterized in that: The preset classifier based on the training uses the clinical text data and / or lung image of the patient to be predicted at the first time to complete the prediction of the acute exacerbation of the patient to be predicted at the second time, including: Obtaining corresponding second local features and second global features according to the clinical text data and / or lung image of the patient to be predicted at the first time; Based on the trained preset classifier, the second local feature and the second global feature are used to complete the prediction of the acute exacerbation of the patient to be predicted at the second time.

5. The prediction method according to claim 4, characterized in that The preset classifier based on the training uses the second local feature and the second global feature to complete the prediction of the acute exacerbation of the patient to be predicted at the second time, including: performing a fusion operation on the second local feature and the second global feature to obtain a second fused feature; Based on the trained preset classifier, the second fusion feature is used to complete the prediction of the acute exacerbation of the patient to be predicted at the second time.

6. The prediction method according to claim 4, characterized in that The obtaining of the corresponding second local feature and second global feature based on the clinical text data and / or lung image of the patient to be predicted at the first time includes: Based on the first local features corresponding to the clinical text data and / or lung images of the multiple patients at the first time, selecting a feature identical to the first local feature from the features corresponding to the clinical text data and / or lung images of the patient to be predicted at the first time, to obtain a second local feature corresponding to the patient to be predicted; Acquire a model for obtaining a corresponding first global feature according to the clinical text data and / or the lung image; Based on the model corresponding to the first global feature, the corresponding second global feature is obtained using the clinical text data and / or lung image of the patient to be predicted at the first time.

7. The prediction method according to claim 5, characterized in that The obtaining of the corresponding second local feature and second global feature based on the clinical text data and / or lung image of the patient to be predicted at the first time includes: Based on the first local features corresponding to the clinical text data and / or lung images of the multiple patients at the first time, selecting a feature identical to the first local feature from the features corresponding to the clinical text data and / or lung images of the patient to be predicted at the first time, to obtain a second local feature corresponding to the patient to be predicted; Acquire a model for obtaining a corresponding first global feature according to the clinical text data and / or the lung image; Based on the model corresponding to the first global feature, the corresponding second global feature is obtained using the clinical text data and / or lung image of the patient to be predicted at the first time.

8. A prediction device for acute exacerbation, characterized in that: include: an acquisition unit, configured to acquire the number of acute exacerbations corresponding to a first time and a second time for a plurality of patients, clinical text data and / or lung images at the first time; Determining whether the clinical text data corresponding to the patient to be predicted includes lung function data; If included, identifying or grading chronic obstructive pulmonary disease in the patient to be predicted based on the lung function data; Otherwise, obtaining a lung image of the patient to be predicted; The lung image is used to identify or grade the chronic obstructive pulmonary disease of the patient to be predicted, comprising: determining the number corresponding to the radiomics features of the lung parenchyma image of the lung image; generating an radiomics feature map using the radiomics features, performing a convolution operation on the radiomics feature map to obtain convolution features corresponding to the number; fusing the radiomics features and the convolution features to obtain a fusion feature; determining a risk factor feature of a graph convolutional network based on the fusion feature; performing a splicing operation on the risk factor feature and the fusion feature to obtain a splicing feature; based on the graph convolutional network, using the splicing feature to achieve identification or grading of the chronic obstructive pulmonary disease of the patient to be predicted; and if the patient to be predicted suffers from the chronic obstructive pulmonary disease or the chronic obstructive pulmonary disease of the patient to be predicted reaches a set grade, then predicting the patient to be predicted; a determining unit, configured to determine a training label based on the number of acute exacerbations corresponding to the first time and the second time; wherein the determining of the training label based on the number of acute exacerbations corresponding to the first time and the second time includes: if the number of acute exacerbations corresponding to the second time is greater than the number of acute exacerbations corresponding to the first time, the training label is acute exacerbation; otherwise, the training label is non-acute exacerbation; a training unit, configured to train a preset classifier using the clinical text data at the first time and / or the lung image at the first time and the corresponding training labels; The prediction unit is used to complete the prediction of acute exacerbation of the patient to be predicted at the second time based on the trained preset classifier and using the clinical text data and / or lung image of the patient to be predicted at the first time.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method for predicting acute exacerbation according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method for predicting acute exacerbation according to any one of claims 1 to 7 is implemented.

Images