Method for generating future lung LDCT image by introducing conceptual coding CSTA-GAN network model

Abstract

The invention provides a method for generating future lung LDCT images by introducing a concept-coded CSTA-GAN network model, and has the following advantages: a CSTA-GAN deep learning network structure introducing clinical concept information is designed, and compared with a traditional generative network, the network introduces a time and space attention mechanism structure, so that the time and space attention mechanism structure is optimized, and the generation efficiency is improved. The guide model focuses on changing cross-sectional images and nodule areas in the LDCT image; conceptual coding is introduced into a CSTA-GAN network model, and conceptual information coding is carried out on the existing information of a patient by combining iconography expression, screening diagnosis results and epidemiological information, so that a future LDCT image generated by the model considers the multi-factor comprehensive influence of the life and the environment of the patient; the CSTA-GAN network generates a future LDCT image of a patient through LDCT of one-time lung cancer screening, so that a diagnosis physician can preliminarily judge the change of future pulmonary nodules of the patient, false positive and false negative results existing during lung cancer diagnosis are reduced, and meanwhile, the problems of radiation exposure, information pressure and cost during reexamination of the patient are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of concept coding and medical image generation, and in particular relates to a method for generating future lung LDCT images by using a CSTA-GAN network model that introduces concept coding. Background Art

[0002] Lung cancer ranks first in the incidence and mortality of all cancers in men and women. Among patients diagnosed with cancer, 28% of deaths are caused by lung cancer. Research institutions have shown that the use of low-dose computed tomography (LDCT) for lung cancer screening can effectively detect 60% to 80% of early lung cancer, and early treatment can effectively reduce the lung cancer mortality rate of patients by 20% to 31%.

[0003] However, there are certain drawbacks in the use of LDCT for lung cancer screening. There are false positive results for some nodules that are suspected but not actually tumors, and some slow-growing lesions that are found may be overdiagnosed; some patients often need regular follow-up examinations after the nodules are found. Although LDCT is used for lung cancer screening, there is also a certain amount of radiation. The accumulated radiation dose from multiple screenings may have potential effects on the body, and will also cause certain psychological pressure and time costs to patients. At this stage, deep learning models are also used to model LDCT images to improve the screening rate of early lung cancer in the lung cancer screening stage. However, only using image data to analyze pixel information ignores the patient's clinical information, such as smoking history, medical history, and family genetic factors, which may cause the model to predict inaccurate results without considering the patient's clinical information.

[0004] In summary, considering that the existing model does not take into account the patient's clinical information, the radiation exposure during follow-up patient examinations, and the psychological pressure and time cost of patients during the examination, there is an urgent need for a generative model that introduces conceptual information to solve the above problems and generate future lung LDCT images. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a method for generating future lung LDCT images based on the above-mentioned background by using a CSTA-GAN network model that introduces concept coding.

[0006] According to the technical solution of the present invention, a method for generating future lung LDCT images by introducing a generative model with concept coding specifically comprises the following steps:

[0007] S1. Data collection: collection and data cleaning of LDCT images of patients undergoing lung cancer screening, clinical information of patients, epidemiological information and screening report data cohorts;

[0008] S2. Data feature engineering, including ROI area delineation, radiomics feature extraction and screening, diagnosis report information extraction, and conceptual information coding of patient clinical information, epidemiology, and diagnosis results;

[0009] S3, model construction, design a generative adversarial network embedded with a spatiotemporal attention mechanism, introduce the concept information of clinical information, epidemiology and diagnostic reports, design a loss function suitable for introducing a concept encoding generation model, and generate a CSTA-GAN network model;

[0010] S4. Model training: Use conventional CT cohorts to pre-train the model, use transfer learning to train the LDCT data cohort 0 for lung cancer screening, and fine-tune the model to optimize model performance by combining numerical evaluation and human subjective evaluation indicators;

[0011] S[, model result display, compares the LDCT image of the subject at screening with the LDCT image generated D days later, marks the suspected nodule area, and assists the physician in diagnosing the future lung changes of the subject.

[0012] Further, according to one aspect of the present invention, a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding is provided, wherein the extracted LDCT radiomics features include morphological features, texture, first-order statistics, and high-order features. The extracted features are subjected to feature dimension reduction, noise removal, and redundant features removal by LASSO regression and statistical methods to obtain radiomics features that are highly correlated with lung cancer; optical character recognition (OCR) is used to identify the features of the lung cancer. Abstract: This paper proposes a novel method for the detection and management of lung cancer in patients with lung cancer. The optical character recognition (OCR) technology is used to extract text content from the patient's lung cancer screening diagnosis report, and the neural network method is used to classify the concepts of nodule lesions based on the imaging manifestations and diagnostic opinions. The patient's imaging genomics information, clinical information, epidemiological data information and diagnostic report information are integrated, and the data are divided into discrete, continuous and disease types according to the data value type. Concept information is encoded through One-Hot encoding, standardization, ICD-10 encoding and Word2Vec to encode discrete, continuous and disease-type data, and the concept information of different data types is mapped into the vector space, which is used as a control generation network to adjust the changes in the nodule area in the lung LDCT images. The design introduces a spatiotemporal attention mechanism to guide the network model to focus on the nodule area in LDCT and highlight the changing characteristics of the lung nodule area at different levels.

[0013] Further, according to one aspect of the present invention, a method for generating future lung LDCT images by using a CSTA-GAN network model that introduces conceptual coding is provided. In step S1, the LDCT image data queue for lung cancer screening of patients includes CT images taken at two different times of the screened person, and data cleaning includes: evaluating the collected lung cancer screening LDCT image queue, removing LDCT image data with poor image quality, missing or incomplete chest key areas in the image, non-compliance with the standard acquisition protocol, image artifacts caused by patient movement, repeated or redundant data, and damaged or unreadable image data; for the collected patient clinical information and epidemiological information, removing missing values, repeated values, and abnormal values ​​in the data of the same patient, checking the consistency of data in the same field, and ensuring the logical correctness of the information; integrating the above-processed data, removing the data of incomplete patient information, and ensuring the integrity, correctness, and uniqueness of the cleaned data.

[0014] Furthermore, according to one aspect of the present invention, a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding is provided. In step S2, Python software is used to convert the lung LCTD in Dicom format into nii.gz, and ITK-SNAP medical image processing software is used to outline each layer of the lung image in LDCT. The outline method is to outline the lung boundary area, and the outlined labels are finally saved in nii format.

[0015] Further, according to one aspect of the present invention, a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding is provided. In step S2, extracting information from a patient's diagnosis report using OCR technology mainly includes:

[0016] Imaging manifestations: location of the detected pulmonary nodules (upper lobe and lower lobe of the left lung, upper lobe, middle lobe and lower lobe of the right lung), size (long axis, short axis and volume), morphological description (ground glass, solid, cystic), edge description (smooth, burred, lobed), and other abnormal descriptions (pulmonary inflammation, pleural effusion, fibrosis, chronic obstructive pulmonary disease);

[0017] Diagnostic opinion: whether the diagnostic expert has found suspected lung cancer lesions, such as "no abnormalities found, regular reexamination is recommended", "small lung nodules found, considered to be benign, regular follow-up is recommended", "lung nodules found, the nature is to be determined, malignant possibility cannot be ruled out, further examination is recommended", the time interval for regular reexamination, and the items for further examination;

[0018] After the diagnostic report information is extracted, the text is vectorized using Word2Vec technology, and a convolutional neural network is built to classify the text. According to the category of information, such as location, morphology, edge, and other abnormalities, the information is written into the corresponding Excel header.

[0019] Further, according to one aspect of the present invention, a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding is provided. In step S2, different concept information encodings are performed according to the types of data in the extracted diagnostic report information, patient clinical information, and epidemiological information:

[0020] One-Hot encoding was performed on discrete variables such as gender, occupational exposure (such as asbestos, radon, coal smoke and tar, polycyclic aromatic hydrocarbons), and family cancer history;

[0021] The continuous variables of age, weight, biochemical indexes, and blood routine were standardized and dimensionless.

[0022] The conceptual type information of pulmonary fibrosis, pulmonary inflammation, chronic obstructive pulmonary disease, and family history of cancer was coded according to the International Statistical Classification of Diseases and Related Health Problems (ICD-10), and then Word2Vec text vectorization was performed on it.

[0023] According to the lung cancer guidelines, the concepts of lung nodule location, morphology, and margins are numerically coded, and the numerical range is defined as 0-1, where 0 represents completely smooth, 1 represents obvious burrs, and intermediate values ​​represent different degrees of burr manifestations.

[0024] Furthermore, according to one aspect of the present invention, a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding is provided. In step S3, a temporal and spatial attention mechanism structure is added as a model framework for generating future lung LDCT images based on a generative adversarial network model structure. The spatial attention mechanism structure enables the model to focus on the distribution of spatial information importance of nodule areas in lung CT, guiding the model to focus on nodule areas in the image; the temporal attention mechanism is used to dynamically assign weights to a series of continuous tomographic images in LDCT images, so that the model pays more attention to tomographic images in which nodules appear or may appear in the lungs, named Concept Feature Spatial-Temporal Attention Generative Adversarial Network (CSTA-GAN).

[0025] Further, according to one aspect of the present invention, a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding is provided, wherein the CSTA-GAN network includes three loss functions, namely, a discriminator loss function L between the generated lung LDCT image and the real image; Discriminator , the perceptual loss function L of the feature map Perceptual and the radiomics mean absolute error loss function L Radiomics , the expressions of the three loss functions are as follows:

[0026]

[0027] Among them, G(x predict_LDCT ) is the future lung LDCT image generated by the CSTA-GAN network, D(x true_LDCT ) is the result of the discriminator’s judgment on the real CT image, D(G(x predict_LDCT )) is the result of the discriminator’s judgment on the future lung LDCT images generated by the network, and the perceptual loss function L of the feature map Perceptual The expression is:

[0028]

[0029] in, is the characteristic map of the lth layer of the Tth layer of the patient's real future LDCT image, The l-th layer feature map of the T-th layer of the CT image of the patient at the future time generated by the CSTA-GAN network, and They are the number of channels, height and width of the feature map, and the mean absolute error loss function L of radiomics Radiomics The expression is:

[0030]

[0031] Among them, n is the number of radiomics features after feature screening, is the i-th radiomics feature value in the patient’s real image, is the i-th radiomics feature value in the CT image of the patient at the future time generated by the CSTA-GAN network.

[0032] Further, according to one aspect of the present invention, a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding is provided. In step S4, since the number of data teams for lung cancer screening that can meet multiple screening needs is far less than that of conventional lung CT data teams, the conventional lung CT data queue is used for pre-training of the CSTA-GAN network model based on CT images for different examination purposes, and for model initialization, thereby solving the problem of scarcity of LDCT image data during lung cancer screening.

[0033] Further, according to one aspect of the present invention, a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding is provided. In step S4, the model is initialized with a conventional lung CT data team, and CSTA-GAN training is performed in the screening low-dose LDCT data using a transfer learning method. The generated future lung LDCT images are comprehensively, accurately and quality evaluated using numerical evaluation and human subjective evaluation indicators. The model hyperparameters are adjusted according to the evaluation results, and the training is repeated to obtain a CSTA-GAN generative model with optimal generalization performance.

[0034] The numerical evaluation uses the mean squared error (MSE) to evaluate the average value of the square of the pixel value difference between the real LDCT and the generated LDCT image, and its expression is as follows:

[0035]

[0036] p ij is the pixel value at position (i, j) in the real lung LDCT image, The pixel value at (i, j) in the lung LDCT image generated by the CSTA-GAN network is finally summed up by the square of its difference and divided by the total number of pixels in the LDCT to obtain the mean square error.

[0037] The human subjective evaluation index uses preference options in which radiologists distinguish between real and generated LDCT images. If the diagnosis is correct, the score is 1, and if the judgment is wrong, the score is 0.

[0038] Furthermore, in step S5, the lung nodule area (if any lung nodules exist) in the future LDCT segmentation is generated using a deep learning (DL) model, and by providing a nodule edge mask area, the physician is assisted in providing the patient's future LDCT diagnosis results based on its development trend.

[0039] The method for generating future lung LDCT images by a CSTA-GAN network model that introduces concept coding of the present invention has the following advantages: first, the present invention designs a CSTA-GAN deep learning network structure that introduces clinical concept information. Compared with the traditional generation network, the network introduces a temporal and spatial attention mechanism structure to guide the model to focus on the changing tomographic images and nodule areas in the LDCT images; second, the concept coding is introduced into the CSTA-GAN network model, and the patient's existing information is encoded with concept information in combination with imaging manifestations, screening diagnosis results, and epidemiological information, so that the future LDCT images generated by the model take into account the comprehensive influence of multiple factors of the patient's life and environment; third, the CSTA-GAN network generates the patient's future LDCT images through a lung cancer screening LDCT, which can enable the diagnostic physician to have a preliminary judgment on the changes in the patient's future lung nodules, reduce the false positive and false negative results in the diagnosis of lung cancer, and reduce the radiation exposure, information pressure and cost problems of the patient during the follow-up examination. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a flowchart of a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention;

[0041] Figure 2 A schematic diagram of concept information extraction-processing-coding of a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention;

[0042] Figure 3 A schematic diagram of the structure of a spatial attention mechanism of a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention;

[0043] Figure 4 A schematic diagram of the structure of a temporal attention mechanism of a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention;

[0044] Figure 5 A schematic diagram of a CSTA-GAN network structure of a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention;

[0045] Figure 6 This is an example diagram of the generation effect of a method for generating future lung images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The embodiments described are intended to further illustrate some preferred embodiments of the present invention, but not all embodiments. Other embodiments based on the present invention made by professionals in the field without creative work all belong to the scope of protection of the present invention. The present invention will be further described below in conjunction with the accompanying drawings.

[0047] The inspection methods without specific conditions in the embodiments are usually carried out under conventional conditions or under conditions recommended by the manufacturer. Figure 1 .

[0048] like Figure 1 As shown, a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding includes:

[0049] Step S1, data collection, collection and data cleaning of LDCT image cohorts of patients' lung cancer screening, patient clinical information, epidemiological information and screening report data cohorts.

[0050] Specifically, the LDCT data collected during lung cancer screening of the subjects are divided into general risk group, high risk group and extremely high risk group. Among them, the general risk group: people aged between 50 and 77 years old, smoking 20 packs / year or more, quitting smoking for less than 15 years, or people with other lung cancer risk factors but not serious, such as radon exposure, occupational exposure, family history of lung cancer, history of chronic lung disease, and the data in their LDCT data queue needs to be at least 1 year or more; high risk group: people who smoke ≥30 packs / year, or have smoked ≥30 packs / year but quit smoking for less than 15 years, and have a history of occupational exposure, such as asbestosis. , beryllium, uranium, radon, and have a history of chronic obstructive pulmonary disease, diffuse pulmonary fibrosis or pulmonary tuberculosis, and a first-degree relative with a family history of lung cancer. The data interval in their LDCT data cohort needs to be six months to one year; Extremely high-risk population: people who have been diagnosed with lung nodules, and the nodules have high-risk characteristics, or people who are closely related to lung cancer genetic genes, such as the subjects with EGFR, KRAS, BRAF gene mutations, ALK or ROS1 gene rearrangements, MET gene amplification or mutations, RET gene fusions. The data interval in their LDCT data cohort needs to be 3 to C months, and the shooting time needs to be re-examined according to the doctor's advice.

[0051] Generally, in this embodiment, the input image data needs to be LDCT image data in the original Dicom format.

[0052] Step S2, data feature engineering, the feature engineering operations include ROI area delineation, radiomics feature extraction and screening, diagnosis report information extraction, and conceptual information coding of patient clinical information, epidemiology, and diagnosis results.

[0053] Specifically, in order to ensure the data standardization and compatibility of LDCT, the LDCT image data was converted into NIfTI format (nii.gz format) using Python language to reduce the complex preprocessing process during image data processing. After the format conversion was completed, the ITK-SNAP medical image processing software was used to have two senior radiologists delineate the lesion area in the LDCT. The Dice coefficient was used to evaluate the average Dice similarity of the two physicians' delineation of the lung area, and its expression is:

[0054]

[0055] Among them, img A and img B The mask area of ​​the lung area in the same LDCT image drawn by doctors A and B, img A_i and img B _ i is the i-th layer in LDCT, n is the number of lung layers outlined in the LDCT, and when the calculated average Dice coefficient is ≥0.85, it indicates that the lung areas outlined by the two physicians have a high consistency. The third senior radiologist adjusts the lung mask areas outlined by the first two physicians to obtain the final lung area ROI mask data, which is saved in nii format for lung imaging feature extraction.

[0056] Step S3, model construction, design a generative adversarial network embedded with a spatiotemporal attention mechanism, introduce conceptual information of clinical information, epidemiology and diagnostic reports, and design a loss function suitable for introducing a concept encoding generation model.

[0057] Specifically, in this embodiment, the spatial attention structure can help the network model focus on the possible lesion areas, morphological edges and internal structures of abnormal tissues in the LDCT images, improve the model's ability to extract and analyze the features of the lung regions with changes, and suppress the model's attention to other normal areas; the temporal attention structure is mainly used to highlight the importance of faults with significant changes when extracting features from continuous LDCT images, so that the model can adapt to the important differences in corresponding faults at different time points, give high weights to layers that may have abnormalities, and weaken the model's attention to other unimportant layers; the Convolutional Neural Networks (CNN) in the network structure are used to extract the shape, texture, edge, low-order and high-order features of the lungs in the LDCT images; the main function of the Multilayer Perceptron (MLP) is to discover the hidden rules between the input data and the output data, and at the same time, the feature dimension of its output data can be changed according to the MLP structure to provide a high-quality data feature basis.

[0058] Among them, in the CSTA-GAN network, the role of the generator is to inversely transform the input LDCT data through the spatial, temporal attention mechanism and the features of the CNN network, fuse the features in the hidden space with the encoded conceptual features, and transform the fused features to generate future LDCT images; the role of the discriminator is to judge the authenticity between the LDCT generated by the generator and the real LDCT of the subject. If the discriminator discriminates the generated LDCT image as a real LDCT, the discriminator outputs 1, otherwise, if the discriminator discriminates the generated LDCT image as an LDCT image generated by the generator, the discriminator outputs 0, and both are evaluated using numerical evaluation and human subjective evaluation indicators. The error between the two is calculated according to the loss function defined in this embodiment, and back-propagation is performed from the output layer to the input layer by back-propagation, the gradient changes between each layer are recalculated, and the weight parameters in the network model are updated to make the generated LDCT closer to the real image data.

[0059] Step S4, model training, pre-train the model using a conventional CT cohort, use transfer learning to train the LDCT data cohort for lung cancer screening, and fine-tune the model to optimize model performance by combining numerical evaluation and human subjective evaluation indicators.

[0060] Specifically, in the pre-training process, a data queue including the subject’s first CT imaging data, CT imaging data after D days, two time intervals D days, diagnosis reports, epidemiological and clinical information data is used as a data queue as the pre-training data of the CSTA-GAN network model, and the parameters of each structural layer of the model are randomly initialized to construct the pre-training model.

[0061] The knowledge gained from training the model in a large-scale CT data group is transferred to the LDCT data group with less data volume for training using transfer learning. The model hyperparameters are fine-tuned. After each training, the generated LDCT data is evaluated using mean square error loss and physician subjective scores until the model converges.

[0062] Among them, in the pre-training and transfer learning training of the CSTA-GAN network model in this embodiment, the target loss function is composed of three loss functions, namely, the discriminator loss function of the generated lung image and the real image

[0063] L Discriminator , the perceptual loss function L of the feature map Perceptual and the radiomics mean absolute error loss function L Radiomics , the objective loss function expression of the CSTA-GAN network model is:

[0064] L CSTA_GAN =L Discriminator +L Perceptual +L Radiomics

[0065] Among them, the input of this embodiment is different from that of the traditional generative adversarial network (GAN). The input in the traditional GAN ​​network is random noise, which has a certain randomness for the generator, while the input of the CSTA-GAN network in this embodiment is the initial lung image data of the subject, and a randomly generated feature map is formed by a convolutional neural network, a spatial attention mechanism, a temporal attention mechanism and temporal features.

[0066] Step S5, model result display, compare the LDCT image of the subject at screening with the LDCT image generated D days in the future, mark the suspected nodule area, and assist the physician in diagnosing the future lung changes of the subject.

[0067] Specifically, the CSTA-GAN network model generates LDCT images D days in the future, uses the DL segmentation model to mark the boundaries of the suspected nodule area (if a nodule exists, the nodule area is marked, if not, the marked area is not displayed), and gives the marked image to the diagnostic physician, who diagnoses the LDCT image D days later and gives the patient diagnostic advice.

[0068] Figure 2 The diagram is a conceptual information extraction-processing-coding diagram of a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention.

[0069] In the process of encoding the grandmother's imaging genomics, diagnostic report information, epidemiological information and basic clinical information: first, the corresponding conceptual information of different types of concept numbers is extracted, and then the extracted conceptual information is conceptually integrated, and all the conceptual information is integrated into four types: numerical type, category type, degree type and disease type. Data preprocessing is performed according to the type of its conceptual information, and finally it is integrated into a complete conceptual feature vector.

[0070] Furthermore, for the conceptual information of radiomics features, 19 first-order statistical features, 14 shape features, 75 texture features, and 8 feature maps after wavelet transformation of high-order features were extracted. The mean, standard deviation, sample size, and degree of freedom between the features were calculated using the independent sample T test. The P value between the features was calculated using the T distribution. If the P value between the features was less than or equal to 0.05, it was considered that there was a significant difference in the radiomics features between the features and they were retained; otherwise, it was considered that there was no significant difference between the omics features and such features were removed. LASSO regression was performed on the significant features retained by the independent sample T test. First, the feature values ​​were normalized to ensure that the features maintained the same dimension. The LAASO regression algorithm model was established. The appropriate regularization parameter lambda was selected by the cross-validation method to determine the optimal lambda value. Finally, the coefficient of the radiomics feature was calculated. When the feature coefficient was 0 or close to 0, the feature was a non-important feature for the target task and was screened out.

[0071] Furthermore, for the conceptual information of the diagnosis report, OCR technology is used to perform image preprocessing, segmentation and sub-segmentation on whether there are nodules, nodule location description, nodule morphology description and other lung abnormality characters in the diagnosis report, and then a mature OCR model is used to extract the shape, structure and texture features of the characters, and a mature OCR model is used to identify and post-process the features. Finally, the identified information is structured using pre-established clinical prior knowledge rules, such as patient: Zhang San; reason for examination; coughing for several months; image description: 1 ground-glass nodule, located in the lower lobe of the left lung, 7mmX8mm in size, with pulmonary fibrosis; regular follow-up is recommended.

[0072] Furthermore, the epidemiological information and basic clinical information of the subjects are collected by the physician through questionnaires, medical records or testing systems, and are sorted by the physician according to different categories of information, and finally integrated into structured text information and stored in an Excel spreadsheet.

[0073] Furthermore, after the conceptual information is integrated, different types of conceptual information are encoded.

[0074] Furthermore, in order to eliminate the impact of different types of numerical data dimensions and improve the convergence speed of the model, the numerical concept information such as energy, entropy, gray value, weight, and age is standardized. The standardized expression is as follows:

[0075]

[0076] In the expression X number～0-1 is the standardized numerical concept feature value, X number is the original numerical concept feature value, μ is the mean value of the data feature of the concept, and σ is the standard deviation of the data feature of the concept.

[0077] Furthermore, in order to prevent the model from mistakenly understanding the relationship between categorical concept data as a certain numerical relationship, the categorical concepts such as asbestos, radon gas, smoking, and drinking are processed with One-Hot encoding, and the expression is as follows:

[0078]

[0079] In the expression, assuming that a certain classification concept has n different categories of concept features, a feature belonging to the i-th category is encoded to obtain a vector of length n, in which the i-th position is 1 and the rest of the n vectors are 0. is the encoded concept vector.

[0080] Furthermore, since computers are better at processing numerical data than algorithm analysts, numerical encoding is performed on the degree of burrs on the edge of nodules, so that non-numerical data can be used more effectively when building models. Linear formulas are used to map conceptual data of different degrees to the degree between 0 and 1 to represent the concept. The expression is as follows:

[0081]

[0082] In the expression, X degree-numer is a coded value, ranging from 0 to 1. degree It is a conceptual level, such as nodule with clear edge-level 1, smooth-level 2, lobed-level 3, burred-level 4, and blurred-level 5.

[0083] Furthermore, in order to ensure that all diseases have uniform standards for diagnosis and classification, and to facilitate the unification of disease naming between different regions and medical institutions, for example, pulmonary fibrosis, lung inflammation, and chronic obstructive pulmonary disease are coded in ICD-10, pulmonary fibrosis-J84.1, pneumonia-J18.91, and chronic obstructive pulmonary disease-J44.9. The Word2Ver method is then used to encode the disease types coded in ICD-10 to obtain the encoded vectors.

[0084] Furthermore, the encoded numerical, categorical, degree and disease-type concept information vectors are finally concatenated to obtain a concept feature vector.

[0085] Figure 3 This is a schematic diagram of the spatial attention mechanism structure of a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention.

[0086] During model training, the input data is an image data of the subject. The preliminary feature map of the image data is obtained through three CNN structures, replacing the random noise features generated and used in the traditional GAN ​​network. The feature map enters the spatial attention branch and is mapped to another latent space. A weight mask is generated for the pixels at each position in the image. The generated weighted mask feature is Hadamard-multiplied with the original input feature map to obtain a mask feature with weighted features, which is then superimposed with the original input feature. The global maximum pooling, global average pooling and convolution operations are used to further enhance the spatial features, so that the focused area is enhanced. Finally, the Sigmoid function is used to generate weights and superimposed with the branched features to obtain the final image spatial feature map.

[0087] Furthermore, in the spatial attention structure, a nonlinear pattern is introduced using convolutional layers and rectified linear unit (ReLU) activation functions to enhance the feature significance of abnormal lung areas. Global maximum pooling and global average pooling are used to reduce the spatial resolution of the feature map while improving the receptive field of the feature map. The MLP layer maps the image pixel values ​​into semantic and abstract feature vectors. Finally, a weight map is generated through a sigmoid logic activation function and superimposed with the original input features to obtain the spatial feature map of the lung LDCT image.

[0088] Figure 4 This is a schematic diagram of the structure of the temporal attention mechanism of a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention.

[0089] The spatial feature map after the spatial attention mechanism enters the temporal attention mechanism structure, allowing the model to focus on image faults where abnormalities may occur, which to a certain extent makes up for the fact that spatial attention cannot focus on faults that require important attention in the timeline.

[0090] Furthermore, the spatial feature map is downsampled through three layers of CNN, and then the flatten operation is used to convert the feature map into a one-dimensional tensor. After passing through two layers of neural network layers, it enters the temporal attention mechanism branch composed of a long short-term memory network (LSTM) structure. The features after the temporal attention mechanism branch are Hadamard-producted with the input, and then superimposed with the input features again to complete the screening of the fault time features by the temporal attention mechanism branch, and then the temporal feature map is obtained by passing through the LSTM partial output again.

[0091] Furthermore, the temporal attention branch is mainly composed of three parts: the forget gate, the input gate, and the output gate. The forget gate is based on the input h of the previous node. t-1 and the input x of this stage t Selectively forget, forget the non-critical faults, and remember the critical faults; the input gate is based on the input x of this stage t The current turntable is updated through the Sigmoid and Tanh activation functions, followed by the Hadamard product and the h of the previous stage. t-1 The states are superimposed to finally obtain the latest focus fault at this time node; the final output gate is the Hadamard product of the current state and the previous time node to determine the input feature information at the next time point.

[0092] Figure 5A schematic diagram of a CSTA-GAN network structure of a method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention.

[0093] The complete CSTA-GAN network consists of two major parts: the generation network and the discriminant network. The generation network includes spatial attention mechanism, temporal attention mechanism, inspection time interval encoding and transposed convolution generation structure. The discriminant network uses the traditional CNN network to distinguish between real and generated lung LDCT images.

[0094] Furthermore, the encoding of the time interval between two examinations is achieved by performing sine and cosine transformations on the number of days of the interval D. The transformed time vector is connected with the time feature vector obtained by the temporal attention mechanism, and finally upsampled to an image size consistent with the input size through a three-layer transposed convolution. The ReLU activation function is used between layers to nonlinearize the feature map, and the original image is gradually upsampled to restore the image details, and finally the generated LDCT image of the next D days is obtained.

[0095] Figure C is an example diagram of the generation effect of a method for generating future lung images using a CSTA-GAN network model that introduces concept coding in an embodiment of the present invention.

[0096] The basic information of the patients using the test data is as follows: male, C3 years old, no family history of lung cancer, a history of smoking for more than 40 years (1 pack / day), no history of drinking, no parasitic infection, no exposure to industrial toxins, no exposure to dust, no history of contact with radioactive substances, no history of traveling, and had been infected with COVID-19.

[0097] Furthermore, the diagnosis result of the lung CT image tested by the model after the first shot was: a solid mass, visible burrs, pleural traction, and unclear edges were visible in the right lung; using the patient's lung CT image, the CT image generated 65 days later by CSTA-GAN was described by the radiologist as: a lobed mass shadow was seen in the right upper lobe, with unclear edges, visible burrs around it, and adjacent pleural traction; the diagnosis result was: a mass in the right upper lobe, which was new compared to the previous one, considered to be a malignant tumor, and emphysema in both lungs.

[0098] Further, through Figure 6In the method observation of the lesion area, the lung CT images generated by the CSTA-GAN network were compared with the real CT images of the patients. The model showed a high degree of consistency with the real images in the generation task. At the same time, the radiologist's diagnosis was consistent with the real CT image results. However, there are still slight differences in the details between the generated lesion area boundary and the real lesion. This may be because the model may experience performance fluctuations when processing extreme data in the data of different patients. However, this situation will only occur when facing edge cases with very small data volumes and highly abnormal features. We are also clearly aware that there are still some areas for improvement in the model. In the future, we will continue to optimize the CSTA-GAN network model in multi-center and multi-instrument data queues to prevent the above situation from happening.

[0099] For the specific calculation of the method for generating future lung LDCT images based on a CSTA-GAN network model that introduces concept coding, please refer to the above description of data collection, concept information preprocessing, concept information coding, CSTA-GAN network model establishment, loss function design, pre-training, retraining and true and false image identification in the invention method, which will not be repeated here. All or part of the above method for generating future lung LDCT images based on a CSTA-GAN network model that introduces concept coding can be implemented by software to facilitate the calculation of the corresponding operations in each step.

[0100] The process embodiment described above is only illustrative, wherein the CSTA-GAN network model that introduces concept coding may also be of other different structures, and the spatial and temporal attention mechanisms, target loss functions, and discrimination methods in the training process may or may not be the structures and methods proposed in this article. Some of them can be selected according to actual needs to achieve the purpose of the embodiment scheme, and at the same time, ordinary technicians in this field can understand and implement it without paying creative labor.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand the various embodiments, and can implement the proposed methods with the help of different languages ​​and deep learning frameworks. Based on this understanding, the essence of the above technical solution or the contribution to the shared technology can be presented in the form of software or a full-stack Web model. The model can be embedded in the software or deployed in a Web network database, such as MySQL, SQL Server, Oracle, DB2, etc., and contains several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0102] The above embodiments are intended to further illustrate some preferred embodiments of the present invention, but not all embodiments. Other embodiments based on the present invention made by professionals in this field without creative work are all within the scope of protection of the present invention.

Claims

1. A method for generating future lung LDCT images using a CSTA-GAN network model that introduces concept coding, characterized in that: The steps of the method are as follows: S1. Data collection: collection and data cleaning of LDCT images of patients undergoing lung cancer screening, clinical information of patients, epidemiological information and screening report data cohorts; S2. Data feature engineering, including ROI area delineation, radiomics feature extraction and screening, diagnosis report information extraction, and conceptual information coding of patient clinical information, epidemiology, and diagnosis results; S3, model construction, design a generative adversarial network embedded with a spatiotemporal attention mechanism, introduce the concept information of clinical information, epidemiology and diagnostic reports, design a loss function suitable for introducing a concept encoding generation model, and generate a CSTA-GAN network model; S4. Model training: Use conventional CT cohorts to pre-train the model, use transfer learning to train the LDCT data cohort for lung cancer screening, and fine-tune the model to optimize model performance by combining numerical evaluation and human subjective evaluation indicators; S5. Model result display: compare the LDCT image of the subject at screening with the LDCT image generated D days in the future, and mark the suspected nodule area.

2. The method for generating future lung LDCT images using the CSTA-GAN network model with concept coding introduced in claim 1, characterized in that: In the method, the LDCT imaging features include morphological features, texture, first-order statistics, and high-order features. The extracted LDCT imaging features are subjected to feature dimension reduction, noise removal, and redundant features through LASSO regression and statistical methods to obtain imaging features highly correlated with lung cancer; text content is extracted from the patient's lung cancer screening diagnosis report, and the imaging manifestations and diagnostic opinions are combined with a neural network method to classify the concepts of nodule lesions; the patient's imaging information, clinical information, epidemiological data information, and diagnostic report information are integrated, and divided into discrete, continuous, and disease types according to the data value type; conceptual information is encoded through One-Hot encoding, standardization, ICD-10 encoding, and Word2Vec to encode discrete, continuous, and disease-type data, and conceptual information of different data types is mapped to the vector space, which is used as a control generation network to adjust the changes in the nodule area in the lung LDCT image; The design introduces a spatiotemporal attention mechanism to guide the network model to focus on the nodule area in LDCT and highlight the changing characteristics of the lung nodule area at different levels.

3. The method for generating future lung LDCT images using the CSTA-GAN network model with concept coding introduced in claim 2 is characterized in that: In the step S1, the LDCT image data queue for lung cancer screening of patients includes CT images taken at two different times of the screened person, and the data cleaning includes: evaluating the collected lung cancer screening LDCT image queue, removing LDCT image data with poor image quality, missing or incomplete key chest areas in the image, non-compliance with the standard acquisition protocol, image artifacts caused by patient movement, repeated or redundant data, and damaged or unreadable image data; for the collected patient clinical information and epidemiological information, removing missing values, repeated values, and abnormal values ​​in the data of the same patient, checking the consistency of data between the same fields, and ensuring the logical correctness of the information; integrating the above-processed data, and removing the data of incomplete patient information.

4. The method for generating future lung LDCT images using the CSTA-GAN network model with concept coding introduced in claim 3 is characterized in that: In step S2, Python software is used to convert the lung LCTD in Dicom format into nii.gz, and ITK-SNAP medical image processing software is used to outline each layer of the lung image in LDCT. The outline method is to outline the lung boundary area, and the final outlined label is saved in nii format.

5. The method for generating future lung LDCT images using the CSTA-GAN network model with concept coding introduced in claim 4 is characterized in that: The diagnostic report information extraction in step S2 can directly read the electronic text information or use OCR technology to extract information from the patient's diagnostic report, including: Imaging findings and diagnostic opinions; After the diagnostic report information is extracted, the text is vectorized using Word2Vec technology, and a convolutional neural network is built to classify the text. According to the category of information, such as location, morphology, edge, and other abnormalities, the information is written into the corresponding Excel header.

6. The method for generating future lung LDCT images using the CSTA-GAN network model with concept coding introduced in claim 5, characterized in that: In step S2, different conceptual information encodings are performed according to the types of data in the extracted diagnostic report information, patient clinical information and epidemiological information: One-Hot encoding was performed on discrete variables such as gender, occupational exposure, and family cancer history; The continuous variables of age, weight, biochemical indexes, and blood routine were standardized and dimensionless. The concept type information of pulmonary fibrosis, pulmonary inflammation, chronic obstructive pulmonary disease, and family history of cancer was coded by ICD-10, and then Word2Vec text vectorization was performed on it after coding; According to the lung cancer guidelines, the concepts of lung nodule location, morphology, and margins are numerically coded, and the numerical range is defined as 0-1, where 0 represents completely smooth, 1 represents obvious burrs, and intermediate values ​​represent different degrees of burr manifestations.

7. The method for generating future lung LDCT images using the CSTA-GAN network model with concept coding as described in claim 6 is characterized in that: The CSTA-GAN network model in step S3 is based on the generative adversarial network model structure, and adds the temporal and spatial attention mechanism structures as the model framework for future lung LDCT image generation. The spatial attention mechanism structure enables the model to focus on the distribution of spatial information importance of the nodule area in the lung CT, and guides the model to focus on the nodule area in the image; the temporal attention mechanism is used to dynamically assign weights to a series of continuous tomographic images in the LDCT image, so that the model pays more attention to the tomographic images of the lungs where nodules appear and where nodules may appear.

8. The method for generating future lung LDCT images using the CSTA-GAN network model with concept coding introduced in claim 7, characterized in that: The CSTA-GAN network contains three loss functions, namely, the discriminator loss function L between the generated lung LDCT image and the real image Discriminator , the perceptual loss function L of the feature map perceptual and the radiomics mean absolute error loss function L Radiomics , the expressions of the three loss functions are as follows: Among them, G(x predict_LDCT ) is the future lung LDCT image generated by the CSTA-GAN network, D(x tre_LDCT ) is the result of the discriminator’s judgment on the real CT image, D(G(x predict_LDCT )) is the result of the discriminator’s judgment on the future lung LDCT images generated by the network, and the perceptual loss function L of the feature map perceptual The expression is: in, is the characteristic map of the lth layer of the Tth layer of the patient's real future LDCT image, The l-th layer feature map of the T-th layer of the CT image of the patient at the future time generated by the CSTA-GAN network, and They are the number of channels, height and width of the feature map, and the mean absolute error loss function L of radiomics Radiomics The expression is: Among them, n is the number of radiomics features after feature screening, is the i-th radiomics feature value in the patient’s real image, is the i-th radiomics feature value in the CT image of the patient at the future time generated by the CSTA-GAN network.

9. The method for generating future lung LDCT images using the CSTA-GAN network model with concept coding as described in claim 8, characterized in that: In step S4, the model initialized by the conventional lung CT data team is trained on the low-dose LDCT data for CSTA-GAN using transfer learning, and the generated future lung LDCT images are comprehensively, accurately and quality evaluated using numerical evaluation and human subjective evaluation indicators. The model hyperparameters are adjusted according to the evaluation results, and the training is repeated to obtain the CSTA-GAN generative model with the best generalization performance; The numerical evaluation uses the mean squared error (MSE) to evaluate the average value of the square of the pixel value difference between the real LDCT and the generated LDCT image, and its expression is as follows: p ij is the pixel value at position (i, j) in the real lung LDCT image, The pixel value at (i, j) in the lung LDCT image generated by the CSTA-GAN network is finally summed up by the square of its difference and divided by the total number of pixels in the LDCT to obtain the mean square error. Among them, the human subjective evaluation index uses the preference option method, and the radiologist distinguishes the real and generated LDCT images. If the diagnosis is correct, the score is 1, and if the judgment is wrong, the score is 0.

10. The method for generating future lung LDCT images using the CSTA-GAN network model with concept coding introduced according to claim 9, characterized in that: In step S5, the lung nodule area in the future LDCT is segmented and generated using the deep learning model. If a lung nodule exists, a nodule edge mask area is given.

Images