Prediction method and device based on lung image processing, equipment and medium

Abstract

The invention provides a prediction method and device based on lung image processing, equipment and a medium. The method comprises the steps that a to-be-processed lung image and breathing parameters of a target user are acquired; performing spatial registration to obtain spatial registration result data; inputting the spatial registration result data into a graph neural network, constructing a functional connection relationship between lung segments, and obtaining a functional contribution weight of each lung segment; performing prediction by adopting a time sequence convolutional network; and inputting the time sequence registration result data and the biomarker data into a cross-modal prediction model to obtain tidal volume change data and hypoxemia risk data of the target user. Compared with an existing technical scheme based on static data, dynamic analysis of the lung state of the patient is achieved, and analysis hysteresis is reduced; in addition, dynamic prediction and analysis are achieved based on biomarkers, lung images and breathing parameters, and the defect that the prior art depends on single index prediction precision can be overcome through multi-modal dynamic modeling.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a prediction method, device, equipment and medium based on lung image processing. Background Art

[0002] Thoracic surgery is an important treatment for a variety of chest diseases, but intraoperative monitoring of vital signs and postoperative lung function recovery directly impact patient prognosis and quality of life. Currently, existing technologies for analyzing a patient's lung status focus on static data, such as preoperative CT scans or postoperative monitoring. These static data-based analysis solutions can only capture a patient's vital signs at a specific moment, exhibit a certain lag, and fail to reflect the changing trends of these vital signs, making them difficult to meet clinical decision-making needs. Summary of the Invention

[0003] The present invention application provides a prediction method, apparatus, device and medium based on lung image processing to solve the technical problem of how to reduce the lag in analyzing the patient's lung status.

[0004] In order to solve the above technical problems, the present invention provides a prediction method based on lung image processing, comprising:

[0005] Acquire a lung image to be processed and respiratory parameters of a target user; the lung image to be processed includes multiple lung segments;

[0006] Performing spatial registration on the lung image to be processed and the respiratory parameters to obtain spatial registration result data;

[0007] Inputting the spatial registration result data into a preset graph neural network to construct a functional connectivity relationship between the multiple lung segments, and then obtaining a functional contribution weight of each lung segment based on the functional connectivity relationship;

[0008] Based on the spatial registration result data and the functional contribution weights of each lung segment, a preset temporal convolutional network is used for prediction to obtain temporal registration result data;

[0009] Obtain biomarker data of the target user; input the temporal registration result data and the biomarker data into a preset cross-modal prediction model to obtain tidal volume change data and hypoxemia risk data of the target user.

[0010] As a preferred solution, the preset cross-modal prediction model includes a Transformer model, and the inputting of the temporal registration result data and the biomarker data into the preset cross-modal prediction model to obtain the tidal volume change data and hypoxemia risk data of the target user includes:

[0011] Input the time series registration result data and the biomarker data into the Transformer model, perform embedding representation according to a preset fusion formula to obtain fusion features, and then obtain the tidal volume change data and hypoxemia risk data of the target user based on the fusion features;

[0012] The preset fusion formula is expressed as:

[0013] W fusion = α·X img + β·X bio ;

[0014] where, W fusion represents the fusion feature, X img is the time series registration result data, and X bio is the biomarker.

[0015] As a preferred solution, the step of inputting the spatial registration result data into a preset graph neural network to construct the functional connection relationship between multiple lung segments, and then obtaining the functional contribution weights of each lung segment based on the functional connection relationship includes:

[0016] According to the spatial registration result data, analyze the ventilation distribution of each lung segment using a density histogram;

[0017] Input the spatial registration result data and the ventilation distribution into the preset graph neural network, so that the preset graph neural network takes the lung segments as nodes and the blood flow data corresponding to the lung segments as edge weights to predict the ventilation weights of the regions corresponding to each lung segment; and obtain the functional contribution weights of each lung segment based on the ventilation weights of the regions corresponding to each lung segment.

[0018] As a preferred solution, the step of predicting the time series registration result data by using a preset time series convolutional network based on the spatial registration result data and the functional contribution weights of each lung segment includes:

[0019] Construct the spatial registration result data and the functional contribution weights of each lung segment into a data set;

[0020] Use XGBoost to extract time series features from the data set;

[0021] Input the time series features into the preset time series convolutional network for prediction to obtain the time series registration result data.

[0022] As a preferred solution, after obtaining the to-be-processed lung image and respiratory parameters of the target user, it further includes:

[0023] Perform finite element meshing on the to-be-processed lung image to obtain a number of lung tissue meshes;

[0024] Load mechanical parameters on the lung tissue grid, simulate the stress distribution of the lung tissue grid, and obtain simulation results;

[0025] Obtain the user's historical data and virtual resection plan of the target user, and input the simulation results, user's historical data and virtual resection plan into a preset generative adversarial network to generate a postoperative compensation simulation diagram.

[0026] As a preferred solution, the preset generative adversarial network includes a generator and a discriminator;

[0027] The generator is used to obtain a generated resection image based on the lung tissue grid, the virtual resection plan and the user's historical data of the target user;

[0028] The discriminator is used to make a discrimination based on the generated resection image and the real resection image to obtain a discrimination result, and then use the discrimination result to generate the postoperative compensation simulation diagram.

[0029] As a preferred solution, the prediction method further includes:

[0030] Monitor the tidal volume change data and hypoxemia risk data of the target user, and dynamically update the functional contribution weights of each lung segment according to the monitoring results.

[0031] Correspondingly, the present invention application also provides a prediction device based on lung image processing, including an acquisition module, a spatial registration module, a weight allocation module, a temporal registration module and a prediction module; wherein,

[0032] The acquisition module is used to obtain the lung image to be processed and respiratory parameters of the target user; the lung image to be processed includes multiple lung segments;

[0033] The spatial registration module is used to perform spatial registration on the lung image to be processed and the respiratory parameters to obtain spatial registration result data;

[0034] The weight allocation module is used to input the spatial registration result data into a preset graph neural network to construct a functional connection relationship between the multiple lung segments, and then obtain the functional contribution weights of each lung segment based on the functional connection relationship;

[0035] The temporal registration module is used to perform prediction using a preset temporal convolutional network based on the spatial registration result data and the functional contribution weights of each lung segment to obtain temporal registration result data;

[0036] The prediction module is used to obtain biomarker data of a target user; input the time series registration result data and the biomarker data into a preset cross-modal prediction model to obtain tidal volume change data and hypoxemia risk data of the target user.

[0037] As a preferred solution, the preset cross-modal prediction model includes a Transformer model. The prediction module inputs the time series registration result data and the biomarker data into the preset cross-modal prediction model to obtain tidal volume change data and hypoxemia risk data of the target user, including:

[0038] The prediction module inputs the time series registration result data and the biomarker data into the Transformer model, performs embedding representation according to a preset fusion formula to obtain a fusion feature, and then obtains tidal volume change data and hypoxemia risk data of the target user based on the fusion feature;

[0039] The preset fusion formula is expressed as:

[0040] W fusion =α·X img +β·X bio ;

[0041] where, W fusion represents the fusion feature, X img is the time series registration result data, and X bio is the biomarker.

[0042] As a preferred solution, the weight assignment module inputs the spatial registration result data into a preset graph neural network to construct a functional connection relationship between the multiple lung segments, and then obtains the functional contribution weights of each lung segment based on the functional connection relationship, including:

[0043] The weight assignment module analyzes the ventilation distribution of each lung segment using a density histogram according to the spatial registration result data;

[0044] Inputs the spatial registration result data and the ventilation distribution into the preset graph neural network, so that the preset graph neural network takes the lung segments as nodes and the blood flow data corresponding to the lung segments as edge weights to predict the ventilation weights of the corresponding regions of each lung segment; and obtains the functional contribution weights of each lung segment based on the ventilation weights of the corresponding regions of each lung segment.

[0045] As a preferred solution, the time series registration module performs prediction using a preset time series convolutional network based on the spatial registration result data and the functional contribution weights of each lung segment to obtain time series registration result data, including:

[0046] The timing registration module constructs the spatial registration result data and the functional contribution weights of each lung segment into a data set;

[0047] XGBoost is used to extract timing features from the data set;

[0048] The timing features are input into the preset timing convolutional network for prediction to obtain timing registration result data.

[0049] As a preferred solution, the prediction device further includes a simulation module, and the simulation module is used after the acquisition module acquires the to-be-processed lung image and respiratory parameters of the target user:

[0050] Perform finite element division on the to-be-processed lung image to obtain a number of lung tissue meshes;

[0051] Load mechanical parameters on the lung tissue meshes, simulate the stress distribution of the lung tissue meshes, and obtain a simulation result;

[0052] Obtain the user historical data and virtual resection plan of the target user, and input the simulation result, user historical data and virtual resection plan into a preset generative adversarial network to generate a postoperative compensation simulation diagram.

[0053] As a preferred solution, the preset generative adversarial network includes a generator and a discriminator;

[0054] The generator is used to obtain a generated resection image based on the lung tissue meshes, the virtual resection plan and the user historical data of the target user;

[0055] The discriminator is used to make a discrimination based on the generated resection image and the real resection image to obtain a discrimination result, and then use the discrimination result to generate the postoperative compensation simulation diagram.

[0056] As a preferred solution, the prediction device further includes a dynamic update module, and the dynamic update module is used to:

[0057] Monitor the tidal volume change data and hypoxemia risk data of the target user, and dynamically update the functional contribution weights of each lung segment according to the monitoring results.

[0058] Correspondingly, the present invention application also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the prediction method based on lung image processing described above.

[0059] Correspondingly, the present invention application also provides a computer-readable storage medium, which includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the above-mentioned prediction method based on lung image processing.

[0060] Compared with the prior art, the present invention application has the following beneficial effects:

[0061] The present invention application provides a prediction method, device, equipment and medium based on lung image processing. The prediction method includes: obtaining a to-be-processed lung image and respiratory parameters of a target user; the to-be-processed lung image includes multiple lung segments; performing spatial registration on the to-be-processed lung image and the respiratory parameters to obtain spatial registration result data; inputting the spatial registration result data into a preset graph neural network to construct a functional connection relationship between the multiple lung segments, and further obtaining the functional contribution weights of each lung segment based on the functional connection relationship; based on the registration result data and the functional contribution weights of each lung segment, using a preset temporal convolutional network for prediction to obtain temporal registration result data; obtaining biomarker data of the target user; inputting the temporal registration result data and the biomarker data into a preset cross-modal prediction model to obtain the tidal volume change data and hypoxemia risk data of the target user. The present invention application performs spatial registration on the to-be-processed lung image and the respiratory parameters to ensure that the CT features and respiratory features of the target user's lungs can be accurately and simultaneously reflected during prediction; inputting the registration result data into a preset graph neural network to construct a functional connection relationship between the multiple lung segments and obtain the functional contribution weights of each lung segment, so that the dynamic respiratory parameters can be mapped to each lung segment partition, improving the accuracy of subsequent prediction; in addition, based on the registration result data and the functional contribution weights of each lung segment, using a preset temporal convolutional network for prediction to obtain temporal registration result data can reduce the temporal deviation between the two. Compared with the existing technical solutions for obtaining static data, it realizes the dynamic analysis of the patient's lung state, reduces the lag of analysis, and can reflect the tidal volume change trend and hypoxemia risk probability of the user; in addition, the present application realizes the dynamic prediction and analysis of the target user's lung state based on biomarkers, lung images and respiratory parameters, and can make up for the deficiency of the prediction accuracy of the prior art relying on a single index (such as FEV1, DLCO) through multi-modal dynamic modeling, reflect the regional lung function distribution and thus accurately reflect the surgical risk. Description of the Drawings

[0062] Figure 1 : It is a schematic flowchart of an embodiment of the prediction method based on lung image processing provided by the present invention application.

[0063] Figure 2: Schematic diagram of the architecture of an embodiment of the medical system provided by this invention application.

[0064] Figure 3 : Schematic diagram of the structure of an embodiment of the prediction device based on lung image processing provided by this invention application. Detailed implementation manners

[0065] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0066] Embodiment 1

[0067] Please refer to Figure 1 , Figure 1 : A prediction method based on lung image processing provided by this invention application, including steps S101 to S105; wherein,

[0068] Step S101, obtaining the lung image to be processed and respiratory parameters of the target user; the lung image to be processed includes multiple lung segments.

[0069] In this step, the respiratory parameters may include the impedance change rate during preoperative deep breathing and the oxygenation index decay curve during intraoperative one-lung ventilation.

[0070] Specifically, the impedance change rate during preoperative deep breathing can be expressed as the respiratory oscillation signal of the dynamic lung function waveform - low frequency (for example, 5 - 15 Hz). For the impedance change rate during preoperative deep breathing, the FFT spectrum feature extraction method can be used to obtain it.

[0071] The oxygenation index decay curve during intraoperative one-lung ventilation can be expressed as the dynamic waveform - SpO2 trend (SpO2 represents blood oxygen saturation). For this oxygenation index decay curve during intraoperative one-lung ventilation, the exponential decay model fitting can be used to obtain it.

[0072] The above-mentioned lung image to be processed can be a Computed Tomography (CT) image, and this lung image to be processed is obtained by processing the original image such as lung segmentation and lung segment segmentation.

[0073] Step S102, performing spatial registration on the lung image to be processed and the respiratory parameters to obtain spatial registration result data.

[0074] In this embodiment, based on the respiratory parameters and the lung image to be processed, the lung parenchyma density distribution characteristics can be extracted, and then the regional ventilation / perfusion weight can be calculated.

[0075] Furthermore, the calculated regional ventilation / perfusion weight and the lung image to be processed are spatially registered so that their three-dimensional coordinates are aligned.

[0076] Exemplarily, in some optimized embodiments of spatial registration, specifically, the lung morphology at each phase (0%-100%) of the respiratory cycle can be obtained through CT four-dimensional reconstruction and aligned with the time stamps of the impedance data collected by FOT (forced oscillation technique); and non-rigid registration, such as using the Demons algorithm to calculate the deformation field from dynamic parameters to the CT space. Then, in step S103, a functional weight is assigned to each lung voxel.

[0077] Step S103: Input the spatially registered result data into a preset graph neural network to construct the functional connection relationships between the multiple lung segments, and then obtain the functional contribution weights of each lung segment based on the functional connection relationships.

[0078] Exemplarily, according to the spatially registered result data, the ventilation distribution of each lung segment is analyzed using a density histogram; the spatially registered result data and the ventilation distribution are input into the preset graph neural network, so that the preset graph neural network takes the lung segments as nodes and the blood flow data corresponding to the lung segments as edge weights to predict the ventilation weights of the regions corresponding to each lung segment; and based on the ventilation weights of the regions corresponding to each lung segment, the functional contribution weights of each lung segment are obtained.

[0079] In this embodiment, different lung segments in the CT image can be distinguished by different colors during the annotation process. The ventilation area and the fibrotic area can be obtained through density histogram analysis, and the high-ventilation area and the fibrotic area among them are annotated. The obtained functional contribution weights of each lung segment can be superimposed and represented in the form of a numerical table, a three-dimensional heat map, etc. In addition, exemplarily, key anatomical landmarks can also be annotated as needed (the interlobar fissure localization error is less than 1.2 mm, and the bronchial grading is B1-B10). It can be understood that the annotation of the lung segment function weights can independently output the results for surgeons' reference.

[0080] The above-mentioned regional ventilation can be expressed by the following formula:

[0081]

[0082] where W i represents the ventilation weight of the i-th lung segment region, V i is the volume of the i-th lung segment, ρ i is the average density of the i-th lung segment, V jrepresents the volume of the j-th lung segment, ρ j represents the average density of the j-th lung segment.

[0083] Step S104: Based on the spatial registration result data and the functional contribution weights of each lung segment, use a preset temporal convolutional network for prediction to obtain temporal registration result data.

[0084] In a preferred embodiment, the spatial registration result data and the functional contribution weights of each lung segment are constructed into a data set; use the XGBoost tool to extract temporal features from the data set; input the temporal features into the preset temporal convolutional network for prediction to obtain temporal registration result data.

[0085] Further, the using the XGBoost tool to extract temporal features from the data set includes:

[0086] Use the XGBoost tool to extract multiple initial features from the data set, sort the initial features according to importance, and screen the temporal features from the sorting results.

[0087] Input the temporal features into the preset temporal convolutional network such as the TCN model for prediction to obtain temporal registration result data. The TCN model can adopt a joint loss function such as the MAE loss function + postoperative complication classification cross-entropy, and can be exemplarily expressed as:

[0088] L = 0.7MAE + 0.3CE;

[0089] where L is the joint loss function and CE is the + postoperative complication classification cross-entropy.

[0090] In addition, the dilation coefficient d of the convolution kernel of the TCN model is 1, 2, 4, and the backpropagation algorithm can be used for optimization. During the training and / or application process of the model, visual output can be performed through the SHAP value interpretation module, such as displaying a bar chart of feature contribution degrees, etc.

[0091] Step S105: Obtain biomarker data of the target user; input the temporal registration result data and the biomarker data into a preset cross-modal prediction model to obtain tidal volume change data and hypoxemia risk data of the target user.

[0092] In this embodiment, the biomarker data is specifically serum data, such as IL-6, CRP, KL-6, and CC-16, etc.

[0093] The preset cross-modal prediction model includes a Transformer model, and the preset cross-modal prediction model can adopt cross-modal attention and gating mechanisms; the step of inputting the time series registration result data and the biomarker data into the preset cross-modal prediction model to obtain the tidal volume change data and hypoxemia risk data of the target user includes:

[0094] Input the time series registration result data and the biomarker data into the Transformer model, perform embedding representation according to a preset fusion formula to obtain a fusion feature, and then obtain the tidal volume change data and hypoxemia risk data of the target user based on the fusion feature;

[0095] The preset fusion formula is expressed as:

[0096] W fusion =α·X img +β·X bio ;

[0097] wherein, W fusion represents the fusion feature, X img is the time series registration result data, X bio is the biomarker, and the sum of α and β is 1.

[0098] In some examples, for example, when the clinical condition is severe emphysema (low CT signal-to-noise ratio), the value of α can be 0.2 to 0.4 at this time, and the corresponding value of β is 0.6 to 0.8; when the clinical condition is acute infection (abnormal serum indicators), the value of α can be 0.7 to 0.9 at this time, and the corresponding value of β is 0.1 to 0.3.

[0099] It is verified that when analyzing only based on images, that is, when the value of α is 1 and the value of β is 0, the predicted MAE of the parameter FEV1% in the prediction result is 8.7%; when analyzing only based on biomarkers, that is, when the value of α is 0 and the value of β is 1, the predicted MAE of the parameter FEV1% in the prediction result is 12.3%. However, in the training and application process of this application, the weights of α and β can be dynamically adjusted, so that the predicted MAE of the parameter FEV1% in the prediction result reaches 5.1%.

[0100] Furthermore, the input data of the Transformer model can also include the medical record data of the target user (as prior knowledge, which affects the prediction result through the gating mechanism) in addition to the time series registration result data and the biomarker data. For example, the medical record data can be the medication record, smoking history, and COPD grading of the target user, etc., so as to realize the fusion of multi-modal data.

[0101] Correspondingly, the output of the Transformer model can be subdivided into 3D image features of CT images (for example, a 512-dimensional vector that can be extracted by 3D ResNet-50 + spatial pyramid pooling, with an output dimension of 512×8×8×8), temporal features of dynamic breathing parameters (for example, a 256-dimensional sequence), and embeddings of serum markers and target user medical record data (a 128-dimensional vector encoded by the Transformer); fusing the above outputs, the fusion method can be expressed as:

[0102] Fused = Concat(W1·imaging emb , W2·dynamic emb , W3·bio emb )

[0103] Among them, Fused represents the fusion result, Concat represents the fusion function, imaging emb represents 3D image features or CT image features, dynamic emb represents dynamic breathing parameters, bio emb represents biomarkers (serum markers), W1, W2, and W3 respectively represent the weights of CT image features, dynamic breathing parameters, and serum markers, and emb, short for embedding, represents the embedding.

[0104] The key features of the fusion include spatial and temporal alignment (spatiotemporal consistency), that is, precise mapping of breathing parameters to CT voxels is achieved through non-rigid registration; and interpretability markers, that is, each feature unit is associated with anatomical localization (such as lung segments S1-S10) and functional parameters (such as local compliance).

[0105] In a preferred embodiment, after obtaining the to-be-processed lung image and breathing parameters of the target user in step S101, the prediction method further includes: performing finite element division on the to-be-processed lung image (using Tetrahedral elements and density adaptation) to obtain a number of lung tissue meshes; loading mechanical parameters (such as an elastic modulus of 50 kPa, a Poisson's ratio of 0.45, and preset boundary conditions) on the lung tissue meshes, simulating the stress distribution of the lung tissue meshes (nonlinear inspiration and expiration and deformation animation sequences can be used), obtaining simulation results; obtaining the user historical data and virtual resection plan of the target user, and inputting the simulation results, user historical data, and virtual resection plan into a preset generative adversarial network to generate a postoperative compensation simulation diagram (the stress distribution results of the finite element analysis can be used as the conditional input of the GAN to constrain the biomechanical rationality of the generated data).

[0106] Among them, the preset generative adversarial network (GAN, Generative Adversarial Networks) includes a generator, a CT generation network (which can adopt a U-Net architecture and residual connections), and a discriminator (which can adopt a PatchGAN architecture and a true / false probability map); the generator is used to obtain a generated resection image based on the lung tissue grid, the virtual resection plan, and the user historical data of the target user; the discriminator is used to make a discrimination based on the generated resection image and the real resection image to obtain a discrimination result, and then use the discrimination result to generate the postoperative compensation simulation map.

[0107] Exemplarily, the data applied by the preset generative adversarial network can be: 3D tensors of CT images (in DICOM format, slice thickness 1 mm, 0.625 mm resolution), stress tensors of finite element analysis, pack-years of smoking, COPD grading (grades 1-4), serum biomarkers such as (KL6, CC16), and the virtual resection method can be represented by encoding the lung segments to be resected (for example, one-hot vectors of S1-S10).

[0108] Furthermore, based on the finite element stress data, patient historical data (smoking history, COPD grading), and the virtual resection plan, the postoperative compensation potential score (ARPI index) can be predicted for the final risk assessment.

[0109] It has been verified that when disabling the data flow of finite element → GAN, the biomechanical rationality of the simulation results decreases (p < 0.01).

[0110] Furthermore, the prediction method further includes: monitoring the tidal volume change data and hypoxemia risk data of the target user, and dynamically updating the functional contribution weights of each lung segment according to the monitoring results. For example, when it is monitored during the operation that the ventilation volume of a certain lung segment decreases by 20%, the functional weight value of this area can be automatically reduced, etc. In addition, the output data of GAN can also be used as an enhanced data set for the re-training data annotation of the Transformer model.

[0111] In some embodiments, the prediction method based on lung image processing described in the present application can be applied to a medical system, such as Figure 2 shown.

[0112] The medical system includes a data acquisition layer, a feature extraction layer, a model calculation layer, and an application layer. The data set acquisition layer includes a CT scanner, a pulmonary function instrument, a serum detection device, and an intraoperative monitor. The data interfaces include a DICOM PACS interface, an IoT device API, and an HL7 protocol interface for electronic medical records. The feature extraction layer includes a lung image segmentation module (U-Net++), 3D ResNet (for CT image feature extraction), and a PACS interface (DICOM routing service), etc. The model calculation layer includes a time series data analysis module (LSTM / TCN), a GAN simulator, and a finite element simulation module. The application layer includes a warning APP, an ICU monitoring large screen, an IoT API, and a surgical navigation system (such as Augmented Reality, abbreviated as AR, enhanced reality), etc.

[0113] Correspondingly, as Figure 3 shown, the present invention application also provides a prediction device 300 based on lung image processing, including an acquisition module 301, a spatial registration module 302, a weight assignment module 303, a time series registration module 304, and a prediction module 305; wherein,

[0114] The acquisition module 301 is configured to acquire the lung image to be processed and respiratory parameters of the target user; the lung image to be processed includes multiple lung segments;

[0115] The spatial registration module 302 is configured to perform spatial registration on the lung image to be processed and the respiratory parameters to obtain spatially registered result data;

[0116] The weight assignment module 303 is configured to input the spatially registered result data into a preset graph neural network to construct a functional connection relationship between the multiple lung segments, and then obtain the functional contribution weights of each lung segment based on the functional connection relationship;

[0117] The time series registration module 304 is configured to perform prediction using a preset time series convolutional network based on the spatially registered result data and the functional contribution weights of each lung segment to obtain time series registered result data;

[0118] The prediction module 305 is configured to acquire biomarker data of the target user; input the time series registered result data and the biomarker data into a preset cross-modal prediction model to obtain the tidal volume change data and hypoxemia risk data of the target user.

[0119] As a preferred solution, the preset cross-modal prediction model includes a Transformer model. The prediction module 305 inputs the time series registered result data and the biomarker data into the preset cross-modal prediction model to obtain the tidal volume change data and hypoxemia risk data of the target user, including:

[0120] The prediction module 305 inputs the time series registration result data and the biomarker data into the Transformer model, performs embedded representation according to a preset fusion formula, obtains fusion features, and further obtains the tidal volume change data and hypoxemia risk data of the target user based on the fusion features;

[0121] The preset fusion formula is expressed as:

[0122] W fusion = α·X img + β·X bio ;

[0123] Among them, W fusion represents the fusion feature, X img is the time series registration result data, and X bio is the biomarker.

[0124] As a preferred solution, the weight allocation module 303 inputs the spatial registration result data into a preset graph neural network to construct the functional connection relationship between the multiple lung segments, and further obtains the functional contribution weights of each lung segment based on the functional connection relationship, including:

[0125] The weight allocation module 303 analyzes the ventilation distribution of each lung segment using a density histogram according to the spatial registration result data;

[0126] The spatial registration result data and the ventilation distribution are input into the preset graph neural network, so that the preset graph neural network uses the lung segments as nodes and the blood flow data corresponding to the lung segments as edge weights to predict the ventilation weights of the corresponding regions of each lung segment; and based on the ventilation weights of the corresponding regions of each lung segment, the functional contribution weights of each lung segment are obtained.

[0127] As a preferred solution, the time series registration module 304 performs prediction using a preset time series convolutional network based on the spatial registration result data and the functional contribution weights of each lung segment, and obtains time series registration result data, including:

[0128] The time series registration module 304 constructs the spatial registration result data and the functional contribution weights of each lung segment into a data set;

[0129] XGBoost is used to extract time series features from the data set;

[0130] The time series features are input into the preset time series convolutional network for prediction to obtain time series registration result data.

[0131] As a preferred solution, the prediction device 300 further includes a simulation module, which is configured to, after the acquisition module 301 acquires the to-be-processed lung image and respiratory parameters of the target user:

[0132] Perform finite element division on the to-be-processed lung image to obtain a number of lung tissue meshes;

[0133] Load mechanical parameters on the lung tissue meshes, simulate the stress distribution of the lung tissue meshes, and obtain a simulation result;

[0134] Obtain the user historical data and virtual resection plan of the target user, and input the simulation result, user historical data, and virtual resection plan into a preset generative adversarial network to generate a postoperative compensation simulation diagram.

[0135] As a preferred solution, the preset generative adversarial network includes a generator and a discriminator;

[0136] The generator is configured to obtain a generated resection image based on the lung tissue meshes, the virtual resection plan, and the user historical data of the target user;

[0137] The discriminator is configured to perform discrimination based on the generated resection image and the real resection image to obtain a discrimination result, and then generate the postoperative compensation simulation diagram by using the discrimination result.

[0138] As a preferred solution, the prediction device 300 further includes a dynamic update module, which is configured to:

[0139] Monitor the tidal volume change data and hypoxemia risk data of the target user, and dynamically update the functional contribution weights of each lung segment according to the monitoring results.

[0140] Correspondingly, the present invention application further provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the prediction method based on lung image processing as described above is implemented.

[0141] The so-called processor may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor is the control center of the terminal and connects various parts of the entire terminal through various interfaces and lines.

[0142] The memory can be used to store the computer program. The processor realizes various functions of the terminal by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the mobile phone (such as audio data, phone book, etc.). In addition, the memory may include high-speed random access memory and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, Flash Card, at least one magnetic disk storage device, flash device, or other volatile solid-state storage devices.

[0143] Correspondingly, the present invention application also provides a computer-readable storage medium. The computer-readable storage medium includes a stored computer program. Among them, when the computer program runs, it controls the device where the computer-readable storage medium is located to execute the described prediction method based on lung image processing.

[0144] Among them, if the module integrated with the prediction device / terminal based on lung image processing is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-mentioned embodiment methods of the present invention, it can also be completed by instructing relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0145] Compared with the prior art, the present invention application has the following beneficial effects:

[0146] The present invention application provides a prediction method, device, equipment and medium based on lung image processing. The prediction method includes: obtaining a lung image to be processed and respiratory parameters of a target user; the lung image to be processed includes multiple lung segments; performing spatial registration on the lung image to be processed and the respiratory parameters to obtain spatially registered result data; inputting the spatially registered result data into a preset graph neural network to construct a functional connection relationship between the multiple lung segments, and further obtaining the functional contribution weights of each lung segment based on the functional connection relationship; based on the registered result data and the functional contribution weights of each lung segment, using a preset temporal convolutional network for prediction to obtain temporally registered result data; obtaining biomarker data of the target user; inputting the temporally registered result data and the biomarker data into a preset cross-modal prediction model to obtain tidal volume change data and hypoxemia risk data of the target user. The present invention application performs spatial registration on the lung image to be processed and the respiratory parameters to ensure that the CT features and respiratory features of the target user's lungs can be accurately and simultaneously reflected during prediction; inputting the registered result data into a preset graph neural network to construct a functional connection relationship between the multiple lung segments and obtaining the functional contribution weights of each lung segment, so that the dynamic respiratory parameters can be mapped to each lung segment partition, improving the accuracy of subsequent prediction; in addition, based on the registered result data and the functional contribution weights of each lung segment, using a preset temporal convolutional network for prediction to obtain temporally registered result data can reduce the temporal deviation between the two. Compared with the existing technical solutions for obtaining static data, it realizes the dynamic analysis of the patient's lung state, reduces the lag of analysis, and can reflect the tidal volume change trend and hypoxemia risk probability of the user; in addition, the present application realizes the dynamic prediction and analysis of the target user's lung state based on biomarkers, lung images and respiratory parameters, and can make up for the deficiency of the prediction accuracy of the existing technology relying on a single index (such as FEV1, DLCO) through multi-modal dynamic modeling, reflecting the regional lung function distribution and thus accurately reflecting the surgical risk.

[0147] In the specific embodiments described above, the purpose, technical solutions and beneficial effects of the present invention have been further described in detail. It should be understood that the above description is only the specific embodiments of the present invention and is not used to limit the protection scope of the present invention. It is particularly pointed out that for those skilled in the art, any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A prediction method based on lung image processing, characterized in that, Including: Obtaining a to-be-processed lung image and respiratory parameters of a target user; The to-be-processed lung image includes multiple lung segments; Performing spatial registration on the to-be-processed lung image and the respiratory parameters to obtain spatially registered result data; Inputting the spatially registered result data into a preset graph neural network to construct a functional connection relationship between the multiple lung segments, and further obtaining the functional contribution weights of each lung segment based on the functional connection relationship; Based on the spatially registered result data and the functional contribution weights of each lung segment, using a preset temporal convolutional network for prediction to obtain temporally registered result data; Obtaining biomarker data of the target user; inputting the temporally registered result data and the biomarker data into a preset cross-modal prediction model to obtain tidal volume change data and hypoxemia risk data of the target user.

2. The prediction method based on lung image processing according to claim 1, wherein The preset cross-modal prediction model includes a Transformer model; the inputting the temporally registered result data and the biomarker data into the preset cross-modal prediction model to obtain the tidal volume change data and the hypoxemia risk data of the target user includes: Inputting the temporally registered result data and the biomarker data into the Transformer model, performing embedded representation according to a preset fusion formula to obtain a fusion feature, and further obtaining the tidal volume change data and the hypoxemia risk data of the target user based on the fusion feature; The preset fusion formula is expressed as: W fusion = α·X img + β·X bio ; Among them, W fusion represents the fusion feature, X img is the time series registration result data, and X bio is the biomarker.

3. The prediction method based on lung image processing according to claim 1, wherein The inputting the spatially registered result data into a preset graph neural network to construct a functional connection relationship between the multiple lung segments, and further obtaining the functional contribution weights of each lung segment based on the functional connection relationship includes: According to the spatially registered result data, analyzing the ventilation distribution of each lung segment using a density histogram; Inputting the spatially registered result data and the ventilation distribution into the preset graph neural network, so that the preset graph neural network takes the lung segments as nodes and the blood flow data corresponding to the lung segments as edge weights to predict the ventilation weights of the corresponding regions of each lung segment; and obtaining the functional contribution weights of each lung segment based on the ventilation weights of the corresponding regions of each lung segment.

4. The prediction method based on lung image processing according to claim 1, wherein The performing prediction using a preset temporal convolutional network based on the spatially registered result data and the functional contribution weights of each lung segment to obtain temporally registered result data includes: Constructing the spatially registered result data and the functional contribution weights of each lung segment into a data set; Using XGBoost to extract temporal features from the data set; Inputting the temporal features into the preset temporal convolutional network for prediction to obtain temporally registered result data.

5. A prediction method based on lung image processing according to any one of claims 1 to 4, characterized in that, After the obtaining the to-be-processed lung image and the respiratory parameters of the target user, the prediction method further includes: Performing finite element division on the to-be-processed lung image to obtain a number of lung tissue meshes; Loading mechanical parameters on the lung tissue meshes to simulate the stress distribution of the lung tissue meshes and obtain a simulation result; Obtaining the user historical data and a virtual resection plan of the target user, and inputting the simulation result, the user historical data and the virtual resection plan into a preset generative adversarial network to generate a postoperative compensation simulation diagram.

6. The prediction method based on lung image processing according to claim 5, wherein The preset generative adversarial network includes a generator and a discriminator; The generator is used to obtain a generated resection image based on the lung tissue mesh, the virtual resection plan, and the user historical data of the target user; The discriminator is used to perform discrimination based on the generated resection image and the real resection image to obtain a discrimination result, and then generate the postoperative compensation simulation diagram by using the discrimination result.

7. A prediction method based on lung image processing according to any one of claims 1 to 4, characterized in that, The prediction method further includes: Monitoring the tidal volume change data and hypoxemia risk data of the target user, and dynamically updating the functional contribution weights of the lung segments according to the monitoring results.

8. A prediction device based on lung image processing, characterized in that, It includes an acquisition module, a spatial registration module, a weight allocation module, a temporal registration module, and a prediction module; wherein, The acquisition module is used to acquire the to-be-processed lung image and respiratory parameters of the target user; the to-be-processed lung image includes multiple lung segments; The spatial registration module is used to perform spatial registration on the to-be-processed lung image and the respiratory parameters to obtain spatial registration result data; The weight allocation module is used to input the spatial registration result data into a preset graph neural network to construct a functional connection relationship between the multiple lung segments, and then obtain the functional contribution weights of the lung segments based on the functional connection relationship; The temporal registration module is used to perform prediction by using a preset temporal convolutional network based on the spatial registration result data and the functional contribution weights of the lung segments to obtain temporal registration result data; The prediction module is used to acquire the biomarker data of the target user; input the temporal registration result data and the biomarker data into a preset cross-modal prediction model to obtain the tidal volume change data and hypoxemia risk data of the target user.

9. A terminal device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a prediction method based on lung image processing according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein when the computer program runs, it controls the device where the computer-readable storage medium is located to execute a prediction method based on lung image processing according to any one of claims 1 to 7.

Images