STRes3dRNN model, evaluation system and evaluation method
The STRes3dRNN model extracts spatiotemporal features from the EIT image sequence, which solves the subjectivity and radiation exposure problems of existing lung function evaluation methods, realizes accurate evaluation of lung function status, and improves the objectivity and accuracy of the evaluation.
Patent Information
- Application Number
- CN202411002226.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-25
AI Technical Summary
The existing lung function assessment methods rely on active cooperation of patients, and there are problems of subjectivity and radiation exposure, which are difficult to be effectively applied in clinical practice, especially in early case screening, which is prone to incorrect assessment.
Using the STRes3dRNN model, combined with EIT image sequence, the temporal and spatial characteristics are extracted through 3D residual network, mixed attention mechanism and two-way long and short-term memory network, to realize automated evaluation of lung function, reduce the impact of redundant features, and enhance attention to key features. Spend.
Accurate assessment of lung function status is achieved, the objectivity and accuracy of the assessment is improved, the risk of radiation exposure is reduced, the evaluation process is simplified, and it is suitable for early case screening.
Smart Images

Figure CN119027722A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of medical imaging and deep learning, and specifically relates to a STRes3dRNN model, an evaluation system, and an evaluation method. Background Art
[0002] Respiratory diseases are the third leading cause of death worldwide. Early screening of lung diseases is crucial to delay disease progression and improve patient prognosis. Electrical impedance tomography (EIT) is a non-invasive medical imaging technology that can visualize the conductivity distribution caused by pulmonary ventilation in real time and provide continuous dynamic information of pulmonary ventilation. Although studies have used EIT technology to confirm the heterogeneity of pulmonary ventilation function in patients with chronic obstructive pulmonary disease (COPD), no one has used EIT image sequences for clinical patient lung function assessment to distinguish between normal, obstructive or damaged states.
[0003] In order to overcome the above problems, researchers evaluated lung function by extracting and processing impedance changes at different time series. After obtaining the time series EIT images of the subject, the regional impedance distribution map containing time characteristics was analyzed, and the distribution characteristic parameters were obtained based on this. By comparing with the standard distribution characteristic parameters, it was determined whether the regional ventilation function was damaged (such as: CN118121183A, CN118121184A).
[0004] Based on the above existing technology research, it is found that the following problems still exist in the application of EIT in etiology diagnosis related technologies:
[0005] (1) Existing lung function assessment methods include lung function tests and medical imaging examinations. All of these methods rely on the patient's active cooperation and cumbersome breathing movements, which have limitations and are difficult to implement in actual clinical applications.
[0006] (2) Chest X-rays and CT scans are usually accompanied by issues of radiation exposure and expensive equipment, which leads to shortcomings in regular examinations and treatment monitoring, limiting their application scenarios and frequency of use.
[0007] (3) Manual interpretation of images obtained by EIT technology is subjective and prone to misjudgment in early case screening. Simple time-series image processing cannot accurately determine lung function status. Summary of the invention
[0008] The purpose of the present invention is to solve the problems existing in the above-mentioned prior art and to provide a STRes3dRNN (3DSpatiotemporal ResidualRecurrent Neural Network) model, an evaluation system, and an evaluation method and system for evaluating pulmonary function using electrical impedance tomography.
[0009] The technical solution of this application is:
[0010] A STRes3dRNN model, comprising: a network input layer, an initial stacking layer, an intermediate stacking layer, a final stacking layer, and a network output layer connected in sequence;
[0011] Among them, the network input layer is used to read the EIT image sequence;
[0012] The initial stacking layer is formed by stacking a 3D residual initial unit, an intermediate 3D residual standard unit, and a final 3D residual standard unit in sequence; the 3D residual initial unit is used to preliminarily extract high-dimensional features of the image, the intermediate 3D residual standard unit is used to reduce the dimension of the high-dimensional image features, and the final 3D residual standard unit is used to reduce the dimension and deepen the feature information;
[0013] Among them, the intermediate stacking layer includes: n sequentially connected monomer intermediate stacking layers; the monomer intermediate stacking layer is stacked by HAM units, 3D residual lower residual units, and final 3D residual standard units; n is a natural number greater than or equal to 1; the HAM unit optimizes the weight of the allocated information, removes redundant information, and strengthens the feature information; the 3D residual lower residual unit increases the dimension of the low-dimensional image features; the final 3D residual standard unit is used to reduce the dimension and deepen the feature information;
[0014] Among them, the final stacking layer is formed by stacking a 3D global average pooling layer, a BiLSTM layer, and a fully connected layer in sequence; the 3D global average pooling layer reads the image sequence information obtained by the middle stacking layer and extracts its feature vector; the BiLSTM layer processes and outputs the feature vector; the fully connected layer maps the input feature vector to the number of target categories and obtains the mapped score value, and the target categories are: healthy, blocked, and damaged.
[0015] Furthermore, the 3D residual initial unit includes a sequentially connected 3D convolution layer, a batch normalization layer, an activation layer, and a 3D maximum pooling layer; the three-dimensional convolution layer is used to convolutionally extract image sequence features; the batch normalization layer is used to normalize the sequence features to accelerate the convergence speed; the activation layer processes the data of the previous step through a nonlinear activation function to improve the representation ability of the features; the 3D maximum pooling layer uses different channels to compress the features to achieve invariance, extract and output the maximum value.
[0016] Furthermore, the middle 3D residual standard unit and the final 3D residual standard unit both include sequentially connected 3D convolutional layers, batch normalization layers, activation layers, 3D convolutional layers, batch normalization layers, additional layers, and activation layers; the 3D convolutional layer is used to convolutionally extract image sequence features; the batch normalization layer is used to normalize the sequence features to speed up the convergence speed; the activation layer processes the data given by the batch normalization layer through a nonlinear activation function to improve the representation ability of the features; the additional layer is used to randomly close certain nodes to prevent overfitting and avoid network degradation.
[0017] Furthermore, the HAM unit includes a spatiotemporal attention module, a channel attention module and a motion attention module; the aforementioned three modules are applied to enhance the feature information in terms of spacetime, channel and motion respectively, and the enhanced information is weighted to obtain new data of fusion enhancement of spacetime, channel and motion information with the number of layers and dimensions remaining unchanged.
[0018] Furthermore, the 3D residual downsampling unit includes a sequentially connected 3D convolution layer, a batch normalization layer, an activation layer, a 3D convolution layer, a batch normalization layer, an additional layer, and an activation layer; the 3D convolution layer is used to convolutionally extract image sequence features; the batch normalization layer normalizes the sequence features to speed up the convergence speed; the activation layer processes the data of the previous step through a nonlinear activation function to improve the representation ability of the features; the additional layer is used to randomly close certain nodes to prevent overfitting and avoid network degradation.
[0019] Furthermore, the HAM unit includes a spatiotemporal attention module, a channel attention module and a motion attention module; the aforementioned three modules are used to enhance the feature information in the spatiotemporal, channel and motion aspects respectively, and the enhanced information is weighted to obtain the fusion enhanced data of the spatiotemporal, channel and motion information with the new number of layers and dimensions remaining unchanged;
[0020] Spatial-Temporal Excitation (STE): After average pooling the channel C dimension, this module uses a 3D convolution kernel of size 3×3×3 to model the information of the temporal and spatial dimensions to capture the spatiotemporal features in the EIT image sequence;
[0021] Channel Excitation (CE): This module performs average pooling on the height H and width W dimensions to reduce the number of channels to 1 / 16 of the original, and then uses a 1D convolution kernel of size 3 between two fully connected layers (FC) to represent the channel features in the time dimension;
[0022] Motion Excitation (ME): This module performs tensor separation on the number of frames T, processes them using a 2D convolution kernel of size 3×3, and adds differential processing of adjacent frames to simulate the motion information of EIT lung ventilation changes.
[0023] Furthermore, the BiLSTM layer is composed of two channels, forward propagation and backward propagation, and performs forward and backward processing simultaneously. The processing time step is determined by the input image sequence, and the time step equal to the number of sequences is automatically cycled; the forward and backward processing results are spliced to obtain a high-dimensional feature vector.
[0024] An evaluation system comprising:
[0025] A storage unit, which is used to store continuous voltage data measured by the EIT device;
[0026] An EIT image sequence generating unit, which reads the continuous voltage data of the storage unit to generate an EIT image sequence;
[0027] STRes3dRNN model, which reads the EIT image sequence of the EIT image sequence generation unit, and its result output is passed to the probability calculation unit;
[0028] Probability calculation unit, which is used to give the probability percentage of health, blockage, and damage.
[0029] Furthermore, the working method of the probability calculation unit is:
[0030]
[0031] Softmax(z i ) represents the predicted probability of the i-th category, represents the raw score of the i-th category.
[0032] An assessment method, which is not intended for direct disease diagnosis, comprises the following steps:
[0033] S100, acquiring continuous voltage data to generate an EIT image sequence; the EIT network input is a 16-frame 64×64×3 EIT image sequence, where 64×64 is the image size and 3 is a three-channel RGB image;
[0034] S200, transmitting the image sequence I to an initial stacking layer; the initial stacking layer is formed by stacking a 3D residual initial unit, an intermediate 3D residual standard unit, and a final 3D residual standard unit;
[0035] S200 includes the following sub-steps:
[0036] S201, the image sequence I is input into the 3D residual initialization unit to obtain a high-dimensional feature of a 64-layer 7×7×7 image sequence;
[0037] S202, the high-dimensional features obtained in step S201 are input into the intermediate 3D residual standard unit to obtain image features after dimensionality reduction;
[0038] S203, the image features obtained in step S202 are input into the last 3D residual standard unit to obtain an image sequence after dimensionality reduction and feature information deepening;
[0039] S300, inputting the result calculated by the initial stacking layer to the intermediate stacking layer; the intermediate stacking layer comprises: 3 sequentially connected monomer intermediate stacking layers; the monomer intermediate stacking layer is formed by stacking a HAM unit, a 3D residual lower residual unit, and a 3D residual standard unit:
[0040] S300 includes the following sub-steps:
[0041] S301, the HAM unit in the middle stacking layer of the first monomer reads a 128-layer 1×1×1 image sequence, optimizes the weight of the allocated information, removes redundant information, strengthens the feature information, and outputs a 128-layer 1×1×1 image sequence;
[0042] S302, the 3D residual downsampling unit of the first monomer intermediate stacking layer reads a 128-layer 1×1×1 image sequence and outputs a 128-layer 3×3×3 image sequence;
[0043] S303, the 3D residual standard unit at the end of the first monomer intermediate stacking layer reads a 128-layer 3×3×3 image sequence and outputs a 256-layer 1×1×1 image sequence;
[0044] S304, the HAM unit of the second monomer intermediate stacking layer reads an image sequence of 256 layers of 1×1×1 image sequences, removes redundant information, strengthens feature information, and outputs an image sequence of 256 layers of 1×1×1 image sequences;
[0045] S305, the 3D residual downsampling unit of the second monomer intermediate stacking layer reads a 256-layer 1×1×1 image sequence and outputs a 256-layer 3×3×3 image sequence;
[0046] S306 , the 3D residual standard unit at the end of the second monomer intermediate stacking layer reads an image sequence of 256 layers of 3×3×3 image sequences and outputs an image sequence of 512 layers of 1×1×1 image sequences;
[0047] S307, the HAM unit in the middle stack layer of the third monomer reads an image sequence of 512 layers of 1×1×1 image sequences, removes redundant information, strengthens feature information, and outputs an image sequence of 512 layers of 1×1×1 image sequences;
[0048] S308, the 3D residual downsampling unit of the third monomer intermediate stacking layer reads a 512-layer 1×1×1 image sequence and outputs a 512-layer 3×3×3 image sequence;
[0049] S309 , the 3D residual standard unit at the end of the third monomer intermediate stacking layer reads an image sequence of 512 layers of 3×3×3 image sequences and outputs a 512 layer 1×1×1 image sequence;
[0050] S400, the result calculated by the middle stacking layer is input to the last stacking layer, and the last stacking layer includes a three-dimensional average layer, a BiLSTM layer, and a fully connected layer connected in sequence; step S400 also includes the following sub-steps:
[0051] S401, a three-dimensional averaging layer reduces the dimension of the 512-layer 1×1×1 image sequence and extracts its feature vector;
[0052] S402, the BiLSTM layer processes and outputs 128 layers of feature vectors;
[0053] S403, the fully connected layer maps the input feature vector to the target classification and obtains the mapped score value; wherein the number of target classifications is 3, which represent health, obstruction, and damage respectively.
[0054] Furthermore, the method further includes: S500, the result of the final stacked layer is input into the Softmax function, and the predicted probability is output, that is, the probability percentage of health, obstruction, and damage:
[0055]
[0056] Among them, Softmax(z i ) represents the predicted probability of the i-th category, represents the raw score of the i-th category.
[0057] The advantages of the technical solution of the present invention are mainly reflected in:
[0058] (1) This application develops a STRes3dRNN model that can effectively extract spatiotemporal features from EIT image sequences and evaluate the functional status of patients' organs (such as the brain and lungs).
[0059] (2) For the three-dimensional residual network, the network degradation problem is prone to occur as the number of network layers increases during its application. Therefore, this application introduces three-dimensional residual learning to optimize the training process and alleviate the degradation problem. The residual connection allows the input to skip the 3D convolution unit of the main branch, simplifies the gradient transfer path, and provides a simpler network flow path, thereby improving network performance and training efficiency.
[0060] The residual unit output function of this application is:
[0061]
[0062] in, represents the input of the 3D residual, f Res3D Represents the residual function, and δ represents the learnable parameter. The residual connection allows the input to skip the 3D convolution unit of the main branch, simplifies the gradient transfer path, and provides a simpler network flow path, thereby improving network performance and training efficiency.
[0063] (3) The hybrid attention mechanism can redistribute the weights between spatiotemporal features, enhance the focus on key features, reduce the adverse effects of redundant features on spatiotemporal classification, and provide more accurate model support for EIT lung function assessment.
[0064] The output function of the hybrid attention mechanism determined in this application is:
[0065]
[0066] in, represents the input of the 3D residual, is a mixed activation function and θ represents all learnable parameters.
[0067] (4) The bidirectional long short-term memory network can capture the long-term dependencies in the EIT image sequence and can also capture information from two different time directions, greatly enhancing the ability to capture the correlation in spatiotemporal features.
[0068] The function of the bidirectional long short-term memory network determined in this application is:
[0069]
[0070] Where ⊙ represents the Hadamard product, i, f, g, and o represent the input gate, forget gate, memory unit, and output gate, respectively, and σ c These gating mechanisms enable BiLSTM to effectively manage the storage and forgetting of information, which is crucial for capturing long-term dependencies.
[0071] (5) The spatiotemporal classification prediction includes a fully connected layer (FC) and a Sofamax function, which can integrate the spatiotemporal feature information extracted in the previous steps and summarize it to form a global representation. By combining a three-dimensional residual network, a hybrid attention mechanism, and a bidirectional long short-term memory network, the spatiotemporal feature information in the EIT pulmonary ventilation changes is effectively extracted and learned. BRIEF DESCRIPTION OF THE DRAWINGS
[0072] The present application is further described in detail below in conjunction with the embodiments in the accompanying drawings, but does not constitute any limitation to the present application.
[0073] Figure 1 This is the architecture design diagram of the STRes3dRNN model in the present invention.
[0074] Figure 2 This is a schematic diagram of the lung function assessment principle of the present invention.
[0075] Figure 3 This is the EIT indicator diagram in the present invention.
[0076] Figure 4 This is the actual test diagram in the present invention.
[0077] Figure 5 This is the training progress of the STRes3dRNN model in the present invention.
[0078] Figure 6 This is the STRes3dRNN confusion matrix classification result in the present invention.
[0079] Figure 7 This is the STRes3dRNN parameter optimization experiment in the present invention.
[0080] Figure 8 The comparison results of EIT spatiotemporal classification performance indicators based on different network models in the present invention. DETAILED DESCRIPTION
[0081] The purpose, advantages and features of the present invention will be explained by the non-limiting description of the following preferred embodiments. These embodiments are only typical examples of the application of the technical solution of the present invention, and any technical solution formed by equivalent replacement or equivalent transformation falls within the scope of protection claimed by the present invention.
[0082] <1: A method for evaluating pulmonary function using electrical impedance tomography based on a three-dimensional spatiotemporal residual recurrent neural network>
[0083] <1.1、Solved technical problems>
[0084] Although previous studies have used EIT technology to confirm the heterogeneity of lung ventilation function in COPD patients, no one has yet used EIT image sequences for clinical lung function assessment to distinguish between normal, obstructive or damaged states.
[0085] <1.2、Technical ideas>
[0086] The technical idea of the pulmonary function assessment method based on electrical impedance tomography of three-dimensional spatiotemporal residual recurrent neural network proposed in this application is:
[0087] (1) Res3d is constructed by stacking multiple 3D convolution kernels and 3D residual units to capture various spatiotemporal correlations in the EIT image sequence and output a feature sequence graph rich in spatiotemporal coupling information;
[0088] (2) Using HAM to redistribute the weights of spatiotemporal features, reduce the adverse effects of redundant features on spatiotemporal classification, and improve accuracy;
[0089] (3) BiLSTM is used to capture long-term dependencies in EIT image sequences, enhancing sensitivity to temporal dynamics and pulmonary ventilation change trends;
[0090] (4) Combining the three-dimensional residual network, hybrid attention mechanism and bidirectional long short-term memory network, the spatiotemporal feature information of EIT pulmonary ventilation changes can be effectively extracted and learned to accurately identify and classify lung diseases.
[0091] <1.3. Overall technical solution>
[0092] The evaluation principle of the method of this application is as follows Figure 2 As shown, the EIT technology applies current excitation on the body surface and measures voltage changes, combines the finite element physical field model to reconstruct the lung impedance distribution image, and uses the functional evaluation method of electrical impedance tomography based on three-dimensional spatiotemporal residual recurrent neural network to perform functional evaluation.
[0093] <1.4、STRes3dRNN architecture design>
[0094] Table 1 STRes3dRNN architecture table of this application
[0095]
[0096]
[0097] It should be noted that the "layer" in "64 layers, 128 layers, 256 layers" in Table 1 refers to the depth of the convolutional neural network, that is, the number of network layers.
[0098] It should be noted that the 3D residual initial unit includes a 3D convolution layer, a batch normalization layer, an activation layer, and a 3D maximum pooling layer connected in sequence; the three-dimensional convolution layer is used to convolutionally extract image sequence features; the batch normalization layer is used to normalize the sequence features to speed up the convergence speed; the activation layer processes the data of the previous step through a nonlinear activation function to improve the representation ability of the features; the 3D maximum pooling layer uses different channels to compress the features to achieve invariance, extract and output the maximum value.
[0099] The 3D residual standard unit (middle and end) includes (which belongs to the prior art, see: https: / / mp.weixin.qq.com / s / Isv9JZp9qjwRgnNqJyjfxg) sequentially connected 3D convolution layer, batch normalization layer, activation layer, 3D convolution layer, batch normalization layer, additional layer, activation layer; the 3D convolution layer is used to convolutionally extract image sequence features; the batch normalization layer is used to normalize the sequence features to speed up the convergence speed; the activation layer processes the data given by the batch normalization layer through a nonlinear activation function to improve the representation ability of the features; the additional layer is used to randomly close certain nodes to prevent overfitting and avoid network degradation.
[0100] It should be noted that the HAM unit includes a spatiotemporal attention module, a channel attention module and a motion attention module; the aforementioned three modules are used to enhance the feature information in terms of spacetime, channel and motion respectively, and the enhanced information is weighted to obtain the fusion enhanced data of spacetime, channel and motion information with the new number of layers and dimensions remaining unchanged.
[0101] It should be noted that the 3D residual downsampling unit includes a sequentially connected 3D convolution layer, a batch normalization layer, an activation layer, a 3D convolution layer, a batch normalization layer, an additional layer, and an activation layer; the 3D convolution layer is used to convolutionally extract image sequence features; the batch normalization layer normalizes the sequence features to speed up the convergence speed; the activation layer processes the data of the previous step through a nonlinear activation function to improve the representation ability of the features; the additional layer is used to randomly close certain nodes to prevent overfitting and avoid network degradation.
[0102] It should be noted that the BiLSTM layer consists of two channels, forward propagation and backward propagation, and performs forward and backward processing at the same time. The processing time step is determined by the input image sequence, and the time step equal to the number of sequences is automatically cycled; the forward and backward processing results are spliced to obtain a high-dimensional feature vector.
[0103] <1.5 Algorithm>
[0104] A working method of a STRes3dRNN model comprises the following steps:
[0105] S100, acquiring continuous voltage data to generate an EIT image sequence I, the EIT network input is a 16-frame 64×64×3 EIT image sequence, 64×64 is the image size, and 3 is a three-channel RGB image;
[0106] S200, transmitting the image sequence I to the initial stacking layer (stack1), the initial stacking layer is formed by stacking 3D residual initial unit, 3D residual standard unit, and 3D residual standard unit in sequence: S200 includes the following sub-steps:
[0107] S201, the image sequence I is input into the 3D residual initialization unit to obtain a high-dimensional feature of a 64-layer 7×7×7 image sequence;
[0108] S202, the high-dimensional features obtained in step S201 are input into the intermediate 3D residual standard unit to obtain image features after dimensionality reduction;
[0109] S203, the image features obtained in step S202 are input into the last 3D residual standard unit to obtain an image sequence after dimensionality reduction and feature information deepening;
[0110] S300, inputting the result of the initial stacking layer (stack1) to the intermediate stacking layers (stack2-stack4);
[0111] The intermediate stacking layer includes: 3 sequentially connected monomer intermediate stacking layers; the monomer intermediate stacking layer is stacked by HAM units, 3D residual lower residual units, and 3D residual standard units:
[0112] The working steps of the monomer intermediate stacking layer are:
[0113] S301, the HAM unit reads the image sequence, removes redundant information, strengthens feature information, and then outputs the image sequence;
[0114] S302, the 3D residual downsampling unit upgrades the low-dimensional features and outputs an image sequence after the upgrade;
[0115] S303, a 3D residual standard unit reduces the dimension of the high-dimensional image sequence to obtain an image sequence with enhanced feature information;
[0116] S400, the result of the middle stacking layer is input to the final stacking layer (stack5), the final stacking layer; the final stacking layer includes the following sequentially connected layers: 3D global average pooling layer, BiLSTM layer, and fully connected layer;
[0117] S400 includes the following sub-steps:
[0118] S401, 3D global average pooling layer reads 512 layers of 1×1×1 image sequence to reduce the dimension and extract its feature vector;
[0119] S402, the BiLSTM layer processes and outputs 128 layers of feature vectors;
[0120] S403, the fully connected layer maps the input feature vector to the number of target categories (the number is 3, healthy, blocked, damaged), and obtains the mapped score value;
[0121] S500, the result of the final stacked layer is input to the Softmax function, which outputs the predicted probability, i.e., the probability percentage of health, blockage, and damage, and the sum of the three is equal to 1:
[0122]
[0123] Among them, Softmax(z i ) represents the predicted probability of the i-th category, Represents the original score (i.e., rating value) of the i-th category.
[0124] When the network is trained, the loss is calculated through the cross entropy loss function. For each training, the Softmax function will output the predicted result. The difference between the predicted result and the actual result is calculated through the loss function formula. When the maximum number of training times is reached or the minimum error reaches the set threshold, the training stops. The loss function formula is:
[0125]
[0126] Among them, y i is the one-hot encoded value of the i-th category in the true label, is the classification probability of the i-th category in the model prediction.
[0127] <2: Clinical Experiment Design>
[0128] <2.1. Subject Description>
[0129] From June 2022 to September 2023, a total of 192 subjects were recruited and their written informed consent was obtained. Before the test, 6 subjects with contraindications to PFT or EIT measurement were excluded. During the test, 9 subjects were unable to effectively complete the lung function test, and another 12 subjects had poor contact with the EIT measurement system. After these exclusions, the respiratory data of a total of 165 subjects were used for subsequent clinical analysis.
[0130] The baseline physical characteristics, PFT values, and EIT indices of different subjects are listed in Tables I and Figure 3Statistically, there were no significant differences among the three groups (P>0.05): 60 subjects with healthy lungs (age 54.68±8.37 years, height 170.98±8.36 cm, weight 67.50±9.08 kg) were classified as the healthy group; 55 subjects with chronic obstructive pulmonary disease (age 69.57±9.09 years, height 163.26±8.57 cm, weight 63.07±9.27 kg) were classified as the obstruction group; and subjects with diseases such as pneumonia, pneumothorax, and pleural effusion (age 65.21±10.37 years, height 168.34±9.26 cm, weight 64.29±11.05 kg) were classified as the injury group.
[0131] The subjects were required to sit for EIT measurement, and after the measurement was stable, they breathed calmly for about 1 to 3 minutes, and the EIT data was recorded simultaneously. This process was repeated three times to collect multiple sets of data for analysis. If the subject felt uncomfortable, the measurement was terminated immediately. Both EIT and PFT measurements were performed after approval by the Medical Ethics Committee of Jinan University (JNUKY-2022-005) and registration of the clinical trial.
[0132] Table 2 Baseline physical characteristics of different subjects
[0133]
[0134]
[0135] <2.2. EIT data collection>
[0136] The EIT signal was collected in real time by a thoracic impedance tomography (EIT-1000, Nanjing Jilun Medical Engineering Intelligent Technology Co., Ltd., Jiangsu, China). Figure 4 As shown in Figure 1, the system consists of wearable electrode sensors, data acquisition hardware, and image reconstruction software. In order to accurately obtain pulmonary ventilation activity information, a fourth-order low-pass digital filter is designed in the system to effectively remove cardiac signals and measurement noise.
[0137] For each breathing cycle, with end-expiration V0 as the reference point, the related voltage signal ΔV i =(V i -V0) / V0(i=1,2,…,T) is used as the input of the GN algorithm for image reconstruction. To enhance lung visualization, the EIT image sequence D={Δσ i}(i=1,2,...,T) are normalized to the range of [0,1]. Assume D min and D max Represent the minimum and maximum pixel values of the EIT image respectively, then the normalized image sequence It is expressed as:
[0138]
[0139] Due to the differences in breathing time of subjects, the inconsistency of EIT image sequences in the temporal dimension poses a challenge to the spatiotemporal classification task based on STRes3dRNN. To solve this problem, 16 consecutive EIT images are extracted at equal intervals as network input. For the case of insufficient breathing time, a zero-padding strategy is used to ensure that the temporal dimension of the EIT image sequences of all patients is consistent.
[0140] <2.3、Model training>
[0141] The clinical data is based on the EIT data of 165 subjects in calm breathing, and the continuous EIT images extracted in each respiratory cycle constitute a sample. The dataset contains a total of N = 7654 EIT image sequences, where the size of the input image is X = 64 × 64 × 16 × 3 (height × width × depth × channel). Pulmonary function assessment involves three diagnostic categories: Y = healthy, obstructive and damaged, which increases the complexity of model prediction.
[0142] During the training process, we used the Adam optimizer, set the mini-batch size to B = 128, the number of iterations to E = 50, and the initial learning rate to α = 0.001. After completing 10 training epochs, the learning rate was gradually reduced by the decay factor β = 0.9, and the other parameters were default. In addition, to enhance the generalization ability of the model, we performed data augmentation on the dataset after each training cycle, including rotation, translation, scaling, flipping, and cropping, to generate new training samples.
[0143] The computer hardware environment used in the experiment includes: Intel Core i9-14900K CPU@3.2GHz processor, NVIDIA RTX A4000 graphics card and 4800MHz DDR5@64GB RAM memory. The network model is implemented using Python language and TensorFlow framework, and image reconstruction is completed through MATLAB. Using the ten-fold cross-validation method, 10% of the data is randomly selected as the test set, and the rest is used as the training set.
[0144] <2.4. Evaluation Criteria>
[0145] To evaluate the classification performance of the STRes3dRNN model, this application uses the following indicators: Accuracy (A), Precision (P), Recall (R), and F1 Score (F1). Accuracy refers to the ratio of the number of samples correctly predicted by the model to the total number of samples, and is an intuitive indicator for measuring classification accuracy. Precision reflects the proportion of samples predicted by the model to be positive that are actually positive. Recall indicates the proportion of positive samples classified by the model to all actual positive samples. F1 Score is the harmonic mean of precision and recall, which comprehensively reflects the performance of these two indicators.
[0146] The calculation formula is:
[0147]
[0148] Among them, TP (true positive), TN (true negative), FP (false positive) and FN (false negative) represent the number of different types of prediction results.
[0149] <2.5、Parameter Optimization Experiment>
[0150] (a) Initial learning rate α and image sequence depth T: In order to explore the impact of different initial learning rates α and image sequence depth T on the classification performance of STRes3dRNN, we selected initial learning rates α as 0.1, 0.01, 0.001 and 0.0001, and image sequence depths T as 8, 16, 32 and 64 for combination testing.
[0151] (b) Training sample size N: The training sample size N is an important parameter in the EIT health diagnosis task. More training samples can provide richer prior information, thereby improving the classification accuracy to a certain extent. In order to verify the robustness of the STRes3dRNN model, we compared the classification accuracy and loss values under different training sample ratios. We randomly selected 60%, 65%, 70%, 75% and 80% of the samples from the EIT image sequence as training sets, and the remaining 20% was used for validation.
[0152] <2.6. Ablation comparison test>
[0153] In order to verify the classification performance of the proposed method in lung function assessment, we designed an ablation comparison experiment. This experiment verifies the effectiveness of each module in the network by removing a module one by one and comparing the classification effect. We mainly compared the following three network configurations: the basic three-dimensional residual network Res3d, STRes3d combined with the HAM module, and STRes3dRNN fused with the BiLSTM module (our method). In addition, we also tested three mainstream network models, including C3D, R(2+1)D, and Variational Auto-Encoders (VAE), for performance comparison.
[0154] <3: Results and Analysis>
[0155] <3.1、Training progress and classification results of STRes3dRNN model>
[0156] The training progress of the STRes3dRNN model is as follows Figure 5 As shown in the figure, the entire training process is nearly 5000 iterations. Each iteration includes the estimation of the network gradient and the update of the weight parameters. The alternating background color in the figure is used to mark each round of training, where one round of training represents a complete traversal of the entire dataset.
[0157] In the initial stage (0-1000 iterations), the model's understanding of EIT data is still shallow, and the classification accuracy and loss value fluctuate greatly, but the overall trend is rapid rise and fall, indicating that the model is learning; in the fluctuation stage (1000-3000 iterations), the model's understanding of EIT data deepens, parameter adjustment is more stable, the classification accuracy gradually increases, and the decline of the loss value slows down, indicating that the model performance is gradually maturing; in the stable stage (after 3000 iterations), the model's classification accuracy reaches 97.40%, and the loss value is less than 0.14, showing that the model is relatively mature; Overall, the training progress process shows the gradual improvement of STRes3dRNN performance and eventually stabilizes. In the left figure, the training accuracy (blue) is smoothed to show the performance improvement trend, and the validation accuracy (black) reflects the performance of the model on unseen data. In the right figure, the training loss (yellow) and validation loss (black) reflect the loss of each mini-batch and the entire validation set, respectively.
[0158] In the trained STRes3dRNN model, a total of 765 EIT cases were used for health diagnosis tasks, and the confusion matrix classification results are shown in Figure 6The figure clearly reveals the relationship between the prediction results and the true labels. The overall accuracy is 97.40%, shown in the cell in the lower right corner of the figure, and the error rate is only 2.6%. The right column and bottom row of the figure show the precision and recall of each category. In the healthy class, 98.4% of the 251 predicted samples were correctly classified and 98.8% of the 250 actual samples were correctly predicted. In the obstruction class, 97.4% of the 268 predicted samples were correctly classified and 96.3% of the 271 actual samples were correctly predicted. In the injury class, 96.3% of the 246 predicted samples were correctly classified and 97.1% of the 271 actual samples were correctly predicted. These classification results show that the STRes3dRNN model has high accuracy and reliability in health diagnosis tasks, and can effectively distinguish individuals under different lung ventilation conditions, thereby providing more favorable technical support for clinical diagnosis and treatment.
[0159] <3.2. Experimental results of STRes3dRNN model parameter optimization>
[0160] The classification accuracy results of the initial learning rate α and the image sequence depth T are as follows Figure 7 As shown in the left figure. When the initial learning rate α is fixed, as the image sequence depth T increases, the classification accuracy usually improves, indicating that the network can learn more complex spatiotemporal features. However, too deep an image sequence may lead to overfitting and affect the accuracy. For example, when α=0.001, T increases from 16 to 64, and the accuracy drops from 97.40% to 95.78%. When the image sequence depth T is fixed, a proper reduction in the initial learning rate α helps the network learn faster, but too low an initial learning rate may cause the network to fall into a local minimum, affecting the classification performance. Among all the test combinations, the network model performs best with an accuracy of 97.40% when the initial learning rate α is 0.001 and the image sequence depth T is 16. This shows that this parameter combination has achieved a good balance between the network's learning ability and generalization ability, effectively avoiding overfitting while making full use of computing resources.
[0161] The comparison results of training sample size N are as follows Figure 7 As shown in the right figure. When the proportion of training samples is only 60%, despite the small number of training samples, our proposed method can effectively extract deep spatiotemporal features, thereby achieving a high classification accuracy (Accuracy = 83.58 ± 0.61% and Loss = 1.04 ± 0.04). As the proportion of training samples increases, the classification accuracy gradually improves and tends to stabilize. When the sample size is sufficient, the classification accuracy can reach the highest. Therefore, especially when the sample size is scarce, our method can also show good classification robustness.
[0162] <3.3. Ablation comparison test results>
[0163] All six network models were tested for classification using the ten-fold cross-validation method. The quantitative results are shown in Figure 8 As shown. In the four evaluation indicators, STRes3dRNN scored the highest, specifically Pre = 97.37 ± 0.86%, Rec = 97.40 ± 1.04%, F1 = 97.38 ± 0.86% and Acc = 97.40%. Among them, for the recognition of normal subjects, the F1 score reached the highest 98.40%, which shows that the STRes3dRNN model has almost no error in judging the health status, and the lung function diagnosis effect of healthy people is very reliable.
[0164] The above embodiments are preferred implementation modes of the present invention and are only used to facilitate the description of the present invention. They are not intended to limit the present invention in any form. Any person with ordinary knowledge in the relevant technical field, if they do not depart from the scope of the technical features of the present invention, can make equivalent embodiments by partial changes or modifications to the technical contents disclosed in the present invention, and they still fall within the scope of the technical features of the present invention without departing from the technical features of the present invention.
Claims
1. A STRes3dRNN model, characterized in that: Includes the sequential connections: network input layer, initial stacking layer, middle stacking layer, final stacking layer, network output layer; Among them, the network input layer is used to read the EIT image sequence; The initial stacking layer is formed by stacking a 3D residual initial unit, an intermediate 3D residual standard unit, and a final 3D residual standard unit in sequence; the 3D residual initial unit is used to preliminarily extract high-dimensional features of the image, the intermediate 3D residual standard unit is used to reduce the dimension of the high-dimensional image features, and the final 3D residual standard unit is used to reduce the dimension and deepen the feature information; Among them, the intermediate stacking layer includes: n sequentially connected monomer intermediate stacking layers; the monomer intermediate stacking layer is stacked by HAM units, 3D residual lower residual units, and final 3D residual standard units; n is a natural number greater than or equal to 1; the HAM unit optimizes the weight of the allocated information, removes redundant information, and strengthens the feature information; the 3D residual lower residual unit increases the dimension of the low-dimensional image features; the final 3D residual standard unit is used to reduce the dimension and deepen the feature information; Among them, the final stacking layer is formed by stacking a 3D global average pooling layer, a BiLSTM layer, and a fully connected layer in sequence; the 3D global average pooling layer reads the image sequence information obtained by the middle stacking layer and extracts its feature vector; the BiLSTM layer processes and outputs the feature vector; the fully connected layer maps the input feature vector to the number of target categories and obtains the mapped score value, and the target categories are: healthy, blocked, and damaged.
2. A STRes3dRNN model according to claim 1, characterized in that: The 3D residual initial unit includes a sequentially connected 3D convolution layer, a batch normalization layer, an activation layer, and a 3D maximum pooling layer; the three-dimensional convolution layer is used to convolutionally extract image sequence features; the batch normalization layer is used to normalize the sequence features to accelerate the convergence speed; the activation layer processes the data of the previous step through a nonlinear activation function to improve the representation ability of the features; the 3D maximum pooling layer uses different channels to compress the features to achieve invariance, extract and output the maximum value.
3. A STRes3dRNN model according to claim 2, characterized in that: The middle 3D residual standard unit and the final 3D residual standard unit both include a 3D convolution layer, a batch normalization layer, an activation layer, a 3D convolution layer, a batch normalization layer, an additional layer, and an activation layer connected in sequence; the 3D convolution layer is used for convolution to extract image sequence features; The batch normalization layer is used to normalize the sequence features and speed up the convergence; the activation layer processes the data given by the batch normalization layer through a nonlinear activation function to improve the representation ability of the features; the additional layer is used to randomly close certain nodes to prevent overfitting and avoid network degradation.
4. A STRes3dRNN model according to claim 3, characterized in that: The HAM unit includes a spatiotemporal attention module, a channel attention module and a motion attention module; the spatiotemporal attention module, the channel attention module and the motion attention module are applied to enhance the feature information in terms of spacetime, channel and motion respectively, and the enhanced information is weighted to obtain the fusion enhanced data of the spatiotemporal, channel and motion information with the new number of layers and dimensions remaining unchanged.
5. A STRes3dRNN model according to claim 4, characterized in that: The 3D residual downsampling unit includes a 3D convolution layer, a batch normalization layer, an activation layer, a 3D convolution layer, a batch normalization layer, an additional layer, and an activation layer connected in sequence; the 3D convolution layer is used for convolution to extract image sequence features; The batch normalization layer normalizes the sequence features to speed up the convergence; the activation layer processes the data of the previous step through a nonlinear activation function to improve the representation ability of the features; the additional layer is used to randomly close certain nodes to prevent overfitting and avoid network degradation.
6. A STRes3dRNN model according to claim 5, characterized in that: The BiLSTM layer consists of two channels, forward propagation and backward propagation, and performs forward and backward processing simultaneously. The processing time step is determined by the input image sequence, and the time step equal to the number of sequences is automatically cycled. The forward and backward processing results are spliced to obtain a high-dimensional feature vector.
7. An evaluation system, characterized in that: include: A storage unit, which is used to store continuous voltage data measured by the EIT device; An EIT image sequence generating unit, which reads the continuous voltage data of the storage unit to generate an EIT image sequence; The STRes3dRNN model according to any one of claims 1 to 6, which reads the EIT image sequence of the EIT image sequence generation unit, and the result output is transmitted to the probability calculation unit; Probability calculation unit, which is used to give the probability percentage of health, blockage, and damage.
8. An evaluation system according to claim 7, characterized in that: The working method of the probability calculation unit is: Softmax(z i ) represents the predicted probability of the i-th category, represents the raw score of the ith category.
9. An evaluation method, which is not aimed at disease diagnosis directly, characterized in that: The steps include: S100, acquiring continuous voltage data to generate an EIT image sequence; the EIT network input is a 16-frame 64×64×3 EIT image sequence, where 64×64 is the image size and 3 is a three-channel RGB image; S200, transmitting the image sequence I to an initial stacking layer; the initial stacking layer is formed by stacking a 3D residual initial unit, an intermediate 3D residual standard unit, and a final 3D residual standard unit; S200 includes the following sub-steps: S201, the image sequence I is input into the 3D residual initialization unit to obtain a high-dimensional feature of a 64-layer 7×7×7 image sequence; S202, the high-dimensional features obtained in step S201 are input into the intermediate 3D residual standard unit to obtain image features after dimensionality reduction; S203, the image features obtained in step S202 are input into the last 3D residual standard unit to obtain an image sequence after dimensionality reduction and feature information deepening; S300, inputting the result calculated by the initial stacking layer to the intermediate stacking layer; the intermediate stacking layer comprises: 3 sequentially connected monomer intermediate stacking layers; the monomer intermediate stacking layer is formed by stacking a HAM unit, a 3D residual lower residual unit, and a 3D residual standard unit: S300 includes the following sub-steps: S301, the HAM unit in the middle stacking layer of the first monomer reads a 128-layer 1×1×1 image sequence, optimizes the weight of the allocated information, removes redundant information, strengthens the feature information, and outputs a 128-layer 1×1×1 image sequence; S302, the 3D residual downsampling unit of the first monomer intermediate stacking layer reads a 128-layer 1×1×1 image sequence and outputs a 128-layer 3×3×3 image sequence; S303, the 3D residual standard unit at the end of the first monomer intermediate stacking layer reads a 128-layer 3×3×3 image sequence and outputs a 256-layer 1×1×1 image sequence; S304, the HAM unit of the second monomer intermediate stacking layer reads an image sequence of 256 layers of 1×1×1 image sequences, removes redundant information, strengthens feature information, and outputs an image sequence of 256 layers of 1×1×1 image sequences; S305, the 3D residual downsampling unit of the second monomer intermediate stacking layer reads a 256-layer 1×1×1 image sequence and outputs a 256-layer 3×3×3 image sequence; S306 , the 3D residual standard unit at the end of the second monomer intermediate stacking layer reads an image sequence of 256 layers of 3×3×3 image sequences and outputs an image sequence of 512 layers of 1×1×1 image sequences; S307, the HAM unit in the middle stack layer of the third monomer reads an image sequence of 512 layers of 1×1×1 image sequences, removes redundant information, strengthens feature information, and outputs an image sequence of 512 layers of 1×1×1 image sequences; S308, the 3D residual downsampling unit of the third monomer intermediate stacking layer reads a 512-layer 1×1×1 image sequence and outputs a 512-layer 3×3×3 image sequence; S309 , the 3D residual standard unit at the end of the third monomer intermediate stacking layer reads an image sequence of 512 layers of 3×3×3 image sequences and outputs a 512 layer 1×1×1 image sequence; S400, the result calculated by the middle stacking layer is input to the last stacking layer, and the last stacking layer includes a three-dimensional average layer, a BiLSTM layer, and a fully connected layer connected in sequence; step S400 also includes the following sub-steps: S401, a three-dimensional averaging layer reduces the dimension of the 512-layer 1×1×1 image sequence and extracts its feature vector; S402, the BiLSTM layer processes and outputs 128 layers of feature vectors; S403, the fully connected layer maps the input feature vector to the target classification and obtains the mapped score value; wherein the number of target classifications is 3, which represent health, obstruction, and damage respectively.
10. An evaluation method according to claim 9, characterized in that: Also included: S500, the result of the final stacked layer is input into the Softmax function, and the predicted probability is output, that is, the probability percentage of health, obstruction, and damage: Among them, Softmax(z i ) represents the predicted probability of the i-th category, represents the raw score of the ith category.
Citation Information
Patent Citations
Behavior recognition device and method based on multi-attention mechanism fusion and storage medium
CN112990116A
Method for classifying pancreatic images based on mixed attention network
CN115760797A
CXR image classification method and system based on residual convolution and multi-head self-attention
CN115995015A
Medical image classification method based on improved attention mechanism
CN117409232A
Unmanned aerial vehicle identification method based on blind source separation and deep learning
CN117668641A
