Method for automatically reconstructing crisis organs of head and neck

Abstract

The invention provides a medical image segmentation and three-dimensional visualization method based on an improved U-Net network, and aims to improve the segmentation precision and the reconstruction effect. According to the method, a polarization self-attention (PSA) mechanism is introduced into an encoder module and a decoder module of the U-Net network, the capturing capacity of a model for global features is enhanced, and the training process of the model is optimized through an improved loss function. And dynamically adjusting network hyper-parameters, such as dropout rate, batch size and optimizer type, in combination with a genetic algorithm to realize automatic optimization of model performance. In the reasoning stage, the segmentation result is rendered by using VTK and ITK tools, a three-dimensional visual image is generated, and the space structure of the target area is visually displayed. According to the method, the representation capability of the segmentation network is enhanced, model parameters are automatically optimized, the accuracy and robustness of medical image segmentation are remarkably improved, and the method is suitable for medical scenes such as tumor boundary detection and organ recognition and has important clinical application value.

Description

Technical Field

[0001] The present invention relates to the application of artificial intelligence (AI), deep learning (DL), convolutional neural networks (CNN), self-attention mechanism (Self-Attention Mechanism), genetic algorithm (GA), medical image processing (Medical Image Processing) and three-dimensional visualization technology (3D visualization) in the direction of semantic segmentation. Background Art

[0002] Radiotherapy is one of the most commonly used and important methods in cancer treatment. It uses high-energy rays to accurately irradiate tumor tissue, destroy the DNA of cancer cells, inhibit their proliferation and eventually lead to the death of cancer cells. According to statistics, about 70% of patients need to receive radiotherapy during cancer treatment, and about 40% of cancers can be cured by radiotherapy. However, since the radiotherapy process involves ionizing radiation from high-energy rays, the risk of accidental injury to normal tissues around the target area has always been an important challenge in clinical practice. Especially in the head and neck region, critical organs (such as the optic nerve, brainstem and spinal cord) are often closely adjacent to the tumor. These organs are extremely sensitive to radiation, and even a small dose over-limit may cause serious or even irreversible functional damage. Therefore, in the radiotherapy plan, it is crucial to accurately segment and reconstruct the critical organs of the head and neck, and then formulate radiation paths that avoid these areas, which is crucial to the treatment effect and quality of life of patients.

[0003] Traditionally, the segmentation of organs at risk in medical images mainly relies on manual delineation, which is a time-consuming and highly subjective task. In addition, rule-based image processing methods are usually difficult to cope with complex anatomical structures and differences in image quality. Therefore, with the rapid development of deep learning technology, researchers have begun to try to apply convolutional neural networks (CNNs) and attention mechanisms to the field of medical image segmentation to improve the accuracy and efficiency of segmentation.

[0004] As a classic medical image segmentation method, U-Net network can effectively capture multi-scale features due to its symmetrical encoding-decoding structure and skip connection mechanism, and performs well in various segmentation tasks. However, the traditional U-Net network still has the following limitations when dealing with complex anatomical structures: 1. Insufficient ability to capture detailed features: Traditional convolution operations are more sensitive to the extraction of local features, but are prone to omissions or misclassifications when dealing with target areas with blurred boundaries or complex shapes. 2. Insufficient use of global information: Since the network design of U-Net is mainly based on local receptive fields, its ability to model global context is limited, which affects the segmentation effect when the target area is small or the background noise is high.

[0005] To overcome these problems, researchers have proposed a series of improvement strategies. Among them, the attention mechanism has become a research hotspot in recent years because it can dynamically focus on important areas in the input features. In particular, the self-attention mechanism can significantly enhance the network's ability to model global information by calculating the correlation between each position in the feature map. Based on this, we further introduce a polarized self-attention mechanism (PSA) and embed it into the encoder and decoder structure of U-Net to improve the ability to capture details and global information. In addition, in order to improve the automatic tuning capability of model parameters, we use a genetic algorithm (GA) to reduce the cost of manual parameter tuning and ensure the stability of network performance.

[0006] At the same time, the importance of 3D reconstruction technology in radiotherapy is becoming increasingly prominent. Compared with 2D image segmentation, 3D reconstruction can provide clinicians with a more intuitive display of anatomical structures, which is convenient for optimizing radiotherapy pathways. Modern 3D reconstruction technology, combined with visualization tools such as VTK and ITK, can efficiently realize the conversion from segmentation results to 3D models.

[0007] Based on the above technical background, the present invention proposes a method for automatic reconstruction of critical organs of the head and neck that combines deep learning and three-dimensional visualization technology. The medical images are segmented by an improved U-Net network, and the parameters are optimized by combining a genetic algorithm, ultimately achieving high-precision three-dimensional reconstruction of critical organs. Experiments show that this method is significantly superior to existing technologies in terms of segmentation accuracy and three-dimensional reconstruction quality, providing important support for the formulation of radiotherapy plans. Summary of the invention

[0008] The purpose of the present invention is to provide an efficient method for automatic reconstruction of critical organs of the head and neck, so as to solve the problems of insufficient precision and low computational efficiency in the field of medical image processing when accurately segmenting and three-dimensionally reconstructing complex anatomical structures of the head and neck. The present invention utilizes deep learning technology, especially for the segmentation task of ultrasound images, makes innovative improvements to the U-Net network, and combines visualization technology to achieve efficient and accurate reconstruction of critical organs of the head and neck.

[0009] In traditional deep learning models, especially U-Net networks, the improvement of feature extraction and target segmentation accuracy faces challenges. Especially when dealing with complex anatomical structures of the head and neck, the segmentation accuracy and efficiency cannot meet the needs of actual medical applications. Through the improvements of the present invention, these problems are solved and the automatic reconstruction effect of critical organs is significantly improved.

[0010] As a classic image segmentation architecture, the U-Net network is widely used in the field of medical image processing. However, due to the particularity of head and neck ultrasound images, such as blurred tissue boundaries and diverse organ morphologies, it is difficult for traditional U-Net networks to achieve ideal segmentation effects. To this end, the present invention makes the following three key improvements to the U-Net network: 1. Improving segmentation accuracy: By improving the U-Net network structure and loss function, the performance of the model is optimized when processing complex target areas, especially the accuracy in blurred boundaries and small area segmentation; 2. Enhancing feature expression capabilities: By introducing the Polarized Self-Attention mechanism, the model's feature expression and attention capabilities for key target areas are improved; 3. Automated parameter optimization: Introducing a genetic algorithm to optimize the network's hyperparameters, reduce manual parameter adjustment work, and improve the efficiency and accuracy of model training.

[0011] After completing the image segmentation, the present invention uses the VTK (Visualization Toolkit) and ITK (Insight Segmentation and Registration Toolkit) libraries to perform 3D reconstruction of the segmented 2D image data. Through volume rendering technology, a 3D visualization model of the critical organs of the head and neck can be generated, providing doctors with intuitive and comprehensive anatomical structure information, which is helpful for accurate diagnosis and surgical treatment.

[0012] In summary, the main contents of the present invention are as follows:

[0013] (1) This paper proposes an improved U-Net network, which achieves high-precision segmentation of critical organs in the head and neck by modifying the loss function, introducing the Polarized Self-Attention mechanism and genetic algorithm tuning.

[0014] (2) Combining VTK and ITK libraries and using volume rendering technology, the segmented image data is converted into a three-dimensional visualization model, providing doctors with a more intuitive and accurate basis for diagnosis.

[0015] (3) This method improves segmentation accuracy and 3D reconstruction efficiency while maintaining computational feasibility and real-time performance, providing an efficient and reliable solution for the automatic reconstruction of critical organs in the head and neck. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The overall flow chart of the present invention is shown

[0017] Figure 2 The improved U-Net network architecture diagram used in this invention is shown

[0018] Figure 3 Flowchart showing the optimization process based on genetic algorithm DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific examples.

[0020] like Figure 1 As shown in the figure, the whole process of automatic reconstruction of head and neck crisis organs mainly includes three stages. First, we use a series of processing methods to convert the head and neck DICOM format data set into input that the network can process; second, we initialize multiple networks through genetic algorithms for model training; finally, we send the inference data into the model for prediction and combine it with VTK / ITK for 3D reconstruction.

[0021] First, data preprocessing.

[0022] First, convert the DICOM format file into an image file in PNG format to meet the needs of subsequent processing steps. Crop the image and its corresponding labeled binary image to a uniform size of 416×416. Then, divide the data set into a training set and a test set in a random arrangement at a ratio of 4:1 to ensure the randomness and rationality of the data division. At the same time, in view of the scarcity of medical data, the present invention further implements a data enhancement strategy. For each image in the training set, a variety of data enhancement methods such as random flipping, random rotation, and brightness adjustment are used to effectively alleviate the overfitting phenomenon that may occur in the deep learning model during the training process by increasing the variability and complexity of the data, thereby improving the prediction accuracy and robustness of the model.

[0023] Second, network structure.

[0024] The core of the network is a semantic segmentation network based on the U-Net framework, whose main modules include encoder, decoder and skip connection. Genetic algorithm is used to automatically adjust the hyperparameter combination (such as dropout_rate, batch_size and optimizer), and the Dice coefficient is used as the evaluation index in the optimization process.

[0025] (1) PSA mechanism: Let the input feature be F∈R C×H×W , the global and local modeling formulas of PSA are:

[0026] Global Modeling Branch:

[0027] W g =σ(W l GAP(F)+b1)

[0028] Among them, W l ∈R C×C is the learnable weight matrix, GAP(F)∈R c represents global average pooling, σ(·) is the Sigmoid activation function

[0029] Weight W g Acting on global features:

[0030] F g =F⊙W g

[0031] where ⊙ represents element-wise multiplication

[0032] Local modeling branch:

[0033] W l =σ(Conv(F))

[0034] Where Conv(·) is the local convolution operation and σ(·) is the activation function

[0035] Weight W l Acting on local features:

[0036] F l =F⊙W l

[0037] Global and local feature fusion:

[0038] F PSA =F g +F l

[0039] (2) Encoder: The PSA mechanism is introduced in each block of the encoder to enhance the network’s ability to adaptively model image features. Each block contains two convolutional layers, a batch normalization layer (BatchNormalization), a ReLU activation function, a PSA mechanism, and a Dropout layer.

[0040] Set the input features The output after the first layer encoder is:

[0041]

[0042] Among them, Conv is the convolution operation, BN is batch normalization, ReLU(·) is the activation function, PSA(·) is the Polarized Self-Attention operation, Represents the output features of the first layer encoder

[0043] The downsampling operation is obtained through maximum pooling:

[0044]

[0045] (2) Decoder: In the decoder, the PSA mechanism is also introduced in each block.

[0046] Assume that the upsampled input of the first layer decoder is The formula is:

[0047]

[0048] Where UpSample(·) is the upsampling operation.

[0049] Skip connection feature with encoder After splicing:

[0050]

[0051] PSA models the concatenated features:

[0052]

[0053] (3) Loss function: Drawing on the experience of target detection tasks, we use the improved Focal loss as the loss function.

[0054]

[0055] The α and γ parameters are introduced. The value range of the α parameter is 0-1. When the α value is greater than 0.5, the loss function will pay more attention to positive samples and reduce the attention to negative samples accordingly, thereby effectively balancing the weights between positive and negative samples.

[0056] The value range of the γ parameter is greater than 1. The purpose is to make the network get more training for samples that are difficult to classify, and reduce the training of samples that are trained accordingly. For example, for positive samples, y′ is close to 1, which means that the easier the sample is to classify, the lower the loss will be; on the contrary, if it is a difficult sample to classify, the loss will increase, and the same is true for negative samples. In the actual training process, Focal loss ensures the stability of training while effectively solving the problem of uneven categories.

[0057] (4) Learning rate: To avoid failure to converge due to excessively high learning rate in the later stages of training, the learning rate is set to decay with the training cycle. The decay formula is as follows:

[0058]

[0059] (5) Hyperparameter optimization of genetic algorithms

[0060] Initialize the population, each individual is a hyperparameter combination:

[0061] h={batch_size, dropout_rate, optimizer}

[0062] Fitness function, calculate the Dice coefficient as fitness:

[0063]

[0064] Among them, p is the inference result and g is the label value

[0065] Crossover operation, randomly select two parent individuals h A and h B , crossover to generate offspring individuals:

[0066] h child ={dropout_rate A , batch_size B , optimizer A}

[0067] Mutation operation randomly adjusts a hyperparameter of the offspring individuals, such as:

[0068] dropout_rate→dropout_rate+δ, δ~(-0.1,0.1)

[0069] Update the population, combine the new individuals generated by crossover and mutation with the parent individuals with the highest fitness to form the next generation population.

[0070] After iteration, the optimal hyperparameter combination is found as:

[0071]

[0072] The final network Dice coefficient reaches:

[0073] Dice optimal =f(h * )

[0074] Third, three-dimensional reconstruction.

[0075] Based on medical image segmentation, the segmentation reasoning results are combined with visualization tools (VTK and ITK) to achieve three-dimensional display of the target area.

[0076] (1) Reading data

[0077] The segmentation results are read as a three-dimensional array SegmentArray through ITK

[0078] SegmentArray=ITK.ReadImage(SegmentFile)

[0079] (2) Format Conversion

[0080] Use ITK to convert volume data to VTK format

[0081] VTKData=ITK.ImageToVTKImageFilter(SegmentArray)

[0082] (3) 3D reconstruction using VTK

[0083] Volume rendering is performed through VTK's vtkVolume and vtkVolumeMapper components:

[0084] Mapping data to renderer: VolumeMapper←SetInputData(VTKData)

[0085] Renderer configuration: Volume←SetMapper(VolumeMapper)

[0086] Transfer function binding: Volume←SetProperty(ColorTransfer, OpacityTransfer)

[0087] Add to the renderer: Render.AddVolume(Volume).

Claims

1. A method for automatic reconstruction of head and neck crisis organs, characterized in that: The following steps are involved: Step 1, converting the DICOM format data set into a PNG format image file by window width and window position processing; Step 2, cropping the image and its corresponding label binary image; Step 3: Divide the data set, with most of the images used as network training sets and a small number of images used as network validation sets; Step 4: For the images in the training set, various data enhancement methods such as random flipping, random rotation, and brightness adjustment are used; Step 5, initializing the network hyperparameter combination according to the genetic algorithm; Step 6: Add the PSA (Polarized Self-Attention) mechanism to the encoder and decoder of U-Net; Step 7: Modify the loss function of U-Net and train and fine-tune the entire network; Step 8: Perform crossover, mutation and iteration based on fitness - Dice coefficient; Step 9: Complete model training and combine the inference results with VTK and ITK for 3D reconstruction.

2. A method for automatic reconstruction of head and neck crisis organs according to claim 1, characterized in that: In step 1, the DICOM format data obtained from the hospital is converted into PNG format images according to the window width and window position.

3. The method for automatic reconstruction of head and neck crisis organs according to claim 1, characterized in that: In step 2, the image and its corresponding label image are cropped to a uniform size of 416×416.

4. The method for automatic reconstruction of head and neck crisis organs according to claim 1, characterized in that: In step 3, the data set is divided to keep the ratio of training set to test set at about 4:

1.

5. The method for automatic reconstruction of head and neck crisis organs according to claim 1, characterized in that: After dividing the data set in step 4, various data enhancement methods such as random flipping, random rotation, and brightness adjustment are used to alleviate the overfitting phenomenon that may occur in the deep learning model during training by increasing the variability and complexity of the data, and all images are normalized before being input into the network.

6. The method for automatic reconstruction of head and neck crisis organs according to claim 1, characterized in that: In step 5, initialize multiple U-Net network hyperparameter combinations, including batch_size, optimizer and dropout_rate, where batch_size is selected from 8, 16, 32, 64; optimizer: {0: SGD, 1: Adam, 2: RMSprop}; dropout_rate: selected from 0.1-0.

7.

7. The method for automatic reconstruction of head and neck crisis organs according to claim 1, characterized in that: In step 6, PSA modules are added to both the Encoder and Decoder modules of U-Net. This attention mechanism does not perform a large degree of compression in both the spatial and channel dimensions, resulting in relatively small information loss and improved ability to capture details and global information. The specific expression is: Among them, Conv is the convolution operation, BN is batch normalization, ReLU(·) is the activation function, PSA(·) is the PolarizedSelf-Attention operation, represents the output features of the first layer encoder, represents the output features of the skip connection between the first layer encoder and decoder, is the output feature of the first layer decoder.

8. The method for automatic reconstruction of head and neck crisis organs according to claim 1, characterized in that: The U-Net loss function in step 7 is: The α parameter ranges from 0 to 1. When the α value is greater than 0.5, the loss function will pay more attention to positive samples and reduce the attention to negative samples accordingly, thereby effectively balancing the weights between positive and negative samples. The value range of the Y parameter is greater than 1. The purpose is to allow the network to get more training for samples that are difficult to classify, and reduce training for trained samples accordingly. For example, for positive samples, y′ is close to 1, which means that the easier the sample is to classify, the lower the loss will be; on the contrary, if it is a difficult sample to classify, the loss will increase. The same is true for negative samples. In the actual training process, Focalloss ensures training stability while effectively solving the problem of uneven categories.

9. The method for automatic reconstruction of head and neck crisis organs according to claim 1, characterized in that: In each round of testing in step 8, according to the Dice coefficient, the specific formula is as follows: Select individuals with high fitness, obtain new offspring through crossover and mutation, and repeat the above steps until the optimal hyperparameter combination is found or the maximum number of iterations is reached.

10. The method for automatic reconstruction of head and neck crisis organs according to claim 1, characterized in that: In step 9, the inference results are combined with VTK and ITK to complete the 3D reconstruction of the critical organs of the head and neck.

Images