Automatic prognosis system for nasopharyngeal carcinoma primary site and lymph node nuclei magnetic resonance image

Abstract

The application discloses an automatic prognosis system for nasopharyngeal carcinoma primary focus and lymph node nuclear magnetic resonance image, which comprises a data preparation module, a segmentation module, a feature sharing module, a data processing module and a prognosis module; the data preparation module puts all nuclear magnetic resonance images of a patient into a folder in a patient unit; the segmentation module is used for processing data obtained by the data preparation module; the feature sharing module uses an expansion convolution group to reduce dimensions of each layer multi-scale deep features obtained by the segmentation module but maintains the scale unchanged; the data processing module uses a prediction map generated by the segmentation module and the nuclear magnetic resonance image obtained by the data preparation module to perform a mask strategy; the prognosis module is used for processing a lesion map obtained by the data processing module and the multi-scale deep features processed by the feature sharing module to obtain a prediction value, finally uses the prediction value to obtain a survival probability of the patient, and realizes risk stratification of the patient and gives a suggestion on whether to accept chemotherapy; the application can automatically realize prognosis prediction of the nasopharyngeal carcinoma patient, realizes automation in the whole process and avoids manual participation.

Description

Technical Field

[0001] The present invention relates to the technical field of prognosis prediction of nasopharyngeal carcinoma primary lesions and lymph nodes, and in particular to an automatic prognosis system for nasopharyngeal carcinoma primary lesions and lymph nodes based on nuclear magnetic resonance imaging. Background Art

[0002] Nasopharyngeal carcinoma prognosis prediction is a doctor's assessment of a patient's condition. It involves predicting and evaluating the disease or its future course, and developing a tailored treatment plan. With the advancement of image processing technology, artificial intelligence (AI) is now capable of predicting nasopharyngeal carcinoma prognosis. However, if only MRI images are used for prognosis prediction, a large number of non-lesion features will interfere with the model, reducing predictive effectiveness.

[0003] Nasopharyngeal carcinoma prognosis prediction currently faces two challenges: First, existing prognostic methods focus solely on the primary lesion. In clinical practice, lymph nodes are often present in nasopharyngeal carcinoma, which plays a crucial role in determining patient survival. Second, lesions are manually delineated in images, and omics or deep features are extracted from these lesions and fed into a network for prediction. Therefore, these nasopharyngeal carcinoma prognosis predictions are not automated and still require human intervention. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and propose an automatic prognosis system for nasopharyngeal carcinoma primary lesions and lymph node magnetic resonance imaging. It can automatically segment the nasopharyngeal carcinoma primary lesions and metastatic lymph nodes in magnetic resonance imaging, extract deep features, and realize automatic prognosis prediction.

[0005] To achieve the above objectives, the present invention provides a technical solution: an automatic prognostic system for nasopharyngeal carcinoma primary lesions and lymph node magnetic resonance imaging, comprising:

[0006] The data preparation module uses 2D MRI images, patient survival time, and event occurrence as data. All MRI images of a patient are placed in a folder so that all images can be processed simultaneously during training.

[0007] The segmentation module processes the data obtained by the data preparation module. This module, comprised of an improved U-Net segmentation network, automatically delineates NPC primary lesions and lymph nodes in MRI images. The improved U-Net segmentation network removes abnormal pixels from the boundaries of the feature map with the smallest scale and the largest number of channels after feature extraction. It also uses boundary control blocks to reduce feature propagation between pixels of different types, thereby enhancing feature propagation between pixels of the same type, enabling the network to better learn multiple targets.

[0008] The feature sharing module is composed of a dilated convolution group. Its input is the multi-scale deep features of each layer obtained by the segmentation module. The dilated convolution group is used to reduce the dimensionality of the multi-scale deep features while maintaining the scale unchanged.

[0009] The data processing module uses the prediction map generated by the segmentation module and the MRI image obtained by the data preparation module to perform a masking strategy, specifically removing non-lesions in the MRI image and maintaining the lesion display in the MRI image to obtain a lesion map;

[0010] The prognosis module is used to process the lesion map obtained by the data processing module and the multi-scale deep features processed by the feature sharing module. The prognosis module is constructed by the U-Net encoder, Transformer and fully connected layer. It relies on the U-Net encoder to downsample the lesion map and connect it with the feature map of each layer. It is sent to the Transformer in the last layer to build a global context, and then sent to the fully connected layer for processing to obtain the predicted value. Finally, the predicted value is used to obtain the patient's survival probability, realize patient risk stratification, and give recommendations on whether to receive chemotherapy.

[0011] Furthermore, the data preparation module specifically performs the following operations:

[0012] The 2D MRI images were preprocessed, including unifying the size and pixel value range. Each patient had multiple MRI images of varying sizes, and the patient's images were stored in the order of MRI scans. For patient survival time and whether an event occurred, each patient had only one data point, which was stored separately.

[0013] Furthermore, the segmentation module processes all MRI images of a patient together and inputs them into the segmentation module at one time. The backbone of the segmentation module is a U-Net network with a five-layer symmetrical structure, divided into an encoder and a decoder. Except for the last layer, which only has convolution, the remaining four layers of the encoder consist of convolution and pooling. With each layer of processing, the number of image channels doubles and the image size decreases by half. Except for the first layer, which only has convolution, each layer of the decoder consists of convolution and upsampling. With each layer of processing, the number of image channels decreases by half and the image size increases by half. The encoder and decoder are connected by skip connections at the same number of layers. The features processed by the last layer of the encoder are sent to the boundary control block, which obtains edge pixels from the boundary, calculates the proportion of the lesion area within the surrounding 10 pixels, and filters pixels with a proportion higher than a preset value to maintain a smooth boundary. The boundary information is used to control the information propagation between different lesions during feature propagation. Strong connections are established between pixels within the same lesion to strengthen information connection and transmit more information. Conversely, weak connections are established between pixels of different lesions to reduce information flow. Finally, convolution upsampling is used to restore the image size and generate a prediction map.

[0014] Furthermore, the feature sharing module takes the multi-scale deep features obtained from each encoder layer of the segmentation module as input, achieves feature size consistency and dimensionality reduction through five groups of dilated convolution groups, and then inputs the multi-scale deep features into the relevant layers of the survival module. The two groups are connected using residual connections to prevent the gradient vanishing problem.

[0015] Furthermore, the masking strategy of the data processing module is to set the pixel value of the lesion area in the prediction image of the segmentation module to 0, and then perform Hadamard multiplication on the prediction image and the magnetic resonance image, so that the pixel value of the non-lesion area in the magnetic resonance image also becomes 0, thereby masking the non-lesion area and obtaining the lesion map.

[0016] Furthermore, the prognosis module uses the encoder part of the U-Net network to extract features from the lesion map and connect the multi-scale deep features from the feature sharing module. In the last layer, the features are sent to the Transformer to construct the global context connection of the entire patient lesion features. After construction, the features are flattened into a one-dimensional feature vector and padded to the baseline. The baseline is set according to the maximum number of magnetic resonance images of a single patient in the data preparation module. After the one-dimensional feature vector is reduced in dimension and extracted in the fully connected layer, the patient's survival probability is obtained.

[0017] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0018] 1. The present invention can realize automatic prognosis prediction from 2D MRI images without the need for human intervention.

[0019] 2. The present invention automatically segments the primary lesion and lymph node area of ​​nasopharyngeal carcinoma in magnetic resonance imaging to obtain lesion characteristics.

[0020] 3. The present invention uses the model to segment the primary nasopharyngeal carcinoma lesions and lymph nodes to perform masking in the original magnetic resonance imaging to remove non-lesion areas.

[0021] 4. The feature sharing module designed in the segmentation and prognosis parts of the present invention reduces the dimensionality of the multi-scale deep features in the segmentation and shares them with the prognosis prediction, thereby improving the performance of nasopharyngeal carcinoma primary lesion and lymph node segmentation and patient prognosis prediction.

[0022] 5. The present invention is designed based on the characteristics of nasopharyngeal carcinoma primary lesions and lymph nodes and task requirements to achieve accurate nasopharyngeal carcinoma primary lesions and lymph node segmentation and patient prognosis prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a logical overall flow chart of the system of the present invention.

[0024] Figure 2This is a flow chart of segmentation of the present invention.

[0025] Figure 3 The figure is a prognosis flow chart of the present invention. DETAILED DESCRIPTION

[0026] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0027] like Figures 1 to 3 As shown, this embodiment discloses an automatic prognostic system for nasopharyngeal carcinoma primary lesions and lymph node magnetic resonance imaging, including:

[0028] The data preparation module uses 2D MRI images, patient survival time, and event occurrence as data. All MRI images of a patient are placed in a folder so that all images can be processed simultaneously during training.

[0029] The segmentation module processes the data obtained by the data preparation module. This module, comprised of an improved U-Net segmentation network, automatically delineates NPC primary lesions and lymph nodes in MRI images. The improved U-Net segmentation network removes abnormal pixels from the boundaries of the feature map with the smallest scale and the largest number of channels after feature extraction. It also uses boundary control blocks to reduce feature propagation between pixels of different types, thereby enhancing feature propagation between pixels of the same type, enabling the network to better learn multiple targets.

[0030] The feature sharing module is composed of a dilated convolution group. Its input is the feature maps of each layer obtained by the segmentation module. The dilated convolution group is used to reduce the dimensionality of multi-scale deep features while maintaining the scale unchanged.

[0031] The data processing module uses the prediction map generated by the segmentation module and the MRI image obtained by the data preparation module to perform a masking strategy, specifically removing non-lesions in the MRI image and maintaining the lesion display in the MRI image to obtain a lesion map;

[0032] The prognosis module is used to process the lesion map obtained by the data processing module and the multi-scale deep features processed by the feature sharing module. The prognosis module is constructed by the U-Net encoder, Transformer and fully connected layer. It relies on the U-Net encoder to downsample the lesion map and connect it with the feature map of each layer. It is sent to the Transformer in the last layer to build a global context, and then sent to the fully connected layer for processing to obtain the predicted value. Finally, the predicted value is used to obtain the patient's survival probability, realize patient risk stratification, and give recommendations on whether to receive chemotherapy.

[0033] Specifically, the data preparation module performs the following operations:

[0034] 2D MRI images were preprocessed, including standardization of size and pixel value range. Each patient had multiple MRI images of varying sizes, and the images were stored in the order of their MRI scans. Each patient had a single data point for survival time and event occurrence, stored separately. Images were sorted by MRI scan site, from the oropharynx to the suprasellar cistern, numbered starting at 1. All images were placed in a folder. Multiple images from each modality were then concatenated using the same standards to form a single slice.

[0035] Specifically, the segmentation module consists of a segmentation backbone, U-Net, which is a general segmentation backbone with multiple upsampling and downsampling operations for feature engineering, and a boundary control block. U-Net is a general segmentation backbone with multiple upsampling and downsampling operations for feature engineering. Both the upsampling and downsampling operations consist of five convolutional groups, each of which contains two 3×3 convolutions with a stride and padding of 1, two batch normalization layers, and two ReLU activation functions. The convolutional groups at different levels are connected by a 2×2 max pooling layer with a stride of 2. The downsampling convolutional group is connected to the upsampling convolutional group via a skip connection. In the last convolutional layer, a boundary control block is designed to smooth the boundaries by filtering outliers and reduce misidentification by controlling feature propagation. Finally, after the final upsampling operation, a three-channel probability map corresponding to the MRI image is obtained.

[0036] The boundary control block consists of two processes: filtering abnormal boundary pixels and feature propagation control. Get edge pixels from the boundary and calculate the ratio of the lesion area within 10 pixels around it. Too large a ratio indicates that the boundary is not smooth and prone to errors, so pixels with too large a ratio are removed to keep the boundary smooth. Controlling feature propagation improves the similarity of lesion features while maintaining the differences in features of different lesions. Use boundary information to control the information propagation of different lesions in feature propagation. Establish strong connections between pixels in the same lesion, strengthen information connection, and transmit more information. On the contrary, establish weak connections between pixels in different lesions, reducing the flow of information. For the feature i of the t-th pixel t and its output y t , its propagation process can be expressed as:

[0037] y t =f(C*i t +K*y t-1 m t-1 +δ)

[0038] Where * represents the convolution operation, f represents the ReLU activation function, C and K are learnable parameters, δ is the learnable bias, and m t-1 is the propagation control of the previous pixel, y t-1 Represents the output of the previous pixel. The propagation control m of the t-th pixelt The generation of can be expressed as:

[0039] m t =1-αg(βb t -γ)

[0040] Where α is a learnable parameter, β and γ are constants, and b t is the probability that the tth pixel belongs to the boundary, and g represents the sigmoid activation function. According to the probability of the boundary, different types of connections are established between pixels to control feature propagation.

[0041] Specifically, the feature sharing module uses the multi-scale deep features of each layer of the segmentation module as input. Feature size consistency and dimensionality reduction are achieved through dilated convolution groups. Specifically, each dilated convolution group contains five convolution layers, including two 1×1 convolution operations, two 3×3 convolution operations, and one 5×5 convolution operation. There are two dilated convolution groups, with dilations of 3 and 1, respectively. In addition, the two groups are connected using residual connections to prevent the gradient vanishing problem. After calculating the two dilated convolution groups, the final multi-scale deep features are input to the prognosis module.

[0042] Specifically, the data processing module's masking strategy removes non-lesion areas from the MRI image while maintaining the lesion image. First, each pixel is classified using the argmax function. Then, a lesion map is generated by taking the Hadamard product of the predicted image and the original MRI image. Thus, in subsequent calculations, non-lesion areas are masked, and only lesion features are considered.

[0043] Specifically, the prognosis module is built by the U-Net encoder, Transformer and fully connected layer. Specifically, the U-Net encoder is used to downsample the lesion map, connect it with the feature map of each layer, send it to the Transformer in the last layer, build the global context, and then send it to the fully connected layer for processing to obtain the predicted value: the prognosis network consists of a U-Net encoder, a Transformer, three linear layers and a sigmoid activation function. The U-Net encoder structure is the same as the segmentation network encoder network. The lesion map is encoded into a feature map, but at each layer it is connected to the multi-scale deep features output by the feature sharing module on the channel. Here, the feature map generated by the U-Net encoder is input into the Transformer. The Transformer consists of multi-head attention, a multi-layer perceptron and layer normalization. Each feature vector in the feature map is regarded as a patch. Residual connections are widely used to stabilize network training and retain prior knowledge. The input patches are divided into two groups: one group enters the multi-head self-attention to capture the global context, and the other group skips the multi-head self-attention through the residual connection. The output Z of this layer of the Transformer i With the output Z of the previous layer i-1 It can be expressed as:

[0044]

[0045]

[0046] Where MSA stands for multi-head self-attention, MLP stands for multi-layer perceptron, and LN stands for layer normalization. represents element-by-element addition, i represents the Transformer of the i-th layer, Z′ i This is the result of element-wise addition of the original input features in a residual structure using multi-head self-attention and layer normalization. Following the Transformer computation, a 3×3 convolutional layer performs dimensionality reduction and flattens the features into a one-dimensional feature vector. Because each patient has a different number of MRI images, the image of the patient with the most MRI images in the dataset is used as the baseline. The one-dimensional feature vector is padded with zeros to ensure that all feature vectors reach the baseline. Finally, a linear classification layer with a sigmoid function generates the patient's survival probability at different time points.

[0047] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. An automatic prognostic system for nasopharyngeal carcinoma primary lesions and lymph node MRI, characterized by: include: The data preparation module uses 2D MRI images, patient survival time, and event occurrence as data. All MRI images of a patient are placed in a folder so that all images can be processed simultaneously during training. The segmentation module processes the data obtained by the data preparation module. This module, comprised of an improved U-Net segmentation network, automatically delineates NPC primary lesions and lymph nodes in MRI images. The improved U-Net segmentation network removes abnormal pixels from the boundaries of the feature map with the smallest scale and the largest number of channels after feature extraction. It also uses boundary control blocks to reduce feature propagation between pixels of different types, thereby enhancing feature propagation between pixels of the same type, enabling the network to better learn multiple targets. The feature sharing module is composed of a dilated convolution group. Its input is the feature maps of each layer obtained by the segmentation module. The dilated convolution group is used to reduce the dimensionality of multi-scale deep features while maintaining the scale unchanged. The data processing module uses the prediction map generated by the segmentation module and the MRI image obtained by the data preparation module to perform a masking strategy, specifically removing non-lesions in the MRI image and maintaining the lesion display in the MRI image to obtain a lesion map; The prognosis module is used to process the lesion map obtained by the data processing module and the feature map processed by the feature sharing module. The prognosis module is constructed by the U-Net encoder, Transformer and fully connected layer. It relies on the U-Net encoder to downsample the lesion map and connect it with the feature map of each layer. It is sent to the Transformer in the last layer to build a global context, and then sent to the fully connected layer for processing to obtain the predicted value. Finally, the predicted value is used to obtain the patient's survival probability, realize patient risk stratification, and give recommendations on whether to receive chemotherapy.

2. The automatic prognostic system for nasopharyngeal carcinoma primary lesions and lymph node MRI according to claim 1, characterized in that: The data preparation module specifically performs the following operations: The 2D MRI images were preprocessed, including unifying the size and pixel value range. Each patient had multiple MRI images of varying sizes, and the patient's images were stored in the order of MRI scans. For patient survival time and whether an event occurred, each patient had only one data point, which was stored separately.

3. The automatic prognostic system for nasopharyngeal carcinoma primary lesions and lymph node MRI according to claim 2, characterized in that: The segmentation module processes all MRI images of a patient together and inputs them into the segmentation module at one time. The backbone of the segmentation module is a U-Net network with a five-layer symmetrical structure, divided into an encoder and a decoder. Except for the last layer, which only has convolution, the remaining four layers of the encoder consist of convolution and pooling. With each layer of processing, the number of image channels doubles and the image size decreases by half. Except for the first layer, which only has convolution, each layer of the decoder consists of convolution and upsampling. With each layer of processing, the number of image channels decreases by half and the image size increases by half. The encoder and decoder are connected by skip connections at the same number of layers. The features processed by the last layer of the encoder are sent to the boundary control block, which obtains edge pixels from the boundary, calculates the proportion of the lesion area within the surrounding 10 pixels, and filters pixels with a proportion higher than a preset value to maintain a smooth boundary. The boundary information is used to control the information propagation of different lesions during feature propagation. Strong connections are established between pixels within the same lesion to strengthen information connection and transmit more information. Conversely, weak connections are established between pixels of different lesions to reduce the flow of information. Finally, convolution upsampling is used to restore the image size and generate a prediction map.

4. The automatic prognostic system for nasopharyngeal carcinoma primary lesions and lymph node MRI according to claim 3, characterized in that: The feature sharing module takes the multi-scale deep features obtained from each encoder layer of the segmentation module as input, achieves feature size consistency and dimensionality reduction through five groups of dilated convolution groups, and then inputs the multi-scale deep features into the relevant layers of the survival module. The two groups are connected using residual connections to prevent the gradient vanishing problem.

5. The automatic prognostic system for nasopharyngeal carcinoma primary lesions and lymph node MRI according to claim 4, characterized in that: The masking strategy of the data processing module is to set the pixel value of the lesion area in the prediction image of the segmentation module to 0, and then perform Hadamard multiplication on the prediction image and the magnetic resonance image, so that the pixel value of the non-lesion area in the magnetic resonance image also becomes 0, thereby masking the non-lesion area and obtaining the lesion map.

6. The automatic prognostic system for nasopharyngeal carcinoma primary lesions and lymph node MRI according to claim 5, characterized in that: The prognosis module uses the encoder part of the U-Net network to extract features from the lesion map and connect the multi-scale deep features from the feature sharing module. In the last layer, the features are fed into the Transformer to construct a global contextual connection of the entire patient's lesion features. After construction, the features are flattened into a one-dimensional feature vector and padded to the baseline. The baseline is set according to the maximum number of magnetic resonance imaging images of a single patient in the data preparation module. After the one-dimensional feature vector is reduced in dimension and extracted in the fully connected layer, the patient's survival probability is obtained.

Images