A method for identifying osteosarcoma based on RTUNet network

Abstract

The present invention discloses a method for identifying osteosarcoma based on an RTUNet network, which belongs to the field of computer vision deep learning and medical image processing. The method specifically includes: step a, constructing an RTUNet neural network model for identifying osteosarcoma; step b, establishing an osteosarcoma data set, which is divided into a training set and a test set; step c, using the training set to train the RTUNet neural network model and tune the network model parameters; step d, using the test set to test and evaluate the obtained network model, and finally realizing automatic and intelligent osteosarcoma tumor recognition and segmentation functions. The network model RTUNet used in the present invention combines the residual network ResNet, Transformer, and U-Net jump connection structure, which can realize automatic and intelligent osteosarcoma tumor recognition and segmentation functions, and make the network model have higher recognition and segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision deep learning and medical image processing, and in particular to an osteosarcoma recognition method based on an RTUNet network. Background Art

[0002] Osteosarcoma is the most common bone-derived malignant tumor in orthopedics, with extremely high mortality and disability rates. It is characterized by high malignancy, strong invasiveness, easy recurrence and easy metastasis. It is more common in adolescents aged 10 to 25 years old, and the pathological mechanism of osteosarcoma has not been fully studied in the medical field.

[0003] The clinical diagnosis of osteosarcoma is mainly based on three aspects: clinical, imaging and histopathology. In clinical practice, the three methods are generally combined to evaluate cases. Among them, magnetic resonance imaging in imaging technology is an important method for determining preoperative treatment of osteosarcoma. However, due to: 1) osteosarcoma accounts for a relatively low proportion of tumors overall, and most of them are sporadic cases, experts outside of orthopedics rarely pay attention to it; 2) even for experienced radiologists, manually depicting tumor tissue from each imaging image is time-consuming and laborious, and the results are subjective and non-reproducible. Therefore, the medical community needs an accurate automatic or semi-automatic osteosarcoma segmentation method, and the segmentation accuracy of the network model used in the existing technology for osteosarcoma segmentation with the help of deep learning is not accurate enough. Summary of the Invention

[0004] The main purpose of the present invention is to provide an osteosarcoma identification method based on the RTUNet network, which can effectively solve the problems mentioned in the background technology.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for identifying osteosarcoma based on an RTUNet network comprises the following steps:

[0007] Step a: Construct an RTUNet neural network model for identifying osteosarcoma. The RTUNet neural network model combines the residual network ResNet, Transformer, and U-Net jump connection structure, and consists of an encoder and a decoder.

[0008] Among them, the input image is first used to extract feature information through the encoder, and then the low-level spatial feature information is input into the decoder layer through the jump connection structure to assist in completing the upsampling work and obtain a segmentation result consistent with the image input size.

[0009] Step b: Establish an osteosarcoma dataset, which is divided into a training set and a test set.

[0010] Step c: Use the training set to train the RTUNet neural network model, and combine the imaging characteristics of osteosarcoma to tune the network model parameters to achieve the optimal effect.

[0011] Step d: Use the test set to test and evaluate the obtained network model, and finally realize the automatic and intelligent osteosarcoma tumor recognition and segmentation function.

[0012] The encoder described in step a is mainly composed of a convolutional residual module and an attention module:

[0013] In the convolutional residual module, a 3-channel image of size 512*512 is input and input into the residual module after a layer of 7*7 convolution. Each residual module includes a skip structure and two 3*3 convolution kernels. The feature structure is retained in the feature map and the maximum pooling operation is performed after the input is input into the skip connection structure. After four downsampling operations, the image scale becomes 1 / 16 of the original image.

[0014] The attention module is mainly composed of the Transformer model. The image is serialized and input into the attention layer after being downsampled 4 times by the convolution residual module. The attention layer includes position encoding and 12 layers of Transformer to obtain the encoded result feature map. The attention module is composed of pixel slices input after 4 downsampling, and the slices x p Mapped to the D-dimensional embedding space, slice embedding is performed. The slice embedding expression is as follows:

[0015] E pos ∈R N*D

[0016] Where, P 2 is the slice size, N = H*W / P 2 , E pos is the location information, C is the number of channels;

[0017] The embedding layer is then fed into the Transformer layer for computation, which includes a multi-head attention module and a multi-layer perceptron module. The output of the Lth layer is represented as follows:

[0018]

[0019] Where MSA represents the multi-head attention module and MLP represents the multi-layer perceptron module.

[0020] The decoder in step a includes residual learning and upsampling.

[0021] The residual learning method uses the feature map addition method, which includes three 3*3 convolution kernels. The feature map is retained before each convolution, and the feature map after three convolutions is added to the three retained feature maps to obtain the feature map after residual learning. Then, skip connection and upsampling are performed:

[0022] Upsampling is performed four times. Before each upsampling, the feature map of the previous layer is concatenated with the feature map from the jump connection. A 3x3 convolution is then performed to halve the number of channels in the feature map, restoring its original size. The feature map is then upsampled by a factor of 2 and subjected to a 3x3 convolution, doubling its resolution and halving its number of channels. Linear rectification is then performed to obtain the upsampled result. This upsampling is repeated four times to restore the feature map to its original resolution.

[0023] The neural network model described in step a contains a total of 4 jump connection structures, corresponding to 4 up and down samplings. The jump connection retains the semantic information of the image before downsampling to achieve the purpose of assisting upsampling to restore the size of the feature map. The way to combine feature maps in the jump connection is splicing. Since the number of up and down sampling layers of the jump connection is corresponding, the resolution scale of the two corresponding feature maps is consistent. The two feature maps with the same resolution are channel-spliced ​​to obtain a feature map with unchanged length and width and doubled number of channels as the result of the jump connection feature map splicing; then the number of feature map channels is halved through 3*3 convolution to achieve the purpose of fusing deep and shallow layer features, and at the same time restore the feature map to its original scale in preparation for the subsequent upsampling of the feature map.

[0024] The step b of establishing an osteosarcoma dataset comprises:

[0025] Step b1: Obtain osteosarcoma medical images and perform tumor annotation to establish an experimental dataset.

[0026] Step b2: Perform image preprocessing, convert the MRI image into a 512*512 PNG format image, improve the image resolution through super-resolution method, and reduce the noise of the image.

[0027] Step b3: Perform data augmentation, including random rotation and mirroring. Specifically, the foot and knee data are mirrored, doubling the sample size to match the oral data. The data for the three body parts are then augmented by rotating them 90°, 180°, and 270°, respectively.

[0028] Step b4: Divide the samples into a training set and a test set in a ratio of 7:3. The training set contains the samples and their annotations and is stored in a unified file format to complete the establishment of the osteosarcoma dataset.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention provides an osteosarcoma recognition method based on the RTUNet network. The network model of this method, the RTUNet network, combines the residual network ResNet, Transformer, and U-Net jump connection structure. The addition of the convolution residual module avoids the problems of gradient vanishing and semantic information loss during training that may occur when deepening the network; the addition of the Transformer module improves the internal limitations of convolution operations in modeling and displaying long-distance relationships; in the encoding process, spatial features are captured by using a convolutional neural network, and the Transformer captures global features. In the decoding process, spatial information is restored through convolution, and the high-level semantic information obtained by encoding is expanded on a two-dimensional level, thereby obtaining an intuitive and accurate segmentation map; at the same time, jump connections are added between the encoding layer and the decoding layer to retain detailed semantic information and assist back propagation; thus, better segmentation performance can be achieved than other commonly used networks in the medical field, such as Unet. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram of the process of an osteosarcoma identification method based on RTUNet network of the present invention;

[0032] Figure 2 This is the overall structure diagram of the RTUNet neural network model of the present invention;

[0033] Figure 3 Schematic diagram of the convolutional residual module of the encoder in the network model of the present invention;

[0034] Figure 4 Schematic diagram of the attention module of the encoder in the network model of the present invention;

[0035] Figure 5 Schematic diagram of the decoder structure in the network model of the present invention;

[0036] Figure 6 Schematic diagram of the jump connection structure in the network model of the present invention;

[0037] Figure 7 This is a schematic diagram of the osteosarcoma dataset of the present invention;

[0038] Figure 8 Schematic diagram of osteosarcoma identification and segmentation results of the present invention. DETAILED DESCRIPTION

[0039] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below in conjunction with specific implementation methods.

[0040] like Figure 1 As shown, a method for identifying osteosarcoma based on the RTUNet network includes the following steps:

[0041] 1. Build a network model: Figure 2 The RTUNet network model is constructed as shown. This network model combines the ResNet residual network, the Transformer, and the U-Net skip connection structure. It consists primarily of an encoder and a decoder. The overall network design is based on the classic U-Net network U-shaped structure, with the encoder at the top and the decoder at the bottom. The encoder primarily consists of a convolutional residual module and an attention module, while the decoder consists of [ 1 ] . The overall network retains the skip structure. The encoder extracts feature information, which is primarily composed of convolutional and attention components. In the encoder structure, the image passes through the convolutional layer to extract low-level spatial two-dimensional features. It is then serialized into the attention module to extract high-level global features. This spatial information is passed to the decoder via skip connections to help restore global information. The decoder layer's main task is to restore the feature map to the input image size. The skip connection structure feeds the low-level spatial feature information into the decoder layer, assisting with upsampling. For an image with a spatial resolution of H*W and a number of channels C, our goal is to predict a label map with a corresponding pixel size of H*W. The specific prediction process is as follows:

[0042] (1) When inputting an image with a pixel size of 512*512 and a channel number of 3, it will first be input into Figure 3 The convolution residual module shown in the figure is input into the residual module after the first layer of 7*7 convolution. Each residual module includes a skip structure and two 3*3 convolution kernels. The feature structure is retained in the feature map. After inputting into the skip connection structure, the maximum pooling operation is performed. After 4 downsampling, a 32*32*512 feature map is output.

[0043] (2) The feature map is serialized into and then input into the attention module. The serialization process is to input pixel slices, the slice size is, the number is N = H * W / P2, and the slice xp is mapped to the D-dimensional embedding space. The slice embedding expression is as follows:

[0044] E pos ∈R N*D

[0045] Where, P 2 is the slice size, N = 32*32 / P 2 , E pos is the location information, C is the number of channels 512;

[0046] The attention module consists of 12 layers of Transformer, each layer of Transformer is the same, such as Figure 4As shown, taking the first layer as an example, after regularization, the multi-head attention module extracts features, then adds them to the input to obtain the intermediate result, and then inputs them into the multi-layer perceptron after regularization again, and then adds them to the input and outputs them. Similarly, the input is input into the second layer of Transformer, and finally the result is output through 12 layers of Transformer. The output formula of the Lth layer is as follows:

[0047]

[0048] Where MSA represents the multi-head attention module and MLP represents the multi-layer perceptron module.

[0049] (3)z 12 After the reshape operation, the feature map of size 32*32*512 is output and then input into Figure 5 The decoder module shown in the figure includes residual connections and upsampling. Upsampling is performed four times, corresponding to the four downsampling times in the encoder, to restore the high-level semantic feature maps obtained in the encoder to the original resolution of the image.

[0050] The decoder first performs residual learning on the input feature map to achieve feature recovery Figure 2 The goal is to enhance network recognition capabilities by integrating multi-scale features and retaining dimensional information. The feature maps input to the decoder are those encoded by the attention module. They contain global information but lack two-dimensional local feature information. Therefore, convolution operations and residual learning are required to restore the two-dimensional features for better upsampling. Residual learning uses feature map addition, consisting of three 3*3 convolution kernels. The feature map is retained before each convolution. The feature map after three convolutions is added to the retained three feature maps to obtain the residual learned feature map. Skip connections and upsampling are then performed.

[0051] Upsampling is performed 4 times, such as Figure 6 Before each upsampling, the feature map of the previous layer is concatenated with the feature map from the skip connection. A 3x3 convolution is then performed to halve the number of channels in the feature map, restoring its original size. The feature map is then upsampled by a factor of 2 and subjected to a 3x3 convolution, doubling its resolution and halving its number of channels. Linear rectification is then performed to obtain the upsampled result. This upsampling cycle is repeated four times to restore the feature map to its original resolution.

[0052] 2. Establish Figure 7 The osteosarcoma dataset shown includes the following steps:

[0053] Step 1: Obtain a certain amount of osteosarcoma medical images and annotate the tumors according to the guidance of professional doctors to establish an experimental dataset.

[0054] Step 2: Perform image preprocessing, convert the MRI image into a 512*512 PNG format image, improve the image resolution through super-resolution method, and reduce the noise of the image.

[0055] Step 3: Perform data augmentation, including random rotation and mirroring. Specifically, the foot and knee data are mirrored, doubling their size to roughly match the oral data sample size. The data for the three body parts are then augmented by rotating them 90°, 180°, and 270°, respectively.

[0056] Step 4: Divide the samples into training set and test set in a ratio of 7:3. The training set contains the samples and their annotations and stores them in a unified file format to complete the establishment of the osteosarcoma dataset.

[0057] 3. Model training: During model training, the RTUNet neural network model was trained using the training set of the osteosarcoma dataset. Combined with the characteristics of osteosarcoma images, the network model parameters were tuned to achieve optimal results. In a GPU environment, 500 epochs were trained, with 41 iterations per epoch, for a total of 20,500 iterations.

[0058] 4. Model testing: The RTUNet neural network model was tested and evaluated using the test set of the osteosarcoma dataset. The Dcie coefficient was used to evaluate the recognition and segmentation accuracy of the model. The Dcie coefficient is a metric based on set similarity, used to calculate the similarity between two samples. The value range is 0-1, and the larger the value, the better the segmentation quality. It is calculated as follows:

[0059]

[0060] Where |X| and |Y| are the areas of the two samples respectively; |X∩Y| is the intersection of X and Y, i.e., the overlap of the two sample areas. A larger d indicates a higher overlap of the two sample areas, which means better segmentation quality.

[0061] After testing, the RTUNet neural network model used in the present invention has a Dcie coefficient of 0.61, 0.84, and 0.91 for foot, knee, and oral recognition and segmentation, respectively, with an average value of 0.79. Compared with the average value of 0.68 obtained by testing using the standard network model UNet in the medical field, the recognition and segmentation accuracy is higher, and the segmentation results are more reasonable. Ultimately, an automatic and intelligent osteosarcoma tumor recognition and segmentation function is achieved. After inputting an osteosarcoma MRI image, the following can be output: Figure 8 The osteosarcoma recognition and segmentation results shown will be of great significance in assisting doctors in the initial diagnosis and treatment of osteosarcoma patients, as well as in subsequent continuous medical observation and prevention of tumor recurrence and metastasis.

[0062] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for identifying osteosarcoma based on the RTUNet network, characterized by: The following steps are involved: Step a: Constructing an RTUNet neural network model for identifying osteosarcoma. The RTUNet neural network model combines a residual network (ResNet), a Transformer, and a U-Net skip connection structure, and is composed of an encoder and a decoder. The input image is first passed through the encoder to extract feature information, and then the low-level spatial feature information is input into the decoder layer through the skip connection structure to assist in upsampling and obtain a segmentation result consistent with the image input size. The encoder is mainly composed of a convolutional residual module and an attention module; In the convolutional residual module, a 3-channel image of size 512*512 is input and input into the residual module after a layer of 7*7 convolution. Each residual module includes a skip structure and two 3*3 convolution kernels. The feature structure is retained in the feature map and the maximum pooling operation is performed after the input into the skip connection structure. After 4 downsampling, the image size becomes 1 / 16 of the original image. The attention module is mainly composed of the Transformer model. The image is serialized and input into the attention layer after being downsampled 4 times by the convolution residual module. The attention layer includes position encoding and 12 layers of Transformer to obtain the encoded result feature map. The attention module is composed of pixel slices input after 4 downsampling, and the slices x p Mapped to the D-dimensional embedding space, slice embedding is performed. The slice embedding expression is as follows: Where, P 2 is the slice size, N = H*W / P 2 , E pos is the location information, C is the number of channels; The embedding layer is then fed into the Transformer layer for computation, which includes a multi-head attention module and a multi-layer perceptron module. The output of the Lth layer is represented as follows: In the formula, MSA represents the multi-head attention module, MLP represents the multi-layer perceptron module; The decoder includes residual learning and upsampling: The residual learning adopts the feature map addition method, which includes three 3*3 convolution kernels. The feature map is retained before each convolution, and the feature map after three convolutions is added to the three retained feature maps to obtain the feature map after residual learning, and then skip connection and upsampling are performed. Upsampling is performed 4 times in total. Before each upsampling, the feature map of the previous layer is concatenated with the feature map obtained by jump connection. Then, the number of channels of the feature map is halved through 3*3 convolution to restore its original size. The feature map is then upsampled by 2 times and convolved by 3*3 to double the resolution and halve the number of channels. After that, linear rectification is performed to obtain the upsampling result. The upsampling is repeated 4 times in total to restore the feature map to its original resolution. Step b: Establish an osteosarcoma dataset, which is divided into a training set and a test set; Step c: Use the training set to train the RTUNet neural network model, and optimize the network model parameters based on the characteristics of osteosarcoma images to achieve the optimal network effect; Step d: Use the test set to test and evaluate the obtained network model, and finally realize the automatic and intelligent osteosarcoma tumor recognition and segmentation function.

2. The osteosarcoma identification method based on the RTUNet network according to claim 1, characterized in that: The neural network model contains a total of 4 skip connection structures, corresponding to 4 up and down sampling; the way to combine feature maps in the skip connection is splicing. Since the number of up and down sampling layers of the skip connection is corresponding, the resolution scale of the two corresponding feature maps is consistent. The two feature maps with the same resolution are channel-spliced, and the result is a feature map with unchanged length and width and doubled number of channels as the result of the skip connection feature map splicing; then the number of feature map channels is halved through 3*3 convolution, and the feature map is restored to its original scale in preparation for the subsequent upsampling of the feature map.

3. The osteosarcoma identification method based on RTUNet network according to claim 2, characterized in that: The step b, establishing an osteosarcoma dataset, includes the following detailed steps: Step b1: Obtain osteosarcoma medical images and perform tumor annotation to establish an experimental dataset; Step b2: Perform image preprocessing to convert the MRI image into a 512*512 PNG format image, improve the image resolution using a super-resolution method, and perform image noise reduction; Step b3: Perform data augmentation, including random rotation and mirroring. Specifically, the foot and knee data are first mirrored to double the sample size, bringing the sample size to the same as the oral data. The data for the three body parts are then rotated 90°, 180°, and 270°, respectively, for data augmentation. Step b4: Divide the samples into a training set and a test set in a ratio of 7:

3. The training set contains the samples and their annotations and is stored in a unified file format to complete the establishment of the osteosarcoma dataset.

Images