Medical imaging brain tumor detection method and system based on improved Faster R-CNN

Abstract

The present invention discloses a medical imaging brain tumor detection method and system based on an improved Faster R-CNN. The method comprises: obtaining a magnetic resonance image dataset from a database; preprocessing the images of the dataset using an optical flow method and a generative adversarial network to perform data enhancement; constructing an improved Faster R-CNN deep network model; using the obtained dataset to train the constructed improved Faster R-CNN deep network model; inputting the magnetic resonance image to be predicted into the trained network model, and outputting the brain tumor target detection result. The present invention adopts a data enhancement method based on an optical flow method and a generative adversarial network, and designs an improved Faster R-CNN target detection network model, adds an image feature pyramid structure and a convolutional gated recurrent unit (ConvGRU) module, modifies the data reading method during network training, enhances the continuity of tumor detection results, and improves the retrieval accuracy and recall rate of brain tumors.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, medical image processing, and machine learning technology, and in particular to a medical imaging brain tumor detection method and system based on an improved Faster R-CNN. Background Art

[0002] Brain tumors are a common disease among modern people, yet most people lack understanding of them. Many patients even die due to a lack of timely diagnosis and treatment. Brain tumors can be categorized as either benign or malignant, with malignant tumors further comprising primary malignant tumors and brain metastases. Brain metastases are the most common, while benign and primary malignant tumors are less common clinically. Brain tumors are typically diagnosed using magnetic resonance imaging (MRI), which produces a 3D grayscale image of brain tissue in a single scan. MRI is more sensitive than computed tomography (CT) for diagnosing brain tumors, can detect early lesions, and can more accurately locate them. However, brain tumors are often difficult to identify, and a single MRI scan typically includes hundreds of two-dimensional slices, making diagnosis from a single slice difficult.

[0003] In recent years, deep learning (DL) technology has rapidly developed. Deep convolutional neural networks (CNNs), among them, are able to effectively capture image features and have been widely applied to various image processing and computer vision tasks. In medical image processing, CNNs can automatically extract features of underlying organs and tissues, playing an important supporting role in tasks such as medical image classification, detection, segmentation, retrieval, image generation, and enhancement. However, existing deep learning object detection algorithms, such as FasterR-CNN, are primarily designed for natural images. Compared to natural images, magnetic resonance images (MRIs) have less distinct differences between lesions and normal areas and require consideration of lateral continuity across multiple slices. Therefore, directly applying existing natural image object detection models can lead to poor performance in predicting continuity and inaccurate detection results. Furthermore, among natural brain tumor patients, the number of tumor types is uneven, with fewer samples of benign tumors and primary malignant tumors, resulting in poor detection performance.

[0004] One existing technology, "A Brain Tumor Detection and Segmentation Method Based on CNN and SVM Fusion," replaces the CNN's softmax classifier with a support vector machine (SVM) and the cross-entropy loss with a margin function. This method constructs classifiers for brain tumors and normal tissue for brain tumor detection. However, this method suffers from an overly simplistic network structure and fails to consider the difficulty in distinguishing lesion areas in medical images and the imbalance in the number of diseased and normal samples. This can lead to low model accuracy and inaccurate detection locations. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of existing methods and propose a method and system for detecting brain tumors in medical images based on an improved Faster R-CNN. The main problems solved by the present invention are: first, the number of different types of tumor samples in natural brain tumor patients is unbalanced, and the difference between lesions and normal parts in magnetic resonance images is not obvious; second, the model coherence of existing deep learning target detection algorithms is poor, and they cannot meet the technical requirements of medical detection that require consideration of the horizontal continuity information between multiple slices. In other words, how to enable the model of the deep learning target detection algorithm to better grasp the contextual information of the slices, learn the connection between adjacent slices, and thus enhance the coherence and accuracy of the prediction results under the conditions of unbalanced tumor sample data, unclear difference between lesions and normal parts in magnetic resonance images, and the need to consider the horizontal continuity information between multiple slices.

[0006] To solve the above problems, the present invention proposes a method for detecting brain tumors in medical images based on an improved Faster R-CNN, the method comprising:

[0007] obtaining a magnetic resonance image dataset from a magnetic resonance image database;

[0008] Preprocessing the images of the acquired dataset and performing data enhancement by using an optical flow method and a generative adversarial network;

[0009] Build and improve the Faster R-CNN deep network model, add an image feature pyramid structure and a Convolutional Gate Recurrent Unit (ConvGRU) module, and modify the data reading method during network training;

[0010] The data set obtained after data enhancement is used to train an improved Faster R-CNN model;

[0011] The magnetic resonance image to be predicted is input into the trained improved Faster R-CNN model to output the brain tumor target detection result.

[0012] Preferably, the image of the acquired data set is preprocessed and data enhancement is performed by using the optical flow method and the generative adversarial network, specifically:

[0013] Find the normal slice sequence that is closest in brightness to the slice with tumor. The slice with tumor is recorded as Query, and the normal slice sequence is recorded as Normal. Specifically, for the slice sequence to which Query belongs, first divide it into 10 groups according to the arrangement order, calculate the mean and standard deviation of the pixel brightness of each group, and obtain a brightness value statistical vector of length 20. The calculation formula is as follows:

[0014]

[0015] in, Represents the brightness value statistics vector of length 20 corresponding to the Query sequence,

[0016] For all normal sequences without tumors in the data set, the corresponding statistical vectors of length 20 are calculated using the same method. Then calculate the Euclidean distance between the brightness value statistical vector of the Normal sequence and the brightness value statistical vector of the sequence to which the Query belongs:

[0017]

[0018] in, Represents the brightness value statistics vector of length 20 corresponding to the Query sequence, Represents a statistical vector of length 20 corresponding to the Normal sequence,

[0019] Among all Normal sequences, the one whose brightness value statistical vector has the smallest distance to the brightness value statistical vector of the sequence to which the query belongs is the Normal sequence that is closest to the query slice in terms of brightness;

[0020] The obtained Normal sequence is operated to obtain the normal slice that is closest to the tumor slice in brightness, which is recorded as Answer_pri. Specifically, the pre-trained ResNet50 model is used to extract features from each slice in the Query and Normal sequences, and then v is calculated. a With v i The cosine similarity between them is calculated as follows:

[0021]

[0022] Among them, v q Indicates the value obtained by feature extraction of Query, v iRepresents the value obtained by feature extraction for each slice in the Normal sequence, i = 1, 2, ..., n,

[0023] Sort the slices in the Normal sequence according to the similarity from large to small, and select the top 10 slices with the highest similarity, which are recorded as N1, N2, ..., N 10 , and then use the RAFT algorithm to calculate the optical flow from the query to the slice, denoted as f1, f2, ..., f 10 , and then use the optical flow to perform image twisting on the slice to obtain the twisted result N′ i , the calculation formula is as follows:

[0024] N′ i (x′ i , y′ i )=N i (x i ,y i )

[0025] x′ i =x i -f i (x i ,y i ,0)

[0026] y′ i =y i -f i (x i ,y i , 1)

[0027] Among them, N i Indicates the first 10 slices selected, N′ i represents the result after the slice is twisted, (x i ,y i ) represents N i The pixel coordinates, (x′ i , y′ i ) represents N′ i The pixel coordinates, f i represents the optical flow from Query to the slice, i=1, 2, ..., 10,

[0028] Then for N′ i Perform histogram matching according to Query, so that N′ i The brightness distribution between the query and the query is similar to eliminate the influence of brightness on similarity judgment. Finally, N′ is calculated. i The mean square error of pixel brightness between the Query and the Query is calculated as follows:

[0029]

[0030] N′ with the smallest error i That is Answer_pri;

[0031] The tumor area is mapped on Answer_pri to obtain a preliminary pseudo-tumor slice, denoted as Answer_warped. Specifically, the optical flow from Query to Answer_pri is first calculated, denoted as f1. The position of the tumor in Query is marked by a bounding box. Using f1, according to the image warping formula, the four vertex coordinates of the bounding box of the tumor position in Query are calculated and mapped to the corresponding coordinates on Answer_pri, thereby determining the distribution area of the pseudo-tumor on Answer_pri. Then, the optical flow from Answer_pri to Query is calculated, denoted as f2. Using f2, according to the image warping formula, the corresponding coordinates (i′, j′) of each pixel point (i, j) in the pseudo-tumor distribution area in Query are found. According to the pixel values of the corresponding coordinates, the bilinear interpolation method is used to preliminarily fill the pixels in the pseudo-tumor distribution area to obtain the preliminary filling value Answer_temp, so as to achieve the effect of "transplanting" the tumor from Query to Answer_pri. The calculation formula is as follows:

[0032] x1=int(i′) y1=int(j′)

[0033] x2=x1 y2=y1+1

[0034] x3=x1+1 y3=y1

[0035] x4=x1+1 y4=y1+1

[0036] u=i′-x1 v=j′-y1

[0037] Answer_temp(i,j)=(1-u)(1-v)×int(Query(x1,y1))+(1-u)v×int(Query(x2,y2))+u(1-v)×int(Query(x3,y3))+uv×int(Query(x4,y4))

[0038] Wherein, (i, j) represents the coordinate value of each pixel point in the pseudo tumor distribution area, (i′, j′) represents the corresponding coordinate value of each pixel point in the pseudo tumor distribution area in the Query,

[0039] The final filling result Answerwarped is defined as the average weighted sum of Answer_pri and Answer_temp to retain the original organization part of Answer_pri. The calculation formula is as follows:

[0040] Answer_warped(i,j)=0.5×Answer_pri(i,j)+0.5×Answer_temp(i,j)

[0041] Wherein, (i, j) represents the coordinate value of each pixel point in the pseudo-tumor distribution area, and (i′, j′) represents the corresponding coordinate value of each pixel point in the pseudo-tumor distribution area in the Query;

[0042] Answerwarped is further processed to obtain the final fake tumor slice, denoted as Answer, so that it is as similar as possible to the slice with a real tumor. Specifically, a generative adversarial network is used to reconstruct the image to eliminate the unnatural parts of Answerwarped. The network takes Answer_warped as input and outputs Answer. The network loss function is modified to include three parts: reconstruction loss, coherence loss, and adversarial loss. The entire loss function is defined as the weighted sum of reconstruction loss, coherence loss, and adversarial loss. The calculation formula is as follows:

[0043] Loss=50×recLoss+50×cohLoss+advLoss

[0044] Among them, recLoss represents reconstruction loss, cohLoss represents coherence loss, and advLoss represents adversarial loss.

[0045] Preferably, the construction improves the Faster R-CNN deep network model, adds an image feature pyramid structure and a ConvGRU module, and modifies the data reading method during network training, specifically:

[0046] Based on Faster R-CNN, the FPN structure is added to improve the feature extraction part of Faster R-CNN. Features are extracted in the form of image feature pyramids, fusing high-resolution shallow information with low-resolution but semantically rich deep information.

[0047] After Faster R-CNN extracts features and before entering the RPN structure, a ConvGRU module is added to memorize the context information of the slice;

[0048] The continuous slices in the MRI dataset are divided into groups of 3, and the upper and lower slices provide 3D context information for the middle slice. Then, during training, the data is read in groups to ensure that the sample data read each time is a continuous slice of a certain case.

[0049] Accordingly, the present invention also provides a medical imaging brain tumor detection system based on an improved Faster R-CNN, comprising:

[0050] a magnetic resonance image data set acquisition unit, configured to acquire a magnetic resonance image data set from a magnetic resonance image database;

[0051] A data enhancement unit, configured to preprocess the images of the acquired data set and perform data enhancement by using an optical flow method and a generative adversarial network;

[0052] Improved the Faster R-CNN model building unit, used to build an improved Faster R-CNN deep network model, adding an image feature pyramid structure and ConvGRU module, and modifying the data reading method during network training;

[0053] An improved Faster R-CNN model training unit is used to train an improved Faster R-CNN model using the data set obtained after data enhancement;

[0054] The target detection result output unit is used to input the magnetic resonance image to be predicted into the trained improved Faster R-CNN model and output the brain tumor target detection result.

[0055] The implementation of the present invention has the following beneficial effects:

[0056] The present invention uses a deep learning method to realize the function of 2D retrieval of brain tumor areas from MRI brain medical images and judgment of the tumor category, which plays an auxiliary role in medical diagnosis and research; the invention adopts a data enhancement method based on optical flow method and generative adversarial network, which can expand the brain MRI medical image data set, thereby alleviating the problem of imbalance of tumor categories in the data set. After adding the generated "fake" data to the training set, the model has significantly improved the retrieval accuracy and recall rate of benign tumors and primary tumors, and the overall average retrieval accuracy and average recall rate scores are also higher; the invention designs an improved Faster R-CNN target detection network model, adds an image feature pyramid structure and a ConvGRU module, and modifies the data reading method during network training, which enables the model to learn multi-scale features and contextual coherence information, enhances the continuity of tumor detection results, and further improves the application effect of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is an overall flow chart of a method for detecting brain tumors in medical images based on an improved Faster R-CNN according to an embodiment of the present invention;

[0058] Figure 2 is a flow chart of tumor image data enhancement according to an embodiment of the present invention;

[0059] Figure 3 This is a flow chart of constructing an improved Faster R-CNN deep network model according to an embodiment of the present invention;

[0060] Figure 4 This is a structural diagram of the improved Faster R-CNN deep network model according to an embodiment of the present invention;

[0061] Figure 5 4 is a structural diagram of a medical imaging brain tumor detection system based on an improved Faster R-CNN according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0063] Figure 1 FIG. 1 is an overall flow chart of a method for detecting brain tumors in medical images based on an improved Faster R-CNN according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0064] S1, obtains a magnetic resonance image dataset from a magnetic resonance image database;

[0065] S2, preprocessing the images of the acquired dataset and performing data enhancement by using an optical flow method and a generative adversarial network;

[0066] S3: Build and improve the Faster R-CNN deep network model, add an image feature pyramid structure and ConvGRU module, and modify the data reading method during network training;

[0067] S4, using the data set obtained after data enhancement to train an improved Faster R-CNN model;

[0068] S5: Input the magnetic resonance image to be predicted into the trained improved Faster R-CNN model and output the brain tumor target detection result.

[0069] Step S1 is as follows:

[0070] S1-1, obtain a magnetic resonance image dataset from a magnetic resonance image database.

[0071] Step S2, such as Figure 2 As shown, the details are as follows:

[0072] S2-1: Find the normal slice sequence that is most similar in brightness to the slice with the tumor. The slice with the tumor is denoted as Query, and the normal slice sequence is denoted as Normal. Specifically, for the slice sequence to which Query belongs, first divide it into 10 groups according to the order of arrangement. Calculate the mean and standard deviation of the pixel brightness of each group to obtain a brightness value statistical vector of length 20. The calculation formula is as follows:

[0073]

[0074] in, Represents the brightness value statistics vector of length 20 corresponding to the Query sequence,

[0075] For all normal sequences without tumors in the data set, the corresponding statistical vectors of length 20 are calculated using the same method. Then calculate the Euclidean distance between the brightness value statistical vector of the Normal sequence and the brightness value statistical vector of the sequence to which the Query belongs:

[0076]

[0077] in, Represents the brightness value statistics vector of length 20 corresponding to the Query sequence, Represents a statistical vector of length 20 corresponding to the Normal sequence,

[0078] Among all Normal sequences, the one whose brightness value statistical vector has the smallest distance to the brightness value statistical vector of the sequence to which the query belongs is the Normal sequence that is closest to the query slice in terms of brightness;

[0079] S2-2, operate the obtained Normal sequence to obtain the normal slice that is closest to the tumor slice in brightness, recorded as Answer_pri. Specifically, first use the pre-trained ResNet50 model to extract features for each slice in the Query and Normal sequences, and then calculate v q With v i The cosine similarity between them is calculated as follows:

[0080]

[0081] Among them, v q Indicates the value obtained by feature extraction of Query, v i Represents the value obtained by feature extraction for each slice in the Normal sequence, i = 1, 2, ..., n,

[0082] Sort the slices in the Normal sequence according to the similarity from large to small, and select the top 10 slices with the highest similarity, which are recorded as N1, N2, ..., N 10 , and then use the RAFT algorithm to calculate the optical flow from the query to the slice, denoted as f1, f2, ..., f 10 , and then use the optical flow to perform image twisting on the slice to obtain the twisted result N′ i , the calculation formula is as follows:

[0083] N′ i (x′ i , y′ i )=N i (x i ,y i )

[0084] x′ i =x i -f i (x i ,y i ,0)

[0085] y′ i =y i -f i (x i ,y i , 1)

[0086] Among them, N i Indicates the first 10 slices selected, N′ i represents the result after the slice is twisted, (x i ,y i ) represents N i The pixel coordinates, (x′ i , y′ i ) represents N′ i The pixel coordinates, f i represents the optical flow from Query to the slice, i=1, 2, ..., 10,

[0087] Then for N′ i Perform histogram matching according to Query, so that N′ i The brightness distribution between the query and the query is similar to eliminate the influence of brightness on similarity judgment. Finally, N′ is calculated. iThe mean square error of pixel brightness between the Query and the Query is calculated as follows:

[0088]

[0089] N′ with the smallest error i That is Answer_pri;

[0090] S2-3, map the tumor area on Answer_pri to obtain a preliminary pseudo-tumor slice, recorded as Answer_warped. Specifically, first calculate the optical flow from Query to Answer_pri, recorded as f1. The position of the tumor in Query is marked by a bounding box. Using f1, according to the image warping formula, calculate the corresponding coordinates of the four vertex coordinates of the bounding box of the tumor position in Query mapped on Answer_pri, thereby determining the distribution area of the pseudo-tumor on Answer_pri. Then calculate the optical flow from Answer_pri to Query, recorded as f2. Using f2, according to the image warping formula, find the corresponding coordinates (i′, j′) of each pixel point (i, j) in the pseudo-tumor distribution area in Query. According to the pixel values of the corresponding coordinates, use bilinear interpolation to preliminarily fill the pixels in the pseudo-tumor distribution area to obtain the preliminary filling value Answer_temp, so as to achieve the effect of "transplanting" the tumor from Query to Answer_pri. The calculation formula is as follows:

[0091] x1=int(i′) y1=int(j′)

[0092] x2=x1 y2=y1+1

[0093] x3=x1+1 y3=y1

[0094] x4=x1+1 y4=y1+1

[0095] u=i′-x1 v=j′-y1

[0096] Answer_temp(i,j)=(1-u)(1-v)×int(Query(x1,y1))+(1-u)v×int(Query(x2,y2))+u(1-v)×int(Query(x3,y3))+uv×int(Query(x4,y4))

[0097] Wherein, (i, j) represents the coordinate value of each pixel point in the pseudo tumor distribution area, (i′, j′) represents the corresponding coordinate value of each pixel point in the pseudo tumor distribution area in the Query,

[0098] The final filling result Answerwarped is defined as the average weighted sum of Answer_pri and Answer_temp to retain the original organization part of Answer_pri. The calculation formula is as follows:

[0099] Answer_warped(i,j)=0.5×Answer_pri(i,j)+0.5×Answer_temp(i,j)

[0100] Wherein, (i, j) represents the coordinate value of each pixel point in the pseudo-tumor distribution area, and (i′, j′) represents the corresponding coordinate value of each pixel point in the pseudo-tumor distribution area in the Query;

[0101] In step S2-4, Answerwarped is further processed to obtain the final fake tumor slice, denoted as Answer, so that it is as similar as possible to the slice with a real tumor. Specifically, a generative adversarial network is used to reconstruct the image to eliminate the unnatural parts of Answer_warped. The network takes Answer_warped as input and outputs Answer. The network loss function is modified to include three parts: reconstruction loss, coherence loss, and adversarial loss. The entire loss function is defined as the weighted sum of reconstruction loss, coherence loss, and adversarial loss. The calculation formula is as follows:

[0102] Loss=50×recLoss+50×cohLoss+advLoss

[0103] Among them, recLoss represents reconstruction loss, cohLoss represents coherence loss, and advLoss represents adversarial loss.

[0104] Step S3, as Figure 3 、 Figure 4 As shown, the details are as follows:

[0105] S3-1, based on Faster R-CNN, adds the FPN structure and improves the feature extraction part of Faster R-CNN. It extracts features in the form of image feature pyramid, fusing high-resolution shallow information with low-resolution but semantically rich deep information.

[0106] S3-2, after Faster R-CNN extracts features and before entering the RPN structure, a ConvGRU module is added to memorize the context information of the slice;

[0107] S3-3, divide the continuous slices in the MRI dataset into groups of 3, and the upper and lower slices provide 3D context information for the middle slice. Then, during training, read the data in groups to ensure that the sample data read each time is a continuous slice in a certain case.

[0108] Step S4 is as follows:

[0109] S4-1, using the data set obtained after data enhancement to train an improved Faster R-CNN model.

[0110] Step S5 is as follows:

[0111] S5-1, input the magnetic resonance image to be predicted into the trained improved Faster R-CNN model, and output the brain tumor target detection result.

[0112] Accordingly, the present invention also provides a medical imaging brain tumor detection system based on improved Faster R-CNN, such as Figure 5 As shown, including:

[0113] The magnetic resonance image data set acquisition unit 1 is configured to acquire a magnetic resonance image data set from a magnetic resonance image database.

[0114] The data enhancement unit 2 is used to preprocess the images of the acquired data set and perform data enhancement by using an optical flow method and a generative adversarial network.

[0115] Specifically, find the normal slice sequence that is closest to the tumor slice in terms of brightness. The tumor slice is recorded as Query, and the normal slice sequence is recorded as Normal. Specifically, for the slice sequence to which Query belongs, first divide it into 10 groups according to the order of arrangement, calculate the mean and standard deviation of the pixel brightness of each group, and obtain a brightness value statistical vector of length 20 For all normal sequences without tumors in the data set, the corresponding statistical vectors of length 20 are calculated using the same method. Then, the Euclidean distance between the brightness value statistical vector of the Normal sequence and the brightness value statistical vector of the sequence to which the Query belongs is calculated. Among all Normal sequences, the one with the smallest distance between its brightness value statistical vector and the brightness value statistical vector of the sequence to which the Query belongs is the Normal sequence that is closest to the Query slice in terms of brightness. The obtained Normal sequence is operated to obtain the normal slice that is closest to the tumor slice in terms of brightness, which is recorded as Answer_pri. Specifically, first, the pre-trained ResNet50 model is used to extract features from the Query to obtain v q , extract features from each slice in the Normal sequence to get vi (i=1,2,...,n), then calculate v a With v i The cosine similarity between them is used to sort the slices in the Normal sequence from large to small according to the similarity, and the top 10 slices with the highest similarity are selected and recorded as N1, N2, ..., N 10 , and then use the RAFT algorithm to calculate the optical flow from the query to the slice, denoted as f1, f2, ..., f 10 , and then use the optical flow to perform image twisting on the slice to obtain the twisted result N′ i , then for N′ i Perform histogram matching according to Query, so that N′ i The brightness distribution between the query and the query is similar to eliminate the influence of brightness on similarity judgment. Finally, N′ is calculated. i The mean square error of pixel brightness between Query and the minimum error N′ iThat is Answer_pri; the tumor area is mapped on Answer_pri to obtain a preliminary pseudo-tumor slice, which is recorded as Answer_warped. Specifically, the optical flow from Query to Answer_pri is first calculated, which is recorded as f1. The position of the tumor in Query is marked by a bounding box. Using f1, according to the image twisting formula, the four vertex coordinates of the bounding box of the tumor position in Query are calculated to be mapped to the corresponding coordinates on Answer_pri, thereby determining the distribution area of the pseudo-tumor on Answer_pri. Then, the optical flow from Answer_pri to Query is calculated, which is recorded as f2. Using f2, according to the image twisting formula, the corresponding coordinates (i′, j′) of each pixel point (i, j) in the pseudo-tumor distribution area in Query are found. According to the pixel values of the corresponding coordinates, the pixels in the pseudo-tumor distribution area are interpolated using bilinear interpolation. Perform preliminary filling to obtain the preliminary filling value Answer, so as to achieve the effect of "transplanting" the tumor from Query to Answer_pri. The final filling result Answer_warped is defined as the average weighted sum of Answer_pri and Answer_temp to retain the original tissue part in Answer_pri. Answer_warped is further processed to obtain the final fake tumor slice, recorded as Answer, so that it is as similar as possible to the slice with a real tumor. Specifically, a generative adversarial network is used to reconstruct the image to eliminate the unnatural part of Answer_warped. The network takes Answer_warped as input and outputs Answer. The network's loss function is modified, including reconstruction loss, coherence loss, and adversarial loss. The entire loss function is defined as the weighted sum of reconstruction loss, coherence loss, and adversarial loss.

[0116] Improved Faster R-CNN model construction unit 3 is used to build an improved Faster R-CNN deep network model, add an image feature pyramid structure and ConvGRU module, and modify the data reading method during network training.

[0117] Specifically, based on Faster R-CNN, the FPN structure is added, and the feature extraction part of Faster R-CNN is improved. Features are extracted in the form of an image feature pyramid, fusing shallow information with high resolution and deep information with low resolution but rich semantic information. After Faster R-CNN extracts features and before entering the RPN structure, a ConvGRU module is added to memorize the contextual information of the slice. The continuous slices in the MRI dataset are divided into groups of 3, and the upper and lower slices provide 3D contextual information for the middle slice. Then, during training, data are read in groups to ensure that the sample data read each time is a continuous slice in a certain case.

[0118] The improved Faster R-CNN model training unit 4 is used to train the improved Faster R-CNN model using the data set obtained after the data enhancement.

[0119] The target detection result output unit 5 is used to input the magnetic resonance image to be predicted into the trained improved Faster R-CNN model and output the brain tumor target detection result.

[0120] Therefore, the present invention uses a deep learning method to realize the function of 2D retrieval of brain tumor areas from MRI brain medical images and judgment of the tumor category, which plays an auxiliary role in medical diagnosis and research; the invention adopts a data enhancement method based on optical flow method and generative adversarial network, which can expand the brain MRI medical image data set, thereby alleviating the problem of imbalance of tumor categories in the data set. After adding the generated "fake" data to the training set, the model has a significant improvement in the retrieval accuracy and recall rate of benign tumors and primary tumors, and the overall average retrieval accuracy and average recall rate scores are also higher; the invention designs an improved Faster R-CNN target detection network model, adds an image feature pyramid structure and a ConvGRU module, and modifies the data reading method during network training, which enables the model to learn multi-scale features and contextual coherence information, enhances the continuity of tumor detection results, and further improves the application effect of the model.

[0121] The above describes in detail the medical imaging brain tumor detection method and system based on the improved Faster R-CNN provided by the embodiments of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is intended only to help understand the method and core concept of the present invention. At the same time, those skilled in the art will appreciate that the specific implementation methods and scope of application may vary based on the concepts of the present invention. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims

1. A method for detecting brain tumors in medical images based on an improved Faster R-CNN, characterized in that: The method comprises: obtaining a magnetic resonance image dataset from a magnetic resonance image database; Preprocessing the images of the acquired dataset using an optical flow method and a generative adversarial network, performing data enhancement, and forming a dataset obtained after data enhancement; Build and improve the Faster R-CNN deep network model, add an image feature pyramid structure and a convolutional gated recurrent unit (ConvGRU) module, and modify the data reading method during network training; The data set obtained after data enhancement is used to train an improved Faster R-CNN model to form a trained improved Faster R-CNN model; Inputting the magnetic resonance image to be predicted into the trained improved Faster R-CNN model and outputting a brain tumor target detection result; Specifically, the image of the acquired data set is preprocessed and data enhancement is performed by using the optical flow method and the generative adversarial network, specifically: Find the normal slice sequence that is closest in brightness to the slice with tumor. The slice with tumor is recorded as Query, and the normal slice sequence is recorded as Normal. Specifically, for the slice sequence to which Query belongs, first divide it into 10 groups according to the arrangement order, calculate the mean and standard deviation of the pixel brightness of each group, and obtain a brightness value statistical vector of length 20. The calculation formula is as follows: in, Represents the brightness value statistics vector of length 20 corresponding to the Query sequence, For all normal sequences without tumors in the data set, the corresponding statistical vectors of length 20 are calculated using the same method. Then calculate the Euclidean distance between the brightness value statistical vector of the Normal sequence and the brightness value statistical vector of the sequence to which the Query belongs: in, Represents the brightness value statistics vector of length 20 corresponding to the Query sequence, Represents a statistical vector of length 20 corresponding to the Normal sequence, Among all Normal sequences, the one whose brightness value statistical vector has the smallest distance to the brightness value statistical vector of the sequence to which the query belongs is the Normal sequence that is closest to the query slice in terms of brightness; The Normal sequence that is closest to the Query slice in brightness is operated to obtain the normal slice that is closest to the tumor slice in brightness, which is recorded as Answer_pri. Specifically, the pre-trained ResNet50 model is first used to extract features from each slice in the Query and Normal sequences, and then v is calculated. q With v i The cosine similarity between them is calculated as follows: Among them, v q Indicates the value obtained by feature extraction of Query, v i Represents the value obtained by feature extraction for each slice in the Normal sequence, i = 1, 2, ..., n, Sort the slices in the Normal sequence according to the similarity from large to small, and select the top 10 slices with the highest similarity, denoted as N1, N2, ..., N 10 , and then use the RAFT algorithm to calculate the optical flow from the query to the slice, denoted as f1,f2,…,f 10 , and then use the optical flow to perform image twisting on the slice to obtain the twisted result N′ i , the image twist formula is as follows: N′ i (x′ i ,and' i )=N i (x i ,and i ) x′ i =x i -f i (x i ,y i ,0) and' i =and i -F i (x i ,and i ,1) Among them, N i Indicates the first 10 slices selected, N′ i represents the result after the slice is twisted, (x i ,y i ) represents N i The pixel coordinates, (x′ i ,y′ i ) represents N′ i The pixel coordinates, f i represents the optical flow from Query to the slice, i=1,2,…,10, Then for N′ i Perform histogram matching according to Query, so that N′ i The brightness distribution between the query and the query is similar to eliminate the influence of brightness on similarity judgment. Finally, N′ is calculated. i The mean square error of pixel brightness between the Query and the Query is calculated as follows: N′ with the smallest error i That is Answer_pri; The tumor area is mapped on Answer_pri to obtain a preliminary pseudo-tumor slice, denoted as Answer_warped. Specifically, the optical flow from Query to Answer_pri is first calculated, denoted as f1. The position of the tumor in Query is marked by a bounding box. Using f1, according to the image warping formula, the four vertex coordinates of the bounding box of the tumor position in Query are calculated and mapped to the corresponding coordinates on Answer_pri, thereby determining the distribution area of the pseudo-tumor on Answer_pri. Then, the optical flow from Answer_pri to Query is calculated, denoted as f2. Using f2, according to the image warping formula, the corresponding coordinates (i′, j′) of each pixel point (i, j) in the pseudo-tumor distribution area in Query are found. According to the pixel values of the corresponding coordinates, the pixels in the pseudo-tumor distribution area are preliminarily filled using bilinear interpolation to obtain the preliminary filling value Answer_temp, so as to achieve the effect of "transplanting" the tumor from Query to Answer_pri. The calculation formula is as follows: x1=int(i′) y1=int(j′) x2=x1 y2=y1+1 x3=x1+1 y3=y1 x4=x1+1 y4=y1+1 u=i′-x1 v=j′-y1 Answer_temp(i,j)=(1-u)(1-v)×int(Query(x1,y1))+(1-u)v×int(Query(x2,y2))+u(1-v)×int(Query(x3,y3))+uv×int(Query(x4,y4)) Wherein, (i, j) represents the coordinate value of each pixel point in the pseudo tumor distribution area, and (i′, j′) represents the corresponding coordinate value of each pixel point in the pseudo tumor distribution area in the Query. The final filling result Answer_warped is defined as the average weighted sum of Answer_pri and Answer_temp to retain the original organization part of Answer_pri. The calculation formula is as follows: Answer_warped(i,j)=0.5×Answer_pri(i,j)+0.5×Answer_temp(i,j) Wherein, (i, j) represents the coordinate value of each pixel point in the pseudo-tumor distribution area, and (i′, j′) represents the corresponding coordinate value of each pixel point in the pseudo-tumor distribution area in the Query; Answer_warped is further processed to obtain the final fake tumor slice, denoted as Answer, making it as similar as possible to the slice with a real tumor. Specifically, a generative adversarial network is used to reconstruct the image to eliminate the unnatural parts of Answer_warped. The network takes Answer_warped as input and outputs Answer. The network loss function is modified to include three parts: reconstruction loss, coherence loss, and adversarial loss. The entire loss function is defined as the weighted sum of reconstruction loss, coherence loss, and adversarial loss. The calculation formula is as follows: Loss=50×recLoss+50×cohLoss+advLoss Among them, recLoss represents reconstruction loss, cohLoss represents coherence loss, and advLoss represents adversarial loss; Specifically, the construction improves the Faster R-CNN deep network model, adds an image feature pyramid structure and a ConvGRU module, and modifies the data reading method during network training, specifically: Based on Faster R-CNN, the FPN structure is added to improve the feature extraction part of Faster R-CNN. Features are extracted in the form of image feature pyramids, fusing high-resolution shallow information with low-resolution but semantically rich deep information. After Faster R-CNN extracts features and before entering the RPN structure, a ConvGRU module is added to memorize the context information of the slice; The continuous slices in the MRI dataset are divided into groups of 3, and the upper and lower slices provide 3D context information for the middle slice. Then, during training, the data is read in groups to ensure that the sample data read each time is a continuous slice of a certain case.

2. A medical imaging brain tumor detection system based on improved Faster R-CNN, characterized by: The system comprises: a magnetic resonance image data set acquisition unit, configured to acquire a magnetic resonance image data set from a magnetic resonance image database; A data enhancement unit is used to preprocess the images of the acquired data set by using an optical flow method and a generative adversarial network, perform data enhancement, and form a data set obtained after data enhancement; Improved the Faster R-CNN model building unit, used to build an improved Faster R-CNN deep network model, adding an image feature pyramid structure and ConvGRU module, and modifying the data reading method during network training; An improved Faster R-CNN model training unit is used to train an improved Faster R-CNN model using the data set obtained after data enhancement to form a trained improved Faster R-CNN model; a target detection result output unit, configured to input the magnetic resonance image to be predicted into the trained improved FasterR-CNN model and output a brain tumor target detection result; Specifically, the data enhancement unit needs to find the normal slice sequence that is closest to the tumor slice in terms of brightness. The tumor slice is recorded as Query and the normal slice sequence is recorded as Normal. Specifically, for the slice sequence to which Query belongs, it is first evenly divided into 10 groups according to the arrangement order, and the mean and standard deviation of the pixel brightness of each group are calculated respectively to obtain a brightness value statistical vector of length 20. For all normal sequences without tumors in the data set, the corresponding statistical vectors of length 20 are calculated using the same method. Then, the Euclidean distance between the brightness value statistical vector of the Normal sequence and the brightness value statistical vector of the sequence to which the Query belongs is calculated. Among all Normal sequences, the one with the smallest distance between its brightness value statistical vector and the brightness value statistical vector of the sequence to which the Query belongs is the Normal sequence that is closest to the Query slice in terms of brightness. The Normal sequence that is closest to the Query slice in terms of brightness is operated to obtain the normal slice that is closest to the tumor slice in terms of brightness, which is recorded as Answer_pri. Specifically, first, the pre-trained ResNet50 model is used to extract features from the Query to obtain v q , extract features from each slice in the Normal sequence to get v i (i=1,2,…,n), then calculate v q With v i The cosine similarity between them is used to sort the slices in the Normal sequence from large to small according to the similarity, and the top 10 slices with the highest similarity are selected and recorded as N1, N2, ..., N 10 , and then use the RAFT algorithm to calculate the optical flow from the query to the slice, denoted as f1,f2,…,f 10 , and then use the optical flow to perform image twisting on the slice to obtain the twisted result N′ i , the image twist formula is as follows: N′ i (x′ i ,and' i )=N i (x i ,and i ) x′ i =x i -f i (x i ,y i ,0) and' i =and i -F i (x i ,and i ,1) Among them, N i Indicates the first 10 slices selected, N′ i represents the result after the slice is twisted, (x i ,y i ) represents N i The pixel coordinates, (x′ i ,y′ i ) represents N′ i The pixel coordinates, f i represents the optical flow from Query to the slice, i=1,2,…,10, Then for N′ i Perform histogram matching according to Query, so that N′ i The brightness distribution between the query and the query is similar to eliminate the influence of brightness on similarity judgment. Finally, N′ is calculated. i The mean square error of pixel brightness between Query and the minimum error N′ i That is Answer_pri; the tumor area is mapped on Answer_pri to obtain a preliminary pseudo-tumor slice, recorded as Answer_warped. Specifically, the optical flow from Query to Answer_pri is first calculated, recorded as f1. The position of the tumor in Query is marked by a bounding box. Using f1, according to the image twisting formula, the four vertex coordinates of the bounding box of the tumor position in Query are calculated to be mapped to the corresponding coordinates on Answer_pri, thereby determining the distribution area of the pseudo-tumor on Answer_pri. Then the optical flow from Answer_pri to Query is calculated, recorded as f2. Using f2, according to the image twisting formula, the corresponding coordinates (i′, j′) of each pixel point (i, j) in the pseudo-tumor distribution area in Query are found. According to the pixel values of the corresponding coordinates, the pixels in the pseudo-tumor distribution area are preliminarily interpolated using bilinear interpolation. Step 1 is used to fill in the data and obtain the preliminary filling value Answer_temp, so as to achieve the effect of "transplanting" the tumor from Query into Answer_pri. The final filling result Answer_warped is defined as the average weighted sum of Answer_pri and Answer_temp to retain the original tissue part in Answer_pri. Answer_warped is further processed to obtain the final fake tumor slice, recorded as Answer, so that it is as similar as possible to the slice with real tumor. Specifically, a generative adversarial network is used to reconstruct the image to eliminate the unnatural parts of Answer_warped. The network takes Answer_warped as input and outputs Answer. The network loss function is modified, including reconstruction loss, coherence loss, and adversarial loss. The entire loss function is defined as the weighted sum of reconstruction loss, coherence loss, and adversarial loss. Specifically, the improved Faster R-CNN model construction unit needs to add an FPN structure on the basis of Faster R-CNN, improve the feature extraction part of Faster R-CNN, extract features in the form of an image feature pyramid, and fuse shallow information with high resolution and deep information with low resolution but rich semantic information; after Faster R-CNN extracts features and before entering the RPN structure, a ConvGRU module is added to memorize the context information of the slice; the continuous slices in the MRI data set are divided into groups of 3, and the upper and lower slices provide 3D context information for the middle slice, and then the data is read in groups during training to ensure that the sample data read each time is a continuous slice in a certain case.

Images