A method and system for automatic segmentation of endoscopic tumor images based on BGNet

Abstract

This invention discloses an automatic segmentation method and system for endoscopic tumor images based on BGNet, applied in the field of medical image processing technology. It acquires images of endoscopic examinations and surgeries of tumor patients and preprocesses these images. A BGNet neural network model is trained to obtain an automatic segmentation model for endoscopic tumor images. Features at different levels are extracted, and combined with acquired boundary enhancement features, the background and the target to be segmented are decoded and predicted separately. The region of interest (ROI) is automatically segmented to obtain the final segmentation result. This invention can automatically segment the ROI of the tumor through feature learning of the BGNet neural network model, with less influence from the doctor's experience and higher accuracy. The ROI can be annotated in real time during surgery, preventing doctors from missing the resection of small tumors, thus improving efficiency.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more specifically to a method and system for automatic segmentation of endoscopic tumor images based on BGNet. Background Technology

[0002] Currently, endoscopy remains the primary method for examining bladder tumors. It can initially differentiate between benign and malignant tumors. Benign papillomas are easily identifiable; they have a clear stalk from which numerous finger-like or villous branches emerge, floating in water. The bladder mucosa surrounding the stalk is normal. If the tumor is sessile, has a broad base, and the surrounding bladder mucosa is uneven, thickened, or edematous and congested, appearing as short, irregular protrusions or a fist-sized mass, with ulceration, bleeding, and grayish-white purulent deposits on the surface, along with a small bladder capacity and cloudy, bloody flushed fluid, all these characteristics suggest the presence of a malignant tumor.

[0003] For superficial bladder tumors, transurethral resection (TURBt) or electrocautery can be used. Tube resection is indicated for tumors differentiated above the medullary canal, staged to T2 or lower, and with a diameter of less than 2 cm. Multiple tumors can be removed in stages. TURBt is an incision-free procedure that can be repeated, causing minimal trauma to the patient and resulting in rapid postoperative recovery. It is widely used both domestically and internationally and has almost completely replaced partial cystectomy.

[0004] In clinical practice, surgeons may miss removing small, easily overlooked tumors or minor lesions during surgery. Furthermore, the extent of resection varies among surgeons; experienced surgeons will remove as little normal area as possible while ensuring complete tumor removal, while less experienced surgeons may remove more normal area or incompletely remove lesions. Therefore, there is an urgent need for an automated endoscopic tumor image segmentation solution based on BGNet to assist surgeons in performing precise surgical treatments and improve their work efficiency. Summary of the Invention

[0005] In view of this, the present invention provides an automatic segmentation method and system for endoscopic tumor images based on BGNet to solve the problems in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] On the one hand, the automatic segmentation method for endoscopic tumor images based on BGNet has the following specific steps:

[0008] Acquire images of endoscopic examinations and surgical procedures in cancer patients, and preprocess the images;

[0009] The model is trained based on the BGNet neural network model to obtain an automatic segmentation model for endoscopic tumor images. Features at different levels are extracted and combined with the obtained boundary enhancement features to decode and predict the background and the target to be segmented respectively.

[0010] The region of interest is automatically segmented to obtain the final segmentation result.

[0011] Preferably, in the above-mentioned automatic segmentation method for endoscopic tumor images based on BGNet, images of tumor patients during endoscopic examinations and surgeries are acquired, and the images are preprocessed, specifically including the following steps:

[0012] The video of a cancer patient undergoing endoscopic examination and surgery is acquired and extracted frame by frame into image data.

[0013] The acquired image data is preprocessed, including data augmentation methods such as normalization, geometric transformation, and color transformation.

[0014] Preferably, in the above-mentioned automatic segmentation method for endoscopic tumor images based on BGNet, the step of normalizing the acquired image data during endoscopic examination and surgery to obtain normalized images specifically includes:

[0015] The acquired images from endoscopic examinations and surgical procedures are stored in a two-dimensional matrix;

[0016] Normalize the two-dimensional matrix;

[0017] Each normalized image from a two-dimensional endoscopic examination and surgical procedure is stored together with its corresponding gold standard in a designated file.

[0018] Preferably, in the above-mentioned automatic segmentation method for endoscopic tumor images based on BGNet, the step of performing geometric and color transformations on the acquired image data from the endoscopic examination and surgery process to obtain the amplified image specifically includes:

[0019] The images from the normalized endoscopic examination and surgical procedure were subjected to image flipping, cropping, rotation and translation, random color processing, brightness variation, and random noise addition.

[0020] The corresponding gold standard images are subjected to image flipping, cropping, rotation and translation, random color processing, brightness and darkness changes, and random noise processing.

[0021] Each image from an endoscopic examination and surgical procedure, after undergoing geometric and color transformation processing, is stored together with its corresponding gold standard image, also after geometric and color transformation processing, in a designated file.

[0022] Preferably, in the above-mentioned automatic segmentation method for endoscopic tumor images based on BGNet, the automatic segmentation of the region of interest using the automatic segmentation model for endoscopic tumor images specifically includes the following:

[0023] Acquire endoscopic images of cancer patients;

[0024] A deep learning method is used to automatically segment lesions in acquired endoscopic tumor images. The deep learning method adopts a BGNet neural network model. By utilizing local features at different levels, the receptive field is expanded to obtain global features. These global features are combined with boundary information, and global features at different levels are fused and decoded and predicted for the background and the target to be segmented, respectively, to obtain the lesion segmentation result of the endoscopic tumor image.

[0025] Preferably, in the above-mentioned automatic segmentation method for endoscopic tumor images based on BGNet, the BGNet-based neural network includes the following four parts: CNN module, P-ViT module, BETM module and BITM module.

[0026] The CNN module uses a resnest network to extract features at different levels. The resnest network produces five outputs of different sizes, from shallow to deep, namely x1, x2, x3, x4, and x5.

[0027] The P-ViT module uses a parallel ViT network to assist the CNN module in capturing long-range feature dependencies. It processes x2, x3, and x4 separately and makes skip connections between the output and x2, x3, and x4, with outputs p2, p3, and p4 respectively.

[0028] The BETM module is used to obtain the boundary information of the image to be segmented. The obtained boundary enhancement features guide the decoding process. After performing three convolutions, BN, and ReLU operations on x5, it is upsampled and added to x2 for fusion. Then, the fused result is subjected to three convolutions, BN, and ReLU operations. Finally, it is subjected to a convolution with a dimension of 1 to obtain the image boundary information Ob.

[0029] The BITM module is divided into two branches, which decode and predict the background and the target to be segmented, respectively. The output of x5 in the BETM module after three convolutions, BN, and ReLU operations is concatenated with the outputs p2 and p3 in the P-ViT module after downsampling to obtain the output BI5. Then, it is processed by positional attention and channel attention respectively and added together. This result is concatenated with Ob and p4 and used as the input of the BITM module, which enters the two branches respectively. The upper branch performs foreground segmentation and recognition, and the lower branch performs background segmentation and recognition. The outputs of the upper and lower branches are added together with the upsampled BI5 to obtain the output BI4. BI4 is processed in the same way as BI5 to obtain BI3. BI3 is processed to obtain BI2. The output of BI2 after a one-dimensional convolution is the final segmentation result.

[0030] Preferably, in the above-mentioned automatic segmentation method for endoscopic tumor images based on BGNet, the specific steps for training an automatic segmentation model for endoscopic tumor images using a BGNet neural network model based on given samples are as follows:

[0031] The given samples are divided into 80% training set and 20% test set;

[0032] The training set is input into the BGNet neural network model for training, resulting in a model for automatic segmentation of endoscopic tumor images;

[0033] The obtained model was tested using a test set, and the parameters of the BGNet neural network model were adjusted based on the test performance.

[0034] The training set is input into the parameter-adjusted BGNet neural network model for training, resulting in an automatic segmentation model for endoscopic tumor images.

[0035] The gold standard image is compared with the predicted result to calculate the final loss function. The parameters are continuously adjusted and the model is trained and tested until the best-performing model for automatic segmentation of endoscopic tumor images is obtained.

[0036] On the other hand, the segmentation system corresponding to the automatic segmentation method for endoscopic tumor images based on BGNet includes:

[0037] The acquisition and preprocessing module acquires images of endoscopic examinations and surgical procedures in tumor patients and performs image preprocessing.

[0038] The region of interest segmentation module trains the BGNet neural network model to obtain an automatic segmentation model for endoscopic tumor images. It extracts features at different levels and combines the acquired boundary enhancement features to decode and predict the background and the target to be segmented.

[0039] The display module automatically segments the region of interest to obtain the final segmentation result.

[0040] Preferably, in the above-mentioned automatic segmentation system for endoscopic tumor images based on BGNet, the region of interest segmentation module includes: a CNN module, a P-ViT module, a BETM module, and a BITM module.

[0041] The CNN module uses a resnest network to extract features at different levels. The resnest network produces five outputs of different sizes, from shallow to deep, namely x1, x2, x3, x4, and x5.

[0042] The P-ViT module uses a parallel ViT network to assist the CNN module in capturing long-range feature dependencies. It processes x2, x3, and x4 separately and makes skip connections between the output and x2, x3, and x4, with outputs p2, p3, and p4 respectively.

[0043] The BETM module is used to obtain the boundary information of the image to be segmented. The obtained boundary enhancement features guide the decoding process. After performing three convolutions, BN, and ReLU operations on x5, it is upsampled and added to x2 for fusion. Then, the fused result is subjected to three convolutions, BN, and ReLU operations. Finally, it is subjected to a convolution with a dimension of 1 to obtain the image boundary information Ob.

[0044] The BITM module is divided into two branches, which decode and predict the background and the target to be segmented, respectively. The output of x5 in the BETM module after three convolutions, BN, and ReLU operations is concatenated with the outputs p2 and p3 in the P-ViT module after downsampling to obtain the output BI5. Then, it is processed by positional attention and channel attention respectively and added together. This result is concatenated with Ob and p4 and used as the input of the BITM module, which enters the two branches respectively. The upper branch performs foreground segmentation and recognition, and the lower branch performs background segmentation and recognition. The outputs of the upper and lower branches are added together with the upsampled BI5 to obtain the output BI4. BI4 is processed in the same way as BI5 to obtain BI3. BI3 is processed to obtain BI2. The output of BI2 after a one-dimensional convolution is the final segmentation result.

[0045] Preferably, in the above-mentioned automatic segmentation system for endoscopic tumor images based on BGNet, the region of interest segmentation module further includes a training module;

[0046] The given samples are divided into units, with 80% of the training set and 20% of the test set.

[0047] The training unit takes the training set as input and trains it based on the BGNet neural network model to obtain a model for automatic segmentation of endoscopic tumor images.

[0048] The first adjustment unit tests the obtained model using a test set and adjusts the parameters of the BGNet neural network model based on the test performance. The training set is then input into the adjusted BGNet neural network model for training, resulting in an automatic segmentation model for endoscopic tumor images.

[0049] The second adjustment unit compares the gold standard image with the prediction results to calculate the final loss function, continuously adjusting parameters and training and testing the model until the optimal model for automatic segmentation of endoscopic tumor images is obtained.

[0050] As can be seen from the above technical solutions, compared with the prior art, the present invention discloses a method and system based on the BGNet neural network model, which automatically segments the region of interest (ROI) of the acquired endoscopic tumor images. By combining the BGNet neural network model and endoscopic tumor images for ROI segmentation, the ROI of the tumor can be automatically segmented through feature learning of the BGNet neural network model, which is less affected by the doctor's experience and is more accurate. The ROI can be marked in real time during the operation, avoiding the doctor from missing the resection of small tumors, thus improving efficiency. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0052] Figure 1 A flowchart of an automatic segmentation method for endoscopic tumor images based on a BGNet neural network model provided in an embodiment of the present invention;

[0053] Figure 2 This is a flowchart of an automatic segmentation scheme according to a specific embodiment of the present invention;

[0054] Figure 3 Endoscopic images of bladder tumors;

[0055] Figure 4 for Figure 3 The corresponding gold standard;

[0056] Figure 5 The results are based on the BGNet neural network model for segmentation.

[0057] Figure 6 This is a structural diagram of a specific embodiment of the present invention based on the BGNet neural network model;

[0058] Figure 7 This is a structural diagram of the P-ViT module according to a specific embodiment of the present invention;

[0059] Figure 8 This is a structural diagram of the BITM module in a specific embodiment of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] Reference Figure 1 This invention provides an automatic segmentation method for endoscopic tumor images based on the BGNet neural network model, comprising the following steps:

[0062] Acquire images of endoscopic examinations and surgical procedures in cancer patients, and preprocess the images;

[0063] The model is trained based on the BGNet neural network model to obtain an automatic segmentation model for endoscopic tumor images. Features at different levels are extracted and combined with the obtained boundary enhancement features to decode and predict the background and the target to be segmented respectively.

[0064] The region of interest is automatically segmented to obtain the final segmentation result.

[0065] This invention is the first to apply the BGNet neural network model to endoscopic tumor images for lesion segmentation. It can automatically segment the region of interest of the tumor through feature learning of the BGNet neural network model, thereby assisting doctors in accurately segmenting the region of interest of endoscopic tumors. It is less affected by the doctor's experience, more accurate, and can mark the tumor area in real time during the operation, avoiding the doctor from missing the removal of small tumors, thus improving efficiency.

[0066] As a further preferred embodiment, the step of acquiring images during endoscopic examination or surgery of tumor patients specifically includes:

[0067] The video of a cancer patient undergoing endoscopic examination and surgery is acquired and extracted frame by frame into image data.

[0068] The acquired image data is preprocessed, including data augmentation methods such as normalization, geometric transformation, and color transformation.

[0069] Specifically, this embodiment can acquire endoscopic image data of tumor patients during preoperative examinations or surgical procedures. Based on prior knowledge and actual clinical conditions, the endoscopic images can reveal richer tumor information and details.

[0070] Normalization is used to limit the acquired image data to a certain range (such as [0,1] or [-1,1]), thereby eliminating the adverse effects caused by outlier data.

[0071] Geometric transformation of an image is the operation of mapping one set of image data to another set of image data through certain mathematical operations. Therefore, the key to geometric transformation is to determine this spatial mapping relationship.

[0072] Geometric transformation is the process of mapping the coordinate positions in one image to new coordinate positions in another image. Performing geometric transformations on images can, to some extent, eliminate geometric distortions caused by angles, perspective, shooting methods, etc., which can prevent computer models or algorithms from correctly recognizing the images.

[0073] Color transformation is the process of mapping the brightness, saturation, and other values ​​of each pixel in an image to another corresponding value at the same location. Color transformation of an image can alleviate color differences caused by different sensors and can also artificially add noise to enhance the robustness of the model.

[0074] As a further preferred embodiment, the step of preprocessing the acquired endoscopic tumor image data specifically includes:

[0075] The images from the normalized endoscopic examination and surgical procedure were subjected to image flipping, cropping, rotation and translation, random color processing, brightness variation, and random noise addition.

[0076] The corresponding gold standard was also processed in the same way.

[0077] Each image from an endoscopic examination and surgical procedure, after undergoing geometric and color transformation processing, is stored together with its corresponding gold standard image, also after geometric and color transformation processing, in a designated file.

[0078] As a further preferred embodiment, the step of normalizing the acquired endoscopic tumor image data to obtain a normalized endoscopic tumor image specifically includes:

[0079] The acquired images from endoscopic examinations and surgical procedures are stored in a two-dimensional matrix;

[0080] Normalize the two-dimensional matrix;

[0081] Each normalized image from a two-dimensional endoscopic examination and surgical procedure is stored together with its corresponding gold standard in a designated file.

[0082] As a further preferred embodiment, the deep learning method employs a BGNet-based neural network model, which includes the following four parts: a CNN module, a P-ViT module, a BETM module, and a BITM module.

[0083] The CNN module uses a resnest network to extract features at different levels. The resnest network produces five outputs of different sizes, from shallow to deep, namely x1, x2, x3, x4, and x5.

[0084] The P-ViT module uses a parallel ViT network to assist the CNN module in capturing long-range feature dependencies. It processes x2, x3, and x4 separately and makes skip connections between the output and x2, x3, and x4, with outputs p2, p3, and p4 respectively.

[0085] The BETM module is used to obtain the boundary information of the image to be segmented. The obtained boundary enhancement features can be used to guide the decoding process. After performing three convolutions, BN, and ReLU operations on x5, it is upsampled and added to x2 for fusion. Then, the fused result is subjected to three convolutions, BN, and ReLU, and then a convolution with dimension 1 is performed to obtain the image boundary information Ob.

[0086] The BITM module is divided into two branches, one for decoding and predicting the background and the other for the target to be segmented. The output of x5 from the BETM module, after undergoing three convolutions, BN, and ReLU operations, is downsampled and concatenated with the outputs p2 and p3 from the P-ViT module to obtain output BI5. This BI5 is then processed with positional attention and channel attention respectively and fused together. This result is concatenated with Ob and p4 and used as input to the BITM module, entering the two branches: the upper branch performs foreground segmentation and recognition, and the lower branch performs background segmentation and recognition. The outputs of both branches are then fused with the upsampled BI5 to obtain output BI4. BI4 undergoes the same processing as BI5 to obtain BI3. BI3 is further processed to obtain BI2. BI2 undergoes a single 1-dimensional convolution to obtain o2, which is the final segmentation result.

[0087] The boundary-guided decoding module includes three complete BITM modules and subsequent convolutional layers;

[0088] Specifically, this embodiment uses a BGNet neural network model to build a network for automatic segmentation of endoscopic tumor images. Directly performing convolution operations on images may not effectively learn global features, instead focusing on local features. The P-ViT module provides a larger receptive field, assisting the CNN in capturing global features. The BGNet neural network model is primarily used for semantic image segmentation, with the BETM module's main function being to refine the segmentation boundaries. This is because typical semantic images contain many categories, and the boundaries of each category are often not simple rectangles or circles, but complex. Therefore, using the BETM module can improve the segmentation results to some extent. The BITM module can fuse information from different levels and decode and predict the background and the target to be segmented separately, contributing to improved segmentation accuracy.

[0089] As a further preferred embodiment, the step of automatically segmenting the region of interest in the acquired endoscopic tumor image using a method based on the BGNet neural network model specifically includes:

[0090] An automatic segmentation model for endoscopic tumor images is trained using a BGNet neural network model based on given samples.

[0091] The acquired endoscopic tumor images are preprocessed and then input into the endoscopic tumor image automatic segmentation model to obtain the region of interest segmentation results of the endoscopic tumor.

[0092] Specifically, the given sample can be actual data from endoscopic tumor patients obtained clinically.

[0093] As a further preferred embodiment, the step of training an automatic segmentation model for endoscopic tumor images based on a BGNet neural network model using given samples specifically includes:

[0094] The given samples are divided into 80% training set and 20% test set;

[0095] The training set is input into the BGNet neural network model for training, resulting in a model for automatic segmentation of endoscopic tumor images;

[0096] The obtained model was tested using a test set, and the parameters of the BGNet neural network model were adjusted based on the test performance.

[0097] The training set is input into the parameter-adjusted BGNet neural network model for training, resulting in an automatic segmentation model for endoscopic tumor images.

[0098] The parameters were continuously adjusted and the model was trained and tested until the optimal model for automatic segmentation of endoscopic tumor images was obtained.

[0099] As a further preferred embodiment, the step of inputting the training set into the BGNet neural network model for model training to obtain a segmentation model for automatic segmentation of endoscopic tumor images specifically includes:

[0100] After processing the endoscopic tumor images in the training set sequentially, the boundary output Ob is obtained in the BETM module;

[0101] The feature map BI4 obtained from the BITM module can be obtained by performing convolution processing to obtain the segmentation result o4.

[0102] The obtained BI4 is processed by the BITM module to obtain the feature map BI3, and after convolution processing, the segmentation result o3 can be obtained.

[0103] The obtained BI3 is processed by the BITM module to obtain the feature map BI2, and after convolution processing, the segmentation result o2 can be obtained.

[0104] The obtained output result o2 is used as the final output result for automatic segmentation of endoscopic tumor images.

[0105] Specifically, the training set contains more than one endoscopic tumor image, so the processing through each module is iterative until all endoscopic tumor images in the training set have been trained.

[0106] Taking bladder tumor resection surgery as an example, this specific embodiment addresses the problem that existing bladder tumor resection surgeries rely heavily on the surgeon's experience, potentially missing small or low-grade bladder tumors during the procedure. It utilizes deep learning technology to design an automatic segmentation system based on the BGNet neural network model for endoscopic bladder tumor lesions, reducing the surgeon's workload and assisting in precise lesion segmentation. Figure 2 As shown, the specific embodiment consists of the following two parts: 1) collection and data preprocessing of endoscopic images of bladder tumor patients; 2) segmentation of endoscopic bladder tumor lesions. These two parts are described in detail below:

[0107] Part 1: Collection and Data Preprocessing of Endoscopic Images from Patients with Bladder Tumors

[0108] This specific embodiment can collect endoscopic video data during examinations or surgeries of bladder cancer patients, extract the data frame by frame, and filter out valid images, such as... Figure 3 As shown, the size of each image varies depending on the endoscope model; we have selected images of various sizes. The gold standard (Groundtruth, GT) for endoscopic bladder tumor lesions is manually drawn by experienced radiologists. The drawn gold standard is shown in the image below. Figure 4As shown. Because tumors vary in size, the number of tumors contained in each image is also different. Therefore, this specific embodiment selects endoscopic images containing tumors of different sizes and numbers for lesion segmentation.

[0109] The data preprocessing process is as follows:

[0110] (1) Normalization: The endoscopic bladder tumor images are stored in a two-dimensional matrix. Then, each element of the two-dimensional matrix is ​​normalized by the difference between the maximum and minimum values ​​of the matrix. The intensity of the original image is set between [0, 1]. Finally, each two-dimensional endoscopic tumor image and its corresponding gold standard are saved to a file.

[0111] (2) Geometric Transformation: Since bladder tumors vary in location, size, and shape within images, geometric transformations of the original image can enhance the robustness of the model. Geometric transformations include image flipping, cropping, rotation, and translation. Image flipping includes vertical and horizontal flipping; cropping involves randomly cutting different regions of the image, generating many new images with different positional information; rotation involves rotating the original image 90 degrees clockwise and 90 degrees counterclockwise; and translation involves moving the original image vertically or horizontally.

[0112] (3) Color Transformation: During surgery, bladder tumor images acquired endoscopically may have different color spaces due to lighting conditions and differences in sensor parameters. Therefore, color transformation of the original images is necessary. The color transformations performed include random color processing, brightness variation, and the addition of random noise. Random color processing includes swapping RGB channels, processing and mapping the values ​​of each channel to other values, etc. Brightness variation includes enhancing or reducing the brightness of the original image. Adding random noise includes randomly adding salt-and-pepper noise or Gaussian noise to the original image to improve the model's anti-interference ability and enhance its robustness.

[0113] (4) Finally, perform the same operation on the gold standard, resize the processed image and the gold standard to the same 256*256 size to facilitate model training and reduce the GPU cache usage.

[0114] Part Two: Endoscopic Division of Bladder Tumor Lesions

[0115] This specific embodiment proposes a BGNet neural network model and applies it to the segmentation of endoscopic bladder tumor lesions. Directly performing convolution operations on images may not effectively learn global features, focusing instead on local features. The P-ViT module, however, provides a larger receptive field, assisting the CNN in capturing global features. The BGNet network model is primarily used for semantic image segmentation, with the BETM module's main function being to refine the segmentation boundaries. This is because typical semantic images contain many categories, and the boundaries of each category are often not simple rectangles or circles, exhibiting complexity. Therefore, using the BETM module can improve the segmentation results to some extent. The BITM module can fuse information at different levels and decode and predict the background and the target to be segmented separately, contributing to improved segmentation accuracy. The specific network structure is as follows: Figure 6 As shown. The BGNet-based network model comprises four parts: a CNN module, a P-ViT module, a BETM module, and a BITM module. Specifically: the CNN module uses a Resnest network to extract features at different levels. The Resnest network produces five outputs of different sizes, x1, x2, x3, x4, and x5 from shallow to deep layers. The P-ViT module uses a parallel ViT network to assist the CNN module in capturing long-range feature dependencies. 4 By dividing a 64*64 x2 area into equal parts of 4, 8, 16, and 32, sub-maps of sizes 16*16, 8*8, 4*4, and 2*2 are generated; similarly, by dividing a 32*32 x3 area into equal parts of 4, 8, 16, and 32, sub-maps of sizes 8*8, 4*4, 2*2, and 1*1 are generated; P-ViT 2A 16*16 image is divided into sub-images of size 4*4 and 2*2 by dividing its length and width into 4 and 8 equal parts respectively. After processing x2, x3, and x4 respectively, the outputs are skip-connected with x2, x3, and x4, resulting in outputs p2, p3, and p4 respectively. The BETM module is used to obtain the boundary information of the image to be segmented. The obtained boundary enhancement features can be used to guide the decoding process. x5 is subjected to three convolutions, BN, and ReLU operations, then upsampled and added to x2 for fusion. The fused result is then subjected to three more convolutions, BN, and ReLU operations, and finally a single convolution of dimension 1 to obtain the image boundary information Ob. The BITM module is divided into two branches, one for decoding and predicting the background and the target to be segmented, respectively. The output of x5 from the BETM module, after undergoing three convolutions, BN, and ReLU operations, is downsampled and concatenated with the outputs p2 and p3 from the P-ViT module to obtain output BI5. This BI5 is then processed by PositionAttention and ChannelAttention and fused together. This fused result is concatenated with Ob and p4 and used as input to the BITM module, entering the two branches: the upper branch performs foreground segmentation and recognition, and the lower branch performs background segmentation and recognition. The outputs of both branches are then fused with the upsampled BI5 to obtain output BI4. BI4 undergoes the same processing as BI5 to obtain BI3. BI3 is then processed to obtain BI2. BI2 undergoes a single 1-dimensional convolution to obtain o2, which is the final segmentation result. Finally, the segmentation result is restored to the original image size. The final output is shown below. Figure 5 As shown.

[0116] In Adoption Figure 6 Before training a network, the data needs to be divided into training and test sets. Approximately 80% of the image data can be randomly selected as the training set, and the remaining image data as the test set.

[0117] First, all data is normalized. During the training phase, the training set data undergoes data augmentation (specifically the "geometric transformation" and "color transformation" described in Part 1) and is used as input to the network. The network's base learning rate (lr) is set to 0.0001, and the loss function used is the weighted cross-entropy loss function Li. CE Weighted miou loss L miou And Dice's loss L Dice They are shown in the following formulas:

[0118]

[0119] The final loss function is calculated as follows:

[0120] Where L=3 indicates that the loss is calculated using the result of one convolution of the outputs of the three BITM modules. i This represents the weights, which are assigned different weight values ​​based on their distance from the boundary, ranging from 5 to 1. N is the number of samples in the training set. i Refers to the gold standard, Refers to the model's predicted value.

[0121] In machine learning, a loss function is a non-negative function used to measure the quality of a trained model's prediction in a single iteration. Cross-entropy loss increases the stability of the trained model and avoids slow learning. MIOU loss is more robust to imbalanced samples; in cases of class imbalance, MIOU loss is more robust than cross-entropy loss and less likely to be overly biased towards certain classes. Dice loss is used to calculate the boundary information (Ob) of the segmented data, allowing the model to learn boundary information better. The optimizer is Adam, also known as Adaptive Moment Estimation. It's an optimization algorithm that updates weights and biased parameters by minimizing (or maximizing) the loss function during network model training, bringing the model parameters closer to their optimal values. The augmented training set was iterated 100 times on the BGNet neural network model, saving the model weights for each round. In the testing phase, this specific embodiment uses the trained segmentation model to segment endoscopic bladder tumor lesions on the test set, and the segmentation results are evaluated against the gold standard. This specific embodiment uses the ratio of intersection to union (MIOU), the mean average precision (mAP), and F. β The segmentation results are evaluated using three metrics: Hausdorff distance (HD).

[0122] The Mean Intersection over Union (MIOU) ratio is a standard metric for semantic segmentation. It is calculated as the average ratio of the intersection to the union of all categories, using the following formula:

[0123]

[0124] The mean average precision (mAP) is used to measure the segmentation quality of multi-class samples, and the formula is as follows:

[0125] in:

[0126] F β It is a configurable single-class metric used to evaluate binary classification models based on predictions of the positive class. It is calculated using precision and recall, and the calculation formula is as follows:

[0127]

[0128] Here, β is set to 2.

[0129] Hausdorff distance (HD) is a measure of the similarity between two sets of points. It is a definition of the distance between two sets of points and is sensitive to the boundary of the segmentation, so it can better measure the segmentation boundary.

[0130] TP: True Positive, the number of positive samples correctly identified when the classifier predicts a positive sample. FP: False Positive, the number of falsely reported negative samples when the classifier predicts a positive sample. TN: True Negative, the number of falsely reported negative samples when the classifier predicts a negative sample. FN: False Negative, the number of falsely reported positive samples when the classifier predicts a negative sample. Classes refers to all categories, specifically background and the target to be segmented. AP calculates the average precision under equal-interval recall, describing the overall precision under different recall levels. N refers to the number of equal-interval recalls, and p1, p2, ..., pn refer to the equal-interval recalls. Precision, also called accuracy, represents the proportion of samples predicted as positive that are actually positive. Recall, also known as the full recall rate, represents the proportion of actual positive samples out of the total number of positive samples in the whole sample.

[0131] The method described in this specific embodiment was used to automatically segment 17,138 endoscopic bladder tumor images. The miou (mean squared accuracy) was 91.22%, mAP (maximum accuracy) was 85.19%, and F (mean squared accuracy) was [not specified in the original text]. β The accuracy is 94.53%, and the HD is 3.90. Therefore, it can be seen that the automatic segmentation method of this specific embodiment has high accuracy and excellent boundary segmentation effect.

[0132] In summary, this specific embodiment, based on the BGNet neural network model and endoscopic bladder tumor images, designs an automatic segmentation method for endoscopic bladder tumors using the BGNet neural network model. This method learns lesion features using the BGNet neural network model and automatically segments the tumor lesion area, making it less affected by the doctor's experience and more accurate. Furthermore, it can mark the tumor area in real time during surgery, preventing doctors from missing the removal of small tumors and improving efficiency.

[0133] and Figure 1Corresponding to the method, this embodiment of the invention also provides an automatic endoscopic bladder tumor segmentation system based on the BGNet neural network model, comprising:

[0134] The acquisition and preprocessing module acquires images of endoscopic examinations and surgical procedures in tumor patients and performs image preprocessing.

[0135] The region of interest segmentation module trains the BGNet neural network model to obtain an automatic segmentation model for endoscopic tumor images. It extracts features at different levels and combines the acquired boundary enhancement features to decode and predict the background and the target to be segmented.

[0136] The display module automatically segments the region of interest to obtain the final segmentation result.

[0137] This invention also provides an automatic segmentation system for endoscopic tumor images based on the BGNet neural network model, comprising:

[0138] At least one processor;

[0139] At least one memory for storing at least one program;

[0140] When the at least one program is executed by the at least one processor, the at least one processor implements the automatic segmentation method for endoscopic tumor images based on the BGNet neural network model described in this invention.

[0141] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0142] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for automatic segmentation of endoscopic tumor images based on BGNet, characterized in that, The specific steps are as follows: Obtain images of a tumor patient in an endoscopic examination and a surgical process, and pre-process the images; Based on a BGNet neural network model, model training is performed to obtain an endoscopic tumor image automatic segmentation model, different levels of features are extracted, boundary enhancement features are obtained, and the background and the target to be segmented are decoded and predicted respectively; An interested region is automatically segmented to obtain a final segmentation result; The BGNet neural network comprises the following four parts: a CNN module, a P-ViT module, a BETM module and a BITM module; The CNN module adopts a resnest network to extract different levels of features, and the resnest network generates five outputs of different sizes, from a shallow layer to a deep layer, x1, x2, x3, x4 and x5; The P-ViT module adopts a parallel ViT network to assist the CNN module in capturing long-range feature dependencies, processes x2, x3 and x4, and performs jump connection on the output results and x2, x3 and x4, and outputs p2, p3 and p4 respectively; The BETM module is used to obtain boundary information of the target to be segmented; the boundary enhancement features obtained are used to guide the decoding process, x5 is subjected to three convolution operations, BN and Relu operations, and then is added and fused with x2 after being up-sampled, and then the fusion result is subjected to three convolution operations, BN, Relu, and finally is subjected to a convolution operation with a dimension of 1 to obtain the boundary information Ob of the target to be segmented; The BITM module is divided into an upper branch and a lower branch, and is used to decode and predict the background and the target to be segmented; the output of x5 in the BETM module after the three convolution operations, BN and Relu operations is respectively down-sampled and spliced with the output results p2 and p3 in the P-ViT module to obtain an output BI5, and then the output BI5 is subjected to position attention and channel attention processing and added and fused, and the fusion result is spliced with Ob and p4 to serve as the input of the BITM module and enter the two branches respectively, the upper branch is used for segmentation and identification of the foreground, and the lower branch is used for segmentation and identification of the background, the output results of the upper branch and the lower branch are added and fused with the up-sampled BI5 to obtain an output BI4, and the BI4 is subjected to the same processing as the BI5 to obtain a BI3, and the BI3 is processed to obtain a BI2; the output of the BI2 after a convolution operation with a dimension of 1 is the final segmentation result. 2.The BGNet-based endoscopic tumor image automatic segmentation method according to claim 1, characterized in that, Obtain images of a tumor patient in an endoscopic examination and a surgical process, and pre-process the images, which specifically includes the following steps: Obtain video data of a tumor patient in an endoscopic examination and a surgical process, and frame the video data into image data; Pre-process the obtained image data, and the pre-processing includes normalization, geometric transformation and color transformation data augmentation methods. 3.The BGNet-based endoscopic tumor image automatic segmentation method according to claim 2, characterized in that, The step of normalizing the obtained image data to obtain normalized images specifically includes: Store the obtained images in an endoscopic examination and a surgical process in a two-dimensional matrix; Normalize the two-dimensional matrix; The normalized image of each two-dimensional endoscopy and surgery process is stored in a designated file together with the corresponding gold standard image. 4.The BGNet-based endoscopic tumor image automatic segmentation method according to claim 3, characterized in that, The data augmentation method of geometric transformation and color transformation on the obtained image data of endoscopy and surgery process, to obtain the augmented image, specifically includes: The normalized image of endoscopy and surgery process is respectively subjected to image flipping, cropping, rotation and translation, random color processing, brightness change and random noise processing; The corresponding gold standard image is respectively subjected to image flipping, cropping, rotation and translation, random color processing, brightness change and random noise processing; The image of each endoscopy and surgery process after geometric transformation and color transformation is stored in a designated file together with the corresponding gold standard image after geometric transformation and color transformation. 5.The BGNet-based endoscopic tumor image automatic segmentation method according to claim 1, wherein, The automatic segmentation of the region of interest using the endoscopic tumor image automatic segmentation model specifically includes the following: Obtain the endoscopic tumor image of the tumor patient; The deep learning method is used to automatically segment the obtained endoscopic tumor image, and the deep learning method uses a BGNet neural network model; different levels of local features are used to expand the receptive field and obtain global features; the global features are combined with boundary information, different levels of global features are fused, and the background and the target to be segmented are decoded and predicted respectively to obtain the lesion segmentation result of the endoscopic tumor image. 6.The BGNet-based endoscopic tumor image automatic segmentation method according to claim 1, wherein, Model training based on the BGNet neural network model to obtain an endoscopic tumor image automatic segmentation model, the specific steps are as follows: Divide the given sample into 80% training set and 20% test set; The training set is input into the BGNet neural network model for training to obtain a model for automatic segmentation of endoscopic tumor images; Test the obtained model using the test set, and adjust the parameters of the BGNet neural network model according to the test performance; The training set is input into the BGNet neural network model with adjusted parameters for training to obtain an endoscopic tumor image automatic segmentation model; Compare the gold standard image with the prediction result to calculate the final loss function, and continuously adjust the parameters and train and test the model until the optimal endoscopic tumor image automatic segmentation model is obtained.

7. The BGNet-based endoscopic tumor image automatic segmentation method according to any one of claims 1-6, wherein the BGNet-based endoscopic tumor image automatic segmentation method corresponds to a segmentation system. It includes: An acquisition and preprocessing module acquires images of endoscopy and surgery of tumor patients and pre-processes the images; An interest region segmentation module trains a model based on a BGNet neural network model to obtain an endoscopic tumor image automatic segmentation model, extracts different levels of features, combines the obtained boundary enhanced features, and decodes and predicts the background and the target to be segmented respectively; A display module automatically segments the region of interest to obtain the final segmentation result.

8. The segmentation system of claim 7, wherein, The interest region segmentation module includes a CNN module, a P-ViT module, a BETM module and a BITM module; Wherein: the CNN module adopts the resnest network to extract features at different levels, and the resnest network will generate five outputs of different sizes, from shallow to deep, x1, x2, x3, x4, x5; The P-ViT module adopts a parallel ViT network to assist the CNN module in capturing long-range feature dependencies, which processes x2, x3, and x4, respectively, and outputs p2, p3, and p4 by performing skip connection with x2, x3, and x4; The BETM module is used to obtain the boundary information of the image to be segmented; the obtained boundary enhanced features are used to guide the decoding process, and x5 is subjected to three convolution, BN, and Relu operations, then up-sampling and added to x2 for fusion, and then the fusion result is subjected to three convolution, BN, Relu, and finally a convolution with a dimension of 1 to obtain the image boundary information Ob; The BITM module has two branches, one for decoding and predicting the background and the other for decoding and predicting the target to be segmented. The output of x5 in the BETM module is subjected to three convolution, BN, and Relu operations, and then down-sampled and spliced with the output results p2 and p3 in the P-ViT module to obtain the output BI5, which is then processed by position attention and channel attention and added to obtain the fusion result, which is spliced with Ob and p4 to serve as the input of the BITM module and enter the two branches. The upper branch performs foreground segmentation and identification, and the lower branch performs background segmentation and identification. The output results of the upper and lower branches are added to the up-sampled BI5 to obtain the output BI4. BI4 is processed in the same way as BI5 to obtain BI3, and BI3 is processed to obtain BI2. The output of BI2 obtained by a convolution with a dimension of 1 is the final segmentation result.

9. The segmentation system of claim 7, wherein, The region of interest segmentation module further includes a training module; The division unit divides the given sample into a training set of 80% and a test set of 20%; The training unit inputs the training set into the BGNet neural network model for training to obtain a model for automatic segmentation of endoscopic tumor images; The first adjustment unit tests the obtained model using the test set, adjusts the parameters of the BGNet neural network model according to the test performance, and inputs the training set into the BGNet neural network model with adjusted parameters for training to obtain an endoscopic tumor image automatic segmentation model; The second adjustment unit compares the gold standard image with the prediction result to calculate the final loss function, and continuously adjusts the parameters and trains and tests the model until an optimal endoscopic tumor image automatic segmentation model is obtained.

Images