Introverted papilloma recognition system, electronic equipment and storage medium

Abstract

The invention relates to a nasal introversion papilloma recognition system, electronic equipment and a storage medium. The system comprises a data preprocessing module used for carrying out standardization processing on an input first nasal endoscope image and a first T2W-MRI image to obtain a second nasal endoscope image and a second T2W-MRI image; the image segmentation model is used for carrying out segmentation processing on the second nasal endoscopic image and the second T2W-MRI image and marking an introversive papilloma area; the image classification module is used for carrying out classification processing on the second nasal endoscope image and the second T2W-MRI image marked with the nasal introversive papilloma area to obtain a first prediction classification result and a second prediction classification result respectively; and the fusion module is used for fusing the first prediction classification result and the second prediction classification result to obtain an image recognition result of the introversive papilloma of the target object. In the segmentation and classification stage, the prediction result of the endoscope model is combined with the prediction result of the T2W-MRI model, and the accuracy of combined prediction of the two modal images is evaluated through a logistic regression method.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic equipment, and in particular to an inverted nasal papilloma identification system, electronic equipment and a storage medium. Background Art

[0002] Inverted papilloma (SIP) is the most common benign tumor type in the nasal cavity and paranasal sinus region, accounting for 7% of all primary tumors of this type. Of particular importance is that approximately 7% to 11% of SIP patients are at risk of malignant transformation to squamous cell carcinoma (SIP-SCC). Given the significant differences in treatment strategies between SIP and SIP-SCC and the poor prognosis of SIP-SCC, accurate differentiation between the two is crucial in treatment decisions. Treatment of SIP-SCC often requires more extensive surgical resection and may be accompanied by postoperative adjuvant radiotherapy or chemoradiotherapy. However, the high local recurrence rate of SIP-SCC is partly attributed to preoperative misdiagnosis, which results in some patients failing to receive appropriate treatment options. Therefore, it is extremely important to accurately distinguish SIP from SIP-SCC in the preoperative stage. Although preoperative puncture biopsy is currently regarded as the gold standard for diagnosis, its limitations cannot be ignored, including lack of accuracy, sampling error, and trauma to patients. In addition, medical image analysis that relies on the human eye also has its limitations, such as the difficulty in fully capturing key information in the image and being easily affected by the evaluator's personal experience and subjective judgment. In view of this, it is particularly urgent to develop a technical solution that can accurately identify images collected related to inverted papilloma of the nose. Such technology can not only improve the accuracy of diagnosis and reduce the risk of misdiagnosis, but also provide patients with more personalized and accurate treatment plans, thereby improving the overall treatment effect and the patient's quality of life. Summary of the invention

[0003] The purpose of the present application is to provide an inverted nasal papilloma identification system, an electronic device and a storage medium, which can realize a solution for accurately identifying images collected related to inverted nasal papilloma.

[0004] According to a first aspect of an embodiment of the present application, a nasal inverted papilloma system is provided, wherein

[0005] A data preprocessing module, used for performing standardization processing on the input first nasal endoscopic image and the first T2W-MRI image to obtain a second nasal endoscopic image and a second T2W-MRI image; wherein the first nasal endoscopic image and the first T2W-MRI image are images collected from the nose of the same target object;

[0006] An image segmentation model, used for performing segmentation processing on the second nasal endoscopic image and the second T2W-MRI image, and marking an inverted nasal papilloma region;

[0007] An image classification model is used to classify the second nasal endoscopic image and the second T2W-MRI image in which the inverted papilloma region is marked, to obtain a first predicted classification result and a second predicted classification result, respectively;

[0008] A fusion module is used to fuse the first prediction classification result and the second prediction classification result to obtain an image recognition result of the inverted nasal papilloma of the target object.

[0009] In one embodiment, the image segmentation model includes: a first image segmentation model and a second image segmentation model;

[0010] The system also includes: an image segmentation model training module, which is used to:

[0011] Inputting the nasal endoscopic image sample into a first model to be trained, and training to obtain a first image segmentation model for segmenting the second nasal endoscopic image;

[0012] The T2W-MRI image samples are input into the second model to be trained, and a second image segmentation model for segmenting the second T2W-MRI image is obtained through training.

[0013] In one embodiment, the image classification model includes: a first image classification model and a second image classification model;

[0014] The system also includes: an image classification model training module, which is used to:

[0015] Inputting the nasal endoscopic segmentation image sample into a third model to be trained, and training to obtain a first image classification model for classifying the second nasal endoscopic image;

[0016] The T2W-MRI segmented image samples are input into the fourth model to be trained, and a model for classifying the second image is obtained through training.

[0017] In one embodiment, the data preprocessing module is also used to standardize the anatomical directions of the T2W-MRI image and its segmentation labels into a RAS coordinate system, and resample the image and segmentation labels to make the voxel sizes of the T2W-MRI image and its segmentation labels consistent.

[0018] In one embodiment, the image segmentation model training module is used to: use a combination of DiceLoss and cross-entropy loss (Cross-Entropy Loss) to obtain DiceCELoss as a loss function during training.

[0019] .

[0021] In one embodiment, the image segmentation model training module is used to post-process the segmentation results using a sliding window technique and a KeepLargestConnectedComponent function to clean up misidentified scattered voxels.

[0022] In one embodiment, the classification model training module adopts a cosine decay learning rate strategy for training, and dynamically adjusts the learning rate to improve the training effect and stability of the model.

[0023] In one embodiment, the system further includes a user interaction interface for displaying the input nasal endoscopic image and T2W-MRI image, segmentation results, classification prediction results, and auxiliary recognition information.

[0024] According to a second aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store a computer program executable by the processor; and the processor is used to execute the computer program in the memory, so as to:

[0025] Performing standardization processing on the input first nasal endoscopic image and the first T2W-MRI image to obtain a second nasal endoscopic image and a second T2W-MRI image; wherein the first nasal endoscopic image and the first T2W-MRI image are images collected from the nose of the same target object;

[0026] performing segmentation processing on the second nasal endoscopic image and the second T2W-MRI image, and marking an inverted nasal papilloma region;

[0027] Classify the second nasal endoscopic image and the second T2W-MRI image in which the inverted papilloma region is marked, to obtain a first predicted classification result and a second predicted classification result, respectively;

[0028] The first prediction classification result and the second prediction classification result are fused to obtain an image recognition result of the inverted nasal papilloma of the target object.

[0029] According to a third aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the executable computer program in the storage medium is executed by a processor, it can be used to:

[0030] Performing standardization processing on the input first nasal endoscopic image and the first T2W-MRI image to obtain a second nasal endoscopic image and a second T2W-MRI image; wherein the first nasal endoscopic image and the first T2W-MRI image are images collected from the nose of the same target object;

[0031] performing segmentation processing on the second nasal endoscopic image and the second T2W-MRI image, and marking an inverted nasal papilloma region;

[0032] Classify the second nasal endoscopic image and the second T2W-MRI image in which the inverted papilloma region is marked, to obtain a first predicted classification result and a second predicted classification result, respectively;

[0033] The first prediction classification result and the second prediction classification result are fused to obtain an image recognition result of the inverted nasal papilloma of the target object.

[0034] Compared with the prior art, the beneficial effects of the present application are: a data preprocessing module is used to perform standardization processing on the input first nasal endoscopic image and the first T2W-MRI image to obtain a second nasal endoscopic image and a second T2W-MRI image; wherein the first nasal endoscopic image and the first T2W-MRI image are images collected from the nose of the same target object; an image segmentation model is used to perform segmentation processing on the second nasal endoscopic image and the second T2W-MRI image to mark the inverted papilloma area; an image classification module is used to classify the second nasal endoscopic image and the second T2W-MRI image marked with the inverted papilloma area to obtain a first prediction classification result and a second prediction classification result respectively; a fusion module is used to perform fusion processing on the first prediction classification result and the second prediction classification result to obtain an image recognition result of the inverted papilloma of the target object. Through the above scheme, in the segmentation and classification stage, the prediction results of the endoscopic model are combined with the prediction results of the T2W-MRI model, and the accuracy of the joint prediction of the two modal images is evaluated by the logistic regression method. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the structure of the inverted nasal papilloma identification system provided in an embodiment of the present application.

[0036] Figure 2 A schematic diagram of a process implemented using a nasal endoscope and T2W-MRI is provided in an embodiment of the present application.

[0037] Figure 3 A data analysis flow chart provided for an embodiment of the present application.

[0038] Figure 4 This is a schematic diagram of an example of segmenting a picture to illustrate an embodiment of the present application.

[0039] Figure 5 A schematic diagram of the ROC curve provided in the embodiment of the present application.

[0040] Figure 6 A schematic diagram of image recognition for label diagnosis provided in an embodiment of the present application.

[0041] Figure 7 This is a schematic diagram of the ROC comparison provided in the examples of the present application.

[0042] Figure 8 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0043] Unless otherwise defined, the technical terms or scientific terms used in this specification and claims shall have the usual meanings understood by persons with ordinary skills in the technical field to which the invention belongs. The specific embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be noted that in the specific description of these embodiments, in order to provide a concise description, it is impossible for this specification to provide a detailed description of all the features of the actual embodiments. Without departing from the spirit and scope of the present invention, those skilled in the art may modify and replace the embodiments of the present invention, and the resulting embodiments are also within the scope of protection of the present invention.

[0044] Magnetic Resonance Imaging (MRI): is an advanced medical imaging technology that uses magnetic fields and radio waves to generate detailed images of the internal structure of the human body. Magnetic resonance imaging is a non-invasive medical imaging examination technology. Its basic principle is to use the magnetic resonance phenomenon that occurs when hydrogen nuclei in human tissues in a static magnetic field are excited by radio frequency pulses of a specific frequency. When the radio frequency pulse ends, the hydrogen nuclei will induce MR signals during the relaxation process. After the MR signal is received, spatially encoded, and image reconstructed, an MR image can be generated.

[0045] White light nasal endoscopy, as a common ENT examination method, mainly uses a slender nasal endoscope equipped with a lighting system to insert into the nasal cavity to observe and evaluate the structure of the nasal cavity and nasopharynx and related disease lesions.

[0046] In the identification process of SIP, magnetic resonance imaging (MRI) and white light nasal endoscopy are two commonly used inspection methods. MRI produces images that can comprehensively observe the tumor, including its overall and internal structure, and provide detailed tumor image information. Nasal endoscopy can directly and microscopically display the characteristics of the tumor surface through images. These two methods complement each other and provide multi-dimensional information for the accurate identification of nasal tumors. However, human visual evaluation has limitations in capturing all useful information in medical images and is easily affected by personal experience, resulting in a certain degree of subjectivity in the evaluation results. In contrast, deep learning has a significant ability to objectively and comprehensively extract deep information from medical images, and has shown broad prospects in the application of nasal lesions. Therefore, a solution is needed that can accurately identify images collected related to inverted papilloma of the nose.

[0047] like Figure 1 This is a schematic diagram of the structure of the inverted nasal papilloma identification system provided in the embodiment of the present application. The system includes:

[0048] A data preprocessing module 1 is used to perform standardization processing on the input first nasal endoscopic image and the first T2W-MRI image to obtain a second nasal endoscopic image and a second T2W-MRI image; wherein the first nasal endoscopic image and the first T2W-MRI image are images collected from the nose of the same target object;

[0049] An image segmentation model 2 is used to segment the second nasal endoscopic image and the second T2W-MRI image, and mark the inverted papilloma area;

[0050] An image classification module 3 is used to classify the second nasal endoscopic image and the second T2W-MRI image marked with the inverted papilloma area, and obtain a first predicted classification result and a second predicted classification result respectively;

[0051] The fusion module 4 is used to fuse the first prediction classification result and the second prediction classification result to obtain an image recognition result of the inverted nasal papilloma of the target object.

[0052] The standardization processing method mentioned here includes at least one of: image direction unification, image interpolation, signal strength standardization, cropping and background removal.

[0053] Among them, in the scheme for accurately identifying the collected images related to inverted papilloma of the nose, the present application proposes to combine the collected images by magnetic resonance imaging (MRI) technology with the images presented by white light nasal endoscope to accurately identify the collected images related to inverted papilloma of the nose. In the process of image recognition, the artificial intelligence model is mainly used to participate in the two key steps of image segmentation and image classification. Below, the above scheme will be explained through specific embodiments.

[0054] Before using the artificial intelligence model to recognize the image, it is necessary to prepare the relevant data. Specifically, the data preprocessing module 1 first performs a series of standardization processing on the input first nasal endoscopic image and the first T2W-MRI image to ensure the consistency and accuracy of the subsequent processing steps. Including: standardizing the anatomical directions of the T2W-MRI image and its segmentation labels to the RAS coordinate system, and resampling the image and segmentation labels to make the voxel size of the T2W-MRI image and its segmentation labels consistent.

[0055] First, when using image processing libraries (such as OpenCV and PIL) to read images, the orientation information of the image (such as the orientation information in the DICOM tag) is automatically detected and uniformly adjusted to the standard anatomical orientation (such as front-to-back, left-to-right, up-to-down), thereby achieving image orientation unification.

[0056] When performing operations such as scaling and rotating images, advanced interpolation methods such as bilinear interpolation or bicubic interpolation are used to reduce pixel distortion and maintain image quality.

[0057] Map the intensity values ​​of MRI images to a uniform range to eliminate differences between different scanning devices or scanning parameters. T2W-MRI signal intensity standardization can be achieved using methods such as histogram matching or Z-score normalization.

[0058] Furthermore, the original image needs to be cropped. The region of interest (ROI), such as the nose and sinus area, is determined based on prior knowledge or automatic detection algorithms, and the area is cropped to remove unnecessary background information. For nasal endoscopic images, a specific background removal algorithm may need to be designed based on lighting conditions and imaging characteristics; for MRI images, image segmentation technology can be used to remove non-tissue parts such as air.

[0059] In addition, noise suppression measures can be taken as needed. For example, the image can be smoothed, such as using median filtering or Gaussian filtering to reduce image noise and improve the signal-to-noise ratio.

[0060] In this application, the image segmentation model 2 uses deep learning or traditional image processing technology to accurately segment the preprocessed image and mark the inverted papilloma area. A suitable segmentation model is selected according to the characteristics of the problem. For example, the segmentation model that can be selected can be a deep learning model such as U-Net, V-Net, or a traditional image processing method based on level set, graph cut, etc.

[0061] The model is trained using the labeled image data set, and the model parameters are continuously optimized to improve the segmentation accuracy. The specific training process will be described in the following embodiments, and will not be repeated here.

[0062] Assuming that the training of the image segmentation model (including the first image segmentation model for segmenting the nasal endoscopic image and the second image segmentation model for segmenting the T2W-MRI image) has been completed, the preprocessed image is input into the trained segmentation model, and the segmentation result of the inverted papilloma region of the nose is output. After completing the segmentation of the T2W-MRI image and the nasal endoscopic image, the segmentation result is subjected to necessary post-processing, such as removing small area noise, filling holes, etc., to improve the accuracy and reliability of the segmentation result.

[0063] After the artificial intelligence model is used to accurately segment the image, the image classification module 3 can also be used to accurately classify and identify the image (including T2W-MRI images and nasal endoscopy images). The image classification module 3 classifies the segmented inverted nasal papilloma area to determine its benign or malignant or other attributes. The details are as follows:

[0064] Fusion module 4 fuses the classification results from different image modalities (T2W-MRI images and nasal endoscopic images) to obtain more accurate and comprehensive image recognition results. Specifically, the classification results from different modules are collected, including the classification results of nasal endoscopic images and the classification results of T2W-MRI images. Design reasonable fusion strategies, such as weighted average, voting mechanism, learning-based fusion method, etc. Select a suitable fusion strategy according to the specific needs of the problem. The collected classification results are fused according to the designed fusion strategy to obtain the final image recognition result.

[0065] After the recognition is completed, the fused image recognition results can be output to the user or a subsequent processing module in an easy-to-understand form.

[0066] For example, Figure 2 The following is a schematic diagram of the process of identifying an inverted nasal papilloma as illustrated in the embodiment of the present application. Figure 2As can be seen in the figure, endoscopic images and T2W-MRI images of the same person are collected by different terminal devices (endoscopic device and magnetic resonance imaging device). Then, the images are processed using the models trained by each. Specifically, the endoscopic images are segmented using the first image segmentation model, and the T2W-MRI images are segmented using the second image segmentation model. The purpose of segmentation here is to focus on selecting the ROI area, so that the classification and recognition can be more accurately achieved during the subsequent classification and processing. Then, the segmented endoscopic images are classified and recognized using the first image classification model, and the segmented T2W-MRI images are classified and recognized using the second image classification model. Finally, the classification and recognition results of the two methods are integrated to obtain accurate recognition of the target person. For example, the target person can be an inverted papilloma of the nose or an inverted papilloma of the nose. Provide users with a basis for subsequent processing. Get rid of the ability requirements for image recognition personnel and improve work efficiency.

[0067] In the above system solution, the advantages of nasal endoscopic images and T2W-MRI images are fully utilized for multimodal fusion processing, which can improve the accuracy and reliability of image recognition results. Specifically, the image segmentation and image classification stages are achieved by combining nasal endoscopic images and T2W-MRI images, thereby improving the overall image recognition accuracy, that is, the accuracy of image recognition related to inverted papilloma can be improved.

[0068] In one or more embodiments of the present application, the image segmentation model 2 includes: a first image segmentation model 21 and a second image segmentation model 22. The system also includes: an image segmentation model training module 23, which is used to: input the nasal endoscopic image sample into the first model to be trained, and train to obtain a first image segmentation model for segmenting the second nasal endoscopic image; input the T2W-MRI image sample into the second model to be trained, and train to obtain a second image segmentation model for segmenting the second T2W-MRI image.

[0069] The image segmentation model training module 23 is used to: use the combination of Dice Loss and Cross-Entropy Loss to obtain DiceCELoss as the loss function during training.

[0070] The image segmentation model training module 23 is also used to adopt sliding window technology and

[0071]

[0072] The KeepLargestConnectedComponent function post-processes the segmentation results and cleans up the scattered voxels that were incorrectly identified.

[0073] In this application, before training the artificial intelligence model, it is necessary to collect training samples. These training samples are valid samples collected in actual work. When collecting samples, MRI and endoscopic examinations are performed on the inspected persons, and those who are identified as inverted papilloma of the nasal cavity and paranasal sinuses are selected. There are many images collected by MRI and endoscopic examinations, which should be selected according to actual needs. For example, the criteria for selecting samples include: ① Personnel who have undergone surgical resection and obtained pathological diagnosis; ② No treatment before imaging examination. Based on these criteria, 230 people were initially included. However, according to the exclusion criteria: ① Recurrent lesions (n ​​= 14); ② Poor endoscopic image quality or artifacts (n = 14); ③ Poor MRI image quality or artifacts (n = 14), some people were excluded. In the end, a total of 174 people provided endoscopic images and MRI images that met the requirements for subsequent training. And they were randomly assigned to the training group (n = 121) and the validation group (n = 53) at a ratio of 2:1 for subsequent research.

[0074] For image acquisition and annotation, MRI examinations were performed on two 3.0T scanners (Magnetom Verio or Prisma, Siemens Healthcare, Erlangen, Germany) using a 12-channel head and neck array coil. Axial fat-suppressed T2-weighted imaging (T2WI) was used for analysis, and the specific MRI acquisition parameters were as follows: axial T2WI (repetition time [TR] / echo time [TE], 4000 ms / 99 ms; matrix, 640×592; field of view, 220 mm×220 mm; thickness, 6 mm; spacing, 0.6 mm). Two experienced radiologists used ITK-SNAP software to delineate the tumor contours layer by layer on the T2W-MRI images of all subjects to obtain the region of interest (ROI).

[0075] Endoscopic examination used a 4mm rigid endoscope and a high-definition television endoscope video processor system. An experienced otolaryngologist, head and neck surgeon first selected a white light endoscopic image with the clearest and most representative lesion for each patient. Then, Labelme software was used to outline the tumor boundary on this endoscopic image.

[0076] After completing the preparation of basic data such as training samples mentioned above, you can further use the samples to train the image segmentation model and verify the model effect.

[0077] like Figure 3The data analysis flow chart provided in the embodiment of the present application is as follows. Figure 3 As shown in the figure, before the segmentation model is trained, the MRI images need to go through a series of preprocessing steps. First, the anatomical directions of the MRI images and their segmentation labels need to be standardized to the RAS (right, front, top) coordinate system. Next, the image is interpolated using the bilinear interpolation method, while the segmentation labels are interpolated using the nearest neighbor interpolation method to ensure that both the image and the segmentation labels are resampled to a voxel size of 1mm×1mm×1mm. Thereafter, the signal intensity is standardized to the range of 0-1600 by linear transformation, and a cropping step can be optionally performed to handle outliers. Finally, the mask image obtained from the T2W-MRI image and the nasal endoscopy image is used to remove the background and separate the foreground.

[0078] In the present application, UNet, DeeplabV3_Resnet101 and FCN_ResNet101 algorithms are preferably used to train the automatic segmentation model of tumors on endoscopic images, that is, the first image segmentation model. At the same time, UNet, UNETR and VNet algorithms are used to train the automatic segmentation model of tumors on T2W-MRI images, that is, the second image segmentation model. During the training phase, in order to ensure the balanced representation of positive and negative labels, the sub-volumes of images and labels are selectively cropped. In order to increase the diversity of the training data set, online data augmentation techniques are applied, including spacing adjustment and random cropping, so as to provide a series of different images for each training iteration. In terms of hyperparameter setting, the Adam optimizer is used, and the initial learning rate is set to 1e -3 , and trained for 600 cycles. To avoid overfitting, early stopping was used after 32 rounds. DiceCELoss was used as the loss function during training, which is a combination of Dice Loss and Cross-Entropy Loss. When predicting the segmentation label, a sliding window technique with a size of 96×96×48 voxels was used. However, during the prediction process, misidentified scattered voxels may be encountered, which need to be cleaned up. To solve this problem, the KeepLargestConnectedComponent function was used (set topk=1 in MONAI), through which the largest four connected ROI regions were selected.

[0079] The Dice similarity coefficient (DICE) and mean intersection over union (mIOU) were used to comprehensively evaluate the segmentation performance of each model, so as to select the best model for automated segmentation of inverted papilloma of the nasal cavity and paranasal sinuses on MRI and endoscopy.

[0080] In addition, DiceCELoss is used as the loss function during training, which is a combination of Dice Loss and Cross-Entropy Loss. This dual approach optimizes class imbalance processing and segmentation accuracy. The formula of the loss function is:

[0081]

[0082] In this formula, N is the batch size, Dicei and CEi are the Dice Loss and cross entropy loss of the i-th sample, respectively, and α is the weight factor that balances the two loss functions. Unlabeled pixels are given zero weights, so that learning focuses on labeled areas and enhances the generalization ability of the entire volume. When predicting segmentation labels, for example, a sliding window technique with a size of 96×96×48 voxels can be used. However, during the prediction process, misidentified scattered voxels may be encountered, which need to be cleaned up. To solve this problem, the KeepLargestConnectedComponent function can be used (set topk=1 in MONAI), which selects the largest four connected ROI regions. The Dice similarity coefficient (DICE) and mean intersection-over-union (mIOU) are used to comprehensively evaluate the segmentation performance of each model, so as to select the best model for automated segmentation of nasal inverted papilloma on MRI and endoscopy.

[0083] In one or more embodiments of the present application, the image classification model 3 includes: a first image classification model 31 and a second image classification model 32. The system also includes: an image classification model training module 33, which is used to:

[0084] Inputting the nasal endoscopic segmentation image sample into a third model to be trained, and training to obtain a first image classification model for classifying the second nasal endoscopic image;

[0085] The T2W-MRI segmented image samples are input into the fourth model to be trained, and a model for classifying the second image is obtained through training.

[0086] In practice, the classification model was trained based on the manually segmented ROI and the ROI predicted by the best segmentation model. On the T2W-MRI images, only the largest slice of the tumor was selected as the analysis object. In order to reduce background noise and focus on the key area, the minimum bounding rectangle around the ROI was retained and expanded outward by 5 pixels (the actual number of pixels expanded can be selected and modified according to user needs) to ensure that the area around the tumor is included. In the preprocessing stage, the images were normalized using the Z-score normalization method to eliminate the intensity differences between different images. During the training process, in order to enhance the diversity of data and improve the generalization ability of the model, data augmentation techniques such as random cropping, horizontal flipping, and vertical flipping were applied. For the images of the test group, only normalization was performed to ensure consistency with the training data. Under the framework of transfer learning, the performance of the two models, DenseNet121 and CrossFormer, was evaluated respectively, and the models were initialized using the weights of ImageNet. In addition, in order to further optimize the model performance and promote better generalization between datasets, the learning rate can be fine-tuned.

[0087] In this application, a cosine decay learning rate strategy is adopted, which can dynamically adjust the learning rate according to the progress of the training process, thereby improving the training effect and stability of the model. It is defined as follows:

[0088]

[0089] In this equation, Set the minimum learning rate, Set the maximum learning rate, T i = 30 represents the number of epochs in the iterative training process. Other important hyperparameters include the use of stochastic gradient descent (SGD) as the optimizer and softmax cross entropy as the loss function.

[0090] Other important hyperparameters include the use of stochastic gradient descent (SGD) as the optimizer and softmax cross entropy as the loss function. The predictions of the endoscopic model were combined with those of the T2W-MRI model, and the performance of the joint prediction of the two modalities was evaluated by logistic regression. Finally, the predictions of the best classification model based on automated segmentation techniques were used as deep learning labels to measure and evaluate the accuracy and effectiveness of the joint prediction. In order to make the decision process of the model more transparent and explore its interpretability, gradient-weighted class activation mapping (Grad-CAM) was applied to visualize the model and highlight the important areas in the classification target image.

[0091] It should be noted that the above-mentioned training samples were authorized by relevant personnel when processing personnel image data, and strictly abided by relevant laws, regulations and ethical standards to ensure the security and privacy of the data.

[0092] In one or more embodiments of the present application, the fusion module 4 is used to fuse the first prediction classification results and the second prediction classification results from different image modalities (which may be based on nasal endoscopic images and T2W-MRI images) to obtain the final image recognition result of the target object inverted nasal papilloma. Specifically, as a key component of the entire image processing and recognition system, the fusion module 4 is responsible for integrating the prediction results from different classifiers (or different image modalities), and through an intelligent fusion strategy, comprehensively evaluating and generating the final image recognition result of the target object inverted nasal papilloma. This process is intended to improve the accuracy and robustness of the recognition results and reduce the errors that may be caused by a single classifier.

[0093] Receive two predicted classification results from the image classification module 3: the first predicted classification result (based on the nasal endoscopic image) and the second predicted classification result (based on the T2W-MRI image). Ensure that the format, category label and other information of the two classification results are consistent for subsequent processing.

[0094] In an optional solution, select an appropriate fusion strategy based on the specific needs of the problem. Common fusion strategies include:

[0095] Weighted average method: Assign different weights to the two classification results and then perform a weighted average. The choice of weights can be based on factors such as the performance of the classifier and the reliability of the image modality.

[0096] Voting mechanism: Use majority voting or weighted voting to vote based on the category labels of the two classification results, and the category with the most votes is used as the final recognition result.

[0097] Bayesian fusion: Using Bayesian theorem, the prior probability and conditional probability are combined to calculate the posterior probability, thereby obtaining the final recognition result.

[0098] Machine learning fusion: Train an additional machine learning model (such as decision tree, random forest, support vector machine, etc.) to take the two classification results as input and output the final recognition result.

[0099] Apply the selected fusion strategy to fuse the two predicted classification results. During the processing, it may be necessary to consider factors such as the confidence and uncertainty of the classification results to optimize the fusion effect. Verify and evaluate the fused recognition results, and use independent test sets or cross-validation methods to evaluate their accuracy and reliability.

[0100] The fused image recognition results are output to the user or a subsequent processing module in an easily understandable form. The results may include a clear judgment of whether the target subject suffers from inverted nasal papilloma, as well as relevant confidence or probability information.

[0101] The multimodal information is fused to make full use of the complementary information between different image modalities. Based on the feedback from actual applications, the fusion strategy and the performance of the classification model are continuously adjusted and optimized to improve the recognition accuracy and robustness of the entire system.

[0102] In one or more embodiments of the present application, the system further includes a user interaction interface for displaying input nasal endoscopic images and T2W-MRI images, segmentation results, classification prediction results, and auxiliary recognition information.

[0103] In actual applications, the user interaction interface can provide users with various detailed information, including basic nasal endoscopic images and T2W-MRI images, as well as segmentation and classification results of nasal endoscopic images and T2W-MRI images, so that users can easily identify the results of automatic processing by the model. The classification prediction results are finally output. In order to simplify the user's recognition workload, corresponding auxiliary recognition information can also be assigned to different recognition results. The auxiliary recognition information mentioned here can be inverted papilloma and non-inverted papilloma. After obtaining this auxiliary recognition information, the user can directly adopt it, or use the image classification results to mark the recognition results themselves.

[0104] After completing the above model training and verification work, the above model can be actually used to complete the image classification work. The following is an explanation through a specific embodiment.

[0105] As mentioned above, when selecting the training samples, 19.8% (24 / 121) of the subjects in the training group and 18.9% (10 / 53) of the subjects in the validation group were pathologically diagnosed with IP-SCC and were found to have occult lymph node metastasis (LNM), with no significant difference between the two centers (p = 0.952).

[0106] In the task of automated segmentation of endoscopic images, the FCN_ResNet101 model performed best, with Dice and IOU of 0.94±0.05 and 0.80±0.08 in the training group, and 0.95±0.03 and 0.90±0.06 in the validation group. In addition, the Dice and IOU of UNet in the training group were 0.70±0.20 and 0.57±0.21, and 0.68±0.21 and 0.55±0.22, respectively. The Dice and IOU of DeeplabV3_Resnet101 in the training group were 0.94±0.04 and 0.89±0.07, respectively, and in the validation group were 0.94±0.04 and 0.89±0.07, respectively.

[0107] In the T2-MRI image automatic segmentation task, the VNet model performed best, with Dice and IOU of 0.91±0.10 and 0.86±0.10 in the training group, and 0.93±0.02 and 0.88±0.03 in the validation group. In addition, the Dice and IOU of UNet in the training group were 0.87±0.09 and 0.81±0.09, and 0.89±0.04 and 0.82±0.05 in the validation group. The Dice and IOU of UNETR in the training group were 0.82±0.11 and 0.75±0.10, and 0.86±0.05 and 0.78±0.06 in the validation group. Figure 4 The schematic diagram of the example segmentation images used to illustrate the embodiments of the present application shows Table 1 for the performance of automatic segmentation of endoscopic and T2W-MRI lesions using deep learning.

[0108] Table 1 Performance of deep learning models in automated tumor segmentation under endoscopy and T2WI-MRI

[0109]

[0110] Whether based on manual segmentation or automated segmentation, the model used jointly for nasal endoscopic images and T2W-MRI images is superior to the model constructed by a single modality. In the validation group, CrossFormer and DenseNet121 had the best diagnostic efficacy based on manual segmentation and automated segmentation, respectively. Based on manual segmentation, the AUC of CrossFormer in the training group and validation group were 0.872 (95% CI: 0.786-0.959) and 0.914 (95% CI: 0.829-0.998), and the accuracy rates were 0.901 and 0.736, respectively. Based on automated segmentation, the AUC of DenseNet121 in the training group and validation group were 0.938 (95% CI: 0.893-0.984) and 0.865 (95% CI: 0.742-0.988), and the accuracy rates were 0.876 and 0.774, respectively. In the training group and the validation group, there was no significant difference in AUC between the optimal models based on manual segmentation and automated segmentation (p = 0.558 and 0.488). Finally, the DenseNet121 model integrating two sequences based on automated segmentation was selected as the automated deep learning label. Figure 5 A schematic diagram of the ROC curve provided in the embodiment of the present application, Figure 6 Schematic diagram of image recognition for label diagnosis provided in the embodiment of the present application. Specifically, the performance of the deep learning model for diagnosing SIP-SCC based on manual annotation and automated segmentation is recommended in Tables 2 and 3, and the corresponding ROC curve is shown in Table 2. Figure 5As shown in Figure 2, ROC curves of models for diagnosing SIP-SCC based on manual annotation and automated segmentation are compared, including single-modality and bimodality models based on densenet121 and crossformer. Representative case images correctly diagnosed using automated deep learning labels are shown in Figure 2. Figure 6 As shown, representative cases correctly diagnosed using automated deep learning labels, Cases 1 and 2 are SIP and SIP-SCC (SIP transformed into squamous cell carcinoma), respectively. From left to right, they are the original T2W-MRI image, the cropped region of interest, and the classification activation map (dark colors represent important areas for classification).

[0111] Table 2 Performance of deep learning models in predicting SIP-SCC based on manual annotation

[0112]

[0113] Table 3 Performance of deep learning models in predicting SIP-SCC based on automated segmentation

[0114]

[0115] The AUC values ​​of residents and attending physicians in the training group were 0.631 (0.524-0.738) and 0.704 (95% CI: 0.597-0.811), and the accuracy was 0.760 and 0.752, respectively. The AUC values ​​of the two doctors in the validation group were 0.672 (95% CI: 0.499-0.845) and 0.707 (95% CI: 0.536-0.877), and the accuracy was 0.717 and 0.774, respectively. With the assistance of automated learning labels, the diagnostic performance of the two doctors in both cohorts was improved. The AUC values ​​in the training group were 0.745 (95% CI: 0.641-0.850) and 0.813 (95% CI: 0.719-0.908), and the accuracy was 0.818 and 0.851, respectively. The AUC values ​​in the validation group were 0.734 (95% CI: 0.571-0.896) and 0.834 (95% CI: 0.717-0.951), and the accuracies were 0.755 and 0.792, respectively. Figure 7 The ROC comparison diagram provided in the embodiment of this application is shown in Table 4. The performance of the automated deep learning label (DenseNet121 dual-modal fusion model based on automated segmentation) and two radiologists with and without the assistance of deep learning labels is shown in Table 4. The corresponding ROC curve comparison is shown in Figure 7 It can be seen that using the trained deep learning model (that is, the first image classification model and the second image classification model) can obtain better and more reliable image classification processing results.

[0116] Table 4 Performance of radiologists of different years in diagnosing SIP-SCC before and after assistance with deep learning signatures (DL signature)

[0117]

[0118] An embodiment of the present application further provides an electronic device, comprising a processor and a memory; the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement:

[0119] Performing standardization processing on the input first nasal endoscopic image and the first T2W-MRI image to obtain a second nasal endoscopic image and a second T2W-MRI image; wherein the first nasal endoscopic image and the first T2W-MRI image are images collected from the nose of the same target object;

[0120] performing segmentation processing on the second nasal endoscopic image and the second T2W-MRI image, and marking an inverted papilloma region of the nose;

[0121] Classify the second nasal endoscopic image and the second T2W-MRI image in which the inverted papilloma region is marked, to obtain a first predicted classification result and a second predicted classification result, respectively;

[0122] The first prediction classification result and the second prediction classification result are fused to obtain an image recognition result of the inverted nasal papilloma of the target object.

[0123] The image segmentation model comprises: a first image segmentation model and a second image segmentation model;

[0124] The system also includes: an image segmentation model training module, which is used to:

[0125] Inputting the nasal endoscopic image sample into a first model to be trained, and training to obtain a first image segmentation model for segmenting the second nasal endoscopic image;

[0126] The T2W-MRI image samples are input into the second model to be trained, and a second image segmentation model for segmenting the second T2W-MRI image is obtained through training.

[0127] The image classification model in the electronic device includes: a first image classification model and a second image classification model;

[0128] The system also includes: an image classification model training module, which is used to:

[0129] Inputting the nasal endoscopic segmentation image sample into a third model to be trained, and training to obtain a first image classification model for classifying the second nasal endoscopic image;

[0130] The T2W-MRI segmented image samples are input into the fourth model to be trained, and a model for classifying the second image is obtained through training.

[0131] The data preprocessing module in the electronic device is also used to standardize the anatomical directions of the T2W-MRI image and its segmentation labels into a RAS coordinate system, and resample the image and segmentation labels to make the voxel sizes of the T2W-MRI image and its segmentation labels consistent.

[0132] The image segmentation model training module in the electronic device is used to: use the combination of Dice Loss and Cross-Entropy Loss (Cross-Entropy Loss) to obtain DiceCELoss as a loss function during training, and the formula of the loss function is:

[0133]

[0134] Where N is the batch size, Dicei and CEi are the Dice Loss and cross entropy loss of the i-th sample, respectively, and α is the weight factor to balance the two loss functions.

[0135] The image segmentation model training module in the electronic device is used to post-process the segmentation results using the sliding window technology and the KeepLargestConnectedComponent function to clean up the scattered voxels that were misidentified.

[0136] The classification model training module in the electronic device adopts the cosine decay learning rate strategy for training, and dynamically adjusts the learning rate to improve the training effect and stability of the model.

[0137] In addition, the electronic device also includes a display screen or a projection component, and there is a display module user interaction interface on the electronic device for displaying the input nasal endoscopic image and T2W-MRI image, segmentation results, classification prediction results and auxiliary recognition information.

[0138] The embodiment of the present application further provides a computer-readable storage medium, which, when an executable computer program in the storage medium is executed by a processor, can be used to:

[0139] Performing standardization processing on the input first nasal endoscopic image and the first T2W-MRI image to obtain a second nasal endoscopic image and a second T2W-MRI image; wherein the first nasal endoscopic image and the first T2W-MRI image are images collected from the nose of the same target object;

[0140] performing segmentation processing on the second nasal endoscopic image and the second T2W-MRI image, and marking an inverted papilloma region of the nose;

[0141] Classify the second nasal endoscopic image and the second T2W-MRI image in which the inverted papilloma region is marked, to obtain a first predicted classification result and a second predicted classification result, respectively;

[0142] The first prediction classification result and the second prediction classification result are fused to obtain an image recognition result of the inverted nasal papilloma of the target object.

[0143] The image segmentation model comprises: a first image segmentation model and a second image segmentation model;

[0144] The system also includes: an image segmentation model training module, which is used to:

[0145] Inputting the nasal endoscopic image sample into a first model to be trained, and training to obtain a first image segmentation model for segmenting the second nasal endoscopic image;

[0146] The T2W-MRI image samples are input into the second model to be trained, and a second image segmentation model for segmenting the second T2W-MRI image is obtained through training.

[0147] The image classification model in the electronic device includes: a first image classification model and a second image classification model;

[0148] The system also includes: an image classification model training module, which is used to:

[0149] Inputting the nasal endoscopic segmentation image sample into a third model to be trained, and training to obtain a first image classification model for classifying the second nasal endoscopic image;

[0150] The T2W-MRI segmented image samples are input into the fourth model to be trained, and a model for classifying the second image is obtained through training.

[0151] The data preprocessing module in the electronic device is also used to standardize the anatomical directions of the T2W-MRI image and its segmentation labels into a RAS coordinate system, and resample the image and segmentation labels to make the voxel sizes of the T2W-MRI image and its segmentation labels consistent.

[0152] The image segmentation model training module in the electronic device is used to: use the combination of Dice Loss and Cross-Entropy Loss (Cross-Entropy Loss) to obtain DiceCELoss as a loss function during training, and the formula of the loss function is:

[0153]

[0154] Where N is the batch size, Dicei and CEi are the Dice Loss and cross entropy loss of the i-th sample, respectively, and α is the weight factor to balance the two loss functions.

[0155] The image segmentation model training module in the electronic device is used to post-process the segmentation results using the sliding window technology and the KeepLargestConnectedComponent function to clean up the scattered voxels that were misidentified.

[0156] The classification model training module in the electronic device adopts the cosine decay learning rate strategy for training, and dynamically adjusts the learning rate to improve the training effect and stability of the model.

[0157] In addition, the electronic device also includes a display screen or a projection component, and there is a display module user interaction interface on the electronic device for displaying the input nasal endoscopic image and T2W-MRI image, segmentation results, classification prediction results and auxiliary recognition information.

[0158] Regarding the device in the above embodiment, the specific manner in which the processor performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0159] Figure 8 900 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 900 may be provided as a server. Figure 8 The device 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as applications, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform the above-mentioned method for image processing.

[0160] The device 900 may also include a power component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0161] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 932 including instructions, which can be executed by the processing component 922 of the device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0162] In the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.

[0163] The above description of the embodiments is to facilitate those of ordinary skill in the art to understand and apply the present application. It is obvious that those familiar with the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without having to pay creative labor. Therefore, the present application is not limited to the embodiments herein, and improvements and modifications made by those skilled in the art based on the contents disclosed in the present application without departing from the scope and spirit of the present application are within the scope of the present application.

Claims

1. A system for identifying inverted nasal papilloma, characterized in that: The system comprises: A data preprocessing module, used for performing standardization processing on the input first nasal endoscopic image and the first T2W-MRI image to obtain a second nasal endoscopic image and a second T2W-MRI image; wherein the first nasal endoscopic image and the first T2W-MRI image are images collected from the nose of the same target object; An image segmentation model, used for performing segmentation processing on the second nasal endoscopic image and the second T2W-MRI image, and marking an inverted nasal papilloma region; An image classification model is used to classify the second nasal endoscopic image and the second T2W-MRI image in which the inverted papilloma region is marked, to obtain a first predicted classification result and a second predicted classification result, respectively; A fusion module is used to fuse the first prediction classification result and the second prediction classification result to obtain an image recognition result of the inverted nasal papilloma of the target object.

2. The system according to claim 1, characterized in that The image segmentation model comprises: a first image segmentation model and a second image segmentation model; The system also includes: an image segmentation model training module, which is used to: Inputting the nasal endoscopic image sample into a first model to be trained, and training to obtain a first image segmentation model for segmenting the second nasal endoscopic image; The T2W-MRI image samples are input into the second model to be trained, and a second image segmentation model for segmenting the second T2W-MRI image is obtained through training.

3. The system according to claim 1, characterized in that The image classification model includes: a first image classification model and a second image classification model; The system also includes: an image classification model training module, which is used to: Inputting the nasal endoscopic segmentation image sample into a third model to be trained, and training to obtain a first image classification model for classifying the second nasal endoscopic image; The T2W-MRI segmented image samples are input into the fourth model to be trained, and a model for classifying the second image is obtained through training.

4. The system according to claim 1, characterized in that The data preprocessing module is also used to standardize the anatomical directions of the T2W-MRI image and its segmentation labels into a RAS coordinate system, and to resample the image and segmentation labels so that the voxel sizes of the T2W-MRI image and its segmentation labels are consistent.

5. The system according to claim 2, characterized in that The image segmentation model training module is used to: use the combination of Dice Loss and Cross-Entropy Loss (Cross-Entropy Loss) to obtain DiceCELoss as the loss function during training Where N is the batch size, Dicei and CEi are the Dice Loss and cross entropy loss of the i-th sample, respectively, and α is the weight factor to balance the two loss functions.

6. The system according to claim 2, characterized in that The image segmentation model training module is used to post-process the segmentation results by using the sliding window technology and the KeepLargestConnectedComponent function to clean up the scattered voxels that are misidentified.

7. The system according to claim 3, characterized in that The classification model training module adopts a cosine decay learning rate strategy for training, and dynamically adjusts the learning rate to improve the training effect and stability of the model.

8. The system according to claim 1, characterized in that The system also includes a user interaction interface for displaying input nasal endoscopic images and T2W-MRI images, segmentation results, classification prediction results, and auxiliary recognition information.

9. An electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor for: The input first nasal endoscopic image and the first T2W-MRI image are standardized to obtain a second nasal endoscopic image and a second T2W-MRI image; wherein, The first nasal endoscopic image and the first T2W-MRI image are images acquired from the nose of the same target object; performing segmentation processing on the second nasal endoscopic image and the second T2W-MRI image, and marking an inverted nasal papilloma region; Classify the second nasal endoscopic image and the second T2W-MRI image in which the inverted papilloma region is marked, to obtain a first predicted classification result and a second predicted classification result, respectively; The first prediction classification result and the second prediction classification result are fused to obtain an image recognition result of the inverted nasal papilloma of the target object.

10. A computer-readable medium having stored thereon at least one instruction, at least one program, code set or instruction set, the at least one instruction, at least one program, code set or instruction set being loaded and executed by a processor for: The input first nasal endoscopic image and the first T2W-MRI image are standardized to obtain a second nasal endoscopic image and a second T2W-MRI image; wherein, The first nasal endoscopic image and the first T2W-MRI image are images acquired from the nose of the same target object; performing segmentation processing on the second nasal endoscopic image and the second T2W-MRI image, and marking an inverted nasal papilloma region; Classify the second nasal endoscopic image and the second T2W-MRI image in which the inverted papilloma region is marked, to obtain a first predicted classification result and a second predicted classification result, respectively; The first prediction classification result and the second prediction classification result are fused to obtain an image recognition result of the inverted nasal papilloma of the target object.

Images