Methods, systems, and terminals for predicting the risk of polyp lesions in endoscopic images

Abstract

This invention provides a method and system for predicting the risk of polyps in endoscopic images. The method involves cropping the output image of an endoscopic device into multiple small blocks and filtering out invalid blocks to obtain small block images. Image data is established based on the relative positional relationships between these small block images. This image data is then input into an endoscopic polyp prediction model, which outputs prediction results for each small block image. These prediction results are then combined to obtain a polyp prediction result for the entire output image. Based on this process, segment risk prediction is performed on endoscopic video data to obtain a polyp risk prediction for the entire video data. This invention is simple to operate and requires no technical expertise. Dividing the endoscopic image into multiple small blocks for prediction allows for simultaneous identification of high-risk areas within the endoscopic image. Using a large model as the base model for extracting endoscopic image features results in high accuracy and strong clinical applicability.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a method, system, and terminal for predicting the risk of polyp lesions in endoscopic images based on NBI magnified images. Background Technology

[0002] In recent years, deep learning technology has developed rapidly and has been widely applied in the field of medical image analysis. Endoscopic images are an important basis for doctors to analyze patients' digestive tract diseases, and many studies have applied deep convolutional neural networks (CNNs) to the screening and detection of lesions, achieving remarkable results and having significant clinical implications.

[0003] Clinically, endoscopists often face challenges in identifying and detecting suspicious lesions based on experience. With the increasing prevalence of AI-assisted diagnosis, its research and application in the early diagnosis of colorectal cancer have received significant attention and development in recent years. Currently, much research focuses on using AI to assist in the real-time detection of intestinal polyps, aiming to improve the detection rate during colonoscopy. For polyps with a risk of malignancy, endoscopists need to collect pathological samples for further examination. However, the observation time during colonoscopy withdrawal is typically only 6-7 minutes, and this time is limited by the operator's work status and experience, making it easy to miss potentially cancerous polyps, which could lead to missing the patient's golden treatment window.

[0004] Currently available colonoscopy-assisted systems and methods focus only on identifying lesions in images acquired by the operating physician and improving polyp detection rates, lacking technologies for predicting the risk of polyp malignancy. Due to the lack of unique texture features in colonic polyps, directly using traditional convolutional neural networks for polyp lesion risk prediction is challenging. Therefore, there is a lack of research on analyzing colonoscopy examinations based on NBI magnified images, and no more effective model architecture exists. Therefore, developing an endoscopic polyp lesion diagnosis method based on NBI magnified images is imperative. Currently, no descriptions or reports of technologies similar to this invention have been found, and similar domestic and international data have not been collected. Summary of the Invention

[0005] To address the aforementioned shortcomings in the existing technology, the purpose of this invention is to provide a method, system, and terminal for predicting the risk of polyp lesions in endoscopic images based on NBI magnified images.

[0006] According to one aspect of the present invention, a method for predicting the risk of polypoid lesions in endoscopic images is provided, comprising:

[0007] The output image from the endoscope is cropped into multiple small blocks, and invalid blocks are filtered out to obtain small block images;

[0008] Graph data is created based on the relative positional relationships between small image blocks;

[0009] A pre-trained endoscopic polyp lesion prediction model is provided. The established image data is input into the endoscopic polyp lesion prediction model, and the prediction results are output to obtain the lesion risk prediction results for each small image patch.

[0010] By combining the prediction results of each small image patch, the prediction result of polyp lesions for the entire output image is obtained.

[0011] Preferably, cropping the output image of the endoscopic device into multiple small blocks and filtering out invalid blocks includes:

[0012] The output image from the endoscope is cropped into multiple 256×256 blocks;

[0013] Convert all small blocks to HSV space, and determine whether each pixel belongs to the shadow area in the HSV space;

[0014] If the proportion of the shaded area in any small patch is greater than or equal to a set threshold, then the small patch is considered an invalid small patch image and is discarded.

[0015] If the proportion of the shaded area in any small patch is less than a set threshold, then the small patch is considered a valid small patch image and is retained for risk prediction.

[0016] Preferably, the step of establishing graph data based on the relative positional relationships between small image patches includes:

[0017] Each small image patch is treated as a node. Based on the relative positions of the small images, small images that are in direct contact with each other are connected by edges, while small images that are not in direct contact with each other are not connected by edges, thus creating graph data with small images as nodes.

[0018] Preferably, the provision of a pre-trained endoscopic polyp lesion prediction model involves inputting the established graphical data into the endoscopic polyp lesion prediction model and outputting prediction results, including:

[0019] A pre-trained endoscopic polyp lesion prediction model is provided, the endoscopic polyp lesion prediction model comprising: a pre-trained feature extraction model and a pre-trained graph neural network model;

[0020] The graph data is processed by the feature extraction model to extract features, thereby obtaining feature vectors for each small image block. The information of the feature vectors is carried by each node, resulting in graph data containing feature information.

[0021] The graph data containing feature information is input into the graph neural network model, and the lesion risk prediction result for each node is output, that is, the lesion risk prediction result for each small image block is obtained.

[0022] Preferably, the pre-trained feature extraction model is trained using massive amounts of endoscopic video data to extract local features from endoscopic images.

[0023] Preferably, the pre-trained graph neural network model is trained using training image patches with texture features, and is used to predict the results of image features.

[0024] Preferably, the graph neural network model comprises two graph convolutional layers and a fully connected layer; wherein, after each graph convolutional layer, a batch normalization layer and a ReLU activation function are applied, and a dropout layer is added between the two graph convolutional layers; the output of the fully connected layer is 2-dimensional, corresponding to the prediction confidence of lesions and non-lesions respectively.

[0025] Preferably, the training image patch with texture features is obtained in the following manner:

[0026] Professional doctors manually annotate the training images to facilitate lesion risk diagnosis and filter out irrelevant areas.

[0027] The bounding rectangle of the labeled area is cropped to obtain training image patches of the same shape and size;

[0028] Professional doctors assign labels to the training image blocks based on their expertise and pathological findings, resulting in training image blocks with texture features.

[0029] Preferably, the prediction results of combining each small image patch include:

[0030] If all the small images are predicted negative, the polyp lesion prediction result for the entire output image is low risk. If any one or more of the small images are predicted positive, the polyp lesion prediction result for the entire output image is high risk.

[0031] Preferably, the above method further includes: based on the above process, performing segment risk prediction on the endoscopic video data to obtain a prediction of the risk of polyp lesions in the entire endoscopic video data.

[0032] Preferably, the endoscopic video data is used for segment risk prediction to obtain a prediction of the risk of polypoid lesions in the entire endoscopic video data, including:

[0033] Each frame of the endoscopic video data is extracted, and several small images of the same size are randomly sampled from each frame. The graph data is established based on the relative positional relationship between the small images, and the lesion risk is predicted on the graph data using a pre-trained endoscopic polyp lesion prediction model to obtain the lesion risk prediction result for each frame.

[0034] A video segment with several consecutive high-risk frames is considered as a high-risk lesion segment; if the number of consecutive high-risk lesion segments in the video data exceeds a set threshold, the video data is judged as high-risk lesion; otherwise, it is judged as low-risk lesion.

[0035] According to another aspect of the present invention, a system for predicting the risk of polypoid lesions in endoscopic images is provided, comprising:

[0036] The image preprocessing module is used to crop the output image of the endoscope into multiple small blocks and filter out invalid small blocks to obtain small block images;

[0037] The graph data construction module is used to build graph data based on the relative positional relationships between small image patches.

[0038] The image risk prediction module provides a pre-trained endoscopic polyp lesion prediction model. The established image data is input into the endoscopic polyp lesion prediction model, and the prediction results are output to obtain the lesion risk prediction results for each small image patch. The prediction results of each small image patch are combined to obtain the polyp lesion prediction results for the entire output image.

[0039] Preferably, the above system further includes:

[0040] The video risk prediction module, based on the image preprocessing module, the image data construction module, and the image risk prediction module, performs segment risk prediction on the endoscopic video data to obtain the risk prediction of polyp lesions in the entire endoscopic video data.

[0041] According to a third aspect of the present invention, an endoscopic polyp risk prediction terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, can be used to perform the method described in any one of the above-mentioned inventions, or to run the system described in any one of the above-mentioned inventions.

[0042] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can be used to perform the method described in any one of the preceding claims of the present invention, or to run the system described in any one of the preceding claims of the present invention.

[0043] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0044] This invention employs a large-scale endoscopic model for feature extraction. This model, trained on massive amounts of endoscopic video data, is therefore highly capable of extracting local features from endoscopic images. Simultaneously, to train the graph neural network and integrate information from different locations within the endoscopic images for more accurate risk prediction, it utilizes training image patches incorporating professional knowledge. The superior large-scale model and high-quality training data enable this invention to achieve predictive results in lesion risk diagnosis that surpass those of medical experts.

[0045] The training dataset in this invention is generated by an expert system or by manual annotation of regions with significant intestinal surface structures and blood vessels in endoscopic images. Using such a high-quality dataset for training can further improve the accuracy of lesion risk prediction.

[0046] This invention can diagnose polyps from magnified endoscopic images without human intervention, and output the risk of polyps to assist medical personnel in making judgments. Furthermore, this invention crops the magnified endoscopic image into multiple small blocks, predicts each block separately, and then combines the prediction results. This block-based prediction method provides the risk for each region, making it easier for doctors to locate and observe high-risk areas in the overall image. Attached Figure Description

[0047] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0048] Figure 1 This is a flowchart of the method for predicting the risk of polyp lesions in endoscopic images according to a preferred embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of the components of the polyp risk prediction system in endoscopic images according to a preferred embodiment of the present invention;

[0050] Figure 3 This is an architecture diagram of a polyp risk prediction system in endoscopic images in a specific application example of the present invention.

[0051] Figure 4 This is a conceptual diagram illustrating the workflow of a method for predicting the risk of polyp lesions in endoscopic images in a specific application example of the present invention.

[0052] Figure 5 A comparison chart showing the recall rate predicted by an endoscopist using the method and system provided by this invention in a specific application example;

[0053] Figure 6The image data is predicted as cancerous and non-cancerous using the method and system provided by the present invention in a specific example; wherein, (a) is a set of non-cancerous images with regularly distributed blood vessels and a conventional tubular surface; and (b) is a set of cancerous images with irregularly distributed blood vessels, a loose surface, and a very blurry surface in amorphous areas. Detailed Implementation

[0054] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0055] To address the lack of research on colonoscopy analysis based on NBI magnified images in existing technologies, and the absence of more effective model architectures, an embodiment of this invention provides a method for predicting the risk of polyp lesions in endoscopic images. This method presents a framework for predicting the risk of lesions in medical endoscopic images, utilizing a large model for feature extraction and training a classifier for classification. This provides assistance in judging the risk of cancer in magnified endoscopic images in hospitals, improving the accuracy of judgment and shortening the judgment time.

[0056] Specifically, such as Figure 1 As shown, the method for predicting the risk of polyp lesions in endoscopic images provided in this embodiment may include the following operations:

[0057] S1, Preprocess the output image of the endoscope: Crop the output image of the endoscope into multiple small blocks and filter out invalid small blocks to obtain small block images;

[0058] S2, Build graph data based on the relative positional relationships between small image blocks;

[0059] S3 provides a pre-trained endoscopic polyp lesion prediction model. The established image data is input into the endoscopic polyp lesion prediction model, and the prediction results are output to obtain the lesion risk prediction results for each small image.

[0060] S4 combines the prediction results of each small image patch to obtain the polyp lesion prediction result for the entire output image.

[0061] In some preferred embodiments, the above method may further include the following operations:

[0062] S5. Based on the above process, segment risk prediction is performed on the endoscopic video data to obtain the risk prediction of polyp lesions in the entire endoscopic video data.

[0063] The method provided in the above embodiments of the present invention acquires endoscopic images in real time and performs lesion risk prediction, providing real-time and efficient prediction results for polyp lesion risk. The detection results can assist doctors in improving clinical diagnostic efficiency, reducing patient suffering, and alleviating the pressure on medical staff. To reduce false positives, the method specifically divides the magnified endoscopic image into multiple small blocks and filters out invalid areas to avoid interfering with the final prediction.

[0064] In some preferred embodiments, the above-mentioned S1 may further include the following operation:

[0065] S11, crop the output image of the endoscope device into multiple 256×256 small blocks;

[0066] S12, convert all small blocks to HSV space, and determine whether each pixel belongs to the shadow area in HSV space;

[0067] S13. If the proportion of the shaded area in any small patch is greater than or equal to the set proportion threshold, then the small patch is considered to be an invalid small patch image and is discarded to avoid interference with the prediction of the whole image.

[0068] S14. If the proportion of the shaded area in any small block is less than the set proportion threshold, then the small block is considered to be a valid small block image and is retained for risk prediction.

[0069] In this preferred embodiment, the preprocessing of the acquired endoscopic images mainly includes cropping into multiple small image patches and image patch filtering. In the raw image data, regions of interest crucial for identification are usually limited to specific areas in the image, which are often accompanied by interfering factors such as shadows, blur, and bubbles. However, these interfering areas do not contribute to the prediction of pathological results. To refine the selection of target regions, the original image is cropped into multiple image patches and invalid regions are filtered out. Specifically, firstly, each original image is cropped into multiple image patches of size 256×256, and the relative positional relationship between these image patches is recorded. Secondly, the image patches are converted to the HSV color space, and the proportion of black pixels is analyzed to evaluate the proportion of shadow areas within the image patch, and image patches with excessive invalid areas are discarded to avoid interference with the prediction of the entire image.

[0070] In some preferred embodiments, the above-mentioned S2 may further include the following operation:

[0071] Each small image patch is treated as a node. The relative positions of the small images are determined by connecting them with edges if they are in direct contact, and by not connecting them with edges. This creates a graph data structure with small images as nodes.

[0072] In this preferred embodiment, since the relative positions of different image blocks from the same original image are recorded during the cropping process, graph data can be constructed based on the position information. Specifically, each image block is used as a node, and adjacent image blocks are connected by edges to construct graph data.

[0073] In some preferred embodiments, the above-described S3 may further include the following operations:

[0074] S31 provides a pre-trained endoscopic polyp lesion prediction model, which includes a pre-trained feature extraction model and a pre-trained graph neural network model.

[0075] S32, the graph data is processed by a feature extraction model to extract features, and feature vectors corresponding to each small image block are obtained in sequence. The information of the feature vectors is carried by each node, resulting in graph data containing feature information.

[0076] S33 inputs graph data containing feature information into the graph neural network model and outputs the lesion risk prediction result for each node, that is, the lesion risk prediction result for each small image block.

[0077] In some preferred embodiments, the above-mentioned S31 may further include the following operations:

[0078] S311, the pre-trained feature extraction model is trained using massive amounts of endoscopic video data to extract local features from endoscopic images;

[0079] S312, the pre-trained graph neural network model uses training image patches with texture features to train a graph data classifier constructed by graph convolutional layers, which is used to predict the results of image features.

[0080] In some preferred embodiments, the above-mentioned graph neural network model comprises two graph convolutional layers and a fully connected layer; wherein, after each graph convolutional layer, a batch normalization layer and a ReLU activation function are applied, and a dropout layer is added between the two graph convolutional layers; the output of the fully connected layer is 2-dimensional, corresponding to the prediction confidence of lesions and non-lesions respectively. The graph neural network model implements a graph data classifier through graph convolutional layers, which can classify and predict each node in the graph, thereby obtaining the lesion risk prediction result for each small patch of image.

[0081] In some preferred embodiments, the above-mentioned S312 may further include the following operation:

[0082] S3121 involves professional doctors manually marking out areas in training images that facilitate lesion risk diagnosis and filtering out irrelevant areas.

[0083] S3122, crop the bounding rectangle of the labeled area to obtain training image patches of the same shape and size;

[0084] S3123 involves professional doctors assigning labels to training image blocks based on their expertise and pathological findings, resulting in training image blocks with distinct texture features.

[0085] In the preferred embodiment described above, the endoscopic polyp prediction model comprises two parts: a feature extractor and a feature classifier. To effectively extract features from NBI images, in a preferred embodiment, the feature extractor uses a large-scale Endo endoscopic model trained under self-supervised conditions on massive endoscopic image data. Each small image patch is input into the feature extractor to obtain the corresponding feature vector, thereby capturing the information of the small image patch in the feature space. The feature classifier can use a linear or nonlinear multilayer perceptron. To improve the overall image judgment accuracy, a graph neural network is preferred, capable of constructing a graph based on the relative positions between image patches and fusing the positional information of the image patches in the feature space for judgment. The graph neural network classifier consists of two graph convolutional layers and one fully connected layer. After each graph convolutional layer, a batch normalization layer and a ReLU activation function are applied. To improve the model's generalization ability and prevent overfitting during training, a dropout layer is added between the two graph convolutional layers. The final output of the fully connected layer of the graph neural network is 2-dimensional, corresponding to the prediction confidence of cancerous and non-cancer lesions in the image, respectively.

[0086] In some preferred embodiments, the above-described S4 may further include the following operations:

[0087] S41, When the prediction results of all small images are negative, the prediction result of polyp lesions of the entire output image is low risk;

[0088] S42, if any one or more of the small image patches have a positive prediction result, then the polyp lesion prediction result of the entire output image is high risk.

[0089] In some preferred embodiments, the above-mentioned S5 may further include the following operation:

[0090] S51, extract each frame of the endoscopic video data, randomly sample several small images of the same size in each frame, establish graph data based on the relative positional relationship between the small images, and use a pre-trained endoscopic polyp lesion prediction model to predict the lesion risk of the graph data, and obtain the lesion risk prediction result of each frame.

[0091] S52, a video segment with several consecutive high-risk frames is considered as a high-risk lesion segment; when the number of consecutive high-risk lesion segments in the video data exceeds a set threshold, the video data is judged as high-risk lesion; otherwise, it is judged as low-risk lesion.

[0092] Based on the same inventive concept, one embodiment of the present invention also provides a system for predicting the risk of polyp lesions in endoscopic images.

[0093] Specifically, such as Figure 2 As shown, the endoscopic imaging polyp risk prediction system provided in this embodiment may include the following modules:

[0094] The image preprocessing module is used to crop the output image of the endoscope into multiple small blocks and filter out invalid small blocks to obtain small block images;

[0095] The graph data construction module is used to build graph data based on the relative positional relationships between small image patches.

[0096] The image risk prediction module provides a pre-trained endoscopic polyp lesion prediction model. The established image data is input into the endoscopic polyp lesion prediction model, and the prediction results are output to obtain the lesion risk prediction results for each small image patch. The prediction results of each small image patch are combined to obtain the polyp lesion prediction results for the entire output image.

[0097] In some preferred embodiments, the above system may further include the following modules:

[0098] The video risk prediction module, based on the image preprocessing module, the graph data construction module, and the image risk prediction module, performs segment risk prediction on endoscopic video data to obtain the risk prediction of polyp lesions in the entire endoscopic video data.

[0099] In the above embodiments, the output images of the endoscopic device can be acquired using a capture card. For example, the capture card connects the endoscopic imaging device and a computer. The computer has a capture card driver installed, and the endoscopic images acquired by the endoscopic imaging device are read by calling the interface in the capture card SDK. The capture card SDK interface is called to check the availability of the input signal, and in each cycle of program execution, it checks whether there are new input video frames in the video input buffer. If so, the new video frames are converted into 8-bit RGB images for each channel, and then the images are sent to the preprocessing stage.

[0100] It should be noted that the steps in the method provided by the present invention can be implemented using the corresponding components in the system. Those skilled in the art can refer to the technical solution of the system to implement the steps of the method, and can also refer to the technical solution of the method to implement the composition of the system. That is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, which will not be elaborated here.

[0101] To better understand the technical solution of this invention, the specific operation process is explained below with reference to a specific application example. It should be understood that this specific example is not intended to limit this invention.

[0102] Reference Figure 3 As shown, the hardware used in this specific application example includes: an endoscopic imaging device, a data acquisition card, a computer, and a display device. The output interface of the endoscopic imaging device is connected to the input interface of the data acquisition card, the output interface of the data acquisition card is connected to the input interface of the computer, and the output interface of the computer is connected to the input interface of the display device. When the software is not used, the output interface of the endoscopic imaging device can also be directly connected to the input interface of the display device.

[0103] In this specific application example, the endoscopic image feature extractor is built based on a deep convolutional neural network, with the structure being the Endo large model (Wang Z, Liu C, Zhang S, et al. Foundation model for endoscopy videoanalysis via large-scale self-supervised pre-train[C]. International Conference on Medical Image Computing and Computer-Assisted Intervention. 2023), specifically implemented using Python and the PyTorch framework. The training data for this model was labeled from raw endoscopic images by professionals using the labelme software (version 4.6.0). Regions with significant intestinal feature patterns were mostly irregular polygonal areas. The bounding rectangles of these irregular regions were extracted, cropped into 256×256 image patches, and the relative positional relationships between image patches from the same image were recorded. The image patches were converted to the HSV (hue, saturation, value) color space, the proportion of black pixels was calculated, and regions with shadow areas greater than a certain threshold were filtered out to obtain an image patch dataset containing smaller shadow areas. Since the number of positive samples in the training set is far less than the number of negative samples, focus loss is used for training to mitigate the impact of data imbalance. The network parameters are optimized using an SGD optimizer with an initial learning rate of 0.01, momentum of 0.9, and weight decay of 0.0005. The network is trained for a total of 300 iterations, with the learning rate reduced by a factor of 0.2 at iterations 100 and 150. Preprocessed small image patches are input into the Endo large model to obtain their corresponding feature vectors.

[0104] In this specific application example, the endoscopic image feature classifier is built based on a graph neural network, as shown in Table 1. Here, bs represents the batch size, conv represents a graph convolutional layer, bn represents a normalization layer, relu represents a ReLU activation layer, dropout represents a dropout layer, and fc represents a fully connected layer. This classifier combines the positional information between image patches, constructing a graph with each image patch as a node, and outputting a prediction for each node. The final prediction of the entire image is obtained by voting on the results of each image patch: if any image patch is predicted positive, then the prediction of the entire image is positive; if all image patches are predicted negative, then the prediction of the entire image is negative.

[0105] Table 1: Graph Neural Network Structure

[0106] number of floors structure enter Output 0 conv bs*768 bs*384 1 bn bs*384 bs*384 2 relu bs*384 bs*384 3 dropout bs*384 bs*384 4 conv bs*384 bs*96 5 bn bs*96 bs*96 6 relu bs*96 bs*96 7 fc bs*96 bs*2

[0107] Ideally, Focal Loss should be used during training of the graph neural network. Focal Loss is a loss function used to address class imbalance, particularly suitable for object detection and image segmentation tasks. Traditional cross-entropy loss functions suffer from problems handling imbalanced data because they contribute less to samples from the minority class during training, making it difficult for the model to effectively learn features from these classes. Focal Loss improves upon the imbalance handling capabilities of the cross-entropy loss function by introducing an adjustable balancing parameter and a correction term. The correction term reduces the weight of easily classified samples, causing the model to focus more on difficult samples, thus improving classification performance for the minority class. Focal Loss is simple yet effective, helping to improve model performance in imbalanced scenarios and achieving significant improvements in many computer vision tasks. Its specific form is as follows:

[0108] FL(p)=-α(1-p) γ log(p)

[0109] Here, 1-p represents the difficulty of classifying the sample. When p approaches 1, it indicates a high confidence level in the current prediction of the sample, a good classification, and that the sample is easily distinguishable. In this case, 1-p approaches 0, and the sample contributes little to the loss. When p is very small, 1-p approaches 1, and has little impact on the loss. 1-p can reduce the contribution of easily distinguishable samples to the loss, α can suppress the imbalance between positive and negative samples, and γ can control the imbalance between easily distinguishable and difficult-to-distinguish samples.

[0110] Reference Figure 4 The diagram shows a flowchart of the polyp diagnosis method in this application example. The specific steps are as follows:

[0111] Step 1: Initialize the endoscopic polyp lesion prediction model;

[0112] Step 2: Check if there is a new video frame (output image of the endoscope device) in the video input buffer. If there is, proceed to step 3; if not, repeat step 2.

[0113] Step 3: Preprocess the input image;

[0114] Step 4: Input the preprocessed image into the endoscopic polyp prediction model;

[0115] Step 5: Based on the results returned in Step 4, display the image and prediction results through the video display window;

[0116] In step 2 of the above specific application example, the acquisition of the output image of the endoscope device can be achieved through a video signal acquisition thread. Video signal acquisition is implemented through a separate thread. During the initialization phase, the driver interface is used to check whether the acquisition card hardware is properly connected. Based on the signal source set by the user, the acquisition card hardware is configured by calling the underlying interface. The format of the video input signal is obtained by calling the underlying driver program. According to the format of the video signal, an input buffer is reserved in memory, the effective data length of the buffer is initialized to zero, and the base address of the buffer is transmitted to the driver program. In subsequent operation, the underlying driver program continuously transmits the video frames acquired by the video acquisition card to memory according to the previous configuration. Whenever a new video frame is available, the signal acquisition thread stores the video in the reserved input buffer and updates the effective data count in the buffer.

[0117] Considering the practical needs of endoscopic examination, the performance of the endoscopic polyp prediction model on video data was further investigated. The video dataset consisted of 159 short videos, each running at 30 frames per second, with durations ranging from 7 to 9 seconds. Of the 159 videos, 41 contained cancerous polyps, while the remaining 118 videos contained only non-cancerous polyps. The raw video data was initially divided into individual frames for image extraction from each frame. Subsequently, five regions (each 512×768 pixels) were randomly sampled from different locations across the entire image in each frame. For each region, it was divided into six 256×256 image patches, and features were extracted using the base model. Graph data was constructed based on the feature vectors and relative positional connections between various image patches within each region. Predictions were then made using the proposed polyp risk prediction method. The prediction results for a single image frame were based on a combined prediction from the five sampled regions. If any region was identified as a cancerous polyp, the frame was labeled as a cancerous frame. To minimize the randomness of predictions for a single frame, a video is classified as cancerous only if at least three consecutive frames are classified as cancerous, meaning that the patient's intestines have a high risk of becoming cancerous.

[0118] Quantitative Experiment Results: This example uses 514 images as the training set, including 107 images of cancerous tissue and 397 images of non-cancer tissue. It also uses 88 images as the test set, including 21 images of cancerous tissue and 67 images of non-cancer tissue. The metrics used are as follows:

[0119]

[0120] Among them, TP (True Positive), TN (True Negative), FP (False Positive), and FN (False Negative) represent the number of true positive, true negative, false positive, and false negative samples, respectively.

[0121] Table 2 shows the results of this example for diagnosing lesion risk in small image patches. The model was trained on 1757 image patches, and the performance was evaluated on 439 image patches. Endo-LaG refers to the method using a large endoscopic Endo model as a feature extractor and a graph neural network as a feature classifier; CNN refers to the method using a convolutional neural network ResNet18 (He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C].CVPR.2016); Endo-L refers to the method using a large endoscopic Endo model as a feature extractor and a linear multilayer perceptron as a feature classifier; and Endo-NL refers to the method using a large endoscopic Endo model as a feature extractor and a nonlinear multilayer perceptron as a feature classifier. As can be seen from the table, the traditional CNN model has limitations in polyp risk prediction. Although the accuracy reaches 0.808, the recall and AUC are very low. The limited dataset size and significant imbalance between negative and positive samples make CNNs prone to overfitting during training. Well-trained CNN models tend to classify samples as negative, resulting in high accuracy but missing a large number of positive samples. Furthermore, CNNs struggle to extract effective information for prediction. This can lead to the model overemphasizing irrelevant or repetitive features, resulting in poor generalization to new samples. Endo-L and Endo-NL models outperform CNN models across all metrics. This indicates that using large models to extract features can effectively identify key information in endoscopic images. Both models achieve higher recall than CNN models. The Endo-L model exhibits higher recall compared to the Endo-NL model, suggesting that nonlinear classifiers are more prone to overfitting during training. The preferred method of the above embodiments of the present invention, Endo-LaG, outperforms traditional methods across all metrics, particularly achieving very high recall while maintaining high accuracy. This is of considerable importance in real-world clinical scenarios, as the implementation of the method provided by the above embodiments of the present invention ensures minimal supervision of lesions or polyps, leading to a significant improvement in patient survival.

[0122] Table 2: Diagnostic Results of Image Patch Lesion Risk

[0123]

[0124]

[0125] Table 3 shows the results of whole-image lesion risk diagnosis in this example. The model was trained on 327 images, and the performance evaluation was conducted on 77 images. Experimental results show that the accuracy of the three traditional methods is similar, but the recall of CNN is significantly lower. As mentioned before, this difficulty is due to the limited ability of CNN models to extract basic information from images, leading to a higher sensitivity to overfitting, especially when dealing with small datasets. Compared with the three traditional methods, the method proposed in the above embodiments of the present invention also achieves better performance in predicting the entire image. Furthermore, it is worth noting that although the same method is used, a higher recall can be obtained when predicting the entire image than when predicting image patches. This is because the prediction result of the entire image is based on information collected from different small image patches. If any image patch is predicted as high-risk for disease, then the entire image containing that patch will also be predicted as high-risk for disease. This enhances the ability of the method proposed in the above embodiments of the present invention to detect high-risk cases of serious diseases during the prediction process, enabling patients to receive timely treatment.

[0126] Table 3: Results of Disease Risk Diagnosis for the Entire Image

[0127]

[0128] This specific application example invited six endoscopists to evaluate the test dataset and record the time they spent. The six endoscopists were divided into two groups based on their years of experience: three experienced endoscopists (over ten years of experience) and three younger endoscopists (less than ten years of experience). The test results for all doctors are shown in Table 4. Without AI assistance, the experienced doctors achieved a precision of 0.738, a recall of 0.654, an FPR of 0.226, a precision of 0.571, and average processing times of 7 minutes and 55 seconds. In contrast, the younger doctors achieved a precision of 0.792, a recall of 0.744, an FPR of 0.188, a precision of 0.625, and average processing times of 11 minutes and 45 seconds. The performance of these doctors on the ROC curve was plotted using the method proposed in the above embodiments of the present invention. Figure 5As shown in the figure, the average performance of these doctors was lower than the ROC curve, indicating that the method proposed in the above embodiments of the present invention is superior to the experience of doctors. With the help of artificial intelligence, experienced doctors achieved an accuracy of 0.875, a recall of 0.884, an FPR of 0.129, a precision of 0.762, and average processing times of 6 minutes and 35 seconds. In contrast, younger doctors achieved an accuracy of 0.864, a recall of 0.821, an FPR of 0.118, a precision of 0.746, and average processing times of 7 minutes and 59 seconds. The results show that, in terms of utility, using AI assistance can significantly improve the diagnostic accuracy of doctors and reduce the occurrence of missed diagnoses. In terms of efficiency, diagnosis without AI assistance, especially for younger doctors, can be a time-consuming process, while AI assistance significantly reduces their diagnostic time.

[0129] Table 4: Prediction Results for Doctors with and without Model Assistance Table of Predictive Results for Doctors of Different Qualifications

[0130]

[0131] On the video dataset, this example (Endo-LaG) achieved an accuracy of 0.767, a recall of 0.732, an FPR of 0.220, and a precision of 0.536. In 41 videos containing cancerous polyps, the method provided in the above embodiments of the present invention accurately predicted 30 of them. Three conventional models (CNN, Endo-L, Endo-NL) were tested under the same experimental settings, and their detection rate for cancerous samples was almost zero.

[0132] Figure 6 The above specific application examples use the method and system provided in the above embodiments of the present invention to predict image data as cancerous and non-cancerous; wherein, (a) a group of images are non-cancerous images, with regular distribution of blood vessel morphology and a conventional tubular surface; (b) a group of images are cancerous images, with irregular distribution of blood vessel morphology, a loose surface, and a very blurry surface in amorphous areas.

[0133] An embodiment of the present invention also provides an endoscopic polyp risk prediction terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to execute any of the methods in the above embodiments of the present invention, or to run any of the systems in the above embodiments of the present invention.

[0134] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.

[0135] A processor is used to execute computer programs stored in memory to implement the various steps of the methods or various modules of the systems involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method and system embodiments.

[0136] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0137] An embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to perform the method of any of the above embodiments of the present invention, or to run the system of any of the above embodiments of the present invention.

[0138] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one place to another. Storage media can be any available medium accessible to a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Alternatively, the ASIC can reside in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0139] The method, system, and terminal for predicting the risk of polyps in endoscopic images provided in the above embodiments of the present invention acquire the output image of the endoscopic device and preprocess the input image, cropping the image into multiple 256×256 small blocks and filtering out invalid small blocks to avoid interference with the overall image prediction; calling the endoscopic polyp prediction model to predict each small block image; and combining the prediction results of each small block to obtain the polyp diagnosis result for the entire endoscopic image. This invention can diagnose polyps based on endoscopic images and videos, and output the cancer risk prediction result for doctors' reference. It has the advantages of simple operation and no technical threshold for use. By dividing the endoscopic image into multiple small blocks for prediction, high-risk areas in the endoscopic image can be simultaneously indicated, facilitating doctors' observation. Because a large model is used as the base model to extract endoscopic image features, the accuracy is high and its clinical applicability is strong.

[0140] Those skilled in the art will understand that, in addition to implementing the system and its various devices provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices of this invention function as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0141] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0142] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.

Claims

1. A method for predicting the risk of polypoid lesions in endoscopic images, characterized in that, include: The output image from the endoscope is cropped into multiple small blocks, and invalid blocks are filtered out to obtain small block images; Graph data is created based on the relative positional relationships between small image blocks; A pre-trained endoscopic polyp lesion prediction model is provided. The established image data is input into the endoscopic polyp lesion prediction model, and the prediction results are output to obtain the lesion risk prediction results for each small image patch. By combining the prediction results of each small image patch, the prediction result of polyp lesions for the entire output image is obtained; The step of cropping the output image of the endoscopic device into multiple small blocks and filtering out invalid blocks includes: The output image from the endoscope is cropped into multiple 256×256 blocks; Convert all small blocks to HSV space, and determine whether each pixel belongs to the shadow area in the HSV space; If the proportion of the shaded area in any small patch is greater than or equal to a set threshold, then the small patch is considered an invalid small patch image and is discarded. If the proportion of the shaded area in any small patch is less than a set threshold, then the small patch is considered a valid small patch image and is retained for risk prediction. The process of establishing graph data based on the relative positional relationships between small image patches includes: Each small image patch is treated as a node. Based on the relative positional relationship between the small images, small images that are in direct contact with each other are connected by edges, while small images that are not in direct contact with each other are not connected by edges, thus creating graph data with small images as nodes. The system provides a pre-trained endoscopic polyp prediction model, inputs the established graphical data into the endoscopic polyp prediction model, and outputs prediction results, including: A pre-trained endoscopic polyp lesion prediction model is provided, the endoscopic polyp lesion prediction model comprising: a pre-trained feature extraction model and a pre-trained graph neural network model; The graph data is processed by the feature extraction model to extract features, thereby obtaining feature vectors for each small image block. The information of the feature vectors is carried by each node, resulting in graph data containing feature information. The graph data containing feature information is input into the graph neural network model, and the lesion risk prediction result for each node is output, that is, the lesion risk prediction result for each small image is obtained. The pre-trained feature extraction model is trained using massive amounts of endoscopic video data to extract local features from endoscopic images. The pre-trained graph neural network model is trained using training image patches with texture features and is used to predict results based on image features. The graph neural network model comprises two graph convolutional layers and a fully connected layer. After each graph convolutional layer, a batch normalization layer and a ReLU activation function are applied, and a dropout layer is added between the two graph convolutional layers. The output of the fully connected layer is 2-dimensional, corresponding to the prediction confidence of lesions and non-lesions, respectively. The training image patch with texture features is obtained in the following way: The training images are manually annotated by professional doctors to facilitate the diagnosis of lesion risks, while irrelevant areas are filtered out. The bounding rectangle of the labeled area is cropped to obtain training image patches of the same shape and size; Professional doctors, combining their expertise and pathological findings, assign labels to the training image blocks to obtain training image patches with texture features. The combined prediction results of each small image patch include: If the prediction results of all small images are negative, the prediction result of polyp lesions of the whole output image is low risk; if the prediction results of any one or more small images are positive, the prediction result of polyp lesions of the whole output image is high risk. It also includes: based on the above process, performing segment risk prediction on endoscopic video data to obtain a risk prediction of polyp lesions in the entire endoscopic video data; in: The endoscopic video data is used for segment risk prediction to obtain a prediction of the risk of polyp lesions in the entire endoscopic video data, including: Each frame of the endoscopic video data is extracted, and several small images of the same size are randomly sampled from each frame. The graph data is established based on the relative positional relationship between the small images, and the lesion risk is predicted on the graph data using a pre-trained endoscopic polyp lesion prediction model to obtain the lesion risk prediction result for each frame. A video segment with several consecutive high-risk frames is considered as a high-risk lesion segment; if the number of consecutive high-risk lesion segments in the video data exceeds a set threshold, the video data is judged as high-risk lesion; otherwise, it is judged as low-risk lesion.

2. A system for predicting the risk of polypoid lesions in endoscopic images for implementing the method of claim 1, characterized in that, include: The image preprocessing module is used to crop the output image of the endoscope into multiple small blocks and filter out invalid small blocks to obtain small block images; The graph data construction module is used to build graph data based on the relative positional relationships between small image patches. The image risk prediction module provides a pre-trained endoscopic polyp lesion prediction model. The established image data is input into the endoscopic polyp lesion prediction model, and the prediction results are output to obtain the lesion risk prediction results for each small image patch. By combining the prediction results of each small image patch, the prediction result of polyp lesions for the entire output image is obtained; Also includes: The video risk prediction module, based on the image preprocessing module, the image data construction module, and the image risk prediction module, performs segment risk prediction on the endoscopic video data to obtain the risk prediction of polyp lesions in the entire endoscopic video data.

3. An endoscopic polyp lesion risk prediction terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it can be used to perform the method of claim 1, or to run the system of claim 2.

Images