Method, system and terminal for predicting polyp lesion risk in endoscope image

Abstract

The invention provides a polyp lesion risk prediction method and system in an endoscope image, and the method comprises the steps: cutting an output image of endoscope equipment into a plurality of small blocks, filtering out invalid small blocks, and obtaining small block images; graph data are established according to the relative position relation between the small images; inputting the image data into an endoscope polyp lesion prediction model, outputting a prediction result to obtain a lesion risk prediction result of each small image, and integrating the prediction results to obtain a polyp lesion prediction result of the whole output image; and based on the process, performing fragment risk prediction on the endoscope video data to obtain polyp lesion risk prediction on the video data. The method is simple to operate and has no technical threshold in use. The endoscope image is divided into a plurality of small blocks for prediction, and high-risk areas in the endoscope image can be indicated at the same time. A large model is adopted as a basic model to extract endoscope image features, the accuracy is high, and the clinical practicability is high.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a method, system and terminal for predicting the risk of polyp lesions in endoscopic images based on NBI magnified images. Background Art

[0002] In recent years, deep learning technology has developed rapidly and is widely used in the field of medical image analysis. Endoscopic images are an important basis for doctors to analyze patients' digestive tract diseases. Many studies have applied deep convolutional neural networks (CNNs) to the screening and detection of lesions, achieving remarkable results and having important clinical significance.

[0003] Clinically, endoscopists face difficulties in identifying and detecting suspicious lesions based on their experience. With the widespread adoption of AI-assisted diagnosis, its research and application in the early diagnosis of colorectal cancer has received significant attention and development in recent years. Currently, a significant amount of research is focused on using AI to assist in the real-time detection of intestinal polyps to improve the detection rate of polyps during colonoscopy. For polyps at risk of cancer, clinically, endoscopists are required to collect pathological data for further testing. However, the observation time after colonoscopy is typically only 6-7 minutes. This is limited by the operating physician's work status and experience, making it easy to miss potentially cancerous polyps, which can lead to missing the patient's golden treatment window.

[0004] The currently disclosed related colonoscopy auxiliary systems and methods only focus on how to identify whether the images collected by the operating physician contain lesions and how to improve the detection rate of polyps, but lack the technology to predict the risk of these polyps becoming cancerous. Due to problems such as the lack of unique texture features of colon polyps, it is challenging to directly use traditional convolutional neural networks to predict the risk of polyp lesions. Therefore, there is currently a lack of research on analyzing colonoscopy examinations based on NBI magnified images, and there is no more effective model architecture. For this reason, it is imperative to develop an endoscopic polyp lesion diagnosis method based on NBI magnified images. At present, no description or report of similar technology to the present invention has been found, and similar information at home and abroad has not yet been collected. Summary of the Invention

[0005] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a method, system and terminal for predicting the risk of polyp lesions in endoscopic images based on NBI magnified images.

[0006] According to one aspect of the present invention, a method for predicting the risk of polyps in endoscopic images is provided, comprising:

[0007] The output image of the endoscope device is cropped into multiple small blocks, and invalid small blocks are filtered out to obtain small block images;

[0008] Create graph data based on the relative position relationship between small image blocks;

[0009] Providing a pre-trained endoscopic polyp lesion prediction model, inputting the established image data into the endoscopic polyp lesion prediction model, and outputting a prediction result to obtain a lesion risk prediction result for each small image block;

[0010] The prediction results of each small image block are combined to obtain the polyp lesion prediction result of the entire output image.

[0011] Preferably, the step of cropping the output image of the endoscope device into a plurality of small blocks and filtering out invalid small blocks comprises:

[0012] The output image of the endoscope device is cropped into multiple 256×256 small blocks;

[0013] Convert all small blocks to HSV space, and determine whether each pixel belongs to the shadow area in the HSV space;

[0014] If the shadow area ratio in any small block is greater than or equal to the set ratio threshold, the small block is considered to be an invalid small block image and discarded;

[0015] If the proportion of the shadow area in any small block is less than the set ratio threshold, the small block is considered to be a valid small block image and is retained for risk prediction.

[0016] Preferably, the step of establishing the graph data according to the relative positional relationship between the small image blocks includes:

[0017] Each small image block is regarded as a node. Based on the relative position relationship between the small image blocks, the small image blocks that are directly in contact with each other are connected by edges, while the small image blocks that are not directly in contact with each other are not connected by edges. Graph data with small image blocks as nodes is established.

[0018] Preferably, the method provides a pre-trained endoscopic polyp lesion prediction model, inputs the established image data into the endoscopic polyp lesion prediction model, and outputs a prediction result, including:

[0019] A pre-trained endoscopic polyp lesion prediction model is provided, wherein the endoscopic polyp lesion prediction model includes: a pre-trained feature extraction model and a pre-trained graph neural network model;

[0020] Performing feature extraction on the graph data through the feature extraction model to obtain feature vectors corresponding to each small image block in sequence, wherein information of the feature vectors is carried by each node, thereby obtaining graph data containing feature information;

[0021] The graph data containing the feature information is input into the graph neural network model, and the lesion risk prediction result for each node is output, that is, the lesion risk prediction result for each small block of the image is obtained.

[0022] Preferably, the pre-trained feature extraction model is trained using massive endoscopic video data to extract local features of endoscopic images.

[0023] Preferably, the pre-trained graph neural network model is trained using training image blocks with texture features to predict results of image features.

[0024] Preferably, the graph neural network model comprises: two graph convolution layers and one fully connected layer; wherein, after each graph convolution layer, a batch normalization layer and a ReLU activation function are applied, and a dropout layer is added between the two graph convolution layers; the output of the fully connected layer is 2-dimensional, corresponding to the prediction confidence of lesions and non-lesions, respectively.

[0025] Preferably, the training image block with texture features is obtained by:

[0026] Professional doctors manually mark the areas in the training images that are convenient for lesion risk diagnosis and filter out irrelevant areas;

[0027] Crop the circumscribed rectangle of the marked area to obtain training image blocks of consistent shape and size;

[0028] Professional doctors set labels for the training image blocks based on professional knowledge and pathological results to obtain training image blocks with texture features.

[0029] Preferably, the synthesizing the prediction results of each small image block includes:

[0030] When the prediction results of all small image blocks are negative, the polyp lesion prediction result of the entire output image is low risk. If the prediction results of any one or more small image blocks are positive, the polyp lesion prediction result of the entire output image is high risk.

[0031] Preferably, the above method further comprises: based on the above process, performing segment risk prediction on the endoscopic video data to obtain a risk prediction of polyp lesions in the entire endoscopic video data.

[0032] Preferably, the endoscopic video data is subjected to segment risk prediction to obtain a risk prediction of polyp lesions in the entire endoscopic video data, including:

[0033] Extract each frame of endoscopic video data, randomly sample several small blocks of the same size from each frame, establish graph data based on the relative positional relationship between the small blocks, and use a pre-trained endoscopic polyp lesion prediction model to perform lesion risk prediction on the graph data to obtain a lesion risk prediction result for each frame of image;

[0034] A video segment in which several consecutive frames are high-risk is regarded as a high lesion risk segment; when the number of consecutive high lesion risk segments in the video data exceeds the set threshold, the video data is judged as high lesion risk, otherwise it is judged as low lesion risk.

[0035] According to another aspect of the present invention, a system for predicting the risk of polyps in endoscopic images is provided, comprising:

[0036] An image preprocessing module is used to crop the output image of the endoscope device into multiple small blocks and filter out invalid small blocks to obtain small block images;

[0037] A graph data construction module, which is used to create graph data based on the relative position relationship between small image blocks;

[0038] An image risk prediction module provides a pre-trained endoscopic polyp lesion prediction model, inputs the established image data into the endoscopic polyp lesion prediction model, and outputs the prediction results to obtain the lesion risk prediction results for each small image block; and integrates the prediction results of each small image block to obtain the polyp lesion prediction results for the entire output image.

[0039] Preferably, the above system further includes:

[0040] The video risk prediction module performs segment risk prediction on the endoscopic video data based on the image preprocessing module, the graph data construction module and the image risk prediction module to obtain a risk prediction of polyp lesions in the entire endoscopic video data.

[0041] According to a third aspect of the present invention, an endoscopic polyp lesion risk prediction terminal is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any one of the above-mentioned methods of the present invention, or to execute any one of the above-mentioned systems of the present invention, when executing the computer program.

[0042] According to a third aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it can be used to perform any of the above methods of the present invention, or to run any of the above systems of the present invention.

[0043] Due to the adoption of the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art:

[0044] This invention utilizes a large endoscope model for feature extraction. This large model, trained on massive amounts of endoscope video data, is highly capable of extracting local features from endoscopic images. Furthermore, to train the graph neural network and integrate information from different locations within the endoscopic image, resulting in more accurate risk prediction, the system utilizes training image blocks incorporating specialized knowledge. This exceptional large model and high-quality training data enable this invention to achieve predictive results in lesion risk diagnosis that surpass those achieved by medical experts.

[0045] The training data set in the present invention is manually marked by an expert system or professionals to identify areas with significant intestinal surface structures and blood vessels in the endoscopic images. Using such a high-quality data set for training can further improve the accuracy of lesion risk prediction.

[0046] The present invention can perform manual polyp lesion diagnosis on input magnified endoscopic images, outputting polyp lesion risk data to assist medical personnel in making these assessments. Furthermore, the present invention crops the magnified endoscopic image into multiple small blocks, performs separate predictions on each block, and then synthesizes the prediction results. This block-by-block prediction approach provides information on the risk of each region, making it easier for doctors to locate and observe high-risk areas within the entire image. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0048] Figure 1 This is a general flow chart of a method for predicting the risk of polyps in endoscopic images in a preferred embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram of the components of a system for predicting polyp lesion risk in endoscopic images in a preferred embodiment of the present invention;

[0050] Figure 3 This is a diagram illustrating the architecture of a system for predicting polyp risk in endoscopic images in a specific application example of the present invention;

[0051] Figure 4 This is a conceptual diagram of the workflow of a method for predicting the risk of polyps in endoscopic images in a specific application example of the present invention;

[0052] Figure 5 This is a comparison chart of the recall rate predicted by the method and system provided by the present invention and that by endoscopists in a specific application example;

[0053] Figure 6The following is a specific example of image data predicted as cancerous and non-cancerous using the method and system provided by the present invention. Among them, one group of images in (a) are non-cancerous images, with regularly distributed blood vessels and a regular tubular surface; the other group of images in (b) are cancerous images, with irregularly distributed blood vessels, a loose surface, and a very blurred surface in the amorphous area. DETAILED DESCRIPTION

[0054] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments will help those skilled in the art further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements may be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0055] In response to the problem that there is a lack of research in the existing technology on analyzing colonoscopy examinations based on NBI magnified images and there is no more effective model architecture, an embodiment of the present invention provides a method for predicting the risk of polyp lesions in endoscopic images. This method provides a lesion risk prediction framework in medical endoscopic images, uses a large model to extract features and train a classifier for classification, provides assistance for the judgment of cancer risk in hospital endoscopic magnified images, improves the judgment accuracy, and shortens the judgment time.

[0056] Specifically, if Figure 1 As shown, the method for predicting the risk of polyps in endoscopic images provided in this embodiment may include the following operations:

[0057] S1, preprocessing the output image of the endoscope device: cropping the output image of the endoscope device into multiple small blocks, and filtering out invalid small blocks to obtain small block images;

[0058] S2, establish graph data based on the relative position relationship between small image blocks;

[0059] S3, providing a pre-trained endoscopic polyp lesion prediction model, inputting the established image data into the endoscopic polyp lesion prediction model, and outputting the prediction results to obtain the lesion risk prediction results for each small image block;

[0060] S4, combining the prediction results of each small image block to obtain the polyp lesion prediction result of the entire output image.

[0061] In some preferred embodiments, the above method may further include the following operations:

[0062] S5. Based on the above process, segment risk prediction is performed on the endoscopic video data to obtain a risk prediction of polyp lesions in the entire endoscopic video data.

[0063] The method provided by the above-mentioned embodiment of the present invention acquires endoscopic images and performs lesion risk prediction in real time, effectively providing real-time prediction results for polyp lesion risk. The test results can help doctors improve clinical diagnostic efficiency, alleviate patient suffering, and relieve the pressure on medical staff. To reduce false positive predictions, this method specifically divides the magnified endoscopic image into multiple small blocks and filters out invalid areas to prevent interference with the final prediction.

[0064] In some preferred embodiments, the above S1 may further include the following operations:

[0065] S11, cropping the output image of the endoscope device into multiple 256×256 small blocks;

[0066] S12, converting all small blocks into HSV space, and determining whether each pixel belongs to the shadow area in the HSV space;

[0067] S13, if the shadow area ratio in any small block is greater than or equal to the set ratio threshold, the small block is considered to be an invalid small block image and is discarded to avoid interference with the prediction of the entire image;

[0068] S14: If the proportion of the shadow area in any small block is less than the set ratio threshold, the small block is considered to be a valid small block image and is retained for risk prediction.

[0069] In this preferred embodiment, the preprocessing of the collected endoscopic images mainly includes cropping into multiple small image blocks and image block filtering. In the original image data, the regions of interest that are critical for identification are usually limited to specific areas in the image, and these areas are often accompanied by interference factors such as shadows, blurs, and bubbles. However, these interference areas do not contribute to the prediction of pathological results. In order to refine the selection of the target area, the original image is cropped into multiple image blocks and invalid areas are filtered out. Specifically, each original image is first cropped into multiple image blocks of size 256×256, and the relative positional relationship between these image blocks is recorded. Secondly, the image blocks are converted to the HSV color space, and the proportion of black pixels is analyzed to evaluate the proportion of shadow areas in the image blocks, and image blocks with too many invalid areas are discarded to avoid interference with the prediction of the entire image.

[0070] In some preferred embodiments, the above S2 may further include the following operations:

[0071] Each small image block is regarded as a node. The relative position relationship between the small images is that the small images that are directly in contact with each other are connected by edges, while the small images that are not directly in contact with each other are not connected by edges, thereby establishing graph data with small images as nodes.

[0072] In this preferred embodiment, since the relative positions of different image blocks from the same original image are recorded during the cropping process, graph data can be constructed based on the position information. Specifically, each image block is used as a node, and adjacent image blocks are connected by edges to construct the graph data.

[0073] In some preferred embodiments, the above S3 may further include the following operations:

[0074] S31, providing a pre-trained endoscopic polyp lesion prediction model, the endoscopic polyp lesion prediction model comprising: a pre-trained feature extraction model and a pre-trained graph neural network model;

[0075] S32: Extract features from the graph data using a feature extraction model to obtain feature vectors corresponding to each small image block. The feature vector information is carried by each node, thereby obtaining graph data containing feature information.

[0076] S33, inputting the graph data containing the feature information into the graph neural network model, outputting the lesion risk prediction result for each node, that is, obtaining the lesion risk prediction result for each small block of the image.

[0077] In some preferred embodiments, the above S31 may further include the following operations:

[0078] S311, a pre-trained feature extraction model is trained using massive endoscopic video data to extract local features of endoscopic images;

[0079] S312, the pre-trained graph neural network model uses training image blocks with texture features to train the graph data classifier constructed by the graph convolution layer to predict the results of image features.

[0080] In some preferred embodiments, the graph neural network model comprises two convolutional layers and a fully connected layer. Each convolutional layer is followed by a batch normalization layer and a ReLU activation function, with a dropout layer added between the two convolutional layers. The fully connected layer outputs a two-dimensional matrix, corresponding to the confidence level of the predictions for lesions and non-lesions, respectively. The graph neural network model implements a graph data classifier through the convolutional layers, enabling classification predictions for each node in the graph, thereby generating lesion risk predictions for each small image patch.

[0081] In some preferred implementations, the above S312 may further include the following operations:

[0082] S3121: Professional doctors manually mark the areas in the training images that are convenient for lesion risk diagnosis and filter out irrelevant areas;

[0083] S3122, cropping the circumscribed rectangle of the marked area to obtain training image blocks of uniform shape and size;

[0084] S3123, a professional doctor sets labels for the training image blocks based on professional knowledge and pathological results to obtain training image blocks with obvious texture features.

[0085] In the above-mentioned preferred embodiment, the endoscopic polyp lesion prediction model comprises two components: a feature extractor and a feature classifier. To effectively extract features from NBI images, in a preferred embodiment, the feature extractor utilizes the Endo endoscopy macromodel, obtained through self-supervised training on massive endoscopic image data. Each small image patch is input into the feature extractor to obtain a corresponding feature vector, thereby capturing the information of the small image patch in the feature space. The feature classifier can utilize a linear or nonlinear multilayer perceptron. To improve the accuracy of overall image judgment, it is preferably based on a graph neural network, which can construct a graph based on the relative positions of image patches and integrate the position information of the image patches in the feature space for judgment. The graph neural network classifier consists of two graph convolutional layers and one fully connected layer. After each graph convolutional layer, a batch normalization layer and a ReLU activation function are applied. To improve the model's generalization ability and prevent overfitting during training, a dropout layer is added between the two graph convolutional layers. The fully connected layer of the graph neural network outputs a two-dimensional value, corresponding to the prediction confidence level of the image as cancerous or non-cancerous, respectively.

[0086] In some preferred embodiments, the above S4 may further include the following operations:

[0087] S41, when the prediction results of all small image blocks are negative, the polyp lesion prediction result of the entire output image is low risk;

[0088] S42: If any one or more of the small image blocks have a positive prediction result, the polyp lesion prediction result of the entire output image is high risk.

[0089] In some preferred embodiments, the above S5 may further include the following operations:

[0090] S51, extracting each frame of the endoscopic video data, randomly sampling a number of small blocks of images of the same size from each frame, establishing graph data based on the relative positional relationship between the small blocks, and performing lesion risk prediction on the graph data using a pre-trained endoscopic polyp lesion prediction model to obtain a lesion risk prediction result for each frame of the image;

[0091] S52, a video segment in which several consecutive frames are high-risk is regarded as a high lesion risk segment; when the number of consecutive high lesion risk segments in the video data exceeds a set threshold, the video data is judged as high lesion risk, otherwise it is judged as low lesion risk.

[0092] Based on the same inventive concept, an embodiment of the present invention further provides a system for predicting the risk of polyps in endoscopic images.

[0093] Specifically, if Figure 2 As shown, the polyp lesion risk prediction system in endoscopic images provided in this embodiment may include the following modules:

[0094] An image preprocessing module is used to crop the output image of the endoscope device into multiple small blocks and filter out invalid small blocks to obtain small block images;

[0095] A graph data construction module, which is used to create graph data based on the relative position relationship between small image blocks;

[0096] The image risk prediction module provides a pre-trained endoscopic polyp lesion prediction model, inputs the established image data into the endoscopic polyp lesion prediction model, and outputs the prediction results to obtain the lesion risk prediction results for each small image block; the prediction results of each small image block are combined to obtain the polyp lesion prediction results for the entire output image.

[0097] In some preferred embodiments, the above system may further include the following modules:

[0098] The video risk prediction module performs segment risk prediction on the endoscopic video data based on the image preprocessing module, the graph data construction module and the image risk prediction module to obtain a risk prediction of polyp lesions in the entire endoscopic video data.

[0099] In the above-described embodiment, capturing the output image of the endoscopic device can be achieved through an acquisition card. For example, the acquisition card is connected to the endoscopic camera and a computer. The acquisition card driver is installed in the computer, and the endoscopic image acquired by the endoscopic camera is read by calling the interface in the acquisition card SDK. The acquisition card SDK interface is called to check the availability of the input signal. During each cycle of program execution, the video input buffer is checked for new input video frames. If so, the new video frame is converted into an 8-bit RBG image with each channel, and the image is then sent to the preprocessing stage.

[0100] It should be noted that the steps in the method provided by the present invention can be implemented by using the corresponding components in the system. Those skilled in the art can refer to the technical solution of the system to implement the step flow of the method, and can also refer to the technical solution of the method to implement the composition of the system. That is, the embodiments in the system and the embodiments in the method can be understood as preferred examples of each other, and will not be elaborated here.

[0101] In order to better understand the technical solution of the present invention, the specific operation process is described below in conjunction with a specific application example. It should be understood that this specific example is not used to limit the present invention.

[0102] Reference Figure 3 As shown, in this specific application example, the hardware used includes: an endoscope camera, an acquisition card, a computer, and a display device. The output interface of the endoscope camera is connected to the input interface of the acquisition card, the output interface of the acquisition card is connected to the input interface of the computer, and the output interface of the computer is connected to the input interface of the display device. If software is not used, the output interface of the endoscope camera can also be directly connected to the input interface of the display device.

[0103] In this specific application example, an endoscopic image feature extractor is built based on a deep convolutional neural network (CNN) using the Endo large-scale model (Wang Z, Liu C, Zhang S, et al. Foundation model for endoscopy video analysis via large-scale self-supervised pre-train [C]. International Conference on Medical Image Computing and Computer-Assisted Intervention. 2023). This model is implemented using Python and the PyTorch framework. The training data for this model was annotated from raw endoscopic images by professionals using Labelme software (version 4.6.0). Regions with significant intestinal characteristic patterns were mostly irregular polygonal areas. The bounding rectangles of these irregular regions were extracted and cropped into 256×256 image patches. The relative positional relationships between image patches from the same image were recorded. The image patches were converted to the HSV (hue, saturation, value) color space, the proportion of black pixels was calculated, and regions with shadow areas greater than a certain threshold were filtered out to obtain a dataset of image patches containing smaller shadow areas. Because the number of positive samples in the training set was significantly smaller than the number of negative samples, focal loss was used for training to mitigate the impact of data imbalance. The network parameters were optimized using the SGD optimizer with an initial learning rate of 0.01, momentum of 0.9, and weight decay of 0.0005. The network was trained for a total of 300 iterations, with the learning rate reduced by a factor of 0.2 at iterations 100 and 150. The preprocessed small image patches were fed into the large Endo model to obtain their corresponding feature vectors.

[0104] In this specific application example, an endoscopic image feature classifier is built based on a graph neural network. The structure is shown in Table 1, where bs refers to the batch size, conv represents the graph convolution layer, bn represents the normalization layer, relu represents the ReLu activation layer, dropout represents the random dropout layer, and fc represents the fully connected layer. This classifier combines the positional information between image patches, constructs a graph with each patch as a node, and outputs a prediction for each node. The final prediction for the entire image is obtained by voting the results of each patch: if any patch is predicted to be positive, the prediction for the entire image is positive; if all patches are predicted to be negative, the prediction for the entire image is negative.

[0105] Table 1: Graph Neural Network Architecture

[0106] Number of layers structure enter Output 0 conv bs*768 bs*384 1 bn bs*384 bs*384 2 relu bs*384 bs*384 3 dropout bs*384 bs*384 4 conv bs*384 bs*96 5 bn bs*96 bs*96 6 relu bs*96 bs*96 7 fc bs*96 bs*2

[0107] Better, when training graph neural networks, use focal loss (FocalLoss) for training. Focal loss is a loss function used to solve the problem of category imbalance, and is particularly suitable for target detection and image segmentation tasks. The traditional cross-entropy loss function has problems when processing category-imbalanced data, because it contributes less to samples of minority categories during training, making it difficult for the model to effectively learn the characteristics of these categories. Focal Loss improves the imbalance handling ability of the cross-entropy loss function by introducing an adjustable balance parameter and a correction term. The correction term can reduce the weight of easy-to-classify samples, so that the model pays more attention to difficult samples, thereby improving the classification performance for minority categories. The design idea of ​​FocalLoss is simple and effective, which can help improve the performance of the model in category-imbalanced scenarios, and has achieved significant improvements in many computer vision tasks. Its specific form is as follows:

[0108] FL(p)=-α(1-p) γ log(p)

[0109] Here, 1-p represents the difficulty of sample classification. When p approaches 1, the current prediction confidence for the sample is high, the classification is good, and the sample is easy to distinguish. At this time, 1-p approaches 0, and the sample's contribution to the loss is small. When p is very small, 1-p approaches 1, and the loss is not significantly affected. 1-p can reduce the contribution of easy-to-distinguish samples to the loss, α can suppress the imbalance in the number of positive and negative samples, and γ can control the imbalance in the number of easy and difficult samples.

[0110] Reference Figure 4 The figure shows a flow chart of the polyp lesion diagnosis method in this application example, which can be performed in the following steps:

[0111] Step 1, initializing the endoscopic polyp lesion prediction model;

[0112] Step 2: Check whether there is a new video frame (output image of the endoscope device) input in the video input buffer. If yes, go to step 3; if not, repeat step 2;

[0113] Step 3: preprocess the input image;

[0114] Step 4, inputting the pre-processed image into the endoscopic polyp lesion prediction model;

[0115] Step 5: Display the image and prediction results through the video display window according to the return result of step 4;

[0116] In step 2 of the above-mentioned specific application example, the acquisition of the output image of the endoscope device can be achieved through the video signal acquisition thread. Video signal acquisition is achieved through a separate thread. During the initialization phase, the driver interface is used to check whether the acquisition card hardware is connected normally. According to the signal source set by the user, the acquisition card hardware is configured by calling the underlying interface. The format of the video input signal is obtained by calling the underlying driver. According to the format of the video signal, an input buffer is reserved for it in the memory, the effective data length of the buffer is initialized to zero, and the base address of the buffer is transmitted to the driver. In the subsequent operation, the underlying driver continuously transfers the video frames collected by the video capture card to the memory according to the previous configuration. Whenever a new video frame is available, the signal acquisition thread stores the video in the reserved input buffer and updates the amount of valid data in the buffer.

[0117] Considering the practical needs of endoscopic inspection, the effectiveness of the endoscopic polyp prediction model on video data was further explored. The video dataset consists of 159 short videos, each running at 30 frames per second and ranging in duration from 7 to 9 seconds. Of the 159 videos, 41 contain cancerous polyps, while the remaining 118 videos contain only non-cancerous polyps. The raw video data is initially divided into individual frames, and images are extracted from each frame. Subsequently, five regions (each 512×768 in size) are randomly sampled from different locations across the entire image of each frame. For each region, it is divided into six 256×256 image patches, and features are extracted using the base model. A graph is constructed based on the feature vectors and relative positions of the various image patches in each region. Subsequently, the proposed polyp risk prediction method is used for prediction. The prediction result for a single image frame is based on the combined predictions from the five sampled regions. If any region is identified as a cancerous polyp, the frame is labeled as cancerous. To minimize the randomness of single frame predictions, a video is classified as cancerous only if at least three consecutive frames are classified as cancerous, meaning that the patient's intestine has a high risk of cancer.

[0118] Quantitative test results: This example uses 514 images as the training set, of which 107 are cancerous images and 397 are non-cancerous images. 88 images are used as the test set, of which 21 are cancerous images and 67 are non-cancerous images. The metrics used are as follows:

[0119]

[0120] Among them, TP (True Positive), TN (True Negative), FP (False Positive), and FN (False Negative) are the number of true positive, true negative, false positive, and false negative samples, respectively.

[0121] Table 2 shows the results of this example for small image patch lesion risk diagnosis. The model was trained on 1757 image patches, and the performance was evaluated on 439 image patches. Endo-LaG refers to a method that uses the endoscopic Endo large model as a feature extractor and a graph neural network as a feature classifier. CNN refers to a method that uses the convolutional neural network ResNet18 (He K, Zhang X, Ren S, et al. Deep residual learning for image recognition [C]. CVPR. 2016). Endo-L refers to a method that uses the endoscopic Endo large model as a feature extractor and a linear multilayer perceptron as a feature classifier. Endo-NL refers to a method that uses the endoscopic Endo large model as a feature extractor and a nonlinear multilayer perceptron as a feature classifier. As can be seen in the table, traditional CNN models have limitations in polyp risk prediction. Although the accuracy reaches 0.808, the recall and AUC are very low. The limited dataset size and significant imbalance between negative and positive samples make CNNs prone to overfitting during training. Well-trained CNN models tend to classify samples as negative, resulting in high precision but missing a large number of positive samples. Furthermore, CNNs struggle to mine effective information for prediction. This can cause the model to overfocus on irrelevant or repetitive features, resulting in poor generalization to new samples. The Endo-L and Endo-NL models outperformed the CNN model across all metrics. This suggests that using large models to extract features can effectively identify key information in endoscopic images. Both models achieved higher recall rates than the CNN model. The Endo-L model exhibited a higher recall rate compared to the Endo-NL model, suggesting that nonlinear classifiers are more susceptible to overfitting during training. The preferred method of the above-mentioned embodiment of the present invention, Endo-LaG, outperformed traditional methods across all metrics, particularly achieving a very high recall rate while maintaining high precision. This is of considerable importance in real-world clinical scenarios, as the implementation of the method provided by the above-mentioned embodiment of the present invention ensures minimal supervision of lesions or polyps, resulting in significantly improved patient survival rates.

[0122] Table 2 Image block lesion risk diagnosis results

[0123]

[0124]

[0125] Table 3 shows the results of this example for whole-image lesion risk diagnosis. The model was trained on 327 images, and the performance was evaluated on 77 images. Experimental results show that the three traditional methods have similar accuracy, but the CNN has a significantly lower recall rate. As mentioned above, this difficulty is due to the limited ability of the CNN model to extract essential information from the image, resulting in a higher susceptibility to overfitting, especially when working with small datasets. Compared with the three traditional methods, the method proposed in the above embodiment of the present invention also achieves better performance in predicting the entire image. Furthermore, it is worth noting that, although the same method is used, a higher recall rate can be achieved when predicting the entire image than when predicting image blocks. This is because the prediction result for the entire image is based on information collected from different small image blocks. If any image block is predicted to be at high risk for disease, the entire image containing that block will also be predicted to be at high risk for disease. This enhances the ability of the method proposed in the above embodiment of the present invention to detect high-risk cases of severe disease during the prediction process, enabling patients to receive timely treatment.

[0126] Table 3 Diagnosis results of lesion risk in the whole image

[0127]

[0128] This specific application example invited six endoscopists to evaluate the test data set and record the time they spent. The six endoscopists were divided into two groups according to their years of practice: three experienced endoscopists (with more than ten years of experience) and three young endoscopists (with less than ten years of experience). The test results of all doctors are shown in Table IV. Without the assistance of artificial intelligence, the experienced doctors had an accuracy of 0.738, a recall rate of 0.654, an FPR of 0.226, a precision of 0.571, and an average processing time of 7 minutes and 55 seconds. In contrast, the young doctors had an accuracy of 0.792, a recall rate of 0.744, an FPR of 0.188, a precision of 0.625, and an average processing time of 11 minutes and 45 seconds. The performance of these doctors on the ROC curve was plotted using the method proposed in the above embodiment of the present invention, as shown in Table 4. Figure 5As shown. The average performance of these doctors is lower than the ROC curve, indicating that the method proposed in the above embodiment of the present invention is superior to the experience of the doctors. With the help of artificial intelligence, the experienced doctors achieved an accuracy of 0.875, a recall of 0.884, an FPR of 0.129, a precision of 0.762, and an average processing time of 6 minutes and 35 seconds. In comparison, the young doctors achieved an accuracy of 0.864, a recall of 0.821, an FPR of 0.118, a precision of 0.746, and an average processing time of 7 minutes and 59 seconds. The results show that in terms of utility, the use of AI assistance can significantly improve the diagnostic accuracy of doctors and reduce the occurrence of missed diagnoses. In terms of efficiency, diagnosis without AI assistance, especially for young doctors, can be a time-consuming process, while AI assistance significantly reduces their diagnostic time.

[0129] Table 4 Prediction results of doctors with different qualifications with and without model assistance

[0130]

[0131] On a video dataset, this example (Endo-LaG) achieved an accuracy of 0.767, a recall of 0.732, an FPR of 0.220, and a precision of 0.536. Of the 41 videos containing cancerous polyps, the method provided by the above embodiment of the present invention accurately predicted 30 of them. Three traditional models (CNN, Endo-L, and Endo-NL) were tested under the same experimental settings, and their detection rates for cancerous samples were almost zero.

[0132] Figure 6 For the above specific application examples, the method and system provided by the above embodiments of the present invention are used to predict image data as cancerous and non-cancerous. Among them, one group of images in (a) are non-cancerous images, with regularly distributed blood vessels and a regular tubular surface; the other group of images in (b) are cancerous images, with irregularly distributed blood vessels, a loose surface, and a very blurred surface in the amorphous area.

[0133] One embodiment of the present invention also provides an endoscopic polyp lesion risk prediction terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it can be used to execute any one of the methods of the above-mentioned embodiments of the present invention, or to run any one of the systems of the above-mentioned embodiments of the present invention.

[0134] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic Random Access Memory, abbreviated: DDR SDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above-mentioned methods), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories. In addition, the above-mentioned computer programs, computer instructions, data, etc. can be called by the processor.

[0135] The processor is configured to execute the computer program stored in the memory to implement the various steps of the method or various modules of the system involved in the above embodiments. For details, please refer to the relevant descriptions in the above method and system embodiments.

[0136] The processor and memory can be independent structures or integrated structures. When the processor and memory are independent structures, the memory and processor can be coupled via a bus.

[0137] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it can be used to execute any method of the above embodiments of the present invention, or to run any system of the above embodiments of the present invention.

[0138] Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of computer programs from one location to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC. Alternatively, the ASIC can be located in a user device. Of course, the processor and storage medium can also exist as discrete components in a communication device.

[0139] The above-mentioned embodiments of the present invention provide a method, system, and terminal for predicting the risk of polyps in endoscopic images. The method collects the output image of the endoscopic device and preprocesses the input image, crops the image into multiple 256×256 small blocks, and filters out invalid small blocks to avoid interference with the prediction of the entire image; calls the endoscopic polyp lesion prediction model to predict each small block of the image; and integrates the prediction results of each small block to obtain the polyp lesion diagnosis result for the entire endoscopic image. The present invention can diagnose polyps based on endoscopic images and videos and output the results of cancer risk prediction for doctors' reference. It has the advantages of simple operation and no technical barriers to use. By dividing the endoscopic image into multiple small blocks for prediction, high-risk areas in the endoscopic image can be indicated simultaneously, which is convenient for doctors to observe. Because a large model is used as the basic model to extract endoscopic image features, the accuracy is high and the clinical practicality is strong.

[0140] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. by logically programming the method steps. Therefore, the system and its various devices provided by the present invention can be considered a hardware component, and the devices included therein for implementing the various functions can also be considered as structures within the hardware component; the devices for implementing the various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0141] Matters not mentioned in the above embodiments of the present invention are well known in the art.

[0142] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various modifications or variations within the scope of the claims without affecting the essence of the present invention. The above preferred features may be used in any combination as long as they do not conflict with each other.

Claims

1. A method for predicting the risk of polyps in endoscopic images, characterized in that: include: The output image of the endoscope device is cropped into multiple small blocks, and invalid small blocks are filtered out to obtain small block images; Create graph data based on the relative position relationship between small image blocks; Providing a pre-trained endoscopic polyp lesion prediction model, inputting the established image data into the endoscopic polyp lesion prediction model, and outputting a prediction result to obtain a lesion risk prediction result for each small image block; The prediction results of each small image block are combined to obtain the polyp lesion prediction result of the entire output image.

2. The method for predicting polyp risk in endoscopic images according to claim 1, characterized in that: The step of cropping the output image of the endoscope device into a plurality of small blocks and filtering out invalid small blocks includes: The output image of the endoscope device is cropped into multiple 256×256 small blocks; Convert all small blocks to HSV space, and determine whether each pixel belongs to the shadow area in the HSV space; If the shadow area ratio in any small block is greater than or equal to the set ratio threshold, the small block is considered to be an invalid small block image and discarded; If the proportion of the shadow area in any small block is less than the set ratio threshold, the small block is considered to be a valid small block image and is retained for risk prediction.

3. The method for predicting polyp risk in endoscopic images according to claim 1, wherein: The step of establishing the graph data according to the relative position relationship between the small image blocks includes: Each small image block is regarded as a node. Based on the relative position relationship between the small image blocks, the small image blocks that are directly in contact with each other are connected by edges, while the small image blocks that are not directly in contact with each other are not connected by edges. Graph data with small image blocks as nodes is established.

4. The method for predicting polyp risk in endoscopic images according to claim 1, wherein: The method provides a pre-trained endoscopic polyp lesion prediction model, inputs the established image data into the endoscopic polyp lesion prediction model, and outputs a prediction result, including: A pre-trained endoscopic polyp lesion prediction model is provided, wherein the endoscopic polyp lesion prediction model includes: a pre-trained feature extraction model and a pre-trained graph neural network model; Performing feature extraction on the graph data through the feature extraction model to obtain feature vectors corresponding to each small image block in sequence, wherein information of the feature vectors is carried by each node, thereby obtaining graph data containing feature information; The graph data containing the feature information is input into the graph neural network model, and the lesion risk prediction result for each node is output, that is, the lesion risk prediction result for each small block of the image is obtained.

5. The method for predicting polyp risk in endoscopic images according to claim 4, characterized in that: Also includes any one or more of the following: -The pre-trained feature extraction model is trained using massive endoscopic video data to extract local features of endoscopic images; -The pre-trained graph neural network model is trained using training image blocks with texture features to predict results of image features; The graph neural network model comprises two graph convolutional layers and a fully connected layer; wherein a batch normalization layer and a ReLU activation function are applied after each graph convolutional layer, and a dropout layer is added between the two graph convolutional layers; the output of the fully connected layer is two-dimensional, corresponding to the prediction confidence of lesions and non-lesions, respectively; The training image block with texture features is obtained by: Professional doctors manually mark the areas of the training images that are convenient for lesion risk diagnosis and filter out irrelevant areas; Crop the circumscribed rectangle of the marked area to obtain training image blocks of consistent shape and size; Professional doctors set labels for the training image blocks based on professional knowledge and pathological results to obtain training image blocks with texture features.

6. The method for predicting polyp risk in endoscopic images according to claim 1, wherein: The prediction results of each small block of image are integrated, including: When the prediction results of all small image blocks are negative, the polyp lesion prediction result of the entire output image is low risk. If the prediction results of any one or more small image blocks are positive, the polyp lesion prediction result of the entire output image is high risk.

7. The method for predicting polyp lesion risk in endoscopic images according to any one of claims 1 to 5, characterized in that: Also includes: Based on the above process, the segment risk prediction of the endoscopic video data is performed to obtain the risk prediction of polyp lesions in the entire endoscopic video data; in: The endoscopic video data is subjected to segment risk prediction to obtain a risk prediction of polyp lesions in the entire endoscopic video data, including: Extract each frame of endoscopic video data, randomly sample several small blocks of the same size from each frame, establish graph data based on the relative positional relationship between the small blocks, and use a pre-trained endoscopic polyp lesion prediction model to perform lesion risk prediction on the graph data to obtain a lesion risk prediction result for each frame of image; A video segment in which several consecutive frames are high-risk is regarded as a high lesion risk segment; when the number of consecutive high lesion risk segments in the video data exceeds the set threshold, the video data is judged as high lesion risk, otherwise it is judged as low lesion risk.

8. A polyp risk prediction system in endoscopic images, characterized in that: include: An image preprocessing module is used to crop the output image of the endoscope device into multiple small blocks and filter out invalid small blocks to obtain small block images; A graph data construction module, which is used to create graph data based on the relative position relationship between small image blocks; An image risk prediction module, which provides a pre-trained endoscopic polyp lesion prediction model, inputs the established image data into the endoscopic polyp lesion prediction model, and outputs the prediction results to obtain the lesion risk prediction results for each small image block; The prediction results of each small image block are combined to obtain the polyp lesion prediction result of the entire output image.

9. The polyp risk prediction system in endoscopic images according to claim 8, characterized in that: Also includes: The video risk prediction module performs segment risk prediction on the endoscopic video data based on the image preprocessing module, the graph data construction module and the image risk prediction module to obtain a risk prediction of polyp lesions in the entire endoscopic video data.

10. An endoscopic polyp risk prediction terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When executing the computer program, the processor can be used to perform the method according to any one of claims 1 to 7, or to run the system according to any one of claims 8 to 9.

Images