Polyp image processing method and device and computer equipment

Abstract

The invention relates to a polyp image processing method and device, computer equipment and a storage medium. The method can be applied to an application scene for processing a polyp image in a vehicle-mounted terminal, a cloud server, a block chain node or other equipment, and comprises the following steps: when a polyp image is detected from a colonoscope image, obtaining a white light polyp image and a narrow-band polyp image; obtaining a first region-of-interest image based on a first polyp detection frame in the white light polyp image; obtaining a second region-of-interest image based on a second polyp detection frame in the narrow-band polyp image; and extracting white light polyp features and narrow-band polyp features of the first region-of-interest image and the second region-of-interest image, and performing classification prediction based on the white light polyp features and the narrow-band polyp features to determine a polyp infiltration degree category to which the polyp image belongs. By adopting the method, the accuracy of a polyp detection result can be effectively improved under the condition of ensuring automatic detection of the polyp.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a polyp image processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of computer technology and the Internet, colorectal cancer (CRC) is a common adenocarcinoma that mainly occurs in the colon or rectum. At least 80%-95% of colon cancers evolve from adenomatous polyps. In clinical practice, colonoscopy is the most effective method for polyp detection, but manual detection is not only time-consuming and labor-intensive, but may also result in missed or false detections. Therefore, the use of artificial intelligence technology to learn from colon polyp images to achieve automatic detection of colon polyps brings great convenience to doctors, can assist doctors in observation, and help diagnose colorectal diseases in the early stages.

[0003] However, the current polyp image processing method is usually based on electronic endoscope images and uses deep learning technology to automatically detect polyps in the colon. Due to the large number of parameters and long inference time of deep learning networks, it is not very efficient in practical applications. Therefore, how to ensure the automatic detection of polyps while effectively improving the accuracy of polyp detection results has become an urgent problem that needs to be solved. Summary of the Invention

[0004] Based on this, it is necessary to provide a polyp image processing method, device, computer equipment and storage medium to address the above technical problems, which can effectively improve the accuracy of polyp detection results while ensuring automatic polyp detection.

[0005] In a first aspect, the present application provides a polyp image processing method. The method includes: when a polyp image is detected from a colonoscopy image, obtaining a white-light polyp image acquired in a white-light mode and a narrow-band polyp image acquired in a narrow-band imaging mode; based on the position of a first polyp detection frame in the white-light polyp image, expanding the first polyp detection frame to obtain a first region of interest image; based on the position of a second polyp detection frame in the narrow-band polyp image, expanding the second polyp detection frame to obtain a second region of interest image; performing feature extraction on the first region of interest image and the second region of interest image respectively to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image; and performing classification prediction based on the white-light polyp features and the narrow-band polyp features to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

[0006] In a second aspect, the present application also provides a polyp image processing device. The device includes: an acquisition module for acquiring, when a polyp image is detected from a colonoscopy image, a white-light polyp image acquired in a white-light mode and a narrow-band polyp image acquired in a narrow-band imaging mode; an expansion module for expanding a first polyp detection frame in the white-light polyp image based on the position of the first polyp detection frame in the white-light polyp image to obtain a first region of interest image; and expanding a second polyp detection frame in the narrow-band polyp image based on the position of the second polyp detection frame in the narrow-band polyp image to obtain a second region of interest image; an extraction module for performing feature extraction on the first region of interest image and the second region of interest image, respectively, to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image; and a determination module for performing classification prediction based on the white-light polyp features and the narrow-band polyp features to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

[0007] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program: when a polyp image is detected from a colonoscopy image, obtaining a white-light polyp image acquired in a white-light mode and a narrow-band polyp image acquired in a narrow-band imaging mode; expanding a first polyp detection frame in the white-light polyp image based on the position of the first polyp detection frame in the white-light polyp image to obtain a first region of interest image; expanding a second polyp detection frame in the narrow-band polyp image based on the position of the second polyp detection frame in the narrow-band polyp image to obtain a second region of interest image; performing feature extraction on the first region of interest image and the second region of interest image, respectively, to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image; and performing classification prediction based on the white-light polyp features and the narrow-band polyp features to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

[0008] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps: when a polyp image is detected from a colonoscopy image, obtaining a white-light polyp image acquired in white-light mode and a narrow-band polyp image acquired in narrow-band imaging mode; expanding a first polyp detection frame in the white-light polyp image based on the position of the first polyp detection frame in the white-light polyp image to obtain a first region of interest image; expanding a second polyp detection frame in the narrow-band polyp image based on the position of the second polyp detection frame in the narrow-band polyp image to obtain a second region of interest image; performing feature extraction on the first region of interest image and the second region of interest image, respectively, to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image; and performing classification prediction based on the white-light polyp features and the narrow-band polyp features to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

[0009] In a fifth aspect, the present application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps: when a polyp image is detected from a colonoscopy image, obtaining a white-light polyp image acquired in a white-light mode and a narrow-band polyp image acquired in a narrow-band imaging mode; based on the position of a first polyp detection frame in the white-light polyp image, expanding the first polyp detection frame to obtain a first region of interest image; based on the position of a second polyp detection frame in the narrow-band polyp image, expanding the second polyp detection frame to obtain a second region of interest image; performing feature extraction on the first region of interest image and the second region of interest image, respectively, to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image; and performing classification prediction based on the white-light polyp features and the narrow-band polyp features to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

[0010] The above-mentioned polyp image processing method, apparatus, computer equipment, storage medium and computer program product, when a polyp image is detected from a colonoscopy image, obtains a white-light polyp image collected in a white-light mode and a narrow-band polyp image collected in a narrow-band imaging mode; based on the position of a first polyp detection frame in the white-light polyp image, the first polyp detection frame is expanded to obtain a first region of interest image; based on the position of a second polyp detection frame in the narrow-band polyp image, the second polyp detection frame is expanded to obtain a second region of interest image; feature extraction is performed on the first region of interest image and the second region of interest image respectively to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image; classification prediction is performed based on the white-light polyp features and the narrow-band polyp features to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories. Since the white-light polyp image and the narrow-band polyp image are collected when the polyp image is detected in the colonoscopy image, the first region of interest image determined based on the first polyp detection frame in the white-light polyp image and the second region of interest image determined based on the second polyp detection frame in the narrow-band polyp image have richer image information, so that when subsequent classification prediction is performed based on the white-light polyp features extracted from the first region of interest image and the narrow-band polyp features extracted from the second region of interest image, more accurate classification prediction results can be obtained, which can effectively improve the accuracy of polyp detection results while ensuring automatic polyp detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A diagram showing an application environment of a polyp image processing method according to an embodiment;

[0012] Figure 2 1 is a flow chart of a polyp image processing method according to an embodiment;

[0013] Figure 3 is a schematic diagram of a polyp detection effect in one embodiment;

[0014] Figure 4 A schematic diagram of the prediction of the degree of infiltration in one embodiment;

[0015] Figure 5 A schematic diagram of capturing a polyp ROI image in one embodiment;

[0016] Figure 6 Schematic diagram of a process for fusion prediction based on white light polyp ROI and NBI polyp ROI in one embodiment;

[0017] Figure 7 A schematic diagram of an application method on the product side in one embodiment;

[0018] Figure 8 1. A schematic diagram of the overall process of a polyp image processing method according to an embodiment;

[0019] Figure 9 A schematic diagram of a ResNet series network architecture table in one embodiment;

[0020] Figure 10 Schematic diagram of a training method of a pre-trained model Model_1 for infiltration degree prediction in one embodiment;

[0021] Figure 11 Schematic diagram of a training method of an infiltration degree prediction model Model_2 for infiltration degree prediction in one embodiment;

[0022] Figure 12 is a schematic diagram of an application method on the product side in another embodiment;

[0023] Figure 13 1 is a schematic diagram of the overall processing flow of an artificial intelligence-based auxiliary diagnosis system for colorectal polyp detection and classification in one embodiment;

[0024] Figure 14 Schematic diagram of the polyp detection effect of an artificial intelligence-based auxiliary diagnosis system for colorectal polyp detection and classification in one embodiment;

[0025] Figure 15 FIG1 is a schematic diagram of an embodiment of an artificial intelligence-based auxiliary diagnosis system for colorectal polyp detection and classification for predicting the degree of invasion;

[0026] Figure 16 A schematic diagram of a polyp ROI cutout in an artificial intelligence-based auxiliary diagnosis system for colorectal polyp detection and classification in one embodiment;

[0027] Figure 17 Schematic diagram of a training method for an infiltration degree prediction pre-training model Model_1 of an artificial intelligence-based auxiliary diagnosis system for colorectal polyp detection and classification in one embodiment;

[0028] Figure 18 Schematic diagram of a training method for the infiltration degree prediction model Model_2 of an artificial intelligence-based auxiliary diagnosis system for colorectal polyp detection and classification in one embodiment;

[0029] Figure 19 is a structural block diagram of a polyp image processing device in one embodiment;

[0030] Figure 20 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0032] Cloud technology refers to a hosting technology that unifies hardware, software, network and other resources within a wide area network or local area network to achieve data computing, storage, processing and sharing.

[0033] Cloud technology is a general term for network technologies, information technologies, integration technologies, management platform technologies, and application technologies based on the cloud computing business model. It can form a resource pool for on-demand, flexible and convenient use. Cloud computing technology will become a crucial support. Services in technical network systems, such as those for video sites, image sites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identifier, requiring transmission to the system for logical processing. Data of different levels will be processed separately, and data from all industries will require a strong system backing, which can only be achieved through cloud computing.

[0034] Cloud storage is a new concept that has been extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0035] A pre-training model, also known as a cornerstone model or a large model, refers to a deep neural network (DNN) with large parameters. It is trained on massive amounts of unlabeled data. Leveraging the function approximation capabilities of large-parameter DNNs, the pretrained machine learning (PTM) extracts common features from the data. Through techniques such as fine tuning, efficient parameter fine tuning (PEFT), and prompt-tuning, it is then adapted for downstream tasks. Therefore, pre-trained models can achieve ideal results in few-shot or zero-shot scenarios. PTMs can be categorized by the data modality they process, including language models (ELMO, BERT, GPT), vision models (swin-transformer, ViT, V-MOE), speech models (VALL-E), and multimodal models (ViBERT, CLIP, Flamingo, Gato). Multimodal models represent features from two or more data modalities. Pre-trained models are important tools for outputting AI-generated content (AIGC) and can also serve as a universal interface for connecting multiple task-specific models.

[0036] Adaptive computing: Automatically adjusts the model's computational load and accuracy based on different input data, improving efficiency while maintaining accuracy. Adaptive computing flexibly adjusts the model's computational load and accuracy based on different input data, achieving a better balance between efficiency and accuracy.

[0037] Model parallel computing: This refers to distributing model computation tasks to multiple computing devices (such as CPUs, GPUs, and TPUs) for simultaneous computation, thereby accelerating model training and inference. Model parallel computing can effectively utilize computing resources, improving model computational efficiency and training speed.

[0038] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0039] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0040] Computer vision (CV) is the science of enabling machines to "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, tracking, and measuring objects, and further image processing to produce images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the field of vision, such as the Swin Transformer, ViT, V-MOE, and MAE, can be fine-tuned to quickly and widely apply to specific downstream tasks. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0041] It should be noted that in the following description, the terms "first, second and third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second and third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0042] The polyp image processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, can be integrated on the server 104, or can be placed on the cloud or other network servers. When the terminal 102 detects a polyp image from a colonoscopy image, the terminal 102 can obtain a white-light polyp image collected in white-light mode from the server 104, and obtain a narrow-band polyp image collected in narrow-band imaging mode; or, when the terminal 102 detects a polyp image from a colonoscopy image, the terminal 102 can also obtain a white-light polyp image collected in white-light mode from the local computer, and obtain a narrow-band polyp image collected in narrow-band imaging mode; further, the terminal 102 can expand the first polyp detection frame based on the position of the first polyp detection frame in the white-light polyp image, A first region of interest image is obtained, and based on the position of the second polyp detection frame in the narrow-band polyp image, the second polyp detection frame is expanded to obtain a second region of interest image; further, the terminal 102 can perform feature extraction on the first region of interest image and the second region of interest image respectively to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image, and perform classification prediction based on the white-light polyp features and the narrow-band polyp features to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

[0043] The terminal 102 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart TV, smart watch, IoT device, or portable wearable device. The IoT device may be a smart car device, etc. The portable wearable device may be a smart watch, smart bracelet, head-mounted device, etc.

[0044] The server 104 may be an independent physical server or a service node in a blockchain system. Each service node in the blockchain system forms a peer-to-peer network. The peer-to-peer protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP).

[0045] In addition, server 104 can also be a server cluster composed of multiple physical servers, and can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN, Content Delivery NetworkCDN), and big data and artificial intelligence platforms.

[0046] The terminal 102 and the server 104 can be connected via Bluetooth, USB (Universal Serial Bus) or network communication connection methods, and this application does not limit this.

[0047] In one embodiment, Figure 2 As shown, a polyp image processing method is provided, which can be executed by a server or a terminal alone, or by a server and a terminal together. Figure 1 The following steps are used as an example to illustrate the terminal in the figure:

[0048] Step 202 : When a polyp image is detected from a colonoscopy image, a white light polyp image acquired in a white light mode and a narrow band polyp image acquired in a narrow band imaging mode are acquired.

[0049] Among them, the colonoscopic image refers to the image in the real-time inspection video collected by the endoscope host. For example, the colonoscopic image in this application can be a multi-frame colonoscopic image obtained by the terminal collecting the real-time colonoscopic video frame by frame.

[0050] A polyp image refers to an image containing polyps. For example, if an image showing a polyp morphology is detected at the lower left position of a certain frame of colonoscopy image, it indicates that a polyp image is detected from the frame of colonoscopy image.

[0051] White light mode refers to a light source mode. The endoscope host in white light mode can provide natural light observation effects. It is usually used to observe the color, texture and details of the mucosa in order to detect lesions and the extent of lesions.

[0052] A white-light polyp image refers to an image captured when the endoscope host is adjusted to be in white-light mode. For example, the white-light polyp image in this application can be an image captured when the terminal transforms the endoscope host into white-light mode and aims at the target object (i.e., the physical form of the polyp) at a better angle.

[0053] Narrowband imaging (NBI), also known as endoscopic narrowband imaging (NBI), is an emerging endoscopic technique that uses filters to remove the broadband spectrum of red, blue, and green light waves emitted by the endoscope's light source, leaving only the narrowband spectrum for diagnosing various gastrointestinal diseases. The main advantage of NBI endoscopic technology is that it can not only accurately observe the morphology of the gastrointestinal mucosal epithelium, such as the structure of the epithelial glandular fovea, but also the morphology of the epithelial vascular network. This new technology can better help endoscopists distinguish between gastrointestinal epithelium, such as changes in vascular morphology in gastrointestinal inflammation and irregular changes in the glandular fovea in early gastrointestinal tumors, thereby improving the accuracy of endoscopic diagnosis.

[0054] A narrow-band polyp image refers to an image captured when the endoscope host is adjusted to a narrow-band imaging mode. For example, the narrow-band polyp image in this application can be an image captured when the terminal transforms the endoscope host into a narrow-band imaging mode and aims at the target object (i.e., the physical form of the polyp) at a better angle.

[0055] Specifically, when the user wants to detect polyps in the colon, the user can open the diagnostic application (Application, APP) on the terminal through a trigger operation, and enter the polyp detection page of the diagnostic application through a selection operation, that is, the user can log in to the diagnostic application through a trigger operation. Furthermore, the user can enter the polyp detection page through a trigger operation in the main interface displayed by the diagnostic application. In the polyp detection page displayed by the terminal, each object using the diagnostic application (such as a doctor) can view the specific content and related classification auxiliary functional information in the polyp detection page, and each object using the diagnostic application can also trigger an automatic detection operation for the polyp image. The terminal then responds to the start-up operation of the diagnostic application by the object using the diagnostic application, and performs an operation of performing polyp image detection on the colonoscopy image. When the terminal detects a polyp image from the colonoscopy image, the terminal can display the colonoscopy image and display a polyp detection box on the colonoscopy image to mark the polyp image detected from the colonoscopy image.

[0056] For example, a colonoscopy image is used as a colonoscopy video frame. Figure 3 As shown in Figure 1, it is a schematic diagram of the polyp detection effect. Assume that the colonoscopy image is Figure 3 In the colon endoscopy video frame picture shown on the left, when the user wants to detect polyps in the colon, user 1 (i.e., doctor A) can trigger the diagnostic application on the terminal and enter the polyp detection page of the diagnostic application through a selection operation. User 1 can trigger the automatic detection operation for the polyp image, and the terminal responds to the user 1's start operation of the diagnostic application and performs the following operations: Figure 3 The three frames of colonoscopy images shown on the left are used for polyp image detection. Figure 3 When a polyp image is detected in the second frame of the colonoscopy image shown in FIG. 1 , the terminal can display the colonoscopy image and display the following information on the colonoscopy image: Figure 3 The polyp detection frame shown on the right side of the image is a polyp detected from the colonoscopy image of the frame (ie, Figure 3 The black solid rectangle shown in the second frame on the right).

[0057] Step 204 : Based on the position of the first polyp detection frame in the white-light polyp image, the first polyp detection frame is enlarged to obtain a first region of interest image.

[0058] The first polyp detection frame refers to the polyp detection frame of the polyp image in the white light polyp image collected in the white light mode, for example, Figure 4 As shown in FIG. 1 , which is a schematic diagram of the prediction of the degree of invasion, the first polyp detection frame can be as follows: Figure 4 The white light image shown in and the rectangular detection box shown in polyp detection.

[0059] A region of interest image is an image obtained by extracting the polyp's region of interest based on the polyp image detection results. In image processing or machine vision, a region of interest (ROI) is a specific area selected from the original image. This area is typically the focus of image analysis. In some cases, the region of interest image in this application may also be referred to as an ROI image.

[0060] The first region of interest image refers to an image obtained by performing a cutout process on the polyp region of interest according to the white light polyp image detection result.

[0061] Step 206 : Based on the position of the second polyp detection frame in the narrow-band polyp image, the second polyp detection frame is enlarged to obtain a second region of interest image.

[0062] The second polyp detection frame refers to the polyp detection frame of the polyp image in the narrow-band polyp image acquired in the narrow-band imaging mode. For example, the second polyp detection frame can be as follows: Figure 4 The NBI image shown in FIG and the rectangular detection frame displayed in the polyp detection. It can be understood that the first polyp detection frame and the second polyp detection frame in the present application are only used to distinguish the polyp detection frames determined in different types of images.

[0063] The second region of interest image refers to the image obtained by intercepting the polyp region of interest based on the narrow-band polyp image detection results. It can be understood that the first region of interest image and the second region of interest image in this application are only used to distinguish the region of interest images determined in different types of images.

[0064] Specifically, after the terminal obtains a white-light polyp image collected in white-light mode and a narrow-band polyp image collected in narrow-band imaging mode, the terminal can first perform image recognition on the obtained white-light polyp image and narrow-band polyp image respectively to obtain a first image type to which the white-light polyp image belongs, and a second image type to which the narrow-band polyp image belongs; further, the terminal can perform a first verification on the consistency between the first image type and the white-light image type, and perform a second verification on the consistency between the second image type and the narrow-band image type. When the first verification and the second verification both indicate that the verification is passed, the terminal can further expand the first polyp detection frame based on the position of the first polyp detection frame in the white-light polyp image in the white-light polyp image to obtain a first region of interest image, and expand the second polyp detection frame based on the position of the second polyp detection frame in the narrow-band polyp image in the narrow-band polyp image to obtain a second region of interest image, so that the final first region of interest image and the second region of interest image contain richer information.

[0065] For example, Figure 5 As shown in the figure, it is a schematic diagram of intercepting the polyp ROI image. Assume that the terminal obtains the white light polyp image collected in the white light mode as Figure 5 The white light image shown on the left and the narrow band polyp image acquired in narrow band imaging mode are Figure 5 The NBI picture shown on the right side of the figure, the terminal can first obtain the following Figure 5 The white light image and the NBI image shown in the figure are subjected to image recognition to obtain a first image type to which the white light image belongs and a second image type to which the NBI image belongs; further, the terminal can perform a first check on the consistency between the first image type and the white light image type, and a second check on the consistency between the second image type and the narrowband image type. When both the first check and the second check indicate that the check is passed, the terminal can further perform the following steps based on the example of FIG. Figure 5 The position of the first polyp detection frame in the white light image shown in FIG is enlarged to obtain the following: Figure 5 The first region of interest image shown in the dotted box in the white light picture on the left; at the same time, the terminal can also be based on Figure 5 The position of the second polyp detection frame in the NBI image shown in FIG is obtained by expanding the second polyp detection frame. Figure 5 The second region of interest image is shown in the dotted box in the NBI image on the right, so that the final first region of interest image and the second region of interest image contain richer information.

[0066] Step 208 : performing feature extraction on the first ROI image and the second ROI image respectively to obtain white light polyp features of the first ROI image and narrow band polyp features of the second ROI image.

[0067] The white light polyp feature refers to the feature obtained by extracting the polyp region of interest in the white light polyp image.

[0068] Narrow-band polyp features refer to features extracted from polyp regions of interest in narrow-band polyp images. It is understood that the white-light polyp features and narrow-band polyp features in this application are only used to distinguish polyp image features extracted from different types of images.

[0069] Step 210 : Perform classification prediction based on the white light polyp feature and the narrow band polyp feature to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

[0070] Among them, the preset multiple polyp infiltration degree categories refer to pre-configured polyp infiltration degree categories. For example, the preset multiple polyp infiltration degree categories in this application include but are not limited to: non-adenomatous polyps, intraepithelial neoplasia, submucosal superficial invasive carcinoma and submucosal deep invasive carcinoma. These four categories may also include other customized categories.

[0071] Specifically, after the terminal obtains the first region of interest image and the second region of interest image, the terminal can perform feature extraction on the first region of interest image and the second region of interest image respectively to obtain the white light polyp feature of the first region of interest image and the narrow-band polyp feature of the second region of interest image. For example, the terminal can perform feature extraction on the first region of interest image and the second region of interest image through a trained prediction model to obtain the white light polyp feature of the first region of interest image and the narrow-band polyp feature of the second region of interest image; further, the terminal can fuse the white light polyp feature and the narrow-band polyp feature through the fully connected layer in the trained prediction model to obtain the polyp fusion feature, and perform classification prediction on the polyp fusion feature to determine the polyp infiltration degree category to which the polyp image detected from the colonoscopy image belongs from a plurality of preset polyp infiltration degree categories.

[0072] For example, Figure 6 As shown in FIG, a flowchart of fusion prediction based on white light polyp ROI and NBI polyp ROI is shown. After the terminal obtains the first region of interest image and the second region of interest image, as shown in FIG. Figure 6 As shown in , the terminal can simultaneously input the first region of interest image and the second region of interest image into the trained prediction model to Figure 6The backbone part of the convolutional neural network (CNN) in the trained prediction model shown in the figure extracts features of the first region of interest image and the second region of interest image respectively to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image; further, the terminal fuses the white-light polyp features and narrow-band polyp features output by the backbone part of the convolutional neural network (CNN) through the fully connected layer in the trained prediction model to obtain polyp fusion features, and classifies and predicts the polyp fusion features through the classification network in the trained prediction model to determine the polyp infiltration degree category to which the polyp image detected from the colonoscopy image belongs from a plurality of preset polyp infiltration degree categories (four categories: non-adenomatous polyps, intraepithelial neoplasia, submucosal superficial invasive carcinoma and submucosal deep invasive carcinoma).

[0073] In this embodiment, when a polyp image is detected from a colonoscopy image, a white-light polyp image acquired in a white-light mode and a narrow-band polyp image acquired in a narrow-band imaging mode are obtained; based on the position of a first polyp detection frame in the white-light polyp image, the first polyp detection frame is expanded to obtain a first region of interest image; based on the position of a second polyp detection frame in the narrow-band polyp image, the second polyp detection frame is expanded to obtain a second region of interest image; feature extraction is performed on the first region of interest image and the second region of interest image, respectively, to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image; and classification prediction is performed based on the white-light polyp features and the narrow-band polyp features to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories. Since the white-light polyp image and the narrow-band polyp image are collected when the polyp image is detected in the colonoscopy image, the first region of interest image determined based on the first polyp detection frame in the white-light polyp image and the second region of interest image determined based on the second polyp detection frame in the narrow-band polyp image have richer image information, so that when subsequent classification prediction is performed based on the white-light polyp features extracted from the first region of interest image and the narrow-band polyp features extracted from the second region of interest image, more accurate classification prediction results can be obtained, which can effectively improve the accuracy of polyp detection results while ensuring automatic polyp detection.

[0074] In one embodiment, the method further comprises:

[0075] Acquire real-time colonoscopy video, collect the real-time colonoscopy video frame by frame, and obtain a colonoscopy image;

[0076] Perform polyp detection on colonoscopy images using a polyp detection model;

[0077] When a polyp image is detected from the colonoscopic image, the colonoscopic image is displayed, and a polyp detection frame marking the polyp image detected from the colonoscopic image is displayed on the colonoscopic image;

[0078] When the polyp image marked by the polyp detection frame does not meet the visual diagnosis conditions, the steps of detecting a polyp image from a colonoscopy image, acquiring a white light polyp image acquired in a white light mode, and acquiring a narrow band polyp image acquired in a narrow band imaging mode are performed.

[0079] Among them, the colonoscopy real-time video refers to the real-time inspection video collected by the endoscope host, that is, the colonoscopy real-time video in this application can also be understood as the electronic endoscope inspection video. The colonoscopy real-time video is the video of the colonoscopy inspection, and the colonoscopy real-time video contains multiple frames of colonoscopy inspection pictures.

[0080] A polyp detection model refers to a pre-trained model for polyp image detection. For example, the polyp detection model in this application can be an image-based target detection model used to detect polyps in colon endoscopy images. That is, the polyp detection model can use a typical network, such as Yolo V3.

[0081] The polyp detection frame refers to a detection frame used to mark the area where the polyp image is located. For example, the polyp detection frame in this application can be as follows: Figure 3 The black solid rectangular box shown in the second frame image on the right.

[0082] Visual diagnostic conditions refer to different pre-configured condition parameters for visual diagnosis. For example, the visual diagnostic conditions in this application include but are not limited to: determining whether the clarity of the polyp image marked by the polyp detection frame reaches a preset clarity threshold, determining whether the image brightness of the polyp image marked by the polyp detection frame reaches a preset brightness threshold, and other condition parameters.

[0083] Specifically, if Figure 7 As shown in FIG, it is a schematic diagram of the application of the method provided by this application on the product side. Figure 7 The endoscope host shown in the figure is in external mode for example. Figure 7 As shown in , the endoscope-assisted system (i.e., diagnostic application) in the terminal can obtain real-time colonoscopy video by connecting to the endoscope host, and collect the real-time colonoscopy video frame by frame to obtain multiple frames of colonoscopy images; further, the endoscope-assisted system in the terminal can perform polyp image detection on the obtained multiple frames of colonoscopy images through a pre-trained polyp detection model; when a polyp image is detected from the colonoscopy image, the terminal can display the colonoscopy image through a display, and display a polyp detection frame on the colonoscopy image to mark the polyp image detected from the colonoscopy image, as shown in FIG. Figure 3As shown in , the polyp detection frame can be as follows Figure 3 The black solid rectangular box shown in the second frame image on the right of the image shown in the figure; further, the terminal can determine whether the polyp image marked by the polyp detection box meets the preset visual diagnosis conditions. For example, the terminal can determine whether the image brightness of the polyp image marked by the polyp detection box reaches the preset brightness threshold. When the image brightness of the polyp image marked by the polyp detection box reaches the preset brightness threshold, the terminal can further determine whether the image size of the polyp image marked by the polyp detection box reaches the preset size threshold, until it is determined that the polyp image marked by the polyp detection box meets all the visual diagnosis condition parameters, indicating that the polyp image marked by the polyp detection box meets the visual diagnosis conditions; otherwise, when the polyp image marked by the polyp detection box does not meet the visual diagnosis conditions, the terminal automatically triggers the execution of the steps of: when the polyp image is detected from the colonoscopy image, obtaining the white light polyp image collected in the white light mode, and obtaining the narrow band polyp image collected in the narrow band imaging mode. It can be understood that the visual diagnostic conditions mentioned in the embodiments of the present application can be visual diagnostic conditions for certain condition parameters, or can be visual diagnostic conditions obtained by combining condition parameters of different dimensions of polyp images, and no specific limitation is made here.

[0084] In this embodiment, the first region of interest image determined based on the first polyp detection frame in the white-light polyp image and the second region of interest image determined based on the second polyp detection frame in the narrow-band polyp image have richer image information, so that when subsequent classification prediction is performed based on the white-light polyp features extracted from the first region of interest image and the narrow-band polyp features extracted from the second region of interest image, more accurate classification prediction results can be obtained, which can effectively improve the accuracy of polyp detection results while ensuring automatic polyp detection.

[0085] In one embodiment, the step of acquiring a white light polyp image acquired in a white light mode and acquiring a narrow band polyp image acquired in a narrow band imaging mode comprises:

[0086] In response to input of a first type of start signal, the imaging mode is set to a white light mode indicated by the first type of start signal, and a polyp image is instantly acquired in the white light mode to obtain a white light polyp image acquired in the white light mode;

[0087] In response to the input of the second type of start signal, the imaging mode is set to the narrowband imaging mode indicated by the second type of start signal, and the polyp image is instantly acquired in the narrowband imaging mode to obtain the narrowband polyp image acquired in the narrowband imaging mode.

[0088] The first type of start signal and the second type of start signal in this application are only used to distinguish different types of input signals. For example, the first type of start signal in this application may be a type A start signal, and the second type of start signal may be a type B start signal.

[0089] Specifically, if Figure 8 As shown, it is a schematic diagram of the overall process of the polyp image processing method provided by the present application. When the terminal obtains real-time colonoscopy video and collects the real-time colonoscopy video frame by frame to obtain multiple frames of colonoscopy images, the terminal can perform polyp image detection on the obtained multiple frames of colonoscopy images through a pre-trained polyp detection model; when a polyp image is detected from the colonoscopy image and it is determined that the polyp image marked by the polyp detection frame does not meet the visual diagnosis conditions, the terminal responds to the input of a Class A start signal, sets the imaging mode of the built-in endoscope to the white light mode indicated by the Class A start signal, and immediately collects polyp images in the white light mode to obtain white-light polyp images collected in the white light mode; further, the terminal responds to the input of a Class B start signal, sets the imaging mode of the built-in endoscope to the narrow-band imaging mode indicated by the Class B start signal, and immediately collects polyp images in the narrow-band imaging mode to obtain narrow-band polyp images collected in the narrow-band imaging mode. As a result, it is possible to automatically extract polyp ROI areas with higher signal-to-noise ratios, and use white light and NBI images for fusion prediction, achieving the technical effect of effectively improving the accuracy of polyp detection results while ensuring automatic polyp detection.

[0090] In one embodiment, the method further comprises:

[0091] performing image recognition on the white-light polyp image and the narrow-band polyp image respectively to obtain a first image type to which the white-light polyp image belongs and a second image type to which the narrow-band polyp image belongs;

[0092] Performing a first check on the consistency between the first image type and the white light image type, and performing a second check on the consistency between the second image type and the narrowband image type;

[0093] In response to both the first check and the second check passing, continuing the process;

[0094] In response to at least one of the first verification or the second verification failing, the process ends.

[0095] The first image type and the second image type are only used to distinguish different image types. For example, the image types in this application include but are not limited to: white light image type and narrowband image type, and may also include other image types.

[0096] The first verification and the second verification are only used to distinguish different verification processes. For example, the first verification in this application can be a verification process for the first image type to which the white light polyp image belongs, and the second verification can be a verification process for the second image type to which the narrow-band polyp image belongs.

[0097] Specifically, if Figure 4 As shown in , it is assumed that the terminal obtains the white light polyp image collected in the white light mode as Figure 4 The white light image shown in , and the narrow band polyp image acquired in narrow band imaging mode are Figure 4 The NBI picture shown in , the terminal can first obtain Figure 4 The white light image and the NBI image shown in the figure are subjected to image recognition to obtain a first image type to which the white light image belongs and a second image type to which the NBI image belongs; further, the terminal can perform a first check on the consistency between the first image type and the white light image type, and a second check on the consistency between the second image type and the narrowband image type. When both the first check and the second check indicate that the check is passed, the terminal can further perform the following steps based on the example of FIG. Figure 4 The first polyp detection frame is located in the white light image as shown in FIG. , and the first polyp detection frame is expanded to obtain the first region of interest image in the white light image; at the same time, the terminal can also be based on the following example Figure 4 The position of the second polyp detection frame in the NBI image shown in FIG is shown in FIG. By expanding the second polyp detection frame, a second region of interest image in the NBI image can be obtained, so that the final first region of interest image and the second region of interest image contain richer information.

[0098] In addition, if Figure 4 As shown in , when at least one of the first check or the second check indicates a failure, the terminal directly terminates the process. That is, the terminal terminates the process in response to at least one of the first check or the second check failing. This allows the captured white light image and NBI image to be identified to ensure that the captured image meets the requirements. Only when the identification results of the white light image and the NBI image match the corresponding image type will the step of executing the infiltration level prediction be triggered. Otherwise, the process is terminated without performing the infiltration level prediction, thereby ensuring the accuracy of the infiltration level prediction result.

[0099] In one embodiment, the steps of performing image recognition on the white-light polyp image and the narrow-band polyp image to obtain a first image type to which the white-light polyp image belongs and a second image type to which the narrow-band polyp image belongs include:

[0100] Performing image classification on the white-light polyp image using a pre-trained polyp image classification model, and outputting a first image type to which the white-light polyp image belongs;

[0101] The narrow-band polyp image is classified by the polyp image classification model, and a second image type to which the narrow-band polyp image belongs is output.

[0102] The polyp image classification model refers to a model used to identify the type of image. For example, the polyp image classification model in this application can use ResNet50, such as Figure 9 As shown in the figure, it is a schematic diagram of the ResNet series network architecture. The network structure of ResNet50 used in the polyp image classification model in this application is as follows Figure 9 Shown in the dotted box.

[0103] Specifically, it is assumed that the terminal obtains the white light polyp image collected in the white light mode as Figure 4 The white light image shown in , and the narrow band polyp image acquired in narrow band imaging mode are Figure 4 The NBI picture shown in , the terminal can first obtain Figure 4 The white light picture and NBI picture shown in the figure are used for image recognition to obtain the first image type to which the white light picture belongs and the second image type to which the NBI picture belongs. For example, the terminal can use pre-trained Figure 9 The polyp image classification model, shown in the dashed box, performs image classification on the acquired white-light polyp image and outputs a first image type to which the white-light polyp image belongs. The polyp image classification model also performs image classification on the acquired narrow-band polyp image and outputs a second image type to which the narrow-band polyp image belongs. This ensures that the acquired white-light image and NBI image meet the requirements by identifying them. Only when the identification results of the white-light image and NBI image match the corresponding image type will the invasion degree prediction step be triggered. Otherwise, the process ends without performing invasion degree prediction, thereby ensuring the accuracy of the invasion degree prediction result.

[0104] In one embodiment, the method further comprises:

[0105] Acquire multiple frames of images acquired by a colonoscopy; some of the multiple frames of images contain polyp images;

[0106] Determine labels for each of the multiple frames, where the labels identify whether the images contain polyp images;

[0107] The initial recognition model is trained using multiple frames of images and their respective labels to obtain a trained image classification model.

[0108] Among the multiple frames of images collected by the colonoscope, some images contain polyp images, while the other images do not contain polyp images.

[0109] The label is used to identify whether each frame of the image contains a polyp image. For example, when a frame of the image contains a polyp image, the label of the frame of the image is determined to be 1; when a frame of the image does not contain a polyp image, the label of the frame of the image is determined to be 0.

[0110] Specifically, before the terminal performs image recognition on white-light polyp images and narrow-band polyp images respectively, the terminal can pre-train an image classification model for recognizing white-light polyp images and narrow-band polyp images. For example, the terminal can obtain multiple frames of images collected by a colonoscope and use the multiple frames of images as a training set; wherein, some of the multiple frames of images in the training set contain polyp images; further, the terminal can determine the labels of the multiple frames of images included in the training set, the labels identifying whether the images contain polyp images, and use the multiple frames of images and their respective labels to train the initial recognition model to obtain a trained image classification model. That is, in the training process of the recognition model for recognizing white-light images and NBI images provided in the embodiment of the present application, it is not necessary for each frame of images to contain a polyp image, but it is required that each frame of images be an image of a colonoscopy. As a result, the acquired white light images and NBI images are identified to ensure that the collected images meet the requirements. Only when the identification results of the white light images and NBI images meet the corresponding image types will the step of executing the infiltration degree prediction be triggered. Otherwise, the process will be terminated and the infiltration degree prediction will not be performed to ensure the accuracy of the infiltration degree prediction results.

[0111] In one embodiment, the step of expanding the first polyp detection frame based on the position of the first polyp detection frame in the white-light polyp image to obtain the first region of interest image includes:

[0112] Determine a length value and a width value of a first polyp detection frame in the white light polyp image;

[0113] Determine the expanded length value based on the length value and the preset ratio;

[0114] Determine the enlarged width value based on the width value and the preset ratio;

[0115] Based on the position, the expanded length value, and the expanded width value of the first polyp detection frame in the white light polyp image, the first polyp detection frame is expanded to obtain a first region of interest image.

[0116] The expanded length value refers to the length value of the expanded detection frame, and the expanded width value refers to the width value of the expanded detection frame.

[0117] Specifically, if Figure 5 As shown in the figure, it is a schematic diagram of intercepting the polyp ROI image. Assume that the terminal obtains the white light polyp image collected in the white light mode as Figure 5The white light image shown on the left and the narrow band polyp image acquired in narrow band imaging mode are Figure 5 When the terminal performs the first verification on the white light image and the second verification on the NBI image, and the verification results both indicate that the verification is passed, the terminal can further perform the verification based on the following example: Figure 5 The first polyp detection frame is positioned in the white light image as shown in the figure, and the first polyp detection frame is expanded. That is, the terminal can first determine that the length value a and the width value b of the first polyp detection frame in the white light polyp image are, and based on the length value a and the preset ratio 50%, determine that the expanded length value is a+50%a; similarly, the terminal can determine that the expanded width value is b+50%b based on the width value b and the preset ratio 50%; further, the terminal can be based on the following example. Figure 5 The position of the first polyp detection frame in the white light image shown in FIG, the expanded length value a+50%a and the expanded width value b+50%b, and the first polyp detection frame are expanded to obtain the following: Figure 5 The first ROI image is shown in the dashed box in the white light image on the left. This allows the resulting first ROI image to contain richer information, thereby improving the signal-to-noise ratio and making the subsequent determination of the polyp invasion degree category based on the first and second ROI images more accurate.

[0118] In one embodiment, the step of expanding the second polyp detection frame based on the position of the second polyp detection frame in the narrow-band polyp image to obtain the second region of interest image includes:

[0119] determining a length value and a width value of a second polyp detection frame in the narrow-band polyp image;

[0120] Determine the expanded length value based on the length value and the preset ratio;

[0121] Determine the enlarged width value based on the width value and the preset ratio;

[0122] Based on the position, the expanded length value, and the expanded width value of the second polyp detection frame in the narrow-band polyp image, the second polyp detection frame is expanded to obtain a second region of interest image.

[0123] Specifically, if Figure 5 As shown in the figure, it is a schematic diagram of intercepting the polyp ROI image. Assume that the terminal obtains the white light polyp image collected in the white light mode as Figure 5 The white light image shown on the left and the narrow band polyp image acquired in narrow band imaging mode are Figure 5 When the terminal performs the first verification on the white light image and the second verification on the NBI image, and the verification results both indicate that the verification is passed, the terminal can further perform the verification based on the following example: Figure 5 The position of the second polyp detection frame in the NBI image shown in is used to expand the second polyp detection frame. That is, the terminal may first determine that the length value a and the width value b of the second polyp detection frame in the NBI image are, and based on the length value a and the preset ratio 50%, determine that the expanded length value is a+50%a; similarly, the terminal may determine that the expanded width value is b+50%b based on the width value b and the preset ratio 50%; further, the terminal may determine that the expanded width value is b+50%b based on the width value b and the preset ratio 50%. Figure 5 The position of the second polyp detection frame in the NBI image shown in FIG, the expanded length value a+50%a and the expanded width value b+50%b, and the second polyp detection frame are expanded to obtain the following: Figure 5 The second ROI image is shown in the dotted box in the NBI image on the left. This allows the resulting second ROI image to contain richer information, thereby improving the signal-to-noise ratio and making the subsequent determination of the polyp invasion degree category based on the first and second ROI images more accurate.

[0124] In one embodiment, feature extraction and classification prediction are achieved through a trained prediction model. The training steps of the prediction model include:

[0125] Based on independent white-light polyp region-of-interest images and independent narrow-band polyp region-of-interest images, an initial convolutional neural network is trained to obtain a pre-trained model;

[0126] The paired white-light polyp region of interest images and narrow-band polyp region of interest images are combined, and the combined white-light polyp region of interest images and narrow-band polyp region of interest images are input into a pre-trained model for training to obtain a trained prediction model.

[0127] Specifically, if Figure 10 As shown in FIG, a schematic diagram of the training method of the pre-trained model Model_1 for infiltration degree prediction is shown, that is, the terminal can first train the initial convolutional neural network based on independent white light polyp region of interest images and independent narrow band polyp region of interest images to obtain the following: Figure 10 The pre-trained model Model_1 shown in ; further, as Figure 11 As shown in FIG. 1 , a schematic diagram of a training method of the invasion degree prediction model Model_2 for invasion degree prediction is shown. The terminal can combine paired white light polyp region of interest images and narrow band polyp region of interest images, and input the combined white light polyp region of interest images and narrow band polyp region of interest images into the training method. Figure 11The pre-trained model Model_1 (CNN) shown in the figure is trained to obtain the trained prediction model Model_2. Thus, a single-input method is first adopted, and a training set consisting of white-light polyp ROI images and NBI polyp ROI images is used for training to form the pre-trained model Model_1. To provide more image detail information, Model_1 is then trained using a dual-input method of a paired combination of white-light polyp ROI images and NBI polyp ROI images to obtain the final model Model_2 for invasion degree prediction. This makes it possible to more accurately determine the polyp invasion degree category of the polyp image after subsequent processing of the first and second region of interest images using this prediction model.

[0128] In one embodiment, the step of training an initial convolutional neural network based on independent white-light polyp region-of-interest images and independent narrow-band polyp region-of-interest images to obtain a pre-trained model includes:

[0129] Acquire a first training set, the first training set including independent white-light polyp region-of-interest images and independent narrow-band polyp region-of-interest images;

[0130] The images in the first training set are randomly input into the initial convolutional neural network for training to obtain a pre-trained model.

[0131] Specifically, if Figure 10 As shown in FIG. 1 , a schematic diagram of a training method of a pre-trained model Model_1 for infiltration degree prediction is shown. That is, the terminal can obtain a first training set, which includes independent white light polyp region of interest images and independent narrow band polyp region of interest images, and randomly input the images in the first training set into the image processing system as shown in FIG. Figure 10 Alternatively, the terminal may obtain a first training set, which includes independent white light polyp region of interest images and independent narrow band polyp region of interest images, and input the images in the first training set into a pre-trained model in a regularly spaced order. Figure 10 The initial convolutional neural network (Model_1) shown in is trained to obtain a pre-trained model. It can be understood that in this application, the images in the first training set are input into the Figure 10 When training the initial convolutional neural network (Model_1) shown in , a single input method can be used. A random single input method or a regular sequential single input method can be used. There is no specific limitation here.

[0132] In this embodiment, it is possible to automatically extract polyp ROI areas with a higher signal-to-noise ratio, and use white light and NBI images for fusion prediction, thereby achieving the technical effect of effectively improving the accuracy of polyp detection results while ensuring automatic polyp detection.

[0133] In one embodiment, the steps of combining a paired white-light polyp region of interest image and a narrow-band polyp region of interest image, inputting the combined white-light polyp region of interest image and the narrow-band polyp region of interest image into a pre-trained model for training, and obtaining a trained prediction model include:

[0134] obtaining a second training set, the second training set including paired white light polyp region of interest images and narrow band polyp region of interest images;

[0135] Inputting the white light polyp region of interest image and the narrow band polyp region of interest image in each pair of training data in the second training set into the pre-training model for training to obtain a trained prediction model;

[0136] The performing feature extraction on the first region of interest image and the second region of interest image respectively to obtain white light polyp features of the first region of interest image and narrow band polyp features of the second region of interest image includes:

[0137] Inputting the first region of interest image and the second region of interest image into the trained prediction model simultaneously, so as to perform feature extraction on the first region of interest image and the second region of interest image respectively using the trained prediction model to obtain white light polyp features of the first region of interest image and narrow band polyp features of the second region of interest image;

[0138] The performing classification prediction based on the white light polyp feature and the narrow band polyp feature to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories includes:

[0139] The white light polyp features and narrow-band polyp features are fused through the fully connected layer in the trained prediction model to obtain the polyp fusion features, and the polyp fusion features are classified and predicted to determine the polyp infiltration degree category to which the polyp image belongs from multiple preset polyp infiltration degree categories.

[0140] Specifically, if Figure 11As shown in FIG. 1 , a schematic diagram of a training method of the invasion degree prediction model Model_2 for invasion degree prediction is shown. The terminal can obtain a second training set, which includes paired white light polyp region of interest images and narrow band polyp region of interest images, and simultaneously input the white light polyp region of interest images and narrow band polyp region of interest images in each pair of training data in the second training set into the training data set. Figure 11 The pre-trained model Model_1 (CNN) shown in is trained to obtain the trained prediction model Model_2; further, as Figure 6 As shown in , the terminal can simultaneously input the obtained first and second ROI images into the trained prediction model Model_2, and then use the backbone (CNN) portion of the trained prediction model to extract features from the first and second ROI images, respectively, to obtain white-light polyp features for the first ROI image and narrow-band polyp features for the second ROI image. The white-light polyp features and narrow-band polyp features are then fused together using the fully connected layer of the trained prediction model to obtain a fused polyp feature. Furthermore, the terminal can also classify and predict the fused polyp feature using the classification network layer of the trained prediction model to determine the polyp infiltration category to which the polyp image detected in the colonoscopy image belongs from among multiple preset polyp infiltration categories. This allows for the automatic extraction of polyp ROI regions with higher signal-to-noise ratios, and the fusion prediction using white-light and NBI images, achieving the technical effect of effectively improving the accuracy of polyp detection results while ensuring automatic polyp detection.

[0141] In one embodiment, the present application further provides an application scenario, in which the above-mentioned polyp image processing method is applied. Specifically, the application of the polyp image processing method in this application scenario is as follows:

[0142] When the user wants to detect polyps in the colon, the above-mentioned polyp image processing method can be used. The method provided in this application can be applied to an online diagnostic system or diagnostic application, so that the accuracy of the polyp detection results can be effectively improved while ensuring the automatic detection of polyps. Different users can open the online diagnostic system on the terminal through a trigger operation, and enter the polyp automatic detection page in the diagnostic system through a selection operation. When the diagnostic system detects a polyp image from a colonoscopy image, the diagnostic system obtains a white-light polyp image collected in white-light mode, and obtains a narrow-band polyp image collected in narrow-band imaging mode; further, the diagnostic system is based on the white-light polyp image. The first polyp detection frame is positioned in the white-light polyp image, and the first polyp detection frame is expanded to obtain a first region of interest image. At the same time, based on the position of the second polyp detection frame in the narrow-band polyp image, the second polyp detection frame can be expanded to obtain a second region of interest image. Furthermore, the diagnostic system performs feature extraction on the first region of interest image and the second region of interest image respectively to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image, and performs classification prediction based on the white-light polyp features and the narrow-band polyp features to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories. Compared with traditional methods, this application proposes a polyp image processing method for auxiliary diagnosis of colorectal polyp detection and classification based on artificial intelligence. The processing method based on this method can reduce the missed detection rate of colonoscopy, improve the effectiveness of examination, and further provide a prediction of the degree of polyp infiltration on the basis of polyp detection, assisting doctors in selecting treatment plans, and can achieve the technical effect of effectively improving the accuracy of polyp detection results while ensuring automatic detection of polyps.

[0143] The method provided in the embodiments of the present application can be applied to various polyp image processing scenarios. The following describes the polyp image processing method provided in the embodiments of the present application using the scenario of an artificial intelligence-based auxiliary diagnosis system for colorectal polyp detection and classification as an example.

[0144] Sensitivity: The proportion of images detected as malignant (i.e., lung cancer) among all malignant images.

[0145] Specificity: The proportion of images detected as benign among all benign (i.e., not containing lung cancer) images.

[0146] Candidate box: A professional term that often appears in target detection, which frames the location of the candidate target area.

[0147] Deep learning technology: a technology that uses deep neural network systems to perform machine learning.

[0148] Traditionally, artificial intelligence technology is used to learn from colon polyp images to achieve automatic detection of colon polyps. However, due to the large number of parameters and long inference time of deep learning networks, their efficiency is not high in practical applications. Therefore, how to ensure the automatic detection of polyps while effectively improving the accuracy of polyp detection results has become an urgent problem that needs to be solved.

[0149] The above traditional methods have the following disadvantages:

[0150] 1) Polyps are detected, which only assists doctors in the discovery of polyps;

[0151] 2) When doctors are treating polyps, the system cannot provide auxiliary information, and the accuracy of polyp detection results is not accurate enough. Therefore, how to effectively improve the accuracy of polyp detection results while ensuring the automatic detection of polyps has become an urgent problem that needs to be solved.

[0152] Therefore, in order to solve the above problems, the present application provides a solution, that is, the present application proposes a colorectal polyp detection and classification auxiliary diagnosis system based on artificial intelligence technology. The system is based on electronic endoscopic images and uses deep learning technology to automatically detect and classify polyps in the colon. When the doctor believes that it is suspected cancer, the infiltration depth is predicted to assist the doctor in choosing a treatment plan. The input of the method provided in the embodiment of the present application is an electronic endoscopic examination video and two types of different input start signals. The inspection video can be a real-time video stream, and the input signal can be any form of signal input through a computer, such as a keyboard key, a mouse button, or a medical foot pedal.

[0153] The system automatically detects colon polyps upon startup. When a polyp is detected, the system displays a rectangular box to alert the physician. Once a polyp is confirmed and suspected of being an adenomatous polyp or carcinoma, the system automatically predicts the degree of invasion. First, the endoscope is in white light mode and aimed at the polyp at a preferred angle to capture video. In response to a Class A activation signal, the auxiliary diagnostic system captures a white light image of the polyp. Subsequently, the endoscope switches to narrow-band imaging (NBI) mode, and the auxiliary diagnostic system, in response to a Class B activation signal, captures an NBI image of the polyp. Finally, the auxiliary diagnostic system fuses the white light and NBI images of the detected polyp (physical morphology) to automatically predict and classify the polyp image as non-adenomatous polyp, intraepithelial neoplasia, superficial submucosal invasive carcinoma, or deep submucosal invasive carcinoma.

[0154] On the product side, the method provided in this application can be applied in two ways: external mode and internal mode. Figure 12The following is a schematic diagram of the product-side application. The business logic of the external mode is to deploy the auxiliary diagnosis system on independent hardware and connect it to the endoscope host to obtain real-time examination video, and then perform real-time auxiliary diagnosis. The business logic of the built-in mode is to deploy the auxiliary diagnosis system on the endoscope system and complete the auxiliary diagnosis there.

[0155] On the technical side, 1.1 Overall framework and data flow

[0156] like Figure 13 The figure shows the overall processing flow diagram of the auxiliary diagnosis system for colorectal polyp detection and classification based on artificial intelligence. The overall flow diagram of the technical solution provided by this application is as follows Figure 13 As shown in . The real-time video of the colon endoscope will be collected frame by frame into video frame images and sent to the polyp detection module. The polyp detection module performs polyp detection on the video frame images. If there are no polyps, the current frame image processing process ends. If there are polyps and the terminal, that is, the system, responds to the input of the Class A start signal, it collects the white light image at that moment; then the terminal, that is, the system, responds to the input of the Class B start signal, collects the NBI image at that moment; the white light image and the NBI image are transmitted to the infiltration depth prediction module to predict the infiltration depth. After the prediction is completed, the auxiliary diagnosis process of the current polyp ends.

[0157] 1.2 Polyp Detection

[0158] like Figure 14 The figure shows the polyp detection effect of the artificial intelligence-based colorectal polyp detection and classification auxiliary diagnosis system. The polyp detection module performs real-time polyp detection based on the colon endoscopy video frame image. The effect is as follows Figure 14 As shown in . The polyp detection model is an image-based target detection model used to detect polyps in colon endoscopy images. A typical network is Yolo V3.

[0159] 1.3 Collecting white light images and collecting NBI images

[0160] When the system has a polyp detection result, and the system determines the presence of polyps based on visual diagnostic conditions and determines the presence of suspected adenomatous polyps or cancers, the prediction process of the degree of infiltration is automatically executed. First, the system adjusts the endoscope to white light mode and aims at the polyp at a better angle to collect video. The system responds to the input of a Class A start signal and collects a white light picture of the current polyp. Subsequently, the system changes the endoscope to narrow band imaging mode (NBI). The system responds to the input of a Class B start signal and collects an NBI picture of the current polyp. After collecting the white light picture and the NBI picture, the system identifies the obtained white light picture and the NBI picture to ensure that the collected pictures meet the requirements. A typical method for the recognition model can use ResNet50, such as Figure 9As shown in . During the training process of the recognition model of white light images and NBI images, it is not necessary for all images to contain polyps, but it is required that each image must be a colonoscopy image. If the recognition results of the white light image and the NBI image are of the corresponding image type, the system automatically executes the process of infiltration degree prediction, otherwise the process ends and the process of infiltration degree prediction is not executed. Figure 15 As shown in Figure 15 Shown is a schematic diagram of the prediction of the degree of infiltration of the auxiliary diagnosis system for colorectal polyp detection and classification based on artificial intelligence.

[0161] 1.4 Prediction of infiltration depth

[0162] After collecting the white light image and NBI image of the polyp, the polyp ROI (region of interest) is first intercepted according to the polyp detection results, such as Figure 16 Figure 2 shows a schematic diagram of a polyp ROI for an AI-based auxiliary diagnosis system for colorectal polyp detection and classification. To include more information, the ROI must encompass both the polyp region and the surrounding area for comparison. Therefore, the intercepted region is larger than the polyp detection frame, for example, by 50% in both length and width. Figure 16 The solid-line frame is the polyp detection frame, and the dotted-line frame is the result of expanding the solid-line frame by 50%, which is also the ROI interception frame.

[0163] After obtaining the white light polyp ROI and NBI polyp ROI, the system can automatically perform fusion invasion degree prediction. First, the system uses the backbone of the convolutional neural network (CNN) to extract features. The typical method is the backbone of ResNet50, such as Figure 9 The solid line frame part. After extracting the corresponding features of white light polyp ROI and NBI polyp ROI respectively, the system performs feature fusion on the two features, using the stacking method to form a unified feature. Finally, the system classifies and predicts the same feature based on the fully connected layer. The categories include non-adenomatous polyps, intraepithelial neoplasia, submucosal superficial invasive carcinoma and submucosal deep invasive carcinoma. The feature extraction CNN first uses a single input method and is trained with a training set consisting of white light polyp ROI and NBI polyp ROI to form a pre-training model Model_1, as shown in Figure 1. Figure 17 The figure shows the training method of the pre-trained model Model_1 for predicting the degree of invasion of the auxiliary diagnosis system for colorectal polyp detection and classification based on artificial intelligence. Then, the dual input method of white light polyp ROI + NBI polyp ROI is used to train Model_1 to obtain the final model Model_2 for predicting the degree of invasion, as shown in Figure 1. Figure 18 The figure shows a schematic diagram of the training method of the infiltration degree prediction model Model_2 of the auxiliary diagnosis system for colorectal polyp detection and classification based on artificial intelligence.

[0164] The innovative points and implementation methods of this solution are summarized as follows:

[0165] a. It has both polyp detection and invasion degree prediction, is fully functional and based on deep learning technology, which can effectively help doctors improve accuracy.

[0166] b. Automatic polyp detection and manual collection of key images by doctors for infiltration degree prediction.

[0167] c. Automatically extract polyp ROI areas with higher signal-to-noise ratios and use white light and NBI images for fusion prediction.

[0168] The beneficial effects of the technical solution of this application include:

[0169] a. Provide doctors with auxiliary diagnosis functions for automatic polyp detection, reduce the missed detection rate of colonoscopy, and improve the effectiveness of examinations;

[0170] b. Based on polyp detection, it further provides a prediction of the degree of polyp infiltration to assist doctors in selecting treatment plans.

[0171] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0172] Based on the same inventive concept, embodiments of the present application also provide a polyp image processing device for implementing the aforementioned polyp image processing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of the one or more script program upgrade device embodiments provided below can be found in the aforementioned limitations of the polyp image processing method and will not be further elaborated here.

[0173] In one embodiment, Figure 19 As shown, a polyp image processing device is provided, including: an acquisition module 1902, an expansion module 1904, an extraction module 1906 and a determination module 1908, wherein:

[0174] The acquisition module 1902 is configured to acquire a white light polyp image acquired in a white light mode and a narrow band polyp image acquired in a narrow band imaging mode when a polyp image is detected from a colonoscopy image.

[0175] The expansion module 1904 is configured to expand the first polyp detection frame in the white-light polyp image based on its position in the white-light polyp image to obtain a first region of interest image; and to expand the second polyp detection frame in the narrow-band polyp image based on its position in the narrow-band polyp image to obtain a second region of interest image.

[0176] The extraction module 1906 is configured to perform feature extraction on the first ROI image and the second ROI image respectively to obtain white-light polyp features of the first ROI image and narrow-band polyp features of the second ROI image.

[0177] The determination module 1908 is configured to perform classification prediction based on the white light polyp feature and the narrow band polyp feature, so as to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

[0178] In one embodiment, the device further includes: an acquisition module, a detection module, a display module and an execution module, wherein the acquisition module is used to acquire real-time colonoscopy video and acquire the real-time colonoscopy video frame by frame to obtain the colonoscopy image; the detection module is used to perform polyp image detection on the colonoscopy image through a polyp detection model; the display module is used to display the colonoscopy image when a polyp image is detected from the colonoscopy image, and to display a polyp detection frame on the colonoscopy image that marks the polyp image detected from the colonoscopy image; the execution module is used to execute the steps of acquiring a white-light polyp image acquired in a white-light mode and a narrow-band polyp image acquired in a narrow-band imaging mode when a polyp image is detected from the colonoscopy image, when the polyp image marked by the polyp detection frame does not meet the visual diagnosis conditions.

[0179] In one embodiment, the device further includes: an acquisition module for, in response to the input of a first type of start signal, setting the imaging mode to the white light mode indicated by the first type of start signal, and instantly acquiring the polyp image in the white light mode to obtain the white light polyp image acquired in the white light mode; in response to the input of a second type of start signal, setting the imaging mode to the narrow-band imaging mode indicated by the second type of start signal, and instantly acquiring the polyp image in the narrow-band imaging mode to obtain the narrow-band polyp image acquired in the narrow-band imaging mode.

[0180] In one embodiment, the device further includes: an identification module, a verification module, and a processing module, wherein the identification module is configured to perform image recognition on the white-light polyp image and the narrow-band polyp image, respectively, to obtain a first image type to which the white-light polyp image belongs, and a second image type to which the narrow-band polyp image belongs; the verification module is configured to perform a first verification on the consistency between the first image type and the white-light image type, and to perform a second verification on the consistency between the second image type and the narrow-band image type; the processing module is configured to continue executing the process in response to both the first verification and the second verification being passed; and terminate the process in response to at least one of the first verification or the second verification being failed.

[0181] In one embodiment, the device further includes: an image classification module, configured to perform image classification on the white-light polyp image using a pre-trained polyp image classification model, and output a first image type to which the white-light polyp image belongs; and to perform image classification on the narrow-band polyp image using the polyp image classification model, and output a second image type to which the narrow-band polyp image belongs.

[0182] In one embodiment, the acquisition module is also used to acquire multiple frames of images collected by a colonoscopy; some of the multiple frames of images contain polyp images; the device also includes: a determination module, used to determine the labels of each of the multiple frames of images, the labels identifying whether the images contain polyp images; a training module, used to train the initial recognition model using the multiple frames of images and the labels of each of the multiple frames of images to obtain the trained image classification model.

[0183] In one embodiment, the determination module is further used to determine the length value and width value of the first polyp detection frame in the white light polyp image; determine the expanded length value based on the length value and a preset ratio; determine the expanded width value based on the width value and the preset ratio; the expansion module is further used to expand the first polyp detection frame based on the position of the first polyp detection frame in the white light polyp image, the expanded length value and the expanded width value to obtain the first region of interest image.

[0184] In one embodiment, the determination module is further used to determine the length value and width value of the second polyp detection frame in the narrow-band polyp image; determine the expanded length value based on the length value and the preset ratio; determine the expanded width value based on the width value and the preset ratio; the expansion module is further used to expand the second polyp detection frame based on the position of the second polyp detection frame in the narrow-band polyp image, the expanded length value and the expanded width value to obtain the second region of interest image.

[0185] In one embodiment, the feature extraction and the classification prediction are achieved through a trained prediction model. The device also includes: a training module and an input module. The training module is used to train the initial convolutional neural network based on independent white-light polyp region of interest images and independent narrow-band polyp region of interest images to obtain a pre-trained model; the input module is used to combine paired white-light polyp region of interest images and narrow-band polyp region of interest images, and input the combined white-light polyp region of interest images and narrow-band polyp region of interest images into the pre-trained model for training to obtain a trained prediction model.

[0186] In one embodiment, the acquisition module is also used to acquire a first training set, which includes independent white-light polyp region of interest images and independent narrow-band polyp region of interest images; the device also includes: a training module for randomly inputting the images in the first training set into the initial convolutional neural network for training to obtain a pre-trained model.

[0187] In one embodiment, the acquisition module is further configured to acquire a second training set, the second training set comprising paired white-light polyp region-of-interest images and narrow-band polyp region-of-interest images; the input module is further configured to simultaneously input the white-light polyp region-of-interest images and the narrow-band polyp region-of-interest images in each pair of training data in the second training set into the pre-trained model for training to obtain a trained prediction model; the extraction module is further configured to simultaneously input the first region-of-interest image and the second region-of-interest image into the trained prediction model, so as to perform feature extraction on the first region-of-interest image and the second region-of-interest image respectively using the trained prediction model to obtain white-light polyp features of the first region-of-interest image and narrow-band polyp features of the second region-of-interest image; the apparatus further comprises: a fusion module configured to fuse the white-light polyp features and the narrow-band polyp features using a fully connected layer in the trained prediction model to obtain a fused polyp feature; and the determination module is further configured to perform classification prediction on the fused polyp feature to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

[0188] Each module in the polyp image processing device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0189] In one embodiment, a computer device is provided. The computer device may be a terminal or a server. In this embodiment, the computer device is described as a terminal. The internal structure diagram thereof may be as follows: Figure 20 As shown. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals via wired or wireless communication, and the wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near field communication), or other technologies. When executed by the processor, the computer program implements a polyp image processing method. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0190] Those skilled in the art will understand that Figure 20 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0191] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0192] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0193] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0194] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0195] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0196] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0197] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A polyp image processing method, characterized in that: The method comprises: When a polyp image is detected from the colonoscopy image, a white light polyp image acquired in a white light mode and a narrow band polyp image acquired in a narrow band imaging mode are acquired; enlarging the first polyp detection frame in the white-light polyp image based on a position of the first polyp detection frame in the white-light polyp image to obtain a first region of interest image; enlarging the second polyp detection frame in the narrow-band polyp image based on a position of the second polyp detection frame in the narrow-band polyp image to obtain a second region of interest image; performing feature extraction on the first region of interest image and the second region of interest image respectively to obtain white light polyp features of the first region of interest image and narrow band polyp features of the second region of interest image; Classification prediction is performed based on the white light polyp feature and the narrow band polyp feature to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

2. The method according to claim 1, characterized in that The method further comprises: Acquiring a real-time colonoscopy video, and collecting the real-time colonoscopy video frame by frame to obtain the colonoscopy image; performing polyp image detection on the colonoscopy image using a polyp detection model; When a polyp image is detected from the colonoscopic image, the colonoscopic image is displayed, and a polyp detection frame marking the polyp image detected from the colonoscopic image is displayed on the colonoscopic image; When the polyp image marked by the polyp detection frame does not meet the visual diagnosis conditions, the steps of: when a polyp image is detected from a colonoscopy image, obtaining a white light polyp image acquired in a white light mode, and obtaining a narrow band polyp image acquired in a narrow band imaging mode are performed.

3. The method according to claim 1, characterized in that The step of acquiring a white-light polyp image acquired in a white-light mode and acquiring a narrow-band polyp image acquired in a narrow-band imaging mode comprises: In response to input of a first type of start signal, setting the imaging mode to a white light mode indicated by the first type of start signal, and instantly acquiring a polyp image in the white light mode to obtain the white light polyp image acquired in the white light mode; In response to the input of the second type of start signal, the imaging mode is set to the narrowband imaging mode indicated by the second type of start signal, and the polyp image is instantly acquired in the narrowband imaging mode to obtain the narrowband polyp image acquired in the narrowband imaging mode.

4. The method according to claim 1, wherein The method further comprises: performing image recognition on the white-light polyp image and the narrow-band polyp image respectively to obtain a first image type to which the white-light polyp image belongs and a second image type to which the narrow-band polyp image belongs; Performing a first check on the consistency between the first image type and the white light image type, and performing a second check on the consistency between the second image type and the narrowband image type; In response to both the first check and the second check passing, continuing the process; In response to at least one of the first verification or the second verification failing, the process ends.

5. The method according to claim 4, wherein performing image recognition on the white-light polyp image and the narrow-band polyp image to obtain the first image type to which the white-light polyp image belongs and the second image type to which the narrow-band polyp image belongs comprises: performing image classification on the white-light polyp image using a pre-trained polyp image classification model, and outputting a first image type to which the white-light polyp image belongs; The narrow-band polyp image is classified by the polyp image classification model, and a second image type to which the narrow-band polyp image belongs is output.

6. The method according to claim 5, characterized in that The method further comprises: Acquire multiple frames of images acquired by a colonoscopy; some of the multiple frames of images contain polyp images; determining a label for each of the plurality of frames of images, the label identifying whether the image contains a polyp image; The initial recognition model is trained using the multiple frames of images and their respective labels to obtain the trained image classification model.

7. The method according to claim 1, characterized in that The step of expanding the first polyp detection frame based on the position of the first polyp detection frame in the white-light polyp image to obtain a first region of interest image includes: Determining a length value and a width value of a first polyp detection frame in the white light polyp image; Determining an expanded length value based on the length value and a preset ratio; Determining an enlarged width value based on the width value and a preset ratio; Based on the position of the first polyp detection frame in the white-light polyp image, the expanded length value, and the expanded width value, the first polyp detection frame is expanded to obtain the first region of interest image.

8. The method according to claim 1, characterized in that The step of expanding the second polyp detection frame based on a position of the second polyp detection frame in the narrow-band polyp image to obtain a second region of interest image includes: determining a length value and a width value of a second polyp detection frame in the narrow-band polyp image; Determining an expanded length value based on the length value and a preset ratio; Determining an enlarged width value based on the width value and a preset ratio; Based on the position of the second polyp detection frame in the narrow-band polyp image, the expanded length value, and the expanded width value, the second polyp detection frame is expanded to obtain the second region of interest image.

9. The method according to any one of claims 1 to 8, characterized in that The feature extraction and classification prediction are achieved through a trained prediction model, and the training steps of the prediction model include: Based on independent white-light polyp region-of-interest images and independent narrow-band polyp region-of-interest images, an initial convolutional neural network is trained to obtain a pre-trained model; The paired white-light polyp region of interest images and narrow-band polyp region of interest images are combined, and the combined white-light polyp region of interest images and narrow-band polyp region of interest images are input into the pre-training model for training to obtain a trained prediction model.

10. The method according to claim 9, wherein the training of the initial convolutional neural network based on the independent white-light polyp region of interest images and the independent narrow-band polyp region of interest images to obtain a pre-trained model comprises: Acquire a first training set, wherein the first training set includes independent white-light polyp region-of-interest images and independent narrow-band polyp region-of-interest images; The images in the first training set are randomly input into the initial convolutional neural network for training to obtain a pre-trained model.

11. The method according to claim 9, wherein the combining the paired white-light polyp region of interest images and narrow-band polyp region of interest images, and inputting the combined white-light polyp region of interest images and narrow-band polyp region of interest images into the pre-trained model for training to obtain a trained prediction model comprises: Acquire a second training set, wherein the second training set includes paired white-light polyp region-of-interest images and narrow-band polyp region-of-interest images; Inputting the white light polyp region of interest image and the narrow band polyp region of interest image in each pair of training data in the second training set into the pre-trained model for training to obtain a trained prediction model; The performing feature extraction on the first region of interest image and the second region of interest image respectively to obtain white light polyp features of the first region of interest image and narrow band polyp features of the second region of interest image includes: Inputting the first ROI image and the second ROI image into the trained prediction model simultaneously, so as to perform feature extraction on the first ROI image and the second ROI image respectively using the trained prediction model to obtain white-light polyp features of the first ROI image and narrow-band polyp features of the second ROI image; The performing classification prediction based on the white light polyp feature and the narrow band polyp feature to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories includes: The white light polyp feature and the narrow-band polyp feature are fused through the fully connected layer in the trained prediction model to obtain a polyp fusion feature, and the polyp fusion feature is classified and predicted to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

12. A polyp image processing device, characterized in that: The device comprises: an acquisition module, configured to acquire, when a polyp image is detected from a colonoscopy image, a white light polyp image acquired in a white light mode and a narrow band polyp image acquired in a narrow band imaging mode; an expansion module, configured to expand the first polyp detection frame in the white-light polyp image based on the position of the first polyp detection frame in the white-light polyp image to obtain a first region-of-interest image; and to expand the second polyp detection frame in the narrow-band polyp image based on the position of the second polyp detection frame in the narrow-band polyp image to obtain a second region-of-interest image; an extraction module, configured to perform feature extraction on the first region of interest image and the second region of interest image, respectively, to obtain white-light polyp features of the first region of interest image and narrow-band polyp features of the second region of interest image; A determination module is configured to perform classification prediction based on the white light polyp feature and the narrow band polyp feature, so as to determine the polyp infiltration degree category to which the polyp image belongs from a plurality of preset polyp infiltration degree categories.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images