Endoscope-assisted inspection method and device based on artificial intelligence

Abstract

The present invention provides an artificial intelligence-based endoscopic assisted inspection method and device, wherein the method includes: inputting the video stream collected by the endoscope into at least one model, using the at least one model to determine the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position, and providing a real-time prompt of the current position of the endoscope; if the current position of the endoscope is not clean, a prompt is provided to clean it; if the current position of the endoscope is a lesion, a prompt is provided to stain the endoscope according to the lesion condition, or lesion information is provided. This improves the quality of endoscopic inspection.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to an endoscope-assisted inspection method and device based on artificial intelligence. Background Art

[0002] Early detection of digestive tract tumors can greatly improve the cure rate and reduce the mortality rate. Gastrointestinal endoscopy (or endoscopy, including gastroscopy and colonoscopy) and pathological biopsy are currently the gold standard for diagnosing digestive tract tumors. High-quality gastrointestinal endoscopy can significantly reduce the morbidity and mortality of digestive tract tumors. The main factors affecting the quality of gastrointestinal endoscopy are whether the gastrointestinal tract is clean and whether there are any missed detections. Whether there is appropriate mucosal staining and the doctor's ability to detect early digestive tract tumors, how to improve the level of each gastrointestinal endoscopy and timely and objectively evaluate the quality of each gastrointestinal endoscopy are quite difficult with the existing management, training and inspection systems. Therefore, it is urgent to improve the inspection quality of each gastrointestinal endoscopy.

[0003] The key factors affecting the quality of endoscopic examinations are bowel cleanliness, thorough full-bowel inspection, and the physician's ability to detect lesions. Current indicators for evaluating endoscopic quality generally include good bowel preparation, access to the examination site, endoscope insertion and removal time, lesion detection rate, and colorectal perforation rate. However, existing endoscopy procedures lack guidance and specificity for individual endoscopic examinations, resulting in low average endoscopic quality. Summary of the Invention

[0004] The present invention provides an artificial intelligence-based endoscope-assisted inspection method, device, equipment, medium and product.

[0005] In the first aspect, the present invention provides an artificial intelligence-based endoscope-assisted inspection method, comprising: inputting the video stream collected by the endoscope into at least one model, and determining the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position through the at least one model, and prompting the current position of the endoscope in real time; if the current position of the endoscope is not clean, it prompts that cleaning is required; if there is a lesion at the current position of the endoscope, it prompts staining according to the lesion condition, or prompts lesion information; in response to receiving an operation corresponding to the prompt for cleaning, staining, or lesion information, determining the operation status, and generating an evaluation report for this assisted inspection according to preset evaluation rules based on the operation status and the arrival status of the endoscope.

[0006] Furthermore, the method also includes: in response to receiving the operation corresponding to the prompt requiring cleaning, prompting staining or prompting lesion information, determining the operation status, and generating an evaluation report for this auxiliary examination through preset evaluation rules based on the operation status and the arrival status of the endoscope.

[0007] Furthermore, the inputting of the video stream collected by the endoscope into at least one model includes: parsing the video stream collected by the endoscope into at least one frame of image, and inputting the at least one frame of image into a feature extraction module after image preprocessing to obtain convolutional neural network features; obtaining and splicing the endoscopic color features, endoscopic texture features and endoscopic shape features of each frame of image to obtain endoscopic image features of each frame of image; and inputting the convolutional neural network features and the endoscopic image features into the at least one model.

[0008] Furthermore, the endoscope includes a colonoscope; and the video stream collected by the endoscope is input into at least one model, and the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position are determined through the at least one model, and the current position of the endoscope is prompted in real time, including: inputting the video stream collected by the colonoscope when the endoscope is withdrawn into at least one model, and the current position of the colon, whether the colon at the current position is clean, and whether there is a lesion at the current position are determined through the at least one model, and the current position of the colon is prompted in real time.

[0009] Furthermore, the at least one model includes an anatomical position recognition model, a cleaning model and a polyp adenoma recognition model; and the at least one model is used to determine the current position of the colon where the colonoscope is located, whether the colon at the current position is clean, and whether there is a lesion in the colon at the current position, including: determining the current position of the colon where the colonoscope is located through the anatomical position recognition model; determining whether the colon at the current position is clean through the cleaning model; and determining whether there are polyp and / or adenoma lesions in the colon at the current position through the polyp adenoma recognition model.

[0010] Furthermore, the at least one model includes a multi-lesion recognition model; and the method also includes: inputting the video stream collected by the colonoscope when the scope is withdrawn into the multi-lesion recognition model to determine whether the colon has lesion characteristics of non-polyps or non-adenoma.

[0011] Furthermore, the polyp and adenoma recognition model includes a white light staining classification model and a polyp and adenoma classification model; and the determining whether there are lesions of polyps and / or adenomas in the colon at the current position through the polyp and adenoma recognition model includes: determining whether there are lesions in the colon at the current position through the polyp and adenoma classification model, and the lesions include the presence of high-suspected polyps and / or adenomas and the presence of low-suspected polyps and / or adenomas; and if there is a lesion at the current position of the endoscope, staining is prompted according to the lesion situation, or lesion information is prompted, including: determining whether the colon at the current position is in a white light state through the white light staining classification model; if the colon at the current position is in a white light state and there are low-suspected polyps and / or adenomas, staining is prompted; if the colon at the current position is a high-suspected polyp and / or adenoma, the presence of polyps and / or adenomas is prompted.

[0012] Furthermore, the method further comprises: inputting the video stream collected by the colonoscope when the scope is withdrawn into a polyp and adenoma boundary recognition model to determine the boundary of the polyp and / or adenoma of the colon at the current position.

[0013] Furthermore, the method further includes: determining a time to withdraw the colonoscope based on the current position of the colonoscope indicated in real time.

[0014] Furthermore, the method of determining the withdrawal time of the colonoscope based on the current position of the colonoscope indicated by the real-time prompt includes: when the current position of the colonoscope indicated by the real-time prompt is the ileocecal portion for the first time, starting to count the withdrawal time of the ascending colon, and ending to count the withdrawal time of the ascending colon when the current position of the colonoscope indicated by the real-time prompt is the transverse colon, and obtaining the withdrawal time of the ascending colon; when the current position of the colonoscope indicated by the real-time prompt is the transverse colon for the first time, starting to count the withdrawal time of the transverse colon, and ending to count the withdrawal time of the transverse colon when the current position of the colonoscope indicated by the real-time prompt is the splenic flexure for the first time, and obtaining the withdrawal time of the transverse colon; when the current position of the colonoscope indicated by the real-time prompt is the ileocecal portion for the first time, ending to count the withdrawal time of the transverse colon, and obtaining the withdrawal time of the transverse colon; When the current position of the colonoscope is the splenic flexure as prompted, the time for withdrawing the scope in the descending colon starts to be counted; when the first real-time prompt appears that the current position of the colonoscope is the sigmoid colon, the time for withdrawing the scope in the descending colon ends to obtain the time for withdrawing the scope in the descending colon; when the first real-time prompt appears that the current position of the colonoscope is the sigmoid colon, the time for withdrawing the scope in the sigmoid colon starts to be counted; when the first real-time prompt appears that the current position of the colonoscope is the rectum junction, the time for withdrawing the scope in the sigmoid colon ends to obtain the time for withdrawing the scope in the sigmoid colon; the time for withdrawing the scope in the colon is determined according to the time for withdrawing the scope in the ascending colon, the time for withdrawing the scope in the transverse colon, the time for withdrawing the scope in the descending colon and the time for withdrawing the scope in the sigmoid colon.

[0015] Furthermore, before inputting the video stream collected by the colonoscope when the scope is withdrawn into at least one model, the method also includes: inputting the video stream collected by the colonoscope when the scope is advanced into the lesion feature extraction network, obtaining the scope advancement lesion features output by the lesion feature extraction network and saving them; and the method also includes: inputting the video stream collected by the colonoscope when the scope is withdrawn into the lesion feature extraction network, obtaining the scope withdrawal lesion features output by the lesion feature extraction network; performing a similarity comparison between the scope advancement lesion features and the scope withdrawal lesion features, and if the comparison result is greater than a threshold value, it indicates that a lesion is detected in the current position when the scope is advanced.

[0016] Furthermore, the endoscope includes a gastroscope, and the at least one model includes the anatomical position recognition model, the cleaning model, and the early cancer recognition model; and determining the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position through the at least one model, includes: determining the current position of the gastroscope through the anatomical position recognition model; determining whether the current position is clean through the cleaning model; and determining whether there is an early cancer lesion at the current position through the early cancer recognition model.

[0017] Furthermore, the method further includes: inputting the video stream collected by the gastroscope into the multi-lesion recognition model to determine whether there are lesion features other than early-stage cancer at the location of the gastroscope.

[0018] Furthermore, the early cancer recognition model includes the white light staining classification model and the early cancer classification model; and determining whether there is an early cancer lesion at the current location using the early cancer recognition model includes: determining whether there is a lesion at the current location using the early cancer classification model, the lesions including the presence of a high suspicion of early cancer and the presence of a low suspicion of early cancer; and if there is a lesion at the current location of the endoscope, prompting staining or prompting lesion information according to the lesion condition includes: determining whether the current location is in a white light state using the white light staining classification model; if the current location is in a white light state and there is a low suspicion of early cancer, prompting staining; if there is a high suspicion of early cancer at the current location, prompting the presence of early cancer lesion information.

[0019] Furthermore, the method further comprises: inputting the video stream collected by the gastroscope into an early cancer boundary recognition model to determine the boundary of the early cancer at the current position.

[0020] In the second aspect, the present invention also provides an endoscope-assisted inspection device based on artificial intelligence, including: a first processing module, used to input the video stream collected by the endoscope into at least one model, and through the at least one model, determine the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position, and prompt the current position of the endoscope in real time; a second processing module, used to prompt that cleaning is required if the current position of the endoscope is not clean; if there is a lesion at the current position of the endoscope, prompt staining according to the lesion condition, or prompt lesion information.

[0021] In a third aspect, the present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-described artificial intelligence-based endoscopic-assisted inspection methods are implemented.

[0022] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described artificial intelligence-based endoscopic-assisted inspection methods.

[0023] In a fifth aspect, the present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned artificial intelligence-based endoscopic-assisted inspection methods.

[0024] The AI-based endoscopic assisted examination method, device, equipment, medium, and product provided by this invention address issues such as mucosal cleanliness, mucosal staining, missed lesions, and the lesion identification ability of junior physicians, which often hinder endoscopic diagnosis. This improves the speed and quality of physician access and the success rate of the endoscope reaching the target site. This helps guide physicians in standardized endoscopic examination procedures, helps compensate for physicians' lack of experience or inattention, facilitates real-time evaluation of each endoscopic examination, and improves the efficiency and quality of endoscopic examinations. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0026] Figure 1 is a flowchart of some embodiments of the artificial intelligence-based endoscope-assisted inspection method provided by the present invention;

[0027] Figure 2 is a schematic diagram of a flow chart of an embodiment of inputting a video stream collected by an endoscope into at least one model according to the present invention;

[0028] Figure 3 1 is a flow chart of some embodiments of a colonoscopy-assisted examination method based on artificial intelligence provided by the present invention;

[0029] Figure 4 1. It is a flowchart of some embodiments of gastroscopy-assisted examination according to the artificial intelligence-based endoscope-assisted examination method provided by the present invention;

[0030] Figure 5-1 This is a schematic diagram of an application scenario of the network structure of a colonoscope;

[0031] Figure 5-2 This is a schematic diagram of an application scenario of the colonoscopy examination process;

[0032] Figure 5-3 This is a schematic diagram of an application scenario of the functional module of a colonoscope;

[0033] Figure 5-4 This is a schematic diagram of another application scenario of the colonoscopy examination process;

[0034] Figure 5-5 This is a flowchart of an application scenario in which an endoscope prompts whether it is clean;

[0035] Figure 5-6 It is a schematic diagram of a device for endoscopy-assisted inspection based on artificial intelligence;

[0036] Figure 5-7 This is a schematic diagram of an application scenario of the gastroscopy examination process;

[0037] Figure 5-8 This is a schematic diagram of an application scenario of the functional module of the gastroscope;

[0038] Figure 5-9 This is a schematic diagram of another application scenario of the gastroscopy examination process;

[0039] Figure 5-10 This is a schematic diagram of an application scenario of the network structure of gastroscopy;

[0040] Figure 6 Schematic diagram of the structure of some embodiments of the artificial intelligence-based endoscope-assisted inspection device provided by the present invention;

[0041] Figure 7 It is a structural schematic diagram of an electronic device provided according to the present invention.

[0042] Attachment Figure 5-6 Description of labels:

[0043] 1: Display; 2: Mucosal cleaning module; 3: Mucosal machine staining module; 4: Foot pedal; 5: Dual-way foot switch. DETAILED DESCRIPTION

[0044] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0045] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features of the embodiments of the present invention may be combined with each other.

[0046] It should be noted that the concepts of "first" and "second" mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0047] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0048] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0049] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0050] See also Figure 1 , Figure 1 Schematic diagram of some embodiments of the artificial intelligence-based endoscope-assisted inspection method provided by the present invention. Figure 1 As shown, the method includes the following steps:

[0051] Step 101: Input the video stream collected by the endoscope into at least one model. Through the at least one model, determine the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position, and prompt the current position of the endoscope in real time.

[0052] An endoscope is a lighted tube that can be passed through the mouth and into the stomach or other natural orifices. It can reveal lesions that X-rays cannot. For example, an endoscope allows doctors to visualize ulcers or tumors in the stomach, allowing them to determine the best treatment plan.

[0053] As an example, at least one model may be a neural network model with different functions that has been pre-trained according to specific needs. These neural network models with different functions can respectively determine the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position.

[0054] As an example, the current position of the endoscope can be determined in real time according to the input of the video stream, and the current position of the endoscope can be prompted in real time.

[0055] For example, before inputting the video stream captured by the endoscope into at least one model, image preprocessing can be performed, and the preprocessed video stream can be input into at least one model. Alternatively, depending on the requirements of different models, the video stream (or the preprocessed video stream) can be passed through different feature extraction networks for feature extraction before being input into the corresponding at least one model.

[0056] Step 102: If the current position of the endoscope is not clean, a prompt is given to clean it; if there is a lesion at the current position of the endoscope, a prompt is given to stain it according to the lesion condition, or lesion information is given.

[0057] As an example, the video stream can be input into a model that can identify whether the part is clean, and whether the current part is clean can be judged in real time. If it is detected that the current part is not clean, the words "need to be cleaned" can be displayed on the display screen.

[0058] As an example, the video stream can be input into a model that can identify whether there is a lesion, and whether there is a lesion in the current part can be judged in real time. The specific lesion condition and type of lesion are determined according to the part of the endoscopic examination. If a lesion is detected in the current part, the words "lesion exists" can be displayed on the display screen. If the current part is the colon, the words "polyp adenoma exists" can also be displayed on the display screen.

[0059] For example, a video stream can be fed into a model capable of identifying the presence of lesions. This model can then determine in real time whether a lesion is present in the current area. If a lesion is present, it can further determine whether the lesion is highly suspected or low-suspicion. If the lesion is low-suspicion, the screen will display "Needs Staining" to remind the doctor to proceed. If the lesion is highly suspected, the screen will display "Polyp Adenoma Present" or "Lesion Present" to remind the doctor that further observation, biopsy, or resection is necessary.

[0060] In some optional implementations, the method further includes: in response to receiving an operation corresponding to a prompt for cleaning, a prompt for staining, or a prompt for lesion information, determining the operation status, and generating an evaluation report for this auxiliary examination through preset evaluation rules based on the operation status and the arrival of the endoscope.

[0061] In general, some evaluation indicators are post-evaluation, and some are targeted at medical institutions, which cannot be evaluated in real time or in a single instance. Using the above method, an evaluation report for this auxiliary examination can be generated, thus achieving a real-time, single evaluation of this examination.

[0062] As an example, the doctor may perform cleaning, staining, or lesion prompts according to the prompts, or perform corresponding operations. For example, the doctor may press the cleaning button according to the prompts. At this time, the cleaning operation can be observed from the video stream of the endoscope (for example, the operation status can be determined by inputting the video stream into the operation recognition model), indicating that the doctor has performed the operation according to the prompts. In addition, the arrival of the endoscope can be determined based on the current position of the endoscope in real-time. For example, if the ileocecal region is detected during a colonoscopy (from the beginning of entering the colon, the endoscope will pass through the rectal junction, rectum, sigmoid colon, descending colon, splenic flexure, transverse colon, hepatic flexure of the colon, ascending colon, ileocecal valve, appendix orifice, and terminal ileum, if any of the ileocecal valve, appendix orifice, or terminal ileum is detected, it means that the ileocecal region has been detected), indicating that the endoscope has currently reached the terminal colon and the arrival status is very good.

[0063] For example, the preset evaluation rules could be: if the doctor performs the corresponding action for each prompt, it indicates the prompt is correct, and one point is added; if no action is detected, one point is subtracted; for the endoscope's arrival, if the endoscope reaches the preset location, it indicates a good arrival, and one point is added; otherwise, one point is subtracted. The base score is zero. The final score is the total score for this auxiliary examination.

[0064] As an example, the prompt information, prompt time, doctor's operation information and operation time, etc. during the examination process can also be recorded. During the examination process, videos or images can also be saved according to the doctor's needs (or other preset image retention rules).

[0065] As an example, the total score of this auxiliary examination and the information saved during this examination (eg, prompt information, prompt time, doctor's operation information, operation time and saved visual image data) can be combined to generate an evaluation report.

[0066] The AI-based endoscopic assisted examination method disclosed in some embodiments of the present invention addresses issues that hinder endoscopic diagnosis, including mucosal cleanliness, mucosal staining, missed lesions, and the lesion identification ability of junior physicians. It improves the speed and quality of physician access and the success rate of the endoscope reaching the target site. It helps guide physicians in standardized endoscopic examination procedures, helps compensate for physicians' lack of experience or inattention, facilitates real-time evaluation of each endoscopic examination, and improves the efficiency and quality of endoscopic examinations.

[0067] See also Figure 2 , Figure 2 FIG. 1 is a flow chart of an embodiment of inputting a video stream collected by an endoscope into at least one model according to the present invention. Figure 2 As shown, the method includes the following steps:

[0068] In step 201, the video stream collected by the endoscope is parsed into at least one frame of image, and the at least one frame of image is input into a feature extraction module after image preprocessing to obtain convolutional neural network features.

[0069] As an example, at least one frame of image may be subjected to image preprocessing operations such as image cropping, image scaling, image standardization, and image normalization before being input into the feature extraction module.

[0070] As an example, after parsing the video stream captured by the endoscope into at least one frame of images, images can be selected according to a preset rule, and the selected images can be preprocessed before being input into the feature extraction module. For example, starting with the first frame, the first frame, the third frame, the fifth frame, and so on, are selected, i.e., the odd-numbered images are selected. This sequentially improves recognition efficiency.

[0071] For example, for each frame (or raw endoscopic image), the image cropping operation is performed by converting the input image into a grayscale image. The Canny edge detection algorithm is used to determine the camera's captured area in the grayscale image. The rectangular area parallel to this area is the cropping area. The raw endoscopic image is then cropped to the position and size of the cropping area.

[0072] As an example, image cropping can also be implemented using an effective image segmentation module. The effective region segmentation module can be composed of a convolutional neural network layer, an activation function layer, a pooling layer, and a deconvolution layer. After the preprocessed image is input into the effective region segmentation module, a binary image (0 / 1) of the same size as the original endoscopic image is obtained. The image cropping area is the parallel circumscribed rectangular area of ​​the area with a pixel value of 1 in this binary image.

[0073] As an example, the image scaling operation is to scale all the cropped images. Then the scaled images are normalized. The image normalization operation requires calculating the mean map of the entire training set, denoted as mean, and the standard deviation map, denoted as std. The calculation method is:

[0074]

[0075]

[0076] where X i Is a (513,513,3) image matrix, N is the number of images in the training set. For each input image X j All are standardized:

[0077] X j =(X j -mean) / std (Formula 3)

[0078] Image normalization is to convert X j The pixel value is transformed to 0~1.

[0079] Step 202 : Acquire and combine the endoscopic color features, endoscopic texture features, and endoscopic shape features of each frame of image to obtain the endoscopic image features of each frame of image.

[0080] As an example, endoscope color features, endoscope texture features, and endoscope shape features. Endoscope color features can be obtained by calculating the color feature vector (i.e., endoscope color features) obtained by calculating the endoscope color histogram and color distribution; endoscope texture features can be calculated by using the gray level co-occurrence matrix texture feature analysis method to calculate the endoscope texture feature vector (i.e., endoscope texture features); endoscope shape features can be calculated using the Fourier shape descriptor to calculate the shape feature vector (i.e., endoscope shape features) of the image observed by the endoscope.

[0081] As an example, the splicing rules can be determined according to specific needs, but the present invention does not limit this.

[0082] Step 203: Input the convolutional neural network features and the endoscopic image features into at least one model.

[0083] As an example, the network structure of at least one model can refer to Figure 5-1 The general classification network in the convolutional neural network is processed by the convolution layer, activation function and fully connected layer of the general classification network to obtain the processed convolutional neural network features. The processed convolutional neural network features are spliced ​​with the endoscopic image features before classification.

[0084] from Figure 2 It can be seen that Figure 1 Compared with the description of some corresponding embodiments, Figure 2 The AI-based endoscopic-assisted inspection method in some corresponding embodiments includes an image processing step before inputting the video stream captured by the endoscope into at least one model. This improves model recognition efficiency by preprocessing each image frame, extracting convolutional neural network features, concatenating the processed convolutional neural network features with endoscopic image features, and inputting the concatenated features into at least one model.

[0085] See also Figure 3 , Figure 3 1 is a flow chart of some embodiments of the colonoscopy-assisted examination method based on artificial intelligence provided by the present invention. Figure 3 As shown, the method includes the following steps:

[0086] Step 301, input the video stream collected by the colonoscope when the scope is withdrawn into at least one model, and through the at least one model, determine the current position of the colon at the colonoscope, whether the colon at the current position is clean, and whether there is a lesion in the colon at the current position, and prompt the current position of the colonoscope in real time.

[0087] The examination of the colon mainly inputs the video stream when the scope is withdrawn into at least one model to determine the current position of the colon where the colonoscope is located, whether the colon at the current position is clean, and whether there are lesions in the colon at the current position, and prompts the current position of the colonoscope in real time.

[0088] As an example, when the colonoscope video stream is detected and it is found that the ileocecal part is reached from the rectal junction (that is, the ileocecal part is reached when any one of the ileocecal valve, appendix orifice and terminal ileum is detected), the video stream collected by the colonoscope when the scope is withdrawn is started.

[0089] In some optional implementations, at least one model includes an anatomical position recognition model, a cleaning model, and a polyp and adenoma recognition model; and through at least one model, the position of the colon where the colonoscope is currently located, whether the colon at the current position is clean, and whether there are lesions in the colon at the current position are determined respectively, including: determining the position of the colon where the colonoscope is currently located through the anatomical position recognition model; determining whether the colon at the current position is clean through the cleaning model; and determining whether there are polyp and / or adenoma lesions in the colon at the current position through the polyp and adenoma recognition model.

[0090] As an example, you can refer to Figure 5-2 The anatomical position recognition model, the cleaning model, and the polyp and adenoma recognition model can adopt the network structure of a general classification network. Other methods or models can also be used to determine the current position of the colon, whether the colon at the current position is clean, and whether there is a lesion in the colon at the current position.

[0091] Step 302: If the current position of the colonoscope is not clean, a prompt is given to clean it; if there is a lesion at the current position of the colonoscope, a prompt is given to stain it according to the lesion condition, or lesion information is given.

[0092] For example, the prompt "needs cleaning" is a message output by the cleaning model after it recognizes the presence of foam or other substances that affect mucosal observation in the current colonoscopy field of view. The doctor can perform the flushing operation according to the prompt.

[0093] For example, the anatomical location (i.e., the colonoscope's current location) is displayed after the anatomical location recognition model identifies the current location within the body where the colonoscope's field of view is located. This information includes 11 categories: terminal ileum, appendiceal orifice, ileocecal valve, ascending colon, hepatic flexure (also known as the hepatic flexure of the colon), transverse colon, splenic flexure, descending colon, sigmoid colon, rectum, and rectal junction. This information appears during both the insertion and removal of the colonoscope during a colonoscopy.

[0094] In some optional implementations, the polyp and adenoma recognition model includes a white light staining classification model and a polyp and adenoma classification model; and determining whether the colon at the current position has lesions of polyps and / or adenomas through the polyp and adenoma recognition model includes: determining whether the colon at the current position has lesions through the polyp and adenoma classification model, the lesions including the presence of high-suspicion polyps and / or adenomas and the presence of low-suspicion polyps and / or adenomas; and if there is a lesion at the current position of the endoscope, prompting staining according to the lesion condition, or prompting lesion information, including: determining whether the colon at the current position is in a white light state through the white light staining classification model; if the colon at the current position is in a white light state and there are low-suspicion polyps and / or adenomas, prompting staining; if the colon at the current position is a high-suspicion polyp and / or adenoma, prompting the presence of polyps and / or adenomas.

[0095] refer to Figure 5-3 and Figure 5-4 , the polyp and adenoma classification model can be used to directly determine whether there are high-suspicion polyps and / or adenomas and low-suspicion polyps and / or adenomas. Alternatively, the polyp and adenoma classification model can be used to determine whether there are polyps and / or adenomas, and then determine whether the polyps and / or adenomas are high-suspicion polyps and / or adenomas or low-suspicion polyps and / or adenomas. For example, a value (e.g., 5) is obtained through the polyp and adenoma classification model. If this value meets a preset threshold range (e.g., the threshold range is greater than 0), it is determined that polyps and / or adenomas exist. Then, if this value meets a value greater than a preset maximum threshold (e.g., the preset maximum threshold is 3), the polyp and / or adenoma is determined to be a high-suspicion polyp and / or adenoma.

[0096] For example, the high suspicion of polyps and / or adenomas and the low suspicion of polyps and / or adenomas are defined based on the output value range of the polyp and adenoma classification module in the polyp and adenoma recognition model. The range is determined as follows:

[0097] This range has a lower limit and an upper limit, which are calculated based on the output results of the polyp and adenoma classification module in the polyp and adenoma recognition model based on an independent test set of polyps and adenomas.

[0098] The independent polyp / adenoma test set can be a collection of colonoscopy image data collected from a hospital. This dataset consists of negative and positive samples. Negative samples are colonoscopy images without polyps / adenomas; positive samples are colonoscopy images with polyps / adenomas. Furthermore, no images in this image dataset, nor any colonoscopy data from any of the cases associated with them, are used in the training of the polyp / adenoma recognition model.

[0099] The lower limit L of the range is determined by inputting all negative samples of the polyp / adenoma independent test set into the software component, statistically analyzing the output results of each negative sample image in the polyp and adenoma classification module, and calculating the mean of all results, denoted as M, and the variance V, denoted as . The lower limit L of the range is calculated as follows:

[0100] L = M + aV (Formula 4)

[0101] Where a is the lower limit variance multiple.

[0102] The upper limit of the range U is determined by inputting all positive samples in the independent test set of polyps / adenomas into the software component, statistically analyzing the output results of each positive sample image in the polyp and adenoma classification module, and calculating the mean M and variance V of all results. The upper limit of the range U is calculated as follows:

[0103] U=M-bV (Formula 5)

[0104] Where b is the upper limit variance multiple.

[0105] The polyp or adenoma prompt is generated by the polyp and adenoma classification module in the polyp and adenoma recognition model after it identifies a highly suspected polyp or adenoma lesion in the current colonoscopy field of view. The polyp and adenoma boundary recognition module also generates prompts on the display screen outlining the boundaries of the highly suspected polyp or adenoma lesion in the current field of view. Doctors can use this prompt to conduct further observation, biopsy, or resection.

[0106] For example, the staining prompt is generated by the polyp / adenoma classification module in the polyp / adenoma recognition model after it identifies a low-suspicion polyp / adenoma lesion in the current colonoscopy field of view. The white light staining classification model then determines whether the current colon is in a white light state. The doctor can then perform the staining procedure based on this prompt.

[0107] Step 303, in response to receiving the corresponding prompt for cleaning, staining or lesion information, determine the operation status, and generate an evaluation report of this auxiliary examination according to the operation status and the arrival of the colonoscope using preset evaluation rules.

[0108] As an example, the auxiliary examination process can be evaluated through the colonoscopy quality assessment system, which automatically generates a colonoscopy quality assessment report based on the doctor's response to prompt information, arrival at the examination site, and the time it takes to withdraw the scope during the examination.

[0109] refer to Figure 5-5 The system can record the doctor's response to the prompt information, including: whether the doctor performs the flushing operation after the system prompts flushing (i.e., prompts that cleaning is required), whether the doctor performs the staining operation after the system prompts staining, and whether the doctor performs the corresponding operation after the system prompts polyps or adenomas.

[0110] The system's inspection site arrival status includes: ileocecal arrival status and other anatomical locations of the colonoscope. The arrival status is determined by whether the inspection site is mapped or whether the anatomical location recognition model has identified the arrival site.

[0111] The statistics of the withdrawal time in this system include: withdrawal time, withdrawal time of ascending colon, withdrawal time of transverse colon, withdrawal time of descending colon, and withdrawal time of sigmoid rectum.

[0112] The colonoscopy quality assessment report consists of two parts: basic information about the examination case and evaluation indicators. These indicators include: response to flushing prompts, response to staining prompts, response to polyp and adenoma prompts, ileocecal reach, scope withdrawal time (minutes), ascending colon withdrawal time, transverse colon withdrawal time, descending colon withdrawal time, and sigmoid rectum withdrawal time. Each component of the evaluation indicator is labeled with a corresponding score for hospital reference.

[0113] The response after the flushing prompt refers to whether the doctor performs flushing after seeing the flushing prompt information output on the screen. The response after the flushing prompt is displayed can be determined by identifying whether a doctor performs a flushing operation in the current colonoscopy field of view through an operation recognition model.

[0114] The response after the staining prompt refers to whether the doctor performs the staining after seeing the staining prompt information output on the screen. The operation recognition model determines whether the doctor performs the staining operation in the current colonoscopy observation field.

[0115] The response to the polyp and adenoma prompt refers to whether the doctor performs further observation, biopsy, or resection after seeing the polyp and adenoma prompt on the screen. The operation recognition model identifies whether the corresponding operation is performed within the current colonoscopy field of view.

[0116] Ileocecal reach refers to whether the endoscope lens reaches and observes the ileocecal region during a colonoscopy. Ileocecal reach is determined by whether the anatomical position recognition model identifies the ileocecal region.

[0117] The scoring for each part of the evaluation index takes the following form:

[0118] 1. The doctor implements the flushing prompt.

[0119] If the doctor flushes after each flushing prompt and does not keep a picture without being prompted to flush, the flushing part will be scored full marks. Otherwise, one point will be deducted for each flushing less than the given score.

[0120] 2. The doctor's reaction after staining.

[0121] Each time the doctor is prompted to perform staining (chemical staining or electronic staining), he will get full marks. Otherwise, one point will be deducted for each missing staining.

[0122] 3. Post-reaction status of polyps and adenomas.

[0123] If the doctor performs further observation (staining and magnification) or biopsy or resection after the polyp adenoma is framed, full marks will be awarded; otherwise, points will be deducted.

[0124] 4. The ileocecal region is reached.

[0125] Full marks will be awarded if the ileocecal region is reached, otherwise marks will be deducted.

[0126] 5. Retraction time.

[0127] Full marks will be awarded if the scope withdrawal time is no less than 6 minutes and is evenly distributed across the ascending colon, transverse colon, descending colon, and sigmoid colon and rectum. Otherwise, points will be deducted. Specific evaluation rules can be adjusted according to actual circumstances.

[0128] from Figure 3 It can be seen that Figure 1 Compared with the description of some corresponding embodiments, Figure 3The AI-based endoscopic assisted examination method in some corresponding embodiments embodies colonoscopy-assisted examination. It addresses issues such as mucosal cleanliness, mucosal staining, missed lesions, and the lesion identification ability of junior physicians, which often hinder endoscopic diagnosis. It improves the speed and quality of physician access and the success rate of the endoscope reaching the target site. It facilitates guiding physicians in standardized endoscopic examination procedures, helps address physicians' lack of experience or inattention, facilitates real-time evaluation of each endoscopic examination, and improves the efficiency and quality of endoscopic examinations.

[0129] In some optional implementations, at least one model includes a multi-lesion recognition model; and the method further includes: inputting the video stream collected by the colonoscope when the scope is withdrawn into the multi-lesion recognition model to determine whether the colon has lesion characteristics that are not polyps or non-adenomas.

[0130] As an example, the video stream can be parsed into at least one frame of image, and each frame of image is subjected to image preprocessing and feature extraction before being input into the multi-lesion recognition model. The image preprocessing process can be referred to Figure 2 Image preprocessing process. Feature extraction can refer to Figure 5-1 As an example, the image's color, texture, and shape features can be combined with the features output by the general feature extraction network and then fed into a multi-lesion recognition model to determine the presence of non-polyp and non-adenoma lesion features in the colon. The structure of the multi-lesion recognition model can refer to the general classification network structure in the figure.

[0131] In some optional implementations, the method further includes: inputting the video stream collected by the colonoscope when the scope is withdrawn into a polyp and adenoma boundary recognition model to determine the boundary of the polyp and / or adenoma in the colon at the current position.

[0132] As an example, the process of determining the boundaries of the polyps and / or adenomas of the colon at the current location can refer to the above process of determining whether there are non-polyp and non-adenoma lesion features in the colon, wherein the structure of the polyp and adenoma boundary recognition model can refer to Figure 5-1 The network structure of the lesion boundary recognition network.

[0133] In some optional implementations, the method further includes: determining a time to withdraw the colonoscope based on the current position of the colonoscope indicated in real time.

[0134] For example, the current position of the colonoscope can be recorded in real time to determine the time it takes to withdraw the colonoscope. Alternatively, the doctor's operation time can be determined using an operation recognition model, where the time calculated based on the current position of the colonoscope recorded in real time is subtracted from the doctor's operation time to determine the time it takes to withdraw the colonoscope.

[0135] In some optional implementations, the time for withdrawing the colonoscope is determined based on the current position of the colonoscope indicated by real-time prompts, including: when the current position of the colonoscope indicated by real-time prompts is the ileocecal region for the first time, the time for withdrawing the ascending colon begins to be counted; when the current position of the colonoscope indicated by real-time prompts is the transverse colon for the first time, the time for withdrawing the ascending colon ends to obtain the time for withdrawing the ascending colon; when the current position of the colonoscope indicated by real-time prompts is the transverse colon for the first time, the time for withdrawing the transverse colon begins to be counted; when the current position of the colonoscope indicated by real-time prompts is the splenic flexure for the first time, the time for withdrawing the transverse colon ends to obtain the time for withdrawing the transverse colon; when the current position of the colonoscope indicated by real-time prompts is the transverse colon for the first time, the time for withdrawing the transverse colon begins to be counted; When the real-time prompt appears and the current position of the colonoscope is the splenic flexure, the time for withdrawing the scope in the descending colon starts to be counted. When the real-time prompt appears for the first time and the current position of the colonoscope is the sigmoid colon, the time for withdrawing the scope in the descending colon ends to obtain the time for withdrawing the scope in the descending colon. When the real-time prompt appears for the first time and the current position of the colonoscope is the sigmoid colon, the time for withdrawing the scope in the sigmoid colon starts to be counted. When the real-time prompt appears for the first time and the current position of the colonoscope is the rectum, the time for withdrawing the scope in the sigmoid colon ends to obtain the time for withdrawing the scope in the sigmoid colon. The time for withdrawing the scope in the colon is determined based on the time for withdrawing the scope in the ascending colon, the time for withdrawing the scope in the transverse colon, the time for withdrawing the scope in the descending colon and the time for withdrawing the scope in the sigmoid colon.

[0136] As an example, the time period for withdrawing the colonoscope (for example, the time period for withdrawing the ascending colon and the time period for withdrawing the transverse colon are counted as one time period, and the time period for withdrawing the descending colon and the time period for withdrawing the sigmoid colon are counted as another time period) can be determined according to the current position of the colonoscope prompted in real time based on specific needs. As an example, the time period for withdrawing the ascending colon, the time period for withdrawing the transverse colon, the time period for withdrawing the descending colon and the time period for withdrawing the sigmoid colon can be directly added together to determine the time for withdrawing the colonoscope. The time period for withdrawing the ascending colon, the time period for withdrawing the transverse colon, the time period for withdrawing the descending colon and the time period for withdrawing the sigmoid colon can also be added together, and then the doctor's operation time can be subtracted to determine the time for withdrawing the colonoscope.

[0137] As an example, the operations performed by the doctor during the mirror removal process include: flushing, staining, biopsy, and surgery. The operation time of the doctor in the mirror removal time statistics is the total time of the four types of operation processes: flushing, staining, biopsy, and surgery. The method for counting the operation time of the doctor during the mirror removal process starts counting the corresponding operation time when the operation recognition model recognizes the above four types of operations. Figure 5-5 Flushing and dyeing operations can also be detected by foot switch signals. When flushing or dyeing is performed, the foot switch signal is transmitted to the software component via the serial port protocol, and the software component counts the flushing or dyeing time.

[0138] In some optional implementations, before inputting the video stream collected by the colonoscope when the scope is withdrawn into at least one model, the method further includes: inputting the video stream collected by the colonoscope when the scope is advanced into a lesion feature extraction network, obtaining the scope advancement lesion features output by the lesion feature extraction network and saving them; and the method further includes: inputting the video stream collected by the colonoscope when the scope is withdrawn into the lesion feature extraction network, obtaining the scope withdrawal lesion features output by the lesion feature extraction network; performing a similarity comparison between the scope advancement lesion features and the scope withdrawal lesion features, and if the comparison result is greater than a threshold, it indicates that a lesion is detected in the current position when the scope is advanced.

[0139] refer to Figure 5-3 After preprocessing, the image can be directly input into the lesion feature extraction network to extract lesion features (lesion features are extracted for both advancing and retreating the scope). Comparing the similarity between the advancing and retreating lesion features helps prevent missed diagnoses. The present invention does not limit the specific similarity comparison method.

[0140] As an example, during the colonoscopy insertion process, the video stream collected during the insertion can be input into at least one model (for example, an anatomical position recognition model, a cleaning model, a polyp adenoma recognition model, a multiple lesion recognition model, or a polyp adenoma boundary recognition model, etc.) to prompt the doctor in real time whether cleaning is needed, whether staining is needed, or lesion information, etc.

[0141] refer to Figure 5-6 . The colonoscopy auxiliary examination device consists of a host, software components, mucosal cleaning components and mucosal machine staining components. The host includes an AI host, a display screen, a mucosal cleaning module, a mucosal machine staining module and a foot switch; the mucosal cleaning component includes a flushing container, a flushing bag, a flushing pipeline and a connector; the mucosal machine staining component includes a staining pipeline, a spray tube and various special staining solutions for mucosal machine staining. The endoscope host can be connected through a video transmission line to transmit the colonoscopy video image data to the host of this system in the form of a video stream through the video transmission line. The software components in the host analyze and process the video stream data, and output prompt information and endoscope withdrawal time on the display screen in real time.

[0142] The specific workflow of software components is as follows Figure 5-3As shown. The software component is composed of multiple intelligent recognition models, including: a cleanliness model, an anatomical location recognition model, a polyp and adenoma recognition model, other lesion recognition models (i.e., a multi-lesion recognition model), and an operation recognition model. The software component is also composed of two major parts: a general module and a functional module. Among them, the general module includes: a preprocessing module and a feature extraction module; the functional module includes: a cleanliness classification module, an anatomical location classification module (i.e., an anatomical location recognition model), a white light staining classification module, a polyp and adenoma classification module, a polyp and adenoma boundary recognition module, a group of other lesion recognition modules, an operation recognition module, and a lesion feature extraction module. The relationship between the model and each module is as follows: the cleanliness recognition model is composed of a general module and a cleanliness classification module; the anatomical location recognition model is composed of a general module and an anatomical location classification module; the polyp and adenoma recognition model is composed of a general module, a white light staining classification module, a polyp and adenoma classification module, and a polyp and adenoma boundary recognition module; the group of other lesion recognition models is composed of a general module and a group of other lesion recognition modules; and the operation recognition model is composed of a general module and an operation recognition module. The Cleanliness Classification Module identifies the cleanliness of the endoscope image currently being fed into the software component and outputs flushing prompts to the display. These prompts are classified as either "Normal" or "Flush Required." The output of the Cleanliness Classification Module is controlled by the White Light Identification Module and is displayed only when the endoscope image currently being fed into the software component is in a white light state.

[0143] Among them, the cleanliness classification module (i.e., the cleanliness model) is a deep neural network model composed of a CNN layer, a fully connected layer, and a Sigmoid function layer.

[0144] The anatomical location classification module identifies the anatomical location of the endoscopic image currently being input into the software component. This module then outputs anatomical location information to the display and calculates the time it takes to withdraw the endoscope. This anatomical location information appears during both the insertion and withdrawal of the endoscope during a colonoscopy.

[0145] Among them, the anatomical position prompt information is 11 anatomical positions: terminal ileum, appendix orifice, ileocecal valve, ascending colon, hepatic flexure (or hepatic flexure of the colon), transverse colon, splenic flexure, descending colon, sigmoid colon, rectum, and rectal junction.

[0146] Among them, the anatomical position classification module is a deep neural network model composed of CNN layer, fully connected layer, and Softmax function layer.

[0147] Among them, the white light staining classification module is used to identify the endoscopic image data currently input into the software component as a white light or staining image, and control the staining prompt information output of the polyp and adenoma classification module.

[0148] The polyp and adenoma classification module's staining prompt output is controlled by the white light staining classification module outputting a Boolean value (0 / 1) to control whether the staining prompt output is displayed on the display. When the software component inputs an endoscopic image in white light, the white light staining classification module outputs a 1; otherwise, it outputs a 0. If the white light staining classification module outputs a 0, the staining prompt output is not displayed on the display.

[0149] Among them, the white light staining classification module is a deep neural network model composed of CNN layer, fully connected layer, and Sigmoid function layer.

[0150] Among them, the polyp and adenoma classification module is used to identify whether there are polyps or adenoma lesions in the endoscopic image currently input to the software component, and can output staining prompt information and polyp and adenoma prompt information to the display screen.

[0151] The staining prompt information is: normal or staining required. The polyp adenoma prompt information is: normal or polyp adenoma.

[0152] Among them, the polyp adenoma classification module is a deep neural network model composed of CNN layer, fully connected layer, and Sigmoid function layer.

[0153] Among them, the polyp and adenoma boundary recognition module is used to identify whether there are polyps or adenoma lesions in the endoscopic image currently input to the software component, and can output polyp and adenoma boundary delineation information to the display screen.

[0154] Among them, the polyp adenoma boundary recognition module is a deep neural network model composed of CNN layer, deconvolution layer, fully connected layer, and Sigmoid function layer.

[0155] Among them, the other lesion recognition module group is used to identify whether there are other lesions in the endoscopic image currently input to the software component, and output prompt information to the display screen when other lesions are present. The other lesion recognition module group will be composed of multiple lesion recognition modules for various lesion recognition tasks.

[0156] Among them, the operation recognition module is used to identify whether a flushing operation, staining operation, biopsy, or surgery is being performed in the endoscopic image currently input to the software component, and then count the duration of the corresponding operation.

[0157] Among them, the operation recognition module can identify five types of operations: normal, flushing operation, staining operation, biopsy, and surgery.

[0158] Among them, the operation recognition module is a deep neural network model composed of CNN layer, fully connected layer, and Softmax function layer.

[0159] Among them, the lesion feature extraction module is used to extract the lesion features when the colonoscope is inserted and the colonoscope withdrawal features when the colonoscope is withdrawn.

[0160] During colonoscopy, if the polyp and adenoma classification module detects low / high suspicion of polyp and adenoma lesions, the software component will automatically retain an image of the current observation field. The current retained image will be processed through the preprocessing module, feature extraction module, and lesion feature extraction module in the software component to obtain lesion features and save them. When the colonoscopy is withdrawn, the video stream image of the withdrawn endoscope is processed through the preprocessing module, feature extraction module, and lesion feature extraction module to obtain a withdrawal feature. This feature is then compared with the retained image feature for similarity to locate the lesion detected during the insertion of the endoscope.

[0161] The cleanliness recognition model (i.e., the cleanliness model) is used to identify the cleanliness of the endoscope image currently input to the model and can output flushing prompt information to the display screen. After the cleanliness recognition model prompts flushing, and the operation recognition model detects the flushing operation or the software component detects the flushing signal of the foot switch, the cleanliness recognition model still prompts flushing in the front observation field. At this time, the software component will trigger an automatic image retention signal to record the parts of the inspection process that still have substances that affect observation after flushing.

[0162] The anatomical position recognition model is used to identify the anatomical position of the endoscopic image currently input into the model, and can output anatomical position prompt information to the display screen.

[0163] The polyp / adenoma recognition model is used to identify whether polyps / adenomas are present in the endoscopic image currently input to the model. It can output staining prompt information and polyp / adenoma prompt information to the display. After the endoscope has been stained, the staining prompt will no longer appear.

[0164] When inserting the colonoscopy, the polyp adenoma classification module detects low / high suspicion of polyp adenoma lesions and automatically records the internal body location of the current colonoscopy field of view. When withdrawing the colonoscopy, the display screen will prompt the doctor if there is a low / high suspicion of polyp adenoma lesion in that area.

[0165] The other lesion recognition model (ie, multi-lesion recognition model) group is used to identify whether there are other lesions in the endoscopic image currently input to the software component, and output other lesion prompt information to the display screen when other lesions exist.

[0166] The operation recognition model is used to identify whether a flushing operation, staining operation, magnification observation, biopsy, or surgery is being performed in the endoscopic image currently input to the software component, and then calculate the duration of the corresponding operation.

[0167] See also Figure 4 , Figure 4 1 is a flow chart of some embodiments of gastroscopy-assisted examination according to the artificial intelligence-based endoscope-assisted examination method provided by the present invention. Figure 4 As shown, the method includes the following steps:

[0168] Step 401: Input the video stream captured by the endoscope into at least one model. The at least one model includes an anatomical position recognition model, a cleaning model, and an early cancer recognition model. The anatomical position recognition model is used to determine the current position of the gastroscope; the cleaning model is used to determine whether the current position is clean; and the early cancer recognition model is used to determine whether there is an early cancer lesion at the current position. The current position of the endoscope is then indicated in real time.

[0169] The process of inserting and withdrawing the endoscope during gastroscopy is the same. The equipment for gastroscopy can also be referred to Figure 5-6 Specific workflow of the software components for gastroscopy Figure 5-7 .

[0170] As an example, the early cancer recognition model can refer to Figure 5-10 The general classification network in . For the specific implementation of the anatomical position recognition model and the cleaning model, please refer to Figure 3 Related description in .

[0171] In some optional implementations, the early cancer recognition model includes a white light staining classification model and an early cancer classification model; and determining whether there is an early cancer lesion at the current location through the early cancer recognition model includes: determining whether there is a lesion at the current location through the early cancer classification model, the lesions including the presence of a high suspicion of early cancer and the presence of a low suspicion of early cancer; and if there is a lesion at the current location of the endoscope, prompting staining or prompting lesion information according to the lesion condition, including: determining whether the current location is in a white light state through the white light staining classification model; if the current location is in a white light state and there is a low suspicion of early cancer, prompting staining; if the current location is in a high suspicion of early cancer, prompting the presence of early cancer lesion information.

[0172] As an example, refer to Figure 5-8 and Figure 5-9 The flushing prompt is a message generated by the cleanliness classification module (i.e., the cleanliness model) when it identifies the presence of foam or other substances that could affect mucosal observation in the current endoscopic field of view. The physician can perform the flushing operation based on the prompt.

[0173] Staining prompt is a prompt message output by the early cancer classification module in the early cancer recognition model after it identifies a low-suspicion early cancer lesion in the current endoscopic observation field. The doctor can perform the staining operation according to the prompt message. The specific implementation method of the staining prompt can be referred to the reference. Figure 3 See the description of hint staining in .

[0174] The low-suspicion early cancer lesions are defined based on the output value range of the early cancer classification module in the early cancer recognition model. The range determination method can be referenced Figure 3 The method in .

[0175] Early cancer alerts are generated by the early cancer classification module in the early cancer recognition model after it identifies a highly suspected early cancer lesion in the current endoscopic field of view. The early cancer boundary recognition module also generates alerts on the display screen, outlining the boundaries of the high-probability early cancer lesion in the current field of view. Doctors can use these alerts to conduct further observation, perform a biopsy, or perform a resection.

[0176] The anatomical position prompt (i.e., the real-time prompt of the current position of the endoscope) is the prompt information output by the anatomical position recognition model after recognizing the position of the current endoscope observation field in the body. Anatomical location information includes 38 categories: oropharynx, upper esophagus, middle esophagus, lower esophagus, esophagogastric junction, upper gastric body, upper gastric body posterior wall, upper gastric body anterior wall, upper gastric body greater curvature, upper gastric body lesser curvature, middle gastric body, middle gastric body posterior wall, middle gastric body anterior wall, middle gastric body greater curvature, middle gastric body lesser curvature, lower gastric body, lower gastric body posterior wall, lower gastric body anterior wall, lower gastric body greater curvature, lower gastric body lesser curvature, antral junction greater curvature, gastric angle, gastric angle posterior wall, gastric angle anterior wall, gastric antrum, gastric antrum posterior wall, gastric antrum anterior wall, gastric antrum greater curvature, gastric antrum lesser curvature, pylorus, duodenal bulb, duodenal papilla, gastric fundus, gastric fundus posterior wall, gastric fundus anterior wall, gastric fundus greater curvature, gastric fundus lesser curvature, and cardia. This anatomical location information appears during both endoscopic advancement and withdrawal.

[0177] Step 402: If the current position of the endoscope is not clean, a prompt is given to clean it; if there is a lesion at the current position of the endoscope, a prompt is given to stain it according to the lesion condition, or lesion information is given.

[0178] Refer to the specific workflow of the software components for gastroscopy Figure 5-9 The software component outputs four types of prompts during the endoscopic examination process, including: prompt for flushing, prompt for staining, prompt for early cancer, prompt for anatomical location, and prompt for other lesions.

[0179] Step 403, in response to receiving the corresponding prompt for cleaning, staining or lesion information, determine the operation status, and generate an evaluation report for this auxiliary examination according to the operation status and the arrival of the endoscope using preset evaluation rules.

[0180] In some embodiments, the specific implementation of step 403 and the technical effects thereof can be referred to in Figure 1 Step 103 in the corresponding embodiment or reference Figure 3 Step 303 in the corresponding embodiment will not be described again here.

[0181] For other specific implementations of the gastroscopy-assisted examination process and the technical effects it brings, please refer to Figure 3 The description of the embodiments in the present invention will not be repeated here.

[0182] from Figure 4 It can be seen that Figure 1 Compared with the description of some corresponding embodiments, Figure 4 The AI-based endoscopic-assisted examination method in some corresponding embodiments embodies the process of gastroscopy-assisted examination. It addresses issues such as mucosal cleanliness, mucosal staining, missed lesions, and the lesion identification ability of junior physicians, which often hinder endoscopic diagnosis. It improves the speed and quality of physician access and the success rate of the endoscope reaching the target site. It facilitates guiding physicians to standardize endoscopic examination procedures, helps compensate for physicians' lack of experience or inattention, and facilitates real-time evaluation of each endoscopic examination, thereby improving the efficiency and quality of endoscopic examinations.

[0183] In some optional implementations, the method further includes: inputting the video stream captured by the gastroscope into a multi-lesion recognition model to determine whether there are lesion features other than early-stage cancer at the location of the gastroscope.

[0184] As an example, the multi-lesion recognition model can be referred to Figure 5-10 The structure of the general classification network in .

[0185] In some optional implementations, the method further includes: inputting the video stream acquired by the gastroscope into an early cancer boundary recognition model to determine the boundary of the early cancer at the current location.

[0186] As an example, the early cancer boundary recognition model can be referred to Figure 5-10 The structure of the lesion boundary recognition network in .

[0187] In summary, the present invention has the following clinical advantages:

[0188] (1) Assist doctors to improve their gastrointestinal endoscopy diagnosis level. Through the organic combination of four proprietary technologies, namely mucosal cleaning, mucosal machine staining, video annotation and mucosal image recognition, the problems of mucosal cleaning, mucosal staining, missed detection of sites, and the lesion recognition ability of junior doctors, which affect the gastrointestinal endoscopy diagnosis level, can be solved.

[0189] (2)Reference Figure 5-6 The system can identify the cleanliness of the mucosa in the gastrointestinal scope's field of view in real time, and prompts you to flush if it is not clean. The system's cleaning module allows for timely flushing and cleaning. Features of this cleaning module include: 1. A 37°C constant-temperature flushing solution with mucus-removing properties; 2. The flushing line can be connected to the gastrointestinal scope via the auxiliary water inlet, clamp port, or suction port.

[0190] (3) When the lesion is identified as a low-suspicion early cancer (the colonoscopy shows a low-suspicion polyp or adenoma), the doctor is prompted to perform mucosal staining. Through the staining prompt, the doctor can implement high-quality and efficient mucosal machine staining through the machine staining module of this system. The characteristics of this staining module are: 1. There are up to three dyeing liquids for the doctor to select by pressing the button, and each dyeing liquid has a remaining amount prompt, 2. The mucosal cleaning function and the mucosal staining function can be switched on demand, 3. The staining is performed by non-contact machine-powered spraying, 4. The doctor controls the start and stop of the staining by the foot switch (refer to Figure 5-6 ), dyeing is uniform and constant, no nurse cooperation is required.

[0191] (4) When a lesion is identified that is highly suspected of being an early cancer (gastroscopy), the lesion area is framed and the doctor is prompted to perform a biopsy; when a lesion is identified that is highly suspected of being a polyp or adenoma (colonoscopy), the lesion area is framed and the doctor is prompted, who can further observe the lesion, perform a biopsy, or remove the lesion.

[0192] (5) When doctors perform gastroscopy, they can be prompted with 38 anatomical parts: oropharynx, upper esophagus, middle esophagus, lower esophagus, esophagogastric junction, upper gastric body, upper gastric body posterior wall, upper gastric body anterior wall, upper gastric body greater curvature, upper gastric body lesser curvature, middle gastric body, middle gastric body posterior wall, middle gastric body anterior wall, middle gastric body greater curvature, middle gastric body lesser curvature, lower gastric body, lower gastric body posterior wall, lower gastric body anterior wall, lower gastric body greater curvature, lower gastric body lesser curvature, antral junction greater curvature, gastric angle, gastric angle posterior wall, gastric angle anterior wall, gastric antrum, gastric antrum posterior wall, gastric antrum anterior wall, gastric antrum greater curvature, gastric antrum lesser curvature, pylorus, duodenal bulb, duodenal papilla, gastric fundus, gastric fundus posterior wall, gastric fundus anterior wall, gastric fundus greater curvature, gastric fundus lesser curvature, and cardia. This can display the missed parts in real time and prompt doctors to recheck.

[0193] (6) When doctors perform colonoscopy, they can identify and indicate the parts that the colonoscope reaches: rectal junction, rectum, sigmoid colon, descending colon, splenic flexure, transverse colon, hepatic flexure, ascending colon, ileocecal valve, appendix orifice, and terminal ileum. This can improve the speed and quality of doctors' examinations and the success rate of reaching the ileocecal region.

[0194] (7) It can identify and automatically determine the withdrawal time of the colonoscope, including the following segmented withdrawal times: the withdrawal time from the end of the ileocecal end to the hepatic flexure of the colon is the withdrawal time of the ascending colon segment, the withdrawal time from the hepatic flexure of the colon to the splenic flexure is the withdrawal time of the transverse colon segment, the withdrawal time from the splenic flexure to the beginning of the sigmoid colon is the withdrawal time of the descending colon segment, and the withdrawal time from the beginning of the sigmoid colon to the withdrawal of the colonoscope from the body is the withdrawal time of the descending colon and rectum segment. This is beneficial to the quality control of colonoscopy, the careful observation of each section of the colorectum, the prevention of missed detection, and the detection of colorectal adenomas. At the same time, the precise positioning of the examined lesions is more conducive to the treatment of the lesions.

[0195] (8) The doctor's behavior after flushing prompts, staining prompts, early cancer or polyp adenoma prompts, missed detection site prompts and endoscope withdrawal time prompts can be used as indicators for evaluating the quality of gastrointestinal endoscopy.

[0196] (9) It is helpful to guide doctors to perform gastrointestinal endoscopy in a standardized manner, to make up for doctors’ lack of experience or inattention, to facilitate real-time evaluation of each gastrointestinal endoscopy, and ultimately to improve the efficiency and quality of gastrointestinal endoscopy.

[0197] See also Figure 6 , Figure 6 The present invention provides a schematic diagram of the structure of some embodiments of the endoscope-assisted inspection device based on artificial intelligence. As an implementation of the methods shown in the above figures, the present invention also provides some embodiments of the endoscope-assisted inspection device based on artificial intelligence. These device embodiments are similar to Figure 1 The embodiments of the methods shown correspond to each other, and the device can be applied to various electronic devices.

[0198] like Figure 6 As shown, some embodiments of the artificial intelligence-based endoscope-assisted inspection device 600 include a first processing module 601 and a second processing module 602: the first processing module 601 is used to input the video stream collected by the endoscope into at least one model, and through at least one model, determine the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position, and prompt the current position of the endoscope in real time; the second processing module 602 is used to prompt that cleaning is required if the current position of the endoscope is not clean; if there is a lesion at the current position of the endoscope, prompt staining according to the lesion condition, or prompt lesion information.

[0199] In an optional implementation of some embodiments, the device also includes a third processing module for determining the operation status in response to receiving an operation corresponding to a prompt for cleaning, a prompt for staining, or a prompt for lesion information, and generating an evaluation report for this auxiliary examination through preset evaluation rules based on the operation status and the arrival status of the endoscope.

[0200] In an optional implementation of some embodiments, the first processing module 601 is also used to: parse the video stream collected by the endoscope into at least one frame of image, and input the at least one frame of image into a feature extraction module after image preprocessing to obtain convolutional neural network features; obtain and splice the endoscopic color features, endoscopic texture features and endoscopic shape features of each frame of image to obtain the endoscopic image features of each frame of image; and input the convolutional neural network features and the endoscopic image features into at least one model.

[0201] In an optional implementation of some embodiments, the endoscope includes a colonoscope; and the first processing module 601 is further used to: input the video stream collected by the colonoscope when the endoscope is withdrawn into at least one model, and through the at least one model, determine the current position of the colon at which the colonoscope is located, whether the colon at the current position is clean, and whether there is a lesion in the colon at the current position, and prompt the current position of the colonoscope in real time.

[0202] In an optional implementation of some embodiments, at least one model includes an anatomical position recognition model, a cleaning model, and a polyp and adenoma recognition model; and the first processing module 601 is further used to: determine the position of the colon where the colonoscope is currently located through the anatomical position recognition model; determine whether the colon at the current position is clean through the cleaning model; and determine whether the colon at the current position has polyp and / or adenoma lesions through the polyp and adenoma recognition model.

[0203] In an optional implementation of some embodiments, at least one model includes a multi-lesion recognition model; and the device also includes a fourth processing module, which is used to: input the video stream collected by the colonoscope when the scope is withdrawn into the multi-lesion recognition model to determine whether the colon has non-polyp or non-adenoma lesion characteristics.

[0204] In an optional implementation of some embodiments, the polyp and adenoma recognition model includes a white light staining classification model and a polyp and adenoma classification model; and the first processing module 601 is also used to: determine whether there is a lesion in the colon at the current position through the polyp and adenoma classification model, and the lesions include the presence of high-suspected polyps and / or adenomas and the presence of low-suspected polyps and / or adenomas; and the second processing module 602 is also used to: determine whether the colon at the current position is in a white light state through the white light staining classification model; if the colon at the current position is in a white light state and there are low-suspected polyps and / or adenomas, staining is prompted; if the colon at the current position is a high-suspected polyp and / or adenoma, the presence of polyps and / or adenomas is prompted.

[0205] In an optional implementation of some embodiments, the device further includes a fifth processing module for inputting the video stream collected by the colonoscope when the scope is withdrawn into a polyp and adenoma boundary recognition model to determine the boundaries of the polyps and / or adenomas in the colon at the current position.

[0206] In an optional implementation of some embodiments, the device further includes a sixth processing module, configured to determine a time to withdraw the colonoscope based on the current position of the colonoscope indicated in real time.

[0207] In some optional implementations of the embodiments, the sixth processing module is further used for: when the current position of the colonoscope is the ileocecal region when a real-time prompt appears for the first time, the ascending colon scope withdrawal time is started; when the current position of the colonoscope is the transverse colon when a real-time prompt appears for the first time, the ascending colon scope withdrawal time is ended, and the ascending colon scope withdrawal time is obtained; when the current position of the colonoscope is the transverse colon when a real-time prompt appears for the first time, the transverse colon scope withdrawal time is started; when the current position of the colonoscope is the splenic flexure when a real-time prompt appears for the first time, the transverse colon scope withdrawal time is ended, and the transverse colon scope withdrawal time is obtained; when the current position of the colonoscope is the splenic flexure when a real-time prompt appears for the first time, the ascending colon scope withdrawal time is ended, and the ascending colon scope withdrawal time is obtained. When the current position of the colonoscope is the splenic flexure, the time for withdrawing the scope in the descending colon starts to be counted. When the first real-time prompt appears that the current position of the colonoscope is the sigmoid colon, the time for withdrawing the scope in the descending colon ends to obtain the time for withdrawing the scope in the descending colon. When the first real-time prompt appears that the current position of the colonoscope is the sigmoid colon, the time for withdrawing the scope in the sigmoid colon starts to be counted. When the first real-time prompt appears that the current position of the colonoscope is the rectum, the time for withdrawing the scope in the sigmoid colon ends to obtain the time for withdrawing the scope in the sigmoid colon. The time for withdrawing the scope in the colonoscope is determined based on the time for withdrawing the scope in the ascending colon, the time for withdrawing the scope in the transverse colon, the time for withdrawing the scope in the descending colon and the time for withdrawing the scope in the sigmoid colon.

[0208] In an optional implementation of some embodiments, the device also includes a seventh processing module, which is used to: input the video stream collected by the colonoscope when the scope is advanced into the lesion feature extraction network, obtain the lesion feature of the scope when advanced output by the lesion feature extraction network and save it; and the device also includes an eighth processing module, which is used to: input the video stream collected by the colonoscope when the scope is withdrawn into the lesion feature extraction network, obtain the lesion feature of the scope when withdrawn output by the lesion feature extraction network; compare the lesion feature of the scope when advanced with the lesion feature of the scope when withdrawn for similarity, and if the comparison result is greater than a threshold, it is indicated that a lesion is detected in the current position when the scope is advanced.

[0209] In an optional implementation of some embodiments, the endoscope includes a gastroscope, and the at least one model includes an anatomical position recognition model, a cleaning model, and an early cancer recognition model; and the first processing module 601 is further used to: determine the current position of the gastroscope through the anatomical position recognition model; determine whether the current position is clean through the cleaning model; and determine whether there is an early cancer lesion at the current position through the early cancer recognition model.

[0210] In an optional implementation of some embodiments, the device further includes a ninth processing module, which is used to: input the video stream collected by the gastroscope into a multi-lesion recognition model to determine whether there are lesion features other than early cancer at the location of the gastroscope.

[0211] In an optional implementation of some embodiments, the early cancer recognition model includes a white light staining classification model and an early cancer classification model; and the first processing module 601 is further used to: determine whether there is a lesion at the current location through the early cancer classification model, where the lesions include the presence of a high suspicion of early cancer and the presence of a low suspicion of early cancer; and the second processing module 602 is further used to: determine whether the current location is in a white light state through the white light staining classification model; if the current location is in a white light state and there is a low suspicion of early cancer, then a staining prompt is given; if the current location is in a high suspicion of early cancer, then a lesion information indicating the presence of early cancer is given.

[0212] In an optional implementation of some embodiments, the apparatus further includes a tenth processing module, configured to input the video stream acquired by the gastroscope into an early cancer boundary recognition model to determine the boundary of the early cancer at the current location.

[0213] It is understandable that the modules described in the device 600 are similar to those described in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the device 600 and the modules and units contained therein, and will not be repeated here.

[0214] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7 As shown, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute an artificial intelligence-based endoscope-assisted inspection method, which includes: inputting the video stream collected by the endoscope into at least one model, and determining the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position through the at least one model, and prompting the current position of the endoscope in real time; if the current position of the endoscope is not clean, it prompts that cleaning is required; if there is a lesion at the current position of the endoscope, it prompts staining according to the lesion condition, or prompts lesion information.

[0215] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0216] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the artificial intelligence-based endoscope-assisted inspection method provided by the above methods, the method including: inputting the video stream collected by the endoscope into at least one model, and through at least one model, determining the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position, and prompting the current position of the endoscope in real time; if the current position of the endoscope is not clean, it prompts that cleaning is required; if there is a lesion at the current position of the endoscope, it prompts staining according to the lesion condition, or prompts lesion information.

[0217] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-mentioned artificial intelligence-based endoscopic assisted inspection methods provided above, the method comprising: inputting the video stream collected by the endoscope into at least one model, and through at least one model, determining the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position, and prompting the current position of the endoscope in real time; if the current position of the endoscope is not clean, it prompts that cleaning is required; if there is a lesion at the current position of the endoscope, it prompts staining according to the lesion condition, or prompts lesion information.

[0218] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0219] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the above methods of each embodiment or certain parts of the embodiments.

[0220] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An artificial intelligence-based endoscope-assisted inspection method, characterized in that: include: Inputting the video stream collected by the endoscope into at least one model, determining the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position through the at least one model, and providing a real-time prompt of the current position of the endoscope; If the current position of the endoscope is not clean, it will prompt that it needs to be cleaned; If there is a lesion at the current location of the endoscope, staining or lesion information will be prompted according to the lesion condition; The endoscope includes a colonoscope; as well as The video stream collected by the endoscope is input into at least one model, and the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position are determined by the at least one model, and the current position of the endoscope is prompted in real time, including: Inputting the video stream collected by the colonoscope during the withdrawal of the colonoscope into at least one model, determining the current position of the colon at the colonoscope, whether the colon at the current position is clean, and whether there is a lesion in the colon at the current position through the at least one model, and providing a real-time prompt of the current position of the colonoscope; The method further comprises: When the first real-time prompt appears that the current position of the colonoscope is the ileocecal region, the ascending colon scope withdrawal time starts to be counted; when the first real-time prompt appears that the current position of the colonoscope is the transverse colon, the ascending colon scope withdrawal time ends to obtain the ascending colon scope withdrawal time; When the first real-time prompt appears that the current position of the colonoscope is the transverse colon, the transverse colon withdrawal time is started; when the first real-time prompt appears that the current position of the colonoscope is the splenic flexure, the transverse colon withdrawal time is stopped to obtain the transverse colon withdrawal time; When the first real-time prompt appears that the current position of the colonoscope is the splenic flexure, the time for withdrawing the colonoscope in the descending colon begins to be counted; when the first real-time prompt appears that the current position of the colonoscope is the sigmoid colon, the time for withdrawing the colonoscope in the descending colon ends to obtain the time for withdrawing the colonoscope in the descending colon; When the first real-time prompt appears that the current position of the colonoscope is the sigmoid colon, the sigmoid colon withdrawal time starts to be counted. When the first real-time prompt appears that the current position of the colonoscope is the rectum junction, the sigmoid colon withdrawal time ends to obtain the sigmoid colon withdrawal time. The colonoscope withdrawal time is determined by adding the ascending colon scope withdrawal time, the transverse colon scope withdrawal time, the descending colon scope withdrawal time and the sigmoid colon scope withdrawal time, and then subtracting the doctor's operation time.

2. The artificial intelligence-based endoscope-assisted inspection method according to claim 1, characterized in that: The method further comprises: In response to receiving the operation corresponding to the prompt for cleaning, staining or lesion information, the operation status is determined, and an evaluation report for this auxiliary examination is generated according to the operation status and the arrival status of the endoscope through preset evaluation rules.

3. The artificial intelligence-based endoscope-assisted inspection method according to claim 1, characterized in that: The step of inputting the video stream collected by the endoscope into at least one model comprises: Parsing the video stream collected by the endoscope into at least one frame of images, and inputting the at least one frame of images into a feature extraction module after image preprocessing to obtain convolutional neural network features; Acquire and combine the endoscopic color features, endoscopic texture features, and endoscopic shape features of each frame of image to obtain the endoscopic image features of each frame of image; The convolutional neural network features and the endoscopic image features are input into the at least one model.

4. The artificial intelligence-based endoscope-assisted inspection method according to claim 1, characterized in that: The at least one model includes an anatomical location identification model, a cleaning model, and a polyp adenoma identification model; and The determining, by means of the at least one model, respectively the position of the colon where the colonoscope is currently located, whether the colon at the current position is clean, and whether there is a lesion in the colon at the current position includes: Determine the current position of the colon where the colonoscope is located by using an anatomical position recognition model; Determine whether the colon at the current location is clean by using a cleanliness model; The polyp and adenoma recognition model is used to determine whether the colon at the current location has polyp and / or adenoma lesions.

5. The artificial intelligence-based endoscope-assisted inspection method according to claim 1, characterized in that: The at least one model comprises a multi-lesion recognition model; and The method further comprises: The video stream collected by the colonoscope when it is withdrawn is input into the multi-lesion recognition model to determine whether there are non-polyp or non-adenoma lesion characteristics in the colon.

6. The artificial intelligence-based endoscope-assisted inspection method according to claim 4, characterized in that: The polyp adenoma identification model includes a white light staining classification model and a polyp adenoma classification model; as well as The method of determining whether the colon at the current location has polyp and / or adenoma lesions by using the polyp and adenoma recognition model includes: Determining whether there is a lesion in the colon at the current position using a polyp and adenoma classification model, wherein the lesion includes a high suspicion of polyps and / or adenomas and a low suspicion of polyps and / or adenomas; and If there is a lesion at the current location of the endoscope, staining is prompted according to the lesion condition, or lesion information is prompted, including: Determine whether the colon at the current location is in a white light state through a white light staining classification model; If the colon at the current location is white light and there is a low suspicion of polyps and / or adenomas, staining is indicated; If the colon at the current location is highly suspected of polyps and / or adenomas, it indicates the presence of polyps and / or adenomas.

7. The artificial intelligence-based endoscope-assisted inspection method according to claim 1, characterized in that: The method further comprises: The video stream collected by the colonoscope when the scope is withdrawn is input into the polyp and adenoma boundary recognition model to determine the boundary of the polyp and / or adenoma of the colon at the current position.

8. The artificial intelligence-based endoscope-assisted inspection method according to claim 1, characterized in that: Before inputting the video stream collected by the colonoscope when the colonoscope is withdrawn into at least one model, the method further includes: Inputting the video stream collected by the colonoscope during the insertion of the colonoscope into the lesion feature extraction network, obtaining the insertion lesion features output by the lesion feature extraction network and saving them; and The method further comprises: Inputting the video stream collected by the colonoscope during the withdrawal of the colonoscope into the lesion feature extraction network to obtain the withdrawal lesion features output by the lesion feature extraction network; The similarity between the features of the lesion when the scope is advanced and the features of the lesion when the scope is withdrawn is compared. If the comparison result is greater than a threshold, it is indicated that a lesion is detected in the current part when the scope is advanced.

9. The artificial intelligence-based endoscope-assisted inspection method according to claim 1, 2 or 3, characterized in that: The endoscope includes a gastroscope, and the at least one model includes an anatomical position recognition model, a cleaning model, and an early cancer recognition model; as well as Determining the current position of the endoscope, whether the current position is clean, and whether a lesion exists at the current position by using the at least one model includes: Determining the current position of the gastroscope by using the anatomical position recognition model; Determine whether the current location is clean by using a cleaning model; The early cancer recognition model is used to determine whether there is an early cancer lesion at the current location.

10. The artificial intelligence-based endoscope-assisted inspection method according to claim 9, characterized in that: The method further comprises: The video stream collected by the gastroscope is input into the multi-lesion recognition model to determine whether there are lesion features other than early cancer at the location of the gastroscope.

11. The artificial intelligence-based endoscope-assisted inspection method according to claim 9, characterized in that: The early cancer recognition model includes a white light staining classification model and an early cancer classification model; as well as Determining whether there is an early cancer lesion at the current location using the early cancer recognition model includes: Determining whether there is a lesion at the current location using an early cancer classification model, wherein the lesion includes a high suspicion of early cancer and a low suspicion of early cancer; and If there is a lesion at the current location of the endoscope, staining is prompted according to the lesion condition, or lesion information is prompted, including: Determining whether the current position is in a white light state by using the white light staining classification model; If the current position is in white light state and there is a low suspicion of early cancer, staining is prompted; If the current location is highly suspected of early cancer, then the presence of early cancer lesion information will be prompted.

12. The artificial intelligence-based endoscope-assisted inspection method according to claim 9, characterized in that: The method further comprises: The video stream collected by the gastroscope is input into the early cancer boundary recognition model to determine the boundary of the early cancer at the current location.

13. An artificial intelligence-based endoscope-assisted inspection device, characterized in that: include: a first processing module, configured to input the video stream collected by the endoscope into at least one model, determine the current position of the endoscope, whether the current position is clean, and whether there is a lesion at the current position through the at least one model, and provide a real-time indication of the current position of the endoscope; The second processing module is used to prompt that the endoscope needs to be cleaned if the current position of the endoscope is not clean; If there is a lesion at the current location of the endoscope, staining or lesion information will be prompted according to the lesion condition; The endoscope includes a colonoscope; and the first processing module is further configured to: input a video stream captured by the colonoscope when the endoscope is withdrawn into at least one model; determine, by means of the at least one model, the current position of the colon at the colonoscope, whether the colon at the current position is clean, and whether there is a lesion in the colon at the current position; and indicate the current position of the colonoscope in real time; The apparatus further includes a sixth processing module, wherein the sixth processing module is configured to: When the first real-time prompt appears that the current position of the colonoscope is the ileocecal region, the ascending colon scope withdrawal time starts to be counted; when the first real-time prompt appears that the current position of the colonoscope is the transverse colon, the ascending colon scope withdrawal time ends to obtain the ascending colon scope withdrawal time; When the first real-time prompt appears that the current position of the colonoscope is the transverse colon, the transverse colon withdrawal time is started; when the first real-time prompt appears that the current position of the colonoscope is the splenic flexure, the transverse colon withdrawal time is stopped to obtain the transverse colon withdrawal time; When the first real-time prompt appears that the current position of the colonoscope is the splenic flexure, the time for withdrawing the colonoscope in the descending colon begins to be counted; when the first real-time prompt appears that the current position of the colonoscope is the sigmoid colon, the time for withdrawing the colonoscope in the descending colon ends to obtain the time for withdrawing the colonoscope in the descending colon; When the first real-time prompt appears that the current position of the colonoscope is the sigmoid colon, the sigmoid colon withdrawal time starts to be counted. When the first real-time prompt appears that the current position of the colonoscope is the rectum junction, the sigmoid colon withdrawal time ends to obtain the sigmoid colon withdrawal time. The colonoscope withdrawal time is determined by adding the ascending colon scope withdrawal time, the transverse colon scope withdrawal time, the descending colon scope withdrawal time and the sigmoid colon scope withdrawal time, and then subtracting the doctor's operation time.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the artificial intelligence-based endoscope-assisted inspection method as described in any one of claims 1 to 12 are implemented.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the artificial intelligence-based endoscope-assisted inspection method according to any one of claims 1 to 12 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the artificial intelligence-based endoscope-assisted inspection method according to any one of claims 1 to 12 are implemented.

Images