An endoscopy coverage evaluation method and system based on three-dimensional reconstruction

Abstract

The present application relates to the technical field of image processing, in particular to a colonoscopy coverage evaluation method and system based on three-dimensional reconstruction, a storage medium and a computer device, comprising the following steps: S01, screening information frame image: input the examination image into the trained classification neural network, perform information frame screening, if the screening result meets the requirements, enter S02, otherwise, issue a warning; S02, colonoscope depth prediction: input the information frame screened in step S01 into the trained depth prediction neural network, obtain a depth map D; S03, colonoscope pose estimation: input the RGB-D data output by step S02 into the trained pose estimation network, output the pose P of the endoscope; S04, reconstruct the three-dimensional model of the colon; S05, calculate the coverage, reconstruct the three-dimensional model of the intestinal tract according to the endoscopic images in the examination process, and calculate the coverage, when the coverage is less than 90%, issue a warning to avoid missed diagnosis and improve the examination efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a colonoscopy coverage rate evaluation method and system based on three-dimensional reconstruction, a storage medium and a computer device. BACKGROUND

[0002] Colorectal cancer is the fifth most common cancer in terms of morbidity and mortality, and is one of the main cancers that endanger life and health. Colonoscopy is the most commonly used screening method for colorectal cancer. It is usually used to examine each part of the colorectum with a colonoscope. During the examination, the endoscopist controls the electronic endoscope to observe whether there are polyps, tumors and other lesions in each part. If abnormalities are found, biopsy can be performed through electronic endoscopy. However, in actual operation, due to the complex structure of the colon and the uneven level of endoscopist operation, it is easy to cause missed diagnosis and lead to decreased examination effect. The main reasons for missed diagnosis are as follows: first, the polyps appear in the field of view but the doctor fails to check them out, which is mainly caused by the doctor's fatigue or lack of experience. The other aspect is that the polyps do not appear in the field of view, which is mainly caused by the complex digestive tract or the lack of experience of the endoscopist, resulting in blind spots during the examination.

[0003] It is generally believed that a qualified colonoscopy process should have a colon examination coverage rate of 90% to 95% or more. However, there is currently no effective measurement mechanism to accurately reflect this indicator. Domestic and foreign endoscopy societies have pointed out that the polyp detection rate and the withdrawal examination time will be used as the standard to ensure the quality of the examination, because the improvement of these two indicators depends on the non-blind area detection during the examination. However, the above two indicators can only provide subjective feedback on the operation of the doctor during the examination. To fundamentally solve the problem of polyps that may not appear in the field of view, it is necessary to quantify and real-time feedback the examination coverage rate. SUMMARY

[0004] The purpose of the present application is to provide a colonoscopy coverage rate evaluation method and system based on three-dimensional reconstruction, a storage medium and a computer device, which reconstructs a three-dimensional model of the colon according to the endoscopic images during the examination, and calculates the coverage rate. When the coverage rate is less than 90%, a warning is given to avoid missed diagnosis and improve examination efficiency.

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] A colonoscopy coverage rate evaluation method based on three-dimensional reconstruction, characterized in that it comprises the following steps:

[0007] S01, input the examination image into a trained classification neural network to perform information frame screening. If the screening result meets the requirements, proceed to S02, otherwise, give a warning;

[0008] S02, input the information frame screened in step S01 into the trained deep prediction neural network to obtain a depth map D;

[0009] S03, input the RGB-D data output in step S02 into the trained pose estimation network to output a pose P of the endoscope;

[0010] S04, input the depth map D and the pose P obtained in steps S02 and S03 and the image information into a three-dimensional reconstruction algorithm to reconstruct a surface of the examined part;

[0011] S05, calculate the coverage according to the three-dimensional model reconstructed in step S04.

[0012] Further, the judgment algorithm of the information frame in step S01 is: converting the RGB image collected by the endoscope into an HSV color space and removing noise by using a Gaussian filter.

[0013] Further, the training method of the deep prediction neural network in step S02 adopts a combination of supervised training and unsupervised training, and the specific steps are:

[0014] S021, unsupervised training is performed by using real endoscopic pictures to ensure that the network can learn the features of the pictures of the examined part;

[0015] S022, freeze the shallow layer of the network, and perform supervised training by using a phantom data set with real depth to ensure the accuracy of the network depth prediction.

[0016] Further, the specific method of the unsupervised training in step S021 is: inputting the image of the examined part into the depth prediction neural network and the pose estimation network to obtain three corresponding depth maps and a pose transformation matrix in the pose estimation network , wherein and are two images, is an original frame, is a conversion frame obtained by converting the original frame by depth and pose, is a target frame, which is a frame at the next moment of the original frame and is also the target frame expected to be converted, is a frame at the next moment, and the mapping relationship between the conversion frame and the original frame is:

[0017] ,

[0018] wherein for a point in the original frame, for a point in the converted frame, for a conversion pose matrix from the original frame to the target frame, for camera intrinsic parameters; the converted frame and the target frame The error between the converted frame and the target frame is minimized, and the unsupervised loss function is as follows (a):

[0019] ,

[0020] wherein, is the mean absolute error loss, is the structural similarity loss; is the value of a pixel p contained in the converted frame is the value of a pixel p contained in the target frame is the value of a pixel p contained in the target frame is the total number of points contained in the picture, is the mean value of the image is the covariance of the image is the variance of the image is the mean value of the image is the covariance of the image is the variance of the image is the variance of the image is usually taken as , L is the gray dynamic range of the image, if the image is in unit8 format, L takes 255.

[0021] Further, the specific steps of step S04 are as follows:

[0022] S041: input the current frame RGB image, convert it to three-dimensional space according to the depth map D and the pose P, and create it as a face element;

[0023] S042: every time a new frame is input, project it to the three-dimensional space according to the depth map D and the pose P of the frame, calculate the threshold interval , is a threshold parameter, taking a value of 0.5, if the existing adjacent pixel points in the space are within the threshold range, the current face element is called a support face element, indicating that the projection point already exists and does not need to be processed, if it is less than the minimum value of the interval, it is called a conflict face element, indicating that the projection point conflicts with the created face element and needs to be discarded, if it is greater than the maximum value of the interval, it is considered as an expanded face element, indicating that the projection point is outside the created face element and needs to be newly created;

[0024] S043: after all the face elements are created, traverse the face elements to perform triangulation, and finally reconstruct the effective three-dimensional model surface;

[0025] S044: The reconstructed three-dimensional model is fused with the RGB image to render color and present a more realistic three-dimensional model of the colon.

[0026] Further, the calculation method of the coverage in step S05 is as follows: the three-dimensional model is a lumen structure, the central axis is calculated in the lumen structure, the average radius is calculated according to the distance between the central axis and the surface of the colon, a cylinder is fitted according to the average radius, and the complete surface area of the colon is estimated according to the surface area of the cylinder; the ratio of the reconstructed area of the completed three-dimensional model of the colon to the estimated complete surface area is the coverage.

[0027] Further, the acquisition frequency of the inspection image in step S01 is once every 10 seconds, the determination method of the screening result is as follows: whether the information frame number in the time period is greater than 90% is determined, if yes, step S02 is entered, otherwise, a warning is issued to remind the operator to check the part again, and if the calculated coverage in step S05 is less than 90%, a warning is issued.

[0028] In another aspect, the present application provides an endoscopy coverage evaluation system based on three-dimensional reconstruction, comprising: a data screening module for screening information frames of an inspection image;

[0029] An image processing module for processing the information frames to obtain a depth map D and a pose P;

[0030] A three-dimensional model reconstruction module for three-dimensionally reconstructing the depth map D and the pose image to obtain the surface of the inspected part;

[0031] A data calculation module for calculating the reconstructed three-dimensional model to obtain the coverage.

[0032] In another aspect, the present application provides a storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the endoscopy coverage evaluation method based on three-dimensional reconstruction.

[0033] In another aspect, the present application provides a computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable by the processor, wherein the computer program is executed by the processor to implement the steps of the endoscopy coverage evaluation method based on three-dimensional reconstruction.

[0034] Compared with the prior art, the technical scheme of the present application has the following beneficial effects:

[0035] (1) By screening non-information frames, the non-information frames of the collected images are excluded, thereby improving the detection quality and efficiency of the operator on the detected part, and effectively ensuring the three-dimensional reconstruction of the detected part;

[0036] (2). The depth prediction neural network adopts supervised and unsupervised training combination, which is beneficial to obtain a depth map with high accuracy. Compared with the prior art, the supervised training method needs corresponding labels, but the real endoscope cannot obtain depth labels, the phantom data with real depth cannot completely simulate the real detected part, and if the real detected part is used for unsupervised training, the accuracy is difficult to guarantee, and the present application has greater advantages.

[0037] (3). The three-dimensional model of the detected part is reconstructed by screening and processing the obtained image, and the coverage rate is calculated, whether there is a missed detection phenomenon is judged according to the set threshold, so as to improve the quality and efficiency of the inspection. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 It is a flow framework schematic diagram of the embodiment of the present application;

[0039] Figure 2 It is a judgment algorithm schematic diagram of the information frame of the embodiment of the present application;

[0040] Figure 3 It is an unsupervised real colon depth prediction framework schematic diagram of the embodiment of the present application;

[0041] Figure 4 It is a colon phantom data with real depth information of the embodiment of the present application;

[0042] Figure 5 It is a supervised phantom data depth prediction framework schematic diagram of the embodiment of the present application;

[0043] Figure 6 It is a colonoscope pose estimation algorithm schematic diagram based on RGB-D of the embodiment of the present application;

[0044] Figure 7 It is a colon three-dimensional surface reconstruction schematic diagram based on surface element of the embodiment of the present application; DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0046] The drawings are only used for illustrative description and cannot be understood as limitation of the patent;

[0047] The terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the description of the application and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0048] "the" is also intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0049] The following description refers to the accompanying drawings. Wherever possible, the same reference numbers in different drawings refer to the same or similar elements. The following exemplary embodiments are described herein with reference to illustrations that are included to provide a better description of the application. These illustrations are not intended to limit the scope of the application, but merely to clarify and make apparent to those skilled in the art the principles of the application.

[0050] In the description of the application, it should be understood that the terms "first", "second", "third" and the like are used only to distinguish similar objects from each other, and do not necessarily indicate or imply a particular order or sequence, nor are they necessarily used to describe a particular order or sequence, nor are they necessarily used to describe a particular order or sequence. relative importance. For those skilled in the art, the specific meaning of the above terms in this application can be understood according to the specific circumstances.

[0051] In addition, in the description of the application, "a plurality of" means two or more, "and / or", "comprises" and "comprising", unless otherwise specified,

[0052] The association relationship between the associated objects is described, which means that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after it. The application will be further described below in conjunction with the drawings and examples.

[0053] In order to solve the limitations of the prior art, the embodiment provides a technical scheme, which will be further described below in conjunction with the drawings and examples.

[0054] Referring to the drawings Figure 1 A three-dimensional reconstruction-based endoscopy coverage evaluation method, which mainly uses the inspection images of the Olympus electronic endoscope widely used in the prior art as input, and the inspection site is the colon. The method comprises the following steps:

[0055] S01, screen information frame image: input the collected inspection image into the trained classification neural network, screen the information frame, if the screening result meets the requirements, enter S02, otherwise, issue a warning, the classification neural network here can be any classification neural network such as VGG16, ResNet101, etc. The VGG16 model is well suited for classification and positioning tasks, and its name comes from the abbreviation of the Visual Geometry Group of Oxford University. According to the size of the convolution kernel and the number of convolution layers, VGG has 6 configurations, A, A-LRN, B, C, D, and E, among which D and E are the most commonly used VGG16 and VGG19. ResNet was proposed in 2015 because it is "simple and practical" and has been widely used in detection, segmentation, identification, and other fields based on ResNet50 or ResNet101.

[0056] The training method of the classification neural network adopts the training method of other classification networks in the prior art, the inspection image is collected at a frequency of once every 10 seconds, and the Olympus endoscope image is collected. The determination method of the screening result is: whether the number of information frames in this period of time is greater than 90%, if greater than 90%, enter step S02, otherwise, issue a warning to remind the operator to check the part again;

[0057] S02, colonoscope depth prediction: input the information frame screened out in step S01 into the trained depth prediction neural network to obtain a depth map D;

[0058] S03, colonoscope pose estimation: input the RGB-D data output by step S02 into the trained pose estimation network to output the pose P of the colonoscope, wherein the RGB-D data is a pair of RGB image and depth map D, and the RGB image is the inspection image of the colon, and the pose estimation network can be any pose estimation network based on RGB-D data, or a traditional pose estimation algorithm such as PnP / ICP, PnP is a method for solving 3D to 2D point motion estimation, and ICP is a pose estimation problem for multiple 3D to 3D points, and the calculation method adopts the common method in the field in the prior art;

[0059] S04, reconstruct the three-dimensional model of the colon: input the depth map D and the pose P and image information obtained in steps S02 and S03 into a three-dimensional reconstruction algorithm to reconstruct the surface of the inspected part;

[0060] S05, calculate the coverage rate: calculate the coverage rate according to the three-dimensional model reconstructed in step S04, and if the calculated coverage rate is less than 90%, issue a warning.

[0061] Referring to the accompanying drawings Figure 2Because the operation of the digestive tract endoscopy process is complex, and is accompanied by water flushing and air inflation steps, it often leads to images that cannot judge the surface features of the colon, such as images when the digestive tract inner wall is at a very close distance, or when the lens is contaminated. Such images are blurred and cannot judge the surface condition of the colon, and the image has no detection significance, which will reduce the detection quality and efficiency of the clinician on the digestive tract examination, and at the same time hinder the subsequent three-dimensional reconstruction of the digestive tract, so it is necessary to exclude non-information frames. The judgment algorithm of the information frame is: convert the RGB image collected by the electronic endoscope to the HSV color space, and use a Gaussian filter to remove noise, and slightly blurred frames can also be classified into the blurred category to smooth the image features. The blurred frames and the information frames are labeled by professional physicians. The blurred frames are usually images that the lens is attached to the intestinal wall or water spraying and other operations cause the picture to be almost unable to identify the features of the intestinal wall. The slightly blurred frame is defined as an image usually collected due to slight defocus or being adhered to the lens by body fluid. Such images can roughly identify the outline of the intestinal wall, but have no use for feature extraction of the intestine. Such images can be excluded by adding filtering. Then the image is input into the trained classification neural network, which can be VGG16, ResNet50, AlexNet or other arbitrary image recognition network, wherein VGG16 / ResNet50 / AlexNet is a classic convolutional neural network widely used in the industry for image classification. The training set for training the network is divided into two categories: information frames and non-information frames.

[0062] Referring to the accompanying drawings Figures 3-5 In step S02, the colonoscope depth prediction, the picture is an information frame containing useful information. The neural network is usually divided into supervised and unsupervised training. The supervised training method needs corresponding labels, but the real colonoscope cannot obtain depth labels. The phantom data with real depth cannot completely simulate the real colon. If real colon is used for unsupervised training, the accuracy is difficult to guarantee. If the phantom data with real depth is used, it often cannot have good prediction effect on the real colon. The training method of the depth prediction neural network of the present application combines supervised training and unsupervised training. The specific steps are as follows:

[0063] S021, unsupervised training is performed using real endoscopic images to ensure that the network can learn the features of the images of the examined part. The structure of the network is divided into a depth prediction network and a pose estimation network, and the two networks are trained together. The depth prediction network and the pose estimation network can be any network used for depth prediction or pose estimation, such as RNN-DP, PoseCNN, etc. RNN-DP is one of the commonly used depth prediction networks, and PoseCNN is one of the widely used pose estimation networks in the industry. The above networks are widely used in the industry and have open source code.

[0064] S022, freeze the shallow layer of the network, use the phantom dataset with real depth for supervised training, ensure the depth prediction accuracy, use the depth prediction network in S021, freeze the shallow layer and then do supervised training.

[0065] Referring to the accompanying drawings Figure 3 , the specific method of unsupervised training in the above step S021 is: inputting the colon image into the depth prediction neural network and the pose estimation network to obtain three corresponding depth maps , the pose transformation matrix in the pose estimation network Since the colonoscopy is a process of withdrawing the mirror, the next frame of image often contains the previous frame of image, so the point on the current frame of image can be converted from the corresponding point in the previous frame according to the pose transformation and the depth map. Wherein and Two frames of images, is the original frame, is the converted frame obtained by converting the original frame through depth and pose, is the target frame, that is, the frame at the next moment of the original frame, which is also the target frame expected to be converted, is the frame at the next moment, the converted frame and the original frame The mapping relationship between the corresponding points between the converted frame

[0066] ,

[0067] Wherein is the point in the original frame, is the point in the converted frame after conversion of the original frame, is the conversion pose matrix from the original frame to the target frame, is the camera intrinsic parameter; according to the above relationship, the original frame outputs the depth through the depth prediction network, the original frame and the target frame input the pose network to output the pose , and then the converted frame is obtained through conversion according to the above mapping relationship, the error between the converted frame and the target frame is the minimum, which is the unsupervised loss function as follows (a):

[0068] ,

[0069] Wherein, is the mean absolute error loss, is the structural similarity loss; is the converted frame is the value of pixel p contained in the picture, is the target frame is the value of pixel p contained in the picture, is the total number of points in the picture; is the mean of image , is is the covariance of image is the mean of image is the variance of image is usually taken as , L is the gray dynamic range of image, if the image is unit8 format, that is, unsigned 8-bit integer, which is the digital type of the stored image, then L takes 255.

[0070] Referring to the accompanying drawings Figure 4 and 5 , S022 is supervised training. The data set is selected to have a real depth information phantom data, the network structure of supervised training is consistent with the depth prediction network in step S021, the parameters of the shallow layer are frozen and unchanged, and only the network parameters of the deep layer are trained. The loss function is calculated between the predicted depth map and the real depth map, which can use the cross entropy loss function or the and used in the last stage.

[0071] Referring to the accompanying drawings Figure 6 , the pair of real colon pictures and depth maps obtained in the last stage are input into the pose estimation network to obtain the endoscope pose of the corresponding frame, which is represented by a rotation matrix R and a translation matrix T. The pose estimation network in this step can be any pose estimation network based on RGB-D data, or traditional pose calculation algorithms such as triangulation, PnP or ICP method.

[0072] Referring to the accompanying drawings Figure 7 , step S04 reconstructs the colon three-dimensional model: inputting the image information data with depth information D and pose P output in the last stage into the surface reconstruction algorithm based on facet, the three-dimensional model of the colon can be obtained. The main steps of the surface reconstruction algorithm based on facet are as follows:

[0073] S041: input the current frame RGB image, convert it to three-dimensional space according to the depth map D and the pose P, and create it as a facet;

[0074] S042: input a new frame, project it to three-dimensional space according to the depth map D and the pose P of the frame, and calculate the threshold interval , The threshold parameter, that is, the threshold value created by the control facet, is 0.5, if the adjacent pixel points existing in the space are within the threshold range, the current facet is called a support facet, indicating that the projection point already exists and does not need to be processed, if it is less than the minimum value of the interval, it is called a conflict facet, indicating that the projection point has a conflict with the created facet, and needs to be discarded, if it is greater than the maximum value of the interval, it is considered to be an expanded facet, indicating that the projection point is outside the created facet, and a new facet needs to be created;

[0075] S043: After all the facets are created, the facets are traversed for triangulation, and finally the effective three-dimensional model surface is reconstructed;

[0076] S044: The reconstructed three-dimensional model is fused with the RGB image to render color, and a more realistic three-dimensional colon model is presented.

[0077] The calculation method of coverage rate in step S05 is as follows: the three-dimensional model is a lumen structure, the central axis is calculated in the lumen structure, and the average radius is calculated according to the distance of the central axis from the colon surface, a cylinder is fitted according to the average radius, and the complete surface area of the colon is estimated according to the surface area of the cylinder; the ratio of the reconstructed area of the reconstructed colon three-dimensional model to the estimated complete surface area is the coverage rate, and the application in the inspection process: the physician collects colonoscope pictures during colonoscopy and inputs them into the above algorithm, reconstructs the three-dimensional surface of the colon according to the above algorithm, and outputs the current coverage rate on the second screen, and gives a warning prompt when the information frame is too small or the current coverage rate is too low.

[0078] The colonoscopy coverage evaluation algorithm provided by the application has the following advantages: (1) by screening non-information frames, the non-information frames of the collected images are excluded, thereby improving the detection quality and efficiency of the operator on the detected part, and effectively ensuring the three-dimensional reconstruction of the detected part;

[0079] (2) The depth prediction neural network adopts a combination of supervised and unsupervised training, which is beneficial to obtaining a depth map with high accuracy, compared with the prior art: the supervised training method needs corresponding labels, but the real endoscope cannot obtain depth labels, the body model data with real depth cannot completely simulate the real detected part, and if the real detected part is used for unsupervised training, the accuracy is difficult to guarantee, and the application has greater advantages.

[0080] (3) By screening and processing the obtained images, the three-dimensional model of the detected part is reconstructed, and the coverage rate is calculated, and whether there is a missed detection phenomenon is judged according to the set threshold, so as to improve the quality and efficiency of the inspection.

[0081] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Numerous modifications to these embodiments would be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without the use of the innovation falling outside the spirit or scope of the application. Therefore, the present application is not intended to be limited to the described embodiments, but rather is intended to encompass any and all modifications within the spirit and scope of the following claims.

[0082] The above descriptions are only the preferred embodiments of the present application, not intended to limit the application, and any modification, equivalent replacement and improvement made within the spirit and principle of the application shall be included in the protection scope of the application.

Claims

1. A three-dimensional reconstruction-based endoscopy coverage assessment method, characterized in that, Comprising the following steps: S01, screening information frame image: input the inspection image into the trained classification neural network, screen the information frame, and if the screening result meets the requirements, enter S02, otherwise, issue a warning; S02, depth prediction: input the information frame screened in step S01 into the trained depth prediction neural network to obtain a depth map D; S03, pose estimation: input the RGB-D data output by step S02 into the trained pose estimation network to output the pose P of the endoscope; S04, reconstructing a three-dimensional model: input the depth map D and the pose P obtained in steps S02 and S03 and the image information into a three-dimensional reconstruction algorithm to reconstruct the surface of the inspected part; S041: input the current frame RGB image, convert it to a three-dimensional space according to the depth map D and the pose P, and create a facet; S042: every input a new frame, according to the depth map D and pose P of the frame to project it into three-dimensional space, according to its depth z, calculate the threshold interval , is the threshold parameter, the value is 0.5, if the existing adjacent pixel points in the space are within the threshold range, the current face element is called support face element, indicating that the projection point already exists and does not need to be processed, if it is less than the minimum value of the interval, it is called conflict face element, indicating that the projection point conflicts with the created face element and needs to be discarded, if it is greater than the maximum value of the interval, it is considered to be an expanded face element, indicating that the projection point is outside the created face element and a new face element needs to be created; S043: after all the facets are created, perform triangular meshing on the facets to finally reconstruct an effective three-dimensional model surface; S044: the reconstructed three-dimensional model is fused with the RGB image to render the color and present a relatively realistic three-dimensional model of the detected part; S05, calculating the coverage rate: calculate the coverage rate according to the three-dimensional model reconstructed in step S04; the calculation method of the coverage rate is: the three-dimensional model is a lumen structure, the central axis is calculated in the lumen structure, and the average radius is calculated according to the distance of the central axis from the colon surface; a cylinder is fitted according to the average radius, and the complete surface area of the colon is estimated according to the surface area of the cylinder; the ratio of the reconstructed area of the colon three-dimensional model to the estimated complete surface area is the coverage rate.

2. The endoscopy coverage evaluation method based on three-dimensional reconstruction according to claim 1, characterized in that, The judgment algorithm of the information frame in step S01 is: converting the RGB image collected by the endoscope to HSV color space and removing noise using a Gaussian filter.

3. The endoscopy coverage evaluation method based on three-dimensional reconstruction according to claim 1 or 2, characterized in that, The training method of the depth prediction neural network in step S02 combines supervised training and unsupervised training, and the specific steps are: S021, unsupervised training using real endoscopic images to ensure that the network can learn the features of the inspected part images; S022, freeze the shallow layer of the network, and perform supervised training using a phantom dataset with real depth to ensure the depth prediction accuracy of the network.

4. The endoscopy coverage evaluation method based on three-dimensional reconstruction according to claim 3, characterized in that, The specific method of unsupervised training in step S021 is: inputting the image of the detected part , inputting the three pictures into the depth prediction neural network and the pose estimation network to obtain three corresponding depth maps , obtaining the pose transformation matrix in the pose estimation network , wherein the known and two frames of images are the original frame, the conversion frame obtained by converting the original frame through depth and pose, the target frame, that is, the frame at the next moment of the original frame, which is also the target frame expected to be converted, is the frame at the next moment, the conversion frame and the mapping relationship of the corresponding points between the original frame is: ， wherein is a point in the original frame, is a point in the converted frame after the conversion of the original frame, is a conversion pose matrix from the original frame to the target frame, is an intrinsic parameter of the camera; the converted frame is the error between the converted frame and the target frame The error between the converted frame and the target frame is minimized, and the unsupervised loss function is as follows (a): ， wherein, is the mean absolute error loss, is the structural similarity loss; is the converted frame the value of a pixel p comprised in the picture, is the target frame the value of a pixel p comprised in the picture, P is the number of points comprised in the picture, is the total number of points in the picture; is the mean value of the image is the mean value of the image is the covariance of the image is the covariance of the image is the variance of the image is the variance of the image is typically taken as , , L is the grey level dynamic range of the image.

5. The endoscopy coverage evaluation method based on three-dimensional reconstruction according to claim 1, characterized in that, The acquisition frequency of the inspection image in step S01 is once every 10 seconds, and the judgment method of the screening result is: whether the number of information frames in this period of time is greater than 90%; if it is greater than 90%, enter step S02, otherwise, issue a warning to remind the operator to recheck the part, and if the calculated coverage rate in step S05 is less than 90%, issue a warning.

6. A three-dimensional reconstruction-based endoscopy coverage assessment system, characterized by, Comprising: a data screening module for screening information frames of inspection images; an image processing module for processing information frames to obtain a depth map D and a pose P; a three-dimensional model reconstruction module for reconstructing the surface of the examined part from the depth map D and the pose P and the image; inputting the current frame RGB image into the three-dimensional space according to the depth map D and the pose P and creating a surface element; inputting a new frame, projecting it into the three-dimensional space according to the depth map D and the pose P of the frame, calculating a threshold interval according to the depth z of the frame , as the threshold parameter, with a value of 0.5; if the existing adjacent pixel points in the space are within the threshold range, the current surface element is called a support surface element, indicating that the projection point already exists and does not need to be processed; if it is less than the minimum value of the interval, it is called a conflict surface element, indicating that the projection point conflicts with the created surface element and needs to be discarded; if it is greater than the maximum value of the interval, it is considered to be an expansion surface element, indicating that the projection point is outside the created surface element and a new surface element needs to be created; after all the surface elements are created, the surface elements are triangulated to finally reconstruct the effective three-dimensional model surface; the reconstructed three-dimensional model is fused with the RGB image to render the color and present a relatively real three-dimensional model of the detected part. The data calculation module is used for calculating the coverage of the reconstructed three-dimensional model, and the calculation method of the coverage is as follows: the three-dimensional model is a lumen structure, the central axis is calculated in the lumen structure, the average radius is calculated according to the distance between the central axis and the surface of the colon, a cylinder is fitted according to the average radius, and the complete surface area of the colon is estimated according to the surface area of the cylinder; the ratio of the reconstructed area of the completed three-dimensional model of the colon to the estimated complete surface area is the coverage.

7. A storage medium having stored thereon a computer program, characterized in that: The computer program, when executed by a processor, implements the steps of the three-dimensional reconstruction-based endoscopy coverage evaluation method according to any one of claims 1 to 5.

8. A computer device, comprising: The computer program, when executed by a processor, implements the steps of the three-dimensional reconstruction-based endoscopy coverage evaluation method according to any one of claims 1 to 5.

Images