Enteroscope polyp size evaluation method and system based on multi-frame matching

Abstract

The invention relates to the technical field of image processing, in particular to an enteroscope polyp size evaluation method and system based on multi-frame matching, and the method comprises the steps: obtaining a to-be-evaluated video stream of an enteroscope, and judging whether a polyp exists in a to-be-evaluated video frame through an intestinal polyp detection model; when the to-be-evaluated video frame exists, acquiring the position information of the polyp and calculating the image similarity of two adjacent frames of N subsequent video frames of the to-be-evaluated video frame; when the image similarity meets a preset condition, generating a corresponding depth map by using a depth estimation model; calculating the polyp size of the subsequent video frame according to the position information of the polyp and the polyp depth information in the depth map; and performing weighted calculation on the polyp size of each subsequent video frame, and outputting the standard polyp size of the to-be-evaluated video stream of the enteroscope. The method can better predict the depth information of the image in the real enteroscope, eliminate the influence of polyp blurring and smooth polyp size output display, more accurately measure the size of the polyp, and help an endoscopy doctor to make a decision better.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a colonoscopic polyp size assessment method and system based on multi-frame matching. Background Art

[0002] As one of the most common malignant tumors in the world, early diagnosis of colorectal cancer is crucial. Intestinal polyps are recognized as an important precancerous lesion. The size of intestinal polyps is not only related to the choice of treatment, but more importantly, it has a profound impact on the formulation of subsequent intervention measures and disease management strategies.

[0003] Currently, most doctors estimate the size of polyps based on their own experience. In patent number CN213588351U, the size of polyps is measured by a biopsy forceps with a spring ruler. Patent number CN114680809A integrates a laser emission light in the colonoscope and uses the laser spot as a reference to measure the size of polyps. Patent number CN111091562B uses the forward jet of water or the biopsy forceps as a reference to calculate the size of the lesion. Patent number CN115578437B provides a method for acquiring intestinal lesion depth data, which estimates the depth map of the real colonoscopy image by simulating and modeling the intestinal model, thereby facilitating the subsequent estimation of the lesion size.

[0004] At present, the method of estimating the size of intestinal polyps by visual inspection by doctors depends on the experience and judgment of endoscopists, which is greatly affected by human subjectivity and easily leads to deviations. Existing methods measure lesions by introducing auxiliary tools or reference objects such as biopsy forceps and laser probes as reference objects, but the implementation cost is high and the operation is inconvenient. The accuracy of the method of estimating the depth of intestinal lesions depends on the simulation of the real colonoscopy scene by the intestinal model. Since the intestinal model is quite different from the real scene environment and no real colonoscopy data is added for correction, it is difficult to accurately estimate the size of intestinal polyps in real colonoscopy examinations. Summary of the invention

[0005] To achieve the above object, the present invention provides the following technical solutions: According to a first aspect of the present invention, the present invention claims a method for evaluating the size of colonoscopic polyps based on multi-frame matching, characterized in that it comprises: Obtain a colonoscopy video stream to be evaluated, and determine whether there is a polyp in the video frame to be evaluated in the colonoscopy video stream to be evaluated by using a colon polyp detection model; When a polyp exists in the video frame to be evaluated, obtaining location information of the polyp and using a perceptual hash algorithm to calculate image similarity between two adjacent frames of N subsequent video frames of the video frame to be evaluated; When the image similarity of two adjacent frames of N subsequent video frames meets a preset condition, a corresponding depth map is generated using a depth estimation model; Calculating the size of the polyp in subsequent video frames according to the position information of the polyp and the depth information of the polyp in the depth map; The polyp size of each subsequent video frame is weightedly calculated, and the standard polyp size of the colonoscopy video stream to be evaluated is output.

[0006] Furthermore, the method of obtaining the colonoscopy video stream to be evaluated and determining whether a polyp exists in the video frame to be evaluated in the colonoscopy video stream to be evaluated by using a colon polyp detection model further includes: The intestinal polyp detection model uses lower digestive tract polyp data as a training set; For the input colonoscopy video stream to be evaluated, the intestinal polyp detection model outputs the upper left and lower right coordinates and confidence of the polyp position in the video frame to be evaluated in the colonoscopy video stream to be evaluated; When the confidence level is greater than a preset threshold, it is considered that polyps exist in the video frame to be evaluated.

[0007] Furthermore, when a polyp exists in the video frame to be evaluated, obtaining the location information of the polyp and using a perceptual hash algorithm to calculate the image similarity of two adjacent frames of N subsequent video frames of the video frame to be evaluated also includes: Performing image preprocessing on N subsequent video frames of the video frame to be evaluated, and calculating and obtaining a DCT coefficient matrix of each subsequent video frame; Extracting submatrices of each DCT coefficient matrix as image low-frequency information of each subsequent video frame, and calculating the mean of each submatrix; Based on the comparison result of the image low-frequency information of each subsequent video frame and the mean value of each sub-matrix, the matching hash value of each subsequent video frame is calculated; The Hamming distance between two adjacent frames of each subsequent video frame is calculated according to the matching hash value, and the image similarity between two adjacent frames of each subsequent video frame is determined based on the Hamming distance.

[0008] Furthermore, when the image similarity of two adjacent frames of the N subsequent video frames meets a preset condition, using the depth estimation model to generate a corresponding depth map also includes: When the Hamming distances between two adjacent frames of the N subsequent video frames are all less than a preset threshold, determining that the image similarity between two adjacent frames of the N subsequent video frames meets a preset condition; The simulated rendered images and real colonoscopy images are used to train a student depth prediction model, and a corresponding depth map is generated based on the student depth prediction model.

[0009] Furthermore, the method of using the simulated rendering image and the real colonoscopy image to train the student depth prediction model also includes: The simulated rendered image is sequentially input into the encoder 1 and the decoder, and the loss is calculated in the rendered intestinal image training according to the real depth label; The real colonoscopy image is subjected to data enhancement processing and then input into encoder 1, and the loss is calculated in the real intestinal image training according to the pseudo depth label; The real colonoscopy image is input into encoder 2, and semantic segmentation features are introduced as auxiliary supervision information, and the output features of encoder 2 are used as auxiliary semantic supervision; The pseudo depth label is predicted by a teacher model on a real colonoscopy image, wherein the teacher model is trained only with rendered images having real depth labels, and the model structure of the teacher model is the same as that of the student model.

[0010] Further, calculating the size of the polyp in the subsequent video frame according to the position information of the polyp and the depth information of the polyp in the depth map also includes: Obtaining the center coordinate depth information of the polyp according to the student depth prediction model; Calculating the initial polyp size of the subsequent video frame according to the center coordinate depth information and the position information; The initial polyp size is corrected to obtain a final polyp size of the subsequent video frame.

[0011] Furthermore, the weighted calculation of the polyp size of each subsequent video frame and output of the standard polyp size of the colonoscopy video stream to be evaluated also includes: A circular queue is constructed in the depth of field of the lens according to the center depth of the polyp in each subsequent video frame; Based on the IoU values ​​of the polyp frames of the adjacent frames of each subsequent video frame and the depth of the polyp center within the depth of field range of the lens, it is determined whether the previous and next frames are the same polyp and the circular queue is directly entered or cleared; When the circular queue is full, a weighted moving average is used to output the standard polyp size of the colonoscopy video stream to be evaluated.

[0012] Further, when the polyp frame IoU values ​​of adjacent frames of each subsequent video frame and the depth of the polyp center are within the depth of field range of the lens, determining whether the previous and next frames are the same polyp and directly entering or clearing the circular queue includes: If the IoU values ​​of the polyp frames of adjacent frames of each subsequent video frame are greater than or equal to the first threshold and the depth of the center of the polyp is within the depth of field of the lens, the previous and next frames are considered to be the same polyp, and the current polyp size is added to the circular queue; If the IoU value of the polyp frame of the adjacent frames of each subsequent video frame is less than the first threshold and the depth of the polyp center is within the depth of field of the lens, it is determined that the previous and next frames are not the same polyp, then the circular queue is cleared, and the polyp size of the video frame to be evaluated is re-added to the cleared circular queue.

[0013] According to a second aspect of the present invention, the present invention claims a colonoscopic polyp size assessment system based on multi-frame matching, comprising: one or more processors; A memory having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the colonoscopic polyp size assessment method based on multi-frame matching.

[0014] The present invention relates to the field of image processing technology, and in particular to a colonoscopy polyp size assessment method and system based on multi-frame matching, obtaining a colonoscopy video stream to be assessed, judging whether there are polyps in the video frame to be assessed by a colon polyp detection model; obtaining the location information of the polyp when it exists and calculating the image similarity of two adjacent frames of N subsequent video frames of the video frame to be assessed; when the image similarity meets the preset conditions, using the depth estimation model to generate the corresponding depth map; calculating the polyp size of the subsequent video frame according to the location information of the polyp and the depth information of the polyp in the depth map; performing weighted calculation on the polyp size of each subsequent video frame, and outputting the standard polyp size of the colonoscopy video stream to be assessed. The present invention can better predict the depth information of the image in the real colonoscopy, eliminate the influence of polyp blurring, and smooth the display of the polyp size output, more accurately measure the size of the polyp, and help endoscopists make better decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A workflow diagram of a colonoscopic polyp size assessment method based on multi-frame matching as claimed in an embodiment of the present invention; Figure 2 A Unity simulation rendering of a colonoscopic polyp size assessment method based on multi-frame matching as claimed in an embodiment of the present invention; Figure 3 A schematic diagram of a model training process of a colonoscopic polyp size assessment method based on multi-frame matching as claimed in an embodiment of the present invention; Figure 4 A schematic diagram of the specific structures of an encoder 1 and a decoder of a colonoscopy polyp size assessment method based on multi-frame matching as claimed in an embodiment of the present invention; Figure 5 A schematic diagram of a fusion block structure and a depth estimation prediction head structure of a colonoscopy polyp size assessment method based on multi-frame matching as claimed in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The terms "first", "second" and "third" in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined as "first", "second" and "third" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. All directional indications (such as up, down, left, right, front, back...) in the embodiments of the present invention are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0018] Reference to an "embodiment" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0019] According to the first aspect of the present invention, the present invention claims a method for evaluating the size of colonoscopic polyps based on multi-frame matching, referring to Figure 1 ,include: Obtain a colonoscopy video stream to be evaluated, and determine whether there is a polyp in the video frame to be evaluated in the colonoscopy video stream to be evaluated by using a colon polyp detection model; When a polyp exists in the video frame to be evaluated, obtaining location information of the polyp and using a perceptual hash algorithm to calculate image similarity between two adjacent frames of N subsequent video frames of the video frame to be evaluated; When the image similarity of two adjacent frames of N subsequent video frames meets a preset condition, a corresponding depth map is generated using a depth estimation model; Calculating the size of the polyp in subsequent video frames according to the position information of the polyp and the depth information of the polyp in the depth map; The polyp size of each subsequent video frame is weightedly calculated, and the standard polyp size of the colonoscopy video stream to be evaluated is output.

[0020] Furthermore, the method of obtaining the colonoscopy video stream to be evaluated and determining whether a polyp exists in the video frame to be evaluated in the colonoscopy video stream to be evaluated by using a colon polyp detection model further includes: The intestinal polyp detection model uses lower digestive tract polyp data as a training set; For the input colonoscopy video stream to be evaluated, the intestinal polyp detection model outputs the upper left and lower right coordinates and confidence of the polyp position in the video frame to be evaluated in the colonoscopy video stream to be evaluated; When the confidence level is greater than a preset threshold, it is considered that polyps exist in the video frame to be evaluated.

[0021] In this embodiment, the intestinal polyp detection model can use a CNN-based single-stage detection network such as YOLOv8, or a transformer-based single-stage detection network such as DETR.

[0022] The detection model uses the lower gastrointestinal polyp data as the training set. For the input video stream image, the detection model outputs the upper left and lower right coordinates of the polyp location in the corresponding image. When the confidence is greater than the set threshold, it is considered that there is a polyp in the frame.

[0023] In a specific actual detection example, the detection model outputs the upper left and lower right coordinates as [269, 215], [509, 416] respectively, with a confidence level of 0.88. The currently set confidence threshold is 0.3, so it is considered that a polyp exists in the current image frame.

[0024] Furthermore, when a polyp exists in the video frame to be evaluated, obtaining the location information of the polyp and using a perceptual hash algorithm to calculate the image similarity of two adjacent frames of N subsequent video frames of the video frame to be evaluated also includes: Performing image preprocessing on N subsequent video frames of the video frame to be evaluated, and calculating and obtaining a DCT coefficient matrix of each subsequent video frame; Extracting submatrices of each DCT coefficient matrix as image low-frequency information of each subsequent video frame, and calculating the mean of each submatrix; Based on the comparison result of the image low-frequency information of each subsequent video frame and the mean value of each sub-matrix, the matching hash value of each subsequent video frame is calculated; The Hamming distance between two adjacent frames of each subsequent video frame is calculated according to the matching hash value, and the image similarity between two adjacent frames of each subsequent video frame is determined based on the Hamming distance.

[0025] In this embodiment, image preprocessing is performed on N subsequent video frames of the video frame to be evaluated, and a DCT coefficient matrix of each subsequent video frame is calculated, which also includes: Image reduction: Reduce the image to a fixed size (e.g. 32x32 pixels) to remove detail information.

[0026] Grayscale processing: convert the reduced image into a grayscale image; Calculate the discrete cosine transform matrix (DCT): Use discrete cosine transform to convert the image from the spatial domain to the frequency domain and get the DCT coefficient matrix.

[0027] Specifically, extracting submatrices of each DCT coefficient matrix as image low-frequency information of each subsequent video frame, and calculating the mean of each submatrix, further comprising: Keep low-frequency coefficients: Keep the upper left 8x8 sub-matrix of the DCT coefficient matrix. These coefficients represent the low-frequency information of the image. Compute Mean: Compute the mean of the 8x8 submatrix.

[0028] Specifically, based on the comparison result of the image low-frequency information of each subsequent video frame and the mean value of each sub-matrix, the matching hash value of each subsequent video frame is calculated, and the following is also included: The image low-frequency information of each subsequent video frame is compared with the mean of each sub-matrix, and values ​​greater than the mean are recorded as 1, and values ​​less than or equal to the mean are recorded as 0, to generate a 64-bit binary hash value.

[0029] Specifically, the Hamming distance between two adjacent frames of each subsequent video frame is calculated according to the matching hash value, and the image similarity between two adjacent frames of each subsequent video frame is determined based on the Hamming distance, and further includes: Calculate the Hamming distance between two images: Compare the number of different bits in the 64-bit hash values ​​of the two images, which is the Hamming distance. The smaller the Hamming distance, the more similar the two images are.

[0030] In a specific embodiment, specifically, N is 5, and the hash values ​​of the subsequent five frames of images are converted to hexadecimal and expressed as 0x9c62512251686b6d, 0x9c625122502a6a6d, 0x9c625122512a6a6d, 0x9c625122510a626c, and 0x9c6251225108626c, respectively. The Hamming distances between two adjacent frames are 4, 1, 3, and 1, respectively.

[0031] Furthermore, when the image similarity of two adjacent frames of the N subsequent video frames meets a preset condition, using the depth estimation model to generate a corresponding depth map also includes: When the Hamming distances between two adjacent frames of the N subsequent video frames are all less than a preset threshold, determining that the image similarity between two adjacent frames of the N subsequent video frames meets a preset condition; The simulated rendered images and real colonoscopy images are used to train a student depth prediction model, and a corresponding depth map is generated based on the student depth prediction model.

[0032] Among them, in this embodiment, taking the hash values ​​of 5 images as an example, the threshold is set to 5, and the Hamming distances of 5 consecutive frames of images (4, 1, 3, 1) are all less than 5, then it is considered that the current 5 frames of images have a high similarity, indicating that the current picture is in a relatively static or slightly shaking state, so the picture is relatively clear, reducing the interference of motion blur on the depth estimation model.

[0033] The depth estimation model not only uses the depth map obtained by the simulated intestinal model, but also adds the scene of real colonoscopy as an auxiliary to improve the accuracy of depth estimation.

[0034] The dataset used in this scheme uses CT colonography to generate a 3D model and uses Unity to simulate the environment and render the image to get the effect closest to the real colonoscopy. Figure 2 Shown is Unity's simulated rendered image and the corresponding real depth of the rendered image. Furthermore, the method of using the simulated rendering image and the real colonoscopy image to train the student depth prediction model also includes: The simulated rendered image is sequentially input into the encoder 1 and the decoder, and the loss is calculated in the rendered intestinal image training according to the real depth label; The real colonoscopy image is subjected to data enhancement processing and then input into encoder 1, and the loss is calculated in the real intestinal image training according to the pseudo depth label; The real colonoscopy image is input into encoder 2, and semantic segmentation features are introduced as auxiliary supervision information, and the output features of encoder 2 are used as auxiliary semantic supervision; The pseudo depth label is predicted by a teacher model on a real colonoscopy image, wherein the teacher model is trained only with rendered images having real depth labels, and the model structure of the teacher model is the same as that of the student model.

[0035] In this embodiment, reference is made to Figure 3, the solid line part is the process of calculating the loss in the training of the rendered intestinal images with real depth labels, and the dotted line part is the process of calculating the loss in the training of the real intestinal images without real depth labels. The teacher model is trained only with the rendered intestinal images with real depth labels, and predicts the real intestinal images to generate pseudo depth labels.

[0036] Data enhancement operations are added to the real intestinal images, including CutMix, color perturbation, Gaussian blur, exposure enhancement, etc. Through the above operations, the real label loss can be obtained and pseudo-label loss .

[0037] In order to allow the student model to capture more semantic information, semantic segmentation features are introduced as auxiliary supervision information. The output features of encoder 2 are used as auxiliary semantic supervision. Encoder 2 here uses the pre-trained frozen DINOv2 encoder.

[0038] DINOv2 uses self-supervised learning technology to learn common semantic information from massive unlabeled images, so that it can be applied to different tasks. Its semantic feature alignment loss is defined as follows:

[0039] Where MLP is a trainable multi-layer perceptron operation, cos is a cosine similarity operation, are the features output by encoder 1 and encoder 2 respectively, α is the threshold constant, H and W are height and width.

[0040] Therefore, the total loss of the student depth prediction model is ,in , , The hyper parameters corresponding to the three losses are as follows. The encoder 1 of the student depth prediction model also adopts the ViT-B structure, and the decoder adopts the DPT decoder structure. Finally, the student depth prediction model is used to predict the depth information of the real colonoscopy image.

[0041] Reference Figure 4 , which shows the structure of encoder 1 and decoder. Non-overlapping slice embedding is used in encoder 1, which is obtained by extracting non-overlapping patches and linearly projecting their flattened representations.

[0042] Each Transformer block in Encoder 1 consists of N repeated Transformer layers, outputting tokens, p represents the size of the original slice, The relationship between the length and width H and W of the original image is , N is 3 in the encoder of this scheme.

[0043] The decoding stage mainly consists of reorganization blocks and fusion blocks. The reorganization operation is expressed as follows:

[0044] in represents the output feature dimension, s is the ratio of the slice dimension of the Transformer block output to the input image, and t represents token, whose dimensions are The Read operation is to Tokens are mapped to token, that is The Concatenate operation changes the dimensions from Convert to , thus transforming it into an image representation similar to the original image. The operation changes the dimension from Convert to The specific operation is to change the channel through 1×1 convolution. When s≥p, 3×3 convolution is used for downsampling. When s is less than p, 3×3 transposed convolution is used for upsampling. Indicates that Read, Concatenate, and Resample operations are performed on t in sequence.

[0045] The fusion block structure and depth estimation prediction head structure are as follows Figure 5 As shown, in this scheme, p=16, and the resolution of the final fusion block 1 output is half of the image, and the output of the depth estimation prediction head is consistent with the original image size.

[0046] Further, calculating the size of the polyp in the subsequent video frame according to the position information of the polyp and the depth information of the polyp in the depth map also includes: Obtaining the center coordinate depth information of the polyp according to the student depth prediction model; Calculating the initial polyp size of the subsequent video frame according to the center coordinate depth information and the position information; The initial polyp size is corrected to obtain a final polyp size of the subsequent video frame.

[0047] In this embodiment, the upper left and lower right coordinates of the polyp rectangular frame output by the intestinal polyp detection model are ; Calculate the image coordinates of the polyp center based on A and B coordinates , calculate the polyp pixel size as , where dx and dy represent the actual size of each pixel in the x and y directions respectively (unit: mm / pixel); According to the student depth prediction model, the depth of the polyp center coordinate C in the current image is obtained ; Calculation of initial polyp size ,in The current focal length of the endoscope lens; because is the distance from the center surface of the polyp to the endoscope lens. The initial polyp size needs to be corrected to obtain the final polyp size of the current image. , where λ is the correction constant, which is affected by the polyp morphology and has a value range of 2-100. It is tentatively set to 3 in this scheme.

[0048] Taking the polyp coordinates in step 1 as an example, the upper left coordinate is A[269, 215], the lower right coordinate is B[509, 416], then C[389,315]. Given that dx and dy are both 0.005 mm / pixel, the calculated polyp pixel length is 1.57 mm. The depth at the coordinate C[389, 315] is obtained from the depth estimation model. is 21.6mm, and the focal length of the endoscope is 5mm. =6.78mm, after correction =7.49mm.

[0049] Furthermore, the weighted calculation of the polyp size of each subsequent video frame and output of the standard polyp size of the colonoscopy video stream to be evaluated also includes: A circular queue is constructed in the depth of field of the lens according to the center depth of the polyp in each subsequent video frame; Based on the IoU values ​​of the polyp frames of the adjacent frames of each subsequent video frame and the depth of the polyp center within the depth of field range of the lens, it is determined whether the previous and next frames are the same polyp and the circular queue is directly entered or cleared; When the circular queue is full, a weighted moving average is used to output the standard polyp size of the colonoscopy video stream to be evaluated.

[0050] Further, when the polyp frame IoU values ​​of adjacent frames of each subsequent video frame and the depth of the polyp center are within the depth of field range of the lens, determining whether the previous and next frames are the same polyp and directly entering or clearing the circular queue includes: If the IoU values ​​of the polyp frames of adjacent frames of each subsequent video frame are greater than or equal to the first threshold and the depth of the center of the polyp is within the depth of field of the lens, the previous and next frames are considered to be the same polyp, and the current polyp size is added to the circular queue; If the IoU value of the polyp frame of the adjacent frames of each subsequent video frame is less than the first threshold and the depth of the polyp center is within the depth of field of the lens, it is determined that the previous and next frames are not the same polyp, then the circular queue is cleared, and the polyp size of the video frame to be evaluated is re-added to the cleared circular queue.

[0051] In this embodiment, in order to eliminate the influence of polyp blur on depth estimation, when the depth at the center C of the polyp in the i-th frame is In the depth of field range of the lens Internal time, l low and l high are the minimum and maximum depth of field of the lens, respectively. The current polyp size is added to a circular queue of length N. middle; If the adjacent frame polyp box IoU ≥ 0.3 and the depth at the polyp center In the depth of field range of the lens When the number of frames is within 1, the previous and next frames are considered to be the same polyp, and the current polyp size is added to a queue of length N. middle; If the adjacent frame polyp box IoU is less than 0.3 and the depth at the polyp center In the depth of field range of the lens If the previous and next frames are not the same polyp, the queue will be cleared. , add the current frame polyp size to a queue of length N middle; When the queue When full, using a weighted moving average, the output shows the size of the polyp: .

[0052] Taking N=5 as an example, the depth of field of the endoscope lens is [9mm, 100mm], and the polyp depths of 5 consecutive frames are [16.6mm, 16.8mm, 17.3mm, 17.8mm, 18.1mm], all within the depth of field of the endoscope, and the IoU is greater than 0.3. The polyp sizes calculated by step 4 are Q=[6.71mm, 7.12mm, 7.23mm, 6.78mm, 6.88mm], respectively. 6.94mm.

[0053] According to a second embodiment of the present invention, the present invention claims a colonoscopic polyp size assessment system based on multi-frame matching, comprising: one or more processors; A memory having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the colonoscopic polyp size assessment method based on multi-frame matching.

[0054] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0055] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units. The above is only an implementation mode of the present invention, and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the present invention specification and drawings, or directly or indirectly used in other related technical fields, is also included in the patent protection scope of the present invention.

[0056] The specific implementation methods of the invention are described in detail above, but they are only examples, and the invention is not limited to the specific implementation methods described above. For those skilled in the art, any equivalent modification or substitution of the invention is also within the scope of the invention, therefore, the equalization, modification, improvement, etc. made without departing from the spirit and principle of the invention should be included in the scope of the invention.

Claims

1. A colonoscopic polyp size assessment method based on multi-frame matching, characterized in that: include: Obtain a colonoscopy video stream to be evaluated, and determine whether there is a polyp in the video frame to be evaluated in the colonoscopy video stream to be evaluated by using a colon polyp detection model; When a polyp exists in the video frame to be evaluated, obtaining location information of the polyp and using a perceptual hash algorithm to calculate image similarity between two adjacent frames of N subsequent video frames of the video frame to be evaluated; When the image similarity of two adjacent frames of N subsequent video frames meets a preset condition, a corresponding depth map is generated using a depth estimation model; Calculating the size of the polyp in subsequent video frames according to the position information of the polyp and the depth information of the polyp in the depth map; The polyp size of each subsequent video frame is weightedly calculated, and the standard polyp size of the colonoscopy video stream to be evaluated is output.

2. The method for evaluating the size of colonoscopic polyps based on multi-frame matching according to claim 1, characterized in that: The method of obtaining the colonoscopy video stream to be evaluated and determining whether there is a polyp in the video frame to be evaluated in the colonoscopy video stream to be evaluated by using the intestinal polyp detection model also includes: The intestinal polyp detection model uses lower digestive tract polyp data as a training set; For the input colonoscopy video stream to be evaluated, the intestinal polyp detection model outputs the upper left and lower right coordinates and confidence of the polyp position in the video frame to be evaluated in the colonoscopy video stream to be evaluated; When the confidence level is greater than a preset threshold, it is considered that polyps exist in the video frame to be evaluated.

3. The method for evaluating the size of colonoscopic polyps based on multi-frame matching according to claim 1, characterized in that: When a polyp exists in the video frame to be evaluated, obtaining the location information of the polyp and using a perceptual hash algorithm to calculate the image similarity of two adjacent frames of N subsequent video frames of the video frame to be evaluated also includes: Performing image preprocessing on N subsequent video frames of the video frame to be evaluated, and calculating and obtaining a DCT coefficient matrix of each subsequent video frame; Extracting submatrices of each DCT coefficient matrix as image low-frequency information of each subsequent video frame, and calculating the mean of each submatrix; Based on the comparison result of the image low-frequency information of each subsequent video frame and the mean value of each sub-matrix, the matching hash value of each subsequent video frame is calculated; The Hamming distance between two adjacent frames of each subsequent video frame is calculated according to the matching hash value, and the image similarity between two adjacent frames of each subsequent video frame is determined based on the Hamming distance.

4. The method for evaluating the size of colonoscopic polyps based on multi-frame matching according to claim 3, characterized in that: When the image similarity of two adjacent frames of the N subsequent video frames meets a preset condition, using the depth estimation model to generate a corresponding depth map also includes: When the Hamming distances between two adjacent frames of the N subsequent video frames are all less than a preset threshold, determining that the image similarity between two adjacent frames of the N subsequent video frames meets a preset condition; The simulated rendered images and real colonoscopy images are used to train a student depth prediction model, and a corresponding depth map is generated based on the student depth prediction model.

5. The method for evaluating the size of colonoscopic polyps based on multi-frame matching according to claim 4, characterized in that: The method of using the simulated rendering image and the real colonoscopy image to train the student depth prediction model also includes: The simulated rendered image is sequentially input into the encoder 1 and the decoder, and the loss is calculated in the rendered intestinal image training according to the real depth label; The real colonoscopy image is subjected to data enhancement processing and then input into encoder 1, and the loss is calculated in the real intestinal image training according to the pseudo depth label; The real colonoscopy image is input into encoder 2, and semantic segmentation features are introduced as auxiliary supervision information, and the output features of encoder 2 are used as auxiliary semantic supervision; The pseudo depth label is predicted by a teacher model on a real colonoscopy image, wherein the teacher model is trained only with rendered images having real depth labels, and the model structure of the teacher model is the same as that of the student model.

6. The method for evaluating the size of colonoscopic polyps based on multi-frame matching according to claim 5, characterized in that: Calculating the size of the polyp in the subsequent video frame according to the position information of the polyp and the depth information of the polyp in the depth map, further comprising: Obtaining the center coordinate depth information of the polyp according to the student depth prediction model; Calculating the initial polyp size of the subsequent video frame according to the center coordinate depth information and the position information; The initial polyp size is corrected to obtain a final polyp size of the subsequent video frame.

7. The method for evaluating the size of colonoscopic polyps based on multi-frame matching according to claim 6, characterized in that: The weighted calculation of the polyp size of each subsequent video frame and outputting the standard polyp size of the colonoscopy video stream to be evaluated also includes: A circular queue is constructed in the depth of field of the lens according to the center depth of the polyp in each subsequent video frame; Based on the IoU values ​​of the polyp frames of the adjacent frames of each subsequent video frame and the depth of the polyp center within the depth of field range of the lens, it is determined whether the previous and next frames are the same polyp and the circular queue is directly entered or cleared; When the circular queue is full, a weighted moving average is used to output the standard polyp size of the colonoscopy video stream to be evaluated.

8. The method for evaluating the size of colonoscopic polyps based on multi-frame matching according to claim 7, characterized in that: The method further includes: determining whether the preceding and following frames are the same polyp and directly entering or clearing the circular queue when the polyp frame IoU values ​​of adjacent frames of each subsequent video frame and the depth of the polyp center are within the depth of field range of the lens; and If the IoU values ​​of the polyp frames of adjacent frames of each subsequent video frame are greater than or equal to the first threshold and the depth of the center of the polyp is within the depth of field of the lens, the previous and next frames are considered to be the same polyp, and the current polyp size is added to the circular queue; If the IoU value of the polyp frame of the adjacent frames of each subsequent video frame is less than the first threshold and the depth of the polyp center is within the depth of field of the lens, it is determined that the previous and next frames are not the same polyp, then the circular queue is cleared, and the polyp size of the video frame to be evaluated is re-added to the cleared circular queue.

9. A colonoscopic polyp size assessment system based on multi-frame matching, characterized in that: include: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a colonoscopic polyp size assessment method based on multi-frame matching according to any one of claims 1 to 8.

Images