Deep learning model-based polyp image size estimation method and system

Abstract

The invention discloses a polyp image size estimation method and system based on a deep learning model. The method comprises the following steps: acquiring a plurality of image frames; the frame with the maximum pixel diameter is used as a representative frame; estimating a target polyp depth value by adopting a depth estimation model, and calculating a representative depth value by adopting a median-based mode; and calculating a product of the representative pixel diameter, the representative depth value and a given scale factor to obtain a size estimation value of the target polyp. According to the unified polyp size estimation method, a depth estimation model and a segmentation model are combined, and the requirement for clear calibration of a camera is eliminated by estimating an absolute depth size scale factor; meanwhile, a plurality of visual angles are collected and analyzed, and a frame capable of presenting the maximum apparent diameter is selected, so that the polyp cross section is measured through an approximately vertical visual angle, the influence caused by the angle problem is effectively solved, and more accurate and reliable polyp size measurement can be provided.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a polyp image size estimation method and system based on a deep learning model. Background Art

[0002] The discovery of polyps from endoscopic images has important medical value. It not only helps to analyze the risk of lesions, but also provides key reference materials for doctors to specify treatment plans.

[0003] After polyps are detected in images, accurately estimating their size is crucial because size is an important parameter in the reference information. For example, smaller polyps tend to have a lower risk of malignancy, while larger polyps tend to indicate an increased risk of malignancy.

[0004] Of course, polyp size is not the only criterion for diagnosing a certain disease, but only a reference standard. The specific evaluation and diagnosis of disease types, stages and potential risks require professional judgment by professional doctors. The evaluation of polyp size only serves as a reference.

[0005] However, although it is reference data, it is still very important to accurately assess the size of polyp targets in the image.

[0006] With the development of artificial intelligence and image processing technology, automated polyp size measurement methods are becoming an effective auxiliary tool to improve the objectivity and consistency of measurements. These technologies, combined with deep learning algorithms and 3D reconstruction methods, make polyp size estimation more accurate. Existing methods usually combine depth estimation models with two-dimensional images, but often require prior knowledge of the intrinsic parameters of the endoscope (especially the focal length) to accurately complete the conversion from pixels to millimeters. This parameter needs to be adjusted according to different cameras and may not be available in some practical application scenarios. In addition, to avoid geometric errors, it is usually required that the optical axis of the camera is as perpendicular as possible to the maximum cross-section of the polyp, otherwise the diameter measured in the image will be difficult to accurately reflect the true size of the polyp. However, in actual operation, it is difficult to control the optical axis of the endoscope to be completely perpendicular to the maximum cross-section of the polyp. This leads to the problem that in some cases where vertical polyp images cannot be obtained, the size cannot be assessed or the size assessment error is large. Summary of the Invention

[0007] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a polyp image size estimation method and system based on a deep learning model.

[0008] To achieve the aforementioned object of the invention, the technical solutions adopted by the present invention include:

[0009] In a first aspect, the present invention provides a method for estimating the size of a polyp image based on a deep learning model, comprising:

[0010] Acquire multiple image frames containing the target polyp from multiple different perspectives;

[0011] Select an image frame with the largest maximum pixel diameter occupied by the target polyp as a representative frame, and the corresponding maximum pixel diameter as a representative pixel diameter;

[0012] Using a depth estimation model, estimating the depth value of each pixel occupied by the target polyp, and calculating a representative depth value based on the depth values corresponding to multiple pixels using a median-based approach;

[0013] The product of the representative pixel diameter, the representative depth value, and a given scale factor is calculated to obtain an estimated size value of the target polyp, wherein the scale factor is determined by optical parameters of a camera used to capture the image frame.

[0014] In a second aspect, the present invention further provides a polyp image size estimation system based on a deep learning model, which is used to implement the above-mentioned polyp image size estimation method, and comprises:

[0015] An image acquisition module, configured to acquire a plurality of image frames containing target polyps at a plurality of different viewing angles;

[0016] a representative diameter module, configured to select an image frame having the largest maximum pixel diameter occupied by the target polyp as a representative frame, and the corresponding maximum pixel diameter as a representative pixel diameter;

[0017] a representative depth module, configured to estimate the depth value of each pixel occupied by the target polyp using a depth estimation model, and calculate a representative depth value based on the depth values corresponding to a plurality of pixels using a median-based method;

[0018] A size estimation module is configured to calculate the product of the representative pixel diameter, the representative depth value, and a given scaling factor to obtain a size estimation value of the target polyp, wherein the scaling factor is determined by optical parameters of a camera used to capture the image frame.

[0019] In a third aspect, the present invention further provides a readable storage medium, wherein a computer program is stored in the readable storage medium, and when the computer program is executed, the steps of the above-mentioned polyp image size estimation method are executed.

[0020] Based on the above technical solution, compared with the prior art, the beneficial effects of the present invention include at least:

[0021] This invention provides a unified polyp size estimation method that combines depth estimation and segmentation models. By estimating the absolute depth scale factor, it eliminates the need for explicit camera calibration. Furthermore, by acquiring and analyzing multiple viewpoints and selecting the frame that exhibits the largest apparent diameter, polyp cross-sections are measured from a near-perpendicular perspective, effectively addressing the impact of angularity. By utilizing state-of-the-art segmentation and depth estimation models, combined with an improved diameter calculation algorithm, this invention provides more accurate and reliable polyp size measurements. This improvement provides clinicians with more precise reference information, helping to more rationally formulate subsequent management and treatment strategies.

[0022] The above description is only an overview of the technical solution of the present invention. In order to enable those skilled in the art to more clearly understand the technical means of this application and implement them according to the contents of the specification, the following is an explanation of the preferred embodiments of the present invention with detailed drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a process diagram of the first step of a polyp image size estimation method provided by a typical embodiment of the present invention;

[0024] Figure 2 2 is a process diagram of the second step of the polyp image size estimation method provided by a typical embodiment of the present invention;

[0025] Figure 3 1 is a process diagram of the third step of the polyp image size estimation method provided by a typical embodiment of the present invention;

[0026] Figure 4 3 is a process diagram of the fourth step of the polyp image size estimation method provided in a typical embodiment of the present invention. DETAILED DESCRIPTION

[0027] In view of the shortcomings of the prior art, the inventors of this case, after long-term research and extensive practice, have proposed the technical solution of the present invention. The following will further explain this technical solution, its implementation process and principles.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0029] Moreover, relational terms such as “first” and “second” are merely used to distinguish one component or method step from another with the same name, but do not necessarily require or imply any actual relationship or order between these components or method steps.

[0030] An embodiment of the present invention provides a method for estimating the size of polyps images based on a deep learning model (hereinafter referred to as the "method"), which includes the following steps:

[0031] Acquire multiple image frames containing the target polyp from multiple different perspectives;

[0032] Select an image frame with the largest maximum pixel diameter occupied by the target polyp as a representative frame, and the corresponding maximum pixel diameter as a representative pixel diameter;

[0033] Using a depth estimation model, estimating the depth value of each pixel occupied by the target polyp, and calculating a representative depth value based on the depth values corresponding to multiple pixels using a median-based approach;

[0034] The product of the representative pixel diameter, the representative depth value, and a given scale factor is calculated to obtain an estimated size value of the target polyp, wherein the scale factor is determined by optical parameters of a camera used to capture the image frame.

[0035] In some embodiments, the process of acquiring the plurality of image frames specifically includes:

[0036] providing a video stream, and continuously identifying the target polyp in the video stream using an image segmentation model;

[0037] When the target polyp is identified, a plurality of video frames in which the target polyp appears at different positions in the image are captured as the image frames.

[0038] Whether executing the above method during the endoscopic examination or in the images recorded after the endoscopic examination, it is a feasible embodiment of the present invention and is not necessarily limited to being performed simultaneously with the endoscopic examination. It should be noted that the present invention is a technical solution for information processing and parameter extraction based on images in a computer. It is based on optical imaging principles and computer principles, rather than biological or medical principles. It does not belong to the diagnosis and treatment of diseases. Some of the data obtained are only evaluation values of physical dimensions and are not direct diagnostic basis. For example, a doctor cannot easily determine the type, stage, prognosis, etc. of the disease based solely on the estimated size provided by the present invention. It must be combined with other key factors, such as morphology, blood flow, etc. for evaluation, and the ultimate gold standard is still the pathology report, not the endoscopic image.

[0039] In the above technical solution, in order to achieve polyp size estimation and integrate depth estimation and segmentation into a single process, thereby overcoming the limitations of existing methods, the present invention proposes a polyp diameter measurement method that does not require calibration and is insensitive to angles. As a typical example, this method can combine advanced target segmentation models (such as YoloSeg, MaskRcnn) with Transformer-based monocular depth estimation models (such as DepthAnythingV2) when applied, and ensure reliable depth measurement through multi-stage fine-tuning. By eliminating the need for manual calibration and minimizing the inaccuracy caused by camera angles, the polyp size measurement process is greatly simplified and can be easily applied to conventional endoscopic images.

[0040] Generally, to ensure assessment accuracy, the shooting distance (i.e., the physical distance between the camera lens and the polyp) of multiple image frames is required to be roughly consistent to facilitate statistical screening in each step. Specifically, in some embodiments, the shooting distance of multiple image frames is generally required to fluctuate within a range of ±25%. Of course, in actual applications, this value may not be the limit and can be adjusted appropriately based on the performance of the endoscope.

[0041] Alternatively, the camera's imaging principle can be used to limit this. Specifically, in some embodiments, the image frames are selected so that the lens-to-polyp shooting distance remains consistent across multiple frames, and the images of the target polyps in the selected frames are clear. When the focal length remains constant and the image is generally clear, the shooting distance is typically relatively fixed, thus ensuring that the shooting distance does not vary significantly across frames. Regarding how to determine image clarity, numerous technical solutions exist in the prior art, and it is unnecessary to list them all here. Generally, clarity is ultimately determined by ensuring that polyp segmentation can be achieved.

[0042] The present invention has made innovations in four core aspects, which further improves the practicality and accuracy of real-time measurement of polyp size.

[0043] One of the core innovations lies in the overall process framework. As a typical example, the main steps of the above method include:

[0044] Step 1: After the segmentation model detects a polyp in the image, it maintains a roughly constant distance from the polyp and captures multiple image frames by tilting the endoscopic camera appropriately in both the vertical and horizontal axes to cover different viewing angles.

[0045] Step 2: For the selected image frame, calculate the convex hull of the polyp boundary point set obtained by the segmentation model in step 1. The maximum Euclidean distance between any two pixels on the convex hull in the image coordinate system is the maximum pixel diameter of the polyp.

[0046] Step 3: Use the depth estimation model to perform depth estimation on the entire image. Then, based on the polyp mask obtained in step 1, retain the depth values corresponding to the polyp area, ignore the background depth values, and use a median-based outlier filtering method to obtain a representative depth value for the polyp (e.g., in millimeters).

[0047] Step 4: Combine the maximum pixel diameter obtained in step 2, the representative depth value obtained in step 3, and the scale factor designed by the present invention, and substitute these parameters into the polyp size estimation formula to calculate the millimeter-level size of the polyp in real space (this is not limited to this, and the size in other units can also be used. The present invention only uses millimeters as a representative example).

[0048] Regarding the specific processes of the above steps, in some embodiments, the calculation process of the maximum pixel diameter specifically includes:

[0049] Constructing a minimum convex polygon that encompasses all pixel points of the target polyp;

[0050] Calculating the pixel distance between each vertex in the minimum convex polygon and the remaining vertices;

[0051] The maximum value among all the pixel distances is selected as the maximum pixel diameter.

[0052] In the process of calculating the maximum diameter and representative depth, in order to improve efficiency and robustness, an embodiment of the present invention designs and implements a convex hull-based method to obtain the maximum pixel diameter. This method can more accurately and efficiently complete the polyp size measurement.

[0053] In more detail, in some embodiments, the calculation process of the maximum pixel diameter is expressed as:

[0054] P={(x1,y1),(x2,y2),…,(x n ,y n )}

[0055] H(P)={(X1,Y1),(X2,Y2),…,(X m , Y m )}

[0056]

[0057] Wherein, P represents the set of boundary pixel points of the target polyp, (x., y.) represents the pixel coordinates of the boundary pixel points, H(P) represents the minimum convex polygon, (X., Y.) represents the coordinates of the vertices, and W max represents the maximum pixel diameter, n represents the number of boundary pixels, m represents the number of vertices, and i and j represent natural numbers.

[0058] To address the angle-related issues in traditional 2D methods and select a view that is approximately perpendicular to the polyp's largest cross-section, a typical implementation case of the present invention designs and implements a method: by analyzing image frames acquired from different perspectives, the frame that is closest to being perpendicular to the polyp's cross-section is automatically selected as the approximately perpendicular frame.

[0059] That is, in some embodiments, the representative frame selection process is expressed as:

[0060] F={F1,F2,…,F l}

[0061]

[0062] Wherein, F represents the set of image frames, F. represents the image frame, and F * represents the representative frame, W k represents the maximum pixel diameter of the kth image frame, l represents the number of image frames, and k represents a natural number.

[0063] In addition, the present invention further adopts a method based on median outlier filtering to obtain representative depth.

[0064] That is, in some embodiments, the calculation process of the representative depth value is expressed as:

[0065] M0=median{d p |p∈Q}

[0066]

[0067] Q′={d p ∈Q||d p -M0|≤3σ0}

[0068] d rep =median{d p |p∈Q}

[0069] Among them, M0 represents the initial median, d prepresents the depth estimation value of the pixel point p corresponding to the target polyp, p represents the pixel point, Q represents the set of pixel points corresponding to the target polyp, σ0 represents the standard deviation, |Q| represents the number of pixel points corresponding to the target polyp, Q′ represents the filtered inlier set, d rep represents the representative depth value.

[0070] To eliminate the need for manual calibration, embodiments of the present invention design and implement a method for converting pixel measurements to true size using a scale factor. This scale factor can be estimated by training a linear model using known true polyp size data. Once the scale factor is determined, it can be used in the same imaging environment.

[0071] That is, in some embodiments, the size estimate is calculated as:

[0072] D mm =α×W max ×d rep

[0073] Among them, D mm represents the estimated size value, α represents the scale factor, and the process of determining the scale factor includes:

[0074] Extracting corresponding representative pixel diameters and representative depth values from multiple training images and calculating the product of the two as training data, and obtaining the true value size corresponding to the training image;

[0075] calculating an initial scale factor based on a plurality of the training data and a true value size;

[0076] Calculating a size estimate in any training image according to the initial scale factor, and calculating a residual difference between the size estimate and the true size;

[0077] Based on the statistics of the residuals corresponding to the plurality of training images, outliers whose average value of the distance residuals exceeds 3 times the standard value are eliminated to obtain retained points;

[0078] According to the retention points, a final scaling factor is calculated based on the corresponding plurality of the training data and the true value size.

[0079] In some embodiments, the scaling factor is determined as follows:

[0080]

[0081] r t =f t -α(init)e t

[0082]

[0083] Among them, α(init) represents the initial scale factor, e t represents the training data, f t represents the true value size, t represents the sample number, T represents the initial sample set, N T represents the total number of samples in set T, r t represents the residual, μ r represents the mean value of the residuals, σ r represents the standard deviation of the residual, S represents the set of retained points, N S represents the total number of samples in set S, α final Represents the final scale factor.

[0084] A second aspect of an embodiment of the present invention further provides a polyp image size estimation system based on a deep learning model, which is used to implement the polyp image size estimation method provided in any of the above embodiments, and includes:

[0085] An image acquisition module, configured to acquire a plurality of image frames containing target polyps at a plurality of different viewing angles;

[0086] a representative diameter module, configured to select an image frame having the largest maximum pixel diameter occupied by the target polyp as a representative frame, and the corresponding maximum pixel diameter as a representative pixel diameter;

[0087] a representative depth module, configured to estimate the depth value of each pixel occupied by the target polyp using a depth estimation model, and calculate a representative depth value based on the depth values corresponding to a plurality of pixels using a median-based method;

[0088] A size estimation module is configured to calculate the product of the representative pixel diameter, the representative depth value, and a given scaling factor to obtain a size estimation value of the target polyp, wherein the scaling factor is determined by optical parameters of a camera used to capture the image frame.

[0089] An embodiment of the present invention further provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed, the steps of the polyp image size estimation method provided in any of the above embodiments are executed.

[0090] The technical solution of the present invention is further described in detail below through several embodiments and in conjunction with the accompanying drawings. However, the selected embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0091] Example 1

[0092] This embodiment proposes a method for accurately measuring the diameter of polyps that does not require calibration and is insensitive to angles. Specifically, this method combines advanced target segmentation models (such as YoloSeg, MaskRcnn, and this embodiment can adopt a variety of target segmentation models and methods, not limited to these specific examples) with a Transformer-based monocular depth estimation model (this embodiment uses DepthAnythingV2, but is not limited to this) to obtain more accurate depth information for the target position. By eliminating the need for manual calibration and minimizing the errors caused by changes in camera angles, the clinical operation process can be simplified and can be seamlessly integrated into common endoscopic surgeries.

[0093] The following are the main steps of this embodiment:

[0094] Step 1: If Figure 1 As shown in Figure 2, the best aligned frame is automatically selected; once the segmentation model S detects polyps in the endoscopic video stream, the system collects multiple frames of images. l}, they are captured at different tilt angles but at a roughly constant distance. For each frame F, the system measures the polyp's pixel diameter using a convex hull-based approach. The frame with the maximum pixel diameter is the view most orthogonal to the polyp's cross section, so the system selects a representative frame:

[0095]

[0096] That is, the image frame with the largest diameter is selected as the representative frame.

[0097] In the above steps, after inputting the image I, the binary mask M∈{0, 1} is output. H×W To mark the polyp area. In actual endoscopic examination, the system acquires a series of frames F = {F1, F2, ..., F l}, where each frame F. is taken at a specific tilt angle. By keeping the distance to the polyp roughly constant, the apparent size change caused by angle change can be decoupled from the size change caused by distance change. For each frame F., the segmentation model S can be used to obtain the polyp edge point set P = {(x i ,y i )|i=1,...,n}. Then calculate the maximum diameter d of the frame i As the camera is tilted about the vertical and horizontal axes, the system records the frame with the largest measured diameter, that is, the viewing angle that is closest to being perpendicular to the main cross-section of the polyp.

[0098] This embodiment does not seek absolute orthogonality, but rather minimizes measurement errors caused by angular deviation. Rather than relying on "guessing" the appropriate angle during endoscopic operation, the method of the present invention can automatically and reliably identify the angle that maximizes the polyp diameter measurement.

[0099] Step 2: If Figure 2 As shown, calculate or retain the maximum pixel diameter of the polyp on the selected representative frame; calculate the convex hull diameter. Assume:

[0100] P={(x i ,y i )|i=1,...,n}

[0101] is the polyp boundary point set (pixel coordinates) obtained by the segmentation model in step 1. The system calculates its convex hull:

[0102] H(P)={(X1, Y1), ..., (X m , Y m )}

[0103] in To obtain the maximum diameter of the polyp in the pixel coordinate system, the maximum Euclidean distance between the convex hull vertices is calculated, denoted as W. max , as the maximum pixel diameter of the polyp.

[0104] In order to improve the efficiency and robustness of calculating the maximum diameter and representative depth, this embodiment designs and implements a method for extracting the maximum pixel diameter based on the convex hull and calculating the representative depth based on median filtering. In the maximum diameter calculation, the shapely library in Python can be used to calculate its convex hull. The convex hull is the smallest convex polygon containing all points. Calculating the distance only for the vertices of the convex hull is more efficient than calculating it on all boundary points, and has better robustness for concave boundaries because the convex hull ignores the inner points or depressions that are not related to the maximum span. Subsequently, the maximum value of the Euclidean distance between the two points is found between the convex hull vertices:

[0105]

[0106] This value is the maximum chord length in pixel coordinates, which represents the maximum pixel diameter of the polyp.

[0107] Step 3: If Figure 3 As shown in , the representative depth of the polyp is calculated on the selected frame; the depth is predicted and the representative depth of the polyp area is calculated. is the pixel plane where the input endoscopic image is located, let is a depth estimation model that maps each pixel x∈Ω to a depth value in millimeters. is the polyp binary mask obtained in step 1, then:

[0108] S={D(x)|x∈M}.

[0109] Use median-based outlier filtering technology to filter S and obtain the representative depth d rep , denoted as d rep =median(S filtered ).

[0110] In the representative depth calculation, let M be the binary mask of the polyp obtained in step 1, and define Q = {p|p∈M} as the set of pixels p in the mask area, corresponding to the area where the polyp is located. For each pixel in the area, there is a depth d in millimeters on the depth map. p .

[0111] a. Initial median:

[0112] M0=median{d p |p∈Q}

[0113] The median is less sensitive to outliers than the mean, so M0 is used as a stable initial value for polyp depth.

[0114] b. Outlier filtering: Calculate the variance around M0:

[0115]

[0116] c. Define the interior point set:

[0117] Q′={d p ∈Q||d p -M0|≤3σ0}

[0118] Any pixel depth that differs from M0 by more than 3σ0 is considered an outlier and is removed.

[0119] d. Final representative depth:

[0120] d rep =median{d p |p∈Q}

[0121] This final median value excludes extreme depth values and provides a stable and accurate estimate of the distance between the polyp and the endoscope.

[0122] Step 4: If Figure 4 As shown, the results are converted to millimeter scale polyp size. Let α be a known scaling factor used to convert pixel measurements to millimeters. The maximum 2D pixel diameter of the polyp is W max , and the polyp representative depth obtained in the previous step is d rep(in millimeters). Then the diameter of the polyp in real space is D mm The calculation formula is:

[0123] D mm =α×W max ×d rep

[0124] This method is most accurate when the camera optical axis is nearly perpendicular to the polyp cross-section plane and the camera is kept at an appropriate distance from the polyp (avoiding getting too close or using a zoom operation that changes the focal length); if these conditions are violated, the zoom factor will not work accurately.

[0125] To avoid camera calibration, this embodiment designs and implements a method for estimating the scale factor for converting pixels to real-world dimensions. Endoscopic images follow perspective projection, and objects at different depths appear at different scales in the image. The principle is as follows:

[0126] Let PolypSize(pixels) be the diameter of the polyp in pixel coordinates, and let Depth(mm) be the corresponding depth estimate in millimeters. The true polyp size can be expressed as

[0127]

[0128] Since it is often inconvenient to calibrate the focal length directly, we can define:

[0129]

[0130] Thus, the above formula can be rewritten as:

[0131] RealSize(mm)=α×PolypSize(pixels)×PredictedDepth(mm).

[0132] The goal of this step is to use the data of the calibrated polyp size to estimate the scaling factor α, which can be reused in subsequent operations when the optical properties of the camera do not change.

[0133] To this end, a large number of observations were collected during the implementation. in:

[0134] yt = actual size of the marked polyp (mm).

[0135] Introducing the linear model y = α × x, the main technical idea is to use the least squares estimation:

[0136]

[0137] in has a closed-form solution.

[0138] However, in order to improve the stability of the scaling factor, this embodiment also uses a method based on residual statistics to eliminate abnormal samples:

[0139] a. Initial fitting: using all sample pairs (e t , f t ) to calculate the initial α(init).

[0140] b. Residual calculation:

[0141] r t =f t -α(init)e t

[0142] c. Standardized residual: Let μ r and σ r They are the residuals {r i}, calculate the mean and standard deviation:

[0143]

[0144] and remove any (e t , f t ) satisfies |z t |>3 points.

[0145] d. Quadratic fitting: Recalculate α on the set of inliers retained after screening to obtain the final estimate α final .

[0146] Once you get α final , allowing for repeated use in subsequent examinations without changing the camera's optical parameters. This method eliminates the need for an explicit focus calibration step. This process is performed offline, one-time, rather than dynamically during the endoscopic procedure. This method is most accurate when the camera's optical axis is nearly perpendicular to the polyp cross-section and the appropriate distance is maintained. The zoom factor accuracy decreases when the lens is too close or when magnification and zooming are used, resulting in significant changes in focus.

[0147] The general operating principle of the above implementation scheme is:

[0148] by Represents a set of multiple frames of endoscopic images. Let s be the target segmentation model, which outputs a set of polyp boundary points for the kth frame. Then calculate the convex hull of these points:

[0149]

[0150] Get the polygon formed by the subset of boundary points. Define the pixel diameter of the kth frame as:

[0151]

[0152] where ||·|| represents the Euclidean norm. The frame with the largest diameter is then selected.

[0153] Deep Model A reliable metric depth Z is predicted for each pixel in the selected image. A mask is applied to the polyp region to filter the corresponding depth values, ignoring the background depth values, and a median-based outlier filtering algorithm is used to obtain a representative depth d rep Finally, the scaling factor α and representative depth d are obtained by fitting the polyp size under the same type of lens. rep , converting the pixel measurements to their true dimensions gives an estimate of the polyp diameter:

[0154]

[0155] Based on the above embodiments, it is clear that the technical solutions provided by the embodiments of the present invention bring many significant benefits, including:

[0156] Accurate and calibration-free polyp size measurement: Precise polyp size measurement can be achieved without manually acquiring endoscopic camera parameters.

[0157] Angle-insensitive analysis: Automatically identifies the closest vertical image frame at different angles to ensure maximum apparent diameter.

[0158] Simplified end-to-end workflow: Integrates advanced object segmentation models with depth estimation models to simplify the operational process of polyp size measurement.

[0159] By reliably capturing and calculating polyp size in real time in this way, suspicious lesions can be risk-graded more consistently and accurately, inconsistencies caused by operator differences can be reduced, and more informed decisions can be made on whether additional diagnostic or therapeutic intervention is needed. This invention significantly improves the efficiency and reliability of conventional endoscopic examinations.

[0160] It should be understood that the above embodiments are merely illustrative of the technical concepts and features of the present invention. Their purpose is to enable those skilled in the art to understand the contents of the present invention and implement them accordingly. They are not intended to limit the scope of protection of the present invention. Any equivalent variations or modifications made in accordance with the spirit and substance of the present invention are intended to be encompassed within the scope of protection of the present invention.

Claims

1. A polyp image size estimation method based on a deep learning model, characterized in that: include: Acquire multiple image frames containing the target polyp from multiple different perspectives; Select an image frame with the largest maximum pixel diameter occupied by the target polyp as a representative frame, and the corresponding maximum pixel diameter as a representative pixel diameter; Using a depth estimation model, estimating the depth value of each pixel occupied by the target polyp, and calculating a representative depth value based on the depth values corresponding to multiple pixels using a median-based approach; The product of the representative pixel diameter, the representative depth value, and a given scale factor is calculated to obtain an estimated size value of the target polyp, wherein the scale factor is determined by optical parameters of a camera used to capture the image frame.

2. The polyp image size estimation method according to claim 1, characterized in that: The process of acquiring the plurality of image frames specifically includes: providing a video stream, and continuously identifying the target polyp in the video stream using an image segmentation model; When the target polyp is identified, a plurality of video frames in which the target polyp appears at different positions and angles in the image are captured as the image frames.

3. The polyp image size estimation method according to claim 2, characterized in that: The floating range of the shooting distance of the multiple image frames is ±25%. When the image frames are selected, the shooting distance from the lens to the polyp in the multiple image frames is kept consistent, and the images of the target polyps in the selected multiple image frames are clear.

4. The polyp image size estimation method according to claim 1, characterized in that: The calculation process of the maximum pixel diameter specifically includes: Constructing a minimum convex polygon that encompasses all pixel points of the target polyp; Calculating the pixel distance between each vertex in the minimum convex polygon and the remaining vertices; The maximum value among all the pixel distances is selected as the maximum pixel diameter.

5. The polyp image size estimation method according to claim 4, characterized in that: The calculation process of the maximum pixel diameter is expressed as: P={(x1,y1),(x2,y2),…,(x n ,y n )} H(P)={(X1,Y1),(X2,Y2),…,(X m ,Y m )} Wherein, P represents the set of boundary pixel points of the target polyp, (x., y.) represents the pixel coordinates of the boundary pixel points, H(P) represents the minimum convex polygon, (X., Y.) represents the coordinates of the vertices, and W max represents the maximum pixel diameter, n represents the number of boundary pixels, m represents the number of vertices, and i and j represent natural numbers.

6. The polyp image size estimation method according to claim 1, characterized in that: The selection process of the representative frame is expressed as: F={F1,F2,…,F l } Wherein, F represents the set of image frames, F. represents the image frame, and F * represents the representative frame, W k represents the maximum pixel diameter of the kth image frame, l represents the number of image frames, and k represents a natural number.

7. The polyp image size estimation method according to claim 1, characterized in that: The calculation process of the representative depth value is expressed as follows: M0=median{d p |p∈Q} Q′={d p ∈Q||d p -M0|≤3σ0} d rep =median{d p |p∈Q} Among them, M0 represents the initial median, d p represents the depth estimation value of the pixel point p corresponding to the target polyp, p represents the pixel point, Q represents the set of pixel points corresponding to the target polyp, σ0 represents the standard deviation, |Q| represents the number of pixel points corresponding to the target polyp, Q′ represents the filtered inlier set, d rep represents the representative depth value.

8. The polyp image size estimation method according to claim 1, characterized in that: The size estimate is calculated as: D mm =α×W max ×d rep Among them, D mm represents the estimated size value, α represents the scale factor, and the process of determining the scale factor includes: Extracting corresponding representative pixel diameters and representative depth values from multiple training images and calculating the product of the two as training data, and obtaining the true value size corresponding to the training image; calculating an initial scale factor based on a plurality of the training data and a true value size; Calculating a size estimate in any training image according to the initial scale factor, and calculating a residual difference between the size estimate and the true size; Based on the statistics of the residuals corresponding to the plurality of training images, outliers whose average value of the distance residuals exceeds 3 times the standard value are eliminated to obtain retained points; According to the retention points, a final scaling factor is calculated based on the corresponding plurality of the training data and the true value size.

9. The polyp image size estimation method according to claim 8, characterized in that: The determination process of the proportional factor is expressed as: r t =f t -α(init)e t Among them, α(init) represents the initial scale factor, e t represents the training data, f t represents the true value size, t represents the sample number, T represents the initial sample set, N T represents the total number of samples in set T, r t represents the residual, μ r represents the mean value of the residuals, σ r represents the standard deviation of the residual, S represents the set of retained points, N S represents the total number of samples in set S, α final Represents the final scale factor.

10. A polyp image size estimation system based on a deep learning model, used to implement the polyp image size estimation method according to any one of claims 1 to 9, characterized in that: include: An image acquisition module, configured to acquire a plurality of image frames containing the target polyp at a plurality of different viewing angles; a representative diameter module, configured to select an image frame having the largest maximum pixel diameter occupied by the target polyp as a representative frame, and the corresponding maximum pixel diameter as a representative pixel diameter; a representative depth module, configured to estimate the depth value of each pixel occupied by the target polyp using a depth estimation model, and calculate a representative depth value based on the depth values corresponding to a plurality of pixels using a median-based method; A size estimation module is configured to calculate the product of the representative pixel diameter, the representative depth value, and a given scaling factor to obtain a size estimation value of the target polyp, wherein the scaling factor is determined by optical parameters of a camera used to capture the image frame.

Images