Method for acquiring information of polyp under endoscope based on monocular vision

Abstract

The invention belongs to the field of medical image processing and endoscope detection, and relates to a method for acquiring information of a polyp under an endoscope based on monocular vision, and the information of the polyp comprises the distance between the polyp and a lens of the endoscope. The acquisition method comprises the following steps: acquiring an in-vivo endoscope image when the biopsy forceps extend out of an endoscope channel, the forceps body is straight, the jaw is closed and the jaw abuts against a polyp, and then converting the in-vivo endoscope image into an aerial view B; finally, the pixel length of the biopsy forceps body in the aerial view B is obtained and substituted into the mapping model to obtain the physical length of the biopsy forceps body in the aerial view B, and the sum of the physical length of the biopsy forceps body in the aerial view B and the physical length of the biopsy forceps head serves as the distance between the polyp and the lens of the endoscope. According to the method, a high-precision pixel-physical space mapping model is established, measurement errors caused by image distortion, view angle variation and shielding interference are effectively overcome, and a polyp size measurement result has millimeter-level precision and good repeatability.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of medical image processing and endoscopic detection, and relates to an information acquisition method for polyps under an endoscope based on monocular vision. BACKGROUND

[0002] In clinical endoscopy, accurate acquisition of polyp-related information plays a key role in the risk assessment of malignant tumors. However, the measurement methods in current clinical practice have obvious limitations: on the one hand, they rely on the subjective visual estimation of doctors, and the measurement accuracy is significantly affected by the experience of the operator; on the other hand, they use postoperative ex vivo measurement, which cannot realize real-time guidance during surgery and will lead to systematical underestimation of the measurement results due to the shrinkage effect of tissue dehydration.

[0003] To solve the above problems, the prior art uses automatic segmentation technology in the field of computer vision to measure polyp-related information, and has made significant progress. However, when monocular cameras (single 2D images) are used for depth estimation, there is a fundamental "scale ambiguity" problem. Specifically, a very small object in an image may be a large object far away or a small object close by, and only pixel information cannot determine its absolute size. Moreover, when the real-world distance corresponding to a specific point in the image is known, this point can serve as an absolute "scale reference" for the entire depth map, and based on this known point, the absolute depth of other regions in the image can be calculated, rather than the relative distance. Once the accurate object distance (i.e. the distance between the polyp and the endoscope lens) is provided, the "scale ambiguity" problem is eliminated.

[0004] In addition, the object distance information can provide important prior knowledge for models based on deep learning (especially monocular depth estimation). For example, when it is used as an input feature, the object distance information can be input into the neural network as an additional input channel together with the image, guiding the network to learn the mapping from image features to real depth values faster and more accurately; as a supervision signal, the object distance information can be used to construct a more accurate loss function during the training of the model (such as ensuring that the predicted depth at the point with the known distance is highly consistent with the true value, thereby constraining the prediction results of the entire depth map); at the same time, with the object distance information, the model may not need to be trained on a super large-scale dataset covering all possible distances, but can perform well within a specific distance range.

[0005] However, the existing data sets all lack the measurement of the object distance information, which is directly caused by the deficiency of the existing measurement technology. On the one hand, the polyp segmentation data set (such as Kvasir-SEG) lacks synchronous depth information and physical size labeling; on the other hand, the 3D reconstruction data set (such as EndoMapper) lacks fine labeling of the lesion level. After in-depth investigation, many existing technical solutions cannot simultaneously obtain the geometric size and three-dimensional space information of the lesion, and thus it is difficult to construct a high-quality data set that can support accurate quantitative analysis.

[0006] This point is particularly prominent in existing patent technologies. For example, the patent application with the application publication number CN104146711A discloses a lesion size measurement method and system based on an endoscope. Although the method performs visual comparison by projecting an "optical ruler", the measurement principle has inherent defects: it relies heavily on the ideal assumption that "the lesion and the ruler are in the same plane", which is almost impossible to meet in actual clinical practice. Once there is a distance difference, the perspective effect will cause a huge error. Therefore, this method is not only unreliable and difficult to automate, but also cannot provide accurate size labeling with spatial context for the data set.

[0007] For another example, the patent application with the application publication number CN211270764U discloses an endoscopic lesion measuring ruler. The contact type mechanical measuring ruler used by the device has the limitation of the lack of information dimension. The device can only obtain the local two-dimensional size of the lesion, and cannot perceive the spatial distance of the lesion from the lens. Therefore, the measurement result provided by the device is "isolated" and is out of the three-dimensional spatial context, which is difficult to be used for calibrating the scale information in the two-dimensional image and cannot provide any help for constructing a data set containing the depth of the lesion. In addition, the contact operation itself also brings efficiency and safety risk problems.

[0008] In summary, the existing technology lacks an effective method for obtaining the distance between the polyp and the endoscope lens, and the lack of this object distance information seriously restricts the accurate acquisition of polyp-related information under the endoscope. Therefore, it is of great significance to propose a method for obtaining the distance between the polyp and the lens under the endoscope based on monocular vision to solve the above problems. SUMMARY

[0009] The purpose of the present application is to solve the problems existing in the prior art and provide a method for obtaining information of a polyp under an endoscope based on monocular vision.

[0010] To achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows:

[0011] A method for obtaining information of a polyp under an endoscope based on monocular vision, the information of the polyp including the distance between the polyp and the lens of the endoscope, the obtaining method comprising the following steps:

[0012] (1) obtaining an in-vivo endoscope image when the biopsy forceps are extended from the endoscope channel, the forceps body is straight, the forceps mouth is closed and the forceps mouth is against the polyp;

[0013] (2) converting the in-vivo endoscope image into a bird's eye view B;

[0014] (3) obtaining the pixel length of the forceps body in the bird's eye view B and bringing it into a mapping model to obtain the physical length of the forceps body in the bird's eye view B, and taking the sum of the physical length of the forceps head as the distance between the polyp and the lens of the endoscope ;

[0015] The construction process of the mapping model is: marking a scale on the forceps body of the biopsy forceps, obtaining an in-vivo endoscope image when the biopsy forceps are extended from the endoscope channel and the forceps body is straight, converting the in-vivo endoscope image into a bird's eye view A, obtaining the pixel length of the forceps body in the bird's eye view A and determining the physical length of the forceps body in the bird's eye view A according to the scale, and then fitting the pixel length of the forceps body in the bird's eye view A with the physical length of the forceps body in the bird's eye view A by a quadratic function, so as to obtain the mapping model.

[0016] As a preferred technical solution:

[0017] The specific steps of obtaining the information of the polyp under the endoscope based on monocular vision as described above are as follows:

[0018] (a) extending the biopsy forceps from the endoscope channel and keeping the forceps body straight, and recording the process by video;

[0019] (b) extracting m in-vivo endoscope images from the video in the order of time from front to back, m≥8;

[0020] (c) obtaining a homography matrix according to the mth in-vivo endoscope image;

[0021] (d) converting the 1st to (m-1)th in-vivo endoscope images into bird's eye views A by using the homography matrix;

[0022] (e) repeating steps (a) to (d) (n-1) times, a total of n×(m-1) bird's eye views A and n homography matrices are obtained, n≥13.

[0023] The process of step (c) in the method for obtaining the information of the polyp under the endoscope based on monocular vision is as follows:

[0024] First, the pixel coordinates of the four vertices of the forceps body are extracted from the mth in-vivo endoscope image;

[0025] Then a target rectangle is defined in the bird's eye view coordinate system, the physical size W, H of which is the physical size of the part of the pliers body being photographed, and the physical coordinates of the four vertices of the target rectangle are (0, 0), (W, 0), (W, H), (0, H) respectively;

[0026] Then the physical coordinates of the four vertices of the target rectangle are multiplied by the scale ratio coefficient to obtain the pixel coordinates of the four vertices of the target rectangle;

[0027] Further, the target rectangle is translated as a whole to be located at the central position of the output bird's eye view, to obtain the pixel coordinates of the four vertices of the updated target rectangle;

[0028] Finally, the pixel coordinates of the four vertices of the pliers body and the pixel coordinates of the four vertices of the updated target rectangle are directly linearly transformed by DLT algorithm to obtain the homography matrix.

[0029] The method for acquiring information of polyps under endoscopy based on monocular vision as described above converts the in-vivo endoscopic image into a bird's eye view B, and a generally applicable homography matrix is used. The steps for obtaining the generally applicable homography matrix are as follows:

[0030] (i) Collect n homography matrices, each of which has r elements, and r = 9;

[0031] (ii) For each homography matrix, the first (r-1) elements are sequentially combined to form an (r-1) dimensional feature vector, and n (r-1) dimensional feature vectors are obtained;

[0032] (iii) Cluster the n (r-1) dimensional feature vectors to find the largest and most concentrated cluster;

[0033] (iv) Determine whether more than 85% of the (r-1) dimensional feature vectors are located in the largest and most concentrated cluster. If yes, proceed to the next step; otherwise, re-collect n homography matrices;

[0034] (v) Calculate the median of all (r-1) dimensional feature vectors in the largest and most concentrated cluster dimension by dimension to obtain an (r-1) dimensional median vector;

[0035] (vi) Add an element 1 after the last 1 element of the (r-1) dimensional median vector to obtain an r dimensional feature vector;

[0036] (vii) Restore the r dimensional feature vector to matrix form to obtain the generally applicable homography matrix.

[0037] The method for acquiring information of a polyp under an endoscope based on monocular vision as described above, the information of the polyp further comprises a physical size of the polyp, and the method further comprises the following steps:

[0038] (4) After contouring the polyp area close to the jaw in the bird's eye view B, the minimum circumscribed rectangle of the contour is acquired, and the pixel size thereof is recorded ;

[0039] (5) The physical size of the polyp is calculated , and the calculation formula is as follows:

[0040] ;

[0041] In the formula, is the equivalent focal length (horizontal or vertical) of the lens of the endoscope.

[0042] The method for acquiring information of a polyp under an endoscope based on monocular vision as described above, the contouring of the polyp area close to the jaw in the bird's eye view B adopts a lightweight segmentation model (such as Segment Anything Tiny, i.e., "Segment Anything" small model).

[0043] The method for acquiring information of a polyp under an endoscope based on monocular vision as described above, The calculation formula is as follows:

[0044] ;

[0045] In the formula, is the pixel size (horizontal or vertical) of the imaging picture of the lens of the endoscope, is the field of view angle (horizontal or vertical), expressed by an angle number.

[0046] The method for acquiring information of a polyp under an endoscope based on monocular vision as described above, The calculation formula is as follows:

[0047] ;

[0048] In the formula, is the physical size (horizontal or vertical) of the imaging picture of the lens of the endoscope, is the focal length.

[0049] Beneficial effects:

[0050] (1) The present application constructs a standardized endoscopic image dataset that simultaneously covers accurate segmentation of polyp contours, physical size and depth information, effectively making up for the deficiencies of existing datasets (such as Kvasir-SEG, Endo Mapper) in lesion-level labeling and multi-modal information fusion. The dataset provides a reliable mapping relationship from pixels to millimeter-level size for deep learning models, significantly enhancing the training reliability and clinical applicability of the model in polyp quantitative analysis.

[0051] (2) The present application uses homography matrix and combines quadratic function fitting and biopsy forceps fixed structure features to establish a high-precision pixel-physical space mapping model, effectively overcoming measurement errors caused by image distortion, variable viewing angles and occlusion interference, so that the polyp size measurement results have millimeter-level precision and good repeatability.

[0052] (3) The present application integrates image acquisition, calibration processing, depth estimation and size conversion into a complete end-to-end measurement process, relying on a lightweight segmentation model and an automated script to extract polyp contours and calculate parameters, significantly reducing manual intervention and improving the efficiency and consistency of the diagnostic process, suitable for efficient operation requirements in actual clinical environments.

[0053] (4) The present application is simple to operate and does not require complex or expensive equipment, but relies only on conventional endoscopes and biopsy forceps to achieve accurate measurement, with strong hospital universality and scene adaptability; and the standardized polyp size data output by the system can provide objective and quantitative decision-making basis for doctors, helping to improve the standardization and intelligent level of polyp diagnosis and treatment under endoscopy. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A flowchart of the present application's method for acquiring information of polyps under endoscopy based on monocular vision;

[0055] Figure 2 An original image of an in-vitro endoscopic image in Example 1 of the present application;

[0056] Figure 3 A schematic diagram of the selection of the forceps body source points (4 red dots in the figure) of the in-vitro endoscopic image in Example 1 of the present application;

[0057] Figure 4 A schematic diagram of transforming the original image of the in-vitro endoscopic image into a bird's eye view A in Example 1 of the present application;

[0058] Figure 5 An original image of an in-vivo endoscopic image in Example 1 of the present application;

[0059] Figure 6 A schematic diagram of transforming the original image of the in-vivo endoscopic image into a bird's eye view B in Example 1 of the present application;

[0060] Figure 7 A polyp outline map obtained in Example 1 of the present application;

[0061] Figure 8 A polyp picture in Example 1 of the present application;

[0062] Figure 9 A schematic diagram of the proportional relationship of similar triangles in the small hole imaging of the present application. DETAILED DESCRIPTION

[0063] The present application will be further described in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application. Furthermore, it should be understood that those skilled in the art can make various modifications or changes to the present application after reading the content taught by the present application, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0064] A monocular vision-based information acquisition method for polyps under endoscopy, as shown in Figure 1 The information of the polyp includes the distance between the polyp and the lens of the endoscope and the physical size of the polyp, and the steps of the acquisition method are as follows:

[0065] (1) Preparation of experimental apparatus and tools;

[0066] Endoscope: medical endoscope;

[0067] Medical gauze;

[0068] Marker pen: chemical solvent-resistant oil-based marker pen, solvent wiping test is performed before use to verify its corrosion resistance, and to ensure that its markings will not be easily erased under the action of disinfectant solvents (such as alcohol or iodophor);

[0069] Millimeter scale caliper: precision is 1 millimeter, its scale line is clear, has no wear and tear, and has locking function to fix the measurement position;

[0070] (2) Mark the scale on the length direction of the biopsy forceps;

[0071] (2.1) Initial alignment: place the blue plastic forceps to be marked horizontally on a stable workbench, ensure that the surface is clean, dust-free and oil-free to prevent deviation or blurring during marking, then tightly align the zero scale end of the millimeter scale caliper with one end of the forceps, use a small clamp or heat-resistant tape to fix the other end of the caliper, ensure that there is no relative sliding between the caliper and the forceps, and optimize the alignment through visual inspection and slight adjustment to ensure the consistency of the subsequent measurement reference;

[0072] (2.2) Subsection marking: Starting from the beginning of the forceps body, draw a short line on the forceps body every 5 mm along the scale line of the caliper as a subsection mark. After completing each mark, use the caliper to recheck and ensure that the mark is located at the integer multiple of 5 mm. If there is a deviation, erase and redraw immediately to maintain high accuracy. Then move the caliper back 5 mm and repeat the drawing step until the entire length of the forceps body is covered. The selection of the mark interval is based on the actual clinical needs. The 5 mm subsection facilitates subsequent depth fitting without being too dense to cause confusion of the marks. When drawing, keep the marker perpendicular to the forceps body surface to ensure uniform line width and high straightness, avoiding deformation caused by inclination. During the movement of the caliper, operate slowly and confirm the fit multiple times to reduce cumulative errors.

[0073] (2.3) Mark verification: After all the marks are completed, conduct a comprehensive review. First, slide the caliper slowly along the forceps body and check the distance between each mark point to ensure that it is consistent with 5 mm. Record any potential uneven distribution. The comprehensive review process is repeated 2-3 times to enhance reliability and ensure that the ruler can provide accurate depth reference in both in-vivo and in-vitro environments.

[0074] (3) Constructing a mapping model and a universally applicable homography matrix;

[0075] (3.1) After marking the scale on the forceps body of the biopsy forceps, use medical gauze to gently wipe the surface of the endoscope lens in a circular motion to avoid straight scratches, maintain the optical performance of the lens, and ensure the removal of all water stains, stains, or fingerprints. Then check the lens by visual inspection or magnifying glass to confirm its cleanliness.

[0076] (3.2) Insert the prepared biopsy forceps ruler into the endoscope channel, extend it at a speed of 1-2 mm per second and keep the forceps body straight to avoid image shaking caused by rapid movement. At the same time, make sure it is in the center of the endoscope field of view. During the collection process, ensure that the scale of the ruler is clearly visible to maintain a clear recording state. Record the images of the ruler at different extension lengths through video to provide depth reference data for verifying the reliability of in-vitro calibration.

[0077] (3.2) Extract m in-vitro endoscopic images from the video in chronological order from front to back, m≥8;

[0078] (3.3) Obtain a homography matrix;

[0079] First, extract the pixel coordinates of the four vertices (source points) of the forceps body from the mth in-vitro endoscopic image;

[0080] Then a target rectangle is defined in the aerial view coordinate system, the physical size W, H of which is the physical size of the part of the forceps body being photographed, and the physical coordinates of the four vertices of the target rectangle are (0, 0), (W, 0), (W, H), (0, H) respectively;

[0081] Then the physical coordinates of the four vertices of the target rectangle are multiplied by the scale ratio coefficient to obtain the pixel coordinates of the four vertices of the target rectangle;

[0082] Further, the target rectangle is translated as a whole to be located at the central position of the output aerial view, and the pixel coordinates of the four vertices of the updated target rectangle are obtained;

[0083] Finally, the pixel coordinates of the four vertices of the forceps body are directly linearly transformed with the pixel coordinates of the four vertices of the updated target rectangle by the DLT algorithm to obtain the homography matrix;

[0084] The determination process of the scale ratio coefficient is as follows:

[0085] (A) Under the condition of selecting a specific endoscope model and setting a fixed working distance (the fixed working distance can be selected according to the scene requirement, and the purpose is to simulate the object distance range of the doctor during the in-vivo surgery), a photo of a standard chessboard calibration plate with precise known size is collected, and then an image processing algorithm is used to automatically extract the sub-pixel level coordinates of each inner corner point of the chessboard calibration plate, which are taken as the pixel coordinates of the feature points;

[0086] (B) A plurality of feature point pairs are selected, the pixel distance of each pair of points on the image plane is calculated, and the corresponding physical distance is determined, then the preliminary scale ratio coefficient of each feature point pair is calculated according to the formula "scale ratio coefficient (px / mm) = pixel distance (px) / physical distance (mm)", and finally, the arithmetic average operation is performed on all the preliminary scale ratio coefficients, and the result is the scale ratio coefficient;

[0087] (3.4) The first to (m-1) extracorporeal endoscope images are converted into aerial views A by using the homography matrix;

[0088] (3.5) Steps (3.2) to (3.4) are repeated (n-1) times, and n×(m-1) aerial views A and n homography matrices are obtained, n≥13;

[0089] (3.6) The pixel length of the biopsy forceps body in the aerial view A is obtained, and the physical length of the biopsy forceps body in the aerial view A is determined according to the scale, and then a quadratic function is used to fit the pixel length of the biopsy forceps body in the aerial view A and the physical length of the biopsy forceps body in the aerial view A, i.e. the mapping model is obtained; wherein the fitted quadratic function is:

[0090] ;

[0091] wherein, is the physical length of the biopsy forceps shaft, is the pixel length of the corresponding biopsy forceps shaft, a, b, c are fitting coefficients determined by fitting techniques such as least square method, , , the values of the fitting function can best fit the data;

[0092] (3.7) obtaining a generally applicable homography matrix;

[0093] (i) collecting n homography matrices obtained in step (3.5), each homography matrix has r elements, r = 9;

[0094] (ii) for each homography matrix, the first (r-1) elements are sequentially composed into an (r-1) dimensional eigenvector, and n (r-1) dimensional eigenvectors are obtained;

[0095] (iii) clustering the n (r-1) dimensional eigenvectors to find the largest and most concentrated cluster;

[0096] (iv) determining whether more than 85% of the (r-1) dimensional eigenvectors are located in the largest and most concentrated cluster, if so, proceed to the next step; otherwise, re-collect n homography matrices;

[0097] (v) calculating the median of all (r-1) dimensional eigenvectors in the largest and most concentrated cluster dimension by dimension to obtain an (r-1) dimensional median vector;

[0098] (vi) adding an element 1 after the last 1 element of the (r-1) dimensional median vector to obtain an r dimensional eigenvector;

[0099] (vii) restoring the r dimensional eigenvector to matrix form, i.e. obtaining a generally applicable homography matrix;

[0100] (4) obtaining an in-vivo endoscope image when the biopsy forceps is extended from the endoscope channel, the shaft is straight, the jaws are closed and the jaws are against the polyp;

[0101] (4.1) clean the endoscope lens: on the basis of meeting the strict hygiene standards of the hospital, use medical gauze to gently wipe the endoscope lens to ensure that the surface is free of any residues, the cleaning process should be carried out in a sterile environment, and ultraviolet light can be used for disinfection after wiping to prevent cross contamination;

[0102] (4.2) Stabilize the endoscope and accurately position the polyp: When the doctor operates the endoscope, the lens stability needs to be maintained, the image blur caused by the slow adjustment of the angle and position is reduced, and then the lens is gradually moved to the target polyp position to ensure that the polyp is centered and the view is directly opposite;

[0103] (4.3) biopsy forceps close to the polyp mark: when the closed jaw of the biopsy forceps is close to the polyp, maintain this action for 2-3 seconds, ensure that the image is stable, keep the biopsy forceps straight and slowly extend the forceps close to the polyp, when the jaw is against the polyp, maintain for another 2-3 seconds; during the operation, pay attention to the control force to avoid damaging the tissue, while ensuring that the jaw is clearly visible in the image;

[0104] (4.4) Data collection repetition: to reduce the single data deviation, repeat the operation of steps (4.1)~(4.3) for 3-5 times for a single polyp, obtain multiple sets of data, each repetition follows the standardized process to ensure the consistency of the action, and the data is recorded independently;

[0105] (5) Convert the in-vivo endoscope image to the bird's eye view B by using the generally applicable homography matrix;

[0106] (6) Obtain the pixel length of the biopsy forceps shaft in the bird's eye view B and bring it into the mapping model to obtain the physical length of the biopsy forceps shaft in the bird's eye view B, and the sum of the physical length of the biopsy forceps head is taken as the distance between the polyp and the lens of the endoscope ;

[0107] (7) Use a lightweight segmentation model to outline the polyp area close to the jaw in the bird's eye view B, then manually calibrate, and finally use the code to calculate the minimum bounding rectangle of the polyp contour and record its pixel size ; Wherein the lightweight segmentation model used is Segment Anything Tiny;

[0108] (8) Calculate the physical size of the polyp , the calculation formula is as follows:

[0109] ;

[0110] ;

[0111] (the principle of this formula is shown in Figure 9 );

[0112] In the formula, is the physical size (horizontal or vertical) of the imaging screen of the endoscope lens, is the focal length, is the pixel size (horizontal or vertical) of the imaging screen of the endoscope lens, The field of view (horizontal or vertical) is expressed in degrees. The equivalent focal length (horizontal or vertical) of the endoscope lens.

[0113] The dimensions in this invention refer to the length in the horizontal or vertical direction.

[0114] Example 1

[0115] The above-mentioned method for obtaining information about polyps under endoscopy based on monocular vision was used to obtain polyp information (e.g. Figure 8 The information shown is as follows:

[0116] The endoscope used in step (1) is an Olympus CF-H290I, and the horizontal physical length of its lens image is... It is 70mm, focal length The horizontal pixel length of the endoscope lens image is 14.3mm. The size is 1157px, the length of the forceps head is 11mm, and the physical distance between the two ends of the opening when the biopsy forceps are open is 5mm.

[0117] The extracorporeal endoscope images obtained in step (3.2) are as follows: Figure 2 As shown, m is 8;

[0118] In step (3.3), the selection of the clamp source point for the external endoscope image is as follows: Figure 3 As shown;

[0119] The converted bird's-eye view A in step (3.4) is as follows Figure 4 As shown;

[0120] In step (3.5), n is 13;

[0121] In the mapping model obtained by fitting in step (3.6), a is 0.02308, b is 3.27, and c is -0.001045;

[0122] The process of obtaining the universally applicable homography matrix in step (3.7) is as follows:

[0123] (i) Collect the 13 homography matrices obtained in step (3.5), each homography matrix having 9 elements;

[0124] (i) For each homography matrix, the first 8 elements are arranged in order to form an 8-dimensional eigenvector, resulting in a total of 13 8-dimensional eigenvectors; where the 8-dimensional eigenvector is represented as: v_i = [h11, h12, h13, h21, h22, h23, h31, h32];

[0125] (i.e., cluster the 13 8-dimensional feature vectors to find the largest and most concentrated cluster;

[0126] (iv) Determine whether more than 85% of the 8-dimensional eigenvectors are located in the largest and most concentrated cluster. If so, proceed to the next step; otherwise, recollect the 13 homography matrices.

[0127] (v) For all 8-dimensional feature vectors in the largest and most concentrated clusters, calculate the median dimension by dimension to obtain an 8-dimensional median vector; the specific process of calculating the median dimension by dimension is as follows: first, calculate the median of the first element of all 8-dimensional feature vectors in the largest and most concentrated clusters, and use it as the first element of the 8-dimensional median vector; then calculate the median of the second element of all 8-dimensional feature vectors in the largest and most concentrated clusters, and use it as the second element of the 8-dimensional median vector, and so on, to obtain the values ​​of each element in the 8-dimensional median vector, whose 8-dimensional median vector v_median = [ 11, 12, 13, twenty one, twenty two, twenty three, 31, 32];

[0128] (vi) After the last element of the 8-dimensional median vector, add an element 1 to obtain a 9-dimensional eigenvector, i.e., h_vector = [ 11, 12, 13, twenty one, twenty two, twenty three, 31, 32, 1];

[0129] (vii) Reconstructing the 9-dimensional eigenvectors into a 3×3 matrix form yields the universally applicable homography matrix; where the obtained universally applicable homography matrix... ;

[0130] The in vivo endoscopic images obtained in step (4) are as follows: Figure 5 As shown;

[0131] In step (5), a universally applicable homography matrix is ​​used to convert the in vivo endoscopic images into a format similar to... Figure 6 The bird's-eye view shown in Figure B;

[0132] In step (6), the pixel length of the biopsy forceps body in the bird's-eye view B is 11px, and its physical length is 3.288mm. The distance between the polyp and the endoscope lens is... It is 14.288mm;

[0133] The polyp outline obtained in step (7) is as follows Figure 7 As shown, its horizontal pixel length It is 99.66px;

[0134] The physical length of the polyp in the horizontal direction calculated in step (8) It is 6.02mm.

[0135] To verify the accuracy of the above-mentioned method for acquiring information about polyps under endoscopy based on monocular vision, the polyps were measured using the same type of endoscope and existing techniques. The specific process is as follows:

[0136] First, acquire clinical images of the biopsy forceps containing the polyp and in standard open position. Then, manually annotate the pixel distance between the two ends of the opening of the biopsy forceps when in standard open position. Then, combining the physical distance (5mm) between the two ends of the opening when the biopsy forceps are standardly open, the conversion factor of the current image is calculated. Then, based on this conversion factor, the pixel length in the horizontal direction of the polyp was... Converted to the physical length in the horizontal direction of the polyp The formula is as follows:

[0137] ;

[0138] .

[0139] The physical length of the polyp in the horizontal direction was calculated to be 6.15 mm using the above method. Comparing this with the result measured in Example 1, the error of the result measured in Example 1 is only 0.13 mm. This verifies that the polyp information measurement method based on monocular vision endoscopy of the present invention has higher accuracy and reliability, and can also provide a more accurate quantitative basis for clinical polyp assessment. At the same time, the output of the present invention simultaneously includes complete data containing the two-dimensional size of the polyp and its spatial distance from the endoscope lens. This directly makes up for the core deficiency of existing clinical datasets (such as Kvasir-SEG) which only have segmentation masks and lack depth information. It provides essential basic data with physical scale annotation for training depth estimation algorithms that can understand the geometric attributes of the endoscopic scene (such as monocular depth estimation and size-aware segmentation networks), thus laying a solid data foundation for finally realizing fully automatic and high-precision intraoperative measurement without relying on external equipment.

[0140] The difference between physical dimensions and actual dimensions in this invention is as follows:

[0141] Physical size: In scenarios involving the measurement and description of object dimensions, the emphasis is on indicating that the dimension value is measured in millimeters (mm). The focus is on the measurement standard, not whether the dimension is calculated or derived from actual measurement. The relative concept is pixel size. Physical length is similar to physical size.

[0142] Actual size: This focuses on describing the objective, true size of an object in the real world. It is not limited to a specific unit of measurement. The core is to represent the true size of the object as accurately as possible. It can be obtained through direct measurement or deduced from relevant information. Actual length is similar to actual size.

Claims

1. A method for acquiring information about polyps under endoscopy based on monocular vision, characterized in that, Information about the polyp includes the distance between the polyp and the endoscope lens, and the method for obtaining this information includes the following steps: (1) Obtain in vivo endoscopic images when the biopsy forceps extend out of the endoscope channel, the forceps body is straight, the forceps jaws are closed and the forceps jaws are against the polyp; (2) Convert the in vivo endoscopic images into a bird's-eye view B; (3) Obtain the pixel length of the biopsy forceps body in the bird's-eye view B and input it into the mapping model to obtain the physical length of the biopsy forceps body in the bird's-eye view B. Add this length to the physical length of the biopsy forceps head as the distance between the polyp and the endoscope lens. ; The mapping model is constructed as follows: After marking the scale along the length direction on the biopsy forceps, an external endoscope image is acquired when the biopsy forceps extend from the endoscope channel and the forceps body is straight. The external endoscope image is then converted into a bird's-eye view A. The pixel length of the biopsy forceps body in bird's-eye view A is then obtained. At the same time, the physical length of the biopsy forceps body in bird's-eye view A is determined according to the scale. Finally, a quadratic function is used to fit the pixel length of the biopsy forceps body in bird's-eye view A with the physical length of the biopsy forceps body in bird's-eye view A, thus obtaining the mapping model.

2. The method for acquiring information about polyps under endoscopy based on monocular vision according to claim 1, characterized in that, The specific steps for acquiring extracorporeal endoscopic images when the biopsy forceps are extended from the endoscope channel and the forceps are straight, and then converting the extracorporeal endoscopic images into bird's-eye view A, are as follows: (a) Extend the biopsy forceps out of the endoscope channel and keep the forceps straight, and record this process via video; (b) Extract m extracorporeal endoscope images from the video in chronological order from front to back, where m ≥ 8; (c) Obtain the homography matrix based on the m-th external endoscopic image; (d) Use the homography matrix to convert the first to (m-1) extracorporeal endoscope images into a bird's-eye view A; (e) Repeat steps (a) to (d) (n-1) times to obtain a total of n×(m-1) bird's-eye view images A and n homography matrices, where n≥13.

3. The method for acquiring information about polyps under endoscopy based on monocular vision according to claim 2, characterized in that, The process of step (c) is as follows: First, extract the pixel coordinates of the four vertices of the clamp from the m-th external endoscope image; Then, define a target rectangle in the bird's-eye view coordinate system. Its physical dimensions W and H are the physical dimensions of the part of the clamp body that is photographed. The physical coordinates of the four vertices of the target rectangle are (0,0), (W,0), (W,H), (0,H). Next, the physical coordinates of the four vertices of the target rectangle are multiplied by the scale factor to obtain the pixel coordinates of the four vertices of the target rectangle; Furthermore, the target rectangle is translated so that it is located in the center of the output bird's-eye view, thus obtaining the updated pixel coordinates of the four vertices of the target rectangle; Finally, the pixel coordinates of the four vertices of the clamp are directly linearly transformed with the pixel coordinates of the four vertices of the updated target rectangle using the DLT algorithm to calculate the homography matrix.

4. The method for acquiring information about polyps under endoscopy based on monocular vision according to claim 2, characterized in that, The intraoperative endoscopic image was converted into a bird's-eye view (B) using a universally applicable homography matrix. The steps for obtaining the universally applicable homography matrix are as follows: (i) Collect n homography matrices, each homography matrix having r elements, where r=9; (ii) For each homography matrix, the first (r-1) elements are arranged in order to form an (r-1)-dimensional eigenvector, resulting in a total of n (r-1)-dimensional eigenvectors; (iii) Cluster the n (r-1) dimensional feature vectors to find the largest and most concentrated cluster; (iv) Determine whether more than 85% of the (r-1) dimensional feature vectors are located in the largest and most concentrated cluster. If so, proceed to the next step. Conversely, if the result is not satisfactory, then collect n homography matrices again; (v) For all (r-1) dimensional eigenvectors in the largest and most concentrated cluster, calculate the median dimension by dimension to obtain an (r-1) dimensional median vector; (vi) After the last element of the (r-1)-dimensional median vector, add an element 1 to obtain an r-dimensional eigenvector; (vii) Restore the r-dimensional eigenvectors to matrix form to obtain the universally applicable homography matrix.

5. The method for acquiring information about polyps under endoscopy based on monocular vision according to claim 1, characterized in that, Information about the polyp also includes its physical size, and the method for obtaining this information includes the following steps: (4) After outlining the polyp area near the pincers in the bird's-eye view B, obtain the minimum bounding rectangle of the outline and record its pixel size. ; (5) Calculate the physical size of the polyp. The calculation formula is as follows: ； In the formula, This is the equivalent focal length of the endoscope lens.

6. The method for acquiring information about polyps under endoscopy based on monocular vision according to claim 5, characterized in that, A lightweight segmentation model was used to outline the polyp region near the pincers in the bird's-eye view B.

7. The method for acquiring information about polyps under endoscopy based on monocular vision according to claim 5, characterized in that, The calculation formula is as follows: ； In the formula, This refers to the pixel size of the image captured by the endoscope lens. The field of view is the angle of view.

8. The method for acquiring information about polyps under endoscopy based on monocular vision according to claim 7, characterized in that, The calculation formula is as follows: ； In the formula, The physical size of the image captured by the endoscope lens. It is the focal length.

Images