Endoscopic Instrument-Assisted Measurement Methods and Devices

Abstract

This application discloses a method and apparatus for endoscopic instrument-assisted measurement, comprising: acquiring a target endoscopic image; identifying tissue segmentation regions on the target endoscopic image and corresponding instrument segmentation regions on the actual instrument; determining the instrument image size parameters of the actual instrument on the target endoscopic image based on the instrument segmentation regions; determining the actual tissue size parameters of the tissue segmentation regions based on the instrument image size parameters, tissue image size parameters, and the actual instrument size parameters; and displaying the actual tissue size parameters. This application uses the instrument tip as a reference surface and the instrument as a reference object to assist in measuring the actual size parameters of any tissue, thereby improving the accuracy of endoscopic tissue size measurement.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to an endoscopic instrument-assisted measurement method and device. Background Technology

[0002] During gastrointestinal endoscopy, doctors often need to estimate the size of various tissues (including human structures and lesions). In clinical practice, doctors often rely heavily on visual estimation, which is highly subjective and prone to significant errors. This results in low accuracy of endoscopic tissue size measurements. Summary of the Invention

[0003] This application provides an endoscopic instrument-assisted measurement method and device, which can improve the accuracy of endoscopic tissue size measurement.

[0004] Firstly, the endoscopic instrument-assisted measurement method provided in this application includes:

[0005] Acquire target endoscopic images;

[0006] Identify the tissue segmentation region on the target endoscope image and the instrument segmentation region corresponding to the actual instrument;

[0007] Based on the instrument segmentation region, determine the instrument image size parameters of the real instrument on the target endoscope image;

[0008] The actual tissue size parameters of the tissue segmentation region are determined based on the instrument image size parameters, tissue image size parameters, and the actual instrument size parameters.

[0009] Displays the actual size parameters of the tissue.

[0010] Optionally, the end of the real instrument is a cylindrical end, and the actual size parameter of the real instrument is the diameter of the cylindrical end. Determining the instrument image size parameter of the real instrument on the target endoscope image based on the instrument segmentation region includes:

[0011] Polygon fitting is performed on the segmented region of the instrument to obtain the instrument polygon;

[0012] Select a first reference edge and a second reference edge from the multiple edges of the polygon of the instrument;

[0013] The target angle bisector is determined based on the first reference side and the second reference side, wherein the target angle bisector passes through the first intersection point, the angle between the target angle bisector and the first reference side is equal to the angle between the target angle bisector and the second reference side, the angle between the target angle bisector and the first reference side is less than 90 degrees, and the first intersection point is the intersection point of the extension line of the first reference side and the extension line of the second reference side.

[0014] The intersection point between the bisector of the target angle and the polygon of the instrument is determined as the second intersection point, which is the closest to the first intersection point.

[0015] Draw a straight line perpendicular to the target angle bisector through the second intersection point, which intersects the extensions of the first reference side and the second reference side at the third and fourth intersection points, respectively.

[0016] The straight-line distance between the third intersection point and the fourth intersection point is determined as the instrument image size parameter of the real instrument on the target endoscope image.

[0017] Optionally, determining the actual tissue size parameters of the tissue segmentation region based on the instrument image size parameters, tissue image size parameters, and the actual instrument size parameters includes:

[0018] The ratio of the tissue image size parameter to the instrument image size parameter is determined as the image size ratio.

[0019] The product of the image size ratio and the instrument's true size parameter is determined as the tissue's true size parameter for the tissue segmentation region.

[0020] Optionally, selecting the first reference edge and the second reference edge from the multiple edges of the device polygon includes:

[0021] Place the multiple sides of the device polygon into the first side set;

[0022] The second set of sides is determined based on the first set of sides;

[0023] The perpendicular distance between the center of the target endoscopic image and the straight line containing each side in the second side set is determined as the first distance of each side in the second side set;

[0024] Add the edges in the second edge set whose first distance is less than a preset distance value to the third edge set;

[0025] Select a first reference edge and a second reference edge from the third set of edges.

[0026] Optionally, selecting the first reference edge and the second reference edge from the third edge set includes:

[0027] The longest and second longest edges in the third set of edges are determined as the first reference edge and the second reference edge.

[0028] Optionally, determining the second edge set based on the first edge set includes:

[0029] Select two edges from the first set of edges and calculate the first included angle between the two edges;

[0030] If the first included angle is less than a preset angle value, then the shorter side of the two sides corresponding to the first included angle is removed from the first side set to obtain the second side set.

[0031] Optionally, identifying the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the actual instrument includes:

[0032] The target endoscope image is input into the instrument segmentation neural network model to obtain the classification probability of each pixel in the target endoscope image belonging to the instrument category;

[0033] The target endoscope image is binarized based on the classification probability of each pixel in the target endoscope image belonging to the instrument category, resulting in an instrument binarized image.

[0034] Convert the binarized image of the instrument into a grayscale image of the instrument;

[0035] Connectivity detection is performed on the grayscale image of the device to obtain multiple connected components of the device;

[0036] The instrument connected region with the largest area is defined as the instrument segmentation region on the target endoscope image.

[0037] Secondly, the endoscopic instrument-assisted measuring device provided in this application includes:

[0038] The acquisition module is used to acquire images of the target endoscope.

[0039] The identification module is used to identify the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the real instrument;

[0040] The first determining module is used to determine the instrument image size parameters of the real instrument on the target endoscope image based on the instrument segmentation region;

[0041] The second determining module is used to determine the actual tissue size parameters of the tissue segmentation region based on the instrument image size parameters, the tissue image size parameters, and the actual instrument size parameters of the actual instrument.

[0042] The display module is used to display the actual size parameters of the tissue.

[0043] Thirdly, the electronic device provided in this application includes a memory and a processor. The memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the endoscopic instrument-assisted measurement method provided in this application.

[0044] Fourthly, the computer-readable storage medium provided in this application stores multiple instructions that are adapted for loading by a processor to implement the steps in the endoscopic instrument-assisted measurement method provided in this application.

[0045] Fifthly, the computer program product provided in this application includes a computer program or instructions that, when executed by a processor, implement the steps in the endoscopic instrument-assisted measurement method provided in this application.

[0046] In this application, compared to related technologies, the following steps are taken: acquiring a target endoscopic image; identifying tissue segmentation regions on the target endoscopic image and corresponding instrument segmentation regions on the actual instrument; determining the instrument image size parameters of the actual instrument on the target endoscopic image based on the instrument segmentation regions; determining the actual tissue size parameters of the tissue segmentation regions based on the instrument image size parameters, tissue image size parameters, and the actual instrument size parameters; and displaying the actual tissue size parameters. This application uses the instrument tip as a reference surface and the instrument as a reference object to assist in measuring the actual size parameters of any tissue, thereby improving the accuracy of endoscopic tissue size measurement. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram of a scenario for the endoscopic instrument-assisted measurement system provided in an embodiment of this application;

[0049] Figure 2 This is a flowchart illustrating one embodiment of the endoscopic instrument-assisted measurement method provided in this application.

[0050] Figure 3 This is a schematic diagram of a target endoscope image in one embodiment of the endoscopic instrument-assisted measurement method provided in this application.

[0051] Figure 4 This is a schematic diagram of the instrument segmentation region on the target endoscope image in one embodiment of the endoscopic instrument-assisted measurement method provided in this application;

[0052] Figure 5 This is a schematic diagram of the instrument polygon in one embodiment of the endoscopic instrument-assisted measurement method provided in this application;

[0053] Figure 6 This is a schematic diagram of the first reference side and the second reference side in one embodiment of the endoscopic instrument-assisted measurement method provided in this application;

[0054] Figure 7 In one embodiment of the endoscopic instrument-assisted measurement method provided in this application, the target angle bisector s3 and the third intersection point are described. and the fourth intersection point A schematic diagram;

[0055] Figure 8 This is a schematic diagram of the endoscopic instrument auxiliary measurement device provided in the embodiments of this application;

[0056] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0057] It should be noted that the principles of this application are illustrated by example in a suitable computing environment. The following description is based on the specific embodiments of this application that are illustrated, and should not be regarded as limiting other specific embodiments not detailed herein.

[0058] In the following description of this application, "some embodiments" are referred to, which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments, and may be combined with each other without conflict.

[0059] In the following description of this application, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0061] To improve the effectiveness of endoscopic instrument-assisted measurements, this application provides an endoscopic instrument-assisted measurement method, an endoscopic instrument-assisted measurement device, an electronic device, a computer-readable storage medium, and a computer program product. The endoscopic instrument-assisted measurement method can be executed by the endoscopic instrument-assisted measurement device or by an electronic device integrating the endoscopic instrument-assisted measurement device.

[0062] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] Please refer to Figure 1 This application also provides an endoscopic instrument-assisted measurement system, such as... Figure 1 As shown, the electronic device 100 of the endoscopic instrument auxiliary measurement system integrates the endoscopic instrument auxiliary measurement device provided in this application.

[0064] Among them, electronic device 100 can be any device equipped with a processor and having processing capabilities, such as mobile electronic devices with processors such as smartphones, tablets, PDAs, laptops, and smart speakers, or fixed electronic devices with processors such as desktop computers, televisions, servers, and industrial equipment.

[0065] In addition, such as Figure 1 As shown, the endoscopic instrument-assisted measurement system may also include a memory 200 for storing raw data, intermediate data, and result data.

[0066] In this embodiment, the memory 200 can be a cloud memory. Cloud storage is a new concept that is extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology and distributed storage file system functions to bring together a large number of storage devices of various types in the network (storage devices are also called storage nodes) through application software or application interfaces to work together and jointly provide data storage and business access functions to the outside world.

[0067] Currently, the storage method in storage systems is as follows: Logical volumes are created, and during creation, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0068] The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0069] It should be noted that, Figure 1 The schematic diagram of the endoscopic instrument-assisted measurement system shown is merely an example. The endoscopic instrument-assisted measurement system and scenario described in this application are for the purpose of more clearly illustrating the technical solutions of this application and do not constitute a limitation on the technical solutions provided in this application. As those skilled in the art will know, with the evolution of endoscopic instrument-assisted measurement systems and the emergence of new business scenarios, the technical solutions provided in this application are also applicable to similar technical problems.

[0070] The following sections provide detailed descriptions of each example. It should be noted that the sequence numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0071] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating one embodiment of the endoscopic instrument-assisted measurement method provided in this application. Figure 2 As shown, the procedure for the endoscopic instrument-assisted measurement method provided in this application is as follows:

[0072] 201. Obtain the target endoscope image.

[0073] In this embodiment of the application, the target endoscopic image can be a target endoscopic image obtained by medical personnel using an endoscope. The target endoscopic image is as follows: Figure 3 shown.

[0074] Specifically, the endoscope captures images at a preset frequency as it moves, and the currently captured endoscope image is designated as the target endoscope image.

[0075] 202. Identify the tissue segmentation region on the target endoscopic image and the corresponding instrument segmentation region of the actual instrument.

[0076] In one specific embodiment, a pre-trained image segmentation model is obtained. The target endoscopic image is input into the image segmentation model to obtain tissue segmentation regions and instrument segmentation regions corresponding to the actual instruments on the target endoscopic image. Specifically, the image segmentation model can be Unet / Unet++, etc., and can be set according to specific circumstances; this application does not limit it. Specifically, multiple endoscopic image samples labeled with tissue segmentation regions and instrument segmentation regions are obtained, and the image segmentation model is trained based on the multiple endoscopic image samples labeled with tissue segmentation regions and instrument segmentation regions to obtain the pre-trained image segmentation model.

[0077] In another specific embodiment, identifying the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the actual instrument includes:

[0078] (1) Input the target endoscope image into the instrument segmentation neural network model to obtain the classification probability of each pixel in the target endoscope image belonging to the instrument category.

[0079] The pixels are classified into either medical device category or non-medical device category. Medical device segmentation neural network model. It can be Unet / Unet++, etc., and can be set according to the specific situation. This application does not limit it.

[0080] Specifically, the classification probability of each pixel belonging to the instrument category is: .

[0081] (2) Binarize the target endoscope image based on the classification probability of each pixel in the target endoscope image belonging to the instrument category to obtain the instrument binarized image.

[0082] In this embodiment of the application, if the classification probability If the value is less than the probability threshold Thr, then the pixel value of the pixel on the instrument binarized image is 0. If the classification probability is... If the value is not less than the probability threshold Thr, then the pixel value of the pixel on the instrument binarized image is 1. The probability threshold Thr can be 0.5 or other values, which can be set according to the specific situation.

[0083] Specifically, the pixel value of a pixel on the binarized image of the device. The following formula is satisfied.

[0084] .

[0085] Where Thr=0.5.

[0086] (3) Convert the binary image of the instrument into a grayscale image of the instrument.

[0087] grayscale values ​​of pixels in a grayscale image of an instrument The following formula is satisfied.

[0088] .

[0089] (4) Perform connected component detection on the grayscale image of the instrument to obtain multiple connected components of the instrument.

[0090] Connectivity detection is performed on the grayscale image of the device to obtain multiple connected components, which are then combined into a set of connected components for the device. .

[0091] (5) The instrument connected region with the largest area is determined as the instrument segmentation region on the target endoscope image.

[0092] There is only one instrument body at any given time; the set of connected components of multiple instruments is taken. The largest connected component in the target endoscope image is used as the instrument segmentation region. Specifically, the instrument segmentation region in the target endoscope image is as follows: Figure 4 shown.

[0093] In another specific embodiment, identifying the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the actual instrument includes:

[0094] (1) Input the target endoscopic image into the tissue segmentation neural network model to obtain the classification probability of each pixel in the target endoscopic image belonging to the tissue category.

[0095] The pixels are classified into tissue category or non-tissue category. (Tissue segmentation neural network model) It can be Unet / Unet++, etc., and can be set according to the specific situation. This application does not limit it.

[0096] Specifically, the classification probability of each pixel belonging to an organization category is: .

[0097] (2) Binarize the target endoscope image based on the classification probability of each pixel in the target endoscope image belonging to the tissue category to obtain the tissue binarized image.

[0098] In this embodiment of the application, if the classification probability If the value is less than the probability threshold Thr, then the pixel value on the tissue binarization map is 0. If the classification probability... If the value is not less than the probability threshold Thr, then the pixel value of the pixel on the tissue binarization map is 1. The probability threshold Thr can be 0.5 or other values, which can be set according to the specific situation.

[0099] Specifically, the pixel value of a pixel in the organization binarized image. The following formula is satisfied.

[0100] .

[0101] Where Thr=0.5.

[0102] (3) Convert the binarized tissue image into a grayscale tissue image.

[0103] grayscale values ​​of pixels in a grayscale image It satisfies the following formula.

[0104] .

[0105] (4) Perform connected component detection on the grayscale image of the tissue to obtain multiple connected components of the tissue.

[0106] Connectivity detection is performed on the grayscale image of the tissue to obtain multiple tissue connected components, which are then combined into a connected component set. .

[0107] (5) The tissue connected region with the largest area is determined as the tissue segmentation region on the target endoscopic image.

[0108] 203. Determine the instrument image size parameters of the real instrument on the target endoscope image based on the instrument segmentation region.

[0109] In this embodiment, the end of the real instrument is a cylindrical end, and the actual size parameter of the real instrument is the diameter of the cylindrical end. Determining the instrument image size parameter of the real instrument on the target endoscope image based on the instrument segmentation region includes:

[0110] (1) Perform polygon fitting on the segmented region of the instrument to obtain the instrument polygon.

[0111] In this embodiment, polygon fitting (Approximating Polygons) approximates a contour into a polygon composed of line segments. Common examples include minimum bounding rectangles, minimum bounding circles, and least squares ellipses. Polygon fitting can be achieved using the cv2.approxPolyDP() function.

[0112] like Figure 5 As shown, polygon fitting is performed on the segmented region of the instrument to obtain the instrument polygon. The approximate accuracy of the instrument polygon is equal to the perimeter of the instrument polygon. In this example, times, .

[0113] (2) Select the first reference edge and the second reference edge from the multiple edges of the device polygon.

[0114] In one specific embodiment, the longest and second longest sides among the multiple sides of the device polygon are determined as the first reference side and the second reference side.

[0115] In another specific embodiment, selecting a first reference edge and a second reference edge from multiple edges of the instrument polygon includes:

[0116] 1-1. Place the multiple sides of the device polygon into the first side set.

[0117] 1-2. Determine the second side set based on the first side set.

[0118] In one specific embodiment, determining the second edge set based on the first edge set includes: removing edges in the first edge set whose length is less than a preset length value to obtain the second edge set. The preset length value can be 50 pixels.

[0119] In another specific embodiment, determining the second side set based on the first side set includes: selecting two sides from the first side set and calculating a first included angle between the two sides; if the first included angle is less than a preset angle value, then removing the shorter side of the two sides corresponding to the first included angle from the first side set to obtain the second side set. The preset angle value can be 5 degrees or other values, set according to the specific circumstances.

[0120] In another specific embodiment, determining the second side set based on the first side set includes: removing sides in the first side set whose length is less than a preset length value; selecting two sides from the first side set and calculating a first included angle between the two sides; if the first included angle is less than a preset angle value, removing the shorter side of the two sides corresponding to the first included angle from the first side set to obtain the second side set. The preset length value can be 50 pixels. The preset angle value can be 5 degrees or other values, set according to specific circumstances.

[0121] 1-3. The perpendicular distance between the center of the target endoscopic image and the straight line containing each side in the second side set is determined as the first distance of each side in the second side set.

[0122] 1-4. Move the edges in the second edge set whose first distance is less than the preset distance value to the third edge set.

[0123] The preset distance value is one-quarter of the length of the shorter side of the target endoscopic image. Finally, interfering line segments that do not point towards the image center are eliminated. Since the instrument openings of the endoscope tubing are aligned with the lens direction, the instruments on the image point radially towards the image center. The perpendicular distance from the image center point to each line segment is calculated, and line segments exceeding the preset distance value are excluded.

[0124] 1-5. Select the first reference edge and the second reference edge from the third edge set.

[0125] In this embodiment of the application, selecting a first reference edge and a second reference edge from the third edge set includes: determining the longest edge and the second longest edge in the third edge set as the first reference edge and the second reference edge.

[0126] like Figure 6 As shown, the longest and second longest edges in the third edge set are determined as the first reference edge s1 and the second reference edge s2.

[0127] (3) Determine the target angle bisector based on the first reference side and the second reference side.

[0128] The target angle bisector passes through the first intersection point. The angle between the target angle bisector and the first reference side is equal to the angle between the target angle bisector and the second reference side. The angle between the target angle bisector and the first reference side is less than 90 degrees. The first intersection point is the intersection of the extension of the first reference side and the extension of the second reference side.

[0129] like Figure 7 As shown, extend the first reference edge s1 and the second reference edge s2 so that they intersect at the first intersection point. Passing through the first intersection point Calculate the angle bisectors of the first reference side s1 and the second reference side s2. Calculate the slopes of the first reference side s1 and the second reference side s2. , Then the slope of the angle bisector is m.

[0130] .

[0131] From the above formula, we know that there are two angle bisectors. We need to choose the one with the smaller angle to the other two lines. The angle between the target angle bisector s3 and the first reference side is less than 90 degrees. (Passing through the first intersection point...) Draw the bisector s3 of the target angle, which is the instrument centerline. Figure 7 shown.

[0132] (4) Among the intersections of the target angle bisector and the instrument polygon, the intersection point closest to the first intersection point is determined as the second intersection point.

[0133] like Figure 7 As shown, the set of intersection points of the target angle bisector s3 and the instrument polygon is... The one closest to the first intersection point The intersection point is the second intersection point. .

[0134] (5) Draw a straight line perpendicular to the bisector of the target angle through the second intersection point, which intersects the extension of the first reference side and the extension of the second reference side at the third and fourth intersection points respectively.

[0135] like Figure 7 As shown, through the second intersection point Draw the perpendicular line to the bisector s3 of the target angle. , respectively with the first reference edge extension of the second reference edge The extensions of the lines intersect at the third intersection point. and the fourth intersection point .

[0136] (6) The straight-line distance between the third and fourth intersection points is determined as the instrument image size parameter of the real instrument on the target endoscope image.

[0137] In one specific embodiment, before determining the instrument image size parameters of the real instrument on the target endoscopic image based on the instrument segmentation region, the method includes: inputting the target endoscopic image into a pre-trained instrument-tissue-touching classification model. The system performs classification to obtain contact classification results, which are categorized as contact type and non-contact type. When the contact classification result is a contact type, the instrument image size parameters of the actual instrument on the target endoscope image are determined based on the instrument segmentation region.

[0138] In another specific embodiment, before determining the instrument image size parameters of the real instrument on the target endoscopic image based on the instrument segmentation region, the method includes: inputting the target endoscopic image into a pre-trained instrument-tissue-touching classification model. The process involves classification to obtain contact classification results, which are categorized as contact type and non-contact type. If the contact classification result is a contact type, a first reference endoscopic image, captured earlier than the target endoscopic image, is acquired. The target endoscopic image and the first reference endoscopic image are aligned, and the instrument segmentation region from the target endoscopic image is overlaid on the first reference endoscopic image to obtain a second reference endoscopic image. The instrument segmentation region from the target endoscopic image is positioned identically on both the target and second reference endoscopic images. A first image similarity score is calculated between the second reference endoscopic image and the target endoscopic image. If the first image similarity score is higher than a first preset similarity score, it is determined that the instrument has touched the surface of the tissue. This indicates that the end of the actual instrument has touched the surface of the tissue. The size parameters of the tissue can be calculated using the size parameters of the end of the actual instrument. Therefore, the instrument image size parameters of the actual instrument on the target endoscopic image are determined based on the instrument segmentation region.

[0139] In yet another specific embodiment, before determining the instrument image size parameters of the real instrument on the target endoscopic image based on the instrument segmentation region, the method includes: inputting the target endoscopic image into a pre-trained instrument-tissue-touching classification model. The system performs classification to obtain contact classification results, which are categorized as contact type and non-contact type. If the contact classification result is contact type, a first reference endoscopic image captured earlier than the target endoscopic image is acquired. The target endoscopic image is divided into four image segmentation regions by the extension lines of the first and second reference edges. The region within the image segmentation region containing the instrument segmentation region, located outside the instrument segmentation region and close to the first intersection point, is defined as the comparison region. The target endoscopic image and the first reference endoscopic image are aligned, and the second image similarity is calculated between the image within the comparison region of the target endoscopic image and the image within the comparison region of the first reference endoscopic image. If the second image similarity is higher than a second preset similarity, it is determined that the instrument has touched the surface of the tissue. This indicates that the end of the actual instrument has touched the surface of the tissue. The size parameters of the tissue can be calculated using the size parameters of the end of the actual instrument. Therefore, the instrument image size parameters of the actual instrument on the target endoscopic image are determined based on the instrument segmentation region.

[0140] Furthermore, the first intersection point Third intersection point and the fourth intersection point The enclosed area serves as the comparison area.

[0141] 204. Determine the true tissue size parameters of the tissue segmentation region based on the instrument image size parameters, tissue image size parameters, and the true instrument size parameters.

[0142] In this embodiment of the application, determining the actual tissue size parameters of the tissue segmentation region based on the instrument image size parameters, tissue image size parameters, and the actual instrument size parameters includes: determining the ratio of the tissue image size parameters to the instrument image size parameters as an image size ratio; and determining the product of the image size ratio and the actual instrument size parameters as the actual tissue size parameters of the tissue segmentation region.

[0143] In one specific embodiment, the major and minor axes of the minimum bounding rectangle of each tissue segmentation region are obtained, and the major and minor axes of the minimum bounding rectangle of the tissue segmentation region are determined as tissue image size parameters.

[0144] Specifically, the actual size parameters of the organization are: The instrument image size parameters are The tissue image size parameters are The actual size parameter of the instrument is d, and the actual size parameter of the tissue is... The following formula is satisfied.

[0145] .

[0146] Wherein, the actual size parameter d of the instrument is the diameter of the end of the cylinder, and in this example, d = 2.3 mm.

[0147] 205. Display the actual size parameters of the tissue.

[0148] In one specific embodiment, the true size parameters of the tissues are the true major and minor diameters of each tissue, and the length values ​​of the true major and minor diameters of each tissue are displayed on the segmented regions of each tissue.

[0149] In another specific embodiment, it is determined whether a historical measurement endoscopic image exists. This historical measurement endoscopic image is an endoscopic image obtained by endoscopic instrument-assisted measurement of a previously captured endoscopic image, recording the true tissue size parameters. If a historical measurement endoscopic image exists, it is determined whether the angle between the target angle bisector in the historical measurement endoscopic image and the target angle bisector in the target endoscopic image exceeds a set angle. If the angle exceeds the set angle, each tissue segmentation region on the historical measurement endoscopic image is acquired. Each tissue segmentation region on the historical measurement endoscopic image is matched one-to-one with each tissue segmentation region on the target endoscopic image to obtain multiple tissue segmentation region groups. Each tissue segmentation region group includes two tissue segmentation regions matched between the historical measurement endoscopic image and the target endoscopic image. The average true tissue size parameters of the two tissue segmentation regions in the tissue segmentation region group are calculated to obtain the average true tissue size parameter, which is then displayed as the true tissue size parameter.

[0150] Specifically, each tissue segmentation region on the historical endoscopic images is matched one by one with each tissue segmentation region on the target endoscopic image to obtain multiple tissue segmentation region groups. This includes: calculating the segmentation region similarity between each tissue segmentation region on the historical endoscopic images and each tissue segmentation region on the target endoscopic image, and determining the tissue segmentation region group with the highest segmentation region similarity to the target tissue segmentation region.

[0151] Furthermore, the average distances between the tissue segmentation regions and the third intersection point, and between the tissue segmentation regions and the fourth intersection point, are obtained. Based on the average distances between two tissue segmentation regions in the group, a regional weight coefficient is determined for each region; a larger average distance corresponds to a smaller regional weight coefficient. The actual tissue size parameters of the two tissue segmentation regions in the group are then weighted and summed based on their regional weight coefficients to obtain a weighted actual tissue size parameter, which is used as the actual tissue size parameter.

[0152] To facilitate better implementation of the endoscopic instrument-assisted measurement method provided in the embodiments of this application, the embodiments of this application also provide an endoscopic instrument-assisted measurement device based on the above-described endoscopic instrument-assisted measurement method. The meanings of the terms used are the same as in the above-described endoscopic instrument-assisted measurement method; for specific implementation details, please refer to the descriptions in the above method embodiments.

[0153] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the endoscopic instrument auxiliary measurement device provided in an embodiment of this application. The endoscopic instrument auxiliary measurement device may include:

[0154] Acquisition module 701 is used to acquire target endoscopic images;

[0155] The recognition module 702 is used to identify the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the real instrument;

[0156] The first determining module 703 is used to determine the instrument image size parameters of the real instrument on the target endoscope image based on the instrument segmentation region;

[0157] The second determining module 704 is used to determine the actual tissue size parameters of the tissue segmentation region based on the instrument image size parameters, the tissue image size parameters, and the actual instrument size parameters.

[0158] Display module 705 is used to display the actual size parameters of the tissue.

[0159] For details on the implementation of each of the above modules, please refer to the previous examples, which will not be repeated here.

[0160] This application also provides an electronic device, including a memory and a processor, wherein the processor executes the steps in the endoscopic instrument-assisted measurement method provided in this embodiment by calling a computer program stored in the memory.

[0161] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0162] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will understand that the electronic device structure shown in the figures does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0163] The processor 101 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 102, and calls data stored in the memory 102, to perform various functions and process data. Optionally, the processor 101 may include one or more processing cores; alternatively, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 101.

[0164] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0165] The electronic device also includes a power supply 103 that supplies power to the various components. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 103 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0166] The electronic device may also include an input unit 104, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0167] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 101 in the electronic device loads one or more executable codes corresponding to computer programs into the memory 102 according to the following instructions, and the processor 101 executes the steps in the endoscopic instrument-assisted measurement method provided in this application, such as:

[0168] Acquire the target endoscopic image; identify the tissue segmentation region on the target endoscopic image and the corresponding instrument segmentation region of the real instrument; determine the instrument image size parameters of the real instrument on the target endoscopic image based on the instrument segmentation region; determine the tissue real size parameters of the tissue segmentation region based on the instrument image size parameters, tissue image size parameters, and the real instrument instrument's actual size parameters; display the tissue real size parameters.

[0169] It should be noted that the electronic device provided in this application embodiment and the endoscopic instrument-assisted measurement method in the above embodiment belong to the same concept. The specific implementation process can be found in the above related embodiments, and will not be repeated here.

[0170] This application also provides a computer-readable storage medium storing a computer program thereon. When the computer program stored thereon is executed on the processor of the electronic device provided in the embodiments of this application, the processor of the electronic device performs the steps in the endoscopic instrument-assisted measurement method provided in this application. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0171] This application also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform various optional implementations of the above-described endoscopic instrument-assisted measurement method.

[0172] The above provides a detailed description of an endoscopic instrument-assisted measurement method and device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

[0173] It should be noted that when the above embodiments of this application are applied to specific products or technologies, user-related data is involved, and user permission or consent is required. Furthermore, the collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

Claims

1. An endoscopic instrument-assisted measurement method, characterized in that, The endoscopic instrument-assisted measurement method includes: Acquire target endoscopic images; Identify the tissue segmentation region on the target endoscope image and the instrument segmentation region corresponding to the actual instrument; Based on the instrument segmentation region, the instrument image size parameters of the real instrument on the target endoscope image are determined. The end of the real instrument is a cylindrical end, and the actual instrument size parameter is the diameter of the cylindrical end. Polygon fitting is performed on the instrument segmentation region to obtain an instrument polygon. A first reference edge and a second reference edge are selected from the multiple edges of the instrument polygon. A target angle bisector is determined based on the first reference edge and the second reference edge, wherein the target angle bisector passes through a first intersection point, and the angle between the target angle bisector and the first reference edge is equal to the angle between the target angle bisector and the second reference edge. The angle between the target angle bisector and the first reference side is less than 90 degrees. The first intersection point is the intersection of the extension of the first reference side and the extension of the second reference side. Among the intersection points of the target angle bisector and the instrument polygon, the intersection point closest to the first intersection point is determined as the second intersection point. A straight line perpendicular to the target angle bisector is drawn through the second intersection point, intersecting the extension of the first reference side and the extension of the second reference side at a third intersection point and a fourth intersection point, respectively. The straight-line distance between the third intersection point and the fourth intersection point is determined as the instrument image size parameter of the actual instrument on the target endoscope image. The actual tissue size parameters of the tissue segmentation region are determined based on the instrument image size parameters, tissue image size parameters, and the actual instrument size parameters. Displays the actual size parameters of the tissue.

2. The endoscopic instrument-assisted measurement method according to claim 1, characterized in that, Determining the true tissue size parameters of the tissue segmentation region based on the instrument image size parameters, tissue image size parameters, and the true instrument size parameters includes: The ratio of the tissue image size parameter to the instrument image size parameter is determined as the image size ratio. The product of the image size ratio and the instrument's true size parameter is determined as the tissue's true size parameter for the tissue segmentation region.

3. The endoscopic instrument-assisted measurement method according to claim 2, characterized in that, The step of selecting the first reference edge and the second reference edge from the multiple edges of the device polygon includes: Place the multiple sides of the device polygon into the first side set; The second set of sides is determined based on the first set of sides; The perpendicular distance between the center of the target endoscopic image and the straight line containing each side in the second side set is determined as the first distance of each side in the second side set; Add the edges in the second edge set whose first distance is less than a preset distance value to the third edge set; Select a first reference edge and a second reference edge from the third set of edges.

4. The endoscopic instrument-assisted measurement method according to claim 3, characterized in that, The step of selecting the first reference edge and the second reference edge from the third edge set includes: The longest and second longest edges in the third set of edges are determined as the first reference edge and the second reference edge.

5. The endoscopic instrument-assisted measurement method according to claim 3, characterized in that, The determination of the second edge set based on the first edge set includes: Select two edges from the first set of edges and calculate the first included angle between the two edges; If the first included angle is less than a preset angle value, then the shorter side of the two sides corresponding to the first included angle is removed from the first side set to obtain the second side set.

6. The endoscopic instrument-assisted measurement method according to claim 1, characterized in that, The process of identifying the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the actual instrument includes: The target endoscope image is input into the instrument segmentation neural network model to obtain the classification probability of each pixel in the target endoscope image belonging to the instrument category; The target endoscope image is binarized based on the classification probability of each pixel in the target endoscope image belonging to the instrument category, resulting in an instrument binarized image. Convert the binarized image of the instrument into a grayscale image of the instrument; Connectivity detection is performed on the grayscale image of the device to obtain multiple connected components of the device; The instrument connected region with the largest area is defined as the instrument segmentation region on the target endoscope image.

7. An endoscopic instrument auxiliary measuring device, characterized in that, include: The acquisition module is used to acquire images of the target endoscope. The identification module is used to identify the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the real instrument; A first determining module is used to determine the instrument image size parameters of the real instrument on the target endoscope image based on the instrument segmentation region, wherein the end of the real instrument is a cylindrical end, and the actual instrument size parameter of the real instrument is the diameter of the cylindrical end. The module performs polygon fitting on the instrument segmentation region to obtain an instrument polygon; selects a first reference edge and a second reference edge from the multiple edges of the instrument polygon; and determines a target angle bisector based on the first reference edge and the second reference edge, wherein the target angle bisector passes through a first intersection point, and the angle between the target angle bisector and the first reference edge is equal to the angle between the target angle bisector and the first reference edge. The included angle between the target angle bisector and the first reference side is less than 90 degrees. The first intersection point is the intersection of the extensions of the first and second reference sides. Among the intersection points of the target angle bisector and the instrument polygon, the intersection point closest to the first intersection point is determined as the second intersection point. A straight line perpendicular to the target angle bisector is drawn through the second intersection point, intersecting the extensions of the first and second reference sides at a third and a fourth intersection point, respectively. The straight-line distance between the third and fourth intersection points is determined as the instrument image size parameter of the actual instrument on the target endoscope image. The second determining module is used to determine the actual tissue size parameters of the tissue segmentation region based on the instrument image size parameters, the tissue image size parameters, and the actual instrument size parameters of the actual instrument. The display module is used to display the actual size parameters of the tissue.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor running the computer program in the memory to perform the steps of the endoscopic instrument-assisted measurement method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the endoscopic instrument-assisted measurement method according to any one of claims 1 to 6.

Images