Auxiliary measurement method and device for instrument under endoscope

Abstract

The invention discloses an auxiliary measurement method and device for an instrument under an endoscope, and the method comprises the steps: obtaining a target endoscope image; identifying a tissue segmentation area on the target endoscopic image and an instrument segmentation area corresponding to the real instrument; determining instrument image size parameters of the real instrument on the target endoscopic image based on the instrument segmentation region; based on the instrument image size parameter, the tissue image size parameter and the instrument real size parameter of the real instrument, determining the tissue real size parameter of the tissue segmentation area; and displaying real size parameters of the tissue. The front end of the instrument serves as a reference surface, the instrument serves as a reference object, real size parameters of any tissue are measured in an auxiliary mode, and the accuracy of tissue size measurement under an endoscope can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an endoscopic instrument-assisted measurement method and device. Background Art

[0002] During gastrointestinal endoscopy, physicians often need to estimate the size of tissues (including human structures and lesions). In clinical practice, these size estimates are often highly subjective and subject to significant error, resulting in low accuracy in endoscopic tissue size measurements. Summary of the Invention

[0003] The embodiments of the present application provide an endoscopic instrument-assisted measurement method and device, which can improve the accuracy of endoscopic tissue size measurement.

[0004] In a first aspect, the present application provides an endoscopic instrument-assisted measurement method, comprising: acquiring a target endoscopic image; Identifying a tissue segmentation region on the target endoscopic image and an instrument segmentation region corresponding to a real instrument; determining an instrument image size parameter of the real instrument on the target endoscopic image based on the instrument segmentation region; determining a real tissue size parameter of the tissue segmentation region based on the instrument image size parameter, the tissue image size parameter, and the instrument real size parameter of the real instrument; Displays the true size parameters of the tissue.

[0005] Optionally, the end of the real instrument is a cylindrical end, and the real instrument size parameter of the real instrument is the diameter of the cylindrical end. The determining the instrument image size parameter of the real instrument in the target endoscopic image based on the instrument segmentation area includes: Performing polygon fitting on the segmented area of ​​the instrument to obtain an instrument polygon; selecting a first reference side and a second reference side from a plurality of sides of the instrument polygon; Determining a target angle bisector based on the first reference side and the second reference side, wherein the target angle bisector passes through a first intersection point, an angle between the target angle bisector and the first reference side is equal to an angle between the target angle bisector and the second reference side, the angle between the target angle bisector and the first reference side is less than 90 degrees, and the first intersection point is an intersection point of an extension line of the first reference side and an extension line of the second reference side; Determine, among the intersection points of the target angle bisector and the instrument polygon, the intersection point closest to the first intersection point as a second intersection point; Draw a straight line perpendicular to the target angle bisector through the second intersection point, intersecting the extended line of the first reference side and the extended line of the second reference side at a third intersection point and a fourth intersection point respectively; The straight-line distance between the third intersection point and the fourth intersection point is determined as an instrument image size parameter of the real instrument in the target endoscopic image.

[0006] Optionally, the determining the real tissue size parameter of the tissue segmentation region based on the instrument image size parameter, the tissue image size parameter and the real instrument size parameter of the real instrument includes: determining a ratio of the tissue image size parameter to the instrument image size parameter as an image size ratio; The product of the image size ratio and the instrument real size parameter is determined as the tissue real size parameter of the tissue segmentation region.

[0007] Optionally, selecting a first reference edge and a second reference edge from the plurality of edges of the instrument polygon includes: placing the plurality of edges of the instrument polygon into a first edge set; determining a second edge set based on the first edge set; determining a perpendicular distance between the center of the target endoscopic image and a straight line on which each edge in the second edge set lies as a first distance of each edge in the second edge set; Adding the edges in the second edge set whose first distance is less than the preset distance value into the third edge set; A first reference edge and a second reference edge are selected from the third edge set.

[0008] Optionally, selecting a first reference edge and a second reference edge from the third edge set includes: The longest side and the second longest side in the third side set are determined as the first reference side and the second reference side.

[0009] Optionally, determining the second edge set based on the first edge set includes: Select two edges from the first edge set and calculate a first angle between the two edges; If the first angle is smaller than a preset angle value, the shorter side of the two sides corresponding to the first angle is removed from the first side set to obtain the second side set.

[0010] Optionally, the identifying the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the real instrument includes: Inputting the target endoscopic image into an instrument segmentation neural network model to obtain a classification probability that each pixel point on the target endoscopic image belongs to an instrument category; Binarizing the target endoscopic image based on the classification probability that each pixel point on the target endoscopic image belongs to the instrument category to obtain an instrument binarization image; Converting the instrument binary image into an instrument grayscale image; Performing connected domain detection on the device grayscale image to obtain multiple device connected domains; The instrument connected domain with the largest area is determined as the instrument segmentation region on the target endoscopic image.

[0011] In a second aspect, the present application provides an endoscopic instrument auxiliary measurement device, comprising: an acquisition module, for acquiring a target endoscopic image; an identification module, configured to identify a tissue segmentation region on the target endoscopic image and an instrument segmentation region corresponding to a real instrument; A first determining module is configured to determine an instrument image size parameter of the real instrument on the target endoscopic image based on the instrument segmentation region; a second determining module, configured to determine a real tissue size parameter of the tissue segmentation region based on the instrument image size parameter, the tissue image size parameter, and the instrument real size parameter of the real instrument; The display module is used to display the actual size parameters of the tissue.

[0012] On the third aspect, the electronic device provided in this application includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the endoscopic instrument assisted measurement method provided in this application.

[0013] Fourthly, the computer-readable storage medium provided in the present application stores a plurality of instructions, which are suitable for loading by a processor to implement the steps in the endoscopic instrument-assisted measurement method provided in the present application.

[0014] In a fifth aspect, the computer program product provided in the present application includes a computer program or instructions, which, when executed by a processor, implements the steps in the endoscopic instrument-assisted measurement method provided in the present application.

[0015] Compared to related technologies, this application acquires a target endoscopic image; identifies a tissue segmentation region on the target endoscopic image and an instrument segmentation region corresponding to a real instrument; determines instrument image size parameters of the real instrument on the target endoscopic image based on the instrument segmentation region; determines the true tissue size parameters of the tissue segmentation region based on the instrument image size parameters, the tissue image size parameters, and the true instrument size parameters of the real instrument; and displays the true tissue size parameters. This application uses the front end of the instrument as a reference surface and the instrument as a reference object to assist in measuring the true size parameters of any tissue, thereby improving the accuracy of endoscopic tissue size measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 This is a schematic diagram of a scenario of an endoscopic instrument-assisted measurement system provided in an embodiment of the present application; Figure 2 This is a flow chart of an embodiment of the endoscopic instrument-assisted measurement method provided in an embodiment of the present application; Figure 3 is a schematic diagram of a target endoscopic image in one embodiment of the endoscopic instrument-assisted measurement method provided in an embodiment of the present application; Figure 4 is a schematic diagram of an instrument segmentation region on a target endoscopic image in one embodiment of the endoscopic instrument-assisted measurement method provided by an embodiment of the present application; Figure 5 Schematic diagram of an instrument polygon in one embodiment of the endoscopic instrument-assisted measurement method provided in an embodiment of the present application; Figure 6 Schematic diagram of a first reference edge and a second reference edge in an embodiment of the endoscopic instrument-assisted measurement method provided in an embodiment of the present application; Figure 7 In one embodiment of the endoscopic instrument-assisted measurement method provided in the present application, the target angle bisector s3 and the third intersection point and the fourth intersection Schematic diagram of; Figure 8 Schematic diagram of the structure of the endoscopic instrument auxiliary measurement device provided in an embodiment of the present application; Figure 9 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] It should be noted that the principles of this application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of this application and should not be considered as limiting other specific embodiments not described in detail herein.

[0019] In the following description of this application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict.

[0020] In the following description of this application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0022] To improve the effectiveness of endoscopic instrument-assisted measurement, embodiments of the present application provide an endoscopic instrument-assisted measurement method, an endoscopic instrument-assisted measurement device, an electronic device, a computer-readable storage medium, and a computer program product. The endoscopic instrument-assisted measurement method can be performed by the endoscopic instrument-assisted measurement device, or by an electronic device incorporating the endoscopic instrument-assisted measurement device.

[0023] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0024] Please refer to Figure 1 , this application also provides an endoscopic instrument auxiliary measurement system, such as Figure 1 As shown, the electronic device 100 of the endoscopic instrument auxiliary measurement system integrates the endoscopic instrument auxiliary measurement device provided by the present application.

[0025] Among them, the electronic device 100 can be any device equipped with a processor and having processing capabilities, such as mobile electronic devices with processors such as smart phones, tablet computers, PDAs, laptops, smart speakers, or fixed electronic devices with processors such as desktop computers, televisions, servers, industrial equipment, etc.

[0026] In addition, if Figure 1 As shown, the endoscopic instrument-assisted measurement system may further include a memory 200 for storing raw data, intermediate data, and result data.

[0027] In the embodiment of the present application, the memory 200 may be a cloud memory. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as the storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of different types of storage devices (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0028] Currently, storage systems utilize a storage method that creates logical volumes. During the creation of a logical volume, physical storage space is allocated for each logical volume. This physical storage space may consist of disks on a storage device or several storage devices. When a client stores data on a logical volume, it stores the data on a file system. The file system divides the data into multiple parts, each of which is an object. An object contains not only the data but also additional information such as the data identifier (ID). The file system writes each object to the physical storage space of the logical volume and records the storage location of each object. Therefore, when a client requests data access, the file system can provide access based on the storage location of each object.

[0029] The storage system allocates physical storage space to logical volumes by pre-dividing the physical storage space into stripes based on the estimated capacity of the objects to be stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the Redundant Array of Independent Disks (RAID) groupings. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0030] It should be noted that Figure 1 The scenario diagram of the endoscopic instrument assisted measurement system shown is only an example. The endoscopic instrument assisted measurement system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the endoscopic instrument assisted measurement system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0031] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0032] Please refer to Figure 2 , Figure 2FIG. 1 is a flow chart of an embodiment of the endoscopic instrument-assisted measurement method provided in the embodiment of the present application. Figure 2 As shown, the process of the endoscopic instrument-assisted measurement method provided in this application is as follows: 201. Acquire a target endoscopic image.

[0033] In the embodiment of the present application, the target endoscopic image can be a target endoscopic image taken by medical personnel using an endoscope. Figure 3 shown.

[0034] Specifically, the endoscope captures an endoscopic image at a preset frequency while moving, and the currently captured endoscopic image is determined as the target endoscopic image.

[0035] 202. Identify the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the real instrument.

[0036] In a specific embodiment, a pre-trained image segmentation model is obtained, and the target endoscopic image is input into the image segmentation model to obtain the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the real instrument. Specifically, the image segmentation model can be Unet / Unet++, etc., which can be set according to the specific situation, and this application is not limited to this. Specifically, a plurality of endoscopic image samples labeled with tissue segmentation regions and instrument segmentation regions are obtained, and the image segmentation model is trained based on the plurality of endoscopic image samples labeled with tissue segmentation regions and instrument segmentation regions to obtain a pre-trained image segmentation model.

[0037] In another specific embodiment, identifying the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the real instrument includes: (1) The target endoscopic image is input into the instrument segmentation neural network model to obtain the classification probability that each pixel point on the target endoscopic image belongs to the instrument category.

[0038] Among them, the classification category of the pixel points is instrument category or non-instrument category. Instrument segmentation neural network model It can be Unet / Unet++, etc., which can be set according to the specific situation. This application does not limit this.

[0039] Specifically, the classification probability of each pixel belonging to the device category is .

[0040] (2) Based on the classification probability that each pixel point on the target endoscopic image belongs to the instrument category, the target endoscopic image is binarized to obtain the instrument binarization image.

[0041] In the embodiment of the present application, if the classification probability If the probability threshold Thr is less than the threshold, the pixel value of the pixel on the device binary image is 0. If the probability threshold Thr is not less than the probability threshold, the pixel value of the pixel on the device binary image is 1. The probability threshold Thr can be 0.5 or other values, which can be set according to the specific situation.

[0042] Specifically, the pixel value of the pixel on the device binary image Satisfies the following formula, .

[0043] Among them, Thr=0.5.

[0044] (3) Convert the binary image of the instrument into a grayscale image of the instrument.

[0045] Grayscale value of pixel points on the instrument grayscale image Satisfies the following formula, .

[0046] (4) Perform connected domain detection on the device grayscale image to obtain multiple device connected domains.

[0047] Perform connected domain detection on the instrument grayscale image to obtain multiple instrument connected domains, which are then combined into an instrument connected domain set. .

[0048] (5) The device connected domain with the largest area is determined as the device segmentation region on the target endoscopic image.

[0049] There is only one device body at the same time, and the device connected domain set composed of multiple device connected domains is taken The largest connected domain of the instrument in the target endoscopic image is taken as the instrument segmentation region on the target endoscopic image. Specifically, the instrument segmentation region on the target endoscopic image is as follows: Figure 4 shown.

[0050] In another specific embodiment, identifying the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the real instrument includes: (1) The target endoscopic image is input into the tissue segmentation neural network model to obtain the classification probability of each pixel point on the target endoscopic image belonging to the tissue category.

[0051] Among them, the classification category of the pixel point is tissue category or non-tissue category. Tissue segmentation neural network model It can be Unet / Unet++, etc., which can be set according to the specific situation. This application does not limit this.

[0052] Specifically, the classification probability of each pixel belonging to the tissue category is .

[0053] (2) Based on the classification probability of each pixel point on the target endoscopic image belonging to the tissue category, the target endoscopic image is binarized to obtain a tissue binarization map.

[0054] In the embodiment of the present application, if the classification probability If the probability is less than the threshold Thr, the pixel value of the pixel on the tissue binary map is 0. If the probability threshold Thr is not less than the probability threshold Thr, the pixel value of the pixel on the tissue binary image is 1. The probability threshold Thr can be 0.5 or other values, which can be set according to specific circumstances.

[0055] Specifically, the pixel value of the pixel on the tissue binary map Satisfies the following formula, .

[0056] Among them, Thr=0.5.

[0057] (3) Convert the tissue binary image into a tissue grayscale image.

[0058] Grayscale value of pixel points on tissue grayscale image , satisfying the following formula, .

[0059] (4) Perform connected domain detection on the tissue grayscale image to obtain multiple tissue connected domains.

[0060] Perform connected domain detection on the tissue grayscale image to obtain multiple tissue connected domains, which are then combined into a connected domain set. .

[0061] (5) The tissue connected domain with the largest area is determined as the tissue segmentation region on the target endoscopic image.

[0062] 203. Determine the instrument image size parameters of the real instrument on the target endoscopic image based on the instrument segmentation region.

[0063] In the embodiment of the present application, the end of the real instrument is a cylindrical end, and the real instrument size parameter of the real instrument is the diameter of the cylindrical end. Determining the instrument image size parameter of the real instrument on the target endoscopic image based on the instrument segmentation area includes: (1) Perform polygon fitting on the segmented area of ​​the instrument to obtain the instrument polygon.

[0064] In the embodiments of this application, polygon approximation involves approximating a contour to a polygon consisting of straight line segments. Common polygon approximations include minimum enclosing rectangle, minimum enclosing circle, and least squares ellipse. Polygon approximation can be implemented using the cv2.approxPolyDP() function.

[0065] like Figure 5 As shown in the figure, polygon fitting is performed on the instrument segmentation area to obtain the instrument polygon. The approximate accuracy of the instrument polygon is the perimeter of the instrument polygon. times, in this case .

[0066] (2) Selecting a first reference edge and a second reference edge from the multiple edges of the instrument polygon.

[0067] In a specific embodiment, the longest side and the second longest side among the multiple sides of the instrument polygon are determined as the first reference side and the second reference side.

[0068] In another specific embodiment, selecting the first reference side and the second reference side from the plurality of sides of the instrument polygon comprises: 1-1. Put multiple edges of the instrument polygon into the first edge set.

[0069] 1-2. Determine a second edge set based on the first edge set.

[0070] In a specific embodiment, determining the second edge set based on the first edge set includes: removing edges in the first edge set whose length is less than a preset length value to obtain the second edge set. The preset length value may be 50 pixels.

[0071] In another specific embodiment, determining the second edge set based on the first edge set includes: selecting two edges from the first edge set and calculating a first angle between the two edges; if the first angle is less than a preset angle value, removing the shorter of the two edges corresponding to the first angle from the first edge set to obtain the second edge set. The preset angle value can be 5 degrees or other values, which can be set according to specific circumstances.

[0072] In another specific embodiment, determining the second edge set based on the first edge set includes: removing edges from the first edge set whose length is less than a preset length value, selecting two edges from the first edge set and calculating a first angle between the two edges; if the first angle is less than a preset angle value, removing the shorter of the two edges corresponding to the first angle from the first edge set to obtain the second edge set. The preset length value may be 50 pixels. The preset angle value may be 5 degrees or another value, which can be set according to specific circumstances.

[0073] 1-3. Determine the perpendicular distance between the center of the target endoscopic image and the straight line on which each edge in the second edge set lies as the first distance of each edge in the second edge set.

[0074] 1-4. Add the edges in the second edge set whose first distance is less than the preset distance value to the third edge set.

[0075] The preset distance is one-quarter the length of the short side of the target endoscopic image. Finally, interfering line segments that are not oriented toward the image center are eliminated. Because the instrument openings of the endoscope pipeline align with the lens orientation, the instruments in the image are radially oriented toward the image center. The vertical distance from the image center to each line segment is calculated, and those exceeding the preset distance are eliminated.

[0076] 1-5. Select the first reference edge and the second reference edge from the third edge set.

[0077] In the embodiment of the present application, selecting the first reference edge and the second reference edge from the third edge set includes: determining the longest edge and the second longest edge in the third edge set as the first reference edge and the second reference edge.

[0078] like Figure 6 As shown, the longest side and the second longest side in the third side set are determined as the first reference side s1 and the second reference side s2.

[0079] (3) Determine the target angle bisector based on the first reference side and the second reference side.

[0080] Among them, the target angle bisector passes through the first intersection point, the angle between the target angle bisector and the first reference side is equal to the angle between the target angle bisector and the second reference side, the angle between the target angle bisector and the first reference side is less than 90 degrees, and the first intersection point is the intersection point of the extension line of the first reference side and the extension line of the second reference side.

[0081] like Figure 7 As shown, the first reference side s1 and the second reference side s2 are extended so that they intersect at the first intersection point . Passing the first intersection Calculate the angle bisector of the first reference side s1 and the second reference side s2. Calculate the slope of the first reference side s1 and the second reference side s2 、 , then the slope of the angle bisector is m, .

[0082] From the above formula, we know that there are two angle bisectors. We need to choose the one with the smaller angle with the other two lines. The angle between the target angle bisector s3 and the first reference side is less than 90 degrees. Make the target angle bisector s3, which is the center line of the instrument. Figure 7 shown.

[0083] (4) The intersection point between the target angle bisector and the instrument polygon that is closest to the first intersection point is determined as the second intersection point.

[0084] like Figure 7 As shown, the intersection point set of the target angle bisector s3 and the instrument polygon is , which is closest to the first intersection The intersection point is the second intersection point .

[0085] (5) Draw a straight line perpendicular to the target angle bisector through the second intersection point, which intersects with the extension lines of the first reference side and the second reference side at the third and fourth intersection points respectively.

[0086] like Figure 7 As shown, through the second intersection Draw a perpendicular line to the target angle bisector s3 , respectively with the first reference edge The extension line and the second reference side The extension line intersects at the third intersection point and the fourth intersection .

[0087] (6) The straight-line distance between the third intersection point and the fourth intersection point is determined as the instrument image size parameter of the real instrument on the target endoscopic image.

[0088] In a specific embodiment, determining the instrument image size parameters of the real instrument on the target endoscopic image based on the instrument segmentation area includes: inputting the target endoscopic image into a pre-trained instrument-touched tissue classification model Classification is performed to obtain a contact classification result, wherein the contact classification result is a contact type and a non-contact type. When the contact classification result is a contact type, an instrument image size parameter of the real instrument on the target endoscopic image is determined based on the instrument segmentation area.

[0089] In another specific embodiment, the instrument image size parameters of the real instrument on the target endoscopic image are determined based on the instrument segmentation area, before which the following steps are included: inputting the target endoscopic image into a pre-trained instrument-touched tissue classification model. Classification is performed to obtain a contact classification result, wherein the contact classification result is a contact type and a non-contact type. If the contact classification result is a contact type, a first reference endoscopic image captured earlier than the target endoscopic image is obtained, the target endoscopic image and the first reference endoscopic image are aligned, and the instrument segmentation region on the target endoscopic image is overlaid on the first reference endoscopic image to obtain a second reference endoscopic image, wherein the instrument segmentation region on the target endoscopic image is located in the same position on the target endoscopic image and the second reference endoscopic image. The first image similarity between the second reference endoscopic image and the target endoscopic image is calculated. If the first image similarity is higher than a first preset similarity, it is determined that the instrument has touched the surface of the tissue. At this time, it can be determined that the end of the real instrument has touched the surface of the tissue. The size parameters of the end of the real instrument can be used to calculate the size parameters of the tissue. The instrument image size parameters of the real instrument in the target endoscopic image are determined based on the instrument segmentation region.

[0090] In another specific embodiment, the instrument image size parameters of the real instrument on the target endoscopic image are determined based on the instrument segmentation area, before which the method includes: inputting the target endoscopic image into a pre-trained instrument-touched tissue classification model. Classification is performed to obtain a contact classification result, wherein the contact classification result is a contact type and a non-contact type. If the contact classification result is a contact type, a first reference endoscopic image whose shooting time is earlier than the target endoscopic image is obtained. The target endoscopic image is divided into four image segmentation areas by the extension line of the first reference side and the extension line of the second reference side, and the area of ​​the image segmentation area where the instrument segmentation area is located, which is located outside the instrument segmentation area and close to the first intersection, is determined as a comparison area. The target endoscopic image and the first reference endoscopic image are aligned, and the second image similarity between the image of the target endoscopic image located in the comparison area and the image of the first reference endoscopic image located in the comparison area is calculated. If the second image similarity is higher than the second preset similarity, it is determined that the instrument touches the surface of the tissue. At this time, it can be determined that the end of the real instrument touches the surface of the tissue. The size parameters of the end of the real instrument can be used to calculate the size parameters of the tissue. The instrument image size parameters of the real instrument on the target endoscopic image are determined based on the instrument segmentation area.

[0091] Furthermore, the first intersection , the third intersection and the fourth intersection The enclosed area serves as the comparison area.

[0092] 204. Determine the actual tissue size parameter of the tissue segmentation region based on the instrument image size parameter, the tissue image size parameter, and the actual instrument size parameter of the actual instrument.

[0093] In an embodiment of the present application, the actual tissue size parameter of the tissue segmentation area is determined based on the instrument image size parameter, the tissue image size parameter, and the actual instrument size parameter of the actual instrument, including: determining the ratio of the tissue image size parameter to the instrument image size parameter as the image size ratio; and determining the product of the image size ratio and the actual instrument size parameter as the actual tissue size parameter of the tissue segmentation area.

[0094] In a specific embodiment, the major diameter and minor diameter of the minimum circumscribed rectangle of each tissue segmentation region are obtained, and the major diameter and minor diameter of the minimum circumscribed rectangle of the tissue segmentation region are determined as tissue image size parameters.

[0095] Specifically, the true size parameter of the tissue is , the instrument image size parameter is , the tissue image size parameter is , the actual size parameter of the device is d, and the actual size parameter of the tissue is Satisfies the following formula, .

[0096] The actual size parameter d of the instrument is the diameter of the cylindrical end, and in this example, d = 2.3 mm.

[0097] 205. Display the true size parameters of the tissue.

[0098] In a specific embodiment, the true size parameters of the tissue are the true long diameter and the true short diameter of each tissue, and the length values ​​of the true long diameter and the true short diameter of each tissue are displayed on each tissue segmentation area.

[0099] In another specific embodiment, a determination is made as to whether a historical endoscopic measurement image exists, wherein the historical endoscopic measurement image is an endoscopic image recorded with true tissue size parameters obtained by performing endoscopic instrument-assisted measurement on an endoscopic image taken before the target endoscopic image. If the historical endoscopic measurement image exists, a determination is made as to whether the angle between the target angle bisector in the historical endoscopic measurement image and the target endoscopic image exceeds a set angle. If the angle between the target angle bisector in the historical endoscopic measurement image and the target endoscopic image exceeds the set angle, each tissue segmentation region in the historical endoscopic measurement image is obtained, and each tissue segmentation region in the historical endoscopic measurement image is matched one-to-one with each tissue segmentation region in the target endoscopic image to obtain multiple tissue segmentation region groups, each tissue segmentation region group including two tissue segmentation regions matched between the historical endoscopic measurement image and the target endoscopic image. The true tissue size parameters of the two tissue segmentation regions in the tissue segmentation region group are averaged to obtain an average true tissue size parameter, and the average true tissue size parameter is displayed as the true tissue size parameter.

[0100] Specifically, each tissue segmentation region on the historical measurement endoscopic image is matched with each tissue segmentation region on the target endoscopic image one by one to obtain a plurality of tissue segmentation region groups. This includes calculating segmentation region similarities between each tissue segmentation region on the historical measurement endoscopic image and each tissue segmentation region on the target endoscopic image, and determining another tissue segmentation region and the tissue segmentation region having the highest segmentation region similarity with the tissue segmentation region as the tissue segmentation region group.

[0101] Furthermore, an average distance value of the distance between the tissue segmentation region and the third intersection point and the distance between the tissue segmentation region and the fourth intersection point is obtained. Regional weight coefficients of the two tissue segmentation regions in the tissue segmentation region group are determined based on the corresponding average distance values ​​of the two tissue segmentation regions in the tissue segmentation region group, wherein the larger the average distance value, the smaller the regional weight coefficient. Based on the regional weight coefficients of the two tissue segmentation regions in the tissue segmentation region group, a weighted sum of the tissue true size parameters of the two tissue segmentation regions in the tissue segmentation region group is performed to obtain a weighted tissue true size parameter, and the weighted tissue true size parameter is used as the tissue true size parameter.

[0102] To facilitate better implementation of the endoscopic instrument-assisted measurement method provided in the embodiments of this application, the embodiments of this application also provide an endoscopic instrument-assisted measurement device based on the above endoscopic instrument-assisted measurement method. The meanings of the terms herein are the same as those in the above endoscopic instrument-assisted measurement method. For specific implementation details, please refer to the description in the above method embodiments.

[0103] Please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of an endoscopic instrument auxiliary measurement device provided in an embodiment of the present application. The endoscopic instrument auxiliary measurement device may include: An acquisition module 701 is used to acquire a target endoscopic image; Identification module 702, for identifying the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the real instrument; A first determining module 703 is configured to determine an instrument image size parameter of a real instrument on a target endoscopic image based on the instrument segmentation region; A second determining module 704 is configured to determine a real tissue size parameter of the tissue segmentation region based on the device image size parameter, the tissue image size parameter, and the device real size parameter of the real device; The display module 705 is used to display the actual size parameters of the tissue.

[0104] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described again here.

[0105] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps of the endoscopic instrument assisted measurement method provided in this embodiment by calling a computer program stored in the memory.

[0106] Please refer to Figure 9 , Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0107] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not limit the electronic device and may include more or fewer components than shown, or combine certain components, or arrange the components differently. Among them: Processor 101 is the control center of the electronic device. It connects all parts of the electronic device using various interfaces and circuits. It executes software programs and / or modules stored in memory 102 and accesses data stored in memory 102 to perform various functions of the electronic device and process data. Optionally, processor 101 may include one or more processing cores. Alternatively, processor 101 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 101.

[0108] Memory 102 can be used to store software programs and modules. Processor 101 executes various functional applications and data processing by running the software programs and modules stored in memory 102. Memory 102 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as sound playback or image playback); the data storage area may store data generated based on the use of the electronic device. Furthermore, memory 102 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory 102 may also include a memory controller to provide processor 101 with access to memory 102.

[0109] The electronic device also includes a power supply 103 for supplying power to various components. Optionally, the power supply 103 can be logically connected to the processor 101 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 103 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0110] The electronic device may further include an input unit 104, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0111] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps of the endoscopic instrument assisted measurement method provided in this application, such as: Acquire a target endoscopic image; identify a tissue segmentation region on the target endoscopic image and an instrument segmentation region corresponding to a real instrument; determine instrument image size parameters of the real instrument on the target endoscopic image based on the instrument segmentation region; determine tissue real size parameters of the tissue segmentation region based on the instrument image size parameters, the tissue image size parameters, and the instrument real size parameters of the real instrument; and display the tissue real size parameters.

[0112] It should be noted that the electronic device provided in the embodiment of the present application and the endoscopic instrument-assisted measurement method in the above embodiment belong to the same concept. The specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0113] This application also provides a computer-readable storage medium having a computer program stored thereon. When the stored computer program is executed on a processor of an electronic device provided in an embodiment of this application, the processor of the electronic device executes the steps of the endoscopic instrument-assisted measurement method provided in this application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0114] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform various optional implementations of the above-mentioned endoscopic instrument-assisted measurement method.

[0115] The above is a detailed introduction to an endoscopic instrument-assisted measurement method and device provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, based on the idea of ​​the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0116] It should be noted that when the above embodiments of this application are applied to specific products or technologies, the relevant user data is involved, and the user's permission or consent must be obtained, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. An endoscopic instrument-assisted measurement method, characterized in that: The endoscopic instrument-assisted measurement method comprises: acquiring a target endoscopic image; Identifying a tissue segmentation region on the target endoscopic image and an instrument segmentation region corresponding to a real instrument; determining an instrument image size parameter of the real instrument on the target endoscopic image based on the instrument segmentation region; determining a real tissue size parameter of the tissue segmentation region based on the instrument image size parameter, the tissue image size parameter, and the instrument real size parameter of the real instrument; Displays the true size parameters of the tissue.

2. The endoscopic instrument-assisted measurement method according to claim 1, characterized in that: The end of the real instrument is a cylindrical end, and the real size parameter of the real instrument is the diameter of the cylindrical end. The determining of the instrument image size parameter of the real instrument on the target endoscopic image based on the instrument segmentation area includes: Performing polygon fitting on the segmented area of ​​the instrument to obtain an instrument polygon; selecting a first reference side and a second reference side from a plurality of sides of the instrument polygon; Determining a target angle bisector based on the first reference side and the second reference side, wherein the target angle bisector passes through a first intersection point, an angle between the target angle bisector and the first reference side is equal to an angle between the target angle bisector and the second reference side, the angle between the target angle bisector and the first reference side is less than 90 degrees, and the first intersection point is an intersection point of an extension line of the first reference side and an extension line of the second reference side; Determine, among the intersection points of the target angle bisector and the instrument polygon, the intersection point closest to the first intersection point as a second intersection point; Draw a straight line perpendicular to the target angle bisector through the second intersection point, intersecting the extended line of the first reference side and the extended line of the second reference side at a third intersection point and a fourth intersection point respectively; The straight-line distance between the third intersection point and the fourth intersection point is determined as an instrument image size parameter of the real instrument in the target endoscopic image.

3. The endoscopic instrument-assisted measurement method according to claim 2, characterized in that: The determining of the real tissue size parameter of the tissue segmentation region based on the instrument image size parameter, the tissue image size parameter, and the real instrument size parameter of the real instrument comprises: determining a ratio of the tissue image size parameter to the instrument image size parameter as an image size ratio; The product of the image size ratio and the instrument real size parameter is determined as the tissue real size parameter of the tissue segmentation region.

4. The endoscopic instrument-assisted measurement method according to claim 3, characterized in that: The selecting the first reference side and the second reference side from the plurality of sides of the instrument polygon comprises: placing the plurality of edges of the instrument polygon into a first edge set; determining a second edge set based on the first edge set; determining a perpendicular distance between the center of the target endoscopic image and a straight line on which each edge in the second edge set lies as a first distance of each edge in the second edge set; Adding the edges in the second edge set whose first distance is less than the preset distance value into the third edge set; A first reference edge and a second reference edge are selected from the third edge set.

5. The endoscopic instrument-assisted measurement method according to claim 4, characterized in that: The selecting the first reference edge and the second reference edge from the third edge set includes: The longest side and the second longest side in the third side set are determined as the first reference side and the second reference side.

6. The endoscopic instrument-assisted measurement method according to claim 4, characterized in that: The determining the second edge set based on the first edge set includes: Select two edges from the first edge set and calculate a first angle between the two edges; If the first angle is smaller than a preset angle value, the shorter side of the two sides corresponding to the first angle is removed from the first side set to obtain the second side set.

7. The endoscopic instrument-assisted measurement method according to claim 1, characterized in that: The identifying of the tissue segmentation region on the target endoscopic image and the instrument segmentation region corresponding to the real instrument includes: Inputting the target endoscopic image into an instrument segmentation neural network model to obtain a classification probability that each pixel point on the target endoscopic image belongs to an instrument category; Binarizing the target endoscopic image based on the classification probability that each pixel point on the target endoscopic image belongs to the instrument category to obtain an instrument binarization image; Converting the instrument binary image into an instrument grayscale image; Performing connected domain detection on the device grayscale image to obtain multiple device connected domains; The instrument connected domain with the largest area is determined as the instrument segmentation region on the target endoscopic image.

8. An endoscopic instrument auxiliary measurement device, characterized in that: include: an acquisition module, for acquiring a target endoscopic image; an identification module, configured to identify a tissue segmentation region on the target endoscopic image and an instrument segmentation region corresponding to a real instrument; A first determining module is configured to determine an instrument image size parameter of the real instrument on the target endoscopic image based on the instrument segmentation region; a second determining module, configured to determine a real tissue size parameter of the tissue segmentation region based on the instrument image size parameter, the tissue image size parameter, and the instrument real size parameter of the real instrument; The display module is used to display the actual size parameters of the tissue.

9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the endoscopic instrument-assisted measurement method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute the steps of the endoscopic instrument-assisted measurement method according to any one of claims 1 to 7.

Images