Method and device for determining direction of tail end point of sheathing canal, processor and electronic equipment

Abstract

The invention discloses a method and a device for determining the direction of a tail end point of a sheathing canal, a processor and electronic equipment. The method comprises the steps that initial two-dimensional images of an endoscope in the process of stretching into a physiological channel are obtained from at least two angles, and the endoscope comprises a sheath tube and an endoscope body; respectively segmenting a first image region containing an endoscope main body and a second image region containing a sheath tube from the initial two-dimensional image; based on the first image area, determining a first target position of an endoscope tail end point of the endoscope main body in a three-dimensional space, and based on the second image area, determining a second target position of a sheathing canal tail end point of the sheathing canal in the three-dimensional space; and based on the first target position and the second target position, determining azimuth information between the tail end point of the sheathing canal and the reference point. The technical problem that the accuracy of determining the direction of the tail end point of the sheathing canal is low is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of endoscope inspection, and in particular to a method, device, processor and electronic device for determining the position of a sheath end point. Background Art

[0002] At present, with the continuous development of endoscopic examination technology, in the process of inserting the endoscope into the physiological channel for endoscopic examination and treatment, the high-definition imaging and flexible operation of the endoscope can help doctors accurately diagnose and treat the lesion point (that is, the reference point). Computed tomography (CT) can quickly determine the lesion point and determine the planned path to reach the lesion point to improve the accuracy and safety of diagnosis and treatment. The lesion point can also be mapped to the C-arm (C_Arm / CArm) perspective view to accurately guide the diagnosis and treatment in real time.

[0003] In the related art, during the diagnosis and treatment process according to the planned path, the CArm perspectives at different angles can be used to determine whether the end point of the sheath reaches the lesion after the physiological channel. However, there are some problems with only confirming through two-dimensional CArm perspectives at different angles. First, the CArm perspective is a two-dimensional image, which is difficult to provide accurate three-dimensional spatial position information. Judging only based on the CArm perspective can easily cause positioning inaccuracy and errors. Second, when the end point of the sheath does not reach the lesion and needs to be adjusted, the CArm perspective is difficult to provide enough information for precise adjustment. Since the CArm perspective cannot display depth and three-dimensional information, multiple adjustments may be required to accurately guide the end point of the sheath to the target position, which increases the difficulty and risk of diagnostic and treatment operations. Therefore, there is still a technical problem of low accuracy in determining the orientation of the end point of the sheath.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, processor and electronic device for determining the position of a sheath end point, so as to at least solve the technical problem of low accuracy in determining the position of the sheath end point.

[0006] According to one aspect of an embodiment of the present invention, a method for determining the orientation of a sheath end point is provided, comprising: acquiring an initial two-dimensional image of an endoscope during extension into a physiological channel from at least two angles, wherein the endoscope comprises a sheath and an endoscope body; respectively segmenting a first image region including the endoscope body and a second image region including the sheath from the initial two-dimensional image; determining a first target position of an end point of the endoscope of the endoscope body in three-dimensional space based on the first image region, and determining a second target position of an end point of the sheath in three-dimensional space based on the second image region; determining orientation information between the end point of the sheath and a reference point based on the first target position and the second target position, wherein the orientation information is used to represent the distance and angle between the end point of the sheath and the reference point, and the reference point is used to represent a position where an abnormal state exists outside the physiological channel.

[0007] Optionally, a first image area including an endoscope body and a second image area including a sheath are respectively segmented from the initial two-dimensional image, including: performing threshold segmentation on any initial two-dimensional image to obtain a first image area, wherein the first image area includes each pixel point whose corresponding pixel value in any initial two-dimensional image is less than a preset first segmentation threshold; performing threshold segmentation on any initial two-dimensional image to obtain a second image area, wherein the second image area includes each pixel point whose corresponding pixel value in any initial two-dimensional image is less than a preset second segmentation threshold.

[0008] Optionally, based on the first image area, a first target position of the endoscope end point of the endoscope body in the three-dimensional space is determined, including: obtaining an updated first image area, and determining the orientation information of the endoscope body based on the updated first image area; combining the orientation information and the updated first image area to determine the first target position of the endoscope end point in the three-dimensional space.

[0009] Optionally, obtaining the updated first image area includes: if the pixel points corresponding to the endoscope body contained in the first image area are in a discrete state, performing mask continuous processing on the first image area to obtain the updated first image area; if the pixel points corresponding to the endoscope body contained in the first image area are in a continuous state, using the first image area as the updated first image area.

[0010] Optionally, the orientation information of the endoscope body is determined based on the updated first image area, including: fitting the pixel points belonging to the endoscope body in the updated first image area to obtain the centerline fitting curve of the endoscope body; determining the orientation information based on the coordinate information of any two pixel points on the centerline fitting curve on the target coordinate axis.

[0011] Optionally, the first target position of the endoscope end point in the three-dimensional space is determined in combination with the orientation information and the updated first image area, including: obtaining the two-dimensional coordinate information of the endoscope end point in the first image area based on the orientation information, the obtained centerline fitting curve and the pixel points belonging to the endoscope body in the first image area; performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the endoscope end point in the first image area corresponding to different initial two-dimensional images to obtain the first target position of the endoscope end point in the three-dimensional space.

[0012] Optionally, based on the second image area, a second target position of the sheath end point of the sheath in the three-dimensional space is determined, including: performing contour detection on the second image area to obtain a centroid data set corresponding to the sheath, wherein the centroid data set includes two-dimensional coordinate information of each centroid corresponding to the sheath in the second image area; screening each centroid contained in the centroid data set according to the centerline fitting curve and the end point of the endoscope; using the screened designated centroid as the sheath end point, and obtaining the two-dimensional coordinate information of the sheath end point in the second image area; performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the sheath end point in the second image area corresponding to different initial two-dimensional images, to obtain the second target position of the sheath end point of the sheath in the three-dimensional space.

[0013] Optionally, the various centroids contained in the centroid data set are screened according to the centerline fitting curve and the end point of the endoscope, including: obtaining the tangent line of the end point of the endoscope on the centerline fitting curve; determining the corresponding center of the circle on the tangent line according to the end point of the endoscope, the orientation information of the endoscope body and the preset radius length; constructing a target circle with the center of the circle and the preset radius length, and based on the target circle, screening the various centroids contained in the centroid data set according to the screening conditions to determine the specified center of mass.

[0014] Optionally, the screening conditions include at least: the specified center of mass is within the target circle; the distance between the specified center of mass and the tangent is less than or equal to a first distance threshold; the distance between the specified center of mass and the end point of the endoscope is greater than or equal to the distance between any center of mass in the center of mass data set and the end point of the endoscope.

[0015] Optionally, based on the first target position and the second target position, the orientation information between the end point of the sheath tube and the reference point is determined, including: obtaining reference coordinate information of the reference point in three-dimensional space and target coordinate information of the target point on the physiological channel in three-dimensional space, wherein the target point is a puncture point determined on the physiological channel based on the planned path; based on the second target position and the reference coordinate information, determining the distance in the orientation information; determining a first straight line from the target point to the reference point according to the target coordinate information and the reference coordinate information, and determining a second straight line from the target point to the end point of the sheath tube according to the target coordinate information and the second target position, and determining the angle between the first straight line and the second straight line as the angle in the orientation information.

[0016] Optionally, the method also includes: determining the distance between the target point and the end point of the endoscope based on the first target position and the target coordinate information, wherein the distance between the target point and the end point of the endoscope is used to determine the degree of match between the actual movement path of the endoscope in the physiological channel and the planned path.

[0017] According to another aspect of an embodiment of the present invention, there is also provided a device for determining the orientation of a sheath end point, comprising: an acquisition unit for acquiring, from at least two angles, an initial two-dimensional image of an endoscope during extension into a physiological channel, wherein the endoscope comprises a sheath and an endoscope body; a segmentation unit for segmenting, from the initial two-dimensional image, a first image region including the endoscope body and a second image region including the sheath; a first determination unit for determining, based on the first image region, a first target position of an end point of the endoscope of the endoscope body in a three-dimensional space, and based on the second image region, determining, in the three-dimensional space, a second target position of a sheath end point of the sheath; and a second determination unit for determining, based on the first target position and the second target position, orientation information between the sheath end point and a reference point, wherein the orientation information is used to indicate a distance and an angle between the sheath end point and the reference point, and the reference point is used to indicate a position where an abnormal state exists outside the physiological channel.

[0018] According to another aspect of an embodiment of the present invention, a processor is further provided, wherein the processor is used to run a program, wherein when the program is run, the method for determining the position of the end point of the sheath tube according to an embodiment of the present invention is executed.

[0019] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded by a processor and executing any one of the above-mentioned methods for determining the position of the end point of the sheath tube.

[0020] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any one of the above-mentioned methods for determining the position of the sheath end point.

[0021] According to another aspect of the present invention, a computer program product is provided, which includes a computer program, and when executed by a processor, the computer program implements the method for determining the position of the sheath tube end point of the present invention.

[0022] In an embodiment of the present invention, an initial two-dimensional image of the endoscope extending into the physiological channel can be obtained from at least two angles. The first image area of ​​the endoscope body and the second image area of ​​the sheath can be segmented from the initial two-dimensional image. From the first image area, the coordinates of the end point of the endoscope in the two-dimensional space can be determined, and the coordinates can be converted into the three-dimensional space to obtain the first target position. From the second image area, the coordinates of the end point of the trachea in the two-dimensional space can be determined, and the coordinates can be converted into the three-dimensional space to obtain the second target position. Through the first target position and the second target position, the distance and angle between the end point of the sheath and the reference point can be determined, so as to determine in real time whether the end point of the sheath reaches the lesion point. In this embodiment, by converting the coordinates from the two-dimensional space to the three-dimensional space, the position of the end point of the endoscope and the end point of the sheath relative to the reference point can be more accurately determined, the accuracy of positioning can be improved, and more three-dimensional information can be provided, including depth and angle, etc., so that the positional relationship between the endoscope and the reference point can be more clearly understood, which helps to guide the diagnosis and treatment operations more accurately. Compared with the confirmation by CArm perspective images at different angles, the above method provides richer three-dimensional information, reduces the limitations in two-dimensional space, and helps to improve the accuracy and safety of diagnosis and treatment. By more intuitively understanding the distance and angle between the end point of the sheath and the reference point, the diagnosis and treatment operations can be guided more accurately, the error can be reduced, and the success rate can be improved. In addition, the technical effect of improving the accuracy of determining the orientation of the end point of the sheath is achieved, and the technical problem of low accuracy in determining the orientation of the end point of the sheath is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0024] Figure 1 is a flow chart of a method for determining the position of a sheath tube end point according to an embodiment of the present invention;

[0025] Figure 2is a flow chart of a method for calculating spatial position of a two-dimensional perspective image back-projection according to an embodiment of the present invention;

[0026] FIG3( a ) is a flow chart of a method for preprocessing data during sheath tip detection according to an embodiment of the present invention;

[0027] FIG3( b ) is a flow chart of a method for segmenting consumables during sheath tip detection according to an embodiment of the present invention;

[0028] FIG3( c ) is a flow chart of a method for acquiring the coordinates of the sheath tip during the sheath tip detection process according to an embodiment of the present invention;

[0029] Figure 4 is a schematic diagram of a CArm perspective image obtained after data preprocessing and transformation according to an embodiment of the present invention;

[0030] Figure 5 is a schematic diagram of a bronchoscope with a continuous mask obtained by corrosion expansion according to an embodiment of the present invention;

[0031] Figure 6 is a schematic diagram of a bronchoscope obtained by contour detection according to an embodiment of the present invention;

[0032] Figure 7 is a schematic diagram of adaptively segmenting the end of a bronchoscope according to an embodiment of the present invention;

[0033] Figure 8 is a schematic diagram of a fitting line of a bronchoscope according to an embodiment of the present invention;

[0034] Fig. 9 is a schematic diagram of a tangent line of a fitting line at a terminal point of a bronchoscope according to an embodiment of the present invention;

[0035] Fig.10 is a schematic diagram of a segmentation result of a sheath obtained by threshold segmentation according to an embodiment of the present invention;

[0036] Fig.11 is a schematic diagram of determining the end point of a sheath tube from a circle according to an embodiment of the present invention;

[0037] FIG12( a ) is a schematic diagram of a bronchoscope tip and a sheath tip according to an embodiment of the present invention;

[0038] FIG12( b ) is a schematic diagram of the spatial position of a sheath tube according to an embodiment of the present invention;

[0039] Fig.13 is a schematic diagram of back-projection calculation of spatial coordinates according to an embodiment of the present invention;

[0040] Fig.14is a schematic diagram of a bronchial tree visualization display according to an embodiment of the present invention;

[0041] Fig.15 is a schematic structural diagram of a device for determining the position of a sheath tube end point according to an embodiment of the present invention;

[0042] Fig.16 is a schematic diagram of an electronic device for a method for determining the position of a sheath end point according to an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0044] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0045] Example 1

[0046] According to an embodiment of the present invention, an embodiment of a method for determining the orientation of a sheath end point is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0047] Figure 1 is a flow chart of a method for determining the position of a sheath end point according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0048] Step S102, acquiring an initial two-dimensional image of the endoscope during the process of extending into the physiological channel from at least two angles.

[0049] In the technical solution provided in the above step S102 of the present invention, the physiological channel can be a working channel of the endoscope. If the part to be examined by the endoscope is the bronchus of the lung, the physiological channel can be the airway of the lung, which can also be called the bronchial channel. It should be noted that the above physiological channel and endoscope are only for illustration, and are not specifically limited here, and can be determined according to the actual location of the lesion and the operation performed.

[0050] Optionally, the initial two-dimensional image may be a two-dimensional CArm perspective image, which may also be referred to as a CArm view or a CArm perspective image. For example, it may be a CArm perspective image obtained after data preprocessing and transformation.

[0051] Optionally, the angle may be a backward (Towards) angle, or a counterclockwise (CounterClockwise, Ccw) angle. It should be noted that the above angles are only for illustration and are not specifically limited here.

[0052] Optionally, the endoscope may include a sheath and an endoscope body. The sheath may protect the lens, light source and other components of the endoscope body from the external environment, prevent damage or contamination, accurately guide the endoscope body to a specific position, ensure the accuracy and safety of surgery or inspection, and fix the position of the endoscope body to prevent unnecessary shaking or displacement during operation, thereby ensuring the clarity and stability of imaging. The sheath may also be referred to as a consumable. The endoscope body may include components such as a lens, a light source, and a sensor, and the situation inside the physiological channel may be observed through its lens and light source.

[0053] In this embodiment, initial two-dimensional images of the endoscope penetrating into the physiological channel can be obtained at at least two angles.

[0054] Optionally, an endoscope is introduced into a patient's physiological channel, such as a bronchus in the lungs, for inspection or treatment operations. Components such as the lens and light source of the endoscope can capture the situation inside the physiological channel in real time and convert it into an image displayed on a corresponding display. By imaging with a C-arm X-ray machine, a CArm perspective image can be obtained when the endoscope is extended into the physiological channel at different angles. The above CArm perspective image can display the position of the endoscope, the structure of the physiological channel, and possible lesions or abnormalities to provide real-time visual guidance.

[0055] Optionally, after obtaining the CArm perspective image, in order to ensure image quality and reduce noise, which is beneficial for subsequent image analysis and processing, data preprocessing can be performed to optimize image quality and accuracy so that the position and condition of the endoscope in the physiological channel can be observed more clearly. Based on the initial two-dimensional image after data preprocessing, the position of the endoscope and the condition of the physiological channel can be observed from different angles to determine the accurate position and direction of the endoscope.

[0056] For example, if the physiological channel targeted is the bronchus of the lung, in the bronchoscopic transparenchymal nodule access surgery, the CArm perspective view is mainly used to guide the surgical operation in real time to confirm the position of the endoscope and whether the sheath has reached the lesion. After the preoperative CT determines the lesion and plans the path, the digitally reconstructed radiograph (DRR) image generated by the intraoperative CT can be registered with the CArm perspective image to map the confirmed lesion to the CArm perspective view. The operator performs the surgery on the CArm perspective view based on this information. During the operation, the CArm perspective view at different angles is used to confirm the position of the endoscope and whether the end point of the sheath has accurately reached the lesion.

[0057] It should be noted that the above-mentioned scenarios of obtaining CArm perspective images at different angles are only examples, and are not specifically limited here and can be determined according to actual conditions.

[0058] Step S104 : segmenting the initial two-dimensional image into a first image region including the endoscope body and a second image region including the sheath.

[0059] In the technical solution provided in step S104 of the present invention, the first image area may be an image area including the endoscope body. The second image area may be an image area including the sheath. The first image area and the second image area may be an image of the endoscope body and an image of the sheath, respectively, which are adaptively segmented according to the orientation and width of the endoscope.

[0060] In this embodiment, after obtaining initial two-dimensional images of the endoscope extending into the physiological channel at at least two angles, local images with the endoscope body and the sheath can be segmented out to obtain a first image area and a second image area.

[0061] Optionally, after data preprocessing of the initial two-dimensional image, threshold segmentation is an important step. The purpose of threshold segmentation is to separate the endoscope and sheath from the initial two-dimensional image by setting different segmentation thresholds, eliminating the influence of other redundant features in the initial two-dimensional image, so as to better perform endoscope fitting, threshold segmentation and contour detection. Compared with performing the above steps on the entire initial two-dimensional image, local segmentation can improve the acquisition efficiency and accuracy of the entire process.

[0062] Step S106, based on the first image area, determine the first target position of the endoscope end point of the endoscope body in the three-dimensional space, and based on the second image area, determine the second target position of the sheath end point of the sheath in the three-dimensional space.

[0063] In the technical solution provided in the above step S106 of the present invention, the endoscope body may include an endoscope end point, which may also be referred to as the end of the endoscope. For example, if the endoscope body is a bronchoscope, the endoscope end point may be the end of the bronchoscope, which may also be referred to as the end of the bronchoscope. The first target position may be the coordinates of the end point of the endoscope in three-dimensional space, and the second target position may be the coordinates of the end point of the sheath in three-dimensional space. The end point of the sheath may also be referred to as the end of the sheath, which may be represented by point P. The end point of the sheath may also be referred to as the end of the consumable.

[0064] In this embodiment, after the first image region including the endoscope body and the second image region including the sheath are segmented from the initial two-dimensional image, the position of the end point of the endoscope in the two-dimensional space can be determined based on the first image region, and then the first target position of the end point of the endoscope in the three-dimensional space can be determined. The position of the end point of the sheath in the two-dimensional space can also be determined based on the second image region, and then the second target position of the end point of the sheath in the three-dimensional space can be determined.

[0065] Optionally, the image areas of the endoscope body and the sheath are segmented from the initial two-dimensional image, and the positions of the end points of the endoscope and the sheath in two-dimensional space can be determined based on these two areas, respectively, and then converted into target positions in three-dimensional space. The above method process helps to improve the accuracy and safety of endoscopic operation. That is, by segmenting the image areas of the endoscope body and the sheath, the positions of the end points of the endoscope and the sheath in the two-dimensional image can be accurately located. By converting the above two-dimensional position into the target position in three-dimensional space, the positional relationship of the end points of the endoscope and the sheath relative to the reference point (such as the lesion point) can be more clearly understood.

[0066] In the embodiment of the present invention, by converting from two-dimensional to three-dimensional, the position of the end point of the endoscope and the end point of the sheath in space can be determined more accurately, thereby improving the accuracy of positioning. Converting the two-dimensional position to a three-dimensional position can provide more stereoscopic information and angles, allowing doctors to more clearly understand the relative position of the endoscope and the sheath in space, which helps to guide the surgical operation more accurately. By determining the position of the end point of the endoscope and the end point of the sheath in three-dimensional space, the risk of miscutting or damaging healthy tissue can be reduced, thereby improving the safety of the operation.

[0067] Therefore, compared with the method in the related art that only uses two-dimensional CArm perspective to determine the movement of the sheath end point to the lesion point, the conversion from two-dimensional to three-dimensional can provide more accurate positioning information and more spatial information, which helps to improve the accuracy and safety of surgical operations, provide better operational guidance and support, and help improve the accuracy and success rate of endoscopic operations.

[0068] Step S108: determining the position information between the end point of the sheath tube and the reference point based on the first target position and the second target position.

[0069] In the technical solution provided in the above step S108 of the present invention, the orientation information can be used to describe the relative positional relationship between the end point of the sheath and the reference point, and can be used to indicate the distance and angle between the end point of the sheath and the reference point. The reference point can be used to indicate the location of an abnormal state outside the physiological channel, and can also be called a lesion point or a target point, which is used to indicate the location of a lesion outside the physiological channel.

[0070] In this embodiment, after determining the first target position based on the first image area and the second target position based on the second image area, the orientation information between the sheath tube end point and the reference point can be determined based on the first target position and the second target position.

[0071] Optionally, by determining the first target position and the second target position, the position information between the sheath end point and the reference point (such as the lesion point), including the distance and angle, can be further determined. The above process is of great significance for positioning and guiding in endoscopic operation.

[0072] Optionally, the positions of the end point of the endoscope and the end point of the sheath in two-dimensional and three-dimensional space are determined based on the first image area and the second image area. The first target position and the second target position respectively represent the coordinates of the end point of the endoscope and the end point of the sheath in space, providing a basis for subsequent orientation information. Using the first target position and the second target position, the distance and angle information between the end point of the sheath and the reference point can be calculated to describe the relative position relationship between them.

[0073] In an embodiment of the present invention, the sheath in the CArm view of any two angles is segmented and located at the end of the sheath, and then its position in the three-dimensional space is determined by converting the two-dimensional space into the three-dimensional space. By calculating the angle and distance between the sheath and the lesion, it is determined whether the sheath has reached the lesion. If the sheath has not reached the lesion, corresponding angle adjustment and distance prompts can be provided to help the operator more easily deliver the sheath to the lesion, thereby improving the accuracy and success rate of the operation. Through the above method, the sheath position is accurately located using three-dimensional spatial information, which can provide more accurate position information and spatial relationships compared to the two-dimensional CArm perspective view in the related art, and provide more intuitive and instructive information for the surgical operator. It can reduce the number of adjustments during the operation, reduce surgical risks, and improve the efficiency and success rate of the operation, which is of great significance for interventional surgeries such as BTPNA surgery.

[0074] In the above steps S102 to S108 of the embodiment of the present invention, the initial two-dimensional image of the endoscope in the process of extending into the physiological channel can be obtained from at least two angles. The first image area of ​​the endoscope body and the second image area of ​​the sheath can be segmented from the initial two-dimensional image. From the first image area, the coordinates of the end point of the endoscope in the two-dimensional space can be determined, and the coordinates can be converted into the three-dimensional space to obtain the first target position. From the second image area, the coordinates of the end point of the trachea in the two-dimensional space can be determined, and the coordinates can be converted into the three-dimensional space to obtain the second target position. Through the first target position and the second target position, the distance and angle between the end point of the sheath and the reference point can be determined, so as to determine in real time whether the end point of the sheath reaches the lesion point. In this embodiment, by converting the coordinates from the two-dimensional space to the three-dimensional space, the position of the end point of the endoscope and the end point of the sheath relative to the reference point can be more accurately determined, the accuracy of positioning can be improved, and more three-dimensional information can be provided, including depth and angle, etc., so that the positional relationship between the endoscope and the reference point can be more clearly understood, which helps to guide the diagnosis and treatment operations more accurately. Compared with the confirmation by CArm perspective images at different angles, the above method provides richer three-dimensional information, reduces the limitations in two-dimensional space, and helps to improve the accuracy and safety of diagnosis and treatment. By more intuitively understanding the distance and angle between the end point of the sheath and the reference point, the diagnosis and treatment operations can be guided more accurately, the error can be reduced, and the success rate can be improved. In addition, the technical effect of improving the accuracy of determining the orientation of the end point of the sheath is achieved, and the technical problem of low accuracy in determining the orientation of the end point of the sheath is solved.

[0075] The embodiment of the present invention is described in detail below in combination with the above steps.

[0076] As an optional embodiment, step S104, respectively segmenting a first image area including an endoscope body and a second image area including a sheath from the initial two-dimensional image, includes: performing threshold segmentation on any initial two-dimensional image to obtain a first image area, wherein the first image area includes each pixel point whose corresponding pixel value in any initial two-dimensional image is less than a preset first segmentation threshold; performing threshold segmentation on any initial two-dimensional image to obtain a second image area, wherein the second image area includes each pixel point whose corresponding pixel value in any initial two-dimensional image is less than a preset second segmentation threshold.

[0077] In this embodiment, in the process of segmenting the first image area and the second image area from the initial two-dimensional image, the initial two-dimensional image can be segmented according to the first segmentation threshold, and the image area containing pixel points whose pixel values ​​are less than the first segmentation threshold is used as the first image area. The initial two-dimensional image can also be segmented according to the second segmentation threshold, and the image area containing pixel points whose pixel values ​​are less than the second segmentation threshold is used as the second image area. The first image area contains each pixel point whose corresponding pixel value in the initial two-dimensional image is less than the preset first segmentation threshold. The first segmentation threshold can be a segmentation threshold for segmenting with the endoscope body. The second image area contains each pixel point whose corresponding pixel value in the initial two-dimensional image is less than the preset second segmentation threshold. The second segmentation threshold can be a segmentation threshold for segmenting with the sheath.

[0078] In an embodiment of the present invention, data preprocessing may be performed before segmenting the initial two-dimensional image. Data preprocessing is an important step that aims to improve image quality and reduce noise, thereby facilitating subsequent image analysis and processing. For example, during data preprocessing, steps such as improving contrast, improving brightness, and removing noise may be performed, which are only illustrative here and are not specifically limited.

[0079] Optionally, the contrast can be improved by calculating an adaptive equalization histogram. Adaptive equalization histogram is a method for enhancing image contrast. In the above process, by analyzing and processing the histogram of the CArm perspective image, the contrast can be adaptively enhanced according to the brightness of the local area of ​​the image, so that the CArm perspective image is clearer and highlights the details.

[0080] Optionally, the brightness can be increased by gamma transformation. By adjusting the brightness and contrast of the image, the visual effect of the image can be improved. In the above process, gamma transformation can effectively increase the brightness of the image, making the sheath tip, the endoscope body, etc. more eye-catching, which is beneficial to subsequent detection and analysis.

[0081] Optionally, noise can be removed by performing median filtering. The noise in the image is reduced by sorting the pixel values ​​around the pixel point in the CArm perspective image and taking the median. In the above process, performing median filtering can effectively remove the noise in the image, making the target area clearer and more accurate, which is beneficial to subsequent target detection and positioning.

[0082] Through the above data preprocessing steps, the image quality can be improved, the noise interference can be reduced, and a more accurate and reliable image basis can be provided for the detection and positioning of the sheath tip, thereby improving the accuracy and efficiency of the detection. It should be noted that the above methods corresponding to improving contrast, improving brightness, and removing noise are only for illustrative purposes and are not specifically limited here. As long as the process and method can improve the image quality of the CArm perspective view, they are within the protection scope of the embodiments of the present invention.

[0083] In an embodiment of the present invention, after data preprocessing, threshold segmentation is an important step. The purpose of threshold segmentation is to segment a local image area with an endoscope body (first image area) and a local image area with a sheath (second image area). The influence of other redundant pixels in the image is eliminated, so as to better perform endoscope fitting, threshold segmentation and contour detection, and finally facilitate the acquisition of the coordinates of the specified center of mass. Compared with performing the above steps on the entire perspective image, local segmentation can improve the efficiency and accuracy of obtaining the end point of the sheath.

[0084] Optionally, in the process of threshold segmentation, the first segmentation threshold and the second segmentation threshold can be determined according to the grayscale distribution characteristics of the initial two-dimensional image by using adaptive binarization and Otsu's method, so as to achieve effective segmentation of the initial two-dimensional image, separate the target (also called foreground, such as the endoscope body and sheath) from the background (such as the physiological channel) area, and provide a more accurate data basis for subsequent image processing and analysis. Among them, adaptive binarization is a method of dynamically adjusting the binarization threshold according to the local characteristics of the image. In adaptive binarization, the algorithm determines the threshold of the pixel according to the grayscale value of the neighborhood around each pixel, thereby realizing adaptive binarization processing of different regions. The above method can effectively deal with areas with large brightness changes in the image and improve the accuracy of segmentation. Otsu's method is a global threshold segmentation method that aims to find a suitable segmentation threshold (that is, the first segmentation threshold and the second segmentation threshold) that can maximize the inter-class variance between the target and the background. The method traverses multiple possible thresholds, calculates the inter-class variance under each threshold, and finds the threshold that maximizes the inter-class variance as the global threshold for segmentation. Otsu's method is suitable for images with obvious grayscale distribution of background and foreground, and can effectively realize binary segmentation of images.

[0085] Optionally, in the process of threshold segmentation using adaptive binarization and Otsu's method, the initial two-dimensional image after data preprocessing can be regarded as a matrix, each element in the matrix is ​​equivalent to a pixel in the initial two-dimensional image, the value of the element is the pixel value, and each pixel value can be a grayscale value. The grayscale value can be between (0,255). The pixel points can be divided into two categories: foreground and background according to the comparison between the segmentation threshold and the pixel value. By calculating the ratio of the number of pixels of the foreground and background, the average grayscale and other indicators, and the inter-class variance, the segmentation threshold is determined to maximize the inter-class variance. Binarization is performed under the segmentation threshold, and the initial two-dimensional image can be segmented into two parts: foreground and background.

[0086] In an embodiment of the present invention, through the combination of Otsu's method and adaptive binarization, the main part of the endoscope can be extracted more accurately, and the effective segmentation of the foreground and background can be achieved, which is helpful for subsequent image analysis and processing. The method can automatically determine the appropriate segmentation threshold, avoid the subjectivity of manually adjusting the threshold, and improve the accuracy and stability of image processing. In summary, through threshold segmentation by adaptive binarization and Otsu's method, the initial image can be effectively segmented, and the required first image area and second image area can be extracted, providing strong support for further image processing and analysis. It can also improve the automation and accuracy of image processing, which helps to extract the first image area and the second image area for subsequent processing.

[0087] For example, if the initial two-dimensional image is an M*N matrix, the segmentation threshold for segmenting the foreground and background is denoted by T. The proportion of pixels belonging to the foreground in the entire initial two-dimensional image is denoted by ω0, and the average grayscale of the grayscale values ​​corresponding to the pixels of the foreground is denoted by μ0. The proportion of pixels belonging to the background in the entire initial two-dimensional image is denoted by ω1, and the average grayscale of the grayscale values ​​corresponding to the pixels of the background is denoted by μ1. The total average grayscale of the initial two-dimensional image is denoted by μ, and the inter-class variance is denoted by σ 2 If the number of pixels in the image whose grayscale value is less than the segmentation threshold T is recorded as A, and the number of pixels whose grayscale value is greater than or equal to the segmentation threshold T is recorded as B, then:

[0088]

[0089] A+B=M*N

[0090] 1=ω0+ω1

[0091] μ=ω0μ0+ω1μ1

[0092] σ 2 =ω0(μ0-μ) 2 +ω1(μ1-μ)

[0093] By combining the above formulas, we can get:

[0094] σ 2 =ω0ω1(μ0-μ1)

[0095] The value of the segmentation threshold T is sequentially traversed through each grayscale value in the value interval (0, 255) to obtain the segmentation threshold T that can make σ reach the maximum value. The segmentation threshold T can be used to perform threshold segmentation on the initial two-dimensional image.

[0096] For example, if the first image area needs to be segmented, the foreground may include the endoscope body portion that enters the physiological channel to the puncture point, and other pixel points of the first segmentation threshold.

[0097] It should be noted that the above process and method for threshold segmentation of the initial two-dimensional image are only illustrative and are not specifically limited here. The above segmentation threshold T can be determined based on the parameters corresponding to whether the foreground to be segmented is the endoscope body or the sheath (for example, A, B), and the parameters of the initial two-dimensional image (for example, μ0, μ1). As long as the entire two-dimensional CArm perspective image can be threshold segmented, the process and method for determining the end point of the endoscope and the end point of the sheath through the local image area are within the protection scope of the embodiments of the present invention.

[0098] As an optional embodiment, step S106, based on the first image area, determines the first target position of the endoscope end point of the endoscope body in the three-dimensional space, including: obtaining an updated first image area, and determining the orientation information of the endoscope body based on the updated first image area; combining the orientation information and the updated first image area to determine the first target position of the endoscope end point in the three-dimensional space.

[0099] In this embodiment, in the process of determining the first target position of the end point of the endoscope in the three-dimensional space based on the first image area, the orientation information of the endoscope body can be determined according to the updated first image area. In combination with the orientation information and the updated first image area, the position of the end point of the endoscope in the two-dimensional space is determined, and the first target position of the end point of the endoscope in the three-dimensional space is further determined. The updated first image area can be the first image area of ​​the endoscope mask. The orientation information can be used to indicate whether the end point of the endoscope is oriented left-right or up-down, which is only an example and is not specifically limited. The orientation information can also be the sheath orientation.

[0100] Optionally, contour detection can be performed from the updated first image area, and the shape with the largest area is found to be the main body of the endoscope. That is, after the contour is detected, the area of ​​each contour can be calculated, and the shape with the largest area can be found. In this image, the shape with the largest area is usually considered to be the main area of ​​the endoscope body, because the shape of the endoscope is usually more regular and has a larger area. By identifying the shape with the largest area as the main body of the endoscope, the position and shape of the endoscope can be accurately located and extracted, which facilitates the determination of orientation information. The above method can help automatically identify and locate the endoscope, providing important information for subsequent surgical navigation or diagnosis. Through continuous mask processing and contour detection, combined with the identification of the shape with the largest area, the position of the endoscope in the image can be effectively found, and automatic detection and positioning of the endoscope can be achieved. This method can improve processing efficiency and accuracy, and provide strong support for medical analysis and diagnosis.

[0101] Optionally, the endoscope body obtained by contour detection in the updated first image area can be used to analyze its orientation information, and the position of the end point of the endoscope in two-dimensional space is determined according to the orientation information, and then converted into three-dimensional space to obtain the first target position.

[0102] As an optional embodiment, obtaining an updated first image area includes: if the pixel points corresponding to the endoscope body contained in the first image area are in a discrete state, performing mask continuous processing on the first image area to obtain an updated first image area; if the pixel points corresponding to the endoscope body contained in the first image area are in a continuous state, using the first image area as the updated first image area.

[0103] In this embodiment, if it is necessary to obtain the updated first image area, the first image area can be detected to determine whether the pixels constituting the endoscope body in the first image area are in a discrete state or a continuous state. If the pixels of the endoscope body are in a discrete state, the first image area can be subjected to mask continuous processing to obtain the updated first image area. If the pixels of the endoscope body are in a continuous state, there is no need to perform mask continuous processing, and the first image area is directly used as the updated first image area. Among them, the pixels constituting the endoscope body can be called the mask of the endoscope body (also referred to as the mask area), for example, it can be the mask of a bronchoscope. Mask continuity generally refers to the continuity and integrity of the target area in the image. If the mask is continuous, it means that the pixels of the target area are interconnected in the image without interruption or breakage, forming a continuous whole. The continuity of the mask is very important for subsequent image analysis and processing because it can ensure the integrity and accuracy of the target.

[0104] Optionally, by using a combination of erosion and dilation operations, the mask region of the endoscope body can be made continuous and more complete, so as to ensure the accuracy of the orientation information of the endoscope body determined by the first image region.

[0105] Optionally, the mask of the endoscope body in the first image area can be first eroded to remove small pixels and noise at the edge, and then dilated to re-expand and connect the mask area, thereby obtaining a more accurate and continuous mask of the endoscope body. The mask after the above processing can be better used for subsequent image analysis and processing, thereby improving the accuracy of the orientation information of the endoscope body.

[0106] Optionally, erosion is an image morphological processing operation that can slide a structural element on the image. When the structural element completely covers the target area, the pixel is retained, otherwise it is set to the background (i.e., black). The erosion operation can make the target area smaller, remove small pixels around the target, smooth the edges of the target, and remove noise and small holes. Dilation is another image morphological processing operation that can slide a structural element on the image. When the structural element overlaps with the target area, the pixel is set to the target (i.e., white), otherwise the original pixel value is retained. The dilation operation can make the target area larger, fill the holes in the target, connect the broken target parts, and increase the area and connectivity of the target.

[0107] In an embodiment of the present invention, if the endoscope body in the obtained first image area is discontinuous, it means that the mask of the endoscope body may not be continuous. The above situation may affect the recognition, positioning and analysis of the endoscope body, because the discontinuous mask will cause the shape and contour of the endoscope body to be incomplete, the information is incomplete, and errors or inaccurate results may be caused. Therefore, in order to better extract and process information such as the orientation information of the endoscope body, it is necessary to make the mask of the endoscope body continuous through image processing technology, such as corrosion and expansion. This means that in the image, the area of ​​the endoscope body should be continuous without interruptions or breaks to ensure accurate recognition and analysis of the endoscope body. By making the mask of the endoscope body continuous, the accuracy and reliability of the extraction and processing of the endoscope body can be improved.

[0108] It should be noted that the above-mentioned process and method for performing continuous masking on the endoscope body in the first image area are only for illustration and are not specifically limited here. As long as the process and method can perform continuous masking and ensure the accuracy of analyzing the orientation information of the endoscope body and the width of the endoscope body and other information, they are within the protection scope of the embodiments of the present invention.

[0109] As an optional embodiment, the orientation information of the endoscope body is determined based on the updated first image area, including: fitting the pixel points belonging to the endoscope body in the updated first image area to obtain the centerline fitting curve of the endoscope body; determining the orientation information based on the coordinate information of any two pixel points on the centerline fitting curve on the target coordinate axis.

[0110] In this embodiment, in the process of determining the orientation information of the endoscope body based on the updated first image area, the pixel points belonging to the endoscope body in the updated first image area can be fitted to obtain the centerline fitting curve of the endoscope body. The orientation information can be determined based on the coordinate information of any two pixel points on the centerline fitting curve on the target coordinate axis. The centerline fitting curve can also be called a fitting curve.

[0111] In an embodiment of the present invention, a fitting curve can be obtained by performing polynomial fitting on the pixel points of the endoscope body in the updated first image area, that is, the shape of the endoscope body is approximated by a polynomial function. Next, two pixel points are taken on the fitting curve for comparison, mainly comparing the size of the coordinate information of the two points on the target coordinate axis to determine the left-right orientation or the up-down orientation of the endoscope body. For example, the pixel points of the endoscope body can be sorted according to the distance between the initial position where the endoscope is initially extended into the physiological channel, and the left-right orientation of the endoscope body can be determined based on the corresponding serial numbers of the two pixel points and the corresponding coordinate information on the x-coordinate axis, and the up-down orientation of the endoscope body can also be determined based on the corresponding serial numbers of the two pixel points and the corresponding coordinate information on the y-coordinate axis.

[0112] It should be noted that the above-mentioned target coordinate axis for analyzing the orientation information, and the method and process for determining the orientation information of the endoscope body are only for illustration and are not specifically limited here and can be set according to actual conditions.

[0113] Optionally, a suitable fitting curve is found in the first image area through a polynomial fitting algorithm so that the curve can better approximate the shape of the endoscope body. The overall shape and contour of the endoscope body can be better described and characterized by the fitting curve. Two pixel points are selected on the fitting curve to represent two specific positions of the endoscope body. Usually, two pixel points are selected to be located at the two end points or two specific feature points of the endoscope body. The size of the coordinate information of these two points on the target coordinate axis can be used to determine the left and right orientation of the endoscope body.

[0114] Optionally, by comparing the size of the coordinate information of the two selected pixel points and the distance between the two pixel points and the initial position where the endoscope initially extends into the physiological channel, it can be determined which point has the larger coordinate information, and thus the left and right orientation of the endoscope body can be determined. For example, the first point is located on the left side of the second point. If the x-coordinate of the first point is larger, and the distance between the first point and the initial position is greater than the distance between the second point and the initial position, it means that the endoscope body is facing left; if the x-coordinate of the second point is larger, and the distance between the second point and the initial position is greater than the distance between the first point and the initial position, it means that the endoscope body is facing right. This is only for illustration and no specific limitation is made.

[0115] In the embodiment of the present invention. Through the above method, the first image area can be analyzed and judged by image processing and curve fitting methods, so as to determine the left and right orientation of the endoscope body. The position and orientation of the endoscope body can be better understood, providing important reference information for subsequent surgical navigation and operation. Through image processing and analysis, the orientation of the endoscope body can be accurately judged, which helps to improve the accuracy and success rate of the surgery.

[0116] As an optional embodiment, the first target position of the end point of the endoscope in the three-dimensional space is determined in combination with the orientation information and the updated first image area, including: obtaining the two-dimensional coordinate information of the end point of the endoscope in the first image area based on the orientation information, the obtained centerline fitting curve and the pixel points belonging to the endoscope body in the first image area; performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the end point of the endoscope in the first image area corresponding to different initial two-dimensional images to obtain the first target position of the end point of the endoscope in the three-dimensional space.

[0117] In this embodiment, in the process of determining the first target position of the end point of the endoscope in the three-dimensional space by combining the orientation information and the updated first image area, the two-dimensional coordinate information of the end point of the endoscope in the two-dimensional space can be obtained based on the orientation information, the centerline fitting curve and the pixel points of the endoscope body in the first image area. The two-dimensional coordinate information can be three-dimensionally back-projected to obtain the first target position.

[0118] In this embodiment, after the endoscope body is detected from the contour in the first image region in the threshold-based segmentation step, non-zero point detection may be performed.

[0119] Optionally, during the non-zero point detection process, the end point of the endoscope, the parameters of the tangent line of the fitting curve at the end point of the endoscope (e.g., slope k, offset b), and another point O on the tangent line may be obtained. That is, non-zero point detection is performed in the first image area after threshold segmentation and contour detection, that is, pixel points are traversed to obtain pixel points constituting the endoscope body in the first image area, and these pixel points may represent the contour or edge of the endoscope body.

[0120] Optionally, if the orientation information of the endoscope body is toward the right, the rightmost point located on the endoscope body and on the centerline fitting curve can be screened out from the pixel points to the right of the pixel points of the endoscope body, and this point is the end point of the endoscope, and the two-dimensional coordinate information of the end point of the endoscope can be determined. Conversely, if the orientation information is toward the left, the leftmost point located on the endoscope body and on the centerline fitting curve can be determined from the pixel points to the left of the pixel points of the endoscope body as the end point of the endoscope. It should be noted that the above process of screening out the end point of the endoscope is only for illustration and is not specifically limited here.

[0121] It should be noted that the above process and method of determining the end point of the endoscope from the various points on the centerline fitting curve is only an example of the endoscope body facing right or left. If the endoscope body is facing other directions, the end point of the endoscope can be selected according to the actual situation of the direction.

[0122] Optionally, after the two-dimensional coordinate information of the end point of the endoscope is determined, the first target position in the three-dimensional space may be obtained by back-projection calculation.

[0123] For example, after determining the two-dimensional coordinate information of the end point of the endoscope at the Towards and Ccw angles, its spatial coordinates can be obtained by back-projection. It is known that the two-dimensional coordinate information of the end point of the endoscope at the Towards angle is (u1, v1), and the two-dimensional coordinate information of the end point of the endoscope at the Ccw angle is (u2, v2). Assuming that the coordinates of the first target position in the corresponding three-dimensional space are (x, y, z), (u1, v1) and (u2, v2) can be dedistorted according to the inverse process of the following formula to obtain the dedistorted (x'1, y'1) and (x'2, y'2):

[0124]

[0125] Among them, x center The abscissa value of the optical center point of the imaging plane of the first image area can be used to represent; center Can be used to represent the ordinate value of the optical center point; x focal It can be used to represent the horizontal focal length of the camera that captures the CArm perspective view; focalCan be used to indicate the focal length in the vertical direction; a ij , b ij It can be a 4-order distortion parameter, representing the lateral distortion coefficient and the longitudinal distortion coefficient respectively. It has been obtained in the CArm calibration stage.

[0126] For another example, during the patient registration stage, the rotation matrix and translation matrix from CT to CArm at the Towards and Ccw angles have been calculated, which are:

[0127]

[0128] in, Can be used to represent the rotation matrix at the Towards angle; Can be used to represent the translation matrix at the Towards angle; Can be used to represent the rotation matrix under the Ccw angle; Can be used to represent the translation matrix under Ccw angle.

[0129] It can be expressed by the formula Among them, P can be used to represent the coordinates of point A in the coordinate system of the CT image, and A can be used to represent the coordinates of point A in the coordinate system of the CArm image, and the following coordinate information is obtained:

[0130]

[0131]

[0132] Combining the above two equations, we can get:

[0133]

[0134] By solving the above linear equations, the coordinates of the end point of the endoscope in three-dimensional space can be obtained, that is, the first target position (x q ,y q ,z q ), which can also be called the starting point of the physiological channel.

[0135] As an optional embodiment, step S106, based on the second image area, determines the second target position of the sheath end point of the sheath in the three-dimensional space, including: performing contour detection on the second image area to obtain a centroid data set corresponding to the sheath, wherein the centroid data set includes two-dimensional coordinate information of each centroid corresponding to the sheath in the second image area; screening each centroid contained in the centroid data set according to the centerline fitting curve and the end point of the endoscope; using the screened designated centroid as the sheath end point, and obtaining the two-dimensional coordinate information of the sheath end point in the second image area; performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the sheath end point in the second image area corresponding to different initial two-dimensional images, to obtain the second target position of the sheath end point of the sheath in the three-dimensional space.

[0136] In this embodiment, in the process of determining the second target position of the end point of the sheath in the three-dimensional space based on the second image area, the contour detection can be performed on the second image area to obtain a centroid data set corresponding to the sheath. The various centroids contained in the centroid data set can be screened according to the centerline fitting curve and the end point of the endoscope. The screened designated centroid is used as the end point of the sheath. The two-dimensional coordinate information of the end point of the sheath in the second image area can be obtained. The two-dimensional coordinate information can be processed into the three-dimensional space by three-dimensional back projection to obtain the second target position of the end point of the sheath in the three-dimensional space, wherein the centroid data set can include the two-dimensional coordinate information of the various centroids corresponding to the sheath in the second image area.

[0137] Optionally, after determining the designated centroid from a large number of centroids, the two-dimensional coordinate information corresponding to the designated centroid can be determined from the centroid data set. The second target position of the end point of the sheath in three-dimensional space can be calculated using the above-mentioned back-projection calculation formula in the same way as determining the first target position of the end point of the endoscope. The above-mentioned embodiment has described the back-projection calculation process in detail, and will not be repeated here.

[0138] Optionally, the bias of the Otsu method is adaptively adjusted for threshold segmentation to obtain the segmentation result of the sheath image, that is, the second image area. By adaptively adjusting the bias, the characteristics of the image can be better adapted to improve the segmentation effect. The Otsu method is a global threshold segmentation method based on maximizing the inter-class variance. In this method, by adaptively adjusting the bias, the threshold can be dynamically adjusted according to the characteristics of the image to adapt to the brightness changes and noise conditions in different regions. According to the adjusted bias, the Otsu method is used for threshold segmentation. The process of threshold segmentation is to divide the image into two parts, the target (for example, the sheath) and the background, according to the grayscale value of the image. Through the bias after adaptive adjustment, the threshold can be determined more accurately to achieve effective segmentation of the bronchoscope image. After the Otsu method threshold segmentation after adaptively adjusting the bias, the second image area is obtained. This result will show the separation of the target area and the background area of ​​the sheath, which is convenient for subsequent processing and analysis.

[0139] By adaptively adjusting the bias of the Otsu method to perform threshold segmentation, more accurate and effective segmentation can be achieved according to the characteristics and requirements of the image, which is helpful to extract the second image area and provide a better data basis for subsequent processing and analysis. The above method can improve the accuracy and efficiency of image segmentation.

[0140] As an optional embodiment, the various centroids contained in the centroid data set are screened according to the centerline fitting curve and the end point of the endoscope, including: obtaining the tangent line of the end point of the endoscope on the centerline fitting curve; determining the corresponding center of the circle on the tangent line according to the end point of the endoscope, the orientation information of the endoscope body and the preset radius length; constructing a target circle with the center of the circle and the preset radius length, and based on the target circle, screening the various centroids contained in the centroid data set according to the screening conditions to determine the specified center of mass.

[0141] In this embodiment, in the process of selecting a designated center of mass from the centers of mass based on the centerline fitting curve and the end point of the endoscope, the tangent of the end point of the endoscope on the centerline fitting curve can be obtained. The center of the target circle can be determined on the tangent based on the end point of the endoscope, the orientation information of the endoscope body and the preset radius length. A target circle can be constructed with the center of the circle and the preset radius length. Based on the target circle, the designated center of mass is selected from a large number of centers of mass based on the screening conditions, wherein the preset radius length can be determined based on the longest sheath length (maxLength), for example, the preset radius length r=maxLength / 2+5, where the preset radius length is only for example and is not specifically limited. The target circle can be represented by C m The longest sheath length can be obtained in advance based on a large amount of surgical data. The centroids can also form a centroid set, for example, a centroid list (massList).

[0142] Optionally, before determining the tangent on the fitting curve, the first image area can be adaptively segmented according to the orientation information and the overall width of the endoscope body, so as to segment out the image with the end point of the endoscope. According to the orientation judgment of the endoscope body, a specific part of the endoscope body can be selected to be segmented for subsequent processing and analysis. That is, adaptive image segmentation is performed according to the orientation and the overall width of the endoscope body. In this case, if the endoscope body faces right, the image of the endoscope body on the right side can be selected to be segmented. By performing adaptive segmentation according to the orientation information and the width information, the image of the endoscope body of the specific part can be better selected and extracted. After adaptive segmentation, the image of the endoscope body on the right side can be obtained. The purpose of the above method is to provide more accurate data for subsequent processing and analysis, so that the fitting effect is better. Segmenting out part of the endoscope body image can make the subsequent processing more accurate and targeted, and help to better understand the shape and characteristics of the endoscope body.

[0143] In the embodiment of the present invention, by adaptively segmenting according to the orientation information and the overall width of the endoscope body, the image of the endoscope body of a specific part (e.g., the end point of the endoscope) can be effectively selected and extracted, providing a more accurate data basis for subsequent processing and analysis. The above method can segment the image in a targeted manner according to the specific situation, which helps to improve the accuracy of recognition and positioning of the endoscope body.

[0144] Optionally, after acquiring the image of the end point of the endoscope segmented in the above-mentioned adaptive manner, a polynomial fitting can be performed to obtain a fitting line of the image of the end point of the endoscope, thereby indicating the moving path of the endoscope. That is, a polynomial fitting is performed on the image of the end point of the endoscope segmented in the previous step. Through the fitting algorithm, a suitable polynomial function can be found to approximate the shape and contour of the end of the endoscope to indicate the moving path and direction of the endoscope. On the basis of the fitting line, the tangent of the tip of the mirror can be taken, and the tangent can approximately represent the direction of the sheath. By calculating the slope or direction of the fitting line, the approximate orientation of the sheath can be determined, providing important information about the position and direction of the sheath. By fitting the polynomial and obtaining the tangent of the tip of the mirror, the direction of the sheath can be determined more accurately, providing important information and guidance for subsequent surgical operations. The above method can better understand the position and moving path of the endoscope body and improve the efficiency and accuracy of the operation.

[0145] Optionally, after obtaining the end point of the endoscope and the tangent line of the fitting curve on the end point of the endoscope through the above non-zero point detection process, a target circle C can be constructed with the end point of the endoscope as a point on the circle and a point located on the tangent line and at a distance of a preset radius length r from the end point of the endoscope as the center O. mThrough the above process, the end point of the endoscope, the tangent parameters (slope k, offset b) of the fitting curve at the end point of the endoscope, and another point O on the tangent can be obtained. This information can help determine the end point position and direction of the sheath, providing an important data basis for subsequent processing and analysis. This method can improve the accuracy of identifying and positioning the endoscope body and provide useful information and guidance for surgical operations.

[0146] Optionally, contour detection may be performed on the second image region obtained by the above threshold segmentation to obtain the two-dimensional coordinate information of each center of mass in the center of mass list of the sheath tube contour:

[0147]

[0148] Among them, C x It can be used to represent the horizontal coordinate of each centroid, that is, the horizontal coordinate in the two-dimensional coordinate information; C y It can be used to represent the ordinate of each centroid, that is, the ordinate of the two-dimensional coordinate information, x, y is the serial number corresponding to each centroid, x, y is a positive integer; M can be used to represent the image geometric moment, specifically, M 10 Can be used to represent the first-order horizontal moment, M 01 Can be used to represent the first-order vertical moment, M 00 It can be used to represent the area of ​​contours in an image.

[0149] As an optional embodiment, the screening conditions include at least: the specified center of mass is located within the target circle; the distance between the specified center of mass and the tangent is less than or equal to a first distance threshold; the distance between the specified center of mass and the end point of the endoscope is greater than or equal to the distance between any center of mass in the center of mass data set and the end point of the endoscope.

[0150] In this embodiment, the screening conditions may include at least the following: the designated centroid may be located within the target circle; the distance between the designated centroid and the switch may be less than or equal to the first distance threshold; the distance between the designated centroid and the end point of the endoscope may be greater than or equal to the distance between any centroid and the end point of the endoscope. The first distance threshold may also be called a tangent threshold, which may be predetermined according to experimental data and may be used to exclude noise centroids and improve the accuracy of the determined designated centroid.

[0151] Optionally, the specified centroid D(x1, y1), that is, the specified centroid in the centroid massList, can be filtered out through the at least three filtering conditions mentioned above.

[0152] Optionally, the first filter condition is to specify that the center of mass is a circle C. m The center of mass specified in this filter must be located within circle C. m The interior, that is, the circular area C with center Om This condition can be used to limit the location range of the specified center of mass, ensuring that the specified center of mass is within the specified circular area for further screening and analysis. m The inner centroid can narrow the search range of the specified centroid, reduce the interference of irrelevant points, and improve the accuracy of screening.

[0153] Optionally, the second screening condition mentioned above is within a threshold range of the distance to the tangent. The distance from the specified centroid to the tangent in this screening condition must be within a specified threshold range, that is, the distance from the tangent cannot exceed a certain threshold. This screening condition is used to limit the distance range from the specified centroid to the tangent to ensure that the proximity between the specified centroid and the tangent meets the requirements. By setting the threshold range of the distance to the tangent, points far from the tangent can be filtered out, and the specified centroid close to the tangent can be retained, thereby improving the accuracy of the screening.

[0154] Optionally, the third filtering condition mentioned above is the designated centroid that is farthest from the end point of the endoscope. In this filtering condition, on the basis of meeting the first two filtering conditions, the designated centroid that is farthest from the end point of the endoscope can be selected, that is, the designated centroid D(x1,y1) in the centroid massList that is farthest from the end point of the endoscope. This filtering condition is used to determine the final selection of the designated centroid D(x1,y1) to ensure that the designated centroid farthest from the end point of the endoscope is selected as the final designated centroid. By selecting the designated centroid farthest from the end point of the endoscope, it can be ensured that the designated centroid is appropriately selected under specific conditions, avoiding other possible interference factors, and improving the precision and accuracy of the designated centroid.

[0155] In summary, by screening the designated centroids using the above three screening conditions, the designated centroid D(x1,y1) can be effectively determined. The setting and application of these conditions can help screen out the designated centroids that meet the requirements and exclude interference points that do not meet the conditions, thereby improving the accuracy of screening and the reliability of the results. The above screening method helps to determine the appropriate selection of the designated centroid to meet specific needs and requirements.

[0156] It should be noted that the above screening conditions are only for illustration and are not specifically limited here. As long as the process and method can screen out a specified centroid from a large number of centroids as the end point of the sheath, it is within the protection scope of the embodiments of the present invention.

[0157] As an optional embodiment, step S108, based on the first target position and the second target position, determines the orientation information between the end point of the sheath tube and the reference point, including: obtaining reference coordinate information of the reference point in three-dimensional space and target coordinate information of the target point on the physiological channel in three-dimensional space, wherein the target point is a puncture point determined on the physiological channel based on the planned path; based on the second target position and the reference coordinate information, determining the distance in the orientation information; determining a first straight line from the target point to the reference point according to the target coordinate information and the reference coordinate information, and determining a second straight line from the target point to the end point of the sheath tube according to the target coordinate information and the second target position, and determining the angle between the first straight line and the second straight line as the angle in the orientation information.

[0158] In this embodiment, in the process of determining the orientation information between the end point of the sheath tube and the reference point based on the first target position and the second target position, the reference coordinate information of the reference point in the three-dimensional space and the target coordinate information of the target point on the physiological channel in the three-dimensional space can be obtained. The distance in the orientation information can be determined based on the second target position and the reference coordinate information. It is also possible to determine the first straight line where the target point and the reference point are located based on the target coordinate information and the reference coordinate information, and to determine the second straight line where the target point and the end point of the sheath tube are located based on the target coordinate information and the second target position. The angle between the first straight line and the second straight line can be used to determine the angle in the orientation information.

[0159] Optionally, the target point can be a puncture point (POE) determined on the physiological channel based on the planned path. For example, the target point can be a puncture point on the bronchial wall. If the target is bronchoscopy or bronchoscopic biopsy, the planned path can be executed by LungPro software, and the location of the appropriate POE point can be calculated, that is, the target coordinate information P of the target point. p =(x P ,y P ,z P ).

[0160] It should be noted that the above-mentioned software for planning the path and the POE point determination process are only for illustration and are not specifically limited here.

[0161] For example, the distance between the second target position and the reference coordinate information can be determined by the following formula:

[0162]

[0163] Wherein, d1 can be used to represent the distance between the second target position and the reference coordinate information, that is, the distance between the end point of the sheath and the lesion point; P t =(x t ,yt ,z t ) can be used to represent reference coordinate information, that is, the coordinates of the lesion point; P s =(x s ,y s ,z s ) can be used to represent the second target position, that is, the coordinates of the sheath end point P.

[0164] For another example, the angle between the first straight line and the second straight line can be determined by the following formula:

[0165]

[0166] Wherein, θ can be used to represent the angle between the first straight line and the second straight line, that is, the angle between POE to the lesion point and POE to the end point of the sheath tube; P t Can be used to represent the coordinates of the lesion point; P P Can be used to represent POE point coordinates; P s Can be used to indicate the coordinates of the sheath end point.

[0167] It should be noted that the above process and method for determining the angle and distance in the orientation information are only for illustration and are not specifically limited here.

[0168] As an optional embodiment, the method also includes: determining the distance between the target point and the end point of the endoscope based on the first target position and the target coordinate information, wherein the distance between the target point and the end point of the endoscope is used to determine the degree of match between the actual moving path of the endoscope in the physiological channel and the planned path.

[0169] In this embodiment, the distance between the target point and the end point of the endoscope can also be determined based on the first target position and the target coordinate information, wherein the distance between the target point and the end point of the endoscope can be used to determine the degree of match between the actual moving path of the endoscope in the physiological channel and the above-mentioned pre-planned planning path.

[0170] For example, the distance between the first target position and the target coordinate information may be determined by the following formula:

[0171]

[0172] Wherein, d2 can be used to represent the distance between the first target position and the target coordinate information, that is, the distance between the end of the bronchoscope and the POE point; P q =(x q ,y q ,z q ) can be used to represent the first target position, that is, the coordinates of the end point of the bronchoscope.

[0173] In an embodiment of the present invention, the size of d2 can be used to determine the error between the current actual moving path and the pre-planned planning path, that is, the degree of match between the two. If d2 is smaller, it can be said that the degree of match between the actual moving path and the pre-planned planning path is higher, which means that the puncture is more accurate. If d is larger, it can be said that the degree of match between the actual moving path and the pre-planned planning path is lower, which means that the accuracy of the puncture is lower and corresponding adjustments are required. The ideal situation is that the POE point coincides with the end point of the bronchoscope, that is, the puncture is accurate, d2 = 0.

[0174] In an embodiment of the present invention, an initial two-dimensional image of the endoscope extending into the physiological channel can be obtained from at least two angles. The first image area of ​​the endoscope body and the second image area of ​​the sheath can be segmented from the initial two-dimensional image. From the first image area, the coordinates of the end point of the endoscope in the two-dimensional space can be determined, and the coordinates can be converted into the three-dimensional space to obtain the first target position. From the second image area, the coordinates of the end point of the trachea in the two-dimensional space can be determined, and the coordinates can be converted into the three-dimensional space to obtain the second target position. Through the first target position and the second target position, the distance and angle between the end point of the sheath and the reference point can be determined, so as to determine in real time whether the end point of the sheath reaches the lesion point. In this embodiment, by converting the coordinates from the two-dimensional space to the three-dimensional space, the position of the end point of the endoscope and the end point of the sheath relative to the reference point can be more accurately determined, the accuracy of positioning can be improved, and more three-dimensional information can be provided, including depth and angle, etc., so that the positional relationship between the endoscope and the reference point can be more clearly understood, which helps to guide the diagnosis and treatment operations more accurately. Compared with the confirmation by CArm perspective images at different angles, the above method provides richer three-dimensional information, reduces the limitations in two-dimensional space, and helps to improve the accuracy and safety of diagnosis and treatment. By more intuitively understanding the distance and angle between the end point of the sheath and the reference point, the diagnosis and treatment operations can be guided more accurately, the error can be reduced, and the success rate can be improved. In addition, the technical effect of improving the accuracy of determining the orientation of the end point of the sheath is achieved, and the technical problem of low accuracy in determining the orientation of the end point of the sheath is solved.

[0175] Example 2

[0176] In the embodiment of the present invention, the method for determining the position of the end point of the sheath tube provided by the present invention is described in detail below in conjunction with another optional specific implementation of the bronchoscope.

[0177] At present, the BTPNA procedure is bronchoscopic transparenchymal nodule access, also known as tunnel technology. By drilling holes in the bronchial wall and then establishing tunnels, the working channel in the lung parenchyma reaches the nodule site, which does not rely on natural bronchial channels. In theory, it can reach the entire lung and provide precise treatment. As an interventional surgery, the BTPNA procedure requires C-arm imaging during the operation. As a perspective imaging system, the C-arm can only present a two-dimensional position reference, which will result in the loss of 3D information. In order to determine the spatial position, the C-arm is rotated to image at multiple angles. However, since the C-arm relies on X-ray imaging, there will be potential risks if the operator and the patient are exposed to radiation for a long time.

[0178] Optionally, during the BTPNA procedure, the operator can obtain the current position information in real time in the airway through the image information collected by the lens of the bronchoscope, but once out of the airway, the position information becomes unknown. At this time, in order to determine the position of the sheath in the airway part, the CArm perspective is usually used. However, since the CArm perspective is a two-dimensional image, it is difficult to obtain the spatial position information of the sheath in the airway part, which makes it difficult to determine whether the lesion has been reached, increasing the complexity and risk of the operation. During the operation, if the biopsy needle, sheath and other tools do not accurately reach the lesion location, the CArm perspective is a two-dimensional image, and it is difficult to provide effective spatial information prompts such as angles and distances. This will cause the operator to lack accurate guidance during the operation, increase the difficulty and duration of the operation, and also increase the risk of the operation.

[0179] Optionally, the BTPNA procedure is a puncture of a lung mass performed via bronchoscopy for the diagnosis and treatment of lung disease. In this type of surgery, a preoperative CT scan is often used to confirm the location and size of the lesion and plan the surgical path. During the operation, the DRR image (digitally reconstructed radiographic image, virtual image) generated by the intraoperative CT is registered with the CArm perspective image (real image), and the target position confirmed in the CT is mapped to the CArm perspective image. The purpose of this is to accurately guide the operator to the predetermined lesion location in the actual surgery.

[0180] In the BTPNA surgery, the operator first passes through the lung tissue under the guidance of a bronchoscope according to a pre-planned path to reach the lesion site for relevant operations, such as taking biopsy samples or performing other treatments. During the operation, the operator uses CArm perspectives from different angles to confirm whether the sheath after the airway has accurately reached the lesion site. Through the CArm perspective, the operator can observe the position of the tool in the body in real time, thereby ensuring the accuracy and safety of the operation.

[0181] During the entire BTPNA surgery, the preoperative CT scan, the registration of the DRR image generated by the intraoperative CT and the CArm perspective image, and the confirmation of the position of the CArm view at different angles are all aimed at improving the accuracy of the surgery, reducing the risk of the surgery, and ensuring that the operator can accurately reach the lesion site, so as to effectively diagnose and treat. This method of combining imaging technology and real-time navigation technology will help improve the success rate and safety of BTPNA surgery.

[0182] In a related technology, confirmation can be made through CArm perspectives at different angles, but CArm perspectives are two-dimensional images and difficult to accurately locate. Moreover, when the sheath has not reached the lesion and needs to be adjusted, the CArm perspective is difficult to provide effective information. Therefore, there is still a technical problem of low accuracy in determining the orientation of the sheath end point.

[0183] The present invention proposes a method for relocating three-dimensional coordinates based on a two-dimensional X-ray image. The method divides the sheath in the CArm view of any two angles, locates the end position of the sheath, and then calculates its position in three-dimensional space by back projection. By calculating the angle and distance between the sheath and the lesion, it is determined whether the sheath has reached the lesion. If the sheath has not reached the lesion, this technology can provide angle adjustment and distance prompts to help the operator more easily deliver the sheath to the lesion, thereby improving the accuracy and success rate of the operation. The three-dimensional spatial information is used to accurately locate the sheath position, which can provide more accurate position information and spatial relationships than the two-dimensional CArm perspective view in the related technology, and provide more intuitive and more instructive information for the surgical operator. Through this innovative method, the number of adjustments during the operation can be reduced, the surgical risk can be reduced, and the efficiency and success rate of the operation can be improved, which is of great significance for interventional operations such as BTPNA surgery. Thereby, the technical effect of improving the accuracy of determining the orientation of the end point of the sheath is achieved, and the technical problem of low accuracy in determining the orientation of the end point of the sheath is solved.

[0184] The method is further described below.

[0185] In this embodiment, Figure 2 FIG. 4 is a flow chart of a method for calculating spatial position of a two-dimensional perspective image back projection according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0186] Step S202, obtaining two CArm perspective views to determine the position of the sheath tube end.

[0187] In this embodiment, CArm perspective images at two angles may be acquired, and the position of the sheath tip may be segmented from the images.

[0188] Alternatively, due to the physical characteristics of the bronchoscope, the airway cannot be reached, and the farthest it can reach is the planned POE point, i.e., the puncture point. In this case, the sheath needs to be first passed through the bronchoscope working channel to the POE point, and then, according to the planned puncture angle and distance, the airway is exited from the POE point and reaches the target through the lung parenchyma.

[0189] FIG3(a) is a flow chart of a data preprocessing method in a sheath tip detection process according to an embodiment of the present invention. As shown in FIG3(a), the method may include the following steps:

[0190] Step S301, improving the contrast of the CArm perspective image.

[0191] In this embodiment, the contrast can be improved by calculating an adaptive equalization histogram.

[0192] Step S302, increasing the brightness of the CArm perspective image.

[0193] In this embodiment, gamma conversion may be performed to increase brightness.

[0194] Step S303, removing noise from the CArm perspective image.

[0195] In this embodiment, median filtering may be performed to remove noise in the CArm perspective image.

[0196] For example, Figure 4 is a schematic diagram of a CArm perspective image obtained after data preprocessing and transformation according to an embodiment of the present invention, such as Figure 4 As shown in the figure, the hook in the CArm perspective image after data preprocessing can be a bronchoscope, and the translucent tube next to it is the airway. Through the above data preprocessing steps, the image quality can be improved, the noise interference can be reduced, and a more accurate and reliable image basis can be provided for the detection and positioning of the sheath tip, thereby improving the accuracy and efficiency of detection.

[0197] FIG3(b) is a flow chart of a method for segmenting consumables during sheath tip detection according to an embodiment of the present invention. As shown in FIG3(b), the purpose of the method is to segment a local CArm perspective image with the end of the bronchoscope and the sheath, eliminate the influence of other redundant images, and better perform bronchoscope fitting, threshold segmentation, and contour detection, and ultimately facilitate the acquisition of the specified center of mass coordinates. Compared with processing the above steps on the entire CArm perspective image, this link improves the efficiency and accuracy of acquiring the specified center of mass coordinates. The method may include the following steps:

[0198] Step S304, using adaptive binarization and Otsu's method to perform threshold segmentation on the CArm perspective image after data preprocessing.

[0199] In this embodiment, by combining Otsu's method and adaptive binarization, the bronchoscope part can be extracted from the CArm perspective image more accurately, and the foreground and background can be effectively segmented, which is helpful for subsequent image analysis and processing.

[0200] Alternatively, taking a grayscale image as an example, for an image, we can regard it as a matrix of size M*N, that is, the pixels in the image, each value is the pixel value, where the pixel value is between (0,255). The segmentation threshold of the foreground (i.e. the target) and the background is denoted by T. The proportion of the number of pixels belonging to the foreground in the entire image is denoted by ω0, and the average grayscale of the foreground is denoted by μ0. The proportion of the number of background pixels in the entire image is denoted by ω1, and its average grayscale is denoted by μ1. The total average grayscale of the image is denoted by μ, and the between-class variance is denoted by σ 2 .

[0201] Assuming that the background of the image is dark, and the size of the image is M*N, the number of pixels in the image whose grayscale value is less than the threshold T is recorded as A, and the number of pixels whose grayscale is greater than or equal to the threshold T is recorded as B, then:

[0202]

[0203] A+B=M*N

[0204] 1=ω0+ω1

[0205] μ=ω0μ0+ω1μ1

[0206] σ 2 =ω0(μ0-μ) 2 +ω1(μ1-μ)

[0207] By combining the above formulas, we can get:

[0208] σ 2 =ω0ω1(μ0-μ1)

[0209] The value of the threshold T is traversed through the grayscale interval (0, 255) in turn, and the value of σ 2 The maximum threshold value T is obtained. The image is threshold segmented using the maximum threshold value T. The foreground includes the bronchoscope portion that enters the bronchial passage to the POE position and other pixels that meet the threshold value.

[0210] Step S305 , making the masks of the bronchoscope and consumables in the image continuous by corrosion and expansion.

[0211] In this embodiment, the combined use of erosion and dilation operations can make the mask area of ​​the bronchoscope continuous and more complete, thereby ensuring the accuracy of determining the sheath orientation in the CArm image.

[0212] Figure 5 is a schematic diagram of a bronchoscope with a continuous mask obtained by corrosion expansion according to an embodiment of the present invention, such as Figure 5 As shown, if the bronchoscope in the obtained image is discontinuous, it means that the mask of the bronchoscope may not be continuous. The above situation may affect the identification, positioning and analysis of the bronchoscope, because the discontinuous mask will cause the shape and contour of the bronchoscope to be incomplete, and the information is incomplete, which may cause errors or inaccurate results. Therefore, in order to better extract and process information such as the sheath orientation, the mask of the bronchoscope can be made continuous by corrosion and expansion. This means that in the image, the mask of the bronchoscope is continuous without interruption or breakage to ensure the accurate identification and analysis of the bronchoscope, which can improve the accuracy and reliability of the extraction and processing of the bronchoscope.

[0213] Step S306, through contour detection, find the shape with the largest area and regard it as the bronchoscope.

[0214] In this embodiment, contour detection is performed to find the shape with the largest area in the image and regard it as the bronchoscope.

[0215] Figure 6 is a schematic diagram of a bronchoscope obtained by contour detection according to an embodiment of the present invention, such as Figure 6 As shown, the white hook in the figure may be a bronchoscope obtained by contour detection.

[0216] Step S307, determining the direction of the sheath tube.

[0217] In this embodiment, a fitting curve is obtained by performing polynomial fitting on the bronchoscope image obtained in step S306. Two points on the curve are compared, for example, the x-coordinates are compared to determine the left-right direction of the sheath.

[0218] Step S308, adaptively segmenting the end of the bronchoscope according to the sheath orientation and the overall width of the bronchoscope.

[0219] In this embodiment, the image with the tip of the bronchoscope is adaptively segmented based on the orientation and the overall width of the mirror.

[0220] Figure 7 is a schematic diagram of adaptively segmenting the end of a bronchoscope according to an embodiment of the present invention, such as Figure 7 As shown, it can be judged that the bronchoscope is facing right at this time, and the right part of the bronchoscope can be adaptively segmented. The purpose of obtaining part of the bronchoscope is to achieve a better fitting effect later.

[0221] FIG3(c) is a flow chart of a method for acquiring the coordinates of the sheath tip during the sheath tip detection process according to an embodiment of the present invention. As shown in FIG3(c), the method may include the following steps:

[0222] Step S309, performing bronchoscope tip fitting.

[0223] In this embodiment, a polynomial is fitted to the bronchoscope tip image segmented in the previous step.

[0224] Figure 8 is a schematic diagram of a fitting line of a bronchoscope according to an embodiment of the present invention, such as Figure 8 As shown, through polynomial fitting, we can get Figure 8 The white fitting line in the figure indicates the moving path of the mirror. The tangent line of the mirror tip is taken based on the fitting line, which can be used as the approximate direction of the sheath.

[0225] Step S310, obtaining the bronchoscope tip, the tangent parameters on the bronchoscope tip, and another point on the tangent.

[0226] In this embodiment, non-zero point detection is performed after contour detection in step S306 in sheath segmentation to obtain the end point A of the bronchoscope, the end point tangent parameters (slope k, offset b), and another point O on the tangent.

[0227] Fig. 9 is a schematic diagram of a tangent line of a fitting line at the end point of a bronchoscope according to an embodiment of the present invention, such as Fig. 9 As shown, traverse the pixels, obtain all the endpoints on the right side of the white image, then retain the x coordinate, filter out the points on the fitting line from all the endpoints on the right side, and use the filtered endpoints as the mirror endpoints, that is, the end points of the bronchoscope. The slope k is the slope of the tangent line at this point on the fitting line. The offset b is obtained according to y=kx+b.

[0228] Step S311, adaptively adjusting the Otsu method bias to perform threshold segmentation.

[0229] In this embodiment, the bias of the Otsu method is adaptively adjusted to perform threshold segmentation. Fig.10 is a schematic diagram of a segmentation result of a sheath obtained by threshold segmentation according to an embodiment of the present invention, such as Fig.10 As shown, the segmentation result of the image can be obtained.

[0230] Step S312, obtaining a centroid list.

[0231] In this embodiment, contour detection may be performed on the segmentation result obtained in the above step S311 to obtain the centroid massList of each contour.

[0232]

[0233] Among them, C xIt can be used to represent the horizontal coordinate of each centroid, that is, the horizontal coordinate in the two-dimensional coordinate information; C y It can be used to represent the ordinate of each centroid, that is, the ordinate of the two-dimensional coordinate information, x, y is the serial number corresponding to each centroid, x, y is a positive integer; M can be used to represent the image geometric moment, specifically, M 10 Can be used to represent the first-order horizontal moment, M 01 Can be used to represent the first-order vertical moment, M 00 It can be used to represent the area of ​​contours in an image.

[0234] Step S313: Filter out the designated centroid from the centroid list according to the filtering condition.

[0235] In this embodiment, according to three conditions—specifying that the center of mass must be the circle C m The inner centroid, within the tangent threshold range, and farthest from the end point of the bronchoscope, selects the specified centroid D(x1,y1) in the centroid massList, where circle C m The end point of the endoscope can be taken as a point on the circle, and a point located on the tangent line and at a distance of a preset radius length r from the end point of the endoscope is taken as the center O of the circle.

[0236] Fig.11 is a schematic diagram of determining the end point of the sheath tube from a circle according to an embodiment of the present invention. Fig.11 As shown, the white circle in the figure is circle C m , circle C m The point that satisfies the above three conditions is the designated centroid.

[0237] Fig. 12(a) is a schematic diagram of a bronchoscope tip and a sheath tip according to an embodiment of the present invention. As shown in Fig. 12(a), in the CArm image obtained at the Towards angle, the positions of the sheath tip and the bronchoscope tip are respectively located. Fig. 12(b) is a schematic diagram of the spatial position of a sheath according to an embodiment of the present invention. As shown in Fig. 12(b), in the CArm image obtained at the Ccw angle, the portion between the sheath tip and the bronchoscope tip can be the spatial position of the sheath.

[0238] Step S204, calculating the spatial position of the airway outlet portion of the sheath tube.

[0239] In this embodiment, after the above step S202 is completed, the coordinates of the bronchial end and the sheath end in the two perspective views of Towards and Ccw can be obtained, and the three-dimensional coordinates in space can be calculated based on the two-dimensional coordinates of the two perspective views. Since the sheath after the airway is punctured directly to the target point, the position of the sheath of the airway part in space can be calculated based on the three-dimensional spatial coordinates of the puncture point and the sheath end.

[0240] In the embodiment of the present invention, the next step begins to explain the orthographic projection process:

[0241] Assume that the coordinates of the point P (the end point of the sheath) in the known space are (x s ,y s ,z s ), the process of finding its projection coordinates onto the two-dimensional plane can be expressed as:

[0242]

[0243] Among them, m R Can be used to represent the rotation matrix from CT to CArm, m T The translation matrix that can be used to represent CT to CArm can be determined during the patient registration stage.

[0244] Scaling can be expressed as:

[0245]

[0246] The coordinates are then converted to coordinates on the projected image, specifically expressed as the x-axis and y-axis components of the normalized coordinate point multiplied by the x-axis and y-axis components of the image's focal coordinates, and then added to the x-axis and y-axis components of the image's center point coordinates to obtain the converted image coordinate point.

[0247]

[0248] Among them, x center The abscissa value of the optical center point of the imaging plane of the first image area can be used to represent; center Can be used to represent the ordinate value of the optical center point; x focal It can be used to represent the horizontal focal length of the camera that captures the CArm perspective view; focal Can be used to indicate the focal length in the vertical direction.

[0249] Since there is distortion in CArm imaging, the distortion also needs to be calculated during projection, which can be expressed as:

[0250]

[0251] Among them, a ij , b ij It can be a 4-order distortion parameter, representing the lateral distortion coefficient and the longitudinal distortion coefficient respectively. It has been obtained in the CArm calibration stage.

[0252] Fig.13 is a schematic diagram of a back-projection calculation of spatial coordinates according to an embodiment of the present invention, such as Fig.13As shown, now we know the coordinates of a point in space projected onto two different planes, and we need to find its coordinates in space. According to the back-projection principle, if two rays in space have an intersection, there is only one intersection. Therefore, we can find the coordinates of the bronchoscope end and the sheath end in the Towards and Ccw views, and then back-project them to find their spatial coordinates. Taking the sheath end point as an example, we know that its coordinates in the Towards view are (u1, v1), and its coordinates in the Ccw view are (u2, v2). Let the corresponding point P in space have coordinates (x, y, z). First, according to the inverse process of the above formulas (3) and (4), (u1, v1) and (u2, v2) are dedistorted to obtain (x'1, y'1) and (x'2, y'2).

[0253] The rotation matrix and translation matrix from CT to CArm at the Towards and Ccw angles have been calculated during the patient registration stage, and are:

[0254]

[0255] in, Can be used to represent the rotation matrix at the Towards angle; Can be used to represent the translation matrix at the Towards angle; Can be used to represent the rotation matrix under the Ccw angle; Can be used to represent the translation matrix under Ccw angle.

[0256] It can be obtained from the above formulas (1) and (2):

[0257]

[0258]

[0259] Combining the above equations (5) and (6), we get:

[0260]

[0261] By solving the above linear equations (7), the coordinates of the sheath end point P (x s ,y s ,z s ). Similarly, the coordinates of the end point of the bronchoscope (x q ,y q ,z q ), and finally obtain the starting point and end point coordinates of the sheath outlet.

[0262] Step S206, calculating the distance and angle from the end point of the sheath to the lesion point, and rendering them on the airway tree to achieve visual display.

[0263] In this embodiment, the distance and angle between the end of the sheath and the lesion can be determined by using the coordinates of the end of the bronchoscope and the end of the sheath, and the distance between the end of the bronchoscope and the POE point can also be determined, so that they can be rendered on the bronchial tree for visual display.

[0264] Optionally, read the POE point (puncture point on the bronchial wall) from the Lungpoint software (x P ,y P ,z P ), and can obtain the coordinates of the lesion point (x t ,y t ,z t ), calculate the distance between the coordinates of point P and the coordinates of the lesion point:

[0265]

[0266] Among them, d1 can be used for the distance between the end point of the sheath and the lesion point; P t =(x t ,y t ,z t ) can be used to represent the coordinates of the lesion point; P s =(x s ,y s ,z s ) can be used to represent the coordinates of the sheath end point P.

[0267] Similarly, the distance between the end of the bronchoscope and the POE point can be determined by the following formula:

[0268]

[0269] Among them, d2 can be used to represent the distance between the end of the bronchoscope and the POE point; P q =(x q ,y q ,z q ) can be used to represent the coordinates of the end point of the bronchoscope.

[0270] Optionally, the size of d2 is used to determine the error between the current actual moving path and the pre-planned planning path, that is, the degree of match between the two. If d2 is smaller, it means that the degree of match between the actual moving path and the pre-planned planning path is higher, which means that the puncture is more accurate. If d2 is larger, it means that the degree of match between the actual moving path and the pre-planned planning path is lower, which means that the accuracy of the puncture is lower and corresponding adjustments are required. The ideal situation is that the POE point coincides with the end point of the bronchoscope, that is, the puncture is accurate, d2 = 0.

[0271] For another example, the following formula can be used to determine the POE to the lesion point and the POE to the end of the sheath (x s ,y s ,z s )’s angle:

[0272]

[0273] Among them, θ can be used to represent the angle between POE to the lesion point and POE to the end point of the sheath; P t Can be used to represent the coordinates of the lesion point; P P Can be used to represent POE point coordinates; P s Can be used to indicate the coordinates of the sheath end point.

[0274] It should be noted that the above process and method for determining the angle and distance in the orientation information are only for illustration and are not specifically limited here.

[0275] Fig.14 is a schematic diagram of a bronchial tree visualization display according to an embodiment of the present invention, such as Fig.14 As shown, POE, lesion point and consumable end can be visually displayed on the bronchial tree to assist the operator in determining whether the consumable end has reached the lesion point, or to indicate the distance and angle difference between the current position of the consumable end and the lesion point.

[0276] In an embodiment of the present invention, an initial two-dimensional image of the endoscope extending into the physiological channel can be obtained from at least two angles. The first image area of ​​the endoscope body and the second image area of ​​the sheath can be segmented from the initial two-dimensional image. From the first image area, the coordinates of the end point of the endoscope in the two-dimensional space can be determined, and the coordinates can be converted into the three-dimensional space to obtain the first target position. From the second image area, the coordinates of the end point of the trachea in the two-dimensional space can be determined, and the coordinates can be converted into the three-dimensional space to obtain the second target position. Through the first target position and the second target position, the distance and angle between the end point of the sheath and the reference point can be determined, so as to determine in real time whether the end point of the sheath reaches the lesion point. In this embodiment, by converting the coordinates from the two-dimensional space to the three-dimensional space, the position of the end point of the endoscope and the end point of the sheath relative to the reference point can be more accurately determined, the accuracy of positioning can be improved, and more three-dimensional information can be provided, including depth and angle, etc., so that the positional relationship between the endoscope and the reference point can be more clearly understood, which helps to guide the diagnosis and treatment operations more accurately. Compared with the confirmation by CArm perspective images at different angles, the above method provides richer three-dimensional information, reduces the limitations in two-dimensional space, and helps to improve the accuracy and safety of diagnosis and treatment. By more intuitively understanding the distance and angle between the end point of the sheath and the reference point, the diagnosis and treatment operations can be guided more accurately, the error can be reduced, and the success rate can be improved. In addition, the technical effect of improving the accuracy of determining the orientation of the end point of the sheath is achieved, and the technical problem of low accuracy in determining the orientation of the end point of the sheath is solved.

[0277] Example 3

[0278] The embodiment of the present invention provides a device for determining the position of the end point of the sheath tube. It should be noted that the document generation device of the embodiment of the present invention can be used to perform Figure 1 The method for determining the position of the end point of the sheath tube provided in the embodiment of the present invention is described below.

[0279] Fig.15 is a schematic diagram of a device for determining the position of a sheath tube end point according to an embodiment of the present invention. Fig.15 As shown, the device may include: an acquisition unit 1502 , a segmentation unit 1504 , a first determination unit 1506 , and a second determination unit 1508 .

[0280] The acquisition unit 1502 is used to acquire an initial two-dimensional image of the endoscope during the process of extending into the physiological channel from at least two angles, wherein the endoscope includes a sheath and an endoscope body.

[0281] The segmentation unit 1504 is used to segment the initial two-dimensional image into a first image region including the endoscope body and a second image region including the sheath.

[0282] The first determination unit 1506 is used to determine a first target position of an end point of the endoscope body in three-dimensional space based on the first image area, and to determine a second target position of a sheath end point of the sheath in three-dimensional space based on the second image area.

[0283] The second determination unit 1508 is used to determine the orientation information between the end point of the sheath tube and the reference point based on the first target position and the second target position, wherein the orientation information is used to indicate the distance and angle between the end point of the sheath tube and the reference point, and the reference point is used to indicate the position where an abnormal state exists outside the physiological channel.

[0284] The device for determining the orientation of the end point of the sheath provided in an embodiment of the present invention acquires an initial two-dimensional image of the endoscope in the process of extending into a physiological channel from at least two angles through an acquisition unit 1502; a first image area including an endoscope body and a second image area including the sheath are respectively segmented from the initial two-dimensional image through a segmentation unit 1504; a first determination unit 1506 determines a first target position of the end point of the endoscope body in three-dimensional space based on the first image area, and determines a second target position of the end point of the sheath in three-dimensional space based on the second image area; a second determination unit 1508 determines the orientation information between the end point of the sheath and a reference point based on the first target position and the second target position, thereby solving the technical problem of low accuracy in determining the orientation of the end point of the sheath and achieving the technical effect of improving the accuracy of determining the orientation of the end point of the sheath.

[0285] Optionally, the segmentation unit 1504 may include: a first segmentation module, used to perform threshold segmentation on any initial two-dimensional image to obtain a first image area, wherein the first image area includes each pixel point whose corresponding pixel value in any initial two-dimensional image is less than a preset first segmentation threshold; a second segmentation module, used to perform threshold segmentation on any initial two-dimensional image to obtain a second image area, wherein the second image area includes each pixel point whose corresponding pixel value in any initial two-dimensional image is less than a preset second segmentation threshold.

[0286] Optionally, the first determination unit 1506 may include: a first acquisition module, used to acquire the updated first image area, and determine the orientation information of the endoscope body based on the updated first image area; a first determination module, used to combine the orientation information and the updated first image area to determine the first target position of the end point of the endoscope in three-dimensional space.

[0287] Optionally, the first acquisition module may include: a first processing submodule, used to perform mask continuous processing on the first image area to obtain an updated first image area if the pixel points corresponding to the endoscope body contained in the first image area are in a discrete state; and a second processing submodule, used to use the first image area as the updated first image area if the pixel points corresponding to the endoscope body contained in the first image area are in a continuous state.

[0288] Optionally, the first acquisition module may include: a second processing submodule, used to perform fitting processing on the pixel points belonging to the endoscope body in the updated first image area to obtain the centerline fitting curve of the endoscope body; a first determination submodule, used to determine the orientation information based on the coordinate information of any two pixel points on the centerline fitting curve on the target coordinate axis.

[0289] Optionally, the first determination module may include: a first acquisition submodule, used to acquire two-dimensional coordinate information of the endoscope end point in the first image area based on orientation information, the obtained centerline fitting curve and pixel points belonging to the endoscope body in the first image area; a third processing submodule, used to perform three-dimensional back projection processing on the acquired two-dimensional coordinate information of the endoscope end point in the first image area corresponding to different initial two-dimensional images, so as to obtain the first target position of the endoscope end point in the three-dimensional space.

[0290] Optionally, the first determination unit 1506 may include: a second acquisition module, used to perform contour detection on the second image area, and obtain a centroid data set corresponding to the sheath, wherein the centroid data set includes two-dimensional coordinate information of each centroid corresponding to the sheath in the second image area; a first screening module, used to screen each centroid contained in the centroid data set according to the centerline fitting curve and the end point of the endoscope; a third acquisition module, used to use the screened specified centroid as the sheath end point, and obtain the two-dimensional coordinate information of the sheath end point in the second image area; a first processing module, used to perform three-dimensional back projection processing on the acquired two-dimensional coordinate information of the sheath end point in the second image area corresponding to different initial two-dimensional images, to obtain the second target position of the sheath end point of the sheath in the three-dimensional space.

[0291] Optionally, the first screening module may include: a second acquisition submodule, used to obtain the tangent of the end point of the endoscope on the centerline fitting curve; a second determination submodule, used to determine the corresponding center of the circle on the tangent according to the end point of the endoscope, the orientation information of the endoscope body and the preset radius length; a third determination submodule, used to construct a target circle with the center of the circle and the preset radius length, and based on the target circle, filter each center of mass contained in the center of mass data set according to the screening conditions to determine the specified center of mass.

[0292] Optionally, the second determination unit 1508 may include: a fourth acquisition module, used to acquire reference coordinate information of the reference point in three-dimensional space and target coordinate information of the target point on the physiological channel in three-dimensional space, wherein the target point is a puncture point determined on the physiological channel based on the planned path; a second determination module, used to determine the distance in the orientation information based on the second target position and the reference coordinate information; a third determination module, used to determine a first straight line from the target point to the reference point according to the target coordinate information and the reference coordinate information, and to determine a second straight line from the target point to the end point of the sheath according to the target coordinate information and the second target position, and determine the angle between the first straight line and the second straight line as the angle in the orientation information.

[0293] Optionally, the device may also include: a third determination unit, used to determine the distance between the target point and the end point of the endoscope based on the first target position and the target coordinate information, wherein the distance between the target point and the end point of the endoscope is used to judge the degree of match between the actual moving path of the endoscope in the physiological channel and the planned path.

[0294] The above-mentioned device for determining the position of the end point of the sheath tube may also include a processor and a memory. The above-mentioned units are stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0295] The processor includes a kernel, and the kernel retrieves the corresponding program unit from the memory. One or more kernels can be set to control the same device type of devices to be shut down to perform graceful shutdown by adjusting kernel parameters.

[0296] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one storage chip.

[0297] The processor contains a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the work efficiency of traders can be improved by adjusting the kernel parameters.

[0298] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0299] Example 4

[0300] According to an embodiment of the present invention, a computer-readable storage medium is further provided, on which a program is stored. When the program is executed by a processor, a method for determining the position of a sheath end point is implemented.

[0301] Example 5

[0302] According to an embodiment of the present invention, a processor is further provided. The processor is used to run a program, wherein the method for determining the position of the end point of the sheath tube is executed when the program is run.

[0303] Example 6

[0304] Fig.16 is a schematic diagram of an electronic device for determining the position of a sheath end point according to an embodiment of the present invention, such as Fig.16 As shown, an embodiment of the present invention further provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the above embodiments are implemented.

[0305] Example 7

[0306] According to an embodiment of the present invention, a computer program product is also provided. Optionally, in this embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the method for determining the position of the sheath end point of the embodiment of the present application is implemented.

[0307] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0308] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0309] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0310] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0311] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0312] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0313] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0314] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0315] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0316] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for determining the position of a sheath end point, characterized in that: include: Acquiring an initial two-dimensional image of the endoscope during the process of extending into the physiological channel from at least two angles, wherein the endoscope comprises a sheath and an endoscope body; Segmenting a first image region including the endoscope body and a second image region including the sheath from the initial two-dimensional image respectively; Based on the first image area, determining a first target position of an end point of the endoscope body in a three-dimensional space, and based on the second image area, determining a second target position of a sheath end point of the sheath in the three-dimensional space; Based on the first target position and the second target position, the orientation information between the end point of the sheath tube and the reference point is determined, wherein the orientation information is used to indicate the distance and angle between the end point of the sheath tube and the reference point, and the reference point is used to indicate the position where an abnormal state exists outside the physiological channel.

2. The determination method according to claim 1, characterized in that: Segmenting a first image region including the endoscope body and a second image region including the sheath from the initial two-dimensional image respectively, comprising: Performing threshold segmentation on any initial two-dimensional image to obtain the first image region, wherein the first image region includes each pixel point whose corresponding pixel value in any initial two-dimensional image is less than a preset first segmentation threshold; Threshold segmentation is performed on any of the initial two-dimensional images to obtain the second image area, wherein the second image area includes each pixel point whose corresponding pixel value in any of the initial two-dimensional images is less than a preset second segmentation threshold.

3. The determination method according to claim 2, characterized in that: Determining a first target position of an end point of the endoscope body in three-dimensional space based on the first image area includes: Acquire the updated first image area, and determine the orientation information of the endoscope body based on the updated first image area; The first target position of the end point of the endoscope in the three-dimensional space is determined by combining the orientation information and the updated first image area.

4. The determination method according to claim 3, characterized in that: Obtaining the updated first image region, including: If the pixel points corresponding to the endoscope body contained in the first image area are in a discrete state, performing mask continuous processing on the first image area to obtain an updated first image area; If the pixel points corresponding to the endoscope body contained in the first image area are in a continuous state, the first image area is used as the updated first image area.

5. The determination method according to claim 3, characterized in that: Determining the orientation information of the endoscope body based on the updated first image area includes: Performing fitting processing on the pixel points belonging to the endoscope body in the updated first image area to obtain a centerline fitting curve of the endoscope body; The orientation information is determined based on coordinate information of any two pixel points on the centerline fitting curve on the target coordinate axis.

6. The determination method according to claim 5, characterized in that: Combining the orientation information and the updated first image region to determine a first target position of the end point of the endoscope in the three-dimensional space includes: Based on the orientation information, the obtained centerline fitting curve and the pixel points belonging to the endoscope body in the first image area, acquiring the two-dimensional coordinate information of the endoscope end point in the first image area; The acquired two-dimensional coordinate information of the end point of the endoscope in the first image area corresponding to different initial two-dimensional images is subjected to three-dimensional back projection processing to obtain a first target position of the end point of the endoscope in the three-dimensional space.

7. The determination method according to claim 6, characterized in that: Determining a second target position of a sheath tube end point of the sheath tube in the three-dimensional space based on the second image area includes: Performing contour detection on the second image area to obtain a centroid data set corresponding to the sheath, wherein the centroid data set includes two-dimensional coordinate information of each centroid corresponding to the sheath in the second image area; Screening each centroid contained in the centroid data set according to the centerline fitting curve and the end point of the endoscope; Using the selected designated centroid as the end point of the sheath tube, and obtaining the two-dimensional coordinate information of the end point of the sheath tube in the second image area; The acquired two-dimensional coordinate information of the sheath tube end point in the second image area corresponding to different initial two-dimensional images is subjected to three-dimensional back projection processing to obtain the second target position of the sheath tube end point in the three-dimensional space.

8. The determination method according to claim 7, characterized in that: The centroids contained in the centroid data set are screened according to the centerline fitting curve and the end point of the endoscope, including: Obtaining a tangent line of the end point of the endoscope on the centerline fitting curve; Determine the corresponding center of the circle on the tangent line according to the end point of the endoscope, the orientation information of the endoscope body and the preset radius length; A target circle is constructed with the center of the circle and a preset radius length, and each center of mass contained in the center of mass data set is screened according to a screening condition based on the target circle to determine the designated center of mass.

9. The determination method according to claim 8, characterized in that: The screening conditions at least include: The designated centroid is located within the target circle; The distance between the specified centroid and the tangent line is less than or equal to a first distance threshold; The distance between the designated centroid and the end point of the endoscope is greater than or equal to the distance between any centroid in the centroid data set and the end point of the endoscope.

10. The determination method according to claim 1, characterized in that: Determining the position information between the end point of the sheath tube and a reference point based on the first target position and the second target position includes: Acquire reference coordinate information of the reference point in the three-dimensional space and target coordinate information of a target point on the physiological channel in the three-dimensional space, wherein the target point is a puncture point determined on the physiological channel based on a planned path; Determine the distance in the orientation information based on the second target position and the reference coordinate information; According to the target coordinate information and the reference coordinate information, a first straight line from the target point to the reference point is determined, and according to the target coordinate information and the second target position, a second straight line from the target point to the end point of the sheath is determined, and the angle between the first straight line and the second straight line is determined as the angle in the orientation information.

11. The determination method according to claim 10, characterized in that: The method further comprises: Based on the first target position and the target coordinate information, the distance between the target point and the end point of the endoscope is determined, wherein the distance between the target point and the end point of the endoscope is used to judge the degree of matching between the actual moving path of the endoscope in the physiological channel and the planned path.

12. A device for determining the position of a sheath end point, characterized in that: include: An acquisition unit, used for acquiring an initial two-dimensional image of the endoscope during the process of extending into the physiological channel from at least two angles, wherein the endoscope comprises a sheath and an endoscope body; a segmentation unit, configured to segment a first image region including the endoscope body and a second image region including the sheath from the initial two-dimensional image; a first determining unit, configured to determine a first target position of an end point of the endoscope body of the endoscope in a three-dimensional space based on the first image area, and to determine a second target position of a sheath end point of the sheath in the three-dimensional space based on the second image area; The second determination unit is used to determine the orientation information between the end point of the sheath tube and the reference point based on the first target position and the second target position, wherein the orientation information is used to represent the distance and angle between the end point of the sheath tube and the reference point, and the reference point is used to represent the position where an abnormal state exists outside the physiological channel.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method for determining the position of the sheath end point according to any one of claims 1 to 11.

14. An electronic device, characterized in that: The device comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method for determining the position of the distal end point of the sheath tube according to any one of claims 1 to 11.

15. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the method for determining the position of the distal end point of the sheath tube according to any one of claims 1 to 11.

Images