Methods, devices, processors, and electronic equipment for determining the orientation of the sheath tip

Abstract

This invention discloses a method, apparatus, processor, and electronic device for determining the orientation of the sheath tip. The method includes: acquiring initial two-dimensional images of an endoscope during insertion into a physiological channel from at least two angles, wherein the endoscope includes a sheath and an endoscope body; segmenting a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional images; determining a first target position in three-dimensional space for the endoscope tip of the endoscope body based on the first image region, and determining a second target position in three-dimensional space for the sheath tip exiting the sheath based on the second image region; and determining the orientation information between the sheath tip exiting the sheath and a reference point based on the first and second target positions. This invention solves the technical problem of low accuracy in determining the orientation of the sheath tip.

Description

Technical Field

[0001] This invention relates to the field of endoscopic examination technology, and more specifically, to a method, apparatus, processor, and electronic device for determining the orientation of the distal end of a sheath. Background Technology

[0002] Currently, with the continuous development of endoscopic examination technology, during endoscopic examinations and treatments, the high-definition imaging and flexible operation of the endoscope help doctors accurately diagnose and treat lesions (i.e., reference points). Computed tomography (CT) can quickly identify lesions and determine the planned path to them, improving the accuracy and safety of diagnosis and treatment. It can also map lesions onto a C-arm (C-Arm / CArm) perspective view to guide diagnosis and treatment in real time.

[0003] In related technologies, during diagnosis and treatment according to a planned path, CArm perspective images at different angles can be used to determine whether the distal end of the sheath has reached the lesion. However, confirming this solely through two-dimensional CArm perspective images at different angles presents several problems. First, CArm perspective images are two-dimensional images, making it difficult to provide accurate three-dimensional spatial location information. Judging solely based on CArm perspective images can easily lead to inaccuracies and errors in positioning. Second, when the distal end of the sheath does not reach the lesion and adjustment is needed, CArm perspective images cannot provide sufficient information for precise adjustment. Because CArm perspective images cannot display depth and three-dimensional information, multiple adjustments may be required to accurately guide the distal end of the sheath to the target position, increasing the difficulty and risk of diagnostic and treatment procedures. Therefore, the technical problem of low accuracy in determining the orientation of the distal end of the sheath remains.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, processor, and electronic device for determining the orientation of the sheath tip, in order to at least solve the technical problem of low accuracy in determining the orientation of the sheath tip.

[0006] According to one aspect of the present invention, a method for determining the orientation of the distal end of a sheath is provided, comprising: acquiring initial two-dimensional images of an endoscope during insertion into a physiological channel from at least two angles, wherein the endoscope includes a sheath and an endoscope body; segmenting a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional images; determining a first target position of the distal end of the endoscope body in three-dimensional space based on the first image region, and determining a second target position of the distal end of the sheath in three-dimensional space based on the second image region; and determining orientation information between the distal end of the sheath and a reference point based on the first target position and the second target position, wherein the orientation information is used to represent the distance and angle between the distal end of the sheath and the reference point, and the reference point is used to represent the location of an abnormal state outside the physiological channel.

[0007] Optionally, segmenting a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional image includes: performing threshold segmentation on any initial two-dimensional image to obtain the first image region, wherein the first image region contains each pixel point in any initial two-dimensional image whose corresponding pixel value is less than a preset first segmentation threshold; performing threshold segmentation on any initial two-dimensional image to obtain the second image region, wherein the second image region contains each pixel point in any initial two-dimensional image whose corresponding pixel value is less than a preset second segmentation threshold.

[0008] Optionally, determining the first target position of the endoscope tip in three-dimensional space based on the first image region includes: acquiring an updated first image region and determining the orientation information of the endoscope body based on the updated first image region; and combining the orientation information and the updated first image region to determine the first target position of the endoscope tip in three-dimensional space.

[0009] Optionally, obtaining the updated first image region includes: if the pixels in the first image region corresponding to the endoscope body are in a discrete state, then the first image region is subjected to continuous masking processing to obtain the updated first image region; if the pixels in the first image region corresponding to the endoscope body are in a continuous state, then the first image region is used as the updated first image region.

[0010] Optionally, the orientation information of the endoscope body is determined based on the updated first image region, including: performing fitting processing on the pixels belonging to the endoscope body in the updated first image region to obtain the centerline fitting curve of the endoscope body; and determining the orientation information based on the coordinate information of any two pixels on the centerline fitting curve on the target coordinate axis.

[0011] Optionally, by combining orientation information and the updated first image region, the first target position of the endoscope tip in three-dimensional space is determined, including: obtaining the two-dimensional coordinate information of the endoscope tip in the first image region based on orientation information, the obtained centerline fitting curve, and the pixels belonging to the endoscope body in the first image region; performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the endoscope tip in the first image region corresponding to different initial two-dimensional images to obtain the first target position of the endoscope tip in three-dimensional space.

[0012] Optionally, determining the second target position of the sheath tip in three-dimensional space based on the second image region includes: performing contour detection on the second image region to obtain a centroid dataset corresponding to the sheath, wherein the centroid dataset includes the two-dimensional coordinate information of each centroid corresponding to the sheath in the second image region; filtering each centroid contained in the centroid dataset according to the centerline fitting curve and the endoscope tip; taking the selected centroid as the sheath tip and obtaining the two-dimensional coordinate information of the sheath tip in the second image region; and performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the sheath tip in the second image region corresponding to different initial two-dimensional images to obtain the second target position of the sheath tip in three-dimensional space.

[0013] Optionally, based on the centerline fitting curve and the endoscope end point, the centroids contained in the centroid dataset are filtered, including: obtaining the tangent line of the endoscope end point on the centerline fitting curve; determining the corresponding circle center on the tangent line based on the endoscope end point, the orientation information of the endoscope body, and the preset radius length; constructing a target circle with the circle center and the preset radius length, and filtering the centroids contained in the centroid dataset according to the filtering conditions based on the target circle to determine the specified centroid.

[0014] Optionally, the filtering criteria may include at least: specifying that the centroid is located within the target circle; specifying that the distance between the centroid and the tangent is less than or equal to a first distance threshold; and specifying that the distance between the centroid and the end-end point is greater than or equal to the distance between any centroid in the centroid dataset and the end-end point.

[0015] Optionally, based on the first target position and the second target position, the orientation information between the sheath tip and the reference point is determined, including: acquiring the reference coordinate information of the reference point in three-dimensional space and the target coordinate information of the target point on the physiological channel in three-dimensional space, wherein the target point is the puncture point determined on the physiological channel based on the planned path; determining the distance in the orientation information based on the second target position and the reference coordinate information; determining the first straight line from the target point to the reference point according to the target coordinate information and the reference coordinate information, and determining the second straight line from the target point to the sheath tip according to the target coordinate information and the second target position, and determining the angle between the first straight line and the second straight line as the angle in the orientation information.

[0016] Optionally, the method further includes: determining the distance between the target point and the endoscope tip based on the first target location and target coordinate information, wherein the distance between the target point and the endoscope tip is used to determine the degree of matching between the actual movement path of the endoscope in the physiological channel and the planned path.

[0017] According to another aspect of the present invention, a device for determining the orientation of the sheath tip is also provided, comprising: an acquisition unit, configured to acquire initial two-dimensional images of an endoscope during insertion into a physiological channel from at least two angles, wherein the endoscope includes a sheath and an endoscope body; a segmentation unit, configured to segment a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional images; a first determination unit, configured to determine a first target position of the endoscope tip of the endoscope body in three-dimensional space based on the first image region, and to determine a second target position of the sheath tip of the sheath in three-dimensional space based on the second image region; and a second determination unit, configured to determine orientation information between the sheath tip and a reference point based on the first target position and the second target position, wherein the orientation information represents the distance and angle between the sheath tip and the reference point, and the reference point represents the location where an abnormal state exists outside the physiological channel.

[0018] According to another aspect of the present invention, a processor is also provided. The processor is used to run a program, wherein the program executes the method for determining the orientation of the sheath tip point according to the embodiments of the present invention.

[0019] According to another aspect of the present invention, a computer-readable storage medium is also provided, which stores a plurality of instructions adapted for a processor to load and execute any one of the above-described methods for determining the orientation of the sheath end point.

[0020] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform any of the above-described methods for determining the orientation of the sheath end point.

[0021] According to another aspect of the present invention, a computer program product is also provided. This computer program product includes a computer program that, when executed by a processor, implements the method for determining the orientation of the sheath tip point described in the embodiments of this application.

[0022] In this embodiment of the invention, initial two-dimensional images of the endoscope during insertion into the physiological channel can be acquired from at least two angles. From the initial two-dimensional image, a first image region of the endoscope body and a second image region of the sheath can be segmented. From the first image region, the coordinates of the endoscope tip in two-dimensional space can be determined, and these coordinates can be transformed into three-dimensional space to obtain a first target position. From the second image region, the coordinates of the tracheal tip in two-dimensional space can be determined, and these coordinates can be transformed into three-dimensional space to obtain a second target position. Using the first and second target positions, the distance and angle between the sheath tip and the reference point can be determined, thereby determining in real time whether the sheath tip has reached the lesion point. In this embodiment, by transforming the coordinates from two-dimensional space to three-dimensional space, the positions of the endoscope tip and the sheath tip relative to the reference point can be determined more accurately, improving the accuracy of positioning and providing more stereoscopic information, including depth and angle, etc., enabling a clearer understanding of the positional relationship between the endoscope and the reference point, and facilitating more precise guidance for diagnostic and treatment operations. Compared to relying on CArm perspective views at different angles for confirmation, the method described above provides richer three-dimensional information, reduces the limitations of two-dimensional space, and helps improve the accuracy and safety of diagnosis and treatment. By more intuitively understanding the distance and angle between the sheath tip and the reference point, diagnostic and treatment procedures can be guided more precisely, reducing errors and increasing the success rate. This achieves the technical effect of improving the accuracy of determining the orientation of the sheath tip, solving the technical problem of low accuracy in determining the orientation of the sheath tip. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0024] Figure 1 This is a flowchart of a method for determining the orientation of the end point of a sheath according to an embodiment of the present invention;

[0025] Figure 2This is a flowchart of a method for calculating spatial position by back projection of a two-dimensional perspective view according to an embodiment of the present invention;

[0026] Figure 3(a) is a flowchart of a data preprocessing method in the detection process of a sheath tip according to an embodiment of the present invention;

[0027] Figure 3(b) is a flowchart of a consumable segmentation method during sheath tip detection according to an embodiment of the present invention;

[0028] Figure 3(c) is a flowchart of a method for obtaining the coordinates of the sheath tip during sheath tip detection according to an embodiment of the present invention;

[0029] Figure 4 This is a schematic diagram of a CArm perspective image obtained after data preprocessing and transformation according to an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of a bronchoscope with a continuous mask obtained by corrosion and expansion according to an embodiment of the present invention.

[0031] Figure 6 This is a schematic diagram of a bronchoscope obtained by contour detection according to an embodiment of the present invention.

[0032] Figure 7 This is a schematic diagram of an adaptive segmentation of the end of a bronchoscope according to an embodiment of the present invention;

[0033] Figure 8 This is a schematic diagram of the fitting line of a bronchoscope according to an embodiment of the present invention.

[0034] Figure 9 This is a schematic diagram of the tangent of a fitting line at the end point of a bronchoscope according to an embodiment of the present invention.

[0035] Figure 10 This is a schematic diagram of the segmentation result of the sheath obtained by threshold segmentation according to an embodiment of the present invention;

[0036] Figure 11 This is a schematic diagram illustrating the determination of the end point of the sheath from within a circle according to an embodiment of the present invention;

[0037] Figure 12(a) is a schematic diagram of the end of a bronchoscope and the end of a sheath according to an embodiment of the present invention;

[0038] Figure 12(b) is a schematic diagram of the spatial position of a sheath according to an embodiment of the present invention;

[0039] Figure 13 This is a schematic diagram of a back projection calculation of spatial coordinates according to an embodiment of the present invention;

[0040] Figure 14This is a schematic diagram of a bronchial tree visualization according to an embodiment of the present invention;

[0041] Figure 15 This is a schematic diagram of a device for determining the orientation of the sheath tip according to an embodiment of the present invention;

[0042] Figure 16 This is a schematic diagram of an electronic device for determining the orientation of the sheath tip according to an embodiment of the present invention. Detailed Implementation

[0043] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0044] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] Example 1

[0046] According to an embodiment of the present invention, a method for determining the orientation of the sheath tip is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] Figure 1 This is a flowchart of a method for determining the orientation of the sheath tip according to an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps:

[0048] Step S102: Obtain initial two-dimensional images of the endoscope during insertion into the physiological channel from at least two angles.

[0049] In the technical solution provided in step S102 of the present invention, the physiological channel can be the working channel of the endoscope. If the area to be examined by the endoscope is the bronchus of the lung, the physiological channel can be the airway of the lung, or it can be called the bronchial channel. It should be noted that the above-mentioned physiological channel and endoscope are only illustrative examples and are not specifically limited here. They can be determined according to the actual location of the lesion and the surgery performed.

[0050] Optionally, the initial two-dimensional image can be a two-dimensional CArm perspective view, also known as a CArm view or CArm perspective image. For example, it can be a CArm perspective image obtained after data preprocessing and transformation.

[0051] Optionally, the angle can be a backward angle or a counter-clockwise angle (Ccw). It should be noted that the above angles are only illustrative examples and are not specifically limited here.

[0052] Optionally, the endoscope may include a sheath and an endoscope body. The sheath protects the endoscope body's lens, light source, and other components from external environmental influences, preventing damage or contamination. It accurately guides the endoscope body to a specific position, ensuring the accuracy and safety of the surgery or examination. It also fixes the endoscope body in place, preventing unnecessary shaking or displacement during operation and ensuring image clarity and stability. The sheath can also be referred to as a consumable. The endoscope body may include components such as a lens, light source, and sensors, allowing observation of the internal conditions of the physiological passage through its lens and light source.

[0053] In this embodiment, initial two-dimensional images of the endoscope penetrating the physiological channel can be obtained from at least two angles.

[0054] Optionally, an endoscope can be inserted into a patient's physiological passage, such as a bronchus in the lungs, for examination or treatment. The endoscope's lens and light source can capture real-time images of the interior of the physiological passage and display them on a monitor. Imaging with a C-arm X-ray machine can obtain CArm fluoroscopic views of the endoscope inserted into the physiological passage at different angles. These CArm fluoroscopic images can show the endoscope's position, the structure of the physiological passage, and any potential lesions or abnormalities, providing real-time visual guidance.

[0055] Optionally, after obtaining the CArm fluoroscopic images, to ensure image quality and reduce noise, thus facilitating subsequent image analysis and processing, data preprocessing can be performed to optimize image quality and accuracy, enabling clearer observation of the endoscope's position and condition within the physiological channel. Based on the preprocessed initial two-dimensional images, the endoscope's position and the physiological channel can be observed from different angles to determine the endoscope's precise location and orientation.

[0056] For example, if the target physiological pathway is the bronchus of the lung, in bronchoscopic transparenchymal nodule access (TDR) surgery, CArm fluoroscopy is primarily used to guide the surgical procedure in real time, confirming the position of the endoscope and whether the sheath has reached the lesion. After preoperative CT scans determine the lesion and plan the pathway, the digitally reconstructed radiograph (DRR) image generated by the intraoperative CT scan can be registered with the CArm fluoroscopy image, mapping the confirmed lesion onto the CArm fluoroscopy. The operator performs the surgery based on this information on the CArm fluoroscopy, using different angles of CArm fluoroscopy during the procedure to confirm the position of the endoscope and whether the tip of the sheath has accurately reached the lesion.

[0057] It should be noted that the above scenarios for obtaining CArm perspective views from different angles are only illustrative examples, and no specific limitations are imposed here. The specific scenarios can be determined according to the actual situation.

[0058] Step S104: From the initial two-dimensional image, a first image region containing the endoscope body and a second image region containing the sheath are segmented.

[0059] In the technical solution provided by step S104 of the present invention, the first image region can be an image region containing the endoscope body. The second image region can be an image region containing the sheath. The first image region and the second image region can be images of the endoscope body and the sheath, respectively, adaptively segmented according to the orientation and width of the endoscope.

[0060] In this embodiment, after acquiring initial two-dimensional images of the endoscope inserted into the physiological channel from at least two angles, local images containing the endoscope body and sheath can be segmented to obtain a first image region and a second image region.

[0061] Optionally, thresholding is an important step after preprocessing the initial 2D image. The purpose of thresholding is to separate the endoscope and sheath from the initial 2D image by setting different segmentation thresholds, eliminating the influence of other redundant features in the initial 2D image, thereby improving endoscope fitting, thresholding, and contour detection. Compared to performing these steps on the entire initial 2D image, local segmentation can improve the efficiency and accuracy of the entire process.

[0062] Step S106: Based on the first image region, determine the first target position of the endoscope tip of the endoscope body in three-dimensional space, and based on the second image region, determine the second target position of the sheath tip of the sheath tube in three-dimensional space.

[0063] In the technical solution provided in step S106 of the present invention, the endoscope body may include the endoscope tip, also known as the endoscope end point. For example, if the endoscope body is a bronchoscope, the endoscope tip may be the bronchoscope end or the bronchoscope tip. The first target position may be the coordinates of the endoscope tip in three-dimensional space, and the second target position may be the coordinates of the sheath tip in three-dimensional space. The sheath tip may also be called the sheath tip and can be represented by point P. The sheath tip may also be called the consumable end.

[0064] In this embodiment, after segmenting a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional image, the position of the endoscope tip in two-dimensional space can be determined based on the first image region, and then the first target position of the endoscope tip in three-dimensional space can be determined. Alternatively, the position of the sheath tip in two-dimensional space can be determined based on the second image region, and then the second target position of the sheath tip in three-dimensional space can be determined.

[0065] Optionally, by segmenting the image regions of the endoscope body and sheath from the initial two-dimensional image, the positions of the endoscope tip and sheath tip in two-dimensional space can be determined based on these two regions, and then converted into target positions in three-dimensional space. This process helps improve the accuracy and safety of endoscopic operations. That is, by segmenting the image regions of the endoscope body and sheath, the positions of the endoscope tip and sheath tip in the two-dimensional image can be accurately located. Converting these two-dimensional positions into target positions in three-dimensional space provides a clearer understanding of the positional relationship between the endoscope tip and sheath tip and a reference point (such as the lesion).

[0066] In this embodiment of the invention, by converting from two-dimensional to three-dimensional space, the spatial positions of the endoscope tip and the sheath tip can be determined more accurately, improving the precision of positioning. Converting the two-dimensional position to a three-dimensional position provides more stereoscopic information and angles, enabling the surgeon to more clearly understand the relative positions of the endoscope and sheath in space, thus facilitating more precise guidance of the surgical procedure. Determining the positions of the endoscope tip and the sheath tip in three-dimensional space reduces the risk of accidental cutting or damage to healthy tissue, improving the safety of the surgery.

[0067] Therefore, compared to related technologies that rely solely on two-dimensional CArm perspective views to determine the movement of the sheath tip to the lesion, the conversion from two-dimensional to three-dimensional can provide more accurate positioning information and more spatial information, which helps improve the precision and safety of surgical procedures, provides better operational guidance and support, and helps improve the accuracy and success rate of endoscopic procedures.

[0068] Step S108: Based on the first target position and the second target position, determine the orientation information between the end point of the sheath tube and the reference point.

[0069] In the technical solution provided by step S108 of the present invention, the orientation information can be used to describe the relative positional relationship between the sheath tip and the reference point, and can be used to represent the distance and angle between the sheath tip and the reference point. The reference point can be used to represent the location of an abnormal state outside the physiological channel, and can also be called a lesion point or target point, used to represent the location of the lesion outside the physiological channel.

[0070] In this embodiment, after determining the first target position based on the first image region and the second target position based on the second image region, the orientation information between the end point of the sheath and the reference point can be determined based on the first target position and the second target position.

[0071] Optionally, by determining the first target location and the second target location, the azimuth information between the sheath tip and the reference point (such as the lesion point), including distance and angle, can be further determined. This process is crucial for positioning and guidance during endoscopic procedures.

[0072] Optionally, the positions of the endoscope tip and the sheath tip in two-dimensional and three-dimensional space are determined based on the first and second image regions. The first and second target positions represent the coordinates of the endoscope tip and the sheath tip in space, respectively, providing a basis for subsequent orientation information. Using the first and second target positions, the distance and angle information between the sheath tip and the reference point can be calculated to describe their relative positional relationship.

[0073] In this embodiment of the invention, the sheath in the CArm view from any two angles is segmented and located at the end of the sheath. Its position in three-dimensional space is then determined by converting from two-dimensional to three-dimensional space. The angle and distance between the sheath and the lesion are calculated to determine if the sheath has reached the lesion. If the sheath has not reached the lesion, appropriate angle adjustments and distance prompts can be provided to help the operator more easily deliver the sheath to the lesion, thereby improving the accuracy and success rate of the surgery. This method, utilizing three-dimensional spatial information for precise sheath positioning, provides more accurate positional information and spatial relationships compared to two-dimensional CArm perspective views in related technologies, offering more intuitive and instructive information for the surgical operator. It can reduce the number of adjustments during surgery, lower surgical risks, and improve surgical efficiency and success rate, which is of great significance for interventional procedures such as BTPNA.

[0074] In steps S102 to S108 of this embodiment of the invention, initial two-dimensional images of the endoscope during insertion into the physiological channel can be acquired from at least two angles. From the initial two-dimensional images, a first image region of the endoscope body and a second image region of the sheath can be segmented. From the first image region, the coordinates of the endoscope tip in two-dimensional space can be determined, and these coordinates can be transformed into three-dimensional space to obtain a first target position. From the second image region, the coordinates of the tracheal tip in two-dimensional space can be determined, and these coordinates can be transformed into three-dimensional space to obtain a second target position. Using the first and second target positions, the distance and angle between the sheath tip and the reference point can be determined, thereby determining in real time whether the sheath tip has reached the lesion point. In this embodiment, by transforming the coordinates from two-dimensional space to three-dimensional space, the positions of the endoscope tip and the sheath tip relative to the reference point can be determined more accurately, improving the accuracy of positioning and providing more stereoscopic information, including depth and angle, etc., enabling a clearer understanding of the positional relationship between the endoscope and the reference point, and helping to more accurately guide diagnostic and treatment operations. Compared to relying on CArm perspective views at different angles for confirmation, the method described above provides richer three-dimensional information, reduces the limitations of two-dimensional space, and helps improve the accuracy and safety of diagnosis and treatment. By more intuitively understanding the distance and angle between the sheath tip and the reference point, diagnostic and treatment procedures can be guided more precisely, reducing errors and increasing the success rate. This achieves the technical effect of improving the accuracy of determining the orientation of the sheath tip, solving the technical problem of low accuracy in determining the orientation of the sheath tip.

[0075] The embodiments of the present invention will now be described in detail with reference to the steps described above.

[0076] As an optional embodiment, step S104, which involves segmenting a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional image, includes: performing threshold segmentation on any initial two-dimensional image to obtain the first image region, wherein the first image region contains each pixel point in any initial two-dimensional image whose corresponding pixel value is less than a preset first segmentation threshold; and performing threshold segmentation on any initial two-dimensional image to obtain the second image region, wherein the second image region contains each pixel point in any initial two-dimensional image whose corresponding pixel value is less than a preset second segmentation threshold.

[0077] In this embodiment, during the process of segmenting the first image region and the second image region from the initial two-dimensional image, the initial two-dimensional image can be segmented according to a first segmentation threshold, with the image region containing pixels with pixel values ​​less than the first segmentation threshold being designated as the first image region. Alternatively, the initial two-dimensional image can be segmented according to a second segmentation threshold, with the image region containing pixels with pixel values ​​less than the second segmentation threshold being designated as the second image region. The first image region contains all pixels in the initial two-dimensional image whose corresponding pixel values ​​are less than the preset first segmentation threshold. The first segmentation threshold can be a segmentation threshold for segmenting the area containing the endoscope body. The second image region contains all pixels in the initial two-dimensional image whose corresponding pixel values ​​are less than the preset second segmentation threshold. The second segmentation threshold can be a segmentation threshold for segmenting the area containing the sheath.

[0078] In this embodiment of the invention, data preprocessing can be performed before segmenting the initial two-dimensional image. Data preprocessing is an important step aimed at improving image quality and reducing noise, thereby facilitating subsequent image analysis and processing. For example, during data preprocessing, steps such as enhancing contrast, increasing brightness, and removing noise can be performed; these are merely illustrative examples and not intended to impose specific limitations.

[0079] Optionally, contrast can be improved by calculating an adaptive equalization histogram. Adaptive equalization histogram is a method for enhancing image contrast. In the above process, by analyzing and processing the histogram of the CArm perspective, adaptive contrast enhancement can be performed based on the brightness of local image regions, thereby making the CArm perspective clearer and highlighting details.

[0080] Optionally, gamma transformation can be used to enhance brightness. By adjusting the brightness and contrast of an image, its visual effect can be improved. In the above process, gamma transformation can effectively enhance the brightness of the image, making the sheath tip, endoscope body, etc., more prominent, which is beneficial for subsequent detection and analysis.

[0081] Optionally, noise can be removed by performing median filtering. This is achieved by sorting the pixel values ​​around a pixel in the CArm perspective image and taking the median value. In this process, median filtering effectively removes noise from the image, making the target area clearer and more accurate, which is beneficial for subsequent target detection and localization.

[0082] The above data preprocessing steps can improve image quality and reduce noise interference, providing a more accurate and reliable image basis for the detection and positioning of the sheath tip, thereby improving the accuracy and efficiency of detection. It should be noted that the methods described above for improving contrast, brightness, and removing noise are merely illustrative examples and are not intended to be specific. Any process or method that can improve the image quality of a CArm perspective view is within the scope of protection of this invention.

[0083] In this embodiment of the invention, threshold segmentation is a crucial step after data preprocessing. The purpose of threshold segmentation is to separate the local image region containing the endoscope body (the first image region) and the local image region containing the sheath (the second image region). This eliminates the influence of other redundant pixels in the image, thereby improving endoscope fitting, threshold segmentation, and contour detection, ultimately facilitating the acquisition of the coordinates of the specified centroid. Compared to performing the above steps on the entire fluoroscopic image, local segmentation improves the efficiency and accuracy of acquiring the sheath's end point.

[0084] Optionally, during thresholding, adaptive binarization and Otsu's method can be used to determine the first and second segmentation thresholds based on the grayscale distribution characteristics of the initial two-dimensional image. This achieves effective segmentation of the initial two-dimensional image, separating the target (also known as the foreground, such as the endoscope body and sheath) from the background (such as the physiological channel), providing a more accurate data foundation for subsequent image processing and analysis. Adaptive binarization is a method that dynamically adjusts the binarization threshold based on local image characteristics. In adaptive binarization, the algorithm determines the threshold for each pixel based on the grayscale values ​​of its surrounding neighborhood, thus achieving adaptive binarization processing for different regions. This method can effectively handle regions with large brightness variations in the image, improving segmentation accuracy. Otsu's method is a global thresholding segmentation method that aims to find a suitable segmentation threshold (i.e., the first and second segmentation thresholds) that maximizes the inter-class variance between the target and the background. This method iterates through multiple possible thresholds, calculates the inter-class variance at each threshold, and finds the threshold that maximizes the inter-class variance as the global threshold for segmentation. Otsu's method is suitable for images with distinct grayscale distributions in the background and foreground, and can effectively achieve image binarization segmentation.

[0085] Optionally, in the process of thresholding using adaptive binarization and Otsu's method, the initial two-dimensional image after data preprocessing can be viewed as a matrix. Each element in this matrix corresponds to a pixel in the initial two-dimensional image, and the value of this element is the pixel value, which can be a grayscale value. The grayscale value can be between (0, 255). By comparing the segmentation threshold with the pixel values, the pixels can be divided into foreground and background categories. By calculating the proportion of pixels in the foreground and background, the average grayscale value, and the inter-class variance, a segmentation threshold is determined to maximize the inter-class variance. Binarization processing is then performed at this segmentation threshold to segment the initial two-dimensional image into foreground and background parts.

[0086] In this embodiment of the invention, the combination of Otsu's method and adaptive binarization allows for more accurate extraction of the endoscope's main body, achieving effective segmentation of the foreground and background, which is beneficial for subsequent image analysis and processing. This method can automatically determine a suitable segmentation threshold, avoiding the subjectivity of manual threshold adjustment and improving the accuracy and stability of image processing. In summary, threshold segmentation using adaptive binarization and Otsu's method can effectively segment the initial image, extracting the required first and second image regions, providing strong support for further image processing and analysis. It also improves the automation and accuracy of image processing, facilitating the extraction of the first and second image regions for subsequent processing.

[0087] For example, if the initial 2D image is an M*N matrix, the segmentation threshold for separating the foreground and background is denoted as T. The proportion of foreground pixels in the entire initial 2D image is denoted as ω0, and the average gray level of the foreground pixels is denoted as μ0. The proportion of background pixels in the entire initial 2D image is denoted as ω1, and the average gray level of the background pixels is denoted as μ1. The total average gray level of the initial 2D image is denoted as μ, and the inter-class variance is denoted as σ. 2 If the number of pixels in the image whose grayscale value is less than the segmentation threshold T is denoted as A, and the number of pixels whose grayscale value is greater than or equal to the segmentation threshold T is denoted as B, then:

[0088]

[0089] A + B = M * N

[0090] 1 = ω0 + ω1

[0091] μ=ω0μ0+ω1μ1

[0092] σ 2 =ω0(μ0-μ) 2 +ω1(μ1-μ)

[0093] By combining the above formulas, we can obtain:

[0094] σ 2 =ω0ω1(μ0-μ1)

[0095] The segmentation threshold T is obtained by iterating through each gray value in the range (0, 255) to find the segmentation threshold T that maximizes σ. This segmentation threshold T can then be used to perform threshold segmentation on the initial two-dimensional image.

[0096] For example, if it is necessary to segment the first image region, the foreground may include the main body of the endoscope that enters the physiological channel up to the puncture point, as well as other pixels of the first segmentation threshold.

[0097] It should be noted that the above-described process and method for thresholding the initial two-dimensional image are merely illustrative examples and are not intended to impose specific limitations. The segmentation threshold T can be determined based on the parameters (e.g., A, B) corresponding to whether the foreground to be segmented is the endoscope body or the sheath, as well as the parameters of the initial two-dimensional image (e.g., μ0, μ1). Any process and method that can perform thresholding on the entire two-dimensional CArm perspective view and determine the endoscope end point and sheath end point through local image regions is within the protection scope of this invention.

[0098] As an optional embodiment, step S106, determining the first target position of the endoscope tip in three-dimensional space based on the first image region, includes: acquiring an updated first image region and determining the orientation information of the endoscope body based on the updated first image region; combining the orientation information and the updated first image region to determine the first target position of the endoscope tip in three-dimensional space.

[0099] In this embodiment, during the process of determining the first target position of the endoscope tip in three-dimensional space based on the first image region, the orientation information of the endoscope body can be determined according to the updated first image region. Combining the orientation information and the updated first image region, the position of the endoscope tip in two-dimensional space is determined, further determining the first target position of the endoscope tip in three-dimensional space. The updated first image region can be a continuous first image region of the endoscope's mask. The orientation information can be used to indicate whether the endoscope tip is oriented left-right or up-down; this is merely an example and not a specific limitation. The orientation information can also be the orientation of the sheath.

[0100] Optionally, contour detection can be performed from the updated first image region to identify the shape with the largest area as the endoscope body. That is, after detecting contours, the area of ​​each contour can be calculated, and the shape with the largest area is found. In this image, the shape with the largest area is usually considered the main area of ​​the endoscope body because the shape of an endoscope is typically relatively regular and has a large area. Identifying the shape with the largest area as the endoscope body allows for accurate location and extraction of the endoscope's position and shape, facilitating the determination of orientation information. The above method can help automate the identification and location of endoscopes, providing important information for subsequent surgical navigation or diagnosis. By combining continuous masking processing and contour detection with the identification of the largest area shape, the position of the endoscope in the image can be effectively found, achieving automatic detection and location of the endoscope. This method can improve processing efficiency and accuracy, providing strong support for medical analysis and diagnosis.

[0101] Optionally, the endoscope body obtained from the contour detection in the updated first image region can be analyzed for its orientation information. Based on the orientation information, the position of the endoscope tip in two-dimensional space is determined, and then converted to three-dimensional space to obtain the first target position.

[0102] As an optional embodiment, obtaining the updated first image region includes: if the pixels in the first image region corresponding to the endoscope body are in a discrete state, then the first image region is subjected to continuous masking processing to obtain the updated first image region; if the pixels in the first image region corresponding to the endoscope body are in a continuous state, then the first image region is used as the updated first image region.

[0103] In this embodiment, if an updated first image region is needed, the first image region can be detected to determine whether the pixels constituting the endoscope body are discrete or continuous. If the pixels of the endoscope body are discrete, a mask continuity processing can be performed on the first image region to obtain the updated first image region. If the pixels of the endoscope body are continuous, no mask continuity processing is needed, and the first image region can be directly used as the updated first image region. The pixels constituting the endoscope body can be called the endoscope body mask (or mask region), for example, the mask of a bronchoscope. Mask continuity generally refers to the continuity and integrity of the target region in the image. If the mask is continuous, it means that the pixels of the target region are connected to each other in the image without interruption or breakage, forming a continuous whole. The continuity of the mask is very important for subsequent image analysis and processing because it ensures the integrity and accuracy of the target.

[0104] Optionally, by combining erosion and dilation operations, the mask area of ​​the endoscope body can be made continuous and more complete, thereby ensuring the accuracy of determining the orientation information of the endoscope body through the first image area.

[0105] Optionally, the mask of the endoscope body in the first image region can first undergo an erosion operation to remove small pixels and noise at the edges, and then a dilation operation can be performed to re-expand and connect the mask region, thereby obtaining a more accurate and continuous mask of the endoscope body. The mask processed in the above way can be better used for subsequent image analysis and processing, improving the accuracy of orientation information of the endoscope body, etc.

[0106] Optionally, erosion is an image morphology operation that uses a structuring element to slide across an image. When the structuring element completely covers the target area, the pixel is preserved; otherwise, it is set to the background (i.e., black). Erosion can reduce the size of the target area, remove small pixels around the target, smooth the edges of the target, and remove noise and pinholes. Dilation is another image morphology operation that uses a structuring element to slide across an image. When the structuring element overlaps with the target area, the pixel is set to the target (i.e., white); otherwise, the original pixel value is preserved. Dilation can enlarge the target area, fill holes within the target, connect broken parts of the target, and increase the area and connectivity of the target.

[0107] In this embodiment of the invention, if the endoscope body in the obtained first image region is discontinuous, it means that the mask of the endoscope body may not be continuous. This situation may affect the identification, localization, and analysis of the endoscope body because a discontinuous mask will result in incomplete shape and contour of the endoscope body, leading to incomplete information and potentially causing errors or inaccurate results. Therefore, in order to better extract and process information such as the orientation of the endoscope body, it is necessary to use image processing techniques, such as erosion and dilation, to make the mask of the endoscope body continuous. This means that the area of ​​the endoscope body in the image should be continuous, without discontinuities or breaks, to ensure accurate identification and analysis of the endoscope body. By making the mask of the endoscope body continuous, the accuracy and reliability of the extraction and processing of the endoscope body can be improved.

[0108] It should be noted that the above-described process and method for performing continuous masking on the endoscope body in the first image area is merely an example and is not specifically limited here. Any process and method that can perform continuous masking to ensure the accuracy of analyzing the orientation information and width information of the endoscope body is within the protection scope of this invention.

[0109] As an optional embodiment, determining the orientation information of the endoscope body based on the updated first image region includes: performing fitting processing on the pixels belonging to the endoscope body in the updated first image region to obtain the centerline fitting curve of the endoscope body; and determining the orientation information based on the coordinate information of any two pixels on the centerline fitting curve on the target coordinate axis.

[0110] In this embodiment, during the process of determining the orientation information of the endoscope body based on the updated first image region, the pixels belonging to the endoscope body in the updated first image region can be fitted to obtain a centerline fitting curve for the endoscope body. The orientation information can be determined based on the coordinate information of any two pixels on the centerline fitting curve on the target coordinate axis. The centerline fitting curve can also be referred to as a fitting curve.

[0111] In this embodiment of the invention, a fitting curve is obtained by performing polynomial fitting on the pixels of the endoscope body in the updated first image region. That is, the shape of the endoscope body is approximated by a polynomial function. Then, two pixels on the fitting curve are compared, primarily comparing the magnitude of their coordinates on the target coordinate axis to determine the left-right or up-down orientation of the endoscope body. For example, the pixels of the endoscope body can be sorted according to their distance from the initial position of the endoscope when it is initially inserted into the physiological channel. The left-right orientation of the endoscope body can be determined based on the corresponding sequence number of two pixels and their corresponding coordinates on the x-axis, and the up-down orientation can be determined based on the corresponding sequence number of two pixels and their corresponding coordinates on the y-axis.

[0112] It should be noted that the target coordinate axis for analyzing orientation information and the methods and processes for determining the orientation information of the endoscope body described above are only illustrative examples and are not subject to specific limitations. They can be set according to actual circumstances.

[0113] Optionally, a suitable fitting curve is found in the first image region using a polynomial fitting algorithm, allowing the curve to better approximate the shape of the endoscope body. The fitted curve provides a better description and characterization of the overall shape and contour of the endoscope body. Two pixels are selected on the fitted curve to represent two specific locations on the endoscope body. Typically, these two pixels are located at the two endpoints or two specific feature points of the endoscope body. The coordinate information of these two points on the target coordinate axis can be used to determine the left-right orientation of the endoscope body.

[0114] Optionally, by comparing the coordinates of two selected pixels and their distances from the initial position of the endoscope inserted into the physiological channel, it can be determined which pixel has a larger coordinate, thus determining the left-right orientation of the endoscope. For example, if the first pixel is to the left of the second pixel, and the first pixel has a larger x-coordinate and its distance from the initial position is greater than the distance between the second pixel and the initial position, then the endoscope is facing left; if the second pixel has a larger x-coordinate and its distance from the initial position is greater than the distance between the first pixel and the initial position, then the endoscope is facing right. This is merely an example and does not impose specific limitations.

[0115] In this embodiment of the invention, the first image region can be analyzed and judged using the above method through image processing and curve fitting, thereby determining the left-right orientation of the endoscope body. This allows for a better understanding of the position and orientation of the endoscope body, providing important reference information for subsequent surgical navigation and operations. Image processing and analysis can accurately determine the orientation of the endoscope body, helping to improve the accuracy and success rate of the surgery.

[0116] As an optional embodiment, the first target position of the endoscope tip in three-dimensional space is determined by combining orientation information and the updated first image region, including: obtaining two-dimensional coordinate information of the endoscope tip in the first image region based on orientation information, the obtained centerline fitting curve and the pixels belonging to the endoscope body in the first image region; and performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the endoscope tip in the first image region corresponding to different initial two-dimensional images to obtain the first target position of the endoscope tip in three-dimensional space.

[0117] In this embodiment, during the process of determining the first target position of the endoscope tip in three-dimensional space by combining orientation information and the updated first image region, the two-dimensional coordinate information of the endoscope tip in two-dimensional space can be obtained based on the orientation information, the centerline fitting curve, and the pixels of the endoscope body in the first image region. The two-dimensional coordinate information can then be processed by three-dimensional backprojection to obtain the first target position.

[0118] In this embodiment, after the endoscope body is obtained from the contour detection in the first image region in the threshold-based segmentation step, non-zero point detection can be performed.

[0119] Optionally, during the non-zero point detection process, the parameters of the endoscope's end point, the tangent line of the fitted curve at the end point of the endoscope (e.g., slope k, bias b), and another point O on the tangent line can be obtained. That is, non-zero point detection is performed in the first image region after thresholding and contour detection, i.e., traversing the pixels to obtain the pixels that make up the endoscope body in the first image region. These pixels can represent the contour or edge of the endoscope body.

[0120] Optionally, if the orientation of the endoscope body is right, then from the pixels to the right of the pixels on the endoscope body, the rightmost point located on the centerline fitting curve of the endoscope body can be selected. This point is the end-effector point, and its two-dimensional coordinates can be determined. Conversely, if the orientation is left, then from the pixels to the left of the pixels on the endoscope body, the leftmost point located on the centerline fitting curve of the endoscope body can be selected as the end-effector point. It should be noted that the above process of selecting the end-effector point is only an example and is not a specific limitation.

[0121] It should be noted that the process and method described above for determining the endoscope tip from various points on the centerline fitting curve is only an example of the endoscope body facing right or left. If the endoscope body faces other directions, the endoscope tip can be selected according to the actual orientation.

[0122] Optionally, after determining the two-dimensional coordinates of the endoscope tip, the first target position in three-dimensional space can be obtained through back projection calculation.

[0123] For example, after determining the two-dimensional coordinates of the endoscope tip at the Towards and Ccw angles, its spatial coordinates can be obtained through back projection. Given that the two-dimensional coordinates of the endoscope tip at the Towards angle are (u1, v1) and the two-dimensional coordinates of the endoscope tip at the Ccw angle are (u2, v2), assuming the coordinates of the first target position in the corresponding three-dimensional space are (x, y, z), (u1, v1) and (u2, v2) can be distorted using the inverse process of the following formula to obtain the distorted (x'1, y'1) and (x'2, y'2):

[0124]

[0125] Where, x center This can be used to represent the abscissa value of the optical center point of the imaging plane of the first image region; y center It can be used to represent the ordinate value of the optical center point; x focal It can be used to represent the focal length of the camera acquiring a CArm perspective view in the horizontal direction; y focalIt can be used to represent the focal length in the vertical direction; a ij b ij These can be fourth-order distortion parameters, representing the transverse and longitudinal distortion coefficients, respectively. This was obtained during the CArm calibration phase.

[0126] For example, during the patient registration phase, the rotation and translation matrices from CT to CArm at the Towards and Ccw angles have been calculated, respectively:

[0127]

[0128] in, It can be used to represent a rotation matrix under the Towards angle; It can be used to represent the translation matrix under the Towards angle; It can be used to represent the rotation matrix at angle Ccw; It can be used to represent the translation matrix at the Ccw angle.

[0129] It can be derived from the formula Where P can be used to represent the coordinates of point A in the coordinate system of the CT image, and A can be used to represent the coordinates of point A in the coordinate system of the CArm image, resulting in the following coordinate information:

[0130]

[0131]

[0132] Solving the two equations above simultaneously, we can obtain:

[0133]

[0134] By solving the above system of linear equations, the coordinates of the endoscope tip in three-dimensional space can be obtained, that is, the first target position (x... q ,y q ,z q This can also be called the starting point of the physiological channel.

[0135] As an optional embodiment, step S106, determining the second target position of the sheath end point in three-dimensional space based on the second image region, includes: performing contour detection on the second image region to obtain a centroid dataset corresponding to the sheath, wherein the centroid dataset includes the two-dimensional coordinate information of each centroid corresponding to the sheath in the second image region; filtering each centroid contained in the centroid dataset according to the centerline fitting curve and the end point of the endoscope; taking the selected centroid as the end point of the sheath and obtaining the two-dimensional coordinate information of the end point of the sheath in the second image region; performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the end point of the sheath in the second image region corresponding to different initial two-dimensional images to obtain the second target position of the sheath end point in three-dimensional space.

[0136] In this embodiment, during the process of determining the second target position of the sheath tip in three-dimensional space based on the second image region, contour detection can be performed on the second image region to obtain a centroid dataset corresponding to the sheath. The centroids included in the centroid dataset can be filtered based on the centerline fitting curve and the endoscope tip. The selected centroids are used as the sheath tip. Two-dimensional coordinate information of the sheath tip in the second image region can be obtained. This two-dimensional coordinate information can then be processed into three-dimensional space through three-dimensional backprojection to obtain the second target position of the sheath tip in three-dimensional space. The centroid dataset may include the two-dimensional coordinate information of each centroid corresponding to the sheath in the second image region.

[0137] Optionally, after determining a specific centroid from a large number of centroids, the two-dimensional coordinate information corresponding to the specified centroid can be determined from the centroid dataset. Similarly, the second target position of the sheath end point in three-dimensional space can be calculated using the aforementioned back-projection calculation formula, similar to determining the first target position of the end point of the endoscope. The back-projection calculation process has been described in detail in the above embodiments and will not be elaborated further here.

[0138] Optionally, the bias of Otsu's method is adaptively adjusted for thresholding to obtain the segmentation result of the sheath image, i.e., the second image region. Adaptive bias adjustment better adapts to the characteristics of the image, improving segmentation accuracy. Otsu's method is a global thresholding method based on maximizing inter-class variance. In this method, the threshold is dynamically adjusted according to the characteristics of the image by adaptively adjusting the bias, adapting to brightness variations and noise levels in different regions. Based on the adjusted bias, thresholding is performed using Otsu's method. Thresholding segmentation divides the image into two parts based on its grayscale values: the target (e.g., the sheath) and the background. The adaptively adjusted bias allows for more accurate threshold determination, achieving effective segmentation of the bronchoscopy image. The second image region is obtained after adaptively adjusting the bias using Otsu's thresholding method. This result shows the separation between the target region and the background region of the sheath, facilitating subsequent processing and analysis.

[0139] Threshold segmentation by adaptively adjusting the bias of the Otsu method can achieve more accurate and effective segmentation based on the characteristics and requirements of the image. This helps to extract the second image region and provides a better data foundation for subsequent processing and analysis. The above method can improve the accuracy and efficiency of image segmentation.

[0140] As an optional embodiment, the centroids in the centroid dataset are filtered based on the centerline fitting curve and the endoscope end point, including: obtaining the tangent line of the endoscope end point on the centerline fitting curve; determining the corresponding circle center on the tangent line based on the endoscope end point, the orientation information of the endoscope body, and the preset radius length; constructing a target circle with the circle center and the preset radius length, and filtering the centroids in the centroid dataset according to the filtering conditions based on the target circle to determine the specified centroid.

[0141] In this embodiment, during the process of selecting a specified centroid from the centroids based on the centerline fitting curve and the endoscope tip, the tangent line of the endoscope tip on the centerline fitting curve can be obtained. The center of the target circle can be determined on the tangent line based on the endoscope tip, the orientation information of the endoscope body, and a preset radius length. A target circle can be constructed using the center and the preset radius length. Based on the target circle, a specified centroid is selected from a large number of centroids using selection criteria. The preset radius length can be determined based on the longest sheath length (maxLength), for example, the preset radius length r = maxLength / 2 + 5. This preset radius length is only an example and is not a specific limitation. The target circle can be represented by C. m This can be represented. The longest sheath length can be pre-calculated based on a large amount of surgical data. Centroids can also form a centroid set, for example, a massList of centroids.

[0142] Optionally, before determining the tangent on the fitted curve, adaptive image segmentation can be performed on the first image region based on the orientation information and overall width of the endoscope body to segment an image containing the endoscope tip. Based on the orientation of the endoscope body, a specific portion of the endoscope body can be segmented for subsequent processing and analysis. That is, adaptive image segmentation is performed based on the orientation and overall width of the endoscope body. In this case, if the endoscope body faces right, the right-side portion of the endoscope body image can be segmented. By performing adaptive segmentation based on orientation and width information, a better selection and extraction of specific portions of the endoscope body image can be achieved. After adaptive segmentation, the right-side portion of the endoscope body image can be obtained. The purpose of the above method is to provide more accurate data for subsequent processing and analysis, resulting in better fitting. Segmenting a portion of the endoscope body image allows for more precise and targeted subsequent processing, contributing to a better understanding of the shape and characteristics of the endoscope body.

[0143] In this embodiment of the invention, by adaptively segmenting based on the orientation information and overall width of the endoscope body, images of specific portions of the endoscope body (e.g., the end point of the endoscope) can be effectively selected and extracted, providing a more accurate data foundation for subsequent processing and analysis. The above method can perform targeted image segmentation according to specific circumstances, which helps improve the accuracy of endoscope body identification and positioning.

[0144] Optionally, after acquiring the image of the endoscope tip obtained through adaptive segmentation, polynomial fitting can be performed to obtain a fitted line of the image of the endoscope tip, which is then used to indicate the movement path of the endoscope. That is, polynomial fitting is performed on the image of the endoscope tip segmented in the previous step. Through the fitting algorithm, a suitable polynomial function can be found to approximate the shape and contour of the endoscope tip, indicating the movement path and direction of the endoscope. Based on the fitted line, a tangent line at the endoscope tip can be taken, which can approximate the direction of the sheath. By calculating the slope or direction of the fitted line, the approximate orientation of the sheath can be determined, providing important information about the sheath's position and direction. By using polynomial fitting and obtaining the tangent line at the endoscope tip, the orientation of the sheath can be determined more accurately, providing important information and guidance for subsequent surgical operations. The above method can better understand the position and movement path of the endoscope body, improving the efficiency and accuracy of the surgery.

[0145] Optionally, after obtaining the endoscope tip and the tangent of the fitted curve at the endoscope tip through the above non-zero point detection process, a target circle C can be constructed with the endoscope tip as a point on the circle and a point on the tangent line at a distance of a preset radius r from the endoscope tip as the center O. mThrough the above process, the end-effector tip, the tangent parameters (slope k, offset b) of the fitted curve at the end-effector tip, and another point O on the tangent line can be obtained. This information helps determine the position and orientation of the sheath tip, providing an important data foundation for subsequent processing and analysis. This method can improve the accuracy of end-effector identification and positioning, providing useful information and guidance for surgical procedures.

[0146] Optionally, contour detection can be performed on the second image region obtained by the above threshold segmentation to obtain the two-dimensional coordinate information of each centroid in the list of centroids of the sheath contour:

[0147]

[0148] Among them, C x It can be used to represent the abscissa of each centroid, that is, the abscissa in two-dimensional coordinate information; C y M can be used to represent the ordinates of each centroid, that is, the ordinates of two-dimensional coordinate information, where x and y are the indices of each centroid, and x and y are positive integers; M can be used to represent the geometric moments of the image, specifically, M 10 It can be used to represent the first-order horizontal moment, M 01 It can be used to represent the first-order perpendicular moment, M 00 It can be used to represent the area of ​​a contour in an image.

[0149] As an optional embodiment, the filtering conditions include at least: specifying that the centroid is located inside the target circle; specifying that the distance between the centroid and the tangent is less than or equal to a first distance threshold; and specifying that the distance between the centroid and the end point of the endoscope is greater than or equal to the distance between any centroid in the centroid dataset and the end point of the endoscope.

[0150] In this embodiment, the screening criteria may include at least the following: the specified centroid may be located within the target circle; the specified centroid may be less than or equal to a first distance threshold between itself and the switch; and the specified centroid may be greater than or equal to the distance between any centroid and the endoscope end point. The first distance threshold, also known as the tangent threshold, can be predetermined based on experimental data and can be used to exclude noisy centroids, improving the accuracy of the determined specified centroids.

[0151] Optionally, the specified centroid D(x1,y1), that is, the specified centroid in the centroid massList, can be filtered out by the above three filtering conditions.

[0152] Optionally, the first filtering criterion above specifies that the centroid is a circle C. m Internal centroid. The centroid specified in this filter must be located within circle C. m The interior, that is, the circular region C centered at O.m Inside. This condition can be used to limit the location range of a specified centroid, ensuring that the specified centroid is within a specified circular area for further filtering and analysis. By restricting the specified centroid to be located within circle C... m Inner centroids can narrow the search range for a specified centroid, reduce interference from irrelevant points, and improve the accuracy of filtering.

[0153] Optionally, the second filtering condition described above is a distance-to-tangent threshold range. This filtering condition specifies that the distance from the centroid to the tangent must be within a specified threshold range; that is, the distance from the centroid to the tangent cannot exceed a certain threshold. This filtering condition is used to limit the distance range from the specified centroid to the tangent, ensuring that the proximity of the specified centroid to the tangent meets the requirements. By setting a threshold range for distance to the tangent, points that are far from the tangent can be filtered out, while retaining the specified centroids that are close to the tangent, thus improving the accuracy of the filtering.

[0154] Optionally, the third filtering condition mentioned above is the centroid furthest from the endoscope tip. This filtering condition, based on meeting the first two conditions, allows selection of the centroid furthest from the endoscope tip, specifically the centroid D(x1,y1) in the centroid massList that is farthest from the endoscope tip. This filtering condition determines the final selection of the centroid D(x1,y1), ensuring that the centroid furthest from the endoscope tip is chosen as the final centroid. By selecting the centroid furthest from the endoscope tip, appropriate selection of the centroid under specific conditions is ensured, avoiding other possible interfering factors and improving the accuracy and precision of the selected centroid.

[0155] In summary, by using the three screening criteria described above to select a specified centroid, the specified centroid D(x1,y1) can be effectively determined. Setting and applying these criteria helps to select a qualified centroid and eliminate interfering points that do not meet the criteria, thus improving the accuracy and reliability of the screening results. The above screening method helps to determine a suitable selection of a specified centroid to meet specific needs and requirements.

[0156] It should be noted that the above screening conditions are only illustrative examples and are not specifically limited here. Any process or method that can screen out a specified centroid as the end point of the sheath from a large number of centroids is within the protection scope of the embodiments of the present invention.

[0157] As an optional embodiment, step S108, based on the first target position and the second target position, determines the azimuth information between the sheath tip and the reference point, including: acquiring the reference coordinate information of the reference point in three-dimensional space and the target coordinate information of the target point on the physiological channel in three-dimensional space, wherein the target point is the puncture point determined on the physiological channel based on the planned path; determining the distance in the azimuth information based on the second target position and the reference coordinate information; determining the first straight line from the target point to the reference point according to the target coordinate information and the reference coordinate information, and determining the second straight line from the target point to the sheath tip according to the target coordinate information and the second target position, and determining the angle between the first straight line and the second straight line as the angle in the azimuth information.

[0158] In this embodiment, during the process of determining the azimuth information between the sheath tip and the reference point based on the first and second target positions, the reference coordinates of the reference point in three-dimensional space and the target coordinates of the target point on the physiological channel in three-dimensional space can be obtained. The distance in the azimuth information can be determined based on the second target position and the reference coordinates. Alternatively, a first straight line containing the target point and the reference point can be determined based on the target coordinates and the reference coordinates, and a second straight line containing the target point and the sheath tip can be determined based on the target coordinates and the second target position. The angle between the first and second straight lines can be used to determine the angle in the azimuth information.

[0159] Optionally, the target point can be a puncture point (POE) determined along the physiological pathway based on the planned path. For example, the target point can be a puncture point on the bronchial wall. If the procedure is for bronchoscopy or bronchoscopic biopsy, the LungPro software can be used to plan the path and calculate the location of the appropriate POE point, i.e., the target coordinate information P of the target point. p =(x P ,y P ,z P ).

[0160] It should be noted that the above-described software for planning paths and the process for determining POE points are merely illustrative examples and are not subject to specific limitations.

[0161] For example, the distance between the second target location and the reference coordinate information can be determined using the following formula:

[0162]

[0163] Where d1 can be used to represent the distance between the second target location and the reference coordinate information, that is, the distance between the sheath tip and the lesion point; P t =(x t ,yt ,z t () can be used to represent reference coordinate information, that is, the coordinates of the lesion point; P s =(x s ,y s ,z s () can be used to represent the location of the second target, that is, the coordinates of the end point P of the sheath.

[0164] For another example, the angle between the first and second lines can be determined using the following formula:

[0165]

[0166] Where θ can be used to represent the angle between the first and second lines, that is, the angle between the POE to the lesion point and the POE to the end of the sheath; P t It can be used to represent the coordinates of the lesion point; P P It can be used to represent the coordinates of a POE point; P s It can be used to represent the coordinates of the end point of the sheath.

[0167] It should be noted that the process and methods for determining the angle and distance in the above-mentioned location information are only illustrative examples and are not subject to specific limitations.

[0168] As an optional embodiment, the method further includes: determining the distance between the target point and the endoscope tip based on the first target location and target coordinate information, wherein the distance between the target point and the endoscope tip is used to determine the degree of matching between the actual movement path of the endoscope in the physiological channel and the planned path.

[0169] In this embodiment, the distance between the target point and the endoscope tip can also be determined based on the first target location and target coordinate information. The distance between the target point and the endoscope tip can be used to determine the degree of matching between the actual movement path of the endoscope in the physiological channel and the pre-planned path.

[0170] For example, the distance between the first target location and the target coordinates can be determined using the following formula:

[0171]

[0172] Where d2 can be used to represent the distance between the first target location and the target coordinate information, that is, the distance between the end of the bronchoscope and the POE point; P q =(x q ,y q ,z q () can be used to represent the location of the first target, that is, the coordinates of the end point of the bronchoscope.

[0173] In this embodiment of the invention, the magnitude of d2 can be used to determine the error between the actual movement path and the pre-planned path, that is, the degree of matching between the two. A smaller d2 indicates a higher degree of matching between the actual movement path and the pre-planned path, suggesting more accurate puncture. A larger d2 indicates a lower degree of matching between the actual movement path and the pre-planned path, suggesting lower puncture accuracy, requiring appropriate adjustments. Ideally, the POE point coincides with the end point of the bronchoscope, i.e., puncture accuracy is achieved, and d2 = 0.

[0174] In this embodiment of the invention, initial two-dimensional images of the endoscope during insertion into the physiological channel can be acquired from at least two angles. From the initial two-dimensional image, a first image region of the endoscope body and a second image region of the sheath can be segmented. From the first image region, the coordinates of the endoscope tip in two-dimensional space can be determined, and these coordinates can be transformed into three-dimensional space to obtain a first target position. From the second image region, the coordinates of the tracheal tip in two-dimensional space can be determined, and these coordinates can be transformed into three-dimensional space to obtain a second target position. Using the first and second target positions, the distance and angle between the sheath tip and the reference point can be determined, thereby determining in real time whether the sheath tip has reached the lesion point. In this embodiment, by transforming the coordinates from two-dimensional space to three-dimensional space, the positions of the endoscope tip and the sheath tip relative to the reference point can be determined more accurately, improving the accuracy of positioning and providing more stereoscopic information, including depth and angle, etc., enabling a clearer understanding of the positional relationship between the endoscope and the reference point, and facilitating more precise guidance for diagnostic and treatment operations. Compared to relying on CArm perspective views at different angles for confirmation, the method described above provides richer three-dimensional information, reduces the limitations of two-dimensional space, and helps improve the accuracy and safety of diagnosis and treatment. By more intuitively understanding the distance and angle between the sheath tip and the reference point, diagnostic and treatment procedures can be guided more precisely, reducing errors and increasing the success rate. This achieves the technical effect of improving the accuracy of determining the orientation of the sheath tip, solving the technical problem of low accuracy in determining the orientation of the sheath tip.

[0175] Example 2

[0176] In this embodiment of the invention, the method for determining the orientation of the sheath tip provided by the present invention will be described in detail below with reference to another optional bronchoscope implementation.

[0177] Currently, BTPNA, or Bronchoscopic Transbronchial Nodule Access, also known as the tunneling technique, involves creating a tunnel by drilling holes in the bronchial wall. The nodule is then accessed through a working channel within the lung parenchyma, bypassing the natural bronchial pathways. Theoretically, this allows for complete lung access and precise treatment. However, as an interventional procedure, BTPNA relies on C-arm imaging. Since the C-arm, as a fluoroscopic imaging system, only provides a two-dimensional positional reference, there is a loss of 3D information. To determine the spatial location, the C-arm is rotated to image from multiple angles. However, because C-arm imaging relies on X-rays, there are potential risks associated with prolonged exposure to radiation for both the operator and the patient.

[0178] Optionally, during BTPNA, the operator can obtain real-time positional information through images captured by a bronchoscope lens within the airway. However, once outside the airway, this positional information becomes unknown. In this case, a CArm perspective image is typically used to determine the sheath position in the airway portion. However, because the CArm perspective image is a two-dimensional image, it is difficult to obtain spatial positional information of the sheath within the airway. This makes it difficult to determine whether the lesion has been reached, increasing the complexity and risk of the procedure. During the procedure, if the biopsy needle, sheath, or other tools do not accurately reach the lesion location, the CArm perspective image, being a two-dimensional image, cannot provide effective spatial information such as angles and distances. This leads to a lack of accurate guidance for the operator, increasing the difficulty and duration of the procedure, and also increasing the risks.

[0179] Optionally, BTPNA is a bronchoscopic lung biopsy used for the diagnosis and treatment of lung diseases. In this procedure, preoperative CT scans are typically used to confirm the location and size of the lesion and plan the surgical path. During the procedure, DRR (digital reconstructed radiography, virtual images) generated by intraoperative CT are registered with CArm fluoroscopic images (real images), mapping the target location confirmed on the CT scan onto the CArm fluoroscopic view. This is done to accurately guide the operator to the predetermined lesion location during the actual surgery.

[0180] In BTPNA surgery, the operator first guides a bronchoscopically-guided instrument through the lung tissue along a pre-planned path to reach the lesion site and perform procedures such as biopsy sampling or other treatments. During the procedure, the operator uses CArm imaging from different angles to confirm that the sheath after exiting the airway has accurately reached the lesion. CArm imaging allows the operator to observe the instrument's position within the body in real time, ensuring the accuracy and safety of the procedure.

[0181] Throughout the BTPNA procedure, steps such as preoperative CT scans, registration of DRR images generated by intraoperative CT with CArm fluoroscopic images, and confirmation of location using CArm views from different angles are all aimed at improving surgical precision, reducing surgical risks, and ensuring that the operator can accurately reach the lesion site for effective diagnosis and treatment. This integrated approach, utilizing imaging technology and real-time navigation technology, will help improve the success rate and safety of BTPNA surgery.

[0182] In one related technique, CArm perspective images at different angles can be used for confirmation. However, CArm perspective images are two-dimensional images, making precise positioning difficult. Furthermore, when the sheath has not reached the lesion location and adjustments are needed, CArm perspective images are insufficient to provide effective information. Therefore, the technical problem of low accuracy in determining the orientation of the sheath tip remains.

[0183] This invention proposes a method for repositioning three-dimensional coordinates based on two-dimensional X-ray images. This method segments the sheath in a CArm view from any two angles, locates the sheath tip, and then calculates its position in three-dimensional space through back projection. By calculating the angle and distance between the sheath and the lesion, it is determined whether the sheath has reached the lesion. If the sheath has not reached the lesion, this technology can provide angle adjustments and distance prompts, helping the operator to more easily deliver the sheath to the lesion, thereby improving the accuracy and success rate of the surgery. Utilizing three-dimensional spatial information for precise sheath positioning, compared to two-dimensional CArm perspective views in related technologies, provides more accurate positional information and spatial relationships, offering more intuitive and guiding information to the surgeon. This innovative method can reduce the number of adjustments during surgery, lower surgical risks, and improve surgical efficiency and success rates, which is of great significance for interventional surgeries such as BTPNA. This achieves the technical effect of improving the accuracy of determining the orientation of the sheath tip, solving the technical problem of low accuracy in determining the orientation of the sheath tip.

[0184] The method will be further described below.

[0185] In this embodiment, Figure 2 This is a flowchart of a method for calculating spatial position by back projection of a two-dimensional perspective view according to an embodiment of the present invention, such as... Figure 2 As shown, the method may include the following steps:

[0186] Step S202: Obtain two CArm perspective views to determine the position of the sheath end.

[0187] In this embodiment, CArm perspective views from two angles can be obtained, and the end position of the sheath tube can be segmented from the views.

[0188] Alternatively, due to the physical characteristics of bronchoscopy, the airway cannot be exited directly; the furthest point reached is the planned point of exit (POE), i.e., the puncture point. In this case, the sheath needs to be moved from the working channel of the bronchoscopy to the POE first, and then, according to the puncture angle and puncture distance of the planned path, the airway exits from the POE and passes through the lung parenchyma to the target point.

[0189] Figure 3(a) is a flowchart of a data preprocessing method for sheath tip detection according to an embodiment of the present invention. As shown in Figure 3(a), the method may include the following steps:

[0190] Step S301: Increase the contrast of the CArm perspective view.

[0191] In this embodiment, contrast can be improved by calculating an adaptive equalization histogram.

[0192] Step S302: Increase the brightness of the CArm perspective view.

[0193] In this embodiment, gamma transformation can be performed to improve brightness.

[0194] Step S303: Remove noise from the CArm perspective view.

[0195] In this embodiment, median filtering can be performed to remove noise from the CArm perspective.

[0196] For example, Figure 4 This is a schematic diagram of a CArm perspective image obtained after data preprocessing transformation according to an embodiment of the present invention, as shown below. Figure 4 As shown, the hook in the CArm fluoroscopic image after data preprocessing and transformation can be considered a bronchoscope, and the translucent tube next to it represents the airway. Through the above data preprocessing steps, image quality can be improved, noise interference reduced, and a more accurate and reliable image basis can be provided for the detection and positioning of the sheath tip, thereby improving the accuracy and efficiency of the detection.

[0197] Figure 3(b) is a flowchart of a consumable segmentation method in the sheath tip detection process according to an embodiment of the present invention. As shown in Figure 3(b), the purpose of this method is to segment a local CArm perspective view containing the bronchoscope tip and sheath, eliminate the influence of other redundant images, and better perform bronchoscope fitting, threshold segmentation, and contour detection, ultimately facilitating the acquisition of specified centroid coordinates. Compared to processing the above steps on the entire CArm perspective view, this step improves the efficiency and accuracy of acquiring specified centroid coordinates. The method may include the following steps:

[0198] Step S304: Adaptive binarization and Otsu's method are used to perform threshold segmentation on the CArm perspective view after data preprocessing.

[0199] In this embodiment, by combining Otsu's method and adaptive binarization, the bronchoscope portion can be extracted more accurately from the CArm perspective view, achieving effective segmentation of the foreground and background, which is helpful for subsequent image analysis and processing.

[0200] Optionally, taking a grayscale image as an example, an image can be viewed as an M*N matrix, representing the pixels in the image, with each value being a pixel value between (0, 255). The segmentation threshold between the foreground (i.e., the target) and background is denoted as T. The proportion of foreground pixels to the entire image is denoted as ω0, and the average grayscale value of the foreground is denoted as μ0. The proportion of background pixels to the entire image is denoted as ω1, and its average grayscale value is denoted as μ1. The overall average grayscale value of the image is denoted as μ, and the inter-class variance is denoted as σ. 2 .

[0201] Assuming the image background is dark and the image size is M*N, let A be the number of pixels with a gray value less than a threshold T, and B be the number of pixels with a gray value greater than or equal to the threshold T. Then:

[0202]

[0203] A + B = M * N

[0204] 1 = ω0 + ω1

[0205] μ=ω0μ0+ω1μ1

[0206] σ 2 =ω0(μ0-μ) 2 +ω1(μ1-μ)

[0207] By combining the above formulas, we can obtain:

[0208] σ 2 =ω0ω1(μ0-μ1)

[0209] By iterating through the grayscale range (0, 255) with the threshold T value, we obtain the result that σ 2 A threshold T is used to obtain the maximum value. The image is then segmented using the maximum threshold T. The foreground includes the bronchoscope portion extending into the bronchial passage up to the POE position and other pixels that meet the threshold.

[0210] Step S305: Through erosion expansion, the mask of the bronchoscope and consumable in the image is made continuous.

[0211] In this embodiment, the combined use of erosion and expansion operations can make the mask area of ​​the bronchoscope continuous and more complete, thereby ensuring the accuracy of determining the sheath orientation through CArm images.

[0212] Figure 5 This is a schematic diagram of a bronchoscope with a continuous mask obtained through corrosion and expansion according to an embodiment of the present invention, as shown below. Figure 5 As shown, if the bronchoscope in the acquired image is discontinuous, it means that the bronchoscope mask may not be continuous. This situation may affect the identification, localization, and analysis of the bronchoscope, because a discontinuous mask will result in incomplete shape and contour of the bronchoscope, leading to incomplete information and potentially causing errors or inaccurate results. Therefore, to better extract and process information such as sheath orientation, the bronchoscope mask can be made continuous through erosion and expansion. This means that the bronchoscope mask is continuous in the image, without discontinuities or breaks, ensuring accurate identification and analysis of the bronchoscope and improving the accuracy and reliability of bronchoscope extraction and processing.

[0213] Step S306: Through contour detection, the shape with the largest area is selected as the bronchoscope.

[0214] In this embodiment, contour detection is performed, and the shape with the largest area in the image is regarded as a bronchoscope.

[0215] Figure 6 This is a schematic diagram of a bronchoscope obtained by contour detection according to an embodiment of the present invention, as shown below. Figure 6 As shown, the white hook in the figure can be a bronchoscope obtained through contour detection.

[0216] Step S307: Determine the orientation of the sheath.

[0217] In this embodiment, a fitting curve is obtained by performing polynomial fitting on the bronchoscopic image obtained in step S306. Two points on the curve are compared, for example, by comparing the magnitude of the x-coordinate, to determine the left-right orientation of the sheath.

[0218] Step S308: Adaptively segment the end of the bronchoscope according to the orientation of the sheath and the overall width of the bronchoscope.

[0219] In this embodiment, the image with the end of the bronchoscope is adaptively segmented based on the orientation and the overall width of the mirror.

[0220] Figure 7 This is a schematic diagram of an adaptive segmentation of the bronchoscope tip according to an embodiment of the present invention, as shown below. Figure 7 As shown, it can be determined that the bronchoscope is facing to the right at this time, and the right side of the bronchoscope can be adaptively segmented. Obtaining part of the bronchoscope is to improve the fitting effect later.

[0221] Figure 3(c) is a flowchart of a method for obtaining the coordinates of the sheath tip during sheath tip detection according to an embodiment of the present invention. As shown in Figure 3(c), the method may include the following steps:

[0222] Step S309: Fit the tip of the bronchoscope.

[0223] In this embodiment, a polynomial fit is performed on the segmented bronchoscopic end image from the previous step.

[0224] Figure 8 This is a schematic diagram of the fitting line of a bronchoscope according to an embodiment of the present invention, as shown below. Figure 8 As shown, polynomial fitting can be used to obtain... Figure 8 The white fitted line in the diagram indicates the movement path of the mirror. The tangent at the tip of the mirror, taken based on the fitted line, can be used as an approximate direction of the sheath.

[0225] Step S310: Obtain the tangent parameters of the bronchoscope tip, the tangent parameters on the bronchoscope tip, and another point on the tangent.

[0226] In this embodiment, non-zero point detection is performed after contour detection in step S306 of sheath segmentation to obtain the end point A of the bronchoscope, the endpoint tangent parameters (slope k, offset b), and another point O on the tangent.

[0227] Figure 9 This is a schematic diagram of the tangent of a fitting line at the end point of a bronchoscope according to an embodiment of the present invention, as shown below. Figure 9 As shown, the process iterates through the pixels to obtain all endpoints on the right side of the white image. Then, retaining the x-coordinate, it filters out points on the fitted line from all right-side endpoints. These filtered endpoints are taken as the mirror endpoints, i.e., the end points of the bronchoscope. The slope k is the slope of the tangent line at that point on the fitted line. The bias b is obtained using y = kx + b.

[0228] Step S311: Adaptively adjust the Otsu method bias for threshold segmentation.

[0229] In this embodiment, the Otsu method bias is adaptively adjusted for threshold segmentation. Figure 10 This is a schematic diagram illustrating the segmentation result of the sheath obtained by threshold segmentation according to an embodiment of the present invention, as shown below. Figure 10 As shown, the segmentation result of the image can be obtained.

[0230] Step S312: Obtain the list of centroids.

[0231] In this embodiment, contour detection can be performed on the segmentation results obtained in step S311 to obtain the centroid massList of each contour.

[0232]

[0233] Among them, C xIt can be used to represent the abscissa of each centroid, that is, the abscissa in two-dimensional coordinate information; C y M can be used to represent the ordinates of each centroid, that is, the ordinates of two-dimensional coordinate information, where x and y are the indices of each centroid, and x and y are positive integers; M can be used to represent the geometric moments of the image, specifically, M 10 It can be used to represent the first-order horizontal moment, M 01 It can be used to represent the first-order perpendicular moment, M 00 It can be used to represent the area of ​​a contour in an image.

[0234] Step S313: Select the specified centroid from the centroid list according to the filtering criteria.

[0235] In this embodiment, based on three conditions—the centroid must be a circle C—... m The centroids are defined as follows: inner centroid, centroid within the tangent threshold range, and centroid furthest from the end of the bronchoscope. A specified centroid D(x1,y1) is selected from the centroid massList. Among these, circle C... m It can be constructed by taking the end-end point as a point on the circle and taking a point on the tangent that is a distance of a preset radius r from the end-end point as the center O.

[0236] Figure 11 This is a schematic diagram illustrating the determination of the sheath end point from a circle according to an embodiment of the present invention, as shown below. Figure 11 As shown, the white circle in the diagram represents circle C. m Circle C m The point that satisfies the above three conditions is the designated centroid.

[0237] Figure 12(a) is a schematic diagram of the end of a bronchoscope and the end of a bronchoscope according to an embodiment of the present invention. As shown in Figure 12(a), the positions of the end of the bronchoscope and the end of the sheath are respectively shown in the CArm image obtained from the Towards angle. Figure 12(b) is a schematic diagram of the spatial position of a sheath according to an embodiment of the present invention. As shown in Figure 12(b), the portion between the end of the sheath and the end of the bronchoscope in the CArm image obtained from the Ccw angle can be considered the spatial position of the sheath.

[0238] Step S204: Calculate the spatial position of the air outlet section of the sheath.

[0239] In this embodiment, after step S202 is completed, the coordinates of the bronchial tip and the sheath tip in the Towards and Ccw perspective views can be obtained. Their three-dimensional coordinates in space can then be calculated based on the two-dimensional coordinates of these two perspective views. Since the sheath penetrates directly into the target point after exiting the airway, the position of the sheath in space within the airway can be determined based on the three-dimensional spatial coordinates of the puncture point and the sheath tip.

[0240] In this embodiment of the invention, the next step is to describe the orthographic projection process:

[0241] Suppose the coordinates of point P (the end point of the sheath) in space are known to be (x... s ,y s ,z s The process of finding the coordinates of its projection onto a two-dimensional plane can be expressed as:

[0242]

[0243] Where, m R It can be used to represent the rotation matrix from CT to CArm, m T It can be used to represent the translation matrix from CT to CArm, and can be determined during the patient registration phase.

[0244] Proportional scaling can be represented as:

[0245]

[0246] Then the coordinates are transformed into coordinates on the projected image. Specifically, the x-axis and y-axis components of the normalized coordinate points are multiplied by the x-axis and y-axis components of the image's focal point coordinates, and then the x-axis and y-axis components of the image's center point coordinates are added to obtain the transformed image coordinate points.

[0247]

[0248] Where, x center This can be used to represent the abscissa value of the optical center point of the imaging plane of the first image region; y center It can be used to represent the ordinate value of the optical center point; x focal It can be used to represent the focal length of the camera acquiring a CArm perspective view in the horizontal direction; y focal It can be used to represent the focal length in the vertical direction.

[0249] Because distortion exists during CArm imaging, it must also be calculated during projection, as shown below:

[0250]

[0251] Among them, a ij b ij These can be fourth-order distortion parameters, representing the transverse and longitudinal distortion coefficients, respectively. This was obtained during the CArm calibration phase.

[0252] Figure 13 This is a schematic diagram of a back projection calculation of spatial coordinates according to an embodiment of the present invention, as shown below. Figure 13As shown, given the coordinates of a point in space projected onto two different planes, we need to find its spatial coordinates. According to the principle of back projection, if two rays in space intersect, there is only one intersection point. Therefore, we can find its spatial coordinates by back projection based on the coordinates of the bronchoscope end and the sheath end in the Towards and Ccw views. Taking the sheath end point as an example, its coordinates in the Towards view are (u1, v1), and its coordinates in the Ccw view are (u2, v2). Let the coordinates of the corresponding point P in space be (x, y, z). First, we remove the distortion from (u1, v1) and (u2, v2) according to the reverse process of the above formulas (3) and (4), and obtain (x'1, y'1) and (x'2, y'2).

[0253] During the patient registration phase, the rotation and translation matrices from CT to CArm at the Towards and Ccw angles have been calculated, and are as follows:

[0254]

[0255] in, It can be used to represent a rotation matrix under the Towards angle; It can be used to represent the translation matrix under the Towards angle; It can be used to represent the rotation matrix at angle Ccw; It can be used to represent the translation matrix at the Ccw angle.

[0256] It can be obtained from the above formulas (1) and (2):

[0257]

[0258]

[0259] Combining equations (5) and (6) above, we get:

[0260]

[0261] By solving the above system of linear equations (7), the coordinates (x, y) of the end point P of the sheath can be obtained. s ,y s ,z s Similarly, the coordinates (x, y) of the end point of the bronchoscope can be obtained. q ,y q ,z q Finally, the coordinates of the starting and ending points of the sheath outlet are obtained.

[0262] Step S206: Calculate the distance and angle from the end point of the sheath to the lesion point, and render it on the airway tree to achieve a visual display.

[0263] In this embodiment, the distance and angle from the end of the bronchoscope to the lesion can be determined using the coordinates of the bronchoscope tip and the sheath tip obtained above. The distance between the bronchoscope tip and the point of origin (POE) can also be determined. This information can then be rendered onto the bronchial tree for visualization.

[0264] Optionally, the POE point (puncture point on the bronchial wall) can be read from Lungpoint software (x P ,y P ,z P It can obtain the coordinates (x, y) of the lesion points that have already been calculated for the path planning nodes. t ,y t ,z t Calculate the distance between the coordinates of point P and the coordinates of the lesion:

[0265]

[0266] Wherein, d1 can be used as the distance between the end of the sheath and the lesion; P t =(x t ,y t ,z t ) can be used to represent the coordinates of the lesion point; P s =(x s ,y s ,z s () can be used to represent the coordinates of the end point P of the sheath.

[0267] Similarly, the distance between the tip of the bronchoscope and the point of POE can be determined using the following formula:

[0268]

[0269] Where d2 can be used to represent the distance between the end of the bronchoscope and the POE point; P q =(x q ,y q ,z q () can be used to represent the coordinates of the end point of the bronchoscope.

[0270] Optionally, the magnitude of d2 can be used to determine the error between the actual travel path and the pre-planned path, i.e., the degree of matching between the two. A smaller d2 indicates a higher degree of matching between the actual travel path and the pre-planned path, suggesting more accurate puncture. A larger d2 indicates a lower degree of matching between the actual travel path and the pre-planned path, suggesting lower puncture accuracy, requiring adjustment. Ideally, the POE point coincides with the end point of the bronchoscope, i.e., puncture accuracy is achieved, and d2 = 0.

[0271] For another example, the distance from the point of origin (POE) to the lesion site and the distance from the point of origin (POE) to the end of the sheath can be determined using the following formula (x s ,y s ,z s The included angle:

[0272]

[0273] Where θ can be used to represent the angle between the POE and the lesion point and the POE and the end point of the sheath; P t It can be used to represent the coordinates of the lesion point; P P It can be used to represent the coordinates of a POE point; P s It can be used to represent the coordinates of the end point of the sheath.

[0274] It should be noted that the process and methods for determining the angle and distance in the above-mentioned location information are only illustrative examples and are not subject to specific limitations.

[0275] Figure 14 This is a schematic diagram of a bronchial tree visualization according to an embodiment of the present invention, such as... Figure 14 As shown, the POE, lesion point, and consumable terminal can be visualized on the bronchial tree to help the operator determine whether the consumable terminal has reached the lesion point, or to indicate the distance and angle difference between the current position of the consumable terminal and the lesion point.

[0276] In this embodiment of the invention, initial two-dimensional images of the endoscope during insertion into the physiological channel can be acquired from at least two angles. From the initial two-dimensional image, a first image region of the endoscope body and a second image region of the sheath can be segmented. From the first image region, the coordinates of the endoscope tip in two-dimensional space can be determined, and these coordinates can be transformed into three-dimensional space to obtain a first target position. From the second image region, the coordinates of the tracheal tip in two-dimensional space can be determined, and these coordinates can be transformed into three-dimensional space to obtain a second target position. Using the first and second target positions, the distance and angle between the sheath tip and the reference point can be determined, thereby determining in real time whether the sheath tip has reached the lesion point. In this embodiment, by transforming the coordinates from two-dimensional space to three-dimensional space, the positions of the endoscope tip and the sheath tip relative to the reference point can be determined more accurately, improving the accuracy of positioning and providing more stereoscopic information, including depth and angle, etc., enabling a clearer understanding of the positional relationship between the endoscope and the reference point, and facilitating more precise guidance for diagnostic and treatment operations. Compared to relying on CArm perspective views at different angles for confirmation, the method described above provides richer three-dimensional information, reduces the limitations of two-dimensional space, and helps improve the accuracy and safety of diagnosis and treatment. By more intuitively understanding the distance and angle between the sheath tip and the reference point, diagnostic and treatment procedures can be guided more precisely, reducing errors and increasing the success rate. This achieves the technical effect of improving the accuracy of determining the orientation of the sheath tip, solving the technical problem of low accuracy in determining the orientation of the sheath tip.

[0277] Example 3

[0278] This invention provides a device for determining the orientation of the sheath tip. It should be noted that the document generation device of this invention can be used to execute... Figure 1 The present invention provides a method for determining the orientation of the sheath end point. The following describes the device for determining the orientation of the sheath end point provided in the embodiments of the present invention.

[0279] Figure 15 This is a schematic diagram of a device for determining the orientation of the sheath tip according to an embodiment of the present invention, as shown below. Figure 15 As shown, the device may include: an acquisition unit 1502, a segmentation unit 1504, a first determination unit 1506, and a second determination unit 1508.

[0280] The acquisition unit 1502 is used to acquire initial two-dimensional images of the endoscope from at least two angles during insertion into the physiological channel, wherein the endoscope includes a sheath and an endoscope body.

[0281] The segmentation unit 1504 is used to segment a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional image, respectively.

[0282] The first determining unit 1506 is used to determine, based on the first image region, the first target position of the endoscope tip of the endoscope body in three-dimensional space, and based on the second image region, the second target position of the sheath tip of the sheath in three-dimensional space.

[0283] The second determining unit 1508 is used to determine the orientation information between the end point of the sheath tube and the reference point based on the first target position and the second target position. The orientation information is used to represent the distance and angle between the end point of the sheath tube and the reference point, and the reference point is used to represent the location where there is an abnormal state outside the physiological channel.

[0284] The device for determining the orientation of the sheath tip provided in this embodiment of the invention acquires initial two-dimensional images of the endoscope during insertion into the physiological channel from at least two angles using an acquisition unit 1502; a segmentation unit 1504 segments the initial two-dimensional images into a first image region containing the endoscope body and a second image region containing the sheath; a first determination unit 1506 determines a first target position of the endoscope tip in three-dimensional space based on the first image region, and determines a second target position of the sheath tip in three-dimensional space based on the second image region; and a second determination unit 1508 determines the orientation information between the sheath tip and a reference point based on the first and second target positions. This solves the technical problem of low accuracy in determining the orientation of the sheath tip and achieves the technical effect of improving the accuracy of determining the orientation of the sheath tip.

[0285] Optionally, the segmentation unit 1504 may include: a first segmentation module, configured to perform threshold segmentation on any initial two-dimensional image to obtain a first image region, wherein the first image region contains each pixel point in any initial two-dimensional image whose corresponding pixel value is less than a preset first segmentation threshold; and a second segmentation module, configured to perform threshold segmentation on any initial two-dimensional image to obtain a second image region, wherein the second image region contains each pixel point in any initial two-dimensional image whose corresponding pixel value is less than a preset second segmentation threshold.

[0286] Optionally, the first determining unit 1506 may include: a first acquiring module, configured to acquire an updated first image region and determine the orientation information of the endoscope body based on the updated first image region; and a first determining module, configured to combine the orientation information and the updated first image region to determine the first target position of the endoscope tip in three-dimensional space.

[0287] Optionally, the first acquisition module may include: a first processing submodule, configured to perform continuous masking processing on the first image region to obtain an updated first image region if the pixels corresponding to the endoscope body contained in the first image region are in a discrete state; and a second processing submodule, configured to use the first image region as the updated first image region if the pixels corresponding to the endoscope body contained in the first image region are in a continuous state.

[0288] Optionally, the first acquisition module may include: a second processing submodule, used to perform fitting processing on the pixels belonging to the endoscope body in the updated first image region to obtain the centerline fitting curve of the endoscope body; and a first determination submodule, used to determine the orientation information based on the coordinate information of any two pixels on the centerline fitting curve on the target coordinate axis.

[0289] Optionally, the first determining module may include: a first acquisition submodule, used to acquire two-dimensional coordinate information of the endoscope tip in the first image region based on orientation information, the obtained centerline fitting curve, and pixels belonging to the endoscope body in the first image region; and a third processing submodule, used to perform three-dimensional back projection processing on the acquired two-dimensional coordinate information of the endoscope tip in the first image region corresponding to different initial two-dimensional images to obtain the first target position of the endoscope tip in three-dimensional space.

[0290] Optionally, the first determining unit 1506 may include: a second acquisition module, used to perform contour detection on the second image region to acquire a centroid dataset corresponding to the sheath, wherein the centroid dataset includes the two-dimensional coordinate information of each centroid corresponding to the sheath in the second image region; a first filtering module, used to filter each centroid contained in the centroid dataset according to the centerline fitting curve and the end-point of the endoscope; a third acquisition module, used to take the selected centroid as the end-point of the sheath and acquire the two-dimensional coordinate information of the end-point of the sheath in the second image region; and a first processing module, used to perform three-dimensional back projection processing on the acquired two-dimensional coordinate information of the end-point of the sheath in the second image region corresponding to different initial two-dimensional images to obtain the second target position of the end-point of the sheath in three-dimensional space.

[0291] Optionally, the first filtering module may include: a second acquisition submodule, used to acquire the tangent line of the endoscope tip on the centerline fitting curve; a second determination submodule, used to determine the corresponding circle center on the tangent line based on the endoscope tip, the orientation information of the endoscope body, and the preset radius length; and a third determination submodule, used to construct a target circle with the circle center and the preset radius length, and to filter each centroid contained in the centroid dataset according to the filtering conditions based on the target circle to determine the specified centroid.

[0292] Optionally, the second determining unit 1508 may include: a fourth obtaining module, used to obtain the reference coordinate information of the reference point in three-dimensional space and the target coordinate information of the target point on the physiological channel in three-dimensional space, wherein the target point is the puncture point determined on the physiological channel based on the planned path; a second determining module, used to determine the distance in the orientation information based on the second target position and the reference coordinate information; and a third determining module, used to determine the first straight line from the target point to the reference point according to the target coordinate information and the reference coordinate information, and to determine the second straight line from the target point to the end point of the sheath according to the target coordinate information and the second target position, and to determine the angle between the first straight line and the second straight line as the angle in the orientation information.

[0293] Optionally, the device may further include: a third determining unit, used to determine the distance between the target point and the endoscope tip based on the first target position and target coordinate information, wherein the distance between the target point and the endoscope tip is used to determine the degree of matching between the actual movement path of the endoscope in the physiological channel and the planned path.

[0294] The aforementioned device for determining the orientation of the sheath end point may also include a processor and a memory. All of the aforementioned units are stored in the memory as program units, and the processor executes the aforementioned program units stored in the memory to achieve the corresponding functions.

[0295] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and by adjusting kernel parameters, graceful shutdown of devices of the same type awaiting shutdown can be controlled.

[0296] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0297] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and adjusting kernel parameters can improve the efficiency of traders.

[0298] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0299] Example 4

[0300] According to an embodiment of the present invention, a computer-readable storage medium is also provided, on which a program is stored, which, when executed by a processor, implements a method for determining the orientation of the sheath end point.

[0301] Example 5

[0302] According to an embodiment of the present invention, a processor is also provided, which is used to run a program, wherein the program executes a method for determining the orientation of the sheath end point during runtime.

[0303] Example 6

[0304] Figure 16 This is a schematic diagram of an electronic device for determining the orientation of the sheath tip according to an embodiment of the present invention, as shown below. Figure 16 As shown in the embodiment of the present invention, an electronic device is also provided. The device includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the above embodiments.

[0305] Example 7

[0306] According to embodiments of the present invention, a computer program product is also provided. Optionally, in this embodiment, the computer program product may include a computer program that, when executed by a processor, implements the method for determining the orientation of the sheath tip point according to the embodiments of the present application.

[0307] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0308] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0309] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0310] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0311] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0312] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0313] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0314] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0315] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0316] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A device for determining the orientation of the end point of a sheath tube, characterized in that, include: The acquisition unit is used to acquire initial two-dimensional images of the endoscope during insertion into the physiological channel from at least two angles, wherein the endoscope includes a sheath and an endoscope body; The segmentation unit is used to segment a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional image, respectively. The first determining unit is used to determine, based on the first image region, a first target position of the endoscope tip of the endoscope body in three-dimensional space, and to determine, based on the second image region, a second target position of the sheath tip of the sheath in three-dimensional space. The second determining unit is used to determine the orientation information between the sheath tip and the reference point based on the first target position and the second target position, wherein the orientation information is used to represent the distance and angle between the sheath tip and the reference point, and the reference point is used to represent the location where there is an abnormal state outside the physiological channel; The first determining unit performs the following steps to determine the second target position: performing contour detection on the second image region to obtain a centroid dataset corresponding to the sheath, wherein the centroid dataset includes two-dimensional coordinate information of each centroid corresponding to the sheath in the second image region; filtering each centroid contained in the centroid dataset according to the fitting curve of the centerline of the endoscope body and the end point of the endoscope; using the selected centroid as the end point of the sheath and obtaining the two-dimensional coordinate information of the end point of the sheath in the second image region; performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the end point of the sheath in the second image region corresponding to different initial two-dimensional images to obtain the second target position of the end point of the sheath in the three-dimensional space.

2. The apparatus according to claim 1, characterized in that, The segmentation unit is used to segment a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional image by performing the following steps: Threshold segmentation is performed on any initial two-dimensional image to obtain the first image region, wherein the first image region contains each pixel point in the any initial two-dimensional image whose corresponding pixel value is less than a preset first segmentation threshold. Threshold segmentation is performed on any initial two-dimensional image to obtain the second image region, wherein the second image region contains each pixel point in the any initial two-dimensional image whose corresponding pixel value is less than a preset second segmentation threshold.

3. The apparatus according to claim 2, characterized in that, The first determining unit is configured to determine, based on the first image region, a first target position in three-dimensional space of the endoscope tip of the endoscope body by performing the following steps: The updated first image region is obtained, and the orientation information of the endoscope body is determined based on the updated first image region; By combining the orientation information and the updated first image region, the first target position of the endoscope tip in the three-dimensional space is determined.

4. The apparatus according to claim 3, characterized in that, The first determining unit is configured to obtain the updated first image region by performing the following steps: If the pixels in the first image region corresponding to the endoscope body are in a discrete state, then the first image region is subjected to continuous masking processing to obtain an updated first image region. If the pixels in the first image region corresponding to the endoscope body are in a continuous state, then the first image region is used as the updated first image region.

5. The apparatus according to claim 3, characterized in that, The first determining unit is configured to determine the orientation information of the endoscope body based on the updated first image region by performing the following steps: The pixels belonging to the endoscope body in the updated first image region are fitted to obtain the centerline fitting curve of the endoscope body; The orientation information is determined based on the coordinate information of any two pixels on the centerline fitting curve on the target coordinate axis.

6. The apparatus according to claim 5, characterized in that, The first determining unit is configured to determine the first target position of the endoscope tip in the three-dimensional space by combining the orientation information and the updated first image region by performing the following steps: Based on the orientation information, the obtained centerline fitting curve, and the pixels belonging to the endoscope body in the first image region, the two-dimensional coordinate information of the endoscope end point in the first image region is obtained; The two-dimensional coordinate information of the endoscope tip in the first image region corresponding to different initial two-dimensional images is subjected to three-dimensional back projection processing to obtain the first target position of the endoscope tip in the three-dimensional space.

7. The apparatus according to claim 1, characterized in that, The first determining unit is configured to perform the following steps to filter the centroids contained in the centroid dataset based on the centerline fitting curve and the endoscope endpoint: Obtain the tangent line at the end point of the endoscope on the centerline fitting curve; Based on the endoscope tip, the orientation information of the endoscope body, and the preset radius length, the corresponding center of the circle is determined on the tangent. A target circle is constructed using the center and a preset radius. Based on the target circle, each centroid in the centroid dataset is filtered according to the filtering conditions to determine the specified centroid.

8. The apparatus according to claim 7, characterized in that, The filtering criteria include at least the following: The designated centroid is located within the target circle; The distance between the specified centroid and the tangent is less than or equal to a first distance threshold. The distance between the specified centroid and the end point of the endoscope is greater than or equal to the distance between any centroid in the centroid dataset and the end point of the endoscope.

9. The apparatus according to claim 1, characterized in that, The second determining unit is configured to perform the following steps to determine the orientation information between the sheath tip and the reference point based on the first target position and the second target position: The reference coordinate information of the reference point in the three-dimensional space and the target coordinate information of the target point on the physiological channel in the three-dimensional space are obtained, wherein the target point is the puncture point determined on the physiological channel based on the planned path; Based on the second target location and the reference coordinate information, determine the distance in the orientation information; Based on the target coordinate information and the reference coordinate information, a first straight line from the target point to the reference point is determined, and based on the target coordinate information and the second target position, a second straight line from the target point to the end point of the sheath is determined. The angle between the first straight line and the second straight line is determined as the angle in the azimuth information.

10. The apparatus according to claim 9, characterized in that, The device is also used to perform the following steps: Based on the first target location and the target coordinate information, the distance between the target point and the endoscope tip is determined, wherein the distance between the target point and the endoscope tip is used to determine the degree of matching between the actual movement path of the endoscope in the physiological channel and the planned path.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the following method for determining the orientation of the sheath tip: acquiring initial two-dimensional images of the endoscope during insertion into the physiological channel from at least two angles, wherein the endoscope includes a sheath and an endoscope body; segmenting a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional images; determining a first target position of the endoscope tip of the endoscope body in three-dimensional space based on the first image region, and determining a second target position of the sheath tip of the sheath in three-dimensional space based on the second image region; determining orientation information between the sheath tip and a reference point based on the first target position and the second target position, wherein the orientation information is used to represent the distance and angle between the sheath tip and the reference point, and the reference point is used to represent the location of an abnormal state outside the physiological channel; The method of determining the second target position of the sheath tip in the three-dimensional space based on the second image region includes: performing contour detection on the second image region to obtain a centroid dataset corresponding to the sheath, wherein the centroid dataset includes the two-dimensional coordinate information of each centroid corresponding to the sheath in the second image region; filtering each centroid contained in the centroid dataset according to the fitting curve of the centerline of the endoscope body and the end point of the endoscope; using the selected centroid as the end point of the sheath and obtaining the two-dimensional coordinate information of the end point of the sheath in the second image region; and performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the end point of the sheath in the second image region corresponding to different initial two-dimensional images to obtain the second target position of the sheath tip in the three-dimensional space.

12. An electronic device, characterized in that, The device includes a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the following method for determining the orientation of the sheath tip: acquiring initial two-dimensional images of the endoscope during insertion into the physiological channel from at least two angles, wherein the endoscope includes a sheath and an endoscope body; segmenting a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional images; determining a first target position of the endoscope tip of the endoscope body in three-dimensional space based on the first image region, and determining a second target position of the sheath tip of the sheath in three-dimensional space based on the second image region; and determining orientation information between the sheath tip and a reference point based on the first and second target positions, wherein the orientation information represents the distance and angle between the sheath tip and the reference point, and the reference point represents the location of an abnormal state outside the physiological channel. The method of determining the second target position of the sheath tip in the three-dimensional space based on the second image region includes: performing contour detection on the second image region to obtain a centroid dataset corresponding to the sheath, wherein the centroid dataset includes the two-dimensional coordinate information of each centroid corresponding to the sheath in the second image region; filtering each centroid contained in the centroid dataset according to the fitting curve of the centerline of the endoscope body and the end point of the endoscope; using the selected centroid as the end point of the sheath and obtaining the two-dimensional coordinate information of the end point of the sheath in the second image region; and performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the end point of the sheath in the second image region corresponding to different initial two-dimensional images to obtain the second target position of the sheath tip in the three-dimensional space.

13. A computer program product, characterized in that, The system includes a computer program that, when executed by a processor, implements the following method for determining the orientation of the sheath tip: acquiring initial two-dimensional images of the endoscope during insertion into the physiological channel from at least two angles, wherein the endoscope includes a sheath and an endoscope body; segmenting a first image region containing the endoscope body and a second image region containing the sheath from the initial two-dimensional images; determining a first target position of the endoscope tip of the endoscope body in three-dimensional space based on the first image region, and determining a second target position of the sheath tip of the sheath in three-dimensional space based on the second image region; and determining orientation information between the sheath tip and a reference point based on the first and second target positions, wherein the orientation information represents the distance and angle between the sheath tip and the reference point, and the reference point represents the location of an abnormal state outside the physiological channel. The method of determining the second target position of the sheath tip in the three-dimensional space based on the second image region includes: performing contour detection on the second image region to obtain a centroid dataset corresponding to the sheath, wherein the centroid dataset includes the two-dimensional coordinate information of each centroid corresponding to the sheath in the second image region; filtering each centroid contained in the centroid dataset according to the fitting curve of the centerline of the endoscope body and the end point of the endoscope; using the selected centroid as the end point of the sheath and obtaining the two-dimensional coordinate information of the end point of the sheath in the second image region; and performing three-dimensional back projection processing on the obtained two-dimensional coordinate information of the end point of the sheath in the second image region corresponding to different initial two-dimensional images to obtain the second target position of the sheath tip in the three-dimensional space.

Images