Navigation system and readable storage medium

Abstract

The application provides a navigation system and a readable storage medium, the navigation system is applied to a surgical robot, the surgical robot comprises a processor, a catheter and / or an instrument, the catheter and / or the instrument are provided with an image device for collecting medical images, and the processor is configured to execute the following steps: acquiring a plurality of medical images of a training object, the plurality of medical images satisfying a first preset condition; training according to the plurality of medical images to obtain an identification network; in response to a navigation instruction, identifying a first pipe segment from an intraoperative medical image by using the identification network; acquiring a planned path, the planned path being composed of a plurality of connected second pipe segments; matching a target pipe segment from the second pipe segments based on the first pipe segment; generating a navigation marker for marking the target pipe segment; and displaying the navigation marker and the target pipe segment on a user interface to guide the movement of the catheter, thereby achieving precise and efficient navigation.

Description

Technical Field

[0001] This application belongs to the field of medical devices, and in particular relates to a navigation system and a readable storage medium. Background Technology

[0002] Minimally invasive medical techniques aim to reduce the amount of tissue damaged during medical procedures, thereby minimizing patient recovery time, discomfort, and harmful side effects. These techniques often involve inserting catheters through natural openings in the patient's anatomy or through surgical incisions. With the assistance of an intraoperative navigation system, these catheters navigate the complex structure of the body's tubes to reach or approach the target.

[0003] In most cases, catheters move along the body's tubules (such as bronchi, blood vessels, ureters, etc.) without freely traversing them in a way that could damage them. Therefore, the movement path of catheters and / or instruments can be considered confined within the body's tubules and similar to their centerline, especially for smaller tubules where the radial range of motion is limited, resulting in an even higher similarity between the movement path and the centerline. However, due to the numerous branches and complex structure of the body's tubules, and the ease with which intraoperative movements, such as heartbeat and respiration, can interfere with tissue deformation, it is difficult for surgeons to maneuver catheters and / or instruments to reach the target lesion. Accurate and real-time navigation guidance is therefore essential.

[0004] Therefore, improving the accuracy and efficiency of real-time intraoperative navigation is a problem that needs to be solved. Summary of the Invention

[0005] This application provides a navigation system and a readable storage medium that can improve the accuracy and efficiency of real-time intraoperative navigation and reduce the impact of the movement of human anatomical structures.

[0006] In a first aspect, embodiments of this application provide a navigation system applied to a surgical robot, the surgical robot including a processor, catheters and / or instruments, the catheters and / or instruments being provided with an imaging device for acquiring medical images, and the processor being configured to perform the following steps:

[0007] Acquire multiple first medical images of the anatomical structure of the training object, wherein the multiple first medical images satisfy a first preset condition, wherein the first preset condition includes a first medical image representing the stretching process state of the anatomical structure and a first medical image representing the contraction process state of the anatomical structure among the multiple first medical images;

[0008] A recognition network is obtained by training based on the multiple first medical images;

[0009] In response to the user's navigation instructions, the first tube segment is identified from the patient's intraoperative medical images acquired by the imaging device using the recognition network;

[0010] Obtain a planned path from the initial position to the target position, wherein the planned path is composed of multiple connected segments of the second pipeline;

[0011] Based on the first pipeline segment, the target pipeline segment is matched from the second pipeline segment;

[0012] Generate navigation markers to mark the target pipeline segments;

[0013] The user interface displays the navigation markers and the target tubing segments to guide the movement of the catheter and / or instruments.

[0014] Optionally, the training subject's body surface is equipped with a body surface sensor, and during the acquisition of multiple first medical images of the training subject's anatomical structure, the processor is configured to perform the following steps:

[0015] At the same time, acquire the first medical image and the body surface data of the training object sensed by the body surface sensor;

[0016] Based on the body surface data, the process state of the first medical image representation is determined, and the process state includes either the stretching process state or the contraction process state.

[0017] Repeat the above steps until the number of first medical images representing multiple different process states meets a preset value. The first preset condition also includes that the number of first medical images representing multiple different process states meets the preset value.

[0018] Optionally, prior to training based on the plurality of first medical images, the processor is configured to perform the following steps:

[0019] A second medical image of one of the stretching states of the training object is acquired, and a first anatomical model of the training object is generated based on the second medical image;

[0020] A third medical image of one of the contraction states of the training object is obtained, and a second anatomical model of the training object is generated based on the third medical image;

[0021] Determine the maximum contraction ratio of the first anatomical model and the second anatomical model.

[0022] Optionally, before training the recognition network based on the plurality of first medical images, the processor is configured to perform the following steps:

[0023] Determine the first scaling ratio based on the maximum shrinkage ratio;

[0024] Based on the first scaling ratio, the first medical image is scaled to obtain a scaled first medical image.

[0025] In the process of training the recognition network based on the plurality of first medical images, the processor is configured to perform the following steps:

[0026] The recognition network is obtained by training based on the plurality of first medical images and the scaled first medical images.

[0027] Optionally, before identifying the first duct segment from intraoperative medical images of the patient acquired by the imaging device using the recognition network in response to a user's navigation command, the processor is configured to perform the following steps in response to a motion compensation command:

[0028] Determine the second scaling ratio based on the maximum shrinkage ratio;

[0029] The intraoperative medical image is scaled based on the second scaling ratio.

[0030] Optionally, the catheter and / or device are provided with a tracking sensor for sensing the position of the catheter and / or device. The tracking sensor is used to acquire the actual path point of the catheter and / or device. In matching the target tubing segment from the second tubing segment based on the first tubing segment, the processor is configured to perform the following steps:

[0031] The actual path points are corrected to obtain the corrected path points;

[0032] Based on the actual path points and the corrected path points, determine the positioning correction amount;

[0033] Determine the relative positions of the tubular segments in the intraoperative medical images;

[0034] Based on the first tube segment, the positioning correction amount, and the relative position of the tube segment in the intraoperative medical image, a third tube segment in the intraoperative medical image is identified, wherein the third tube segment is different from the first tube segment;

[0035] Based on the first pipeline segment and the third pipeline segment, the target pipeline segment is matched from the second pipeline segment.

[0036] Optionally, the catheter and / or device are provided with a tracking sensor for measuring the position of the catheter and / or device. The sensor is used to acquire the actual path point of the catheter and / or device, and the processor is configured to perform the following steps:

[0037] Acquire preoperative medical images of the patient and generate an anatomical model of the patient based on the preoperative medical images, wherein the preoperative medical images characterize the contraction or relaxation process of the patient's anatomical structures;

[0038] Obtain the transformation matrix, which is used to characterize the transformation relationship between the surgical environment coordinate system and the anatomical model coordinate system of the patient;

[0039] Based on the actual path point and the transformation matrix, determine the simulated path point of the catheter and / or device in the anatomical model of the patient corresponding to the actual path point; or, correct the actual path point to obtain a corrected path point, and based on the corrected path point and the transformation matrix, determine the simulated path point of the catheter and / or device in the anatomical model of the patient corresponding to the actual path point.

[0040] Mark the simulated path points in the anatomical model of the patient;

[0041] The user interface displays the patient's anatomical model and the simulated path points.

[0042] Optionally, in correcting the actual path points to obtain corrected path points, the processor is configured to perform the following steps:

[0043] Based on the actual path points, feature points that match the actual path points are determined, and the feature points are points in the anatomical structure of the patient.

[0044] Obtain the denoising weight matrix associated with the feature points, denoise the actual path points based on the denoising weight matrix to obtain denoised path points, and use the denoised path points as the corrected path points.

[0045] Optionally, a surface sensor is provided on the body surface of the training object. The tracking sensor acquires sampling points from the anatomical structure of the training object, and the surface sensor acquires test points from the body surface of the training object. In acquiring the denoising weight matrix associated with the feature points, the processor is configured to perform the following steps:

[0046] Obtain a subset of sampling points of the training object, wherein the subset of sampling points of the training object includes multiple sampling points;

[0047] Obtain the test point set of the training object, wherein the test point set of the training object includes multiple test points;

[0048] Based on the subset of sampling points and the set of test points, the denoising weight matrix corresponding to the feature points of the training object is determined.

[0049] Optionally, in correcting the actual path points to obtain corrected path points, the processor is configured to perform the following steps:

[0050] Acquire data on the movement of the catheter within the patient's anatomical network of channels;

[0051] The corrected path point is determined based on the actual path point, the movement data, and the intraoperative medical images.

[0052] Optionally, the intraoperative medical image represents the contraction or relaxation state of the patient's anatomical structures, a surface sensor is provided on the patient's body surface, and the processor is configured to perform the following steps:

[0053] At the same time, the intraoperative medical images and the patient's body surface data sensed by the body surface sensor are acquired;

[0054] Based on the body surface data, vital signs data of the patient's anatomical structure are determined, and the vital signs data characterize the patient's anatomical structure under different process states;

[0055] Based on the vital signs data, a vital signs curve of the patient's anatomical structure is generated and displayed on the user interface. The vital signs curve includes a respiratory curve and / or a heart rate curve.

[0056] Optionally, in the second aspect, embodiments of this application provide a navigation method, including:

[0057] In response to the user's navigation instructions, the first tube segment is identified from the patient's intraoperative medical images acquired by the imaging device using the recognition network;

[0058] The actual path points are corrected to obtain the corrected path points;

[0059] Based on the actual path points and the corrected path points, determine the positioning correction amount;

[0060] Determine the relative positions of the tubular segments in the intraoperative medical images;

[0061] Based on the first tube segment, the positioning correction amount, and the relative position of the tube segment in the intraoperative medical image, a third tube segment in the intraoperative medical image is identified, wherein the third tube segment is different from the first tube segment;

[0062] Obtain a planned path from the initial position to the target position, wherein the planned path is composed of multiple connected segments of the second pipeline;

[0063] Based on the first pipeline segment and the third pipeline segment, the target pipeline segment is matched from the second pipeline segment.

[0064] Generate navigation markers to mark the target pipeline segments;

[0065] The user interface displays the navigation markers and the target tubing segments to guide the movement of the catheter and / or instruments.

[0066] Thirdly, embodiments of this application provide a navigation method, including:

[0067] Acquire multiple first medical images of the anatomical structure of the training object, wherein the multiple first medical images satisfy a first preset condition, wherein the first preset condition includes a first medical image representing the stretching process state of the anatomical structure and a first medical image representing the contraction process state of the anatomical structure among the multiple first medical images;

[0068] A recognition network is obtained by training based on the multiple first medical images;

[0069] In response to the user's navigation instructions, the first tube segment is identified from the patient's intraoperative medical images acquired by the imaging device using the recognition network;

[0070] Obtain a planned path from the initial position to the target position, wherein the planned path is composed of multiple connected segments of the second pipeline;

[0071] Based on the first pipeline segment, the target pipeline segment is matched from the second pipeline segment;

[0072] Generate navigation markers to mark the target pipeline segments;

[0073] The user interface displays the navigation markers and the target tubing segments to guide the movement of the catheter and / or instruments.

[0074] Fourthly, embodiments of this application provide a computer-readable storage medium that stores a program that, when executed by a processor, implements some or all of the steps of the navigation methods described in the second and third aspects.

[0075] Fifthly, embodiments of this application provide a computer program product that, when run on a surgical robot, causes the surgical robot to perform some or all of the steps of the navigation method described in any of the second and third aspects above.

[0076] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0077] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment acquires multiple first medical images of the anatomical structure of the training object, wherein the multiple first medical images satisfy a first preset condition, the first preset condition including a first medical image representing the stretching process state of the anatomical structure and a first medical image representing the contraction process state of the anatomical structure among the multiple first medical images; a recognition network is obtained by training based on the multiple first medical images; in response to the user's navigation command, the recognition network is used to identify a first tube segment from the intraoperative medical images of the patient acquired by the imaging device; a planned path from the initial position to the target position is obtained, wherein the planned path consists of multiple connected second tube segments; a target tube segment is matched from the second tube segments based on the first tube segment; a navigation mark is generated to mark the target tube segment; and the navigation mark and the target tube segment are displayed on the user interface to guide the movement of the catheter and / or instrument, thereby achieving precise and efficient navigation. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is a schematic diagram of the structure of a catheter robot provided in one embodiment of this application;

[0080] Figure 2a This is a schematic diagram of the structure of the conduit assembly and power unit provided in an embodiment of this application;

[0081] Figure 2b This is a schematic diagram of the structure of a catheter assembly provided in one embodiment of this application;

[0082] Figure 2c This is a schematic diagram of the structure of a catheter assembly provided in one embodiment of this application;

[0083] Figure 3 A schematic flowchart of a navigation method for a catheter robot provided in an embodiment of this application;

[0084] Figure 4 This is a schematic diagram of the first lung segment in a navigation method according to an embodiment of this application;

[0085] Figure 5 This is a schematic diagram of the path planning in a navigation method according to an embodiment of this application;

[0086] Figure 6 This is a schematic diagram of the target lung segment in a navigation method according to an embodiment of this application;

[0087] Figure 7 This is a schematic diagram of a preoperative image annotated in a navigation method according to an embodiment of this application;

[0088] Figure 8a This is a schematic diagram of the location of a surface sensor on a patient's body surface according to an embodiment of this application;

[0089] Figure 8b A schematic diagram of the respiratory coefficient obtained from EM data of at least one respiratory cycle sensed by a body surface sensor, according to an embodiment of this application.

[0090] Figure 9a A schematic diagram of a lung segment provided in an embodiment of this application;

[0091] Figure 9b A schematic diagram of the target lung segment provided in an embodiment of this application;

[0092] Figure 10 This is a schematic diagram illustrating the effect of denoising the catheter movement path before and after one embodiment of this application.

[0093] Figure 11 A simplified schematic diagram of a user interface for user operation provided in an embodiment of this application;

[0094] Figure 12 A flowchart illustrating a navigation method for a catheter robot provided in another embodiment of the application;

[0095] Figure 13 A schematic diagram of simulated feature points in an anatomical model provided in an embodiment of this application;

[0096] Figures 14-15 A flowchart illustrating a navigation method for a catheter robot provided in another embodiment of the application;

[0097] Figure 16 This is a schematic diagram of the control device of a remote medical system according to an embodiment of this application. Detailed Implementation

[0098] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0099] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0100] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0101] The navigation system and computer-readable storage medium according to embodiments of this application are described below with reference to the accompanying drawings.

[0102] Figure 1 A catheter system 1000 according to an embodiment of this application is shown. The catheter system 1000 includes an imaging cart 100, a trolley 200 connected to the imaging cart 100, a main controller 300, a catheter assembly 400 that can be coupled to the trolley 200, a sensor system 500 connected to the trolley 200, and a control system 600 for controlling the catheter assembly 400, the main controller 300, the sensor system 500, and the imaging cart 100. The main controller 300 can be wired or wirelessly connected to the trolley 200. When an operator performs various procedures on a patient next to the trolley 200, they can trigger control commands by operating the main controller 300, which, driven by the trolley 200, controls the catheter assembly 400 to move forward, retract, and bend / turn.

[0103] The trolley 200 can typically be moved to the side of the operating table to engage the catheter assembly 400. Under control commands, it controls the catheter assembly 400 to move vertically, horizontally, or in both vertical and horizontal directions, thus providing a better preoperative preparation angle for the operation of the catheter assembly 400. These control commands can be triggered by the operator operating the main controller 300, or by the operator directly clicking or pressing buttons on the trolley 200. In other embodiments, the control commands can also be voice control or commands triggered via force feedback mechanisms.

[0104] like Figure 1As shown, the trolley 200 may further include a base 210, a sliding seat 220 that can move up and down along the base 210, and two robotic arms 230 fixedly connected to the sliding seat 220. Each robotic arm 230 may include multiple arm segments connected at joints, providing multiple degrees of freedom for the robotic arm 230, for example, seven degrees of freedom corresponding to seven arm segments. A power unit (not shown) is installed at the end of each robotic arm 230. The power unit of the robotic arm 230 is used to engage the conduit assembly 400 and, under the driving action of the power unit, controls the end of the conduit assembly 400 to bend and turn accordingly. The two robotic arms 230 may have identical or partially identical structures; one robotic arm 230 is used to engage the inner conduit assembly 410, and the other robotic arm 230 is used to engage the outer conduit assembly 420. During installation, the outer conduit assembly 420 can be installed first. After the outer conduit assembly 420 is installed, the conduit of the inner conduit assembly 410 is inserted into the conduit of the outer conduit assembly 420.

[0105] The sensor system 500 has one or more subsystems for receiving information about the catheter assembly 400. The subsystems may include: a position sensor system; a shape sensor system for determining the position, orientation, velocity, rate, pose, and / or shape of the distal end of the catheter assembly 400 and / or along one or more segments of the catheter that may constitute the catheter assembly 400; and / or a visualization system for capturing images from the distal end of the catheter assembly 400. The visualization system may include an imaging device, such as a camera.

[0106] The imaging vehicle 100 may be equipped with a display system 110 and a flushing system (not shown in the figure), etc. The display system 110 is used to display images or representations of the surgical site and catheter assembly 400 generated by the subsystems of the sensor system 500. It can also display real-time images of the surgical site and catheter assembly 400 captured by a visualization system. Image data from imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound can also be used to present images of the surgical site recorded preoperatively or intraoperatively. Preoperative or intraoperative medical image data can be presented as two-dimensional, three-dimensional, or four-dimensional (e.g., time-based or rate-based information) images and / or as images from models created based on preoperative or intraoperative medical image datasets, and virtual navigation images can also be displayed. In the virtual navigation images, the actual position of the catheter assembly 400 is registered with the preoperative images to present a virtual image of the catheter assembly 400 within the surgical site to the operator from the outside.

[0107] The control system 600 includes at least one memory and at least one processor. It is understood that the control system 600 can be integrated into the trolley 200 or the imaging cart 100, or it can be set up independently. The control system 600 can support wireless communication protocols such as IEEE 802.11, IrDA, Bluetooth, HomeRF, DECT, and wireless telemetry. The control system 600 can transmit one or more signals instructing the catheter assembly 400 to move, which is then moved by the power unit. The catheter assembly 400 can extend to the surgical site within the body via an opening in the patient's natural body cavity or a surgical incision.

[0108] Furthermore, the control system 600 may include a mechanical control system (not shown in the figure) and an image processing system (not shown in the figure). The mechanical control system is used to control the movement of the catheter assembly 400, and therefore can be integrated into the trolley 200. The image processing system is used for virtual navigation path planning, and therefore can be integrated into the imaging vehicle 100. Of course, the various subsystems of the control system 600 are not limited to the specific cases listed above, and can be reasonably set according to actual conditions. The image processing system can image the surgical site based on images of the surgical site recorded before or during surgery, using the aforementioned imaging technology. Software used in conjunction with manual input can also convert the recorded images into two-dimensional or three-dimensional composite images of parts or entire anatomical organs or segments. During the virtual navigation procedure, the sensor system 500 can be used to calculate the position of the catheter assembly 400 relative to the patient's anatomical structures.

[0109] The internal catheter assembly 410 and the external catheter assembly 420 have largely the same structure, each having a slender and flexible internal catheter 41 and an external catheter 42, respectively. The diameter of the external catheter 42 is slightly larger than that of the internal catheter 41, so that the internal catheter 41 can pass through the external catheter 42 and provide some support for the internal catheter 41. This allows the internal catheter 41 to reach the target location in the patient's body, so as to facilitate operations such as tissue or cell sampling from the target location.

[0110] Certain movements of the master controller 300 can cause corresponding movements of the catheter assembly 400. For example, when the operator moves the directional lever of the master controller 300 up or down, the movement of the directional lever can be mapped to a corresponding pitch movement of the end of the catheter assembly 400; when the operator moves the directional lever of the master controller 300 left or right, the movement of the directional lever can be mapped to a corresponding yaw movement of the end of the catheter assembly 400. In this embodiment, the master controller 300 can control the end of the catheter assembly 400 to move within a 360° spatial range.

[0111] Figures 2a-2c A catheter assembly 400 according to an embodiment of this application is shown. The catheter assembly 400 is configured to engage with a power unit 240 of a robotic arm 230. The catheter assembly 400 includes an instrument cartridge 45 configured to engage with the power unit 240 and a catheter 48 connected to the instrument cartridge 45. "Engagement" refers to a state where, when the instrument cartridge 45 is installed in the power unit 240, the driving force of the power unit 240 can be transmitted to the instrument cartridge 45, enabling the catheter 48 to move normally. For example, under the driving force of the power unit 240, the end of the catheter 48 can bend or change direction. The catheter 48 may include the aforementioned inner catheter 41 and outer catheter 42. In this embodiment of the application, unless otherwise specified, the catheter is not distinguished as an inner catheter or an outer catheter.

[0112] The catheter assembly 400 also includes a channel tube 12, an imaging device 13, and a tip 14. The tip 14 is connected to the distal end of the catheter 48. The imaging device 13 may be disposed at the tip 14 and is used to acquire medical images. The imaging device 13 may include a light source 13b and an image module 13a. The light source 13b is used to provide supplemental lighting for the environment in which the image module 13a is located.

[0113] The catheter 48 may also be equipped with a tracking sensor 15 for locating the distal end of the catheter 10. The tracking sensor 15 is at least partially fixed to the tip 14. For example, the tracking sensor 15 may include an EM (Electromagnetic) sensor. A detection magnetic field is arranged in the patient's environment. This detection magnetic field can be generated by a magnetic field generator arranged on one side of the patient. When the position and angle of the EM sensor are different, the current generated in the coil of the EM sensor is different, thereby locating the position and orientation of the distal end of the catheter 10. In other embodiments, the positioning sensor may be replaced by other sensing modules, such as a light sensor, an ultrasound probe, a gyroscope, etc. The number of tracking sensors 15 can be set as needed. Optionally, the tracking sensor 15 is disposed in the inner catheter 41.

[0114] The catheter assembly 400 also includes an instrument 30, which is detachably inserted into and removed from the working channel 10H where the channel tube 12 is located. The instrument 30 includes, but is not limited to, various surgical instruments, such as cauterization instruments, clamping instruments, cutting instruments, suturing instruments, electrocautery hooks, irrigation instruments, etc. Figure 2c The device 30 shown is a flushing device. The device 30 has an axially opened fluid channel 321 for fluid to pass through. By inserting the device 30 along the working channel 10H of the conduit 10, the target location can be flushed.

[0115] For example, the tracking sensor 15 can be fabricated on the distal end face of the instrument 30, and the tracking sensor 15 can be used to provide navigation for the instrument 30, for example, providing real-time navigation during rinsing. Figure 2c The device 30 includes an elongated tube 32 and a tracking sensor 15. The device 30 is configured to be detachably inserted into a working channel 10H. The elongated tube 32 has a fluid channel 321 extending along its length. The tracking sensor 15 is fixed to the distal end of the elongated tube 32 and is disposed adjacent to the fluid channel 321. Through this fluid channel 321, the injection and extraction of fluids such as flushing fluid can be achieved. The tracking sensor 15 disposed on the device 30 can be the same as the tracking sensor 15 disposed on the catheter, including but not limited to an EM sensor, a light sensor, an ultrasonic probe, and a gyroscope. The working channel 10H and the elongated tube 32 are contour-matched, with a clearance fit between them.

[0116] For example, the elongated tube 32 is provided with an installation channel 322 extending through its length and an arc-shaped partition 323 extending through its length and arching toward the installation channel 322. The tracking sensor 15 is fixed in the installation channel 322, and the installation channel 322 is separated from the fluid channel 321 by the partition 323. The signal line of the tracking sensor 15 can be led out from the installation channel 322 to the proximal end.

[0117] The inner catheter 41 and outer catheter 42 typically move in tandem, but they can also move independently. The instrument 30 usually moves within the catheter; therefore, as long as the catheter is accurately positioned, the instrument 30 will also be accurately positioned. The instrument 30 can also move independently of the catheter, and it can be equipped with a tracking sensor 15. Similarly, the instrument 30 can also be equipped with an imaging device 13. The tracking sensor 15 and the imaging device 13 can individually position and navigate the outer catheter 42, inner catheter 41, or instrument 30, or they can simultaneously position and navigate them.

[0118] In this application, the end, also referred to as the distal end or head, refers to the end away from the instrument box 45; the anterior end, also referred to as the proximal end or tail, refers to the end close to the instrument box 45.

[0119] Before surgery, preoperative examinations are typically performed, and an anatomical model of the human body is constructed based on the preoperative medical images obtained. This model is used for navigation during the actual operation. However, the human body's vascular network can deform due to movement or compression. Therefore, the actual state of the vascular network during surgery may vary significantly over time and differ considerably from the medical images taken during the preoperative examinations. Thus, using an anatomical model of the human body constructed based on preoperative medical examinations for navigation may lead to errors. Valves can include, for example, bronchi, blood vessels, ureters, and intestines. Movement includes periodic and non-periodic movements. Periodic movements include actions such as heartbeat and breathing, while non-periodic movements include actions such as coughing.

[0120] Preoperative medical examinations are performed on the patient, such as scanning the target area to obtain medical images (e.g., CT, MRI, OCT, or ultrasound scans). Three-dimensional reconstruction is then performed on these preoperative images to obtain a three-dimensional preoperative image. Appropriate image processing is then applied to this three-dimensional preoperative image to obtain the patient's anatomical model. This anatomical model of the patient's anatomical structure is typically a three-dimensional or four-dimensional model. In some embodiments, taking the bronchus as an example and the anatomical model as a three-dimensional model, the steps for obtaining the anatomical model of the anatomical structure are as follows:

[0121] Preoperative images of anatomical structures are obtained through methods such as CT, MRI, OCT, or ultrasound scans (or imaging). During the scan, the patient typically needs to take a deep breath and hold it until the scan is complete. The bronchi are largest and their terminal branches are easier to image when the patient is inhaling.

[0122] Preoperative medical images of anatomical structures are segmented and reconstructed. For example, segmentation algorithms such as region growing and convolutional neural networks can be used to segment the patient's medical images to identify the bronchi in the lungs; then, the segmented images are reconstructed into a three-dimensional model using algorithms such as the moving cube algorithm.

[0123] Image segmentation is used to extract surgical pathways from 3D images. To facilitate intraoperative navigation, the skeleton of these pathways is extracted to obtain their centerlines. The centerline, also known as the skeleton of the human body's pathways, is a curve describing some of the geometric features of these pathways. Located in the middle of the pathways, it shares the same topological structure as the original pathways and is typically a single pixel wide. The centerline provides the information needed for intraoperative navigation, and compared to the original human pathways, its data volume is significantly reduced, making processing easier and facilitating real-time navigation.

[0124] The pipeline centerline is a three-dimensional curve. In practical applications, the pipeline centerline is often stored and used as multiple three-dimensional points. In other words, the pipeline centerline includes these three-dimensional points, which can be called skeleton points. The set of skeleton points can be called a skeleton point set or skeleton point cloud. Preprocessing can be performed on the pipeline centerline. Preprocessing may include extracting feature skeleton points from the skeleton point set and / or resampling the pipeline centerline. Feature skeleton points generally include bifurcation points and terminal points, used to segment the pipeline centerline. Each segment can be called a pipeline centerline segment or simply a pipeline segment. For example, for the lungs, bifurcation points form connecting segments, i.e., lung segments. Generally speaking, most people have similar lung segment distributions, but subtle differences cannot be ruled out. Therefore, for the accuracy of medical operations, in this embodiment, lung segments can also be confirmed for each patient. That is, after identifying the bifurcation points, all lung segments are determined based on the principle that adjacent bifurcation points constitute lung segments.

[0125] This application aims to eliminate the impact of deformation caused by movement or compression on the pipeline network, ensuring that the pre-trained recognition network can effectively identify pipeline segments from intraoperative medical images during actual surgery, and then perform navigation based on the identified pipeline segments. The effective identification of pipeline segments in this application includes two approaches: training a recognition network that can remove the impact of deformation during the training process; or removing the impact of deformation during the identification process using the recognition network.

[0126] This application provides a navigation system applied to a surgical robot. The surgical robot includes a processor, catheters, and / or instruments. The catheters and / or instruments are equipped with imaging devices for acquiring medical images. The processor is configured to perform the following steps to implement the navigation method provided in one embodiment of this application: Figure 3 The diagram shows a flowchart of a navigation method provided in one embodiment of this application. The navigation method includes: pre-training a recognition network, comprising steps S10 and S20. The recognition network can be used to identify duct segments in intraoperative medical images acquired by an imaging device.

[0127] Step S10: Acquire multiple first medical images of the anatomical structure of the training subject. These multiple first medical images satisfy a first preset condition, which includes a first medical image representing the expansion state of the anatomical structure and a first medical image representing the contraction state of the anatomical structure among the multiple first medical images. The training subject may be the patient or a non-patient.

[0128] The periodic movements of anatomical structures generally encompass two types of process states: contraction and relaxation. Furthermore, each process state includes multiple contraction / relaxation states of varying degrees. Each cycle consists of multiple moments, and the anatomical structure corresponds to different process states at different moments, resulting in different vital sign data. For example, when the periodic movement is respiratory, the anatomical structure may include the bronchi. The bronchi may correspond to a contraction state (expiration) or a relaxation state (inspiration) at different moments. Different degrees of expiration are also considered different process states, and different degrees of inspiration are also considered different process states. That is, the contraction state includes many different degrees of contraction, and the relaxation state includes many different degrees of relaxation; the process state corresponding to different moments within a cycle is unique. The medical images acquired at different moments or in different process states are also different; therefore, medical images can also characterize the process states of anatomical structures. Taking respiratory movement as an example, the multiple first medical images include a first medical image representing the inspiratory state of the bronchus and a first medical image representing the expiratory state of the bronchus.

[0129] Step S20: Train the recognition network based on the plurality of first medical images to obtain a recognition network. In this embodiment, during the training process of the recognition network, the first medical images used for training include both first medical images in the expansion state and first medical images in the contraction state. This allows the trained recognition network to recognize intraoperative medical images in both the expansion and contraction states, better eliminating the influence of deformation caused by periodic movement or compression of the duct network, and improving the recognition network's ability to identify duct segments.

[0130] Furthermore, the first preset condition also includes that the number of first medical images representing multiple different process states meets the preset value, the body surface of the training object is provided with a body surface sensor, and the acquisition of multiple first medical images of the anatomical structure of the training object specifically includes steps (1) to (3):

[0131] (1) Acquire the first medical image and the surface data of the training object sensed by the surface sensor at the same time, that is, the time when the first medical image is acquired is the same as the time corresponding to the surface data, or the process state represented by the first medical image is the same as the process state represented by the surface data. By obtaining the medical image and surface data at the same time or in the same process state, the specific process state represented by the medical image can be determined based on the surface data.

[0132] The body surface data of the training subject is acquired through surface sensors, and the sampling period for acquiring the data includes at least one cycle. The duration of a normal respiratory cycle is typically 3-5 seconds. Taking a surgical robot as an example (e.g., a catheterization machine), the robot also includes surface sensors disposed on the body surface of the training subject; there are at least three surface sensors used to acquire the body surface data. These surface sensors can generally be position sensors or pose sensors, and are typically non-invasively and stably positioned on the body surface corresponding to the anatomical structure of the training subject; that is, the surface sensors are generally exposed on the body surface. In addition to EM sensors, other surface sensors such as optical positioning sensors can also be used.

[0133] In some embodiments, such as when the anatomical structure is the bronchus, the respiratory amplitude varies across different parts of the lungs. The surface sensor located in the middle of the main bronchus actually measures the respiratory amplitude at the front end of the main bronchus, which is relatively small. The surface sensors on the left and right sides of the bronchial distal end are mainly used to measure the respiratory amplitude at the distal ends of the left and right lungs, where the diaphragm movement at the lower lung segment has the largest amplitude during respiration. Therefore, the surface sensors are positioned on the subject's body surface at a relatively stable point in the middle of the main bronchus, and at points on the left and right sides of the bronchial distal end where the amplitude changes more significantly. As the subject's respiratory state changes, the anatomical structure changes in six dimensions: vertical, horizontal, front-back, and vertical. Theoretically, the more sensors there are, the richer the changes in respiratory state are obtained; that is, the more surface sensors there are, the larger the area of ​​the subject's body surface covered, and the more accurate the sensing of respiratory movements of the anatomical structure. For example, at least three surface sensors are deployed on the chest surface of the subject. One of the surface sensors can be placed in the middle of the trainee's chest, and the other two surface sensors can be placed in the area of ​​the trainee's chest corresponding to the 7th rib on the left and the area of ​​the trainee's chest corresponding to the 7th rib on the right, respectively. During the operation, adhesive tape or similar materials are needed to fix the surface sensors in place to prevent them from sliding during the operation.

[0134] (2) Based on the body surface data, determine the process state of the first medical image representation, wherein the process state includes either the stretching process state or the contraction process state;

[0135] First, based on the surface data, vital sign data of the training subject's anatomical structure are determined. This vital sign data characterizes the training subject's anatomical structure under different process states. The vital sign data may include the area S, slope, or motion coefficient of the shape enclosed by the surface sensors. The area of ​​the shape enclosed by the surface sensors represents the size of the shape as the training subject's breathing changes; the slope represents the rate of change of the area; and the motion coefficient reflects the proportional relationship between the area in the current state and the areas in other process states. Specifically:

[0136] The area of ​​the shape enclosed by the surface sensors is determined based on the surface data of the training subject. This area includes a first area, a second area, and a third area. The first area includes the area of ​​the shape enclosed by the surface sensors in the first process state; the second area includes the area of ​​the shape enclosed by the surface sensors in the second process state; and the third area is the area S of the shape enclosed by the surface sensors in the current state. The first process state corresponds to a state where the anatomical structure is relatively extended, i.e., the extended process state; the second process state corresponds to a state where the anatomical structure is relatively contracted, i.e., the contracted process state. For example, for the lungs, when the first process state is a full inhalation state, the second area is the maximum area S enclosed by the surface sensors. max When the second process state is a complete exhalation state, the second area is the minimum area S enclosed by the body surface sensors. min .

[0137] Using three surface sensors of the training subjects as examples, we determine the motion coefficients in the current state. The positions of the surface sensors of the three training subjects are P1(X1, Y1, Z1), P2(X2, Y2, Z2), and P1(X3, Y3, Z3), respectively. We calculate the distances between each pair of surface sensors, i.e., the side lengths A, B, and C.

[0138] Calculate the lengths of A, B, and C using Euclidean distance. For example, to calculate the side length A:

[0139]

[0140] The lengths of B and C are calculated in the same way as those of A, and will not be repeated here.

[0141] Calculate the semi-perimeter P of the shape enclosed by the surface sensors of the training subject:

[0142] P = (A + B + C) / 2

[0143] The area S of the shape enclosed by the surface sensors in the current state:

[0144]

[0145] Based on the maximum area S max Minimum area S min The motion coefficient F in the current state is determined by the area S of the shape enclosed by the surface sensors of the training object.

[0146]

[0147] Where Fmax represents the maximum value of the designed motion coefficient, and Fmin represents the minimum value of the designed motion coefficient. For example, in some embodiments, if the design formula is based on the motion coefficient within the range of [-1, 1], then the motion coefficient F is:

[0148]

[0149] For example, in some embodiments, if the design formula is based on the motion coefficient being within the range of [-100, 100], then the motion coefficient F is:

[0150]

[0151] The respiration coefficient is set as the motion coefficient, with the respiration coefficient ranging from -1 to 1. The horizontal axis represents the number of EM data points acquired in real time by the surface sensors of the training subject as the respiratory duration or respiratory cycle changes. The vertical axis, from top to bottom, represents the area, slope, and respiration coefficient of the shape enclosed by the surface sensors of the training subject.

[0152] This application embodiment uses a respiratory coefficient F to intuitively and quickly determine the breathing state of the training subject, and enables or prohibits the movement of the catheter within the anatomical structure based on the breathing state. For example, if the respiratory coefficient F is positive, it indicates that the human chest cavity is in an inspiratory state in the first process state, with the chest cavity full and the bronchi in a relaxed state. The larger the value, the fuller the chest cavity, enabling the movement of the catheter within the anatomical structure. Alternatively, when the respiratory coefficient F is negative, the training subject's breathing state is determined to be expiratory, prohibiting the movement of the catheter within the anatomical structure. Another example is that the breathing state of the training subject is determined based on the respiratory coefficient F, and a prompt indicating whether the movement of the catheter within the anatomical structure is permitted is generated based on the breathing state. When the respiratory coefficient F is positive, the training subject's breathing state is determined to be inspiratory, generating a prompt allowing the movement of the catheter within the anatomical structure; or, when the respiratory coefficient F is negative, the training subject's breathing state is determined to be expiratory, generating a prompt prohibiting the movement of the catheter within the anatomical structure. Yet another example is that the breathing state of the training subject is determined based on the respiratory coefficient F, and a prompt indicating whether the movement of the catheter within the anatomical structure is permitted is generated based on the breathing state. When the respiratory coefficient F is positive, the training subject's breathing state is determined to be inspiratory, generating a prompt that allows the catheter to move within the anatomical structure; conversely, when the respiratory coefficient F is negative, the training subject's breathing state is determined to be expiratory, generating a prompt that prohibits the catheter's movement within the anatomical structure. For example, when the anatomical structure is a bronchus, the visualization system assisting the physician can detect when the surgical instrument (e.g., a needle) at the catheter tip reaches the vicinity of the target tissue location and when the training subject is inspiratory. The system then prompts the physician that percutaneous puncture can be performed, as puncture is safer when the training subject is inspiratory. If the respiratory coefficient is negative, it indicates that the chest cavity is in the expiratory state (the second process state), and the chest cavity and bronchi are in a constricted state; the system then prompts the physician that the procedure should not be performed.

[0153] Then, based on the vital signs data of the anatomical structure of the training subjects, the process state of the first medical image representation is determined.

[0154] (3) Repeat the above steps until the number of first medical images representing multiple different process states meets the preset value.

[0155] The more training samples available, the better the training effect. Therefore, training can be performed only when there are a large number of medical images that meet the preset conditions. In this embodiment, the process state of the first medical image of the training object is determined, and then the number of first medical images in different process states is determined to meet the preset value. Training is then performed based on multiple first medical images to obtain a better recognition network.

[0156] Optionally, embodiments of this application can also acquire multiple first medical images for training by controlling the time or frequency of acquiring the first medical image, specifically including:

[0157] Acquiring medical images of different process states that meet preset values ​​aims to obtain medical images under different process states. For example, acquiring multiple medical images within a cycle at a certain frequency; if the frequency is high enough, multiple medical images under different process states can be obtained and used for training. For example, in some embodiments, at least for this type of anatomical structure, catheters and / or instruments can be manipulated to remain at feature points in the anatomical structure for a certain duration to continuously acquire their positions. In some embodiments, this duration may include at least one cycle of the intrinsic movement of the anatomical structure. For the bronchus, it may include at least one respiratory cycle, such as one, two, or more; a normal respiratory cycle typically lasts 3 to 5 seconds. As another example, for (cardiovascular) vessels, the duration may include at least one heartbeat cycle, such as one, two, or more; a normal heartbeat cycle typically lasts 0.5 to 1 second. The longer the duration of acquiring medical images or the more cycles included, the more continuous sampling within such a duration, and the greater the number of sampling points, thus avoiding individual differences caused by different cycles.

[0158] Optionally, for example, regarding the respiratory motion of the bronchi, the magnitude of the deformation caused by respiratory motion varies in different process states, that is, the magnitude of the deformation caused by motion differs at different time points within the cycle. For example, taking the bronchus as an example in the respiratory cycle, when the first process state is the patient's full inspiration state, when taking the first medical CT image during the inspiration phase, the patient needs to take a deep breath and hold it until the CT image is completed. Since the bronchi are largest during inspiration, the fine branches at the ends of the bronchi are easier to image on the CT. Therefore, selecting a time period within the cycle with less deformation influence to acquire medical images for training can yield a recognition network with better recognition performance. That is, acquiring medical images for network training within a preset time period, such as a time period with less deformation influence. The time period with less deformation influence can be a time period of the relaxation process state, such as a time period within the inspiration state. Here, the time period with less deformation influence can also be called the process state with less deformation influence.

[0159] Furthermore, medical images can be acquired at non-fixed frequencies. Medical images can be acquired at a higher frequency during the aforementioned preset time period and at a lower frequency during non-preset time periods. For example, medical images can be acquired at a higher frequency during the aforementioned stretching process and at a lower frequency during the contraction process. This ensures that the first medical image includes medical images from both the contraction and stretching processes, with a larger number of medical images from the stretching process. This allows for the preparation of multiple representative first medical images that are less affected by deformation for training, thereby enhancing the recognition capabilities of the trained recognition network.

[0160] The navigation method provided in this application embodiment, after obtaining the identified network, further includes:

[0161] Step S30: In response to the user's navigation command, the first duct segment is identified from the patient's intraoperative medical images acquired by the imaging device using the recognition network.

[0162] Intraoperative medical imaging refers to images acquired by imaging devices in scenarios requiring navigation, including examinations or surgical procedures such as bronchoscopic biopsies or cancer resections. Imaging devices can be medical endoscopes, such as the camera inside a bronchoscope.

[0163] The recognition network extracts features from intraoperative medical images that distinguish different ductal segments, such as lung segments, especially the features of the lung segment openings, including geometric shape, texture, and color. Based on the lung segment opening features and the feature set in the recognition network, one or more lung segments in the intraoperative medical image are identified and output, and the identified lung segment is referred to as the first lung segment. Figure 4 The diagram shown is a schematic representation of the first lung segment in a navigation method according to an embodiment of this application. The left main bronchus 201 and the right main bronchus 301 are both identified as the first lung segments. The number of the first lung segments may be 0, 1, or multiple. The feature set refers to multiple feature parameters of the recognition network, which can be determined through network training.

[0164] Optionally, after identifying and outputting one or more of the first duct segments in the intraoperative medical image in step S20, the procedure may further include:

[0165] The system acquires identifiers for one or more first ductal segments and multiple second ductal segments. For example, when a ductal segment is a lung segment, the identifier is used to distinguish each lung segment. The identifier includes the lung segment's location, name, and / or identification number. The lung segment location indicates the segment's spatial position within the lung, and the identification number indicates the segment's sequence number among all lung segments. After identifying the first ductal segments in the intraoperative medical image, the recognition network outputs the identifier corresponding to the first ductal segment.

[0166] This application embodiment identifies the first duct segment and obtains its corresponding identifier, thereby uniquely determining the first duct segment in the intraoperative medical image.

[0167] Step S40: Obtain the planned path from the initial position to the target position. The planned path consists of multiple connected second pipeline segments. Taking a lung segment as an example, the specific steps include:

[0168] The first step: Obtain preoperative images of the patient's anatomical structures.

[0169] The second step: Based on the depth-first or breadth-first algorithm, determine the patient's duct segments according to the preoperative images of the patient's anatomical structure.

[0170] Meanwhile, in order to identify each different pipeline segment, an identifier is recorded for each pipeline segment. The identifier may include the pipeline segment location, pipeline segment name, etc.

[0171] Taking lung segments as an example, a depth-first search algorithm or a breadth-first search algorithm is used to traverse the identifiers of each lung segment, thereby assigning each lung segment a unique identification number. In this embodiment, this identification number is referred to as AirwayID. The identification number can also serve as a marker. The identification number corresponds one-to-one with the lung segment name and lung segment location, both used to uniquely identify each lung segment. The one-to-one correspondence between the identification number and the lung segment name and location can be stored in any form, as long as it allows retrieval of the others when any one of them is known during subsequent use.

[0172] Step 3: Based on the initial position and the target position, determine the second tubing segment included in the planned path from the patient's tubing segments. Specifically, based on all the patient's tubing segments and the target position, determine the second tubing segment, which is used to form the planned path. The initial position can be understood as the starting point, or the starting position for navigation.

[0173] The target location can be the location of the lesion or a location defined by the doctor and related to the medical procedure. Given the patient's tubing segments, initial location, and target location, a planned path to the target location can be planned. In this embodiment, the tubing segments included in the planned path are referred to as the second tubing segments. Multiple second tubing segments are interconnected. This embodiment does not specifically limit the method for planning the path; it can employ shortest distance planning or shortest time planning methods, etc. The planned path can be represented by lung segment identifiers, optionally using identification numbers, lung segment names, etc. For example, the planned path is 3-3 to 3-7 to 4-3 to 5-5 to 6-4, where 3-3, 3-7, 4-3, 5-5, and 6-4 are the identification numbers of the second lung segments. Figure 5 The diagram shows a planned path in a navigation method according to an embodiment of this application. The lung segments in the planned path are labeled as follows: right main bronchus 501 to right intermediate bronchus 502 to right lower lobar bronchus 503 to right lower lobar lateral segment 504 (B9) to right lower lobar lateral segment subsegment 505 (B9b). The lung segments are connected, but not completely connected in the diagram for easy distinction.

[0174] Path planning is typically done preoperatively, but if the system's computing power is sufficient, it can also be done intraoperatively. For example, at the beginning of surgery, because the duct network is relatively thick, the user does not need navigation. As the duct network gradually thins, the user needs navigation, and the system can perform path planning based on the current real-time location and the target location. The real-time location at this point serves as the initial location. Alternatively, if the user needs to change the target location during surgery, path planning can be performed based on the initial location and the updated target location.

[0175] Step S50: Based on the first pipeline segment, match the target pipeline segment from the second pipeline segment;

[0176] Target tube segments are used to represent tube segments jointly included in intraoperative medical images and planned pathways. For example, a target tube segment can be characterized by the intersection of segments identified from the first tube segment in the intraoperative medical image and the second tube segment in the planned pathway.

[0177] Optionally, since the identifier can uniquely identify a lung segment, the processor is also configured to match one or more target lung segments based on the identifiers of one or more of the first lung segments with the identifiers of a plurality of second lung segments.

[0178] If the first duct segment can be identified in step S30, but the target duct segment cannot be matched in step S50, there may be two reasons for this: the identification network identifies and outputs an incorrect first duct segment, or it has reached the wrong position. To rule out the possibility of an incorrect first duct segment, taking a lung segment as an example, this embodiment of the application further includes:

[0179] Verify the correctness of the first lung segment based on the tree structure of the bronchial tree.

[0180] Because the bronchial tree has a tree-like structure, it branches progressively, becoming increasingly finer, resembling a tree branch. This can be understood as the finer branches being child nodes of thicker branches, and the thicker branches being parent nodes of finer branches. The correct navigation path follows the parent node to the child node, while the reverse is incorrect. Therefore, by identifying whether the first lung segment conforms to the parent-child node relationship, we can determine whether the first lung segment is correct. For example, if the current frame of the intraoperative medical image identifies lung segments 5-6, while the previous frame identified lung segments 6-2, this clearly does not conform to the tree structure; therefore, the first lung segment is incorrect.

[0181] Optionally, the verification of whether the first lung segment is correct can be performed after the first lung segment is identified in step S20, or after the target lung segment cannot be matched in step S40.

[0182] This application embodiment verifies whether the first lung segment is correct based on the tree structure of the bronchial tree, thereby eliminating the problem of incorrect identification of the first lung segment and making it more conducive to successfully completing path navigation.

[0183] Step S60: Generate navigation markers to label the target duct segment. Navigation markers can be generated from intraoperative medical images or from a patient's anatomical model. The navigation markers are used to label the target duct segment to distinguish it from other duct segments in the medical image, thereby enabling navigation based on the target duct segment. Navigation markers can include any one or a combination of graphics, curves, highlights, colors, and text, as long as they can be distinguished from other duct segments in the medical image. For example... Figure 6 The diagram shown is a schematic of the target lung segment, where the right main bronchus 301 is the lung segment jointly included by the first lung segment and the second lung segment, and the block diagram is a navigation marker.

[0184] Step S70: Display the navigation markers and the target tubing segment on the user interface to guide the movement of the catheter and / or instrument.

[0185] When no target tubular segment is present, it indicates that the intraoperative medical image does not contain a target for the next step of motion, and the catheter and / or instrument can continue. When a target tubular segment is included, that segment becomes the target for the next step of motion, guiding the catheter and / or instrument towards it. When multiple target tubular segments are included, one segment can be selected as the target for the next step of motion according to a selection strategy, guiding the catheter and / or instrument towards the selected segment. It is understood that the target tubular segments identified and selected based on each intraoperative medical image are all part of the navigation path, collectively forming the entire navigation path. The selection strategy may include selecting target tubular segments with a high probability of correct identification, or selecting target tubular segments with earlier identification numbers.

[0186] This application embodiment identifies a first tubing segment from intraoperative medical images of the patient acquired by the imaging device, and matches a target tubing segment from a plurality of second tubing segments included in the planned path based on the first tubing segment. Then, a navigation path is formed according to the target tubing segment to guide the catheter and / or instrument to move toward the target tubing segment, thereby achieving fast and accurate navigation.

[0187] In one embodiment, the navigation system provided in this application is further configured to pre-train a recognition network, mainly including two cases:

[0188] (a) Training based on preoperative images

[0189] (ii) Training based on intraoperative medical images

[0190] Let's take the lungs as an example.

[0191] In the first scenario, preoperative lung images of the training subject can be annotated to obtain annotation information. In this embodiment, the subject undergoing medical procedures is referred to as the patient; for example, the subject of surgery or bronchoscopy can be a patient. Medical procedures may include the aforementioned examinations or surgical treatments. Optionally, preoperative images are typically taken before performing medical procedures on the patient, and these images can be used for training. Optionally, in this embodiment, the training subject may be the patient or someone other than the patient; for example, the patient undergoing medical procedures is A, and the training subject may be A, B, C, and / or D. Preoperative images may include computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound, and can be presented as two-dimensional, three-dimensional, or four-dimensional (e.g., time-based or rate-based information) images. A bronchial tree model can be established based on the preoperative images, and slice images based on this bronchial tree model can reflect the actual basic geometric structure and texture features at each corresponding real location. Slice images can be understood as images obtained by observing a bronchial tree model from various viewing positions and angles using a virtual imaging device, such as a virtual endoscope. The number of slice images is sufficient for network training; a portion of the slice images can be selected as the training set, and another portion as the test set.

[0192] Of course, sagittal, coronal, or horizontal images based on this bronchial tree model can also be used. These images can also reflect the actual basic geometric structure and texture features at the corresponding real locations.

[0193] Annotation of preoperative images can be performed through automatic identification or manual selection of lung segments. Since navigation is typically required before entering a lung segment, and once a segment is entered, the process continues, the features of the lung segment openings are particularly valuable for navigation. A lung segment opening can be understood as the visible portion of the lung segment before entry. For example, annotation can be performed automatically using a pre-trained recognition network or manually by identifying lung segment openings. The purpose of annotation is to create learning samples for network training, thus ensuring successful network training. By automatically or manually identifying features in preoperative images, lung segments are identified and annotated, establishing a correlation between preoperative image features and lung segments. To characterize this correlation, annotation information can be recorded, including lung segment location, location information, bounding rectangle, lung segment name, and / or identification number. The annotation information can include the same information as the aforementioned identifiers, such as lung segment name, identification number, or lung segment location. Location information refers to the position of the lung segment in the preoperative image. Optionally, a specific location within a lung segment can be chosen to represent its position, such as the segmental orifice. The segmental orifice is typically circular or elliptical. The circumscribed rectangle of the orifice, its length and width, can be determined first, and then a point within that rectangle can be selected as the location of the lung segment, such as the upper left or upper right corner. For easier observation, annotation information can also be displayed in the preoperative images, such as... Figure 7 The image shown is a labeled preoperative image, which includes the circumscribed rectangle of the lung segment opening and its identification number. The labeled preoperative image can be displayed on a display device, which can be mounted on the surgical robot or elsewhere.

[0194] Then, the recognition network is trained based on preoperative images and annotation information. During the model training phase, training images and annotation information are input into the recognition network, which extracts features of each lung segment, especially the features of the lung segment openings, through convolutional operations. It can be understood that network training is a gradual process. Based on the initial network architecture, the initial network is trained using preoperative images and annotation information, and continuously improved, ultimately resulting in a recognition network that can achieve the expected recognition effect.

[0195] Optionally, to increase the number of learning samples and improve the effectiveness of network training, the preoperative images in this embodiment are enhanced. The enhancement process may include rotation, cropping, translation, or scaling. Since the original preoperative image has already been labeled, the enhanced preoperative image also has corresponding labeling information and can therefore be used as a learning sample for network training. The recognition network can be obtained simply by training based on the enhanced preoperative image and the labeling information. Alternatively, the recognition network can be obtained by training based on the medical image, the enhanced medical image, and the labeling information. Furthermore, the enhancement process for the preoperative image may include:

[0196] Determine the first scaling ratio;

[0197] Based on the first scaling ratio, the first medical image is scaled to obtain a scaled first medical image.

[0198] Determining the first scaling ratio may specifically include:

[0199] A second medical image of one of the training objects in a stretching state is acquired, and a first anatomical model of the training object is generated based on the second medical image; a third medical image of one of the training objects in a contraction state is acquired, and a second anatomical model of the training object is generated based on the third medical image. The purpose of the anatomical model here is to determine the scaling ratio; therefore, the first medical image of the training object used when training the recognition network can be used. For example, a medical image of a stretching state can be selected from the first medical images to build the first anatomical model, and a medical image of a contraction state can be selected from the first medical images to build the second anatomical model; other medical images can also be acquired, and this embodiment does not limit this. One stretching state can be selected as the maximum stretching state, and one contraction state can be selected as the minimum contraction state.

[0200] Determining the maximum shrinkage ratio (ShrinkMax) of the first and second anatomical models specifically includes:

[0201] S131, obtain the first geodesic distance of the first anatomical model and the second geodesic distance of the second anatomical model.

[0202] In this embodiment, when the anatomical structure is a bronchus (usually referring to the pulmonary bronchus), the first geodesic distance from the main carina to the lower lobe terminal point in the first anatomical model and the second geodesic distance from the main carina to the lower lobe terminal point in the second anatomical model are calculated respectively. The main carina refers to the bifurcation point of the left and right bronchi, and the lower lobe terminal point refers to the point in the bronchial model closest to the diaphragm. In some embodiments, when the first anatomical model is a three-dimensional mesh model, the geodesic distance is the shortest path distance along the mesh surface from the main carina to the lower lobe terminal point in the first anatomical model; in another embodiment, when the first anatomical model is a three-dimensional point cloud model, a mesh-like surface structure needs to be constructed using all points in the three-dimensional point cloud, and the geodesic distance is calculated by finding the shortest path from the main carina to the lower lobe terminal point in the graph. The first or second anatomical model contains the target tissue, such as... Figure 7 As shown, in the first anatomical model, the geodesic line 10 formed from the main carina to the end point of the lower lobe is a line on the grid surface. The length of the geodesic line 10 is the geodesic distance. The target tissue 20 (e.g., lesion) is located exactly at the end point of the lower lobe.

[0203] S132, determine the maximum shrinkage ratio of the first anatomical model or the second anatomical model based on the first geodesic distance and the second geodesic distance.

[0204] In some embodiments, for example, the first geodesic distance GDIn of the first anatomical model obtained preoperatively is 166 mm; the second geodesic distance GDEx of the second anatomical model obtained preoperatively is 127 mm; then the maximum shrinkage ratio of the first or second anatomical model is determined according to the ratio of the first geodesic distance to the second geodesic distance: ShrinkMax = GDEx / GDIn, where ShrinkMax is approximately 0.76.

[0205] Then, the first scaling ratio is determined based on the maximum shrinkage ratio ShrinkMax, where the first scaling ratio = K1 * ShrinkMax, and K1 is a coefficient, taken as a natural number. For example, if ShrinkMax is 0.76 and k1 is 1.1, the first scaling ratio = 0.76 * 1.1 = 0.83. The specific value of K1 can be flexibly determined based on experience. Optionally, multiple different coefficients K1 can be determined to obtain multiple first scaling ratios. Multiple different first scaling ratios can be used to scale the same first medical image, thereby obtaining more training samples.

[0206] This application embodiment improves the recognition ability of the recognition network by scaling the first medical image of the training object using a scaling ratio, and then training the network based on the plurality of first medical images and the scaled first medical image.

[0207] Optionally, the navigation method provided in this application embodiment further includes:

[0208] The preoperative images of the training subjects are input into the feature transfer network to obtain the style-transferred preoperative images.

[0209] The style-transferred preoperative images were annotated to obtain annotation information;

[0210] The recognition network is trained based on the style-transferred preoperative images and annotation information.

[0211] The feature transfer network is used to perform style transfer on preoperative images, that is, to render preoperative images in the style of intraoperative medical images, which is the process of transferring the style of the virtual endoscopic image domain to the real endoscopic image domain. This includes two stages: style transfer and geometric smoothing. In the style transfer stage, geometric information needs to be preserved to the greatest extent possible. The basic principles in the geometric smoothing stage include ensuring that pixels with the same geometric shape have the same image style and minimizing the global geometric feature offset of the generated image. Style differences between preoperative images and intraoperative medical images can include color, local brightness, or contrast. The feature transfer convolutional network renders the preoperative image according to the style effect of the image device, minimizing changes to the geometric structure and texture of the preoperative image. The feature transfer network can use a GAN (Generative Adversarial Net) network, which includes a generator function and a discriminator function. The feature transfer network can use existing feature transfer networks; this application does not impose specific limitations. For example, the preoperative image may be black and white, and the style-transferred preoperative image may be color, or the preoperative image may be color, and the style-transferred preoperative image may be black and white.

[0212] The principle of annotating preoperative images after style transfer is the same as that of annotating preoperative images by automatic recognition or manual selection of lung segments, as mentioned above. For the sake of brevity, it will not be repeated here.

[0213] In this embodiment, the pre-trained recognition network is obtained by using the preoperative image of the training subject as input feature transfer network for style transfer, followed by annotation, and then training based on the style-transferred image.

[0214] Optionally, the navigation method provided in this application embodiment further includes:

[0215] The preoperative images of the training subjects were labeled to obtain the labeling information;

[0216] The preoperative image is input into the feature transfer network to obtain the style-transferred preoperative image;

[0217] The recognition network is trained based on the style-transferred preoperative images and annotation information.

[0218] In this embodiment, the pre-operative images of the training subjects are labeled and then input into a feature transfer network for style transfer. The network is then trained based on the style-transferred pre-operative images to obtain a pre-trained recognition network.

[0219] In the second scenario, one option is to first acquire intraoperative medical images of the training subjects, annotate these images, and obtain annotation information. The training subjects include individuals other than the patients themselves. In medical practice, intraoperative medical images of the training subjects can be obtained in certain scenarios, such as during surgery. Surgeons can acquire and accumulate these images for subsequent network training. In this case, because the input is intraoperative medical images, they are closer to the intraoperative medical images to be recognized in actual applications, resulting in high recognition efficiency. However, this requires collecting a large number of intraoperative medical images for network training to improve the network's generalization ability. Therefore, relying solely on acquiring intraoperative medical images of the patients themselves for network training is insufficient to quickly meet the requirements, and given the time constraints of surgery, there is usually not enough time for sufficient network training. The annotation of intraoperative medical images can be performed using methods similar to those described above, such as automatic recognition or manual selection of lung segments from preoperative images. The principle is the same, and for the sake of brevity, it will not be elaborated further here.

[0220] Then, the recognition network is trained based on intraoperative medical images and annotation information. During the model training phase, the intraoperative medical images and annotation information are input into the recognition network, which extracts features of each duct segment, such as the features of the lung segment opening, through convolutional operations to complete the lung segment identification.

[0221] This application embodiment uses intraoperative medical images of the training subjects and corresponding annotation information to train the network, thereby obtaining a pre-trained recognition network.

[0222] In one embodiment, the navigation method provided in this application further includes: in response to a user's navigation command, identifying a first duct segment from intraoperative medical images of the patient acquired by the imaging device using the recognition network, including:

[0223] The principle for determining the maximum contraction ratio is the same as that for determining the ShrinkMax maximum contraction ratio, and will not be repeated here for the sake of brevity. Optionally, the maximum contraction ratio can be determined based on the patient's own anatomical model or that of someone other than the patient.

[0224] The second scaling ratio is determined based on the maximum shrinkage ratio. The specific principle is the same as that for determining the first scaling ratio, and will not be repeated here for the sake of brevity.

[0225] The intraoperative medical image is scaled based on the second scaling ratio; optionally, multiple different second scaling ratios may be included, and then the same intraoperative image is scaled using multiple different scaling ratios to obtain intraoperative images with different scaling degrees.

[0226] The recognition network is used to identify the first duct segment from the scaled intraoperative medical images. When multiple scaled intraoperative images are included, the duct segment can be identified from each scaled intraoperative medical image first, and then the recognition results of each scaled intraoperative medical image can be combined. The combination strategy can be flexibly determined, for example, merging the recognition results of each scaled intraoperative medical image.

[0227] This application embodiment determines the second contraction ratio by utilizing the maximum contraction ratio, thereby scaling the intraoperative image. The recognition network is then used to identify the first duct segment from the scaled intraoperative medical image for navigation, thereby improving the recognition accuracy of the recognition network.

[0228] Optionally, determining the second scaling ratio based on the maximum shrinkage ratio may further include:

[0229] At the same time, the intraoperative medical images and the patient's body surface data sensed by the body surface sensor are acquired;

[0230] The vital signs data of the patient's anatomical structures in the current state of intraoperative medical imaging characterization are determined, and the vital signs data characterize the patient's anatomical structures in different procedural states;

[0231] The second scaling factor is determined based on vital sign data and the maximum contraction ratio.

[0232] Determining the patient's vital signs and anatomical structures in the current state of intraoperative medical imaging representation can be achieved through the following steps:

[0233] (1) The patient's body surface is equipped with a body surface sensor. The location of the body surface sensor is the same as the principle of the body surface sensor of the training subject mentioned above, and will not be repeated here.

[0234] (2) Based on the surface data, determine the vital signs data of the patient's anatomical structure. The vital signs data characterize the patient's anatomical structure under different process states. The vital signs data include the area, slope, and motion coefficient of the shape enclosed by the surface sensors. For example, determine the area of ​​the shape enclosed by the surface sensors based on the patient's surface data. The area includes a first area, a second area, and a third area. The first area includes the area of ​​the shape enclosed by the surface sensors in a patient's expansion state. The second area includes the area of ​​the shape enclosed by the surface sensors in a patient's contraction state. The third area is the area S of the shape enclosed by the surface sensors in the patient's current state. For example, for the lungs, when a expansion state can be a fully inspiratory state, the second area is the maximum area S enclosed by the surface sensors. max When a contraction process is in a state of complete exhalation, the second area is the minimum area S enclosed by the body surface sensors. min .

[0235] like Figure 8a As shown, three surface sensors are used as an example to determine the motion coefficient of the patient in the current state. The three surface sensors of the patient are located at points P1(X1, Y1, Z1), P2(X2, Y2, Z2), and P1(X3, Y3, Z3), respectively. The distances between each pair of surface sensors are calculated, i.e., the side lengths A, B, and C.

[0236] Calculate the lengths of A, B, and C using Euclidean distance. For example, to calculate the side length A:

[0237]

[0238] The lengths of B and C are calculated in the same way as those of A, and will not be repeated here.

[0239] Calculate the semi-perimeter P of the shape enclosed by the patient's surface sensors:

[0240] P = (A + B + C) / 2

[0241] The area S of the shape enclosed by the surface sensors in the patient's current state:

[0242]

[0243] Based on the maximum area S max Minimum area S min The patient's motion coefficient F in the current state is determined by the area S of the shape enclosed by the surface sensors in the current state.

[0244]

[0245] Where Fmax represents the maximum value of the designed motion coefficient, and Fmin represents the minimum value of the designed motion coefficient. For example, in some embodiments, if the design formula is based on the motion coefficient within the range of [-1, 1], then the motion coefficient F is:

[0246]

[0247] For example, in some embodiments, if the design formula is based on the motion coefficient being within the range of [-100, 100], then the motion coefficient F is:

[0248]

[0249] like Figure 8b As shown, the respiratory coefficient is designed with the motion coefficient as the respiratory coefficient, ranging from [-1, 1]. The horizontal axis represents the number of EM data points acquired in real time by the surface sensors as the respiratory duration or respiratory cycle changes. The vertical axis, from top to bottom, represents: the area of ​​the shape enclosed by the surface sensors, the area slope, and the respiratory coefficient. The area of ​​the shape enclosed by the surface sensors represents the size of the area enclosed by the surface sensors as the patient's respiratory state changes; the area slope represents the rate of change of the area; and the respiratory coefficient characterizes the patient's different respiratory states.

[0250] Optionally, the second scaling factor Shrink is determined based on the motion coefficient F and the model's maximum shrinkage factor ShrinkMax, using the following formula:

[0251]

[0252] Optionally, after determining the second scaling ratio, it can be floated within a certain range, which may include multiple different scaling ratios. Then, the same intraoperative image can be scaled using multiple different scaling ratios to obtain intraoperative images with different degrees of scaling. Optionally, vital sign curves of the patient's anatomical structures can also be generated based on the vital sign data, and the vital sign curves, including respiratory curves and / or heart rate curves, can be displayed on the user interface.

[0253] This application embodiment utilizes body surface data sensed by the patient's body surface sensors to determine the motion coefficient, and then determines a second contraction ratio based on the maximum contraction ratio and the motion coefficient to scale the intraoperative image. The recognition network is then used to identify the first duct segment from the scaled intraoperative medical image to achieve navigation, thereby improving the recognition accuracy of the recognition network.

[0254] In one embodiment, the navigation method provided in this application includes a tracking sensor on the catheter and / or device for sensing the position of the catheter and / or device. The tracking sensor is used to acquire the actual path point of the catheter and / or device. In response to a user's navigation command, the method uses the recognition network to identify a first tubing segment from the patient's intraoperative medical images acquired by the imaging device, including:

[0255] (I) Correct the actual path points to obtain the corrected path points.

[0256] The data sensed by the tracking sensor corresponds to the actual path point location. By correcting the sensor data, the corrected path point corresponding to the corrected sensor data is obtained. There are two methods for correcting the actual path point, which will be described in detail later.

[0257] (II) Determine the positioning correction amount based on the actual path point and the corrected path point, that is, determine the positioning correction amount based on the sensor data and the corrected sensor data. The difference between the sensor data and the corrected sensor data reflects the influence of motion on the sensor, and therefore the influence of motion can be removed by eliminating this difference. This difference can be denoted as the positioning correction amount.

[0258] (III) Determine the relative positions of the tubular segments in the intraoperative medical image, wherein the relative positions may include the distance and direction between the various tubular segments. Determining the relative positions of the various tubular segments in the medical image can be achieved using existing techniques, and this application embodiment does not impose any limitations on this method.

[0259] (iv) Based on the first tube segment, the positioning correction amount, and the relative position of the tube segment in the intraoperative medical image, identify a third tube segment in the intraoperative medical image, wherein the third tube segment is different from the first tube segment.

[0260] When intraoperative medical images include multiple ductal segments, the probability of identification varies because the completeness of the displayed segments may differ. In this embodiment, the ductal segment that the identification network can directly identify from the intraoperative image is referred to as the first ductal segment.

[0261] By incorporating positioning corrections, intraoperative medical images are corrected. If, after correction, more unidentified duct segments are displayed in the intraoperative images, relative positions can be further considered. For example, the relative position can be compared with the pre-stored relative positions of duct segments to identify the closest relative position as the inferred duct segment, which is referred to as the third duct segment. If, after correction, fewer unidentified duct segments are displayed in the intraoperative images, these unidentified segments can be discarded. The third duct segment can differ from the first duct segment; this difference can include being completely different or partially different.

[0262] Intraoperative medical image correction, which incorporates positioning correction, can include adjusting the intraoperative medical image by the same or proportional amount as the positioning correction. For example, if the positioning correction is to move the image upwards by 5mm, then the intraoperative medical image can be moved upwards by 5mm or 4.5mm.

[0263] For example, such as Figure 9a The image shown is of the lung segments that should have been included in the intraoperative image when breathing was not a factor. These segments include lung segments 901, 902, and 903. Due to breathing, lung segment 903 is partially obscured due to displacement. Figure 9b As shown, the identification network can identify lung segments 901 and 902 as the first duct segment. The positioning correction amount can be determined based on the difference between the actual path point and the corrected path point. The relative positions of lung segments 901, 902, and 903 can be determined based on intraoperative images. Based on lung segments 901 and 902 identified by the identification network, the positioning correction amount, and the relative positions of lung segments 901, 902, and 903, lung segment 903 can be identified, and lung segment 903 is inferred as the third duct segment. Then, based on lung segments 901, 902, and 903, the target duct segment can be matched from the planned path, and navigation is then performed based on the target duct segment.

[0264] This application embodiment corrects the intraoperative image by adjusting the positioning correction amount before identification, thereby accurately identifying pipe segments in the intraoperative image that cannot be fully displayed due to deformation, thus improving navigation accuracy and efficiency.

[0265] In one embodiment, the catheter and / or device are provided with a sensor for measuring the position of the catheter and / or device. The sensor is used to obtain the actual path points of the catheter and / or device. The navigation method provided in this application embodiment further includes:

[0266] Acquire preoperative medical images of the patient and generate an anatomical model of the patient based on the preoperative medical images, wherein the preoperative medical images characterize the contraction or relaxation process of the patient's anatomical structures;

[0267] Obtain the transformation matrix, which is used to characterize the transformation relationship between the surgical environment coordinate system and the anatomical model coordinate system of the patient;

[0268] Based on the actual path points and the transformation matrix, the simulated path points of the catheter and / or instrument in the anatomical model corresponding to the actual path points are determined;

[0269] Mark the simulated path points in the anatomical model of the patient;

[0270] The user interface displays the anatomical model and the simulated path points.

[0271] The simulated path points reflect the position of the catheter / instrument tip in the anatomical model. Combined with the display parameters of the surgical site, the position of the catheter / instrument tip within the displayed surgical site can be obtained, thus fusing the displayed surgical site with the catheter and / or instruments to achieve intraoperative navigation. The surgical site can be displayed as preoperative images and / or intraoperative medical images. If preoperative image data is used to display the surgical site, since the anatomical model is obtained by processing three-dimensional preoperative images, the position of the catheter / instrument tip within the surgical site can be obtained based on the coordinate transformation relationship of the displayed surgical site relative to the three-dimensional preoperative image. If intraoperative medical image data is used to display the surgical site, the intraoperative medical image needs to be registered with the three-dimensional preoperative image / anatomical model to obtain the coordinate transformation relationship of the displayed surgical site relative to the three-dimensional preoperative image, and then the position of the catheter / instrument tip within the surgical site can be obtained. The coordinate transformation relationship can be represented by a transformation matrix, which can be determined using existing technology, or by referring to 202310026323.7 Catheter Robot and its Registration Method and Readable Storage Medium. This application embodiment does not impose any limitations on this.

[0272] Optionally, the actual path point is corrected to obtain a corrected path point, and based on the corrected path point and the transformation matrix, the simulated path point corresponding to the actual path point in the anatomical model of the catheter and / or instrument is determined; then the simulated path point is marked in the anatomical model of the patient; and the anatomical model and the simulated path point are displayed in the user interface.

[0273] This application embodiment determines the simulated path point corresponding to the actual path point / corrected path point based on the actual path point / corrected path point and coordinate transformation relationship, and displays the simulated path point on the user interface. It can be displayed in a stable form at a certain position of the anatomical model without fluctuating or drifting around, making it more convenient for users to observe the current position of the catheter and / or instrument, which is beneficial for navigation to the target position.

[0274] In one embodiment, the navigation method provided in this application includes a sensor on the catheter and / or device for measuring the position of the catheter and / or device. The sensor is used to acquire the actual path point of the catheter and / or device. The process of correcting the actual path point to obtain a corrected path point specifically includes:

[0275] In response to the user's motion compensation command, based on the actual path point, a feature point matching the actual path point is determined. This feature point is a point within the anatomical structure. Determining the feature point matching the actual path point may include identifying the feature point closest to the actual path point. Some types of anatomical structures, such as bronchi and (cardiac) vessels, have high-frequency and large-amplitude internal movements, such as respiratory and heartbeat movements, which can easily cause significant fluctuations in the position of catheters and / or instruments at feature points, leading to inconsistencies with preoperative medical images. For example, when the anatomical structure is a bronchus, the amplitude of the respiratory movement affecting the catheter and / or instrument varies in different lung regions. For instance, the fluctuation amplitude caused by respiratory movement in the upper lobe region may be 10 mm, while in the lower lobe region it may reach 20 mm. Therefore, the impact of the internal anatomical movement on the catheter and / or instrument typically varies depending on its position within the anatomical structure. Since feature points and subsets of sampling points are correlated, different denoising weight matrices can be constructed for different feature points or different subsets of sampling points. This allows for subsequent denoising of the subset using the corresponding denoising weight matrix, minimizing the impact of the inherent motion of the anatomical structure on the sampling points. The denoised subset can also be called the motion-compensated subset.

[0276] In this embodiment, each selected feature point can be considered to represent a region of an anatomical structure. Within this region, the internal movement of that region can be considered to have a substantially similar impact on the position of the catheter and / or instrument. That is, it is necessary to determine the feature point closest to each actual path point. For example, the Euclidean distance between each actual path point and the mean PosTube of the sampling points in the corresponding sampling point set can be compared to obtain the feature point closest to it.

[0277] Obtain the denoising weight matrix associated with the feature points. In some embodiments, a KD-tree can be constructed using the mean point PosTube of the sampling points in the subset corresponding to all feature points as nodes for subsequent retrieval. The KD-tree constructed here is a key-value search tree in three-dimensional Euclidean space, used for range search and nearest neighbor search. By inputting the real-time location of the catheter and / or device (i.e., the real-time actual path point), the feature point corresponding to the nearest node can be quickly found through the KD-tree, and the denoising weight matrix corresponding to that nearest feature point can be obtained. By constructing the KD-tree, the process of obtaining the denoising weight matrix is ​​greatly accelerated.

[0278] Based on the denoising weight matrix, the actual path points are denoised to obtain denoised path points. The denoised path points are then used as the corrected path points. In other words, the sensor data is denoised based on the denoising weight matrix to obtain denoised sensor data. The denoised sensor data is then used as the motion-compensated sensor data, and the motion-compensated sensor data corresponds to the corrected path points.

[0279] exist Figure 10 In the diagram, the point cloud composed of hollow circles represents the ductal movement path before denoising (i.e., before compensation), the point cloud composed of pentagram-shaped dots represents the ductal movement path after denoising, and the continuous path in the middle is a path pieced together from multiple bronchial centerlines. For example... Figure 10 As shown, the denoised point cloud has less offset (i.e., better convergence) relative to the bronchial centerline, meaning the influence of respiratory motion is significantly reduced. Optionally, the correction of the actual path points to obtain the corrected path points may further include:

[0280] Acquire the movement data of the catheter in the patient's anatomical network of channels, wherein the catheter movement data in the channels includes command data instructing the catheter tip / instrument tip to reach a specific anatomical site and / or change its orientation within the channel network, insertion data indicating the catheter tip / instrument tip, etc.

[0281] Based on the actual path point, the catheter's movement data within the tubing, and the intraoperative medical images, a corrected path point is determined, and the position data corresponding to the corrected path point is referred to as the fused sensor data. The insertion data of the catheter tip / instrument tip may include the catheter extension / retraction amount. The catheter extension / retraction amount reflects the depth of catheter insertion into the body and can be obtained from the robot's control commands. Optionally, dynamic weights are assigned to the sensor data, catheter control data, and intraoperative medical images. The position data of the corrected path point is calculated based on these weights and used as the fused sensor data. For example, the weight of the catheter control data can be higher at narrower bronchial points or bends.

[0282] This application embodiment obtains a corrected path point by correcting the actual path point. For example, it processes sensor data to obtain motion-compensated sensor data or fused sensor data. Then, based on the coordinate transformation relationship, it determines the simulated path point corresponding to the actual path point and displays the simulated path point on the user interface. This can improve the stability of the data obtained by the sensor. For example, the sensor data obtained after noise reduction filters out the jitter caused by the inherent movement of the anatomical structure. After being transformed into the anatomical model coordinate system, it can be displayed in a stable form at a certain position of the anatomical model without fluctuating and drifting. This makes it easier for users to observe the current position of the catheter and / or instrument, which is beneficial for navigation to the target position.

[0283] In one embodiment, the user interface scrolls through at least two of a virtual image device view, an anatomical model, a partial view, a slice view, and an aiming view, wherein the virtual image device view may include the navigation markers.

[0284] The user interface can use image data from imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound to present images of the surgical site recorded preoperatively or intraoperatively. Preoperative or intraoperative medical image data can be presented as two-dimensional, three-dimensional, or four-dimensional (e.g., time-based or rate-based information) images, and / or as images from models created based on preoperative or intraoperative medical image datasets. Preoperative images can be presented as slice views, anatomical model views, or local views. Local views are oblique views based on slice views. Slice views include axial, sagittal, and coronal views. Intraoperative medical images can be presented as intraoperative real-views, also known as intraoperative real-field views or camera views.

[0285] The user interface can also display a virtual navigation image. In the virtual navigation image, the actual position of the catheter assembly 400 is registered with the preoperative image to present a virtual image of the catheter assembly 400 within the surgical site to the operator from the outside. This can also be understood as a virtual representation of the endoscope tip within the anatomical model based on the sensor position.

[0286] Furthermore, in this embodiment of the application, determining the virtual image device view includes:

[0287] Pre-calibration of the endoscope, as detailed in 202310154245.9 Endoscope Registration Method, Apparatus and Calibration System, can yield the conversion matrix between the sensor and the endoscope lens, as well as the lens's intrinsic parameters.

[0288] The transformation matrix between the sensor coordinate system and the anatomical model coordinate system is obtained in advance through registration.

[0289] Based on the sensor's pose in the sensor's coordinate system, and combined with the transformation matrix between the sensor and the endoscope lens, the pose of the endoscope lens in the sensor's coordinate system is obtained. Then, combined with the transformation matrix between the sensor's coordinate system and the anatomical model's coordinate system, the pose of the endoscope lens in the anatomical model's coordinate system is obtained, i.e., the pose of the virtual endoscope lens in the anatomical model's coordinate system. Then, combined with the endoscope lens's intrinsic parameters, and using the principle of camera imaging, the contents of the anatomical model are imaged in the virtual lens to obtain a virtual field of view, which can also be called a virtual image device view.

[0290] The method for determining the virtual image device view is not limited to the method in the embodiments of this application, and can also adopt the prior art. The embodiments of this application do not limit it in any way.

[0291] Optionally, navigation markers are generated in the anatomical model and displayed in the virtual image device view.

[0292] The user interface can display an aiming view. The aiming view is used to display the target, such as a nodule, and can include a track view and a head view. The track view allows viewing the alignment around the nodule from different angles. The head view displays the target's position from the endoscope's perspective.

[0293] Depending on the user's preferences, the aforementioned intraoperative real-time view, virtual imaging device view, anatomical model view, partial view, slice view, and aiming view can be flexibly arranged in the user interface, for example, displayed in separate areas. Users can adjust the layout of specific areas themselves. Optionally, at least two of the above views can be displayed in a scrolling manner, allowing users to see different views in a timely manner and make timely adjustments to the medical procedure.

[0294] Optionally, in response to a user's command to display vital sign curves, the vital sign curves can also be displayed on the user interface. The vital sign curves characterize the scaling of the ductal network caused by internal energy flow; the vital sign curves are determined based on surface data sensed by surface sensors placed on the patient's body surface, and in conjunction with 8b, for example, the vital sign curves can reflect the area, slope, and respiratory coefficient of the graphic enclosed by the aforementioned surface sensors. By observing the vital sign curves, the user can intuitively understand the effects of exercise. Vital sign curves can be respiratory curves, heart rate curves, etc.

[0295] Optionally, depending on the magnitude of the impact, it can be determined whether to automatically trigger a motion compensation command to enter motion compensation mode. In motion compensation mode, the following can be performed: (I) before inputting the intraoperative medical image into the pre-trained recognition network, the intraoperative medical image is screened, and then the recognition network is used to identify the first tube segment from the screened intraoperative medical image; (II) the intraoperative medical image is scaled, and the recognition network is used to identify the first tube segment from the scaled intraoperative medical image; (III) the actual path point is corrected to obtain a corrected path point to remove the influence of breathing as much as possible; and (IV) the vital signs curve is displayed on the user interface. Optionally, at least one of (I), (II), (III), and (IV) can be performed. When scaling, it can be based on the aforementioned second scaling ratio.

[0296] Optionally, depending on whether the vital signs data are within the preset range, if the impact of breathing is considered significant and the exercise compensation mode needs to be entered, otherwise, if the impact of exercise is considered negligible and the exercise compensation mode does not need to be entered.

[0297] Optionally, the user may trigger a motion compensation command to enter motion compensation mode and execute at least one of (Ⅰ), (Ⅱ), (Ⅲ), and (ⅳ).

[0298] Optionally, after entering motion compensation mode, the specific one or combination of (Ⅰ), (Ⅱ), (Ⅲ), and (ⅳ) to be executed can be based on the soft default settings or on further user commands. For example, the user can select one or combination of (Ⅰ), (Ⅱ), (Ⅲ), and (ⅳ) in the user interface.

[0299] The application of the embodiments of this application is as follows: Figure 11 A schematic diagram of the user interface 800. The user interface 800 may display a camera view window 810, a virtual image device view 820, a global anatomical model view 830, an indicator window 840, and a control window 850.

[0300] The camera view window 810 displays camera data captured by the imaging device of the medical device. For example, the camera data may include camera image fields of view or camera video data captured at the end of a catheter by a stereo camera or single-field-of-view camera mounted on an endoscope.

[0301] One or more indicator windows 840 can display current operation prompts and screen recording buttons, as well as the status of the catheter, such as displaying the inner or outer bend angle of the catheter, or prompting the doctor to proceed with the surgical procedure or pause the surgical procedure.

[0302] The control window 850 can display the connection status of each device, such as the operating handle, conduit, magnetic navigation, suction, flushing, and locked or activated status.

[0303] In some embodiments, the virtual imaging device view 820 displays images from a local anatomical model corresponding to the catheter tip to simulate an endoscopic view. When the anatomical model is a bronchus, virtual bronchial model image data can be generated by a virtual visualization system using, for example, preoperative CT images. The virtual bronchial model image data can show the real-time position of the catheter within the patient's bronchus.

[0304] In some embodiments, the anatomical model view 830 shows the user observing the complete anatomical model from a global perspective; for example, the user can observe the position of the catheter in the bronchial model from the overall bronchial model.

[0305] In one embodiment, the denoising weight matrix obtained in the navigation method provided by this application can include two sources corresponding to the subset of sampling points.

[0306] The first type can originate from the patient themselves.

[0307] The second type can be derived from other patients with the same or similar physical characteristics as the patient in question. Similarity allows for a certain degree of deviation, for example, defined as an allowable deviation within 5%. Patients with the same or similar physical characteristics corresponding to the same anatomical features have essentially the same amplitude of internal anatomical movement, and thus have essentially the same impact on the placement of catheters and / or instruments. Physical characteristics include, for example, body shape and / or physiological characteristics. For anatomical structures that are bronchi or (cardiovascular) vessels, body shape characteristics include at least chest circumference; for anatomical structures that are bronchi, physiological characteristics include at least the amplitude of respiratory movements; for anatomical structures that are (cardiovascular) vessels, physiological characteristics include at least the amplitude of heartbeat movements.

[0308] In some embodiments, for the case where the denoising weight matrix originates from the second type, such as Figure 12 As shown, determining the denoising weight matrix may include:

[0309] Step S1201: Obtain the patient's physical characteristics.

[0310] Physical characteristics include, for example, body shape and / or physiological characteristics.

[0311] Step S1202: Determine the type of the patient's anatomical structure.

[0312] The types of anatomical structures can include, for example, bronchi or (cardiac) vessels, and of course, other organs. The type can be manually entered or automatically identified.

[0313] Step S1203: Based on the patient's physical characteristics and anatomical structure type, match target patients with the same anatomical structure type and the same or similar physical characteristics.

[0314] This involves constructing one or more databases storing information on multiple patients who have previously received treatment at this or other hospitals. This information includes, but is not limited to, the patient's name, gender, age, physical characteristics, physiological features of anatomical structures, and identical or different denoising weight matrices for different feature points of one or more anatomical structures. Therefore, it is advantageous to achieve precise matching through physical characteristics, physiological characteristics, and identical or different denoising weight matrices for different feature points of one or more anatomical structures.

[0315] Step S1204: Determine the feature points associated with the subset of patient sampling points.

[0316] Step S1205: Based on the feature points, match the denoising weight matrix of the target patient associated with the feature points.

[0317] In this context, the feature points of the same anatomical structures from different patients are usually identical or highly similar when selected. Taking the bronchus as an example, these feature points are usually selected from the most easily identifiable bifurcation points of the bronchus, i.e., the carina at each level. For example, when the feature points of the sampling point subset correspond to the main carina, the denoising weight matrix matched in step S1204 should also be the denoising weight matrix of the main carina of the target patient; when the feature points of the sampling point subset correspond to the first-level carina of the left lobe, the denoising weight matrix matched in step S1204 should also be the denoising weight matrix of the first-level carina of the left lobe of the target patient.

[0318] Feature points can be selected from anatomical structures that are distinctive and easily identifiable. For example, when the anatomical structure is a bronchus (usually referring to the bronchus of the lungs), feature points can be selected from carinae of various grades. For instance, the main carina, the first-grade carina of the left lobe, and the second-grade carina of the right lobe can be selected.

[0319] When selecting feature points, it is advisable to cover as much of the anatomical structure as possible. For example, when the anatomical structure is the bronchus, multiple feature points can be selected to cover as much of the bronchus as possible. For instance, one or more feature points can be selected from one or more regions among the main carina, right upper lobe, right middle lobe, right lower lobe, left upper lobe, left middle lobe, and left lower lobe. For example, one feature point can be selected from each of the above regions to cover the entire bronchus, such as... Figure 13 As shown, in the anatomical model, feature point 1 representing the main carina, feature point 2 representing the carina of the left upper lobe, feature point 3 representing the carina of the left middle lobe, feature point 4 representing the carina of the left lower lobe, feature point 5 representing the carina of the right upper lobe, feature point 6 representing the carina of the right middle lobe, and feature point 7 representing the carina of the right lower lobe were selected, for a total of seven feature points.

[0320] Each subset of sampling points corresponds to a feature point of an anatomical structure (matches it), and different subsets of sampling points can correspond to different feature points of the anatomical structure. The sampling point subsets and simulated feature points are associated with each other through these feature points. Each subset of sampling points may include at least one sampling point, which is generally described by the coordinates of the catheter or instrument tip in a world coordinate system (surgical environment coordinate system or physical space coordinate system). Sampling points can be acquired using tracking sensors, which may include position sensors and / or shape sensors.

[0321] For example, a visualization system can guide the distal end of a catheter inserted into an anatomical structure and / or the distal end of an instrument to a feature point, and a position sensor located on the catheter and / or instrument can be used to obtain the position of the catheter and / or instrument. One of the position sensors can be a component of an electromagnetic positioning system. The electromagnetic positioning system can further include a magnetic field generating component and a magnetic field detection component. The magnetic field generating component generates a magnetic field, and the position sensor is an EM sensor (i.e., an electromagnetic sensor). The position sensor induces a change in the magnetic field in the magnetic field, and the magnetic field detection component can detect this change, thereby detecting the pose of the position sensor relative to the magnetic field / magnetic field generating component. Combining this with a pre-set or calibrated coordinate transformation relationship between the position sensor and the catheter and / or instrument, the position of the catheter and / or instrument relative to the magnetic field / magnetic field generating component can be calculated. Furthermore, combining this with the coordinate transformation relationship between the magnetic field / magnetic field generating component and the world coordinate system, the position of the catheter and / or instrument in the world coordinate system, i.e., the sampling point, can be calculated. This position sensor can sense at least three translational degrees of freedom within the magnetic field.

[0322] For example, a shape sensor system can be used to acquire the position of catheters and / or instruments inserted into anatomical structures. For instance, the shape sensor may include an optical fiber aligned with the catheter, and the fiber optic bending sensor formed by the fiber can provide feedback on the shape of the catheter. Based on this, the position of the catheter and / or instrument relative to the base of the shape sensor can be calculated. Combined with the position of the shape sensor base in the world coordinate system, the position of the catheter and / or instrument in the world coordinate system, i.e., the sampling point, can be calculated.

[0323] In some embodiments, for the case where the denoising weight matrix originates from the second type, such as Figure 14 As shown, the denoising weight matrix also includes:

[0324] Step S1201': Obtain the patient's physical characteristics.

[0325] Step S1203': Based on the patient's physical characteristics, match target patients with the same or similar physical characteristics.

[0326] Step S1204': Determine the feature points associated with the subset of the patient's sampling points.

[0327] Step S1205': Based on the feature point, match the denoising weight matrix of the target patient associated with the feature point.

[0328] Through the steps S1201–S1205 described above, it is unnecessary to reconstruct the denoising weight matrix for different feature points of the anatomical structure for the patient. Instead, by using the existing denoising weight matrix for the corresponding feature points of the anatomical structure of other patients with similar or identical physical characteristics, the preoperative preparation time can be significantly reduced. This time includes at least the time for setting up sensors to detect the internal movement of the anatomical structure. Of course, in scenarios with ample time, individual differences between patients can also be taken into account, and a denoising weight matrix for different feature points of the anatomical structure can be constructed separately for each patient.

[0329] Optional, such as Figure 15 As shown, for the training object, whether based on the patient itself or other patients, for each of the multiple feature points, the method for initially constructing the denoising weight matrix for each feature point can include:

[0330] Step S1211: Obtain a subset of sampling points corresponding to a feature point from the anatomical structure.

[0331] The subset of sampling points includes multiple sampling points, which are obtained by the same tracking sensor at different times. In this embodiment, the multiple sampling points typically include tens, hundreds, or even more sampling points.

[0332] Step S1212: Obtain a set of test points from the body surface corresponding to the anatomical structure.

[0333] Each test point set comprises multiple test point subsets, and each test point subset includes at least one test point. Test points are obtained by sampling from surface sensors deployed on the patient's body surface corresponding to the anatomical structures. Test points are generally described using the coordinates of the surface sensors in the surgical environment coordinate system. A single test point corresponding to a different test point subset is typically obtained by sampling from a corresponding surface sensor at different times.

[0334] The number of surface sensors is configured in the same way according to the expected number of test points in each test point subset. For example, if the expected number of test points in each test point subset is one, the number of surface sensors is one; or if the expected number of test points in each test point subset is two or three, the number of surface sensors is two or three respectively.

[0335] In this embodiment, multiple test point subsets typically include test point sets of ten, one hundred, or even more.

[0336] In some embodiments, the number of sample points in the sample point subset usually needs to be the same as the number of test points in the test point subset to facilitate subsequent calculations. However, this does not necessarily require obtaining the same number through sampling; the same number can also be maintained through data processing methods such as interpolation or filtering.

[0337] In some embodiments, the same feature point can be sampled separately to obtain a subset of sampling points and a set of test points respectively; however, from the perspective of saving preoperative time, sampling can also be performed simultaneously and at the same frequency to obtain a subset of sampling points and a set of test points.

[0338] In some embodiments, when the anatomical structure is a bronchus or (cardiac)vascular vessel, at least one surface sensor is positioned on the patient's chest. Exemplarily, the surface sensor may include three: the first surface sensor may be positioned in the middle of the patient's chest, the second surface sensor may be positioned in the area corresponding to the seventh rib on the left side of the patient's chest, and the third surface sensor may be positioned in the area corresponding to the seventh rib on the right side of the patient's chest. The more surface sensors there are, the more area of ​​the patient's body surface is covered, and the more accurate the sensing of the internal movement of the anatomical structure.

[0339] Step S1213: Based on the test point set and the sampling point subset in the same surgical environment coordinate system, determine the denoising weight matrix corresponding to the sampling point subset, that is, determine the denoising weight matrix corresponding to the feature point.

[0340] In some embodiments, the surface sensor and the tracking sensor are both EM sensors, which may belong to the same electromagnetic positioning system or different electromagnetic positioning systems. When they belong to the same electromagnetic positioning system, the positions sensed by both are in the same surgical environment coordinate system; when they belong to different electromagnetic positioning systems, the positions sensed by both can be transformed to the same surgical environment coordinate system through coordinate transformation. In this case, the surgical environment coordinate system can also be called the electromagnetic coordinate system or the sensor coordinate system.

[0341] In some embodiments, step S1213 described above can be implemented more specifically by the following steps:

[0342] For ease of description, the location data (i.e., test points) of the test point set acquired by the body surface sensor is defined as body surface sensor data Ref, and the location data (i.e., sampling points) of the sampling point subset acquired by the tracking sensor is defined as in vivo catheter data Tube.

[0343] (1) Perform mean-reduction processing on the surface sensor data Ref to obtain the surface sensor data. The in vivo catheter data (Tube) was then processed to remove the mean, thus obtaining the in vivo catheter data.

[0344] (2) Acquiring data from body surface sensors The inverse solution. For example, this inverse solution can be obtained using the Singular Value Decomposition (SVD) method. The obtained inverse solution can typically be expressed as:

[0345]

[0346] in, This is the inverse solution of the surface sensor data, where V is the right singular matrix, S is the singular value matrix, and U is the left singular matrix.

[0347] (3) Inverse solution based on body surface sensor data and in-body catheter data Determine the denoising weight matrix for each feature point. The obtained denoising weight matrix associated with each feature point can be expressed as:

[0348]

[0349] Where Weight is the denoising weight matrix.

[0350] In some embodiments, taking the bronchus as an example, three surface sensors can be used to acquire a set of test points over two or more respiratory cycles (typically 6-10 seconds), and a subset of sampling points associated with feature points can be acquired simultaneously by a tracking sensor. The surface sensor data (Ref) is an N x 9 matrix, where each row can be formatted as {X1, Y1, Z1, X2, Y2, Z2, X3, Y3, Z3}, and each set of three columns corresponds to the position data sensed by one surface sensor. The internal catheter data (Tube) is an N x 3 matrix, where each row can be formatted as {X0, Y0, Z0}, and these three columns correspond to the position data of the catheter and / or device sensed by the tracking sensor.

[0351] Furthermore, each feature point corresponds to a defined denoising weight matrix. For example, this can be stored using the data structure ST = {PosTube, PosRef, Weight}. Here, PosTube represents the mean of the sampled points in the sampling point set corresponding to each feature point. For instance, when using an electromagnetic positioning system, this mean is the mean of the electromagnetic coordinate points of the catheter and / or instrument, in the form of... PosRef is the mean of the test points in the test point set corresponding to each feature point. For example, when using an electromagnetic positioning system, this mean is the mean of the electromagnetic coordinate points of the three surface sensors, in the form of... weight is a 9x3 matrix.

[0352] When acquiring data for sampling points corresponding to a specific feature point, the control device can label these sampling points or the subset of sampling points corresponding to them, for example, by assigning them serial numbers, to establish a correlation between the sampling points or subset of sampling points and the corresponding feature points. Furthermore, when determining the corresponding denoising weight matrix based on the correlation between feature points, sampling points (or subsets of sampling points), and the denoising weight matrix, it is extremely convenient and fast to match the sampling points (or subsets of sampling points) and the denoising weight matrix.

[0353] Optionally, specifically when denoising the sampling points in the subset of sampling points based on the denoising weight matrix, for the same feature point, the sampling point can be denoised by combining the corresponding pair of sampling points and test points, the denoising weight matrix weight, and the mean PosRef of the test points in the test point set. Examples include:

[0354] (1) Determine the offset between the current test point associated with the same feature point and the mean of the test points in the test point set. For example, this offset can be determined using the following formula:

[0355]

[0356] in, Ro is the offset, and Ro is the current test point. For example, when there are 3 surface sensors, this offset is a row vector with 9 elements.

[0357] (2) Denoise the current sampling point based on the feature vectors associated with the same feature point, the current sampling point, and the denoising weight matrix. For example, denoising can be performed using the following formula:

[0358]

[0359] Where Tf is the current sampling point after denoising; To is the current sampling point.

[0360] In this embodiment, the corresponding pair of sampling points and test points can refer to sampling points and test points obtained at the same time in the same cycle of the anatomical structure, for example, the pair of points are both collected at the same time in the same cycle; or they can refer to sampling points and test points obtained at the same time in different cycles of the anatomical structure.

[0361] This application embodiment uses data from a body surface sensor to determine a denoising weight matrix, thereby enabling the aforementioned denoising of sensor data using the denoising weight matrix to be realized.

[0362] In one embodiment, the navigation method provided in this application includes:

[0363] Acquire multiple first medical images of the anatomical structure of the training object, wherein the multiple first medical images satisfy a first preset condition, wherein the first preset condition includes a first medical image representing the stretching process state of the anatomical structure and a first medical image representing the contraction process state of the anatomical structure among the multiple first medical images;

[0364] A recognition network is obtained by training based on the multiple first medical images;

[0365] In response to the user's navigation instructions, the first tube segment is identified from the patient's intraoperative medical images acquired by the imaging device using the recognition network;

[0366] Obtain a planned path from the initial position to the target position, wherein the planned path is composed of multiple connected segments of the second pipeline;

[0367] Based on the first pipeline segment, the target pipeline segment is matched from the second pipeline segment;

[0368] Generate navigation markers to mark the target pipeline segments;

[0369] The user interface displays the navigation markers and the target tubing segments to guide the movement of the catheter and / or instruments.

[0370] In one embodiment, the navigation method provided in this application includes:

[0371] In response to the user's navigation instructions, the first tube segment is identified from the patient's intraoperative medical images acquired by the imaging device using the recognition network;

[0372] Obtain a planned path from the initial position to the target position, wherein the planned path is composed of multiple connected segments of the second pipeline;

[0373] Based on the first pipeline segment, the target pipeline segment is matched from the second pipeline segment;

[0374] Generate navigation markers to mark the target pipeline segments;

[0375] The user interface displays the navigation markers and the target tubing segments to guide the movement of the catheter and / or instruments.

[0376] The identification of the first duct segment from intraoperative medical images of the patient acquired by the imaging device using the recognition network includes:

[0377] The intraoperative medical images of the patient acquired by the imaging device are scaled;

[0378] The first duct segment is identified from the scaled intraoperative medical image using the recognition network.

[0379] In one embodiment, the navigation method provided by this application includes:

[0380] In response to the user's navigation instructions, the first tube segment is identified from the patient's intraoperative medical images acquired by the imaging device using the recognition network;

[0381] The actual path points are corrected to obtain the corrected path points;

[0382] Based on the actual path points and the corrected path points, determine the positioning correction amount;

[0383] Determine the relative positions of the tubular segments in the intraoperative medical images;

[0384] Based on the first tube segment, the positioning correction amount, and the relative position of the tube segment in the intraoperative medical image, a third tube segment in the intraoperative medical image is identified, wherein the third tube segment is different from the first tube segment;

[0385] Obtain a planned path from the initial position to the target position, wherein the planned path is composed of multiple connected segments of the second pipeline;

[0386] Based on the first pipeline segment and the third pipeline segment, the target pipeline segment is matched from the second pipeline segment.

[0387] Generate navigation markers to mark the target pipeline segments;

[0388] The user interface displays the navigation markers and the target tubing segments to guide the movement of the catheter and / or instruments.

[0389] The specific implementation follows the same principle as described above, and will not be repeated here for the sake of brevity.

[0390] This application also provides a control device. For example... Figure 16 As shown, the control device may include: a processor 501, a communications interface 502, a memory 503, and a communications bus 504.

[0391] The processor 501, communication interface 502, and memory 503 communicate with each other through the communication bus 504.

[0392] The communication interface 502 is used to communicate with other network elements such as various sensors, rotary motors, solenoid valves, or other clients or servers.

[0393] The processor 501 is used to execute program 505, which can specifically perform the relevant steps in the above method embodiments.

[0394] Specifically, program 505 may include program code that includes computer operation instructions.

[0395] The processor 505 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), one or more integrated circuits configured to implement the embodiments of this application, an FPGA, or a graphics processing unit (GPU). The control device includes one or more processors, which may be processors of the same type, such as one or more CPUs or one or more GPUs; or they may be processors of different types, such as one or more CPUs and one or more GPUs.

[0396] Memory 503 is used to store program 505. Memory 503 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0397] Specifically, program 505 can be used to cause processor 501 to perform the steps of the method described in any of the above embodiments.

[0398] This application also provides a computer-readable storage medium storing a computer program configured to be loaded by a processor and executed to implement the steps of the method as described in any of the above embodiments.

[0399] This application also provides a computer program product, including computer instructions that, when executed on a computer, cause the computer to perform the method described in any of the above embodiments.

[0400] It should be noted that other sorting schemes that can be easily conceived by those skilled in the art within the technical scope disclosed in this invention should also be within the protection scope of this invention, and will not be elaborated here.

[0401] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional units or modules as needed, that is, the internal structure of the mobile terminal can be divided into different functional units or modules to complete all or part of the functions described above. The functional modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the modules in the mobile terminal can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0402] For example, a computer program can be divided into one or more modules / units, one or more of which are stored in memory and executed by a processor to perform the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a surgical robot.

[0403] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0404] The memory can be an internal storage unit of the surgical robot, such as its hard drive or RAM. It can also be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory can include both internal and external storage units. The memory is used to store computer programs and other programs and data required by terminal devices. It can also be used to temporarily store data that has been output or will be output.

[0405] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0406] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0407] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determination" or "if the described condition or event is detected" may be interpreted, depending on the context, as "once determination," "in response to determination," "once the described condition or event is detected," or "in response to the detection of the described condition or event."

[0408] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0409] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0410] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0411] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0412] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0413] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0414] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0415] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A navigation system, characterized in that, The navigation system is applied to a surgical robot, which includes a processor, catheters and / or instruments, the catheters and / or instruments being equipped with imaging devices for acquiring medical images, and the processor being configured to perform the following steps: Multiple first medical images of the anatomical structure of a training subject are acquired. A surface sensor is provided on the surface of the training subject to sense the surface data of the training subject. Based on the surface data, the process state represented by the first medical image is determined. The process state includes either a relaxation process state or a contraction process state. The multiple first medical images satisfy a first preset condition. The first preset condition includes that the multiple first medical images include a first medical image representing the relaxation process state of the anatomical structure and a first medical image representing the contraction process state of the anatomical structure. A recognition network is obtained by training based on the multiple first medical images; In response to the user's navigation instructions, the first tube segment is identified from the patient's intraoperative medical images acquired by the imaging device using the recognition network; Obtain a planned path from the initial position to the target position, wherein the planned path is composed of multiple connected segments of the second pipeline; Based on the first pipeline segment, the target pipeline segment is matched from the second pipeline segment; Generate navigation markers to mark the target pipeline segments; The user interface displays the navigation markers and the target tubing segments to guide the movement of the catheter and / or instruments.

2. The navigation system as described in claim 1, characterized in that, In acquiring multiple first medical images of the anatomical structure of the training subject, the processor is configured to perform the following steps: At the same time, acquire the first medical image and the body surface data of the training object sensed by the body surface sensor; Repeat the above steps until the number of first medical images representing multiple different process states meets a preset value. The first preset condition also includes that the number of first medical images representing multiple different process states meets the preset value.

3. The navigation system as described in claim 1, characterized in that, Prior to training based on the plurality of first medical images, the processor is configured to perform the following steps: A second medical image of one of the stretching states of the training object is acquired, and a first anatomical model of the training object is generated based on the second medical image; A third medical image of one of the contraction states of the training object is obtained, and a second anatomical model of the training object is generated based on the third medical image; Determine the maximum contraction ratio of the first anatomical model and the second anatomical model.

4. The navigation system as described in claim 3, characterized in that, Before training the recognition network based on the plurality of first medical images to obtain the recognition network, the processor is configured to perform the following steps: Determine the first scaling ratio based on the maximum shrinkage ratio; Based on the first scaling ratio, the first medical image is scaled to obtain a scaled first medical image. In the process of training the recognition network based on the plurality of first medical images, the processor is configured to perform the following steps: The recognition network is obtained by training based on the plurality of first medical images and the scaled first medical images.

5. The navigation system as described in claim 3, characterized in that, Before identifying the first duct segment from intraoperative medical images of the patient acquired by the imaging device using the recognition network in response to a user's navigation command, the processor is configured to perform the following steps in response to a motion compensation command: Determine the second scaling ratio based on the maximum shrinkage ratio; The intraoperative medical image is scaled based on the second scaling ratio.

6. The navigation system as described in claim 1, characterized in that, The catheter and / or device are provided with a tracking sensor for sensing the position of the catheter and / or device. The tracking sensor is used to acquire the actual path point of the catheter and / or device. In matching the target tubing segment from the second tubing segment based on the first tubing segment, the processor is configured to perform the following steps: The actual path points are corrected to obtain the corrected path points; Based on the actual path points and the corrected path points, determine the positioning correction amount; Determine the relative positions of the tubular segments in the intraoperative medical images; Based on the first duct segment, the positioning correction amount, and the relative position, a third duct segment in the intraoperative medical image is identified, wherein the third duct segment is different from the first duct segment; Based on the first pipeline segment and the third pipeline segment, the target pipeline segment is matched from the second pipeline segment.

7. The navigation system as described in claim 1, characterized in that, The catheter and / or device are equipped with a tracking sensor for measuring the position of the catheter and / or device, the tracking sensor being used to acquire the actual path points of the catheter and / or device, and the processor is configured to perform the following steps: Acquire preoperative medical images of the patient and generate an anatomical model of the patient based on the preoperative medical images, wherein the preoperative medical images characterize the contraction or relaxation process of the patient's anatomical structures; Obtain the transformation matrix, which is used to characterize the transformation relationship between the surgical environment coordinate system and the anatomical model coordinate system of the patient; Based on the actual path points and the transformation matrix, the simulated path points of the catheter and / or instrument in the anatomical model of the patient corresponding to the actual path points are determined; Alternatively, the actual path point can be corrected to obtain a corrected path point, and based on the corrected path point and the transformation matrix, the simulated path point of the catheter and / or instrument in the anatomical model of the patient corresponding to the actual path point can be determined. Mark the simulated path points in the anatomical model of the patient; The user interface displays the patient's anatomical model and the simulated path points.

8. The navigation system as described in claim 6 or 7, characterized in that, In correcting the actual path points to obtain corrected path points, the processor is configured to perform the following steps: Based on the actual path points, feature points that match the actual path points are determined, and the feature points are points in the anatomical structure of the patient. Obtain the denoising weight matrix associated with the feature points, denoise the actual path points based on the denoising weight matrix to obtain denoised path points, and use the denoised path points as the corrected path points.

9. The navigation system as claimed in claim 8, characterized in that, The tracking sensor acquires sampling points from the anatomical structure of the training object, and the surface sensor acquires test points from the surface of the training object. In acquiring the denoising weight matrix associated with the feature points, the processor is configured to perform the following steps: Obtain a subset of sampling points of the training object, wherein the subset of sampling points of the training object includes multiple sampling points; Obtain the test point set of the training object, wherein the test point set of the training object includes multiple test points; Based on the subset of sampling points and the set of test points, the denoising weight matrix corresponding to the feature points of the training object is determined.

10. The navigation system as claimed in claim 6 or 7, characterized in that, In the process of correcting the actual path points to obtain corrected path points, the processor is configured to perform the following steps: Acquire data on the movement of the catheter within the patient's anatomical network of channels; The corrected path point is determined based on the actual path point, the movement data, and the intraoperative medical images.

11. The navigation system as claimed in claim 1, characterized in that, The intraoperative medical images characterize the contraction or relaxation processes of the patient's anatomical structures. Surface sensors are installed on the patient's body surface. The processor is configured to perform the following steps: At the same time, the intraoperative medical images and the patient's surface data sensed by the surface sensors are acquired; Based on the body surface data, vital signs data of the patient's anatomical structure are determined, and the vital signs data characterize the patient's anatomical structure under different process states; Based on the vital signs data, a vital signs curve of the patient's anatomical structure is generated and displayed on the user interface. The vital signs curve includes a respiratory curve and / or a heart rate curve.

12. A computer-readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements navigation methods, which include: Multiple first medical images of the anatomical structure of the training object are acquired. Based on the body surface data of the training object, the process state represented by the first medical images is determined. The process state includes either a stretching process state or a contraction process state. The multiple first medical images satisfy a first preset condition. The first preset condition includes that the multiple first medical images include a first medical image representing the stretching process state of the anatomical structure and a first medical image representing the contraction process state of the anatomical structure. A recognition network is obtained by training based on the multiple first medical images; In response to the user's navigation instructions, the first tube segment is identified from the patient's intraoperative medical images acquired by the imaging device using the recognition network; Obtain a planned path from the initial position to the target position, wherein the planned path is composed of multiple connected segments of the second pipeline; Based on the first pipeline segment, the target pipeline segment is matched from the second pipeline segment; Generate navigation markers to mark the target pipeline segments; The user interface displays the navigation markers and the target tubing segments to guide the movement of catheters and / or instruments.

Images