Navigation system and readable storage medium

Abstract

The invention provides a navigation system and a readable storage medium, the navigation system is applied to a surgical robot, the surgical robot comprises a processor, a catheter and / or an instrument, the catheter and / or the instrument is provided with an image device used for collecting medical images, and the processor is configured to execute the following steps: obtaining a plurality of medical images of a training object, the plurality of medical images meet a first preset condition; training according to the plurality of medical images to obtain a recognition network; in response to the navigation instruction, identifying a first pipeline segment from the intraoperative medical image by using an identification network; a planned path is obtained, wherein the planned path is composed of a plurality of communicated second pipeline segments; matching a target pipeline segment from the second pipeline segments based on the first pipeline segment; generating a navigation mark for marking the target pipeline segment; and the navigation mark and the target pipeline segment are displayed on the user interface and are used for guiding the movement of the catheter, so that accurate and efficient navigation is realized.

Description

Technical Field

[0001] The present application belongs to the field of medical devices, and in particular, relates to a navigation system and a readable storage medium. Background Art

[0002] Minimally invasive medical technology intends to reduce the amount of tissue damaged during medical procedures to reduce patient recovery time, discomfort and harmful side effects. In minimally invasive medical technology, it is often necessary to insert a catheter through a natural orifice in the patient's anatomical structure or through a surgical incision, and with the cooperation of an intraoperative navigation system, pass through the complex structure of the human body to reach or approach the target.

[0003] In most cases, the catheter moves along the human body's ducts (such as bronchi, blood vessels, ureters, etc.) and does not shuttle freely in a way that damages the ducts. Therefore, it can be considered that the movement path of the catheter and / or instrument is limited within the human body's ducts and is similar to the centerline of the human body's ducts. Especially for smaller ducts, due to the limited radial range of motion, the movement path is more similar to the centerline. However, due to the large number of branches and complex structures of ducts in the human body, and the fact that they are easily affected by the body's own intrinsic movements during surgery, such as tissue deformation caused by heartbeat and breathing, it is difficult for doctors to operate the catheter and / or instrument to reach the target lesion, and accurate and real-time navigation guidance is required.

[0004] Therefore, how to improve the accuracy and efficiency of intraoperative real-time navigation is a problem that needs to be solved at present. Summary of the invention

[0005] The embodiments of the present application provide a navigation system and a readable storage medium, which can improve the accuracy and efficiency of real-time navigation during surgery and reduce the impact of the intrinsic movement of the human anatomical structure.

[0006] In the first aspect, an embodiment of the present application provides a navigation system, which is applied to a surgical robot, wherein the surgical robot includes a processor, a catheter and / or an instrument, wherein an image device for acquiring medical images is provided on the catheter and / or the instrument, and the processor is configured to perform the following steps:

[0007] Acquire a plurality of first medical images of the anatomical structure of the training object, wherein the plurality of first medical images satisfy a first preset condition, wherein the first preset condition comprises that the plurality of first medical images include a first medical image representing a stretching process state of the anatomical structure and a first medical image representing a contracting process state of the anatomical structure;

[0008] Performing training according to the plurality of first medical images to obtain a recognition network;

[0009] In response to a user's navigation instruction, using the recognition network to identify a first pipeline segment from an intraoperative medical image of the patient acquired by the imaging device;

[0010] Acquire a planned path from an initial position to a target position, wherein the planned path is composed of a plurality of connected second pipeline segments;

[0011] Based on the first pipeline segment, matching a target pipeline segment from the second pipeline segment;

[0012] generating a navigation mark marking the target pipeline segment;

[0013] The navigation mark and the target channel segment are displayed on a user interface to guide the movement of the catheter and / or instrument.

[0014] Optionally, a body surface sensor is provided on the body surface of the training object, and in acquiring a plurality of first medical images of the anatomical structure of the training object, the processor is configured to perform the following steps:

[0015] Acquire the first medical image and the body surface data of the training object sensed by the body surface sensor at the same time;

[0016] Based on the body surface data, determining a process state represented by the first medical image, wherein the process state includes any one of a stretch process state and a contraction process state;

[0017] The above steps are repeatedly performed until the number of first medical images representing multiple different process states meets a preset value, and the first preset condition also includes that the number of first medical images representing multiple different process states meets the preset value.

[0018] Optionally, before performing training according to the plurality of first medical images, the processor is configured to perform the following steps:

[0019] Acquire a second medical image of the training object in a stretching process state, and generate a first anatomical model of the training object based on the second medical image;

[0020] Acquire a third medical image of one of the contraction process states of the training object, and generate a second anatomical model of the training object based on the third medical image;

[0021] A maximum shrinkage ratio of the first anatomical model and the second anatomical model is determined.

[0022] Optionally, before training according to the plurality of first medical images to obtain a recognition network, the processor is configured to perform the following steps:

[0023] Determining a first scaling ratio according to the maximum shrinkage ratio;

[0024] Based on the first scaling ratio, scaling the first medical image to obtain a scaled first medical image;

[0025] In the step of training the plurality of first medical images to obtain a recognition network, the processor is configured to perform the following steps:

[0026] The recognition network is obtained by training according to the multiple first medical images and the scaled first medical image.

[0027] Optionally, before identifying the first pipeline segment from the intraoperative medical image of the patient acquired by the imaging device using the recognition network in response to the user's navigation instruction, the processor is configured to perform the following steps in response to the motion compensation instruction:

[0028] Determining a second scaling ratio according to the maximum shrinkage ratio;

[0029] The intraoperative medical image is scaled based on the second scaling ratio.

[0030] Optionally, the catheter and / or the instrument is provided with a tracking sensor for sensing the position of the catheter and / or the instrument, and the tracking sensor is used to obtain the actual path point of the catheter and / or the instrument. In matching the target pipeline segment from the second pipeline segment based on the first pipeline segment, the processor is configured to perform the following steps:

[0031] Correcting the actual path point to obtain a corrected path point;

[0032] Determining a positioning correction amount according to the actual path point and the corrected path point;

[0033] determining relative positions of the duct segments in the intraoperative medical image;

[0034] identifying a third pipeline segment in the intraoperative medical image according to the first pipeline segment, the positioning correction amount, and a relative position of the pipeline segment in the intraoperative medical image, wherein the third pipeline segment is different from the first pipeline segment;

[0035] Based on the first pipeline segment and the third pipeline segment, a target pipeline segment is matched from the second pipeline segment.

[0036] Optionally, the catheter and / or the instrument is provided with a tracking sensor for measuring the position of the catheter and / or the instrument, and the sensor is used to obtain the actual path point of the catheter and / or the instrument. The processor is configured to perform the following steps:

[0037] Acquire a preoperative medical image of the patient, and generate an anatomical model of the patient based on the preoperative medical image, wherein the preoperative medical image represents a contraction process state or an extension process state of the anatomical structure of the patient;

[0038] Acquire a transformation matrix, where the transformation matrix is ​​used to represent the conversion relationship between the surgical environment coordinate system and the patient's anatomical model coordinate system;

[0039] Based on the actual path points and the transformation matrix, determine the simulated path points of the catheter and / or instrument in the anatomical model of the patient corresponding to the actual path points; or, correct the actual path points to obtain corrected path points, and determine the simulated path points of the catheter and / or instrument in the anatomical model of the patient corresponding to the actual path points based on the corrected path points and the transformation matrix;

[0040] marking the simulated waypoints in an anatomical model of the patient;

[0041] An anatomical model of the patient and the simulated waypoints are displayed on the user interface.

[0042] Optionally, in correcting the actual path point to obtain the corrected path point, the processor is configured to perform the following steps:

[0043] Based on the actual path point, determining a feature point that matches the actual path point, the feature point being a point in the anatomical structure of the patient;

[0044] A denoising weight matrix associated with the feature point is obtained, and based on the denoising weight matrix, the actual path point is denoised to obtain a denoised path point, and the denoised path point after denoising is used as the corrected path point.

[0045] Optionally, a body surface sensor is provided on the body surface of the training object, the tracking sensor acquires sampling points from the anatomical structure of the training object, the body surface sensor acquires test points from the body surface of the training object, and in acquiring a denoising weight matrix associated with the feature points, the processor is configured to perform the following steps:

[0046] Acquire a sampling point subset of the training object, wherein the sampling point subset of the training object includes a plurality of sampling points;

[0047] Acquire a test point set of the training object, wherein the test point set of the training object includes a plurality of test points;

[0048] Based on the sampling point subset and the test point set, the denoising weight matrix corresponding to the feature points of the training object is determined.

[0049] Optionally, in correcting the actual path point to obtain the corrected path point, the processor is configured to perform the following steps:

[0050] Acquire movement data of the catheter in the conduit network of the patient's anatomical structure;

[0051] The revised pathpoints are determined based on the actual pathpoints, the movement data and the intraoperative medical image.

[0052] Optionally, the intraoperative medical image represents a contraction process state or an extension process state of the anatomical structure of the patient, a body surface sensor is provided on the body surface of the patient, and the processor is configured to perform the following steps:

[0053] Acquire the intraoperative medical image and the body surface data of the patient sensed by the body surface sensor at the same time;

[0054] Determine vital sign data of the patient's anatomical structure according to the body surface data, wherein the vital sign data represents data of the patient's anatomical structure in different process states;

[0055] A vital sign curve of the anatomical structure of the patient is generated according to the vital sign data, and the vital sign curve is displayed on the user interface, wherein the vital sign curve includes a breathing curve and / or a heartbeat curve.

[0056] Optionally, in the second aspect, an embodiment of the present application provides a navigation method, including:

[0057] In response to a user's navigation instruction, using the recognition network to identify a first pipeline segment from an intraoperative medical image of the patient acquired by the imaging device;

[0058] Correcting the actual path point to obtain a corrected path point;

[0059] Determining a positioning correction amount according to the actual path point and the corrected path point;

[0060] determining relative positions of the duct segments in the intraoperative medical image;

[0061] identifying a third pipeline segment in the intraoperative medical image according to the first pipeline segment, the positioning correction amount, and a relative position of the pipeline segment in the intraoperative medical image, wherein the third pipeline segment is different from the first pipeline segment;

[0062] Acquire a planned path from an initial position to a target position, wherein the planned path is composed of a plurality of connected second pipeline segments;

[0063] Based on the first pipeline segment and the third pipeline segment, a target pipeline segment is matched from the second pipeline segment.

[0064] generating a navigation mark marking the target pipeline segment;

[0065] The navigation mark and the target channel segment are displayed on a user interface to guide the movement of the catheter and / or instrument.

[0066] In a third aspect, an embodiment of the present application provides a navigation method, including:

[0067] Acquire a plurality of first medical images of the anatomical structure of the training object, wherein the plurality of first medical images satisfy a first preset condition, wherein the first preset condition comprises that the plurality of first medical images include a first medical image representing a stretching process state of the anatomical structure and a first medical image representing a contracting process state of the anatomical structure;

[0068] Performing training according to the plurality of first medical images to obtain a recognition network;

[0069] In response to a user's navigation instruction, using the recognition network to identify a first pipeline segment from an intraoperative medical image of the patient acquired by the imaging device;

[0070] Acquire a planned path from an initial position to a target position, wherein the planned path is composed of a plurality of connected second pipeline segments;

[0071] Based on the first pipeline segment, matching a target pipeline segment from the second pipeline segment;

[0072] generating a navigation mark marking the target pipeline segment;

[0073] The navigation mark and the target channel segment are displayed on a user interface to guide the movement of the catheter and / or instrument.

[0074] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which may store a program, and when the program is executed by a processor, implements part or all of the steps of the navigation method of the second and third aspects above.

[0075] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on a surgical robot, enables the surgical robot to execute part or all of the steps of the navigation method of any one of the second and third aspects described above.

[0076] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0077] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain multiple first medical images of the anatomical structure of the training object, wherein the multiple first medical images satisfy a first preset condition, wherein the first preset condition includes that the multiple first medical images include a first medical image representing the stretching process state of the anatomical structure and a first medical image representing the contraction process state of the anatomical structure; training is performed based on the multiple first medical images to obtain a recognition network; in response to a user's navigation instruction, the recognition network is used to identify a first pipeline segment from an intraoperative medical image of the patient acquired by the image device; a planned path from an initial position to a target position is obtained, wherein the planned path consists of multiple connected second pipeline segments; based on the first pipeline segment, a target pipeline segment is matched from the second pipeline segment; a navigation mark is generated to mark the target pipeline segment; the navigation mark and the target pipeline segment are displayed on a user interface to guide the movement of the catheter and / or instrument, thereby achieving accurate and efficient navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0079] Figure 1 A schematic diagram of the structure of a catheter robot provided in one embodiment of the present application;

[0080] Figure 2a A schematic diagram of the structure of a catheter assembly and a power unit provided in one embodiment of the present application;

[0081] Figure 2b A schematic diagram of the structure of a catheter assembly provided in one embodiment of the present application;

[0082] Figure 2c A schematic diagram of the structure of a catheter assembly provided in one embodiment of the present application;

[0083] Figure 3 A schematic flow chart of a navigation method for a catheter robot provided in one embodiment of the present application;

[0084] Figure 4 A schematic diagram of the first lung segment in the navigation method of an embodiment of the present application;

[0085] Figure 5 A schematic diagram of a planned path in a navigation method according to an embodiment of the present application;

[0086] Figure 6 This is a schematic diagram of a target lung segment in a navigation method in an embodiment of the present application;

[0087] Figure 7 A schematic diagram of a preoperative image after annotation in a navigation method according to an embodiment of the present application;

[0088] Figure 8a A schematic diagram of the position of body surface sensors on the patient's body surface provided in one embodiment of the present application;

[0089] Figure 8b A schematic diagram of a respiratory coefficient obtained by using EM data of at least one respiratory cycle sensed by a body surface sensor according to an embodiment of the present application;

[0090] Figure 9a A schematic diagram of a lung segment in an intraoperative image provided in an embodiment of the present application;

[0091] Figure 9b A schematic diagram of a target lung segment provided in an embodiment of the present application;

[0092] Fig.10 A schematic diagram of the effect before and after denoising of the catheter motion path provided by an embodiment of the present application;

[0093] Fig.11 A simplified schematic diagram of a user interface for user operation provided by an embodiment of the present application;

[0094] Fig.12 A schematic flow chart of a navigation method for a catheter robot provided in another embodiment of the application;

[0095] Fig.13 A schematic diagram of simulated feature points in an anatomical model provided by an embodiment of the present application;

[0096] Figures 14-15 A schematic flow chart of a navigation method for a catheter robot provided in another embodiment of the application;

[0097] Fig.16 This is a schematic diagram of the principle of a control device of a telemedicine system according to an embodiment of the present application. DETAILED DESCRIPTION

[0098] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0099] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0100] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0101] The following describes a navigation system and a computer-readable storage medium according to an embodiment of the present application in conjunction with the accompanying drawings.

[0102] Figure 1 The catheter system 1000 provided in one embodiment of the present application is shown. The catheter system 1000 includes an imaging vehicle 100, a trolley 200 and a main controller 300 respectively connected to the imaging vehicle 100, a catheter assembly 400 that can be coupled to the trolley 200, a sensor system 500 connected to the trolley 200, and a control system 600 for realizing control between the catheter assembly 400, the main controller 300, the sensor system 500 and the imaging vehicle 100. Among them, the main controller 300 can be connected to the trolley 200 by wire or wirelessly. When the operator performs various procedures on the patient next to the trolley 200, the control instruction can be triggered by operating the main controller 300, and the catheter assembly 400 can be controlled to advance, retract, bend and turn, etc. through the drive of the trolley 200.

[0103] The trolley 200 can usually be moved to the side of the operating bed to engage the catheter assembly 400, and control the catheter assembly 400 to move up and down in the vertical direction, or translate in the horizontal direction, or move in non-vertical and non-horizontal directions under the control command, so as to provide a better preoperative preparation angle for the operation of the catheter assembly 400. Among them, the control command can be a command triggered by the operator by operating the main controller 300, or it can be a command triggered by the operator directly clicking or pressing a button set on the trolley 200. Of course, in other embodiments, the control command can also be a command triggered by voice control or a force feedback mechanism.

[0104] like Figure 1As shown, further, the trolley 200 may include a base 210, a sliding seat body 220 that can be lifted and moved along the base 210, and two mechanical arms 230 fixedly connected to the sliding seat body 220. The mechanical arm 230 may include a plurality of arm segments connected at a joint, and the plurality of arm segments provide a plurality of degrees of freedom for the mechanical arm 230, for example, seven degrees of freedom corresponding to seven arm segments. A power unit (not shown) is provided at the end of the mechanical arm 230, and the power unit of the mechanical arm 230 is used to engage the catheter assembly 400, and the end of the catheter assembly 400 is controlled to be bent and turned accordingly under the driving action of the power unit. Wherein, the two mechanical arms 230 may be structures that are completely the same or partially the same, one mechanical arm 230 is used to engage the inner catheter assembly 410, and the other mechanical arm 230 is used to engage the outer catheter assembly 420. During installation, the outer catheter assembly 420 may be installed first, and when the outer catheter assembly 420 is installed, the catheter of the inner catheter assembly 410 is inserted into the catheter of the outer catheter assembly 420 .

[0105] The sensor system 500 has one or more subsystems for receiving information about the catheter assembly 400. The subsystems may include: a position sensor system; a shape sensor system for determining the position, orientation, speed, velocity, posture and / or shape of the distal end of the catheter assembly 400 and / or along one or more sections of the catheter that may constitute the catheter assembly 400; and / or a visualization system for capturing images from the distal end of the catheter assembly 400. The visualization system may include an imaging device, which may be a camera or the like.

[0106] The imaging vehicle 100 can be provided with a display system 110 and a flushing system (not shown in the figure), etc. The display system 110 is used to display the images or representations of the surgical site and the catheter assembly 400 generated by the subsystem of the sensor system 500. The real-time images of the surgical site and the catheter assembly 400 captured by the visualization system can also be displayed. The image data from the imaging technology can also be used to present the image of the surgical site recorded before or during the operation, and the imaging technology is such as computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound, etc. The preoperative or intraoperative medical image data can be presented as a two-dimensional, three-dimensional or four-dimensional (such as time-based or rate-based information) image and / or presented as an image from a model created according to the preoperative or intraoperative medical image data set, and a virtual navigation image can also be displayed. In the virtual navigation image, the actual position of the catheter assembly 400 is registered with the preoperative image to present the virtual image of the catheter assembly 400 in the surgical site to the operator from the outside.

[0107] The control system 600 includes at least one memory and at least one processor. It is understandable that the control system 600 can be integrated in the trolley 200 or the imaging vehicle 100, or can be independently arranged. The control system 600 can support wireless communication protocols such as IEEE 802.11, IrDA, Bluetooth, HomeRF, DECT and wireless telemetry. The control system 600 can transmit one or more signals indicating that the catheter assembly 400 moves and that the catheter assembly 400 is moved by the power unit. The catheter assembly 400 can extend to the surgical position in the body via the opening of the patient's natural cavity or surgical incision.

[0108] Further, the control system 600 may include a mechanical control system (not shown in the figure) and an image processing system (not shown in the figure), and the mechanical control system is used to control the movement of the catheter assembly 400, so it can be integrated in the trolley 200. The image processing system is used for virtual navigation path planning, so it can be integrated in the imaging vehicle 100. Of course, the various subsystems of the control system 600 are not limited to the specific circumstances listed above, and can also be reasonably set according to actual conditions. Among them, the image processing system can be based on the image of the surgical site recorded before or during the operation, and the surgical site is imaged using the above-mentioned imaging technology. The software used in combination with manual input can also convert the recorded image into a two-dimensional or three-dimensional synthetic image of a part or the entire anatomical organ or segment. During the virtual navigation program, the sensor system 500 can be used to calculate the position of the catheter assembly 400 relative to the patient's anatomical structure.

[0109] The inner catheter assembly 410 and the outer catheter assembly 420 have substantially the same structural composition, and each comprises a slender and flexible inner catheter 41 and an outer catheter 42, wherein the diameter of the outer catheter 42 is slightly larger than that of the inner catheter 41, so that the inner catheter 41 can pass through the outer catheter 42 and provide a certain support for the inner catheter 41, so that the inner catheter 41 can reach the target location in the patient's body, so as to facilitate operations such as tissue or cell sampling from the target location.

[0110] Certain movements of the main controller 300 may cause corresponding movements of the catheter assembly 400. For example, when the operator operates the direction lever of the main controller 300 to move upward or downward, the movement of the direction lever of the main controller 300 may be mapped to the corresponding pitch movement of the end of the catheter assembly 400; when the operator operates the direction lever of the main controller 300 to move left or right, the movement of the direction lever of the main controller 300 may be mapped to the corresponding yaw movement of the end of the catheter assembly 400. In this embodiment, the main controller 300 may control the end of the catheter assembly 400 to move within a 360° spatial range.

[0111] Figure 2a to 2c The catheter assembly 400 provided in one embodiment of the present application is shown. The catheter assembly 400 is configured to engage with the power unit 240 of the mechanical arm 230, and the catheter assembly 400 includes an instrument box 45 configured to engage with the power unit 240 and a catheter 48 connected to the instrument box 45. Among them, the "engagement" refers to the state in which the driving force of the power unit 240 can be transmitted to the instrument box 45 when the instrument box 45 is installed to the power unit 240, and the catheter 48 can move normally. For example, under the driving force of the power unit 240, the end of the catheter 48 can be bent and turned. The catheter 48 may include the aforementioned inner catheter 41 and outer catheter 42. Unless otherwise specified in the embodiment of the present application, the catheter is not distinguished as an inner catheter or an outer catheter.

[0112] The catheter assembly 400 further includes a channel tube 12, an image device 13, and a tip portion 14. The tip portion 14 is connected to the distal end of the catheter 48. The image device 13 may be disposed at the tip portion 14, and the image device 13 is used to collect medical images. The image device 13 may include a light source 13b and an image module 13a, and the light source 13b is used to provide supplementary light for the environment where the image module 13a is located.

[0113] The catheter 48 may also be provided with a tracking sensor 15 for locating the position of the distal end of the catheter 10. The tracking sensor 15 is at least partially fixed on the tip 14. For example, the tracking sensor 15 may include an EM (Electro Magnetic) sensor. A detection magnetic field is arranged in the environment where the patient is located. The detection magnetic field can be generated by a magnetic field generator arranged on one side of the patient. When the position and angle of the EM sensor are different, the current generated in the coil of the EM sensor is different, thereby locating the position and orientation of the distal end of the catheter 10. In other embodiments, the positioning sensor may also be replaced by other sensor modules, such as a light sensor, an ultrasonic probe, a gyroscope, etc. The number of tracking sensors 15 can be set as needed. Optionally, the tracking sensor 15 is arranged in the inner catheter 41.

[0114] The catheter assembly 400 further includes an instrument 30, which is detachably inserted into and removed from the working channel 10H where the channel tube 12 is located. The instrument 30 includes but is not limited to various surgical instruments, such as cauterization instruments, clamping instruments, cutting instruments, suturing instruments, electric hooks, and irrigation instruments. Figure 2c The device 30 shown is a flushing device. A fluid channel 321 for fluid to pass through is opened axially of the device 30. By inserting the device 30 along the working channel 10H of the catheter 10, the target site can be flushed.

[0115] For example, the tracking sensor 15 can be made on the distal end surface of the instrument 30, and the tracking sensor 15 can be used to track and provide navigation for the instrument 30, for example, to provide navigation in real time during the flushing process. Figure 2c The instrument 30 includes a slender tube body 32 and a tracking sensor 15. The instrument 30 is configured to be detachably inserted into the working channel 10H. The slender tube body 32 is provided with a fluid channel 321 running through its length. The tracking sensor 15 is fixed at the distal end of the slender tube body 32 and is arranged adjacent to the fluid channel 321. Through the fluid channel 321, the injection and extraction of fluids such as flushing fluid can be achieved. The tracking sensor 15 provided in the instrument 30 can be the same as the tracking sensor 15 provided in the catheter, including but not limited to an EM sensor, a light sensor, an ultrasonic probe, and a gyroscope. The contours of the working channel 10H and the slender tube body 32 match, and the two are clearance-matched.

[0116] Exemplarily, the slender tube body 32 is provided with a mounting channel 322 running through its length direction and an arc-shaped partition 323 running through its length direction and arched toward the mounting channel 322. The tracking sensor 15 is fixed in the mounting channel 322, and the mounting channel 322 and the fluid channel 321 are separated by the partition 323. The signal line of the tracking sensor 15 can be led out from the mounting channel 322 to the proximal end.

[0117] The inner catheter 41 and the outer catheter 42 usually move in a linked manner, but they can also move independently. The instrument 30 usually moves in the catheter, so as long as the catheter is positioned accurately, the instrument 30 is positioned accurately. The instrument 30 can also move independently of the catheter, and the instrument 30 can also be provided with a tracking sensor 15. Similarly, the instrument 30 can also be provided with an imaging device 13. The tracking sensor 15 and the imaging device 13 can perform positioning navigation for the outer catheter 42, the inner catheter 41 or the instrument 30 separately, or can perform positioning navigation for them simultaneously.

[0118] The end in this application may also be referred to as the distal end or the head, which refers to the end away from the instrument box 45 ; the front end may also be referred to as the proximal end or the tail, which refers to the end close to the instrument box 45 .

[0119] Before surgery, a preoperative examination is usually performed and an anatomical model of the human body is constructed based on the preoperative medical images obtained during the preoperative examination for navigation guidance during the actual surgery. However, the human body's pipeline network will deform due to movement or compression, so the actual state of the pipeline network during surgery may change over time, which is quite different from the medical images taken during the preoperative examination. Therefore, if the anatomical model of the human body constructed based on the preoperative medical examination is still used for navigation, there may be errors. The pipelines may include, for example, bronchi, blood vessels, ureters, intestines, etc. Movement includes periodic movement and non-periodic movement. Periodic movement includes, for example, heartbeat, breathing, etc., and non-periodic movement includes coughing, etc.

[0120] Before surgery, the patient is subjected to a medical examination, for example, a target area of ​​the patient is scanned to obtain a medical image (such as CT, MRI, OCT or ultrasound scan, etc.), the preoperative image is three-dimensionally reconstructed to obtain a three-dimensional preoperative image, and the three-dimensional preoperative image is appropriately processed to obtain an anatomical model of the patient. The anatomical model of the patient's anatomical structure can generally be a three-dimensional or four-dimensional model. In some embodiments, taking the anatomical structure as a bronchus and the anatomical model as a three-dimensional model as an example, the steps for obtaining the anatomical model of the anatomical structure are implemented as follows:

[0121] Preoperative images of anatomical structures are obtained by scanning (or photographing) such as CT, MRI, OCT or ultrasound. During the scan, the patient usually needs to take a deep breath and hold it until the scan is completed. The bronchi are largest and the fine branches are easier to image when the human body is in the inhalation state.

[0122] Segment and reconstruct preoperative medical images of anatomical structures. For example, the medical images of patients can be segmented by segmentation algorithms such as region growing and convolutional neural networks to segment the bronchi of the lungs; and then the segmented images can be reconstructed into a three-dimensional model by, for example, marching cube algorithms.

[0123] Image segmentation is used to extract the human ducts involved in the surgery from the three-dimensional image. To facilitate intraoperative navigation, the skeleton of the human duct is extracted to obtain the duct centerline. The duct centerline is the centerline of the human duct, which can also be called the skeleton of the human duct. It is a curve used to describe some geometric features of the human duct. It is located in the middle of the human duct and has the same topological structure as the human duct. Its width is generally a single pixel. The duct centerline can provide the information required for intraoperative navigation. At the same time, compared with the original human duct, its data volume is significantly reduced, and it is more convenient to process, which is conducive to real-time navigation.

[0124] The pipeline centerline is a three-dimensional curve. In practical applications, the pipeline centerline is often stored and used in the form of multiple three-dimensional points. In other words, the pipeline centerline includes these three-dimensional points, which can be called skeleton points, and the set of skeleton points can be called a skeleton point set or a skeleton point cloud. The pipeline centerline can be preprocessed. Preprocessing can include extracting characteristic skeleton points and / or resampling the pipeline centerline from the skeleton point set. The characteristic skeleton points generally include bifurcation points and end points, which are used to segment the pipeline centerline. Each segment can be called a pipeline centerline segment or simply a pipeline segment. For example, for the lungs, the bifurcation points and the bifurcation points constitute a connecting segment, i.e., a lung segment. Generally speaking, the lung segments of most people are distributed in the same way, but slight differences are not excluded. Therefore, for the accuracy of medical operations, the lung segments of each patient can also be confirmed in the embodiment of the present application, that is, after identifying the bifurcation points, all lung segments are determined based on the principle that adjacent bifurcation points constitute lung segments.

[0125] The embodiment of the present application aims to remove the influence of deformation of the pipeline network due to movement or squeezing, and ensure that the pre-trained recognition network can effectively identify pipeline segments from intraoperative medical images in actual surgery, and then navigate based on the identified pipeline segments. The method of achieving effective recognition of pipeline segments in the embodiment of the present application includes two directions: in the training process, training a recognition network that can remove the influence of deformation; or in the recognition process using the recognition network, the influence of deformation can be removed.

[0126] The present application embodiment provides a navigation system, which is applied to a surgical robot. The surgical robot includes a processor, a catheter and / or an instrument. The catheter and / or the instrument are provided with an imaging device for acquiring medical images. The processor is configured to perform the following steps to implement the navigation method provided in an embodiment of the present application, such as: Figure 3 As shown, a flowchart of a navigation method provided by an embodiment of the present application is provided, and the navigation method includes: pre-training a recognition network, including step S10 and step S20. The recognition network can be used to recognize pipeline segments in intraoperative medical images acquired by an imaging device.

[0127] Step S10: Acquire multiple first medical images of the anatomical structure of the training subject, wherein the multiple first medical images satisfy a first preset condition, wherein the first preset condition includes a first medical image representing the stretching process state of the anatomical structure and a first medical image representing the contraction process state of the anatomical structure in the multiple first medical images. The training subject may include the patient himself or a non-patient himself.

[0128] For the periodic motion of the anatomical structure, there are two types of process states as a whole: contraction process state and stretch process state. Further, each type of process state includes multiple contraction / stretch process states of different degrees. Each cycle is composed of multiple moments, and the anatomical structure corresponds to different process states at different moments, and the vital signs data shown by the anatomical structure are also different. For example, when the periodic motion is a respiratory motion, the anatomical structure may include bronchi, and the bronchi may correspond to a contraction process state (expiratory state) or a stretch process state (inspiratory state) at different moments. Different degrees of exhalation states are also regarded as different process states, and different degrees of inspiratory states are also regarded as different process states, that is, the contraction process state includes many contraction process states of different degrees, and the stretch process state includes many stretch process states of different degrees, that is, the process states corresponding to different moments in the cycle are unique. At different moments or different process states, the collected medical images are also different, so the medical images can also represent the process states of the anatomical structure. Taking respiratory movement as an example, the multiple first medical images include a first medical image representing the inhalation state of the bronchus and a first medical image representing the exhalation state of the bronchus.

[0129] Step S20: training according to the plurality of first medical images to obtain a recognition network. In the training process of the recognition network, the first medical images used for training include both the first medical images in the stretching process state and the first medical images in the contraction process state, so that the trained recognition network can recognize the intraoperative medical images in the stretching process state and the intraoperative medical images in the contraction process state, better eliminate the influence of deformation of the pipeline network due to periodic motion or compression, and improve the recognition network's recognition ability for pipeline segmentation.

[0130] Furthermore, the first preset condition also includes that the number of first medical images representing multiple different process states satisfies the preset value, the body surface of the training object is provided with a body surface sensor, and the acquiring of multiple first medical images of the anatomical structure of the training object specifically includes steps (1) to (3):

[0131] (1) The first medical image and the body surface data of the training object sensed by the body surface sensor are acquired at the same time, that is, the time when the first medical image is acquired is the same as the time corresponding to the body surface data, or the process state represented by the first medical image is the same as the process state represented by the body surface data. By acquiring the medical image and the body surface data at the same time or the same process state, the specific process state represented by the medical image is determined based on the body surface data.

[0132] The surface data of the training object is obtained through the surface sensor, and the sampling period for obtaining the surface data of the training object includes at least one period. The duration of a normal breathing cycle is usually 3 to 5 seconds. Taking the surgical robot as a catheter machine as an example, the catheter robot also includes a surface sensor configured on the surface of the training object, and there are at least three surface sensors for obtaining the surface data of the training object. Among them, the surface sensor can generally be a position sensor or a posture sensor, which is usually non-invasively and stably arranged on the surface of the training object corresponding to the anatomical structure, that is, the surface sensor is generally exposed on the surface of the training object. The surface sensor can not only use EM sensors, but also optical positioning sensors.

[0133] In some embodiments, for example, when the anatomical structure is a bronchus, since the respiratory amplitudes of various parts of the lungs are different, the surface sensor in the middle of the main bronchus actually measures the respiratory amplitude of the front end of the main bronchus, and its amplitude is relatively small, while the surface sensors on the left and right sides of the bronchial end are mainly used to measure the respiratory amplitude of the left and right lung ends. The movement amplitude of the diaphragm at the lower end of the lung segment is the largest during breathing. Therefore, the surface sensors are arranged at a position point in the middle of the main bronchus of the lung on the body surface of the training subject that does not change much, and at a position point on the left and right sides of the bronchial end that changes greatly. As the breathing state of the training subject changes, the anatomical structure changes in six dimensions: up and down, left and right, and front and back. In theory, the more sensors there are, the richer the breathing state changes obtained, that is, the more surface sensors there are, the more areas of the training subject's body surface are covered, and the more accurate the breathing movement of the anatomical structure is sensed. For example, at least three surface sensors are arranged on the chest surface of the training subject. Among them, one surface sensor can be arranged in the middle of the training subject's chest, and the other two surface sensors can be arranged in the area of ​​the training subject's chest corresponding to the 7th rib on the left, and in the area of ​​the training subject's chest corresponding to the 7th rib on the right. During the operation, the surface sensors need to be fixed with tape to avoid slipping during the operation.

[0134] (2) determining a process state represented by the first medical image based on the body surface data, wherein the process state includes any one of an expansion process state and a contraction process state;

[0135] First, based on the body surface data, the vital sign data of the anatomical structure of the training object is determined, and the vital sign data represents the data of the anatomical structure of the training object in different process states. The vital sign data may include the area S, area slope or motion coefficient of the figure enclosed by the body surface sensors, wherein the area of ​​the figure enclosed by the body surface sensors represents the area size of the figure enclosed by the body surface sensors with different breathing states of the training object; the area slope represents the rate of change of the area; and the motion coefficient reflects the proportional relationship between the area of ​​the current state and the area of ​​other process states. Specifically as follows:

[0136] The area of ​​the figure enclosed by the body surface sensors is determined according to the body surface data of the training object, and the area includes the first area, the second area and the third area. The first area includes the area of ​​the figure enclosed by the body surface sensors in the first process state, the second area includes the area of ​​the figure enclosed by the body surface sensors in the second process state, and the third area is the area S of the figure enclosed by the body surface sensors in the current state. The first process state is a state corresponding to when the anatomical structure is relatively stretched, that is, a stretching process state; the second process state is a state corresponding to when the anatomical structure is relatively contracted, that is, a contracting process state; for example, for the lungs, when the first process state is a fully inhaled state, the second area is the maximum area S enclosed by the body surface sensors. max When the second process state is the complete exhalation state, the second area is the minimum area S enclosed by the body surface sensor min .

[0137] Take the body surface sensors of the training subjects as an example to determine the motion coefficient in the current state. The body surface sensors of the three training subjects are located at P1 (X1, Y1, Z1), P2 (X2, Y2, Z2), and P1 (X3, Y3, Z3), respectively. Calculate the distance between the body surface sensors, that is, the side lengths A, B, and C.

[0138] Use Euclidean distance to calculate the lengths of A, B, and C. For example, to calculate the length of side A:

[0139]

[0140] The length calculation method of B and C is the same as that of A, which will not be repeated here.

[0141] Calculate the semi-perimeter P of the figure enclosed by the training subject's body surface sensors:

[0142] P=(A+B+C) / 2

[0143] The area S of the figure enclosed by the body surface sensors in the current state:

[0144]

[0145] According to the maximum area S max , minimum area S min , and the area S of the figure enclosed by the body surface sensors of the training object in the current state determine the motion coefficient F in the current state:

[0146]

[0147] Wherein, Fmax represents the maximum value of the designed motion coefficient, and Fmin represents the minimum value of the designed motion coefficient. For example, in some embodiments, if the design formula is based on the range of the motion coefficient in the interval [-1,1], the motion coefficient F is:

[0148]

[0149] For example, in some embodiments, if the formula is designed according to the range of the motion coefficient in the interval [-100, 100], the motion coefficient F is:

[0150]

[0151] The motion coefficient is used as the breathing coefficient, and the breathing coefficient is designed in the range of [-1,1]. The horizontal axis represents the number of EM data obtained by the training subject's body surface sensor in real time as the breathing duration or breathing cycle changes. The vertical axis represents from top to bottom: the area of ​​the figure enclosed by the training subject's body surface sensors, the area slope, and the breathing coefficient.

[0152] The embodiment of the present application uses the breathing coefficient F to intuitively and quickly determine the breathing state of the training object at this time, and enables or prohibits the movement of the catheter in the anatomical structure based on the breathing state. For example, if the breathing coefficient F is a positive value, it means that the human chest cavity is in the inhalation state under the first process state, the chest cavity is full and the bronchi are in a stretched state, the larger the value, the fuller it is, and the movement of the catheter in the anatomical structure is enabled; or, when the breathing coefficient F is a negative value, the breathing state of the training object is determined to be an exhalation state, and the movement of the catheter in the anatomical structure is prohibited. For another example, the breathing state of the training object is determined based on the breathing coefficient F, and a prompt is generated based on the breathing state to indicate whether the movement of the catheter in the anatomical structure is allowed. When the breathing coefficient F is a positive value, the breathing state of the training object is determined to be an inhalation state, and a prompt is generated to allow the movement of the catheter in the anatomical structure; or, when the breathing coefficient F is a negative value, the breathing state of the training object is determined to be an exhalation state, and a prompt is generated to prohibit the movement of the catheter in the anatomical structure. For another example, the breathing state of the training object is determined based on the breathing coefficient F, and a prompt is generated based on the breathing state to indicate whether the movement of the catheter in the anatomical structure is allowed. When the breathing coefficient F is a positive value, the breathing state of the training subject is determined to be an inhalation state, and a prompt is generated to allow the movement of the catheter in the anatomical structure; or, when the breathing coefficient F is a negative value, the breathing state of the training subject is determined to be an exhalation state, and a prompt is generated to prohibit the movement of the catheter in the anatomical structure. For example, when the anatomical structure is a bronchus, the visualization system assisting the doctor in the operation can detect that the surgical instrument (e.g., a needle) at the end of the catheter reaches the vicinity of the target tissue position and detects that the training subject is in an inhalation state. The system allows the doctor to perform a percutaneous puncture operation, and it is safer to perform the puncture in the inhalation state of the training subject; if the breathing coefficient is a negative value, it means that the human chest cavity is in the exhalation state under the second process state, and the chest cavity and bronchi are in a contracted state. The system allows the doctor to prohibit the operation.

[0153] Then, based on the vital sign data of the anatomical structure of the training subject, a process state represented by the first medical image is determined.

[0154] (3) Repeat the above steps until the number of first medical images representing multiple different process states meets a preset value.

[0155] The more training samples there are, the better the training effect is. Therefore, training can be performed when there are more medical images that meet the preset conditions. The embodiment of the present application determines the process state of the first medical image of the training object, and then determines that the number of first medical images with different process states meets the preset value, and then performs training based on multiple first medical images, thereby obtaining a better recognition network.

[0156] Optionally, the embodiment of the present application may also acquire multiple first medical images for training by controlling the time or frequency of acquiring the first medical images, specifically including:

[0157] Acquiring medical images of different process states that meet preset values ​​is intended to obtain medical images under different process states. For example, multiple medical images are acquired within a cycle at a certain frequency. If the frequency is high enough, multiple medical images under different process states can be obtained and used for training. For example, in some embodiments, at least for this type of anatomical structure, the catheter and / or instrument can be manipulated to stay at a characteristic point in the anatomical structure for a certain period of time to continuously acquire its position. In some embodiments, the duration may include at least one cycle of the intrinsic movement of the anatomical structure. For the bronchi, at least one respiratory cycle, such as one, two or more, is usually 3 to 5 seconds for a normal person. For another example, for (cardio)vascular vessels, the duration may include at least one heartbeat cycle, such as one, two or more, is usually 0.5 to 1 second for a normal person. The longer the duration of acquiring medical images or the more cycles included, the more sampling points will be obtained during such a duration, and individual differences caused by different cycles can be avoided.

[0158] Optionally, for example, for the respiratory movement of the bronchi, the magnitude of the deformation of the bronchi caused by the respiratory movement is different in different process states, that is, the magnitude of the deformation caused by the movement is different in different time periods (different moments) within the cycle. For example, for the respiratory cycle, for example, taking the pipe as an example of the bronchi, when the first process state is the patient's full inhalation state, when taking the first medical CT image in the inhalation phase, the patient needs to take a deep breath and then hold his breath until the CT shooting is completed. Since the bronchi of the human body are the largest in the inhalation state, the fine branches at the end of the bronchus are easier to image on CT. Therefore, a time period that is less affected by deformation in the cycle is selected, and medical images are obtained for training, so that a recognition network with better recognition effect can be obtained. That is, in a preset time period, such as a time period that is less affected by deformation, medical images are obtained for network training. The time period that is less affected by deformation can be a time period of the stretching process state, such as a time period in the inhalation state. Among them, the time period that is less affected by deformation can also be called a process state that is less affected by deformation.

[0159] Furthermore, medical images can also be acquired at a non-fixed frequency, that is, medical images are acquired at a higher frequency in the aforementioned preset time period, and medical images are acquired at a lower frequency in a non-preset time period. For example, medical images are acquired at a higher frequency in the aforementioned stretching process state, and medical images are acquired at a lower frequency in the contraction process state, so that the first medical image includes both medical images in the contraction process state and medical images in the stretching process state, and there are more medical images in the stretching process state, so that a plurality of representative first medical images that are less affected by deformation are prepared for training, thereby providing recognition capabilities of the trained recognition network.

[0160] The navigation method provided in the embodiment of the present application, after obtaining the identified network, further includes:

[0161] Step S30: In response to a user's navigation instruction, using the recognition network to identify a first pipeline segment from an intraoperative medical image of the patient acquired by the imaging device.

[0162] Intraoperative medical images refer to images obtained by imaging devices in scenarios that require navigation, including inspection or surgical treatment processes, such as bronchoscopic biopsy or cancer resection. The imaging device can be a medical endoscope, such as a camera in a bronchoscope.

[0163] The recognition network extracts features that can distinguish each pipeline segment in the intraoperative medical image. For example, when the pipeline segment is a lung segment, in particular, the features of the lung segment opening, such as geometric shape, texture, and color, are extracted. According to the features of the lung segment opening and the feature set in the recognition network, one or more lung segments in the intraoperative medical image are identified and output, and the identified lung segment is referred to as the first lung segment. Figure 4 The figure shows the first lung segment in the navigation method of one embodiment of the present application, where the left main bronchi 201 and the right main bronchi 301 are both identified first lung segments. The number of the first lung segments may be 0, 1 or more. The feature set refers to multiple feature parameters of the recognition network, which can be determined through network training.

[0164] Optionally, after identifying and outputting one or more first pipeline segments in the intraoperative medical image in step S20, the following may also be included:

[0165] Obtain identifiers of one or more of the first pipeline segments and multiple of the second pipeline segments. For example, when the pipeline segment is a lung segment, the identifier is used to distinguish each lung segment, and the identifier includes the lung segment position, name and / or identification number of the lung segment. The lung segment position can indicate the spatial position of the lung segment in the lung, and the identification number can indicate the sequence number of the lung segment in all lung segments. After identifying the first pipeline segment of the intraoperative medical image, the recognition network can output the identifier corresponding to the first pipeline segment.

[0166] The embodiment of the present application can uniquely determine the first pipeline segment in the intraoperative medical image by identifying the first pipeline segment and obtaining the corresponding identifier.

[0167] Step S40: Obtaining a planned path from the initial position to the target position, wherein the planned path is composed of a plurality of connected second pipeline segments. Taking the pipeline segment as a lung segment as an example, the specific steps include:

[0168] The first step is to obtain a preoperative image of the patient's anatomical structure.

[0169] The second step: determining the patient's duct segmentation according to the preoperative image of the patient's anatomical structure based on a depth-first algorithm or a breadth-first algorithm.

[0170] At the same time, in order to identify different pipeline sections, an identifier is recorded for each pipeline section, and the identifier may include the pipeline section position, pipeline section name, etc. of the pipeline section.

[0171] Taking the segmentation of the pipeline into lung segments as an example, the identification of each lung segment is traversed using a depth-first algorithm or a breadth-first algorithm, so as to assign a unique identification number to each lung segment. In the embodiment of the present application, the identification number is called AirwayID. At the same time, the identification number can also be used as an identification. The identification number can correspond one-to-one with the lung segment name and the lung segment position, and both are used to uniquely identify each lung segment. The one-to-one correspondence between the identification number and the lung segment name and the lung segment position can be stored. The storage form is not limited, as long as any one of them is known in the subsequent use process, the other can be found.

[0172] Step 3: According to the initial position and the target position, the second pipeline segment included in the planned path is determined from the pipeline segments of the patient. Specifically, the second pipeline segment is determined according to all pipeline segments of the patient and the target position, wherein the second pipeline segment is used to form the planned path. The initial position can be understood as a starting point, or a starting position that needs to be navigated.

[0173] The target position can be the lesion position, or it can be a position related to the medical operation customized by the doctor. When the patient's pipeline segmentation, initial position and target position are known, a planning path to reach the target position can be planned. In the embodiment of the present application, the pipeline segment included in the planning path is called the second pipeline segment. Multiple second pipeline segments are connected to each other. The embodiment of the present application does not specifically limit the method for planning the path, and the shortest distance planning method or the shortest time planning method can be used. The planned path can be represented by the identification of the lung segment, and can optionally be represented by an identification number, a lung segment name, etc. For example, the planned path is 3-3 to 3-7 to 4-3 to 5-5 to 6-4, where 3-3, 3-7, 4-3, 5-5 and 6-4 are the identification numbers of the second lung segments. Figure 5 The figure shows a schematic diagram of the planned path in the navigation method of an embodiment of the present application, in which the names of the lung segments in the planned path are identified: right main bronchi 501 (right main bronchi) to right intermediate bronchus 502 (intermediate bronchus) to right lower lobe bronchus 503 (inferior lobar bronchus) to right lower lobe outer base segment 504 (B9) to right lower lobe outer base segment subsegment 505 (B9b). The lung segments are connected, but for the sake of distinction, they are not fully connected in the figure.

[0174] Path planning is usually done before surgery, but if the system has sufficient computing power, it can also be done during surgery. For example, at the beginning of surgery, the pipeline network is relatively thick and the user does not need navigation. As the pipeline network gradually becomes thinner, the user needs navigation. The system can plan the path based on the real-time position and the target position at this time. The real-time position at this time is equivalent to the initial position. If the user needs to change the target position during surgery, the path planning can be performed based on the initial position and the updated target position.

[0175] Step S50: Based on the first pipeline segment, matching a target pipeline segment from the second pipeline segment;

[0176] The target pipeline segment is used to represent the pipeline segment included in the intraoperative medical image and the planned path. For example, the target pipeline segment is represented by the intersection of the first pipeline segment identified in the intraoperative medical image and the second pipeline segment in the planned path.

[0177] Optionally, since the identifier can uniquely identify a lung segment, the processor is further configured to match and obtain one or more target lung segments based on the identifiers of one or more of the first lung segments and the identifiers of multiple of the second lung segments.

[0178] When the first pipeline segment can be identified in step S30, but the target pipeline segment cannot be matched in step S50, there may be two reasons for this: the first pipeline segment identified and output by the recognition network is wrong or has reached the wrong position. In order to eliminate the factor of the first pipeline segment error, taking the pipeline segment as a lung segment as an example, the embodiment of the present application also includes:

[0179] Verify whether the first lung segment is correct based on the tree structure of the bronchial tree.

[0180] Since the bronchial tree is a tree structure, it repeatedly branches step by step, becoming finer and finer, like a tree branch. It can be understood that the finer branches are child nodes of the thicker branches, and the thicker branches are parent nodes of the thinner branches. The correct navigation path can go from the parent node to the child node, but the reverse is wrong. Therefore, by identifying whether the first lung segment conforms to the relationship between the parent node and the child node, it can be determined whether the first lung segment is correct. For example, according to the intraoperative medical image of the current frame, the 5-6 lung segment is identified, and the intraoperative medical image of the previous frame identifies the 6-2 lung segment, which obviously does not conform to the tree structure, so the first lung segment is wrong.

[0181] Optionally, verification of whether the first lung segment is correct may be performed after the first lung segment can be identified in step S20, or after a target lung segment cannot be matched in step S40.

[0182] The embodiment of the present application verifies whether the first lung segment is correct based on the tree structure of the bronchial tree, thereby eliminating the problem of incorrect identification of the first lung segment, which is more conducive to successfully completing the path navigation.

[0183] Step S60: Generate a navigation mark to mark the target pipeline segment. The navigation mark can be generated in the intraoperative medical image, or in the anatomical model of the patient. The navigation mark is used to mark the target pipeline segment to distinguish and identify it from other pipeline segments in the medical image, thereby realizing the function of navigation according to the target pipeline segment. The navigation mark can include any one or a combination of graphics, curves, highlights, colors and texts, as long as it can be distinguished from other pipeline segments in the medical image. Figure 6 It is a schematic diagram of the target lung segment, wherein the right main bronchus 301 is a lung segment included in both the first lung segment and the second lung segment, and the box diagram is a navigation mark.

[0184] Step S70: Displaying the navigation mark and the target pipeline segment on the user interface to guide the movement of the catheter and / or instrument.

[0185] When there is no target pipeline segment, it indicates that the intraoperative medical image does not contain the next moving target, and the catheter and / or instrument can continue to execute; when a target pipeline segment is included, the target pipeline segment is the next moving target, and the catheter and / or instrument is guided to move toward the target pipeline segment; when multiple target pipeline segments are included, one of the target pipeline segments can be selected as the next moving target according to the selection strategy, and the catheter and / or instrument is guided to move toward the selected target pipeline segment. It can be understood that the target pipeline segments identified and selected according to each intraoperative medical image are all part of the navigation path, and they together constitute the entire navigation path. Among them, the selection strategy may include selecting a target pipeline segment with a high probability of correct identification, or including selecting a target pipeline segment with a preceding identification number.

[0186] The embodiment of the present application identifies a first pipeline segment from an intraoperative medical image of the patient acquired by the imaging device, matches a target pipeline segment from a plurality of second pipeline segments included in a planned path based on the first pipeline segment, and then forms a navigation path based on the target pipeline segment to guide the catheter and / or instrument to move toward the target pipeline segment, thereby achieving fast and accurate navigation.

[0187] In one embodiment, in the navigation system provided by the embodiment of the present application, the processor is further configured to pre-train the recognition network, mainly including two situations:

[0188] (I) Training based on preoperative images

[0189] (II) Training based on intraoperative medical images

[0190] Let’s take the lungs as an example.

[0191] For the first case, the preoperative lung images of the training object can be annotated first to obtain the annotation information. In the embodiment of the present application, the object undergoing medical operation is referred to as a patient, for example, the object to be operated on can be a patient, and the object to be bronchoscopically examined can also be a patient. Medical operation can include the aforementioned examination or surgical treatment, etc. Optionally, before performing a medical operation on the patient, a preoperative image is usually taken first, so the preoperative image can be used for training. Optionally, in the embodiment of the present application, the training object can be the patient himself or not the patient himself, for example, the patient to be operated on is A, and the training object can be A, B, C and / or D. The preoperative image can include such as computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound, etc., which can be presented as a two-dimensional, three-dimensional or four-dimensional (such as time-based or rate-based information) image. A bronchial tree model can be established based on the preoperative image, and the slice image based on the bronchial tree model can reflect the actual basic geometric structure and texture characteristics at each corresponding real position. The slice images can be understood as images obtained by observing the bronchial tree model at various observation positions and angles using a virtual imaging device, such as a virtual endoscope. The number of slice images is sufficient for network training, and a portion of the slice images can be selected as a training set and a portion of the slice images can be selected as a test set.

[0192] Of course, sagittal images, coronal images or horizontal images based on the bronchial tree model may also be used, and these images may also reflect the actual basic geometric structure and texture characteristics at the corresponding real positions.

[0193] Among them, the preoperative image can be annotated by automatically identifying or manually selecting pipeline segments, such as lung segments. Since it is usually necessary to navigate whether to enter which lung segment before entering the lung segment, and just keep moving forward after entering a certain lung segment, the features of the lung segment mouth are particularly valuable for navigation. The lung segment mouth can be understood as the part of the lung segment that can be seen before entering the lung segment. Exemplarily, the lung segment mouth is automatically identified by using a preliminarily trained recognition network for annotation, or manually identified by manually identifying the lung segment mouth. The purpose of annotation is to form a learning sample for network training, so as to successfully complete the network training. By automatically or manually identifying the features in the preoperative image, the lung segment in the preoperative image is identified, and the identified lung segment is annotated, so as to establish an association between the features of the preoperative image and the lung segment. In order to characterize this association, it can include recording annotation information, and the annotation information includes the lung segment position, position information, circumscribed rectangle, lung segment name and / or identification number, etc. Among them, the annotation information and the aforementioned identification can include the same information, such as the lung segment name, identification number or lung segment position. Among them, the position information refers to the position of the lung segment in the preoperative image. Optionally, a certain part of the lung segment is selected to represent the position of the lung segment, for example, the position of the lung segment opening is used to represent the position of the lung segment. The lung segment opening is usually circular or elliptical. The circumscribed rectangle of the lung segment opening and the length and width of the circumscribed rectangle can be determined first, and a certain point in the circumscribed rectangle can be selected as the position of the lung segment, such as the upper left corner or upper right corner of the rectangle. For easy observation, the annotation information can also be displayed in the preoperative image, such as Figure 7 The preoperative image after annotation is shown, and the image includes the circumscribed rectangle of the lung segmental opening and the identification number. The preoperative image containing the annotation information can be displayed on a display device, and the display device can be set on a surgical robot or other location.

[0194] Then, the recognition network is trained based on the preoperative images and annotation information. In the model training stage, the training images and annotation information are input into the recognition network, and the recognition network extracts the features of each lung segment, especially the features of the lung segmental opening, through convolution operations. It can be understood that network training is a gradual process. Based on the initial network architecture, the initial network is trained using preoperative images and annotation information, and continuously improved, so as to obtain a recognition network that can ultimately achieve the expected recognition effect.

[0195] Optionally, in order to increase the learning sample and improve the effect of network training, the preoperative image is enhanced in the embodiment of the present application, wherein the enhancement may include rotation, cropping, translation or scaling, etc. When the original preoperative image has been labeled, the enhanced preoperative image also has the corresponding annotation information, so it can also be used as a learning sample for network training. It is only necessary to train based on the enhanced preoperative image and the annotation information to obtain the recognition network. Of course, it is also possible to train based on the medical image, the enhanced medical image and the annotation information to obtain the recognition network. Among them, the enhancement of the preoperative image may also include:

[0196] determining the first scaling ratio;

[0197] The first medical image is scaled based on the first scaling ratio to obtain a scaled first medical image.

[0198] Determining the first scaling ratio may specifically include:

[0199] A second medical image of one of the stretching process states of the training object is obtained, and a first anatomical model of the training object is generated based on the second medical image; a third medical image of one of the contraction process states of the training object is obtained, and a second anatomical model of the training object is generated based on the third medical image. The purpose of the anatomical model here is to determine the scaling ratio, so the first medical image of the training object can be used when training the recognition network, for example, a medical image of the stretching process state is selected from the first medical image to establish a first anatomical model, and a medical image of the contraction process state is selected from the first medical image to establish a second anatomical model; other medical images can also be obtained separately, and the embodiments of the present application are not limited to this. A stretching process state can be optionally a maximum stretching state, and a contraction process state can be optionally a minimum contraction state.

[0200] Determining a maximum shrinkage ratio ShrinkMax of the first anatomical model and the second anatomical model specifically includes:

[0201] S131: Acquire the first geodesic distance of the first anatomical model and the second geodesic distance of the second anatomical model.

[0202] In this embodiment, when the anatomical structure is a bronchus (usually referring to the bronchus of the lung), the first geodesic distance (Geodesic Distance) from the main carina point to the end point of the lower lobe in the first anatomical model and the second geodesic distance from the main carina point to the end point of the lower lobe in the second anatomical model are calculated respectively. The main carina point refers to the bifurcation point of the left and right bronchi, and the end point of the lower lobe refers to the point in the bronchial model that is closest to the human diaphragm. In some embodiments, when the first anatomical model is a three-dimensional mesh model, the geodesic distance is the shortest path distance from the main carina point to the end point of the lower lobe along the mesh surface in the first anatomical model; in another embodiment, when the first anatomical model is a three-dimensional point cloud model, it is necessary to use all points in the three-dimensional point cloud to construct a surface structure similar to a mesh, and then calculate the geodesic distance by finding the shortest path from the main carina point to the end point of the lower lobe in the graph. The first anatomical model or the second anatomical model contains target tissue, such as Figure 7 As shown, the geodesic line 10 formed from the main carina point to the end point of the lower lobe in the first anatomical model is a line on the grid surface. The length of the geodesic line 10 is the geodesic distance. The target tissue 20 (such as a lesion) is exactly at the end point of the lower lobe.

[0203] S132: Determine a maximum shrinkage ratio of the first anatomical model or the second anatomical model according to the first geodesic distance and the second geodesic distance.

[0204] In some embodiments, for example, the first geodesic distance GDIn of the first anatomical model obtained before surgery, GDIn is 166 mm; the second geodesic distance GDEx of the second anatomical model obtained before surgery, GDEx is 127 mm; then the maximum shrinkage ratio of the first anatomical model or the second anatomical model is determined according to the ratio of the first geodesic distance to the second geodesic distance: ShrinkMax = GDEx / GDIn, at this time, ShrinkMax is approximately 0.76.

[0205] Then, the first scaling ratio is determined according to the maximum shrinkage ratio ShrinkMax, the first scaling ratio = K1 * ShrinkMax, K1 is a coefficient, and takes a natural number. For example, ShrinkMax is 0.76, k1 is 1.1, and the first scaling ratio = 0.76 * 1.1 = 0.83. Among them, the specific value of K1 can be flexibly determined based on experience. Optionally, multiple different coefficients K1 can be determined to obtain multiple first scaling ratios, and multiple different first scaling ratios can be used to scale the same first medical image, so as to obtain more training samples.

[0206] In the embodiment of the present application, the first medical image of the training object is scaled by using a scaling ratio, and then training is performed based on the multiple first medical images and the scaled first medical image to obtain the recognition network, thereby improving the recognition ability of the recognition network.

[0207] Optionally, the navigation method provided in the embodiment of the present application further includes:

[0208] Input the preoperative image of the training object into the feature transfer network to obtain the preoperative image after style transfer;

[0209] Annotate the preoperative image after style transfer to obtain annotation information;

[0210] The recognition network is obtained by training based on the pre-operative images and annotation information after style transfer.

[0211] Among them, the feature transfer network is used to transfer the style of the preoperative image, that is, to render the preoperative image into the style of the intraoperative medical image, that is, to transfer the style of the virtual endoscopic image domain to the real endoscopic image domain, including two stages such as style transfer and geometric shape smoothing. In the style transfer stage, it is necessary to retain the geometric shape information to the greatest extent. The basic principles in the geometric shape smoothing stage include having the same image style around the pixels of the same geometric shape; limiting the global geometric feature offset of the generated image as much as possible. The style difference between the preoperative image and the intraoperative medical image may include color, local brightness or contrast. The feature transfer convolutional network renders the preoperative image according to the style effect of the image device, and does not change the geometric shape structure and texture of the preoperative image to the greatest extent. The feature transfer network can adopt a GAN (Generative Adversatial Net) network, including a generation function and a discriminant function, and the feature transfer network can adopt a feature transfer network in the prior art, which is not specifically limited in the embodiment of the present application. For example, the preoperative image is black and white, and the preoperative image after style transfer is color, or the preoperative image is color, and the preoperative image after style transfer is black and white.

[0212] The implementation principle of labeling the preoperative image after style transfer is the same as the labeling of the preoperative image by automatic recognition or manual selection of the lung segment orifice mentioned above. For the sake of brevity of the whole article, it will not be repeated here.

[0213] In the embodiment of the present application, a pre-trained recognition network is obtained by inputting a pre-operative image of a training object into a feature transfer network for style transfer and then labeling, and then training is performed based on the image after style transfer, thereby obtaining a pre-trained recognition network.

[0214] Optionally, the navigation method provided in the embodiment of the present application further includes:

[0215] Annotating the preoperative images of the training object to obtain annotation information;

[0216] Input the preoperative image into the feature transfer network to obtain the preoperative image after style transfer;

[0217] The recognition network is obtained by training based on the pre-operative images and annotation information after style transfer.

[0218] In the embodiment of the present application, pre-operative images of the training object are used for annotation and then input into a feature transfer network for style transfer, and then network training is performed based on the pre-operative images after style transfer, thereby obtaining a pre-trained recognition network.

[0219] For the second case, it is optional to first obtain the intraoperative medical image of the training object, annotate the intraoperative medical image of the training object, and obtain the annotation information. The training object includes non-patients themselves. In medical practice activities, in certain scenarios, intraoperative medical images of the training object can be obtained. For example, during the operation of the training object, the doctor can obtain the intraoperative medical image of the training object and accumulate it for subsequent network training. In this case, since the network training input is the intraoperative medical image, it is closer to the intraoperative medical image to be identified in the actual application, so the recognition efficiency is high, but a large number of intraoperative medical images need to be collected for network training to improve the generalization recognition ability of the network; therefore, only relying on obtaining the intraoperative medical image of the patient himself for network training, the number of intraoperative medical images is difficult to quickly meet the requirements, and the operation time is urgent, and there is usually not enough time for network training. Among them, the annotation of the intraoperative medical image can be performed by automatically identifying the preoperative image as before or manually selecting the lung segment orifice. The implementation principle is the same. For the sake of brevity of the whole text, it will not be repeated here.

[0220] Then, the recognition network is trained based on the intraoperative medical images and annotation information. In the model training stage, the intraoperative medical images and annotation information are input into the recognition network, and the recognition network extracts the features of each duct segment through convolution operation, such as the features of the lung segment opening, to complete the recognition of the lung segment.

[0221] The embodiment of the present application performs network training by using intraoperative medical images of training subjects and corresponding annotation information, thereby obtaining a pre-trained recognition network.

[0222] In one embodiment, the navigation method provided in the embodiment of the present application further includes: in response to the navigation instruction of the user, using the recognition network to identify the first pipeline segment from the intraoperative medical image of the patient acquired by the imaging device, including:

[0223] The specific principle of determining the maximum shrinkage ratio is the same as that of determining the maximum shrinkage ratio ShrinkMax mentioned above, and will not be repeated here for the sake of the brief introduction of the full text. Optionally, the maximum shrinkage ratio can be determined based on an anatomical model of the patient himself or a non-patient himself.

[0224] The second scaling ratio is determined according to the maximum shrinkage ratio. The specific principle is the same as that of determining the first scaling ratio, and will not be described here for the sake of simplicity.

[0225] Based on the second scaling ratio, the intraoperative medical image is scaled; optionally, a plurality of different second scaling ratios may be included, and then the same intraoperative image is scaled using the plurality of different scaling ratios, thereby obtaining intraoperative images with different scaling degrees.

[0226] The recognition network is used to identify a first pipeline segment from the scaled intraoperative medical image. When multiple scaled intraoperative images are included, the pipeline segment can be first identified from each scaled intraoperative medical image, and then the recognition results of each scaled intraoperative medical image are integrated, wherein the integration strategy can be flexibly determined, for example, the recognition results of each scaled intraoperative medical image are combined.

[0227] The embodiment of the present application uses the maximum shrinkage ratio to determine the second shrinkage ratio, thereby achieving scaling of the intraoperative image, and uses a recognition network to identify the first pipeline segment from the scaled intraoperative medical image to achieve navigation, thereby improving the recognition accuracy of the recognition network.

[0228] Optionally, determining the second scaling ratio according to the maximum shrinkage ratio may further include:

[0229] Acquire the intraoperative medical image and the body surface data of the patient sensed by the body surface sensor at the same time;

[0230] Determine vital sign data of the anatomical structure of the patient in a current state represented by the intraoperative medical image, wherein the vital sign data represents data of the anatomical structure of the patient in different process states;

[0231] The second scaling ratio is determined according to the vital sign data and the maximum contraction ratio.

[0232] Determining the vital sign data of the patient's anatomical structure in the current state represented by the intraoperative medical image can be achieved by the following steps:

[0233] (1) The patient's body surface is provided with a body surface sensor. The location of the body surface sensor is the same as the setting principle of the body surface sensor of the aforementioned training subject, and will not be repeated here.

[0234] (2) Determining the vital sign data of the patient's anatomical structure based on the body surface data, the vital sign data representing the data of the patient's anatomical structure in different process states, and the vital sign data including the area of ​​a figure enclosed by the body surface sensors, the area slope, and the motion coefficient. For example, determining the area of ​​a figure enclosed by the body surface sensors based on the patient's body surface data, the area including the first area, the second area, and the third area. The first area includes the area of ​​the figure enclosed by the body surface sensors in a stretch process state of the patient, the second area includes the area of ​​the figure enclosed by the body surface sensors in a contraction process state of the patient, and the third area is the area S of the figure enclosed by the body surface sensors in the patient's current state. For example, for the lungs, when a stretch process state may be a fully inhaled state, the second area is the maximum area S enclosed by the surface sensors. max When a contraction process state is a complete exhalation state, the second area is the minimum area S enclosed by the body surface sensor min .

[0235] like Figure 8a As shown, taking three body surface sensors as an example, the motion coefficient of the patient in the current state is determined. The three body surface sensors of the patient are located at P1 (X1, Y1, Z1), P2 (X2, Y2, Z2), and P1 (X3, Y3, Z3), respectively, and the distance between the body surface sensors is calculated, that is, the side lengths A, B, and C.

[0236] Use Euclidean distance to calculate the lengths of A, B, and C. For example, to calculate the length of side A:

[0237]

[0238] The length calculation method of B and C is the same as that of A, which will not be repeated here.

[0239] Calculate the semi-circumference P of the figure enclosed by the patient's body surface sensors:

[0240] P=(A+B+C) / 2

[0241] The area S of the figure enclosed by the body surface sensors in the patient's current state:

[0242]

[0243] According to the maximum area S max , minimum area S min , and the area S of the figure enclosed by the body surface sensors in the current state determine the patient's motion coefficient F in the current state:

[0244]

[0245] Wherein, Fmax represents the maximum value of the designed motion coefficient, and Fmin represents the minimum value of the designed motion coefficient. For example, in some embodiments, if the design formula is based on the range of the motion coefficient in the interval [-1,1], the motion coefficient F is:

[0246]

[0247] For example, in some embodiments, if the formula is designed according to the range of the motion coefficient in the interval [-100, 100], the motion coefficient F is:

[0248]

[0249] like Figure 8b As shown in the figure, the motion coefficient is used as the respiratory coefficient, and the respiratory coefficient is designed in the range of [-1,1]. The horizontal axis represents the number of EM data acquired by the surface sensor in real time as the breathing duration or breathing cycle changes. The vertical axis represents from top to bottom: the area of ​​the figure enclosed by the surface sensor, the area slope, and the respiratory coefficient. The area of ​​the figure enclosed by the surface sensor represents the area size of the figure enclosed by the surface sensor as the patient's breathing state changes; the area slope represents the rate of change of the area; and the respiratory coefficient represents the different breathing states of the patient.

[0250] Optionally, the second scaling ratio Shrink is determined according to the motion coefficient F and the maximum shrinkage ratio ShrinkMax of the model, and the calculation formula is:

[0251]

[0252] Optionally, after determining the second scaling ratio, it can be floated within a certain range, and a plurality of different scaling ratios can be included in the floating range, and then the same intraoperative image is scaled using the plurality of different scaling ratios, thereby obtaining intraoperative images with different scaling degrees. Optionally, a vital sign curve of the anatomical structure of the patient can be generated based on the vital sign data, and the vital sign curve can be displayed on the user interface, wherein the vital sign curve includes a respiratory curve and / or a heartbeat curve.

[0253] The embodiment of the present application determines the motion coefficient by utilizing the surface data sensed by the patient's surface sensor, and then determines the second contraction ratio based on the maximum contraction ratio and the motion coefficient to scale the intraoperative image, and utilizes the recognition network to identify the first pipeline segment from the scaled intraoperative medical image to achieve navigation, thereby improving the recognition accuracy of the recognition network.

[0254] In one embodiment, in the navigation method provided in the embodiment of the present application, the catheter and / or the instrument is provided with a tracking sensor for sensing the position of the catheter and / or the instrument, and the tracking sensor is used to obtain the actual path point of the catheter and / or the instrument. In response to the navigation instruction of the user, the first pipeline segment is identified from the intraoperative medical image of the patient acquired by the imaging device using the recognition network, including:

[0255] (I) Correcting the actual path points to obtain corrected path points.

[0256] The data sensed by the tracking sensor corresponds to the position of the actual path point, and the data sensed by the sensor is corrected to obtain the corrected path point corresponding to the corrected sensor data. There are two methods for correcting the actual path point, which will be described in detail below.

[0257] (II) Determine the positioning correction amount based on the actual path point and the corrected path point, that is, determine the positioning correction amount based on the sensor data and the corrected sensor data. The difference between the sensor data and the corrected sensor data reflects the effect of the motion on the sensor, so the effect of the motion can be removed by removing the difference. The difference can be recorded as the positioning correction amount.

[0258] (III) Determine the relative position of the pipeline segments in the intraoperative medical image, wherein the relative position may include the distance and direction between the pipeline segments. The relative position of each pipeline segment in the medical image may be determined using existing technology, and the embodiments of the present application do not impose any limitation on this.

[0259] (iv) identifying a third pipeline segment in the intraoperative medical image based on the first pipeline segment, the positioning correction amount, and the relative position of the pipeline segment in the intraoperative medical image, wherein the third pipeline segment is different from the first pipeline segment.

[0260] When an intraoperative medical image includes multiple pipeline segments, the multiple pipeline segments may be displayed with different completeness, and thus have different probabilities of being recognized. In the embodiment of the present application, the pipeline segment that the recognition network can directly recognize from the intraoperative image is referred to as the first pipeline segment.

[0261] Combined with the positioning correction amount, the intraoperative medical image is corrected. If after the correction, more unrecognized pipeline segments in the intraoperative image are displayed, the relative position can be further combined, for example, the relative position is compared with the relative position of the pre-stored pipeline segments, and the pipeline segment with the closest relative position is found as the inferred and identified pipeline segment, and the inferred and identified pipeline segment is called the third pipeline segment; if after the correction, the unrecognized pipeline segment in the intraoperative image is displayed less, the unrecognized pipeline segment in the intraoperative image can be discarded. The third pipeline segment can be different from the first pipeline segment, and the difference can include completely different or partially different.

[0262] The correction of the intraoperative medical image in combination with the positioning correction amount may include performing a correction on the intraoperative medical image that is the same as or proportional to the positioning correction amount. For example, if the positioning correction amount is to move upward by 5 mm, the intraoperative medical image may be moved upward by 5 mm or 4.5 mm.

[0263] For example, Figure 9a FIG. 1 shows the lung segments that should be included in the intraoperative image when it is not affected by breathing, including lung segment 901, lung segment 902, and lung segment 903. Due to the influence of breathing, lung segment 903 is partially blocked due to displacement, as shown in FIG. Figure 9b As shown. The recognition network can identify lung segments 901 and 902 as the first pipeline lung segments. According to the difference between the actual path point and the corrected path point, the positioning correction amount can be determined. According to the intraoperative image, the relative positions of lung segments 901, 902, and 903 can be determined. According to the lung segments 901 and 902 identified by the recognition network, the positioning correction amount and the relative positions of lung segments 901, 902, and 903, lung segment 903 can be identified, and lung segment 903 is the inferred and identified third pipeline segment. Then, according to lung segments 901, 902, and 903, the target pipeline segment can be matched from the planned path, and then navigation can be performed based on the target pipeline segment.

[0264] The embodiment of the present application corrects the intraoperative image according to the positioning correction amount before identification, thereby accurately identifying the pipeline segments that cannot be fully displayed in the intraoperative image due to deformation, thereby improving navigation accuracy and efficiency.

[0265] In one embodiment, the catheter and / or the instrument is provided with a sensor for measuring the position of the catheter and / or the instrument, and the sensor is used to obtain the actual path point of the catheter and / or the instrument. The navigation method provided in the embodiment of the present application also includes:

[0266] Acquire a preoperative medical image of the patient, and generate an anatomical model of the patient based on the preoperative medical image, wherein the preoperative medical image represents a contraction process state or an extension process state of the anatomical structure of the patient;

[0267] Acquire a transformation matrix, where the transformation matrix is ​​used to represent the conversion relationship between the surgical environment coordinate system and the patient's anatomical model coordinate system;

[0268] Based on the actual path points and the transformation matrix, determining simulated path points of the catheter and / or instrument in the anatomical model corresponding to the actual path points;

[0269] marking the simulated waypoints in an anatomical model of the patient;

[0270] The anatomical model and the simulated waypoints are displayed on the user interface.

[0271] Among them, the simulated path point reflects the position of the catheter end / instrument end in the anatomical model. Combined with the display parameters of the surgical site, the position of the catheter end / instrument end in the displayed surgical site can be obtained, thereby integrating the displayed surgical site with the catheter and / or instrument therein to achieve intraoperative navigation. The surgical site can be displayed in the form of a preoperative image and / or an intraoperative medical image. If the preoperative image data is used to display the surgical site, since the anatomical model is obtained by processing the three-dimensional preoperative image, the position of the catheter end / instrument end in the surgical site can be obtained according to the coordinate transformation relationship of the displayed surgical site relative to the three-dimensional preoperative image. If the surgical site is displayed using intraoperative medical image data, it is necessary to align the intraoperative medical image with the three-dimensional preoperative image / anatomical model to obtain the coordinate transformation relationship of the displayed surgical site relative to the three-dimensional preoperative image, and then obtain the position of the catheter end / instrument end in the surgical site. The coordinate transformation relationship can be represented by a transformation matrix, which can be determined by the prior art, or can refer to 202310026323.7 Catheter robot and its alignment method, readable storage medium, and the embodiment of the present application does not impose any limitation on this.

[0272] Optionally, the actual path points are corrected to obtain corrected path points, and based on the corrected path points and the transformation matrix, simulated path points of the catheter and / or instrument corresponding to the actual path points in the anatomical model of the patient are determined; the simulated path points are then marked in the anatomical model of the patient; and the anatomical model and the simulated path points are displayed on the user interface.

[0273] The embodiment of the present application determines the simulated path points corresponding to the actual path points / corrected path points based on the actual path points / corrected path points and the coordinate conversion relationship, and displays the simulated path points on the user interface. The simulated path points can be displayed at a certain position of the anatomical model in a stable form without fluctuating and drifting everywhere, which makes it easier for the user to observe the current position of the catheter and / or instrument and facilitates navigation to the target position.

[0274] In one embodiment, in the navigation method provided in the embodiment of the present application, the catheter and / or the instrument is provided with a sensor for measuring the position of the catheter and / or the instrument, and the sensor is used to obtain the actual path point of the catheter and / or the instrument, and the actual path point is corrected to obtain the corrected path point, which specifically includes:

[0275] In response to the user's motion compensation instruction, based on the actual path point, a feature point matching the actual path point is determined, and the feature point is a point in the anatomical structure. Wherein, determining the feature point matching the actual path point may include: determining the feature point closest to the actual path point. Some types of anatomical structures such as bronchi, (cardio) blood vessels, etc., have high frequency and large amplitude of intrinsic motion such as respiratory motion and heartbeat motion, which easily cause large fluctuations in the position of the catheter and / or instrument at the feature point, and thus easily lead to problems inconsistent with preoperative medical images. Exemplarily, when the anatomical structure is a bronchus, the catheter and / or instrument are affected by different amplitudes of respiratory motion in different lung regions. For example, the amplitude of fluctuation caused by respiratory motion in the upper lobe region may be 10 mm, while the amplitude of fluctuation caused by respiratory motion in the lower lobe region may reach 20 mm. It can be seen that corresponding to the different positions of the catheter and / or instrument in the anatomical structure, the influence of the intrinsic motion of the anatomical structure on them is usually different. Since the feature points are associated with the sampling point subsets, different denoising weight matrices can be constructed for different feature points or for different sampling point subsets, so that the sampling point subset can be denoised later by the denoising weight matrix corresponding to the sampling point subset to eliminate the influence of the intrinsic motion of the anatomical structure on the sampling points as much as possible. The denoised subset can also be called a motion compensated subset.

[0276] In the embodiment of the present application, each selected feature point can be considered to represent an area of ​​the anatomical structure. At different positions in the area, it can be simply considered that the intrinsic movement of the area has basically the same effect on the position of the catheter and / or the instrument. That is, it is necessary to determine the feature point closest to each actual path point. For example, the Euclidean distance between each actual path point and the mean PosTube of the sampling points in the sampling point set corresponding to each feature point can be compared to obtain the feature point closest to it.

[0277] Obtain a denoising weight matrix associated with the feature point. In some embodiments, a KD search tree (KD-tree) can be constructed with the mean point PosTube of the sampling points in the sampling point subset corresponding to all feature points as a node for subsequent retrieval. The KD tree constructed here is a key-value search tree in a three-dimensional Euclidean space, which is used for range search and nearest neighbor search. By inputting the real-time position of the catheter and / or instrument (i.e., the real-time actual path point), the feature point corresponding to the nearest node can be quickly queried through the KD search tree, and then the denoising weight matrix corresponding to the nearest feature point can be obtained. By constructing a KD search tree, the process of obtaining the denoising weight matrix is ​​greatly accelerated.

[0278] Based on the denoising weight matrix, the actual path points are denoised to obtain denoised path points, and the denoised path points after denoising are used as the corrected path points. That is, the sensor data is denoised based on the denoising weight matrix to obtain denoised sensor data, and the denoised sensor data is used as the motion compensated sensor data, and the motion compensated sensor data corresponds to the corrected path points.

[0279] exist Fig.10 In the figure, the point cloud composed of hollow circle-shaped points represents the catheter movement path before denoising (i.e. before compensation), the point cloud composed of five-pointed star-shaped points represents the catheter movement path after denoising, and the continuous path in the middle is the path spliced ​​from multiple bronchial center lines. Fig.10 As shown, it can be seen that the denoised point cloud has a smaller deviation (i.e., better convergence) relative to the bronchial centerline, that is, the influence of respiratory movement is greatly weakened. Optionally, in correcting the actual path point to obtain the corrected path point, it can also specifically include:

[0280] Acquire movement data of the catheter in the pipeline network of the patient's anatomical structure, wherein the movement data of the catheter in the pipeline includes command data instructing the catheter tip / instrument tip to reach a specific anatomical site and / or change its orientation within the pipeline network, insertion data indicating the catheter tip / instrument tip, etc.

[0281] Based on the actual path point, the movement data of the catheter in the pipeline and the intraoperative medical image, the corrected path point is determined, and the position data corresponding to the corrected path point is referred to as the fused sensor data. The insertion data of the catheter end / instrument end may include the amount of catheter expansion and contraction. The amount of catheter expansion and contraction can reflect the depth of the catheter entering the human body and can be obtained from the robot's control command. Optionally, dynamic weights are set for the sensor data, the catheter control data and the intraoperative medical image, and the position data of the corrected path point is calculated based on the weight, and the position data is used as the fused sensor data. For example, the weight of the catheter control data can be higher at thinner or turning locations of the bronchus.

[0282] The embodiment of the present application corrects the actual path point to obtain the corrected path point, for example, the sensor data is processed to obtain the motion compensated sensor data or the fused sensor data, and then the simulated path point corresponding to the actual path point is determined according to the coordinate conversion relationship, and the simulated path point is displayed on the user interface. This can improve the stability of the data obtained by the sensor. For example, the data obtained by the sensor after denoising filters out the jitter caused by the intrinsic movement of the anatomical structure, and after conversion to the anatomical model coordinate system, it can be displayed in a certain position of the anatomical model in a stable form without fluctuating and drifting everywhere, which makes it more convenient for the user to observe the current position of the catheter and / or instrument and facilitates navigation to the target position.

[0283] In one embodiment, at least two of the virtual imaging device view, the anatomical model, the local view, the slice view, and the aiming view are scroll-displayed on the user interface, and the virtual imaging device view may include the navigation mark.

[0284] The user interface can use image data from imaging technologies such as computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound to present images of the surgical site recorded preoperatively or intraoperatively. Preoperative or intraoperative medical image data can be presented as two-dimensional, three-dimensional, or four-dimensional (such as time-based or rate-based information) images, and / or presented as images from models created based on preoperative or intraoperative medical image data sets. Preoperative images can be presented as slice views, anatomical model views, and local views. Local views are oblique views based on slice views. Slice views include axial, sagittal, and coronal views. Intraoperative medical images can be presented as intraoperative real views, which can also be called intraoperative real fields of view or camera views.

[0285] The user interface can also display a virtual navigation image. In the virtual navigation image, the actual position of the catheter assembly 400 is registered with the preoperative image to present a virtual image of the catheter assembly 400 in the surgical site to the operator from the outside, which can also be understood as a virtual representation of the end of the endoscope based on the sensor position in the anatomical model.

[0286] Furthermore, in the embodiment of the present application, determining the virtual image device view includes:

[0287] The endoscope is calibrated in advance, see 202310154245.9 Endoscope alignment method and device and calibration system for details, and the conversion matrix between the sensor and the endoscope lens, as well as the internal parameters of the lens can be obtained.

[0288] The transformation matrix between the sensor's coordinate system and the anatomical model's coordinate system is obtained in advance through registration.

[0289] According to the pose of the sensor in the sensor's coordinate system and the conversion matrix between the sensor and the endoscope lens, the pose of the endoscope lens in the sensor's coordinate system is obtained. Then, combined with the conversion matrix between the sensor's coordinate system and the anatomical model's coordinate system, the pose of the endoscope lens in the anatomical model's coordinate system is obtained, that is, the pose of the virtual endoscope lens in the anatomical model's coordinate system. Combined with the internal parameters of the endoscope lens and using the principle of camera imaging, the contents in the anatomical model are imaged in the virtual lens to obtain a virtual field of view, which can also be called a virtual imaging device view.

[0290] The method for determining the virtual image device view is not limited to the method in the embodiment of the present application, and may also be adopted in the prior art. The embodiment of the present application does not impose any limitation on this.

[0291] Optionally, a navigation marker is generated in the anatomical model and marked in the virtual imaging device view.

[0292] The user interface may display an aiming view. The aiming view is used to display a target, such as a nodule, and may include an orbital view and a head end view. The orbital view is used to allow alignment to be viewed from different angles around the nodule. The head end view is used to display the target position from the end of the endoscope.

[0293] According to the user's preference, the above-mentioned intraoperative real view, virtual imaging device view, anatomical model view, local view, slice view and aiming view can be flexibly arranged in the user interface, such as being displayed in different areas. The layout of the specific area can be adjusted by the user. Optionally, at least two of the above-mentioned views can also be displayed in a scrolling view, so that the user can see different views in time and make timely adjustments to the medical operation.

[0294] Optionally, in response to a user's vital sign curve display instruction, the vital sign curve can also be displayed on the user interface. The vital sign curve is used to characterize the degree of scaling of the pipeline network caused by intrinsic luck; the vital sign curve is determined based on the surface data sensed by the surface sensor that can be placed on the patient's body surface. In combination with 8b, for example, the vital sign curve can reflect the area, area slope, and respiratory coefficient of the figure enclosed by the aforementioned surface sensor. By observing the vital sign curve, the user can intuitively know the impact of exercise. The vital sign curve can be a breathing curve, a heartbeat curve, etc.

[0295] Optionally, based on the magnitude of the impact, it is determined whether it is necessary to automatically trigger a motion compensation instruction to enter a motion compensation mode. In the motion compensation mode, the following can be performed: (I) before the intraoperative medical image is input into a pre-trained recognition network, the intraoperative medical image is screened, and then the recognition network is used to identify the first pipeline segment from the screened intraoperative medical image; (II) the intraoperative medical image is scaled, and the recognition network is used to identify the first pipeline segment from the scaled intraoperative medical image; (III) the actual path point is corrected to obtain a corrected path point to remove the impact of breathing as much as possible; and (IV) the vital sign curve is displayed on the user interface. Optionally, at least one of (I), (II), (III) and (IV) can be performed. When scaling, it can be based on the aforementioned second scaling ratio.

[0296] Optionally, depending on whether the vital sign data is within a preset range, if it is within the preset range, it is considered that the influence of breathing is large and it is necessary to enter the motion compensation mode; otherwise, it is considered that the influence of movement is negligible and it is not necessary to enter the motion compensation mode.

[0297] Optionally, the user may trigger a motion compensation instruction to enter the motion compensation mode and execute at least one of (I), (II), (III) and (iv).

[0298] Optionally, after entering the motion compensation mode, which one or a combination of (Ⅰ)(Ⅱ)(Ⅲ) and (ⅳ) is executed can be based on a soft default setting, or based on further user commands, for example, the user can be allowed to select which one or a combination of (Ⅰ)(Ⅱ)(Ⅲ) and (ⅳ) on the user interface.

[0299] The application of the embodiments of the present application is as follows Fig.11 Schematic diagram of a user interface 800. The user interface 800 may display a camera view window 810, a virtual imaging device view 820, a global anatomical model view 830, an indicator window 840, and a control window 850.

[0300] The camera view window 810 displays camera data captured by an image device of the medical device. For example, the camera data may include a camera image field of view or camera video data captured by a stereo camera or a monoscopic camera mounted on an endoscope at the end of a catheter.

[0301] One or more indicator windows 840 can display the current operation prompt and screen recording button, and can also display the status of the catheter, such as displaying the inner bend angle or outer bend angle information of the catheter, and prompting the doctor to perform the surgical operation or pause the surgical operation.

[0302] The control window 850 may display the status of each device connection, such as the operating handle, catheter, magnetic navigation, suction, flushing, and locked or activated status.

[0303] In some embodiments, the virtual image device view 820 displays an image from a local anatomical model corresponding to the end of the catheter to simulate an endoscope's field of view image. When the anatomical model is a bronchus, the virtual visualization system can generate virtual bronchial model image data using, for example, a preoperative CT image. The virtual bronchial model image data can display the real-time position of the catheter inside the patient's bronchus.

[0304] In some embodiments, the anatomical model view 830 shows the user observing the complete anatomical model from a global perspective. For example, the user can observe the position of the catheter in the bronchial model from the overall bronchial model.

[0305] In one embodiment, in the navigation method provided by the embodiment of the present application, the denoising weight matrix corresponding to the sampling point subset obtained may include two sources.

[0306] The first type may be derived from the patient himself.

[0307] The second type may be derived from other patients with the same or similar physical features as the patient, and a certain deviation is allowed. For example, the allowable deviation is defined as within 5%. For patients with the same or similar physical features corresponding to the same feature points in the anatomical structure, the amplitude of the intrinsic movement of the anatomical structure is basically the same, and the influence on the position of the catheter and / or instrument is basically the same. Among them, the physical features include, for example, body shape features and / or physiological features. When the anatomical structure is a bronchus or a (cardio)vascular vessel, the body shape features at least include chest circumference; when the anatomical structure is a bronchus, the physiological features at least include the amplitude of respiratory movement; when the anatomical structure is a (cardio)vascular vessel, the physiological features at least include the amplitude of heartbeat movement.

[0308] In some embodiments, for the case where the denoising weight matrix is ​​derived from the second type, such as Fig.12 As shown, determining the denoising weight matrix may include:

[0309] Step S1201, obtaining the patient's physical characteristics.

[0310] Physical characteristics include, for example, body shape characteristics and / or physiological characteristics.

[0311] Step S1202, determining the type of the patient's anatomical structure.

[0312] The type of anatomical structure includes, for example, bronchial tubes or (cardio)vascular tubes, and of course, other organs. The type can be determined by manual input or automatically identified.

[0313] Step S1203, based on the patient's physical features and anatomical structure type, target patients with the same anatomical structure type and the same or similar physical features are matched.

[0314] Among them, one or more databases can be constructed, which store information of multiple patients who have been treated in this hospital or other hospitals. This information includes but is not limited to the patient's name, gender, age, physical characteristics, physiological characteristics of anatomical structures, and the same or different denoising weight matrices of different feature points of one or more anatomical structures. Therefore, it is advantageous to achieve accurate matching through the same or different denoising weight matrices of different feature points of physical characteristics, physiological characteristics, and one or more anatomical structures.

[0315] Step S1204: determining feature points associated with the patient's sampling point subset.

[0316] Step S1205: Based on the feature points, a denoising weight matrix of the target patient associated with the feature points is matched.

[0317] Among them, the feature points of the same anatomical structure of different patients are usually identical or highly similar when selected. Taking the bronchus as an example, these feature points are usually selected from the most easily recognizable bifurcation points of the bronchus at various levels, that is, the carina at various levels. Exemplarily, when the feature points of the sampling point subset correspond to the main carina, the denoising weight matrix matched in step S1204 should also be the denoising weight matrix of the main carina of the target patient; when the feature points of the sampling point subset correspond to the first-level carina of the left lung lobe, the denoising weight matrix matched in step S1204 should also be the denoising weight matrix of the first-level carina of the left lung lobe of the target patient.

[0318] The feature points may be selected from points in the anatomical structure that have obvious features and are easy to identify. For example, when the anatomical structure is a bronchus (usually referring to a bronchus in the lung), the feature points may be selected from various levels of carina, for example, the main carina, the first level carina of the left lobe and the second level carina of the right lobe, etc. may be selected.

[0319] When selecting feature points, it is possible to cover as many areas of the anatomical structure as possible. For example, when the anatomical structure is a bronchus of the lung, multiple feature points can be selected to cover as many bronchus of the lung as possible. For example, one or more feature points can be selected from one or more areas of the main carina, right upper lobe, right middle lobe, right lower lobe, left upper lobe, left middle lobe, and left lower lobe. Exemplarily, one feature point can be selected from each of the above areas to cover the entire bronchus, such as Fig.13 As shown, in the anatomical model, feature point 1 representing the main carina, feature point 2 representing the carina of the left upper lobe, feature point 3 representing the carina of the left middle lobe, feature point 4 representing the carina of the left lower lobe, feature point 5 representing the carina of the right upper lobe, feature point 6 representing the carina of the right middle lobe, and feature point 7 representing the carina of the right lower lobe are selected, a total of seven feature points.

[0320] Each sampling point subset may correspond to (match) a feature point of the anatomical structure, different sampling point subsets may correspond to different feature points of the anatomical structure, and the sampling point subsets and the simulated feature points are associated through the feature points. Each sampling point subset may include at least one sampling point, and the sampling point is generally described by the coordinates of the catheter end or the instrument end in the world coordinate system (surgical environment coordinate system or physical space coordinate system). The sampling point can be acquired by a tracking sensor, and the tracking sensor may include a position sensor and / or a shape sensor.

[0321] For example, the visualization system can be used to guide the end of the catheter inserted into the anatomical structure and / or the end of the instrument to move to the feature point, and the position of the catheter and / or the instrument can be obtained by setting a position sensor in the catheter and / or the instrument. Among them, a position sensor can be a component in the electromagnetic positioning system. The electromagnetic positioning system can further include a magnetic field generating component and a magnetic field detecting component. The magnetic field generating component is used to generate a magnetic field, the position sensor is an EM sensor (i.e., an electromagnetic sensor), the position sensor will cause a change in the magnetic field in the magnetic field, and the magnetic field detection component can detect the change in the magnetic field to detect the position of the position sensor relative to the magnetic field / magnetic field generating component. Combined with the coordinate conversion relationship between the position sensor and the catheter and / or the instrument obtained by presetting or calibration, the position of the catheter and / or the instrument relative to the magnetic field / magnetic field generating component can be calculated, and then combined with the coordinate conversion relationship between the magnetic field / magnetic field generating component and the world coordinate system, the position of the catheter and / or the instrument in the world coordinate system, i.e., the sampling point, can be calculated. The position sensor can sense at least three translational degrees of freedom in the magnetic field.

[0322] For another example, a shape sensor system can be used to obtain the position of a catheter and / or instrument inserted into an anatomical structure. For example, the shape sensor can include an optical fiber aligned with the catheter, and the optical fiber bending sensor formed by the optical fiber can feedback the shape of the catheter, based on which the position of the catheter and / or instrument relative to the base of the shape sensor can be calculated. Combined with the position of the base of the shape sensor in the world coordinate system, the position of the catheter and / or instrument in the world coordinate system, i.e., the sampling point, can be calculated.

[0323] In some embodiments, for the case where the denoising weight matrix is ​​derived from the second type, such as Fig.14 As shown, determining the denoising weight matrix also includes:

[0324] Step S1201', obtaining the patient's physical characteristics.

[0325] Step S1203', based on the patient's physical characteristics, matching target patients with the same or similar physical characteristics.

[0326] Step S1204 ′, determining feature points associated with the patient's sampling point subset.

[0327] Step S1205 ′: based on the feature points, a denoising weight matrix of the target patient associated with the feature points is matched.

[0328] Through the above steps S1201 to S1205, there is no need to reconstruct the denoising weight matrix of different feature points of the anatomical structure for the patient himself, and the denoising weight matrix of the corresponding feature points of the existing anatomical structure of other patients with the same or similar body features as the patient can be used to greatly reduce the preoperative preparation time, which at least includes the time for arranging sensors for detecting the intrinsic movement of the anatomical structure. Of course, in a scenario with more time, it is also possible to take into account the individual differences between patients and construct the denoising weight matrix of different feature points of the anatomical structure for the patient alone.

[0329] Optional, such as Fig.15 As shown, for a training object, whether based on the patient itself or on other patients, for each of a plurality of feature points, a method for initially constructing a denoising weight matrix for each feature point may include:

[0330] Step S1211: acquiring a sampling point subset corresponding to a feature point from the anatomical structure.

[0331] The sampling point subset includes multiple sampling points, and the multiple sampling points are obtained by sampling the same tracking sensor at different times. In this embodiment, the multiple sampling points generally include sampling points in the order of ten, one hundred or even more.

[0332] Step S1212: acquiring a test point set from the body surface corresponding to the anatomical structure.

[0333] Each test point set includes multiple test point subsets, and each test point subset includes at least one test point. The test point is sampled by a surface sensor arranged on the patient's body surface corresponding to the anatomical structure. The test point is generally described by the coordinates of the surface sensor in the surgical environment coordinate system. A corresponding test point of different test point subsets is usually sampled by a corresponding surface sensor at different times.

[0334] Corresponding to the expected number of test points in each test point subset, the number of body surface sensors is configured in the same manner. For example, when the expected number of test points in each test point subset is one, the number of body surface sensors is one; for another example, when the expected number of test points in each test point subset is two or three, the number of body surface sensors is correspondingly two or three.

[0335] In this embodiment, the plurality of test point subsets generally include test point sets in the order of ten, one hundred or even more.

[0336] In some embodiments, the number of sampling points in the sampling point subset is usually required to be the same as the number of test point subsets in the test point set for subsequent calculations. However, this does not necessarily require that the same number must be obtained by sampling, and the same number of the two can also be maintained by data processing such as interpolation, filtering, etc.

[0337] In some embodiments, for the same feature point, sampling can be performed separately and successively to obtain a sampling point subset and a test point set respectively; however, from the perspective of saving preoperative time, sampling can also be performed simultaneously and at the same frequency to obtain a sampling point subset and a test point set.

[0338] In some embodiments, when the anatomical structure is a bronchus or a (cardio)vascular vessel, at least one body surface sensor is arranged on the patient's chest. Exemplarily, the body surface sensors may include three, the first body surface sensor may be arranged in the middle of the patient's chest, the second body surface sensor may be arranged in the area of ​​the patient's chest corresponding to the seventh rib on the left, and the third body surface sensor may be arranged in the area of ​​the patient's chest corresponding to the seventh rib on the right. The more body surface sensors there are, the more areas of the patient's body surface are covered, and the more accurate the sensing of the intrinsic movement of the anatomical structure is.

[0339] Step S1213, based on the test point set and the sampling point subset in the same surgical environment coordinate system, determine the denoising weight matrix corresponding to the sampling point subset, that is, determine the denoising weight matrix corresponding to the feature point.

[0340] In some embodiments, the body surface sensor and the tracking sensor are EM sensors, and the two may belong to the same electromagnetic positioning system or different electromagnetic positioning systems. When the two belong to the same electromagnetic positioning system, the positions sensed by the two are in the same surgical environment coordinate system; when the two belong to different electromagnetic positioning systems, the positions sensed by the two may be transformed to the same surgical environment coordinate system by coordinate conversion. In this case, the surgical environment coordinate system may also be referred to as an electromagnetic coordinate system, or as a sensor coordinate system.

[0341] In some embodiments, the above step S1213 can be more specifically implemented by the following steps:

[0342] For ease of description, the position data of the test point set acquired by the body surface sensor (ie, test points) is defined as the body surface sensor data Ref, and the position data of the sampling point subset acquired by the tracking sensor (ie, sampling points) is defined as the in-vivo catheter data Tube.

[0343] (1) De-average the body surface sensor data Ref to obtain the body surface sensor data And de-average the in vivo catheter data Tube to obtain the in vivo catheter data

[0344] (2) Obtaining body surface sensor data The inverse solution of . For example, the inverse solution can be obtained by SVD (Singular Value Decomposition) method. The obtained inverse solution can usually be expressed as:

[0345]

[0346] in, is the inverse solution of the body surface sensor data, V is the right singular matrix, S is the singular value matrix, and U is the left singular matrix.

[0347] (3) Inverse solution based on body surface sensor data and in vivo catheter data Determine the denoising weight matrix of the corresponding feature point. The denoising weight matrix associated with the corresponding feature point can be expressed as:

[0348]

[0349] Among them, Weight is the denoising weight matrix.

[0350] In some embodiments, still taking the anatomical structure of the bronchus as an example, three body surface sensors can be used to obtain a test point set in more than two respiratory cycles (usually 6 to 10 seconds), and a subset of sampling points associated with the feature points can be simultaneously obtained through the tracking sensor. Among them, the body surface sensor data Ref is a matrix of N rows and 9 columns, and the format of each row can be expressed as {X1, Y1, Z1, X2, Y2, Z2, X3, Y3, Z3}, for example, and each 3 columns correspond to the position data sensed by a body surface sensor. The in vivo catheter data Tube is a matrix of N rows and 3 columns, and the format of each row can be expressed as {X0, Y0, Z0}, for example, and the 3 columns correspond to the position data of the catheter and / or instrument sensed by the tracking sensor.

[0351] Furthermore, each feature point corresponds to a certain denoising weight matrix. Exemplarily, the data structure ST = {PosTube, PosRef, Weight} can be used for storage. Among them, PosTube is the mean value of the sampling points in the sampling point set corresponding to each feature point. For example, when an electromagnetic positioning system is used, the mean value is the mean value of the electromagnetic coordinate points of the catheter and / or the instrument, as shown in PosRef is the mean value of the test points in the test point set corresponding to each feature point. For example, when an electromagnetic positioning system is used, the mean value is the mean value of the electromagnetic coordinate points of the three body surface sensors, as shown in Weight is a matrix with 9 rows and 3 columns.

[0352] When acquiring data of sampling points corresponding to a certain feature point, the control device can mark these sampling points or the sampling point subsets corresponding to these sampling points, for example, assign serial numbers to establish an association between the sampling points or the sampling point subsets and the corresponding feature points. Furthermore, when determining the corresponding denoising weight matrix based on the association between the feature points and the sampling points (or sampling point subsets) and the denoising weight matrix, the sampling points (or sampling point subsets) and the denoising weight matrix can be matched extremely conveniently and quickly.

[0353] Optionally, when denoising the sampling points in the sampling point subset based on the denoising weight matrix, for the same feature point, the sampling point can be denoised by combining a corresponding pair of sampling points and a test point, the denoising weight matrix weight, and the mean value PosRef of the test points in the test point set. Exemplary examples include:

[0354] (1) Determine the offset between the current test point associated with the same feature point and the mean of the test points in the test point set. For example, the offset can be determined by the following formula:

[0355]

[0356] in, is the offset, and Ro is the current test point. Exemplarily, when the number of body surface sensors is 3, the offset is a row vector containing 9 elements.

[0357] (2) De-noising the current sampling point based on the feature vector associated with the same feature point, the current sampling point, and the denoising weight matrix. For example, denoising can be performed using the following formula:

[0358]

[0359] Wherein, Tf is the current sampling point after denoising; To is the current sampling point.

[0360] In this embodiment, a corresponding pair of sampling points and test points may refer to sampling points and test points that are sampled at the same moment in the same cycle of the anatomical structure, for example, the pair of points are collected at the same moment in the same cycle; or may refer to sampling points and test points that are sampled at the same moment in different cycles of the anatomical structure.

[0361] The embodiment of the present application uses the data of the body surface sensor to determine the denoising weight matrix, so that the aforementioned denoising of the sensor data using the denoising weight matrix can be achieved.

[0362] In one embodiment, the present application provides a navigation method, the method comprising:

[0363] Acquire a plurality of first medical images of the anatomical structure of the training object, wherein the plurality of first medical images satisfy a first preset condition, wherein the first preset condition comprises that the plurality of first medical images include a first medical image representing a stretching process state of the anatomical structure and a first medical image representing a contracting process state of the anatomical structure;

[0364] Performing training according to the plurality of first medical images to obtain a recognition network;

[0365] In response to a user's navigation instruction, using the recognition network to identify a first pipeline segment from an intraoperative medical image of the patient acquired by the imaging device;

[0366] Acquire a planned path from an initial position to a target position, wherein the planned path is composed of a plurality of connected second pipeline segments;

[0367] Based on the first pipeline segment, matching a target pipeline segment from the second pipeline segment;

[0368] generating a navigation mark marking the target pipeline segment;

[0369] The navigation mark and the target channel segment are displayed on a user interface to guide the movement of the catheter and / or instrument.

[0370] In one embodiment, the navigation method provided by the present application includes:

[0371] In response to a user's navigation instruction, using the recognition network to identify a first pipeline segment from an intraoperative medical image of the patient acquired by the imaging device;

[0372] Acquire a planned path from an initial position to a target position, wherein the planned path is composed of a plurality of connected second pipeline segments;

[0373] Based on the first pipeline segment, matching a target pipeline segment from the second pipeline segment;

[0374] generating a navigation mark marking the target pipeline segment;

[0375] The navigation mark and the target channel segment are displayed on a user interface to guide the movement of the catheter and / or instrument.

[0376] The method of using the recognition network to identify the first pipeline segment from the intraoperative medical image of the patient acquired by the imaging device includes:

[0377] scaling an intraoperative medical image of a patient acquired by the imaging device;

[0378] A first duct segment is identified from the scaled intraoperative medical image using the recognition network.

[0379] In one embodiment, the navigation method provided by the embodiment of the present application includes:

[0380] In response to a user's navigation instruction, using the recognition network to identify a first pipeline segment from an intraoperative medical image of the patient acquired by the imaging device;

[0381] Correcting the actual path point to obtain a corrected path point;

[0382] Determining a positioning correction amount according to the actual path point and the corrected path point;

[0383] determining relative positions of the duct segments in the intraoperative medical image;

[0384] identifying a third pipeline segment in the intraoperative medical image according to the first pipeline segment, the positioning correction amount, and a relative position of the pipeline segment in the intraoperative medical image, wherein the third pipeline segment is different from the first pipeline segment;

[0385] Acquire a planned path from an initial position to a target position, wherein the planned path is composed of a plurality of connected second pipeline segments;

[0386] Based on the first pipeline segment and the third pipeline segment, a target pipeline segment is matched from the second pipeline segment.

[0387] generating a navigation mark marking the target pipeline segment;

[0388] The navigation mark and the target channel segment are displayed on a user interface to guide the movement of the catheter and / or instrument.

[0389] The specific implementation is the same as the above principle, and will not be repeated here for the sake of brevity.

[0390] The present application also provides a control device. Fig.16 As shown, the control device may include: a processor (processor) 501, a communication interface (Communications Interface) 502, a memory (memory) 503, and a communication bus 504.

[0391] The processor 501 , the communication interface 502 , and the memory 503 communicate with each other via the communication bus 504 .

[0392] The communication interface 502 is used to communicate with other devices such as various sensors, rotating motors, solenoid valves, or network elements of other clients or servers.

[0393] The processor 501 is used to execute the program 505, and specifically can execute the relevant steps in the above method embodiment.

[0394] Specifically, the program 505 may include program codes, which include computer operation instructions.

[0395] The processor 505 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application, or an FPGA, or a graphics processing unit (GPU). The one or more processors included in the control device may be processors of the same type, such as one or more CPUs, or one or more GPUs; or may be processors of different types, such as one or more CPUs and one or more GPUs.

[0396] The memory 503 is used to store the program 505. The memory 503 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0397] The program 505 can be specifically used to enable the processor 501 to execute the steps of the method described in any of the above embodiments.

[0398] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program is configured to be loaded by a processor and executed to implement the steps of the method described in any of the above embodiments.

[0399] The present application also provides a computer program product, including computer instructions, which, when executed on a computer, enable the computer to execute the method described in any one of the above embodiments.

[0400] It should be noted that other sorting schemes that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention should also be within the protection scope of the present invention and will not be described in detail here.

[0401] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the mobile terminal can be divided into different functional units or modules to complete all or part of the functions described above. The functional modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the modules in the above-mentioned mobile terminal can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0402] Exemplarily, the computer program can be divided into one or more modules / units, one or more modules / units are stored in a memory and executed by a processor to complete the present invention. One or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the surgical robot.

[0403] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0404] The memory may be an internal storage unit of the surgical robot, such as a hard disk or memory of the surgical robot. The memory may also be an external storage device of the surgical robot, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the surgical robot. Furthermore, the memory may include both an internal storage unit and an external storage device of the surgical robot. The memory is used to store computer programs and other programs and data required by the terminal device. The memory may also be used to temporarily store data that has been output or is to be output.

[0405] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0406] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0407] As used in the specification of this application and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if the described condition or event is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce the described condition or event is detected" or "in response to detecting the described condition or event" depending on the context.

[0408] In addition, in the description of the present application specification and the appended claims, the terms "the first", "the second", "the third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0409] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0410] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0411] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0412] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0413] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0414] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.

[0415] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A navigation system, characterized in that: The navigation system is applied to a surgical robot, the surgical robot comprising a processor, a catheter and / or an instrument, the catheter and / or the instrument being provided with an imaging device for acquiring medical images, the processor being configured to perform the following steps: Acquire a plurality of first medical images of the anatomical structure of the training object, wherein the plurality of first medical images satisfy a first preset condition, wherein the first preset condition comprises that the plurality of first medical images include a first medical image representing a stretching process state of the anatomical structure and a first medical image representing a contracting process state of the anatomical structure; Performing training according to the plurality of first medical images to obtain a recognition network; In response to a user's navigation instruction, using the recognition network to identify a first pipeline segment from an intraoperative medical image of the patient acquired by the imaging device; Acquire a planned path from an initial position to a target position, wherein the planned path is composed of a plurality of connected second pipeline segments; Based on the first pipeline segment, matching a target pipeline segment from the second pipeline segment; generating a navigation mark marking the target pipeline segment; The navigation mark and the target channel segment are displayed on a user interface to guide the movement of the catheter and / or instrument.

2. The navigation system according to claim 1, characterized in that The body surface of the training object is provided with a body surface sensor. In acquiring a plurality of first medical images of the anatomical structure of the training object, the processor is configured to perform the following steps: Acquire the first medical image and the body surface data of the training object sensed by the body surface sensor at the same time; Based on the body surface data, determining a process state represented by the first medical image, wherein the process state includes any one of a stretch process state and a contraction process state; The above steps are repeatedly performed until the number of first medical images representing multiple different process states meets a preset value, and the first preset condition also includes that the number of first medical images representing multiple different process states meets the preset value.

3. The navigation system according to claim 1, characterized in that: Before the training is performed according to the plurality of first medical images, the processor is configured to perform the following steps: Acquire a second medical image of the training object in a stretching process state, and generate a first anatomical model of the training object based on the second medical image; Acquire a third medical image of one of the contraction process states of the training object, and generate a second anatomical model of the training object based on the third medical image; A maximum shrinkage ratio of the first anatomical model and the second anatomical model is determined.

4. The navigation system according to claim 3, characterized in that: Before training according to the plurality of first medical images to obtain a recognition network, the processor is configured to perform the following steps: Determining a first scaling ratio according to the maximum shrinkage ratio; Based on the first scaling ratio, scaling the first medical image to obtain a scaled first medical image; In the step of training the plurality of first medical images to obtain a recognition network, the processor is configured to perform the following steps: The recognition network is obtained by training according to the multiple first medical images and the scaled first medical image.

5. The navigation system according to claim 3, characterized in that: Before identifying the first pipeline segment from the intraoperative medical image of the patient acquired by the imaging device using the recognition network in response to the user's navigation instruction, the processor is configured to perform the following steps in response to the motion compensation instruction: Determining a second scaling ratio according to the maximum shrinkage ratio; The intraoperative medical image is scaled based on the second scaling ratio.

6. The navigation system according to claim 1, wherein: The catheter and / or the instrument is provided with a tracking sensor for sensing the position of the catheter and / or the instrument, and the tracking sensor is used to obtain the actual path point of the catheter and / or the instrument. In matching the target pipeline segment from the second pipeline segment based on the first pipeline segment, the processor is configured to perform the following steps: Correcting the actual path point to obtain a corrected path point; Determining a positioning correction amount according to the actual path point and the corrected path point; determining relative positions of the duct segments in the intraoperative medical image; identifying a third pipeline segment in the intraoperative medical image according to the first pipeline segment, the positioning correction amount, and the relative position, wherein the third pipeline segment is different from the first pipeline segment; Based on the first pipeline segment and the third pipeline segment, a target pipeline segment is matched from the second pipeline segment.

7. The navigation system according to claim 1, characterized in that: The catheter and / or the instrument is provided with a tracking sensor for measuring the position of the catheter and / or the instrument, and the tracking sensor is used to obtain the actual path point of the catheter and / or the instrument. The processor is configured to perform the following steps: Acquire a preoperative medical image of the patient, and generate an anatomical model of the patient based on the preoperative medical image, wherein the preoperative medical image represents a contraction process state or an extension process state of the anatomical structure of the patient; Acquire a transformation matrix, where the transformation matrix is ​​used to represent the conversion relationship between the surgical environment coordinate system and the patient's anatomical model coordinate system; Determining, based on the actual path points and the transformation matrix, simulated path points of the catheter and / or instrument in the anatomical model of the patient corresponding to the actual path points; Alternatively, the actual path point is corrected to obtain a corrected path point, and based on the corrected path point and the transformation matrix, a simulated path point of the catheter and / or instrument corresponding to the actual path point in the anatomical model of the patient is determined; marking the simulated waypoints in an anatomical model of the patient; An anatomical model of the patient and the simulated waypoints are displayed on the user interface.

8. The navigation system according to claim 6 or 7, characterized in that: In correcting the actual path point to obtain the corrected path point, the processor is configured to perform the following steps: Based on the actual path point, determining a feature point that matches the actual path point, the feature point being a point in the anatomical structure of the patient; A denoising weight matrix associated with the feature point is obtained, and based on the denoising weight matrix, the actual path point is denoised to obtain a denoised path point, and the denoised path point after denoising is used as the corrected path point.

9. The navigation system as claimed in claim 8, characterized in that The training object's body surface is provided with a body surface sensor, the tracking sensor acquires sampling points from the training object's anatomical structure, the body surface sensor acquires test points from the training object's body surface, and in acquiring a denoising weight matrix associated with the feature points, the processor is configured to perform the following steps: Acquire a sampling point subset of the training object, wherein the sampling point subset of the training object includes a plurality of the sampling points; Acquire a test point set of the training object, wherein the test point set of the training object includes a plurality of the test points; Based on the sampling point subset and the test point set, the denoising weight matrix corresponding to the feature points of the training object is determined.

10. The navigation system according to claim 6 or 7, characterized in that In the step of correcting the actual path point to obtain the corrected path point, the processor is configured to perform the following steps: Acquire movement data of the catheter in the conduit network of the patient's anatomical structure; The revised pathpoints are determined based on the actual pathpoints, the movement data and the intraoperative medical image.

11. The navigation system of claim 1, wherein: The intraoperative medical image represents the contraction process state or the expansion process state of the anatomical structure of the patient, the body surface of the patient is provided with a body surface sensor, and the processor is configured to perform the following steps: Acquiring the intraoperative medical image and the body surface data of the patient sensed by the body surface sensor at the same time; Determine vital sign data of the patient's anatomical structure according to the body surface data, wherein the vital sign data represents data of the patient's anatomical structure in different process states; A vital sign curve of the anatomical structure of the patient is generated according to the vital sign data, and the vital sign curve is displayed on the user interface, wherein the vital sign curve includes a breathing curve and / or a heartbeat curve.

12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by the processor, a navigation method is implemented, and the navigation method includes: Acquire a plurality of first medical images of the anatomical structure of the training object, wherein the plurality of first medical images satisfy a first preset condition, wherein the first preset condition comprises that the plurality of first medical images include a first medical image representing a stretching process state of the anatomical structure and a first medical image representing a contracting process state of the anatomical structure; Performing training according to the plurality of first medical images to obtain a recognition network; In response to a user's navigation instruction, using the recognition network to identify a first pipeline segment from an intraoperative medical image of the patient acquired by an imaging device; Acquire a planned path from an initial position to a target position, wherein the planned path is composed of a plurality of connected second pipeline segments; Based on the first pipeline segment, matching a target pipeline segment from the second pipeline segment; generating a navigation mark marking the target pipeline segment; The navigation mark and the target channel segment are displayed on a user interface to guide the movement of a catheter and / or an instrument.

Images