Navigation system and method and storage medium

Abstract

The invention discloses a navigation system and method and a storage medium, and the system comprises the steps: recognizing the first positioning information of a flexible instrument in an anatomical model according to a first positioning algorithm and first positioning data in a memory, the first positioning data at least comprising image data collected by an image positioning device; determining second positioning information of the flexible instrument in the anatomical model according to the first positioning information, a second positioning algorithm and the second positioning data; and updating the navigation information based on the second positioning information. According to the method, the navigation precision under lung movement is improved, and the accuracy of image navigation is also improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of medical devices, in particular to a navigation system, method and storage medium. BACKGROUND

[0002] In the existing bronchoscope navigation system, CT data is collected before the operation and the bronchial tree is reconstructed to provide image reference of virtual bronchoscope during the operation. The registration conversion between the image position in the real world and the corresponding image position on the virtual bronchial tree is completed by manual or accessory positioning of the positioning device to provide real-time guidance for the doctor's operation. In the actual operation process, due to the interference of respiratory motion, heartbeat, bronchoscope extrusion, etc., the position of the bronchial wall will deform accordingly, thereby affecting the positioning and registration accuracy of the bronchoscope, causing errors in the position estimation of the virtual endoscope, especially in the bronchus below the 5th lung segment. The accumulation of errors caused by deformation due to respiratory motion and other external forces makes it difficult to accurately position the virtual endoscope during the operation, affecting the navigation accuracy and effect provided by the system.

[0003] In the existing scheme, the periodicity of the sagittal plane respiratory motion is usually extracted to compensate for the position after registration periodically to reduce the positioning error; the positioning device can also be set on the body surface to capture the respiratory correlation model established in vivo / out of the body to estimate the influence of respiratory motion and make error compensation. However, due to the complexity and randomness of the interference factors, it is difficult to consider all deformation interference factors in this indirect compensation method estimated by the model, and the accuracy after compensation is limited. SUMMARY

[0004] To solve the problem of insufficient intraoperative navigation accuracy caused by existing lung motion, the present application proposes the following technical solutions:

[0005] The first aspect of the present application provides a navigation system, comprising: a flexible instrument, provided with a first positioning device and a second positioning device, the first positioning device is used to collect first positioning data, and the second positioning device is used to collect second positioning data, and the first positioning device at least includes an image positioning device; a display device configured to display navigation information when the flexible instrument enters a lung bronchus; a memory storing a plurality of positioning algorithms, the positioning algorithms including a first positioning algorithm and a second positioning algorithm, wherein the generation of each positioning algorithm is related to different constituent elements of an anatomical model of the lung bronchus; a processor configured to: identify first positioning information of the flexible instrument in the anatomical model according to the first positioning algorithm and the first positioning data, wherein the first positioning data at least includes image data collected by the image positioning device; determine second positioning information of the flexible instrument in the anatomical model according to the first positioning information, the second positioning algorithm and the second positioning data; and update the navigation information based on the second positioning information.

[0006] Wherein the first positioning device and the second positioning device further include at least one of a magnetic positioning device and a mechanical positioning device, and the first positioning data and the second positioning data include at least one of magnetic positioning data collected by the magnetic positioning device and robot data collected by the mechanical positioning device, wherein the flexible instrument is detachably installed on a robot arm, and the robot data refers to relevant data generated when the robot arm physically drives the flexible instrument; and the different constituent elements of the anatomical model of the lung bronchus include at least two of model voxels, a first bronchial tree, a second bronchial tree and a skeleton center point set, wherein the first bronchial tree and the second bronchial tree are defined differently in the anatomical model at a bifurcation position.

[0007] Wherein, according to the first positioning algorithm and the first positioning data, the first positioning information of the flexible instrument in the anatomical model includes: according to the first positioning algorithm related to the first bronchial tree and the image data, the position of the flexible instrument is detected to obtain a first detection result, and the first bronchial tree includes a virtual endoscopic image collected at each bifurcation position in the anatomical model; if the first detection result meets a preset first condition, the first positioning information of the flexible instrument in the anatomical model is determined based on the first detection result; or, according to the first positioning algorithm related to the second bronchial tree and the image data, the position of the flexible instrument is detected to obtain a second detection result, and the second bronchial tree includes the visualization relationship of a plurality of branches in the anatomical model; if the second detection result meets the preset first condition, the first positioning information is determined based on the second detection result; wherein the preset first condition at least includes a preset first branch number.

[0008] wherein, after obtaining the first detection result or the second detection result, further comprising: if the first detection result or the second detection result meets a preset second condition, performing position detection on the flexible instrument according to a first positioning algorithm related to the skeleton center point set and the magnetic positioning data to obtain a third detection result, and determining first positioning information based on the third detection result; wherein the skeleton center point set includes an inflection point of a curved segment in the anatomical model, the first positioning information is related to the inflection point, the curved segment refers to a segment in the anatomical model that is greater than a preset bending degree, and the preset second condition at least includes a preset second branch number.

[0009] wherein, the first positioning algorithm includes a first target recognition model trained by the first bronchial tree, and the position detection on the flexible instrument according to the first positioning algorithm related to the first bronchial tree and the image data to obtain the first detection result includes: inputting the image data into the first target recognition model, and using the first target recognition model to perform position detection on the flexible instrument to obtain the first detection result; training the first target recognition model according to the first bronchial tree, specifically including: acquiring real endoscopic images collected at each bifurcation position in the lung bronchus; extracting feature information in each virtual endoscopic image, and extracting style information in each real endoscopic image; synthesizing the feature information and the style information at the same bifurcation position and performing smoothing processing to obtain a style image corresponding to each virtual endoscopic image; labeling first branch information of each style image, the first detection result including parameters in the first branch information; training the first target recognition model based on the style image and the labeled information.

[0010] wherein, the second positioning information of the flexible instrument in the anatomical model detected according to the first positioning information, the second positioning algorithm and the second positioning data includes: when the first positioning information is determined by the first detection result or the second detection result, it is determined that the first positioning information includes parameters of at least two first branches, and the at least two first branches meet a preset first display effect in the image data, the first display effect being determined by a preset first condition; performing three-dimensional reconstruction on the image data to generate a first model voxel of a lung segment in the image data; intercepting a second model voxel of the at least two first branches in the anatomical model; projecting the first model voxel to the second model voxel, and determining the second positioning information of the flexible instrument in the anatomical model according to the projection result, the second positioning information including pose information of the flexible instrument.

[0011] The first positioning algorithm further includes a second target recognition model trained by the second bronchial tree, and the position detection of the flexible instrument according to the first positioning algorithm related to the second bronchial tree and the image data to obtain a second detection result includes: inputting the image data into the second target recognition model, and using the second target recognition model to detect the position of the flexible instrument to obtain a second detection result; training the second target recognition model according to the second bronchial tree, including: obtaining a training data set, the training data set including second branch information labeled for each training image, and the second detection result including parameters in the second branch information; based on the training result of the training data set, the number of each branch and other branches appearing in the same training image is counted; based on the counted number, a knowledge graph is constructed, the knowledge graph being used to represent the visual relationship of each branch in the anatomical model; and generating a second target recognition model according to the training result and the knowledge graph.

[0012] The second positioning information of the flexible instrument in the anatomical model according to the first positioning information, the second positioning algorithm and the second positioning data includes: when the first positioning information is determined by the first detection result or the second detection result, determining that the first positioning information includes the recognition probability of at least two second branches, and the at least two second branches satisfy a preset second display effect, the second display effect being determined by a preset first condition; adjusting the recognition probability according to the visual relationship of the at least two second branches; and / or, adjusting the recognition probability according to the distance between the at least two second branches; and determining the second positioning information of the flexible instrument in the anatomical model based on the adjusted recognition probability, wherein the second positioning information includes the parameters of the second branch within the visual field range of the image positioning device.

[0013] The position detection of the flexible instrument according to the first positioning algorithm related to the skeleton center point set and the magnetic positioning data to obtain a third detection result includes: according to the magnetic positioning data, mapping the position of the flexible instrument to a first target center point in the skeleton center point set, and determining a turning point and a second target center point in the skeleton center point set in the advancing direction of the flexible instrument, the second target center point being in a different branch relative to the first target center point; determining the relative position relationship between the first target center point, the turning point and the second target center point, and determining the third detection result of the flexible instrument according to the relative position relationship, the third detection result at least including: the flexible instrument entering the curved segment and leaving the curved segment.

[0014] The determining the relative position relationship among the first target center point, the inflection point and the second target center point, and determining a third detection result of the flexible instrument according to the relative position relationship comprises: calculating a curve integral distance between the first target center point and the inflection point, and determining whether the flexible instrument enters a bending segment according to the curve integral distance; if the flexible instrument enters the bending segment, calculating a bending angle among the first target center point, the inflection point and the second target center point, and determining whether the flexible instrument is in a turning area in the bending segment according to the bending angle; determining whether the first target center point exceeds the second target center point, and determining whether the flexible instrument leaves the bending segment according to the determination result; wherein the third detection result further comprises that the flexible instrument is in the turning area.

[0015] The detecting the second positioning information of the flexible instrument in the anatomical model according to the first positioning information, the second positioning algorithm and the second positioning data comprises: setting a reference zero point in the set of skeleton center points based on the magnetic positioning data and / or the image data when the flexible instrument is in the turning area; taking the reference zero point as a starting position, calculating an offset of the flexible instrument according to the robot data; mapping a position of the flexible instrument to a third target center point in the set of skeleton center points according to the offset; determining the second positioning information of the flexible instrument in the anatomical model according to the third target center point, and the second positioning information comprises position information of the flexible instrument.

[0016] The first positioning algorithm comprises a classification network trained by the first bronchial tree or the second bronchial tree, and the classification network is used to identify a quality category of the image data, and the first detection result and the second detection result further comprise the quality category; wherein the quality category comprises a first category and a second category, image quality corresponding to the first category is higher than image quality corresponding to the second category, the first category satisfies the preset first condition, and the second category satisfies the preset second condition.

[0017] The quality categories further include a third category, the image quality corresponding to the third category is lower than the image quality corresponding to the second category, the third category satisfies a preset third condition, after obtaining the first detection result or the second detection result, further comprising: if the first detection result or the second detection result satisfies the preset third condition, performing position detection on the flexible instrument according to a first positioning algorithm related to the skeleton center point set and the robot data to obtain a fourth detection result; determining the extension amount of the flexible instrument according to the robot data, and calculating a ratio of the extension amount relative to a preset reference extension amount; setting a weight of the first detection result or the second detection result according to the ratio, and setting a weight of the third detection result and the fourth detection result according to the ratio; performing weighted processing on one of the first detection result and the second detection result in combination with the third detection result and the fourth detection result according to the set weights to obtain first positioning information.

[0018] The evaluation indexes of the image quality of the first category include that the image definition satisfies a preset definition threshold, the image pollution degree satisfies a preset first pollution degree, the shooting angle satisfies a preset first angle, and the display completeness of the branch lung segment orifice satisfies a preset first completeness; the evaluation indexes of the image quality of the second category include that the image definition satisfies a preset definition threshold, the image pollution degree satisfies a preset second pollution degree, the shooting angle satisfies a preset second angle, and the display completeness of the branch lung segment orifice satisfies a preset second completeness; the evaluation indexes of the image quality of the third category include at least one of the following: the image definition does not satisfy a preset definition threshold, the image pollution degree satisfies a preset third pollution degree, the shooting angle satisfies a preset third angle, and the display completeness of the branch lung segment orifice satisfies a preset third completeness; wherein the image pollution degrees are in ascending order of the preset first pollution degree, the preset second pollution degree, and the preset third pollution degree, the shooting angles are in ascending order of the preset first angle, the preset second angle, and the preset third angle, and the completenesses are in descending order of the preset first completeness, the preset second completeness, and the preset third completeness.

[0019] In a second aspect, the present application provides a navigation method, comprising: obtaining different types of positioning data collected by a plurality of positioning devices in real time, wherein the plurality of positioning devices are arranged on a flexible instrument, and at least include an image positioning device, and the flexible instrument is controlled to enter a lung bronchus; identifying first positioning information of the flexible instrument in an anatomical model of the lung bronchus according to at least one first positioning algorithm in a plurality of preset positioning algorithms and at least one type of first positioning data, wherein the first positioning data at least include image data collected by the image positioning device, and each of the preset positioning algorithms is generated in relation to a different constituent element of the anatomical model; selecting a second positioning algorithm related to one of the constituent elements and at least one type of second positioning data according to the first positioning information, wherein the second positioning data at least include part of the first positioning data; detecting second positioning information of the flexible instrument in the anatomical model according to the second positioning algorithm and the second positioning data, and updating navigation information based on the second positioning information.

[0020] In the method, the plurality of positioning devices further include at least one of a magnetic positioning device and a mechanical positioning device, and the different types of positioning data further include at least one of magnetic positioning data collected by the magnetic positioning device and robot data collected by the mechanical positioning device, wherein the flexible instrument is detachably installed on a robot arm, and the robot data refers to relevant data generated when the robot arm physically drives the flexible instrument; and the different constituent elements of the anatomical model include at least two of a model voxel, a first bronchial tree, a second bronchial tree and a skeleton center point set, wherein the first bronchial tree and the second bronchial tree are different in definition of the anatomical model at a bifurcation position.

[0021] The first positioning information of the flexible instrument in the anatomical model of the lung bronchus is identified according to at least one first positioning algorithm in a plurality of preset positioning algorithms and at least one type of first positioning data, including: performing position detection on the flexible instrument according to a first positioning algorithm related to the first bronchial tree and the image data to obtain a first detection result, the first bronchial tree including a virtual endoscope image collected at each bifurcation position in the anatomical model; if the first detection result meets a preset first condition, determining the first positioning information of the flexible instrument in the anatomical model based on the first detection result; or, performing position detection on the flexible instrument according to a first positioning algorithm related to the second bronchial tree and the image data to obtain a second detection result, the second bronchial tree including a visual relationship of a plurality of branches in the anatomical model; if the second detection result meets the preset first condition, determining the first positioning information based on the second detection result; wherein the preset first condition at least includes a preset first branch quantity.

[0022] After obtaining the first detection result or the second detection result, if the first detection result or the second detection result meets a preset second condition, position detection is performed on the flexible instrument according to a first positioning algorithm related to the skeleton center point set and the magnetic positioning data to obtain a third detection result, and the first positioning information is determined based on the third detection result; wherein the skeleton center point set includes an inflection point of a curved segment in the anatomical model, the first positioning information is related to the inflection point, the curved segment refers to a segment greater than a preset bending degree in the anatomical model, and the preset second condition at least includes a preset second branch quantity.

[0023] The first positioning algorithm includes a first target recognition model trained by the first bronchial tree, and the position detection on the flexible instrument according to the first positioning algorithm related to the first bronchial tree and the image data to obtain the first detection result includes: inputting the image data into the first target recognition model, and using the first target recognition model to perform position detection on the flexible instrument to obtain the first detection result; the first target recognition model is trained according to the first bronchial tree, specifically including: acquiring real endoscope images collected at each bifurcation position in the lung bronchus; extracting feature information in each virtual endoscope image and style information in each real endoscope image; synthesizing the feature information and the style information at the same bifurcation position and performing smoothing processing to obtain a style image corresponding to each virtual endoscope image; labeling first branch information of each style image, the first detection result including parameters in the first branch information; and training the first target recognition model based on the style image and the labeling information.

[0024] The second positioning information of the flexible instrument in the anatomical model is detected according to the second positioning algorithm and the second positioning data, and the detecting the second positioning information of the flexible instrument in the anatomical model according to the second positioning algorithm and the second positioning data comprises: when the first positioning information is determined by the first detection result or the second detection result, it is determined that the first positioning information comprises parameters of at least two first branches, and the at least two first branches satisfy a preset first display effect in the image data, the first display effect being determined by a preset first condition; the image data is three-dimensionally reconstructed to generate a first model voxel of a pulmonary segment in the image data; a second model voxel of the at least two first branches in the anatomical model is intercepted; the first model voxel is projected to the second model voxel, and the second positioning information of the flexible instrument in the anatomical model is determined according to a projection result, the second positioning information comprising pose information of the flexible instrument.

[0025] The first positioning algorithm further comprises a second target recognition model trained from the second bronchial tree, and the position of the flexible instrument is detected according to the first positioning algorithm related to the second bronchial tree and the image data to obtain a second detection result, which comprises: the image data is input into the second target recognition model, and the position of the flexible instrument is detected by using the second target recognition model to obtain a second detection result; the second target recognition model is trained according to the second bronchial tree, which comprises: a training data set is obtained, the training data set comprising second branch information labeled for each training image, and the second detection result comprising parameters in the second branch information; based on a training result of the training data set, the number of each branch and other branches appearing in the same training image is counted; based on the counted number, a knowledge graph is constructed, the knowledge graph being used to represent a visual relationship of each branch in the anatomical model; and according to the training result and the knowledge graph, the second target recognition model is generated.

[0026] The second positioning information of the flexible instrument in the anatomical model is detected according to the second positioning algorithm and the second positioning data, and the detecting the second positioning information of the flexible instrument in the anatomical model according to the second positioning algorithm and the second positioning data comprises: when the first positioning information is determined by the first detection result or the second detection result, it is determined that the first positioning information comprises recognition probabilities of at least two second branches, and the at least two second branches satisfy a preset second display effect, the second display effect being determined by a preset first condition; the recognition probabilities are adjusted according to the visual relationship of the at least two second branches, and / or the recognition probabilities are adjusted according to distances between the at least two second branches; and the second positioning information of the flexible instrument in the anatomical model is determined based on the adjusted recognition probabilities, wherein the second positioning information comprises parameters of the second branches within a field of view of the image positioning device.

[0027] The position detection of the flexible instrument according to the first positioning algorithm related to the set of skeleton center points and the magnetic positioning data to obtain a third detection result includes: according to the magnetic positioning data, mapping the position of the flexible instrument to a first target center point in the set of skeleton center points, and determining a turning point and a second target center point in the set of skeleton center points in the advancing direction of the flexible instrument, the second target center point being in a different branch of the first target center point relative to the turning point; determining the relative position relationship among the first target center point, the turning point and the second target center point, and determining the third detection result of the flexible instrument according to the relative position relationship, the third detection result at least including: the flexible instrument entering the curved segment and leaving the curved segment.

[0028] The determination of the relative position relationship among the first target center point, the turning point and the second target center point, and the determination of the third detection result of the flexible instrument according to the relative position relationship include: calculating the curve integral distance between the first target center point and the turning point, and determining whether the flexible instrument enters a curved segment according to the curve integral distance; if the flexible instrument enters the curved segment, calculating the bending angle among the first target center point, the turning point and the second target center point, and determining whether the flexible instrument is in a turning area in the curved segment according to the bending angle; judging whether the first target center point exceeds the second target center point, and determining whether the flexible instrument leaves the curved segment according to the result of the judgment; wherein the third detection result further includes that the flexible instrument is in the turning area.

[0029] The detection of the second positioning information of the flexible instrument in the anatomical model according to the second positioning algorithm and the second positioning data includes: when the flexible instrument is in the turning area, setting a reference zero point in the set of skeleton center points based on the magnetic positioning data and / or the image data; taking the reference zero point as a starting position, calculating the offset of the flexible instrument according to the robot data; according to the offset, mapping the position of the flexible instrument to a third target center point in the set of skeleton center points; determining the second positioning information of the flexible instrument in the anatomical model according to the third target center point, the second positioning information including the position information of the flexible instrument.

[0030] The first positioning algorithm includes a classification network trained by the first bronchial tree or the second bronchial tree, and the classification network is used to identify a quality category of the image data. The first detection result and the second detection result further include the quality category. The quality category includes a first category and a second category. The image quality corresponding to the first category is higher than the image quality corresponding to the second category. The first category satisfies the preset first condition, and the second category satisfies the preset second condition.

[0031] The quality category further includes a third category. The image quality corresponding to the third category is lower than the image quality corresponding to the second category. The third category satisfies a preset third condition. After obtaining the first detection result or the second detection result, the method further includes: if the first detection result or the second detection result satisfies the preset third condition, performing position detection on the flexible instrument according to a first positioning algorithm related to the skeleton center point set and the robot data to obtain a fourth detection result; determining an extension amount of the flexible instrument according to the robot data, and calculating a ratio of the extension amount relative to a preset reference extension amount; setting a weight of the first detection result or the second detection result according to the ratio, and setting weights of the third detection result and the fourth detection result according to the ratio; and performing weighted processing on one of the first detection result and the second detection result in combination with the third detection result and the fourth detection result according to the set weights to obtain first positioning information.

[0032] The evaluation indexes of the image quality of the first category include that the image definition satisfies a preset definition threshold, the image pollution degree satisfies a preset first pollution degree, the shooting angle satisfies a preset first angle, and the display completeness of the branch lung segment orifice satisfies a preset first completeness. The evaluation indexes of the image quality of the second category include that the image definition satisfies a preset definition threshold, the image pollution degree satisfies a preset second pollution degree, the shooting angle satisfies a preset second angle, and the display completeness of the branch lung segment orifice satisfies a preset second completeness. The evaluation indexes of the image quality of the third category include at least one of the following: the image definition does not satisfy the preset definition threshold, the image pollution degree satisfies a preset third pollution degree, the shooting angle satisfies a preset third angle, and the display completeness of the branch lung segment orifice satisfies a preset third completeness. The image pollution degrees are preset first pollution degree, preset second pollution degree, and preset third pollution degree in ascending order. The shooting angles are preset first angle, preset second angle, and preset third angle in ascending order. The completenesses are preset first completeness, preset second completeness, and preset third completeness in descending order.

[0033] The third aspect of the present application provides a readable storage medium, and the readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps of the above navigation method.

[0034] The beneficial effects of the present application are: according to the model data of a plurality of different model elements, a plurality of positioning algorithms such as image positioning, magnetic positioning, robot positioning, etc. are trained, when the flexible instrument enters the pulmonary bronchus such as the pulmonary cavity in the operation, the positioning of the flexible instrument is performed in two stages through the collection of a plurality of different positioning data, and the navigation information is accurately output; the image positioning algorithm of the image data is used as the guide, the positioning information of the flexible instrument is detected, and whether the image data meets the preset condition is also detected to determine whether the image positioning is accurate or whether the image data contains sufficient significant features; when the image positioning is accurate enough, the calibration is performed in another image manner algorithm, and the accuracy of the image positioning is further improved; when the image positioning is not accurate enough, the other positioning manner is used to complete. The accuracy of the intraoperative navigation is guaranteed, and the accuracy is improved compared with the existing navigation mode. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is an embodiment schematic diagram of a catheter system in the present application;

[0036] Figure 2 is an embodiment schematic diagram of a catheter instrument in the present application;

[0037] Figure 3 is an embodiment schematic diagram of a navigation system in the present application;

[0038] Figure 4 is a flowchart schematic diagram of the first positioning stage in the present application;

[0039] Figure 5 is an embodiment schematic diagram of a first bronchial tree in the present application;

[0040] Figure 6 is an embodiment schematic diagram of a second bronchial tree in the present application;

[0041] Figure 7 is a framework schematic diagram of generating a style image in the present application;

[0042] Figure 8 is an embodiment schematic diagram of a first positioning result in the present application;

[0043] Figure 9 is a scene schematic diagram of inflection point determination in the present application;

[0044] Figure 10 is a flowchart schematic diagram of the second positioning stage in the present application;

[0045] Figure 11 is an execution flow diagram of a second positioning algorithm in the present application;

[0046] Figure 12 is a schematic diagram of an embodiment of a second positioning result in the present application;

[0047] Figure 13 is a schematic diagram of an embodiment of an overall navigation flow in the present application;

[0048] Figure 14 is a schematic diagram of another embodiment of an overall navigation flow in the present application. DETAILED DESCRIPTION

[0049] In order to facilitate the understanding of the present application, the present application will be described in more detail below in conjunction with the accompanying drawings and specific embodiments. The preferred embodiments of the present application are shown in the drawings. However, the present application can be implemented in many different forms and is not limited to the embodiments described in the specification. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0050] It should be noted that, unless otherwise defined, all technical and scientific terms used in the specification have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. For example, the term "a plurality of" includes two or more.

[0051] I. Catheter system

[0052] Figure 1 A catheter system 100 provided by an embodiment of the present application is shown. The catheter system 100 includes an image cart 110, a trolley 120 and a master 130 connected with the image cart 110 respectively, a catheter instrument 140 which can be coupled to the trolley 120, a positioning device system 150 connected with the trolley 120, and a control system 160 for realizing control among the catheter instrument 140, the master 130, the positioning device system 150 and the image cart 110, etc. Among them, the master 130 can be connected with the trolley 120 in wired or wireless manner. When an operator performs various procedures on a patient beside the trolley 120, the operator can trigger a control instruction by operating the master 130, and the catheter instrument 140 is controlled to advance, retract and bend and turn, etc. through driving of the trolley 120.

[0053] The trolley 120 can be generally moved to the side of the operating bed for engaging the catheter instrument 140 and controlling the catheter instrument 140 to be lifted in the vertical direction, or to be translated in the horizontal direction, or to be moved in a direction other than the vertical and horizontal directions, under the control of the control instruction, so as to provide a better preoperative preparation angle for the operation of the catheter instrument 140. The control instruction can be triggered by the operator through the operation of the master controller 130, or can be triggered by the operator directly through clicking or pressing the keys arranged on the trolley 120. Of course, in other embodiments, the control instruction can also be a voice control or a force feedback mechanism triggered instruction.

[0054] As shown in Figure 1 Further, the trolley 120 can include a base 121, a sliding seat body 122 that can be lifted and moved along the base 121, and two mechanical arms 123 fixedly connected with the sliding seat body 122. The mechanical arm 123 can include a plurality of arm segments coupled at joints, which provide the mechanical arm 123 with a plurality of degrees of freedom, for example, seven degrees of freedom corresponding to seven arm segments. The distal end of the mechanical arm 123 is provided with a power part (not shown in the figure), which is used to engage the catheter instrument 140 and control the distal end of the catheter instrument 140 to be correspondingly bent and turned under the driving action of the power part. The two mechanical arms 123 can be completely identical or partially identical structures, one of which is used to engage the inner catheter instrument 141 and the other of which is used to engage the outer catheter instrument 142. When installed, the outer catheter instrument 142 can be installed first, and after the outer catheter instrument 142 is installed, the catheter of the inner catheter instrument 141 is inserted into the catheter of the outer catheter instrument 142.

[0055] The positioning device system 150 has one or more subsystems for receiving information about the catheter instrument 140. The subsystems can include a position positioning device system, a shape positioning device system for determining the position, orientation, speed, rate, pose, and / or shape of the distal end of the catheter instrument 140 and / or along one or more segments of the catheter that can constitute the catheter instrument 140, and / or a visualization system for capturing images from the distal end of the catheter instrument 140.

[0056] The cart 110 can be provided with a display system 111 and a flushing system (not shown) among others. The display system 111 is used to display images or representations of the surgical site and the catheter instrument 140 generated by the subsystems of the positioning device system 150. Real-time images of the surgical site and the catheter instrument 140 captured by the visualization system can also be displayed. Image data from imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), optical coherence tomography (OCT), and ultrasound, among others, can also be used to present images of the preoperatively or intraoperatively recorded surgical site. Preoperative or intraoperative image data can be presented as two-dimensional, three-dimensional or four-dimensional (e.g., time-based or velocity-based information) images and / or as images from models created from preoperative or intraoperative image data sets, and virtual navigation images can also be displayed. In the virtual navigation images, the actual position of the catheter instrument 140 is registered with the preoperative images to present the operator with a virtual image of the catheter instrument 140 within the surgical site from the outside.

[0057] The control system 160 includes at least one memory and at least one processor. It can be appreciated that the control system 160 can be integrated in the trolley 120 or the cart 110 or can be provided independently. The control system 160 can support wireless communication protocols such as IEEE 802.11, IrDA, Bluetooth, HomeRF, DECT, and wireless telemetry, among others. The control system 160 can transmit one or more signals instructing the movement of the catheter instrument 140 by the powered portion moving the catheter instrument 140. The catheter instrument 140 can extend to the surgical location in the body via an opening of the patient's bronchial tubes or a surgical incision.

[0058] Further, the control system 160 can include a mechanical control system (not shown) for controlling the movement of the catheter instrument 140, which can be integrated in the cart 120, and an image processing system (not shown) for virtual navigation path planning, which can be integrated in the image cart 110. Of course, the subsystems of the control system 160 are not limited to the above-mentioned specific cases, and can be reasonably arranged according to actual conditions. The image processing system can use the above-mentioned imaging technology to image the surgical site based on the images of the surgical site recorded before or during the operation. Software that can be used in combination with manual input can convert the recorded images into two-dimensional or three-dimensional composite images of part or the entire anatomical organ or section. During the virtual navigation procedure, the positioning device system 150 can be used to calculate the position of the catheter instrument 140 relative to the patient's anatomical structure, which can be used to generate external tracking images and internal virtual images of the patient's anatomical structure, realize the registration of the actual position of the catheter instrument 140 with the preoperative images, and thus present the virtual image of the catheter instrument 140 in the surgical site to the operator from the outside.

[0059] The internal catheter instrument 141 and the external catheter instrument 142 have substantially the same structure and composition, and each has an elongated flexible internal catheter 41 and an external catheter 42, wherein the diameter of the external catheter 42 is slightly larger than that of the internal catheter 41, so that the internal catheter 41 can pass through the external catheter 42 and provide certain support for the internal catheter 41, so that the internal catheter 41 can reach the target position in the patient's body to facilitate tissue or cell sampling and other operations from the target position.

[0060] Some movements of the master controller 130 can cause corresponding movements of the catheter instrument 140. For example, when the operator moves the direction lever of the master controller 130 upward or downward, the movement of the direction lever of the master controller 130 can be mapped to the corresponding pitch movement of the tip of the catheter instrument 140; when the operator moves the direction lever of the master controller 130 to the left or to the right, the movement of the direction lever of the master controller 130 can be mapped to the corresponding yaw movement of the tip of the catheter instrument 140. In this embodiment, the master controller 130 can control the tip of the catheter instrument 140 to move within a 360° spatial range.

[0061] Figure 2A catheter instrument 140 provided by an embodiment of the present application is shown. The catheter instrument 140 is configured to be engaged with a powered part 124 of a mechanical arm 123, and the catheter instrument 140 includes an instrument box 43 configured to be engaged with the powered part 124 and a catheter 44 connected with the instrument box 43. Wherein, the "engagement" refers to a state that when the instrument box 43 is mounted to the powered part 124, the driving force of the powered part 124 can be transmitted to the instrument box 43 and can cause the catheter 44 to normally move. For example, under the action of the driving force of the powered part 124, the tip of the catheter 44 can be bent and turned, etc.

[0062] In the present application, the tip can also be referred to as the distal end or the head, and the front end can also be referred to as the proximal end or the tail.

[0063] II. Navigation system

[0064] For the convenience of understanding, the navigation system of the catheter system 100 is introduced first, as shown in Figure 3 The composition structure and main process of the navigation system are described, as shown below.

[0065] The navigation system 300 includes: a flexible instrument 310, provided with a plurality of positioning devices 320, each of the positioning devices 320 is used to collect different types of positioning data in real time, and at least includes an image positioning device; a display device 330, configured to display navigation information in a preset display area 331 when the flexible instrument 310 enters a lung bronchus 340; a storage 350, storing a plurality of positioning algorithms 351, wherein each of the positioning algorithms 351 is generated in association with a different constituent element of an anatomical model of the lung bronchus 340. A processor 360 is configured to execute a navigation program 361, call the positioning algorithm 351 from the storage, and update the navigation information displayed in the display device 330. Wherein, the storage 350 and the processor 360 can be arranged in a control system 370. Wherein, each of the positioning devices 320 can be connected with the display device 330 and the storage 350 respectively, and transmit the real-time collected positioning data to the two. In addition to directly displaying the positioning data, the display device 330 can also be connected with the storage 350 and the processor 370 respectively, and display the images or videos transmitted by the two. The processor 370 is also connected with the storage 350, and in addition to directly displaying the processed images or videos on the display device 330, it can also be stored in the storage 350.

[0066] Wherein, the flexible instrument can be Figure 1The flexible instrument is shown as an outer catheter instrument 142 or an inner catheter instrument 141. Exemplarily, the flexible instrument is an outer catheter instrument, and multiple inner catheter instruments are arranged therein, and the distal end of each inner catheter instrument is arranged with a different positioning device; the flexible instrument is an inner catheter instrument, and the distal end of the inner catheter instrument is arranged with an image positioning device, and other different types of positioning devices can also be integrated; in addition, the positioning device can also be arranged at the proximal end position of the flexible instrument, and the proximal end position and the distal end position of the elongated shaft.

[0067] When the processor executes the navigation program, the first positioning information of the flexible instrument in the anatomical model is identified according to the first positioning algorithm related to at least one component in the memory and at least one type of first positioning data collected by the positioning device, wherein the first positioning data at least includes image data collected by the image positioning device; the second positioning algorithm related to one of the components is obtained from the memory according to the first positioning information, and at least one type of second positioning data is obtained from the positioning device, wherein the second positioning data at least includes part of the first positioning data; the second positioning information of the flexible instrument in the anatomical model is detected according to the second positioning algorithm and the second positioning data, and the navigation information is updated based on the second positioning information.

[0068] The present application mainly uses image navigation mode as the guide and uses other navigation modes as the auxiliary to complete the whole navigation process of the flexible instrument entering the pulmonary bronchus. Therefore, the flexible instrument is arranged with at least one image positioning device, such as an endoscope, and image navigation is performed based on the collected image data. The positioning device used for auxiliary navigation includes at least one of a magnetic positioning device and a mechanical positioning device, and at least one of the magnetic positioning data collected by the magnetic positioning device and the robot data collected by the mechanical positioning device is used to perform supplementary navigation on the pulmonary bronchus segment for which the image navigation is not accurate enough.

[0069] In the embodiment, different components of the anatomical model of the pulmonary bronchus have different advantages and disadvantages for navigation in different lumen scenarios, and here the positioning algorithm related to different components of the anatomical model is applied to select the best positioning algorithm for navigation in different lumen scenarios, so as to overcome the complex scenarios encountered in the pulmonary bronchus and further improve the accuracy of navigation. The different components of the anatomical model at least include at least two of the model voxels, the first bronchial tree, the second bronchial tree and the skeleton center point set, wherein the first bronchial tree and the second bronchial tree are different in definition of the anatomical model at the bifurcation position.

[0070] The positioning algorithm related to the components of the anatomical model refers to a positioning algorithm iteratively generated using model data related to the components of the anatomical model, or model data related to the components of the anatomical model needs to be used when the positioning algorithm is executed. For example, a positioning algorithm is iteratively generated using model voxels, a first bronchial tree, a second bronchial tree, and a skeleton center point set, and the positioning data is only input into the positioning algorithm for execution when the algorithm is applied; for example, when the algorithm is applied, one of the model voxels, the first bronchial tree, the second bronchial tree, and the skeleton center point set, and the positioning data are simultaneously input into the positioning algorithm for execution.

[0071] In this embodiment, the entire navigation process is divided into two positioning stages, the first positioning stage is a coarse registration process, and the second positioning stage can be a fine registration process, or the second positioning stage can be another coarse registration process to optimize and correct the coarse registration result of the first positioning stage. Among them, the navigation process usually starts with image navigation mode, and the first positioning stage is executed; based on the positioning result, the second positioning stage of the image navigation mode is further executed, or the first positioning stage is executed again in the auxiliary navigation mode.

[0072] The first positioning stage uses first positioning data collected by a first positioning device, and the second positioning stage uses second positioning data collected by a second positioning device. The first positioning device and the second positioning device can each include one or more of the various positioning devices described above, and are only used to distinguish the positioning devices used in the first positioning stage and the second positioning stage, and are not two different positioning devices. For example, the first positioning device can include an image positioning device, a magnetic positioning device, and a mechanical positioning device, and the second positioning device can also include an image positioning device, a magnetic positioning device, and a mechanical positioning device.

[0073] Among them, the positioning algorithm related to the model voxels, the first bronchial tree, and the second bronchial tree is used for image navigation mode, and the skeleton center point set is used for auxiliary navigation mode. The positioning algorithms used in the image navigation mode of the two positioning stages and the associated model components can be the same or different. For example, the first positioning stage uses a positioning algorithm related to the first bronchial tree, and the second positioning stage can use another positioning algorithm related to the first bronchial tree, or use a positioning algorithm related to the model voxels or the second bronchial tree.

[0074] In addition, the first positioning stage is a global registration process of the flexible instrument in the anatomical model. A plurality of positioning algorithms related to different components can be used to perform registration, avoiding navigation position jumping caused by the inapplicability of the navigation mode to the scene, and improving the scene applicability. The second positioning stage belongs to a local registration process, and only one positioning algorithm is used to perform registration, avoiding the influence of the less accurate navigation mode on the more accurate navigation mode, and improving the navigation accuracy.

[0075] Specifically, the first positioning algorithm refers to the positioning algorithm used in the first positioning stage. The starting position is related to the first bronchial tree or the second bronchial tree. The first positioning data refers to the positioning data used in the first positioning stage, and the starting position is the image data. The first positioning information refers to the positioning information output by the first positioning stage, which can specifically include the bifurcation position and branch position of the flexible instrument in the anatomical model.

[0076] Specifically, the second positioning algorithm refers to the positioning algorithm used in the second positioning stage. According to the first positioning information, the positioning algorithm related to the model voxel, the first bronchial tree, the second bronchial tree or the skeleton center point set can be used. The second positioning data refers to the positioning data used in the second positioning stage, which can use image data, or use at least two of image data, magnetic positioning data and robot data.

[0077] The flexible instrument can be detachably installed on the robot arm, and the robot data refers to the related data generated when the robot arm physically drives the flexible instrument. The second positioning information refers to the positioning information output by the second positioning stage, which can specifically include the pose, position or branch label photographed by the image positioning device of the flexible instrument in the anatomical model.

[0078] In the first positioning stage, the second positioning stage and / or based on the updated navigation information, the preoperative navigation planning path in the anatomical model can be associated, the position of the flexible instrument relative to the preoperative navigation planning path can be positioned, and the navigation information can be used to prompt the operator to control the movement of the flexible instrument along the preoperative navigation planning path.

[0079] III. The first positioning stage

[0080] Please refer to Figure 4 , which provides a preferred embodiment of the first positioning stage. The following describes an embodiment of the flow of the first positioning stage of the surgical instrument, as shown below:

[0081] S41, image data-first bronchial tree detection;

[0082] In this embodiment, a first detection result is obtained by performing position detection on the flexible instrument based on a first positioning algorithm and the image data associated with the first bronchial tree, wherein the first bronchial tree includes a virtual endoscopic image acquired at each bifurcation position in the anatomical model;

[0083] Specifically, such as Figure 5 A schematic diagram of a first bronchial tree 500 is shown. Based on the topological relationship of a standard bronchial tree of pulmonary bronchi, locations within the bronchial tree containing multiple bronchopulmonary segments are defined as bifurcation nodes (e.g., bifurcation node 510), and the lung segments connecting two bifurcation nodes are defined as branches (e.g., branch 520) within the first bronchial tree. An image dataset 530 is then constructed within the anatomical model for all bifurcation locations of the first bronchial tree's bifurcation nodes. Image dataset 530 includes virtual endoscopic images captured at each bifurcation location.

[0084] When acquiring virtual endoscopic images, images are acquired at certain intervals or spatial points at each bifurcation location, along with the corresponding position, camera orientation, and angle of the image. The data element for each bifurcation node in the first bronchial tree 500 contains information such as the virtual endoscopic image, position, and posture of the bifurcation location.

[0085] In this embodiment, the first localization algorithm associated with the first bronchial tree can be: 1) a localization algorithm trained preoperatively based on virtual endoscopic images acquired at each bifurcation location in the anatomical model; or 2) a localization algorithm that uses virtual endoscopic images acquired at each bifurcation location in the anatomical model and image data for intraoperative position detection. Either of the first localization algorithms associated with the first bronchial tree can be used as needed.

[0086] In one embodiment, the first positioning algorithm includes a first target recognition model trained by the first bronchial tree, and performing position detection on the flexible device based on the first positioning algorithm related to the first bronchial tree and the image data to obtain a first detection result includes: inputting the image data into the first target recognition model, and performing position detection on the flexible device using the first target recognition model to obtain the first detection result;

[0087] The first target recognition model is trained according to the first bronchial tree, specifically comprising: acquiring real endoscopic images collected at each bifurcation position in the lung bronchus; extracting feature information in each virtual endoscopic image, and extracting style information in each real endoscopic image; synthesizing the feature information and the style information at the same bifurcation position and performing smoothing processing to obtain a style image corresponding to each virtual endoscopic image; labeling first branch information of each style image, and the first detection result comprises parameters in the first branch information; and training the first target recognition model based on the style image and the labeled information.

[0088] Specifically, in the training phase of the first target recognition model, the virtual endoscope is preprocessed for training of the first target recognition model. The preprocessing model includes a style conversion module and a smoothing processing module for the virtual endoscopic image. The style conversion module includes an encoder and a decoder, the encoder is used to extract feature information of the virtual endoscopic image, and the decoder is used to synthesize the feature information and style information of the real endoscopic image. The initial style image obtained by synthesis includes geometric, texture, structure and other key features of the lung segment in the anatomical model, and also contains the style style in the real scene, which completes the mapping of the feature containing geometric shape information of the virtual endoscopic image to the corresponding real endoscopic image domain.

[0089] As shown in the style image generation framework schematic diagram, Figure 7 for the virtual endoscopic image I C and the real endoscopic image I S , after style synthesis and smoothing processing, the style image I R can be obtained. The style synthesis of feature information and style information can be performed by the following formula to obtain the initial style image Y:

[0090]

[0091] Wherein, Pc represents the whitening conversion result of the virtual endoscopic image Ic, Ps represents the anti-whitening conversion result (i.e. style information) of the real endoscopic image Is; Hc is the feature information output by the virtual endoscopic image Ic after the encoder; indicates an encoding process mainly composed of up-sampling layers, for enlarging the spatial resolution of the initial style image;

[0092] Secondly, the smoothing processing module performs geometric shape smoothing processing on the initial style image, and a preset pixel correlation evaluation method is used to perform smoothing processing on the initial style image according to the geometric information of the virtual endoscopic image reflected by the pixel value. The maximum pixel correlation w ij , can be expressed as:

[0093]

[0094] In the formula, I i , I j represents the pixel value in the neighborhood around the pixel point (i, j); and δ represents the global scaling factor of the image.

[0095] The basic principle of the smoothing process is: (1) the same geometric shape around the pixel point has the same image style; (2) the global geometric feature offset generated in the final style image is limited as much as possible. The smoothing process can be represented by the following objective equation:

[0096] R * = F2 (Y, Ic) = (1-α) (I-αS) -1 Y

[0097] wherein, S represents I c the standard Laplacian matrix calculated, and the formula is

[0098] According to the similar content of each pixel point in the local neighborhood, the pixels should have similar styles while maintaining the effect of the global style. The above preprocessing model can be represented by a unified objective equation as follows:

[0099]

[0100] wherein, y i is the pixel value of the i-th position of the initial style image; r i is the expected smoothing value of the i-th pixel; d ii =∑ j w ij represents the diagonal elements of the angle matrix; λ is the balance coefficient of the two factor terms; after the final generated style image is filtered by wavelet transform, the geometric shape, texture and other information of the original virtual endoscope image are maintained to the greatest extent.

[0101] On the generated style image, the labeling of the lung segment name is completed, the endoscope style image training data set obtained by preprocessing all the bifurcation positions in the first bronchial tree is prepared, and the training of the first target recognition model is completed. After the image data is input into the first target recognition model, the lung segment of the first bronchial tree bifurcation position can be recognized and judged, and the first detection result can be obtained, which can include the position, name, lung segment segmentation contour and other information of the lung segment.

[0102] In one embodiment, during intraoperative navigation, the image data and the first bronchial tree are input into a positioning algorithm, and the aforementioned synthesis process and smoothing process are sequentially performed on the style information of the image data and the features of the virtual endoscopic image at each bifurcation position in the first bronchial tree to obtain a real-time style image, which is then matched with the image data, and the annotation information of the real-time style image with the highest matching degree is obtained as the first detection result.

[0103] S42, image data - second bronchial tree detection;

[0104] In this embodiment, the flexible instrument is positionally detected based on a first positioning algorithm and the image data related to the second bronchial tree to obtain a second detection result, wherein the second bronchial tree includes a visualized relationship between branches in the anatomical model;

[0105] Specifically, such as Figure 6 A schematic diagram of a second bronchial tree 600 is shown, also including bifurcation nodes (such as bifurcation node 610) and branches (such as branch 620). At each bifurcation node in the anatomical model, a weight or probability is constructed for each branch appearing simultaneously with other branches within the field of view of the image acquisition device. Finally, a corresponding weight or probability matrix 630 is set at each bifurcation node to visualize the relationship between the branches in the anatomical model, thereby obtaining the second bronchial tree 600.

[0106] In one embodiment, the first positioning algorithm further includes a second object recognition model trained by the second bronchial tree, and performing position detection on the flexible device based on the first positioning algorithm related to the second bronchial tree and the image data to obtain a second detection result includes: inputting the image data into the second object recognition model, and performing position detection on the flexible device using the second object recognition model to obtain a second detection result;

[0107] Among them, training the second target recognition model according to the second bronchial tree specifically includes: obtaining a training data set, the training data set including the second branch information annotated for each training image, and the second detection result including the parameters in the second branch information; based on the training results of the training data set, counting the number of each branch and other branches appearing in the same training image; constructing a knowledge graph based on the counted number, the knowledge graph being used to represent the visual relationship between the branches in the anatomical model; and generating a second target recognition model according to the training results and the knowledge graph.

[0108] Specifically, when the second object recognition model begins training, the parameters in the knowledge graph are initialized. The number of times each branch appears in the same training image as other branches is counted. Based on the bifurcation location in the training image, the parameters at the corresponding knowledge graph location are set and updated in the second bronchial tree, represented as probabilities or a weight matrix. At each bifurcation location, the branch preceding the bifurcation is used as the current branch of the bifurcation node, and the branch following the bifurcation is used as the other branch. The corresponding location in the knowledge graph is set.

[0109] It should be noted that when the first target recognition model is applied in the first positioning stage, there is no need to use the visualization relationship for positioning correction. The role of this visualization relationship in the first positioning stage is to assist in adjusting the weights of the feature extraction network during training, so that the first detection result of the image data is more accurate.

[0110] S43, image data - first bronchial tree positioning;

[0111] In this embodiment, if the first detection result satisfies the preset first condition, the first positioning information of the flexible instrument in the anatomical model is determined based on the first detection result; wherein, the preset first condition includes the preset first number of branches of the lung bronchi; the number of first branches refers to the number of extended visible branches in addition to the current branch where the flexible instrument is located; preferably, the number of first branches is greater than or equal to 2, which means that the extended visible position is at a bifurcation position.

[0112] S44, image data - second bronchial tree positioning;

[0113] In this embodiment, if the second detection result satisfies the preset first condition, the first positioning information is determined based on the second detection result; wherein the preset second condition includes a preset number of second branches of the pulmonary bronchus. Preferably, the number of second branches is equal to 1, indicating that the extended visible position is not at a bifurcation position.

[0114] For example, Figure 8 The first positioning information generated based on the first detection result or the second detection result is displayed on the display device in a manner that, according to the position, name, and segment segmentation contour of the lung segment branches contained in the first detection result or the second detection result, the two lung segment branches at the bifurcation position are encircled by a box, as well as the corresponding names. Figure 8 The two lung segments shown branch into the left and right mainstem bronchi.

[0115] Before that, the first positioning algorithm includes a classification network trained by the first bronchial tree or the second bronchial tree, the classification network is used to identify the quality category of the image data, and the first detection result and the second detection result include the quality category; wherein the quality category includes a first category, a second category and a third category from high to low of image quality, the first category meets the preset first condition, the second category meets the preset third condition, and the third category meets the preset second condition.

[0116] When the classification network discriminates each quality category of the image data, reference is made to indicators such as image sharpness, image pollution degree, shooting angle, and display completeness of branch lung segment port, and the specific evaluation indicators of each quality category are as follows:

[0117] The evaluation indicators of the image quality of the first category include that the image sharpness meets a preset sharpness threshold, the image pollution degree meets a preset first pollution degree, the shooting angle meets a preset first angle, and the display completeness of the branch lung segment port meets a preset first completeness;

[0118] The evaluation indicators of the image quality of the second category include that the image sharpness meets a preset sharpness threshold, the image pollution degree meets a preset second pollution degree, the shooting angle meets a preset second angle, and the display completeness of the branch lung segment port meets a preset second completeness;

[0119] The evaluation indicators of the image quality of the third category include at least one of the following: the image sharpness does not meet the preset sharpness threshold, the image pollution degree meets a preset third pollution degree, the shooting angle meets a preset third angle, and the display completeness of the branch lung segment port meets a preset third completeness;

[0120] Wherein, the image pollution degree is in order from small to large as the preset first pollution degree, the preset second pollution degree, and the preset third pollution degree, the shooting angle is in order from small to large as the preset first angle, the preset second angle, and the preset third angle, and the completeness is in order from large to small as the preset first completeness, the preset second completeness, and the preset third completeness. The shooting angle refers to the angle between the endoscope angle and the lung bronchial lumen.

[0121] According to the above evaluation indicators, the quality category of the image data is determined, the first category meets the preset first condition (set to display all complete lung segments), the second category meets the preset second condition (set to display partial complete lung segments), and the third category meets the preset third condition (set to be unable to display partial complete lung segments).

[0122] Further, the preset first condition, the preset second condition, and the preset third condition can be further specifically set as the evaluation indexes of the image quality of the three kinds.

[0123] In an implementation, when the classification network is trained, the training image set is labeled, and the degrees of the clarity, the pollution degree, the shooting angle, and the completeness of the branch lung segment port are self-defined as levels 1-10 of normalization, and the larger the number is, the greater the degree is. In the application stage, according to the input image data, the clarity, the pollution degree, the shooting angle, and the completeness of the branch lung segment port of any level in 1-10 are output.

[0124] Specifically, the clarity threshold can be preset as [9, 10], the first pollution degree can be preset as [1, 2], the first shooting angle can be preset as [1, 2], and the first completeness can be preset as [9, 10]; the second pollution degree can be preset as [3, 5], the second shooting angle can be preset as [3, 5], and the second completeness can be preset as [6, 8]; the third pollution degree can be preset as [6, 10], the third shooting angle can be preset as [6, 10], and the third completeness can be preset as [1, 5].

[0125] When the image data meets the preset clarity, the endoscope angle is parallel to the central lung bronchus lumen, and the lung holes and related features are all displayed completely (without pollution and occlusion), the image data contains sufficient significant features for image positioning, and the accuracy of image positioning is ensured.

[0126] In another implementation, when the first detection result meets the preset first condition and the second detection result does not meet the preset first condition, or the second detection result meets the preset first condition and the first detection result does not meet the preset first condition, the first detection result or the second detection result that meets the preset first condition is directly used to generate the first positioning information.

[0127] The first positioning algorithm related to the first bronchial tree is suitable for the scene in which the lung holes are displayed completely and the related features of each lung hole are displayed obviously. The first positioning algorithm related to the second bronchial tree is more suitable for the scene in which part of the lung holes are displayed completely and part of the lung holes are occluded.

[0128] S45, magnetic positioning data-skeleton center point set positioning;

[0129] In the embodiment, when the first detection result or the second detection result meets the preset second condition, the flexible instrument is positionally detected according to the first positioning algorithm related to the skeleton center point set and the magnetic positioning data, a third detection result is obtained, and the first positioning information is determined based on the third detection result.

[0130] In this embodiment, if the image data does not meet the preset definition, the endoscope angle, and the distance between the pulmonary bronchus lumen and the central parallel line is greater than the preset angle, and the complete lung hole and related features cannot be displayed (the degree of pollution obstruction is serious), it means that sufficient significant features cannot be extracted from the image data, so that it can be identified that the image data is collected at which position. In this case, it may cause a large positioning deviation, or even jump to other bifurcation positions. Therefore, the first detection result of the first bronchial tree positioning or the second detection result of the second bronchial tree positioning is not accurate enough, and the skeleton center point set can be used for positioning.

[0131] Specifically, for the collection of the skeleton center point set, first, the skeleton center line of the anatomical model is resampled, and the sampling interval Δd (preferably 0.1 mm) is set according to the requirement. After cubic spline interpolation, the resampled data is obtained. Further, according to the preoperative navigation planning path, the number set {id n} of the skeleton center point set of the planning path can be obtained.

[0132] The conversion relationship between the preoperative construction of the skeleton center point set and the magnetic positioning data can be selected from a plurality of (preferably 4-5) skeleton center points with obvious features in the bronchial anatomical structure, such as selecting the main carina and the first and second level carina of the left lung and the right lung. The electromagnetic positioning data of each selected skeleton center point is obtained by the magnetic positioning device, and the conversion matrix T is used to minimize the objective function to complete the registration and obtain the conversion matrix T0.

[0133] After obtaining the conversion matrix T0, the real-time collected magnetic positioning data P i can be converted to obtain the electromagnetic positioning point P i* in the anatomical model coordinate system, and T0 is used as the actual positioning point of the magnetic positioning data to perform projection to the skeleton center point set; through the Euclidean distance calculation, the nearest skeleton center point EM i of the actual positioning point on the planning path can be found, and according to the number set {id n} of the skeleton center point set of the planning path, the corresponding id number is found, which is the mapping serial number ID EMi of the electromagnetic positioning point on the planning path.

[0134] Specifically, the skeleton center point set includes the inflection points of the curved segments in the anatomical model, the first positioning information is related to the inflection points, and the curved segment refers to the segment in the anatomical model that is greater than the preset bending degree. That is, the electromagnetic positioning is mainly used for the positioning of the segment in the anatomical model that is greater than the preset bending degree.

[0135] Furthermore, the aforementioned projection method of electromagnetic positioning may cause projection jumps at positions with large bending angles, such as Figure 9 As shown, when the magnetic positioning data is converted to point P on the anatomical model coordinate system, the aforementioned projection method can be used to jump from the skeleton center point A to the skeleton center point B, resulting in a navigation jump. At this time, the path feature inflection point can be used to determine the inflection point to perform electromagnetic positioning in the curved segment of the anatomical model.

[0136] In one embodiment, the position of the flexible instrument is detected based on the first positioning algorithm related to the skeleton center point set and the magnetic positioning data, and the third detection result obtained includes: mapping the position of the flexible instrument to the first target center point in the skeleton center point set according to the magnetic positioning data, and determining the inflection point and the second target center point in the forward direction of the flexible instrument from the skeleton center point set, the second target center point being located on a different branch of the first target center point relative to the inflection point; determining the relative position relationship between the first target center point, the inflection point and the second target center point, and determining the third detection result of the flexible instrument based on the relative position relationship, the third detection result including at least: the flexible instrument entering the curved segment and leaving the curved segment.

[0137] Specifically, a set of inflection points {K n}, in the forward direction of the flexible device, the inflection point K closest to the center point of the first target i , i∈n, and determining the magnetic positioning data in addition to the branch where the first target center point is located, another center point projected on other branches, determining that the magnetic positioning data is easy to jump to the other center point, and setting it as the second target center point.

[0138] In this embodiment, determining the relative position relationship between the first target center point, the inflection point and the second target center point, and determining the third detection result of the flexible device based on the relative position relationship includes: calculating the curve integral distance between the first target center point and the inflection point, and determining whether the flexible device enters the curved segment based on the curve integral distance; if the flexible device enters the curved segment, calculating the bending angle between the first target center point, the inflection point and the second target center point, and determining whether the flexible device is in the turning area in the curved segment based on the bending angle; judging whether the first target center point exceeds the second target center point, and determining whether the flexible device leaves the curved segment based on the judgment result; wherein, the third detection result also includes that the flexible device is in the turning area.

[0139] For details, see Figure 9, first, according to the real-time positioning point P on the anatomical model magnetic positioning data, using the nearest point projection to obtain the first target center A, and calculate the current first target center A and the curve integral distance of the nearest corner C, when the distance is less than the threshold value ε (preferably set to 5mm), the first target center point A has entered the curved segment.

[0140] In the curved segment, according to the first target center A, the corner C, and the second target center point B, the three points are determined by the bending angle, and the specific determination is: Greater than 0 and less than 1, it is considered that the flexible instrument is in the corner area, otherwise it is not in the corner area; wherein the second target center point B can also be the skeleton center point with a distance of threshold value ε from the corner C. When the first target center point A exceeds the second target center point B, it is determined that the flexible instrument leaves the curved segment.

[0141] S46, multi-data fusion positioning;

[0142] In this embodiment, if the first detection result or the second detection result satisfies the preset third condition, the position of the flexible instrument is detected according to the first positioning algorithm related to the skeleton center point set and the robot data, and the fourth detection result is obtained; the extension amount of the flexible instrument is determined according to the robot data, and the ratio of the extension amount to the preset reference extension amount is calculated; according to the ratio, the weight of the first detection result or the second detection result is set, and the weight of the third detection result and the fourth detection result is set; according to the set weight, the first detection result or the second detection result, and the third detection result and the fourth detection result are executed weighted processing, and the first positioning information is obtained.

[0143] In this embodiment, if the image data satisfies the preset definition, but the endoscope angle and the distance between the pulmonary bronchial lumen and the central parallel line are within the preset angle, and part of the lung hole can be displayed completely (the degree of pollution blocking is not serious), it is indicated that part of the salient feature can be extracted in the image data. At this time, the image positioning method has a certain accuracy, but the accuracy is not high enough, and other positioning methods such as electromagnetic positioning method and robot positioning method can be fused.

[0144] Specifically, after obtaining the detection results of multiple positioning methods, different positioning results can be dynamically modified according to the different lung bronchial regions where the flexible instrument is located. The weight distribution is not only related to the depth of the region where the flexible instrument is located, but also related to the root mean square error of the magnetic positioning data.

[0145] For example, the weight of the magnetic positioning data decreases as the flexible instrument enters the deep lung bronchial area, due to the influence of organ movement in the human body and the respiratory movement of the deep lung segment lumen; while the weight of the robot data and the image data is not affected by the depth of the area, so the weight relationship of the three can be divided according to the above factors, as shown below:

[0146] The weight expression of the fourth detection result is: W Rob = ΔL / (L plan -L0)*0.5;

[0147] The weight expression of the third detection result is: W EM =(1-Δ L / (L plan -L0))*θ EM ;

[0148] The weight expression of the first or second detection result is: W Img = ΔL / (L plan -L0)*0.5;

[0149] Wherein, Δ L is the incremental data of the flexible instrument relative to the reference expansion L0, L plan is the total length of the preoperative navigation planning path. The reference expansion L0 determines the influence degree of the depth of the flexible instrument entering the lung bronchial area on the weight distribution.

[0150] In addition, due to the complexity of the lung bronchus, the magnetic positioning data has a registration accuracy problem, so an adjustment factor θ EM is given to it, which is related to the current registration accuracy index σ of the magnetic positioning data: When θ EM > σ0, wherein σ0 refers to the standard value of the root mean square error of registration, which can be freely set according to different scene requirements, and is preferably set to 3.

[0151] Therefore, the final positioning ID result is: Next, according to the ID, the corresponding positioning point in the number set {id n} of the aforementioned skeleton center point set is obtained, and the final first positioning information is obtained.

[0152] Four, the second positioning stage

[0153] Please refer to Figure 10 , the following describes an embodiment of the flow of the second positioning stage of the surgical instrument, as shown below:

[0154] S101, image data-first bronchoscope positioning;

[0155] In the embodiment, when the first positioning information is determined by the first detection result or the second detection result, the first positioning information includes parameters of at least two first branches, and the at least two first branches satisfy a preset first display effect in the image data, the first display effect is determined by a preset first condition; the image data is subjected to three-dimensional reconstruction to generate a first model voxel of a lung segment in the image data; a second model voxel of the at least two first branches in the anatomical model is intercepted; the first model voxel is projected to the second model voxel, and according to a projection result, second positioning information of the flexible instrument in the anatomical model is determined, the second positioning information including pose information of the flexible instrument.

[0156] Specifically, the first positioning information generated based on the first detection result or the second detection result includes parameters of at least two first branches, indicating that the first detection stage detects that the image data includes the at least two first branches, and the parameters include position, name, lung segment segmentation contour and other information.

[0157] For example, if the indicators of the preset first condition include that the image satisfies a preset clarity threshold, an endoscope angle and a lung bronchus cavity included angle are between 0 and A, a lung hole and related features satisfy a larger first completeness M, and a pollution block satisfies a smaller first pollution degree N, then the first display effect is indicated by the following indicators: the image satisfies the preset clarity threshold, the endoscope angle and the lung bronchus cavity included angle are between A / 2 and A, the lung hole and related features satisfy the larger first completeness M, and the pollution block satisfies the smaller first pollution degree N / 2. In addition, other indicators can also be used for indication, for example, the number of lung holes is less than or equal to a preset number (preferably 3).

[0158] In the embodiment, for the image data collected by the image collection device such as a pure monocular endoscope, there are problems such as low accuracy and loss of information when performing visual reconstruction, so the method of computer vision and graphics can be used to improve the efficiency of detail preservation and three-dimensional matching in the reconstruction process through target detection.

[0159] Specifically, the process of monocular visual three-dimensional reconstruction mainly includes camera calibration, feature extraction, feature matching, motion estimation and three-dimensional reconstruction: using a feature point detection algorithm such as SIFT, SURF, etc., to detect feature points in multiple image data, and using a feature point matching algorithm to match the feature points of each image data; according to the matching results of the feature points and the camera parameters, the point cloud volume features of the lung segment in the field of view are reconstructed by using the triangulation method, and the anatomical model corresponding to the image data is obtained after optimization, forming the first model voxel of the lung segment.

[0160] The first positioning stage locates the initial position of the instrument, such as the bifurcation position, and directly segments all first branches of the bifurcation position on the anatomical model to obtain a second model voxel. At this time, the second model voxel contains the complete first branch, and the first model voxel contains the partial first branch. The first model voxel is projected into the second model voxel, and the shooting position of the image data is determined according to the known spatial position information of the overlapping part of the projection in the second model voxel, and the pose information of the flexible instrument is determined according to the virtual endoscope pose for shooting the overlapping part of the projection.

[0161] S102, image data-second bronchoscope positioning;

[0162] In an embodiment, when the first positioning information is determined by the first detection result or the second detection result, the first positioning information includes the identification probability of at least two second branches, and the at least two second branches satisfy a preset second display effect, the second display effect is determined by a preset first condition; the identification probability is adjusted according to the visual relationship of the at least two second branches; and / or, the identification probability is adjusted according to the distance between the at least two second branches; and the second positioning information of the flexible instrument in the anatomical model is determined based on the adjusted identification probability, wherein the second positioning information includes the parameters of the second branch within the field of view of the image positioning device.

[0163] Specifically, the first positioning information generated based on the first detection result or the second detection result includes the identification probability of at least two second branches, preferably the identification probability of all branches in the anatomical model, indicating the possibility that the second branch contained in the image data is identified as which branch in the anatomical model in the first positioning stage.

[0164] For example, if the indicators of the preset first condition include: the image satisfies a preset clarity threshold, the endoscope angle and the lung bronchus cavity included angle are between [0, A], the lung hole and related features satisfy a larger first completeness M, and the pollution shielding satisfies a smaller first pollution degree N; the first display effect is indicated by the following indicators: the image satisfies the preset clarity, the endoscope angle and the lung bronchus cavity included angle are between [0, A / 2], the lung hole and related features satisfy a smaller first completeness M / 2, and the pollution shielding satisfies a larger first pollution degree N. In addition, it can also be indicated by other indicators, for example: the number of lung holes is greater than a preset number (preferably 3).

[0165] In one embodiment, based on the visual relationships between the second branches, as reflected in the knowledge graph, the output of the second localization algorithm takes into account two pieces of empirical information: the detection results of other lung segments in the current image and the detection status of the previous prediction result. This improves the stability of the second localization algorithm and enhances the accuracy of lung segment detection in special situations such as lung segment deformation and occlusion, thereby improving the accuracy of image navigation.

[0166] In this embodiment, the knowledge graph is stored in the form of a weight matrix. The basic rule of the weight matrix is ​​that branches connected to the current branch are weighted more heavily than branches not connected to it, and branches with closer connections are weighted more heavily than branches with more distant connections. The minimum weight can be preset to 0, and the maximum weight is determined by actual performance testing of the first object detection model during the prediction phase.

[0167] In addition, for the case where different lung segment branches are displayed between the previous and subsequent image data, a directed graph or an undirected graph is constructed with the lung segment bifurcations as nodes and the lung segments as directed edges; according to each second branch in the directed or undirected graph, the inverse of the number of edges passing through the shortest path relative to the second branch displayed by the earlier collected image data is used as a reference and dynamically adjusted to a certain threshold to adjust the recognition probability.

[0168] like Figure 11 As shown, after the image data 1110 is input into the first target recognition model or the second target recognition model 1120, the recognition probability matrix 1130 is output; the second bronchial tree is named 1140, such as FC1, FC2..., FC12, etc. are the names of the bifurcation nodes, such as Trachea, RMB, LMB,..., B5, B10L, etc. are the names of the branches, and the knowledge graph weight matrix 1150 of each branch is constructed at the same time; the recognition probability matrix 1130 is adjusted using the weight matrix 1150 to obtain the adjusted recognition probability matrix 1160, and the parameters of the second branch contained in the image data are set according to the branch with the highest probability in the matrix 1160.

[0169] Further, such as Figure 12As shown, according to the second positioning information, a planning path prompt is displayed on the display device, when the parameters of the second branch contained in the image data 1210 are identified, such as the lung segment classification Airway ID: left main bronchus (LMB), right main bronchus (RMB); and the lung segment classification Airway ID set passed in the preoperative navigation planning path 1220 (dashed line part) in the bronchial tree has an intersection, it can be judged that the segment RMB in the image data 1210 contains the lung segment RMB on the planning path 1220. In the image data 1230 displayed on the display device, the name of the lung segment port is displayed as the name of the right main bronchus i. Wherein, the name of the lung segment is determined by the dictionary of the classification Airway ID and the name. The operator only needs to mark according to the mark on the real-time endoscopic image, and the lung segment where the lesion is located can be reached.

[0170] S103, robot data-skeleton center point positioning

[0171] In an embodiment, when the flexible instrument is in the bending area, a reference zero point is set in the skeleton center point set based on the magnetic positioning data and / or the image data; a displacement of the flexible instrument is calculated according to the robot data, taking the reference zero point as the starting position; a position where the flexible instrument is located is mapped to a third target center point in the skeleton center point set according to the displacement; and the second positioning information of the flexible instrument in the anatomical model is determined according to the third target center point, the second positioning information including the position information of the flexible instrument.

[0172] In this embodiment, for robot data positioning, a reference zero point needs to be determined first, which can be determined by using magnetic positioning data or image data to assist positioning, and using nearest point mapping or projection mapping on the skeleton center point set to map the position as the reference zero point; subsequent changes in the mechanical arm stretching amount represented by the robot data are calculated based on the reference zero point to obtain the second positioning information of the robot data.

[0173] In an embodiment, the reference zero point is set by using magnetic positioning data or image data to assist, and the reference stretching amount L0 of the current flexible instrument is recorded; the stretching amount L of the flexible instrument is obtained according to the robot data i , the displacement ΔL=L i -L0 is calculated; the offset position along the skeleton center point set based on the reference zero point is calculated according to the displacement ΔL, which is taken as the positioning point Rob i of the robot data; after the positioning point is obtained, the corresponding id number of the skeleton center point set can be found according to the id number set {id n}, and the id is the mapping serial number ID Robi of the robot data positioning point on the planning path. According to the mapping serial number ID RobiThe position coordinates of the flexible instrument in the coordinate system of the anatomical model can be determined.

[0174] V. Overall navigation process

[0175] Referring to Figure 13 The overall navigation process of the surgical instrument is described as follows:

[0176] S131, acquire different types of positioning data collected in real time by a plurality of positioning devices, the plurality of positioning devices are arranged on the flexible instrument, and at least include an image positioning device, and the flexible instrument is controlled to enter the lung bronchus;

[0177] S132, according to at least one first positioning algorithm in a plurality of preset positioning algorithms and at least one type of first positioning data, identify first positioning information of the flexible instrument in the anatomical model of the lung bronchus, wherein the first positioning data at least includes image data collected by the image positioning device, and each of the preset positioning algorithms is generated in relation to different constituent elements of the anatomical model;

[0178] S133, according to the first positioning information, select a second positioning algorithm related to one of the constituent elements and at least one type of second positioning data, wherein the second positioning data at least includes part of the first positioning data;

[0179] S134, according to the second positioning algorithm and the second positioning data, detect second positioning information of the flexible instrument in the anatomical model, and update the navigation information based on the second positioning information.

[0180] For example, as Figure 14 shown, an embodiment schematic diagram of an overall navigation process is provided. For the first positioning algorithm, the first positioning algorithm related to the first bronchial tree and the first positioning algorithm related to the second bronchial tree are used to perform position detection with the image data respectively; when the first detection result or the second detection result meets a preset first condition, the first detection result or the second detection result is directly used to perform positioning; if a preset second condition is met, the first positioning algorithm related to the skeleton center point set needs to be used again to perform position detection with the magnetic positioning data; if a preset third condition is met, the multi-data fusion scheme of image positioning, magnetic positioning and robot positioning needs to be used again to perform positioning.

[0181] For the second positioning algorithm, if the first detection result or the second detection result meets a preset second condition, the image data is not accurate enough, so the second positioning algorithm related to the skeleton center point set is used to perform positioning in combination with the magnetic positioning data and the robot data. The multi-data fusion mode indicates that multiple positioning data are available, so each second positioning algorithm can perform deep position calibration. Finally, the second positioning algorithm related to the first bronchial tree and the second positioning algorithm related to the second bronchial tree are used in different scenarios, and the related second positioning algorithm is performed based on the first display effect or the second display effect. Finally, appropriate second positioning information is output.

[0182] It should be noted that other sorting schemes that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should also be within the protection scope of the present application, and will not be described here.

[0183] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional units or modules according to needs, that is, the internal structure of the mobile terminal is divided into different functional units or modules to complete all or part of the functions described above. Each functional module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of the functional modules are only for the convenience of mutual differentiation, and do not limit the protection scope of the present application. The specific working process of the modules in the mobile terminal can refer to the corresponding process in the foregoing method embodiments, and will not be described here.

[0184] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0185] The embodiment of the present application provides a computer program product, which, when running on a mobile terminal, enables the mobile terminal to execute the steps in the above-mentioned various method embodiments.

[0186] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0187] It should also be understood that, in the description of the present application and in the appended claims, the term "and / or" is used to mean one or more of the associated listed items, as well as the sum of all possible combinations of the associated listed items. It should also be understood that, in the description of the present application and in the appended claims, the term "comprises" or "comprising" or "includes" or "including" is used to mean one or more of the associated listed items, as well as the sum of all possible combinations of the associated listed items.

[0188] As used in the description of the application and the appended claims, the term "if' can be interpreted to mean "when" or "upon" or "in response to determining" or "in response to detecting." Similarly, the phrase "if a described condition or event occurs" can be interpreted to mean "if a determined" or "in response to determining" or "if the described condition or event is detected" or "in response to detecting the described condition or event."

[0189] In addition, in the description of the present application and in the appended claims, the terms "first", "second", "third", etc. are used merely to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0190] Reference in the specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms "comprising," "including," "having" and their variations, as used in the specification and in the appended claims, mean "including but not limited to," unless otherwise expressly specified and as otherwise evident from context.

[0191] It can be clearly understood by a person skilled in the art that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for description, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction, and do not limit the protection scope of the application. The specific working process of the units and modules in the system can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0192] In the above embodiments, the description of each embodiment is focused on, and the parts not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0193] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0194] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0195] If the integrated module / unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by computer programs instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0196] The above examples are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing examples, those ordinarily skilled in the art should understand: the technical solutions recorded in the foregoing examples can still be modified, or some technical features can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A navigation system, characterized in that: The navigation system includes: A flexible apparatus is provided with a first positioning device and a second positioning device, wherein the first positioning device is used to collect first positioning data, and the second positioning device is used to collect second positioning data, the first positioning device at least includes an image positioning device, and the first positioning data at least includes image data collected by the image positioning device; a display device configured to display navigation information when the flexible instrument enters the lung bronchus; A memory storing a plurality of positioning algorithms, wherein the positioning algorithms include a first positioning algorithm and a second positioning algorithm, wherein generation of each positioning algorithm is related to a different component of the anatomical model of the lung bronchi; The processor is configured to: identifying first positioning information of the flexible instrument on the anatomical model according to the first positioning algorithm and the first positioning data; determining second positioning information of the flexible instrument on the anatomical model according to the first positioning information, the second positioning algorithm, and the second positioning data; The navigation information is updated based on the second positioning information.

2. The navigation system according to claim 1, wherein: The first positioning device and the second positioning device further include at least one of a magnetic positioning device and a mechanical positioning device, the first positioning data and the second positioning data include at least one of the following: magnetic positioning data collected by the magnetic positioning device, and robot data collected by the mechanical positioning device, wherein the flexible instrument is detachably mounted on a robot arm, and the robot data refers to relevant data generated when the robot arm physically drives the flexible instrument; The different components of the anatomical model of the lung bronchi include: model voxels, a first bronchial tree, a second bronchial tree, and at least two items of a skeleton center point set, wherein the first bronchial tree and the second bronchial tree define the anatomical model at a bifurcation position differently.

3. The navigation system according to claim 2, characterized in that The identifying, according to the first positioning algorithm and the first positioning data, first positioning information of the flexible instrument on the anatomical model comprises: performing position detection on the flexible instrument according to a first positioning algorithm associated with the first bronchial tree and the image data to obtain a first detection result, the first bronchial tree comprising a virtual endoscopic image acquired at each bifurcation location in the anatomical model; If the first detection result satisfies a preset first condition, determining first positioning information of the flexible instrument on the anatomical model based on the first detection result; or, performing position detection on the flexible instrument based on a first positioning algorithm associated with the second bronchial tree and the image data to obtain a second detection result, the second bronchial tree including a visualized relationship of a plurality of branches in the anatomical model; If the second detection result satisfies the preset first condition, determining first positioning information based on the second detection result; The preset first condition at least includes a preset first branch quantity.

4. The navigation system according to claim 3, characterized in that After obtaining the first detection result or the second detection result, the method further includes: If the first detection result or the second detection result satisfies a preset second condition, performing position detection on the flexible device according to a first positioning algorithm related to the skeleton center point set and the magnetic positioning data to obtain a third detection result, and determining first positioning information based on the third detection result; Among them, the skeleton center point set includes the inflection point of the curved segment in the anatomical model, the first positioning information is related to the inflection point, the curved segment refers to the segment in the anatomical model that is greater than a preset degree of curvature, and the preset second condition at least includes a preset number of second branches.

5. The navigation system according to claim 3, characterized in that The first positioning algorithm includes a first target recognition model trained by the first bronchial tree, and performing position detection on the flexible instrument based on the first positioning algorithm related to the first bronchial tree and the image data to obtain a first detection result includes: inputting the image data into the first target recognition model, and performing position detection on the flexible instrument using the first target recognition model to obtain the first detection result; Training the first target recognition model according to the first bronchial tree includes: Acquiring a real endoscopic image collected at each bifurcation position in the pulmonary bronchus; extracting feature information from each of the virtual endoscopic images, and extracting style information from each of the real endoscopic images; synthesizing the feature information and the style information at the same bifurcation position and performing smoothing processing to obtain a style image corresponding to each virtual endoscopic image; marking first branch information of each of the style images, wherein the first detection result includes parameters in the first branch information; The first object recognition model is obtained by training based on the style image and the annotation information.

6. The navigation system according to claim 3 or 5, characterized in that: Detecting second positioning information of the flexible instrument on the anatomical model according to the first positioning information, the second positioning algorithm, and the second positioning data includes: When the first positioning information is determined by the first detection result or the second detection result, determining that the first positioning information includes parameters of at least two first branches, and the at least two first branches satisfy a preset first display effect in the image data, where the first display effect is determined by a preset first condition; Performing three-dimensional reconstruction on the image data to generate a first model voxel of a lung segment in the image data; intercepting second model voxels of the at least two first branches in the anatomical model; The first model voxel is projected onto the second model voxel, and second positioning information of the flexible instrument in the anatomical model is determined according to the projection result, where the second positioning information includes posture information of the flexible instrument.

7. The navigation system according to claim 3, characterized in that The first positioning algorithm further includes a second target recognition model trained by the second bronchial tree, and performing position detection on the flexible device based on the first positioning algorithm related to the second bronchial tree and the image data to obtain a second detection result includes: inputting the image data into the second target recognition model, and performing position detection on the flexible device using the second target recognition model to obtain a second detection result; Training the second target recognition model according to the second bronchial tree includes: Acquire a training data set, where the training data set includes second branch information annotated for each training image, and the second detection result includes parameters in the second branch information; Based on the training results of the training data set, counting the number of each branch and other branches that appear in the same training image; Constructing a knowledge graph based on the statistical quantity, wherein the knowledge graph is used to represent the visual relationship between each branch in the anatomical model; A second target recognition model is generated based on the training results and the knowledge graph.

8. The navigation system according to claim 3 or 7, characterized in that: Detecting second positioning information of the flexible instrument on the anatomical model according to the first positioning information, the second positioning algorithm, and the second positioning data includes: When the first positioning information is determined by the first detection result or the second detection result, determining that the first positioning information includes an identification probability of at least two second branches, and the at least two second branches satisfy a preset second display effect, and the second display effect is determined by a preset first condition; adjusting the recognition probability according to the visual relationship between the at least two second branches; and / or adjusting the recognition probability according to the distance between the at least two second branches; Based on the adjusted recognition probability, second positioning information of the flexible instrument on the anatomical model is determined, wherein the second positioning information includes parameters of the second branch within the field of view of the image positioning device.

9. The navigation system according to claim 4, characterized in that The performing position detection on the flexible device according to the first positioning algorithm related to the skeleton center point set and the magnetic positioning data to obtain a third detection result includes: Mapping the position of the flexible device to a first target center point in the skeleton center point set based on the magnetic positioning data, and determining an inflection point and a second target center point in a forward direction of the flexible device from the skeleton center point set, wherein the second target center point is located on a different branch of the first target center point relative to the inflection point; Determine the relative position relationship between the first target center point, the inflection point and the second target center point, and determine a third detection result of the flexible instrument based on the relative position relationship, the third detection result at least including: the flexible instrument entering the curved segment and leaving the curved segment.

10. The navigation system according to claim 9, characterized in that Determining the relative positional relationship between the first target center point, the inflection point, and the second target center point, and determining a third detection result of the flexible device based on the relative positional relationship includes: calculating a curve integral distance between the first target center point and the inflection point, and determining whether the flexible instrument enters a curved segment based on the curve integral distance; If the flexible device enters the curved segment, calculating a bending angle between the first target center point, the inflection point, and the second target center point, and determining whether the flexible device is in a turning area in the curved segment based on the bending angle; determining whether the first target center point exceeds the second target center point, and determining whether the flexible device leaves the curved segment based on the determination result; The third detection result further includes that the flexible device is in the turning area.

11. The navigation system according to claim 10, characterized in that Detecting second positioning information of the flexible instrument on the anatomical model according to the first positioning information, the second positioning algorithm, and the second positioning data includes: When the flexible device is in the turning area, a reference zero point is centrally set at the center point of the skeleton based on the magnetic positioning data and / or the image data; Taking the reference zero point as a starting position, and calculating the offset of the flexible instrument according to the robot data; Mapping the position of the flexible device to a third target center point in the skeleton center point set according to the offset; Second positioning information of the flexible instrument on the anatomical model is determined according to the third target center point, where the second positioning information includes position information of the flexible instrument.

12. The navigation system according to claim 4, characterized in that The first positioning algorithm includes a classification network trained by the first bronchial tree or the second bronchial tree, the classification network is used to identify a quality category of the image data, and the first detection result and the second detection result also include the quality category; The quality category includes a first category and a second category, the image quality corresponding to the first category is higher than the image quality corresponding to the second category, the first category meets the preset first condition, and the second category meets the preset second condition.

13. The navigation system according to claim 12, characterized in that The quality category further includes a third category, the image quality corresponding to the third category is lower than the image quality corresponding to the second category, and the third category meets a preset third condition. After obtaining the first detection result or the second detection result, the method further includes: If the first detection result or the second detection result satisfies the preset third condition, performing position detection on the flexible device according to the first positioning algorithm related to the skeleton center point set and the robot data to obtain a fourth detection result; determining an extension and contraction amount of the flexible device according to the robot data, and calculating a ratio of the extension and contraction amount to a preset reference extension and contraction amount; Setting a weight for the first detection result or the second detection result according to the ratio, and setting a weight for the third detection result and the fourth detection result according to the ratio; According to the set weight, weighted processing is performed on one of the first detection result and the second detection result in combination with the third detection result and the fourth detection result to obtain first positioning information.

14. The navigation system according to claim 13, wherein: The evaluation indicators of the image quality of the first category include: the image clarity meets a preset clarity threshold, the image pollution level meets a preset first pollution level, the shooting angle meets a preset first angle, and the completeness of the display of the branch lung segment meets a preset first completeness level; The evaluation indicators of the second category of image quality include: the image clarity meets a preset clarity threshold, the image pollution level meets a preset second pollution level, the shooting angle meets a preset second angle, and the completeness of the display of the branch lung segment meets a preset second completeness level; The evaluation index of the image quality of the third category includes at least one of the following: the image clarity does not meet the preset clarity threshold, the image pollution level meets the preset third pollution level, the shooting angle meets the preset third angle, and the completeness of the display of the branch lung segment meets the preset third completeness level; Among them, the image pollution levels are, from small to large, a preset first pollution level, a preset second pollution level, and a preset third pollution level; the shooting angles are, from small to large, a preset first angle, a preset second angle, and a preset third angle; and the completeness levels are, from large to small, a preset first completeness level, a preset second completeness level, and a preset third completeness level.

15. A navigation method, characterized in that: The method comprises: Acquiring different types of positioning data collected in real time by multiple positioning devices, wherein the multiple positioning devices are provided on a flexible instrument and include at least an image positioning device, and the flexible instrument is controlled to enter the lung bronchus; identifying first positioning information of the flexible instrument in the anatomical model of the lung bronchus based on at least one first positioning algorithm among a plurality of preset positioning algorithms and at least one type of first positioning data, wherein the first positioning data at least includes image data acquired by the image positioning device, and each of the preset positioning algorithms is generated in association with a different component of the anatomical model; selecting, based on the first positioning information, a second positioning algorithm associated with one of the components and at least one type of second positioning data, wherein the second positioning data includes at least a portion of the first positioning data; According to the second positioning algorithm and the second positioning data, second positioning information of the flexible instrument on the anatomical model is detected, and navigation information is updated based on the second positioning information.

16. A readable storage medium, characterized in that The readable storage medium stores a computer program, which implements the steps of the navigation method according to claim 15 when executed by a processor.

Images