Catheter positioning method, interventional surgery system, electronic device and storage medium

Abstract

The present invention provides a catheter positioning method, an interventional surgery system, an electronic device, and a storage medium. The catheter positioning method includes obtaining a current frame real endoscopic image collected by an image acquisition device in a target organ; obtaining a feature vector of the current frame real endoscopic image; searching for a feature vector that matches the feature vector of the current frame real endoscopic image in a pre-obtained feature vector set; and obtaining the current pose of the end of the catheter according to the pose corresponding to the virtual endoscopic image corresponding to the matching feature vector in the feature vector set. The present invention does not need to rely on other positioning devices, such as an electromagnetic positioning system, so it can effectively reduce the constraints on the environment and patients by the interventional surgery system and expand the applicable range of the interventional surgery system.

Description

Technical Field

[0001] The present invention relates to the technical field of navigation, and particularly relates to a catheter positioning method, an interventional surgery system, an electronic device, and a storage medium. Background Art

[0002] Minimally invasive interventional technology is a surgical method that, under the joint guidance of preoperative medical images (such as CT) and intraoperative imaging technologies (such as medical electronic endoscopes, minimally invasive interventional magnetic resonance, and ultrasonic imaging), places instruments or drugs into the diseased tissue with the least trauma for diagnosis or treatment. Minimally invasive interventional technology has been widely used worldwide due to its advantages of less bleeding, less trauma, fewer complications, safety and reliability, and rapid postoperative recovery. As the most important tool in interventional technology, the doctor remotely controls the movement of the catheter to reach the diseased position for interventional treatment or diagnosis with the help of image navigation.

[0003] Catheter surgery involves inserting a long and slender catheter with a bendable tip into the lower respiratory tract of a patient through the mouth or nose, that is, entering the trachea, bronchi, and more distal parts through the glottis, directly observing the lesions of the trachea and bronchi through an image acquisition device installed at the tip of the catheter, and performing corresponding examinations and treatments according to the lesions. In the actual operation process, the doctor holds the catheter, observes the situation inside the bronchus in real time through the images collected by the image acquisition device, controls the telescopic or bending of the catheter, and performs examinations and treatments after reaching the target position.

[0004] For bronchial examinations based on a positioning system, an external positioning system, such as an electromagnetic positioning system, can be used to establish the mapping relationship between the real human lungs and medical images. When the doctor manipulates the catheter, the position and posture of the catheter tip in the lungs can be prompted in real time. However, introducing an electromagnetic positioning system will expand the range of patient contraindications, such as being unable to perform examinations on patients with implanted cardiac pacemakers or defibrillators, and the electromagnetic positioning system is also relatively sensitive to the surrounding environment. In addition, the existing electromagnetic positioning systems cannot be used in combination with X-ray machines / CT machines during the operation, and after the registration of the existing electromagnetic positioning systems is completed, the relative position and posture between the patient and the magnetic field generator cannot be changed, otherwise it will greatly affect the navigation accuracy and the surgical effect.

[0005] In addition, in the prior art, the movement process of the catheter is mainly controlled manually. However, during traditional catheter surgery, it is difficult to manually operate the catheter to perform telescopic or bending movements inside the human body, and manually controlling the movement of the catheter is likely to cause harm to the patient due to poor movement accuracy, resulting in low safety. In addition, when manually controlling the movement of the catheter, the movement accuracy and surgical effect of the catheter are determined by the experience and ability of the operator, and the surgical effects vary greatly. Summary of the Invention

[0006] The object of the present invention is to provide a catheter positioning method, an interventional surgical system, an electronic device and a storage medium, which can achieve intraoperative positioning only based on a vision method without relying on other positioning devices, effectively reduce the constraints of the interventional surgical system on the environment and patients, and expand the applicable range of the interventional surgical system.

[0007] To achieve the above object, the present invention provides a catheter positioning method, an image acquisition device is installed at the end of the catheter, and the positioning method includes:

[0008] Obtain the current frame real endoscopic image collected by the image acquisition device in the space provided by the target organ;

[0009] Obtain the feature vector of the current frame real endoscopic image;

[0010] Search for the feature vector in the pre-obtained feature vector set that matches the feature vector of the current frame real endoscopic image;

[0011] According to the pose corresponding to the virtual endoscopic image corresponding to the matching feature vector in the feature vector set, obtain the current pose of the end of the catheter.

[0012] Optionally, the obtaining the feature vector of the current frame real endoscopic image includes:

[0013] Perform feature extraction on the current frame real endoscopic image to obtain the feature points of the current frame real endoscopic image;

[0014] According to the feature points and the pre-obtained feature dictionary, obtain the feature vector of the current frame real endoscopic image.

[0015] Optionally, the performing feature extraction on the current frame real endoscopic image to obtain the feature points of the current frame real endoscopic image includes:

[0016] Adopt the ORB algorithm to perform feature extraction on the current frame real endoscopic image to obtain the feature points of the current frame real endoscopic image.

[0017] Optionally, the obtaining the feature vector of the current frame real endoscopic image according to the feature points and the pre-obtained feature dictionary includes:

[0018] According to the feature points and the pre-obtained feature dictionary, count the frequency of each feature type in the pre-obtained feature dictionary that appears in the current frame real endoscopic image;

[0019] According to the frequency of each feature type that appears in the current frame real endoscopic image, obtain the feature histogram of the current frame real endoscopic image;

[0020] Obtain the feature vector of the current-frame real endoscopic image according to the feature histogram of the current-frame real endoscopic image.

[0021] Optionally, the feature dictionary is obtained through the following steps:

[0022] Obtain the virtual endoscopic image sequence of the target organ;

[0023] Extract features from each of the virtual endoscopic images to obtain the feature points of the virtual endoscopic images;

[0024] Cluster the feature points of all the virtual endoscopic images, and generate a feature dictionary according to the clustering result;

[0025] The feature vector set is obtained through the following steps:

[0026] Count the frequency of occurrence of each feature type in the feature dictionary in the virtual endoscopic images;

[0027] Obtain the feature histogram of the virtual endoscopic images according to the frequency of occurrence of each feature type in the virtual endoscopic images;

[0028] Obtain the feature vector of the virtual endoscopic image according to the feature histogram of the virtual endoscopic image;

[0029] Obtain a feature vector set according to the feature vectors of all the virtual endoscopic images.

[0030] Optionally, the clustering of the feature points of the virtual endoscopic images and generating a feature dictionary according to the clustering result includes:

[0031] Divide the three-dimensional model of the target organ obtained in advance into multiple organ regions;

[0032] Cluster the feature points of all the virtual endoscopic images corresponding to each organ region, and generate a feature dictionary corresponding to the organ region according to the clustering result.

[0033] Optionally, the target organ is the bronchus, and the positioning method further includes:

[0034] Divide the three-dimensional model of the bronchus obtained in advance into multiple airway regions according to the pre-obtained bronchial tree topological structure.

[0035] Optionally, the feature vector sets corresponding to all the airway regions are stored in a tree-shaped data structure according to the bronchial tree topological structure, wherein the feature vector sets corresponding to the same airway region are stored in the same node of the tree-shaped data structure.

[0036] Optionally, the node further stores one or more of the following information: the pose information corresponding to the virtual endoscopic image corresponding to each feature vector in the feature vector set, the number information of each feature vector, the number information of the airway where it is located, the number information of the sub-airway of the airway where it is located, the parent node information of the node where it is located, and the child node information of the node where it is located.

[0037] Optionally, after obtaining the current pose of the end of the catheter, the catheter positioning method further includes:

[0038] According to the current pose of the end of the catheter, determine whether the end of the catheter is located at the end position of the current airway. If so, obtain the feature vector set corresponding to all sub-airways of the current airway.

[0039] Optionally, the step of searching for a feature vector that matches the feature vector of the current frame real endoscopic image in the pre-obtained feature vector set corresponding to the organ region includes:

[0040] Calculate the total matching score between the nearest n frame endoscopic images and each of the sub-airways, where n≥2;

[0041] Select the sub-airway with the highest matching score as the current airway where the end of the catheter is located;

[0042] Search for a feature vector that matches the feature vector of the current frame real endoscopic image in the feature set corresponding to the current airway.

[0043] Optionally, the virtual endoscopic image sequence is obtained by rendering the three-dimensional model of the target organ along a preset path and in different directions according to the pre-obtained virtual imaging parameter information and illumination model.

[0044] To achieve the above object, the present invention further provides an interventional surgery system, including a robot and a controller connected by communication. The robot includes a trolley and a robotic arm mounted on the trolley. The end of the robotic arm is used to mount a catheter, and an image acquisition device is mounted at the end of the catheter. The controller is configured to implement the catheter positioning method described above.

[0045] Optionally, the interventional surgery system further includes a display device communicatively connected to the control system. The display device is used to display the current pose of the end of the catheter and / or display the endoscopic image reconstructed according to the current pose of the end of the catheter.

[0046] To achieve the above object, the present invention further provides an electronic device, including a processor and a memory. A computer program is stored on the memory. When the computer program is executed by the processor, the catheter positioning method described above is implemented.

[0047] To achieve the above object, the present invention further provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the catheter positioning method described above is implemented.

[0048] Compared with the prior art, the catheter positioning method, interventional surgery system, electronic device, and storage medium provided by the present invention have the following advantages: The present invention obtains the current frame real endoscopic image collected by an image acquisition device installed at the end of the catheter in the space provided by the target organ, and obtains the feature vector of the current frame real endoscopic image; then searches for the feature vector that matches the feature vector of the current frame real endoscopic image in a pre-acquired feature vector set; finally, according to the pose corresponding to the virtual endoscopic image corresponding to the matching feature vector in the feature vector set, obtains the current pose of the end of the catheter. It can be seen that the catheter positioning method provided by the present invention is a vision-based catheter positioning method and does not need to rely on other positioning devices, such as an electromagnetic positioning system. Therefore, it can effectively reduce the constraints on the environment and patients of the interventional surgery system and expand the applicable range of the interventional surgery system. In addition, since the present invention does not need to use additional positioning devices, the structure of the entire interventional surgery system is simplified, the user operation is more convenient, the surgical speed is effectively increased, and the patient's surgical time is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a schematic diagram of a partial structure of a catheter in an embodiment of the present invention;

[0050] Figure 2 It is a schematic diagram of the flow of a catheter positioning method in an embodiment of the present invention;

[0051] Figure 3 It is a schematic diagram of the extraction of feature points of the current frame real endoscopic image in a specific example of the present invention;

[0052] Figure 4 It is a schematic diagram of the principle of extracting key points in a specific example of the present invention;

[0053] Figure 5 It is a schematic diagram of a feature histogram in a specific example of the present invention;

[0054] Figure 6 It is a schematic diagram of the acquisition process of a feature dictionary in an embodiment of the present invention;

[0055] Figure 7 It is a schematic diagram of the extraction of feature points of a virtual endoscopic image in a specific example of the present invention;

[0056] Figure 8Schematic diagram of the process for obtaining a virtual endoscopy image sequence in an embodiment of the present invention;

[0057] Figure 9 Schematic diagram of the selection of the rendering position in a specific example of the present invention;

[0058] Figure 10 Schematic diagram of the process for obtaining the bronchial tree topological structure in an embodiment of the present invention;

[0059] Figure 11 Schematic diagram of the skeleton of the extracted bronchus in a specific example of the present invention;

[0060] Figure 12 Partial schematic diagram of the bronchial tree topological structure in a specific example of the present invention;

[0061] Figure 13 Schematic diagram of the mapping relationship between the bronchial tree topological structure and the tree - shaped data structure in a specific example of the present invention;

[0062] Figure 14 Schematic diagram of the specific process of positioning in an embodiment of the present invention;

[0063] Figure 15 Schematic diagram of the loading of the feature dictionary and the feature vector set in a specific example of the present invention;

[0064] Figure 16 Schematic diagram of the matching between the current frame real endoscopy image and the virtual endoscopy image in a specific example of the present invention;

[0065] Figure 17 Schematic diagram of the process of the motion control method of the catheter in an embodiment of the present invention;

[0066] Figure 18 Partial structural schematic diagram of the catheter in another embodiment of the present invention;

[0067] Figure 19 Schematic diagram of obtaining a virtual endoscopy image in another embodiment of the present invention;

[0068] Figure 20 Schematic diagram of the specific process for obtaining the current pose deviation information of the end of the catheter in an embodiment of the present invention;

[0069] Figure 21 Schematic diagram of the extracted first feature point set in a specific example of the present invention;

[0070] Figure 22 Schematic diagram of the extracted second feature point set in a specific example of the present invention;

[0071] Figure 23Schematic diagram of the matching process between the first feature point set and the second feature point set in an embodiment of the present invention;

[0072] Figure 24 Schematic diagram of the matching result between the first feature point set and the second feature point set in a specific example of the present invention;

[0073] Figure 25 Schematic diagram of the process for obtaining the mapping relationship between the 3D model coordinate system and the world coordinate system in an embodiment of the present invention;

[0074] Figure 26 Schematic diagram of the registration scenario provided by a specific example of the present invention;

[0075] Figure 27 Schematic diagram of the motion state of catheter initialization in a specific example of the present invention;

[0076] Figure 28 Schematic diagram of the motion state of the catheter when it deviates from the target motion path 200 in a specific example of the present invention;

[0077] Figure 29 Schematic diagram of the motion state of the catheter when it does not deviate from the target motion path in a specific example of the present invention;

[0078] Figure 30 Schematic diagram of the application scenario of the interventional surgery system in an embodiment of the present invention;

[0079] Figure 31 Schematic diagram of the connection relationship structure between the catheter and the robotic arm in an embodiment of the present invention;

[0080] Figure 32 Schematic diagram of the application scenario of the interventional surgery system in another embodiment of the present invention;

[0081] Figure 33 Schematic diagram of the block structure of the electronic device in an embodiment of the present invention.

[0082] Among them, the reference numerals are as follows:

[0083] Catheter - 100; Image acquisition device - 110; Position sensor - 120; Guide wire - 130;

[0084] Target motion path - 200; Path point - 210; Real endoscopic image - 300; First feature point - 310; Virtual endoscopic image - 400; Second feature point - 410; 3D model - 500; Target marking point - 510; Target organ - 600; Feature points - 1, 2;

[0085] Robot - 10; trolley - 11; robotic arm - 12; controller - 20; first driving device - 30; mounting plate - 40; fixing seat - 50; display device - 60; magnetic field generator - 70;

[0086] Processor - 101; communication interface - 102; memory - 103; communication bus - 104. Detailed implementation mode

[0087] The following further elaborates in detail on the catheter positioning method, catheter movement control method, interventional surgery system, electronic device, and storage medium proposed by the present invention in combination with the accompanying drawings and specific implementation modes. According to the following description, the advantages and features of the present invention will be clearer. It should be noted that the accompanying drawings are in a very simplified form and all use non - precise scales, only for conveniently and clearly assisting in explaining the purpose of the implementation mode of the present invention. In order to make the purpose, features, and advantages of the present invention more obvious and understandable, please refer to the accompanying drawings. It should be noted that the structures, scales, sizes, etc. shown in the drawings of this specification are only used to cooperate with the content disclosed in the specification for those familiar with this technology to understand and read, and are not used to limit the limiting conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship, or adjustment of the size, in the case of being the same or similar to the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope covered by the technical content disclosed by the present invention.

[0088] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non - exclusive inclusion, so that a process, method, article, or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such a process, method, article, or device. Without further limitations, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article, or device including the said element.

[0089] The core idea of the present invention is to provide a catheter positioning method, a catheter movement control method, an interventional surgery system, an electronic device, and a storage medium to solve the problems of the existing electromagnetic positioning system that cannot be used in combination with an X - ray machine / CT machine during surgery, and after the registration of the existing electromagnetic positioning system, the relative pose between the patient and the magnetic field generator cannot be changed, otherwise it will greatly affect the navigation accuracy and the surgical effect; or to solve the problems in the existing technology that manual control of catheter movement has high operation difficulty, low safety, and low control accuracy.

[0090] It should be noted that the catheter positioning method and the catheter movement control method of the embodiments of the present invention can be applied to the electronic device of the embodiments of the present invention. The electronic device can be configured on an interventional surgical system. Among them, the electronic device can be a personal computer, a mobile terminal, etc. The mobile terminal can be a hardware device such as a mobile phone, a tablet computer, etc. with various operating systems. In addition, it should be noted that although the catheter of the bronchoscope is taken as an example for illustration in this article, as can be understood by those skilled in the art, the catheter in the present invention can also be a catheter of other types of endoscopes, such as catheters for gastroscopes, colonoscopes, laryngoscopes, etc. The present invention does not limit this.

[0091] To implement the above idea, the present invention provides a catheter positioning method. Please refer to Figure 1 , which schematically shows a partial structural diagram of a catheter provided by an embodiment of the present invention. As Figure 1 shown, an image acquisition device 110 is installed at the end of the catheter 100, and the end of the catheter 100 can be bent. Please continue to refer to Figure 3 , which schematically shows a flowchart of a catheter positioning method provided by an embodiment of the present invention. As Figure 3 shown, the catheter positioning method includes the following steps:

[0092] Step S110: Obtain the current frame real endoscopic image collected by the image acquisition device in the space provided by the target organ.

[0093] Step S120: Obtain the feature vector of the current frame real endoscopic image.

[0094] Step S130: Search for the feature vector that matches the feature vector of the current frame real endoscopic image in the pre-obtained feature vector set.

[0095] Step S140: Obtain the current pose of the end of the catheter according to the pose corresponding to the virtual endoscopic image corresponding to the matching feature vector in the feature vector set.

[0096] Therefore, the catheter positioning method provided by the present invention is a vision-based catheter positioning method that does not rely on other positioning devices, such as an electromagnetic positioning system. Thus, it can effectively reduce the constraints on the environment and patients by the interventional surgery system and expand the applicable range of the interventional surgery system. In addition, since the present invention does not require the use of additional positioning devices, the structure of the entire interventional surgery system is more simplified, the user operation is more convenient, the surgical speed is effectively increased, and the patient's surgical time is reduced. It should be noted that, as can be understood by those skilled in the art, the current frame is dynamically changing, that is, the current frame of real endoscopic image changes over time. In addition, it should be noted that, as can be understood by those skilled in the art, as introduced later, the virtual endoscopic image in the present invention is obtained by rendering the three-dimensional model of the target organ. Thus, the pose corresponding to the virtual endoscopic image can be the pose in the three-dimensional model coordinate system. At this time, according to the pose corresponding to the virtual endoscopic image corresponding to the matching feature vector in the feature vector set, the pose of the end of the catheter 100 in the three-dimensional model coordinate system can be directly obtained. Of course, it can also be converted to the pose in the world coordinate system according to the mapping relationship between the three-dimensional model coordinate system and the world coordinate system obtained in advance. The present invention does not limit this. Regarding how to obtain the mapping relationship between the three-dimensional model coordinate system and the world coordinate system, reference can be made to the relevant descriptions in the following text, so it will not be elaborated here.

[0097] Further, the step S120, obtaining the feature vector of the current frame of real endoscopic image, includes:

[0098] Performing feature extraction on the current frame of real endoscopic image to obtain the feature points of the current frame of real endoscopic image;

[0099] According to the feature points and the pre-obtained feature dictionary, obtaining the feature vector of the current frame of real endoscopic image.

[0100] Thus, by obtaining the feature vector of the current frame of real endoscopic image according to the feature points of the current frame of real endoscopic image and the pre-obtained feature dictionary, the present invention can reduce the calculation amount and improve the positioning speed.

[0101] In an exemplary embodiment, the performing feature extraction on the current frame of real endoscopic image to obtain the feature points of the current frame of real endoscopic image includes:

[0102] Adopting the ORB algorithm to perform feature extraction on the current frame of real endoscopic image to obtain the feature points of the current frame of real endoscopic image.

[0103] Thus, by using the ORB (Oriented FAST and Rotated BRIEF) algorithm for feature point extraction, not only the extraction speed is improved, but also it is to a certain extent not affected by noise and image transformation, such as rotation and scaling transformation. Please refer to Figure 3 , which schematically shows a schematic diagram of the extraction of feature points of the current frame real endoscopic image in a specific example of the present invention. As Figure 3 shown, by using the ORB algorithm to perform feature extraction on the current frame real endoscopic image, the feature points 1 of the current frame real endoscopic image can be accurately and quickly extracted.

[0104] Specifically, the ORB algorithm is to detect key points and calculate the directions of the key points. By randomly selecting pairs of pixel points in the neighborhood of the key points for comparison, a binary descriptor of the key points is obtained, thereby obtaining ORB feature points. An ORB feature point can include two parts: a FAST corner point (i.e., a key point) and a BRIEF descriptor (i.e., a binary descriptor). Among them, the FAST corner point refers to the position of the ORB feature point in the image. The FAST corner point mainly detects areas where the local pixel grayscale changes significantly, and the calculation speed is fast. Its core idea is that if a pixel is significantly different from the pixels in its neighborhood (too dark or too bright), then this pixel is a corner point. Specifically, please refer to Figure 4 , which schematically shows a schematic diagram of the principle of extracting key points in a specific example of the present invention. As Figure 4 shown, each pixel point in the current frame real endoscopic image can be compared with 16 neighboring pixel points with similar distances around it. If the brightness difference between this pixel point and more than 8 neighboring pixel points is large, then this pixel point is regarded as a corner point. The BRIEF descriptor is a vector represented by binary. This vector describes the information of the pixels around the FAST corner point in a manually set manner. That is to say, the vector of the BRIEF descriptor is composed of multiple 0s and 1s, representing the relationship between the pixel value sizes of the FAST corner point and the nearby neighboring pixels. Specifically, in the neighborhood of a corner point, n pairs of pixel points pi, qi (i = 1, 2,..., n) are selected. Then the pixel values I of each pair of points are compared. If I(pi) > I(qi), then a 1 is generated in the binary string, otherwise it is 0. All pairs of points are compared, and a binary string with a length of n is generated. Generally, n takes 128, 256 or 512, and opencv defaults to 256. In this embodiment, in order to increase the anti-noise performance of the feature descriptor, before using the ORB algorithm to perform feature extraction on the current frame real endoscopic image, the current frame real endoscopic image is first subjected to Gaussian smoothing processing. Thus, when obtaining the BRIEF descriptors of each corner point, the selected pixel point pi follows a Gaussian distribution of N(μ, σ), and the selected pixel point qi follows a Gaussian distribution of N(μ, σ / 2).

[0105] It should be noted that, as can be understood by those skilled in the art, after the corners are extracted by FAST, defining a direction for them can achieve the rotation invariance of the ORB feature points. In addition, it should be noted that although this article takes the extraction of feature points using the ORB algorithm as an example for illustration, as can be understood by those skilled in the art, in some other embodiments, other feature point extraction methods in the prior art can also be used for feature point extraction, such as the harris corner detection method, the SIFT algorithm, the SURF algorithm, etc.

[0106] Preferably, before using the ORB algorithm to extract features from the current frame of real endoscopic image, the method further includes:

[0107] Preprocessing the current frame of real endoscopic image to convert the current frame of real endoscopic image into a grayscale image.

[0108] Correspondingly, the using the ORB algorithm to extract features from the current frame of real endoscopic image specifically includes:

[0109] Using the ORB algorithm to extract features from the current frame of real endoscopic image.

[0110] Thus, by first converting the current frame of endoscopic image into a grayscale image and then extracting features from the grayscale image, the computational amount can be effectively reduced and the positioning speed can be further improved.

[0111] Further, the obtaining the feature vector of the current frame of real endoscopic image according to the feature points and the pre-acquired feature dictionary includes:

[0112] According to the feature points and the pre-acquired feature dictionary, counting the frequency of each feature type in the feature dictionary that appears in the current frame of real endoscopic image;

[0113] According to the frequency of each feature type that appears in the current frame of real endoscopic image, obtaining the feature histogram of the current frame of real endoscopic image;

[0114] According to the feature histogram of the current frame of real endoscopic image, obtaining the feature vector of the current frame of real endoscopic image.

[0115] Please refer to Figure 5 , which schematically shows the schematic diagram of the feature histogram in a specific example of the present invention. As Figure 5As shown, the abscissa of the feature histogram represents the feature type, and the ordinate of the feature histogram represents the frequency of occurrence of the feature type. Assuming that the obtained feature dictionary includes four feature types, and the frequencies of occurrence of these four feature types in the current frame of the real endoscopic image are 100, 10, 20, and 10 respectively, then the feature vector of the current frame of the real endoscopic image is (100, 10, 20, 10).

[0116] Further, please refer to Figure 6 , which schematically shows a flowchart of obtaining a feature dictionary provided by an embodiment of the present invention. As Figure 6 shown, the feature dictionary is obtained through the following steps:

[0117] Obtain a sequence of virtual endoscopic images of the target organ;

[0118] Extract features from each of the virtual endoscopic images to obtain feature points of the virtual endoscopic images;

[0119] Cluster the feature points of all the virtual endoscopic images, and generate a feature dictionary according to the clustering result.

[0120] Specifically, any feature extraction method in the prior art can be used to extract features from the virtual endoscopic images. However, as can be understood by those skilled in the art, the same method should be used to extract features from the current frame of the real endoscopic image and the virtual endoscopic images. For example, if the ORB algorithm is used to extract features from the virtual endoscopic images, then the ORB algorithm is correspondingly used to extract features from the current frame of the real endoscopic image; if the SIFT algorithm is used to extract features from the virtual endoscopic images, then the SIFT algorithm is correspondingly used to extract features from the current frame of the real endoscopic image. In addition, it should be noted that clustering algorithms such as the K-means algorithm can be used to cluster the feature points of the virtual endoscopic images, and a feature dictionary is generated according to the clustering result. The feature dictionary is a set of clustering centers obtained by clustering, that is, the feature types in the feature dictionary correspond one-to-one with the clustering centers obtained by clustering. Please refer to Figure 7 , which schematically shows a diagram of extracting feature points of a virtual endoscopic image in a specific example of the present invention. As Figure 7 shown, by extracting features from the virtual endoscopic image, feature point 2 can be accurately extracted on the virtual endoscopic image.

[0121] Please continue to refer to Figure 8 , which schematically shows a flowchart of obtaining a sequence of virtual endoscopic images provided by an embodiment of the present invention. As Figure 8 shown, the sequence of virtual endoscopic images of the target organ is obtained through the following steps:

[0122] Obtain a mask image of the target organ;

[0123] Perform three-dimensional reconstruction based on the mask image to obtain a three-dimensional model of the target organ;

[0124] Render the three-dimensional model along a preset path and in different directions to obtain a sequence of virtual endoscopic images of the target organ.

[0125] Specifically, image segmentation methods such as threshold segmentation method or region generation method can be used to segment a pre-acquired medical image (such as a lung CT image) containing the target organ (such as the bronchus) to obtain a mask image of the target organ (such as the bronchus) (that is, a binary image of the target organ, in which the pixel values of the pixel points in the area where the target organ is located are 1, and the pixel values of the pixel points in other areas are 0). Among them, the virtual imaging parameter information can be obtained through calibration. The specific calibration process includes: using the image acquisition device 110 to acquire multiple checkerboard images, calculating the corner coordinates and sub-corner coordinates in each checkerboard image, and calculating the internal parameter matrix and distortion coefficients (including the focal length of the lens, the optical center, and the radial distortion coefficients, etc.) of the image acquisition device 110 based on the corner coordinates and sub-corner coordinates to obtain virtual imaging parameter information. By using parameters such as the position of the light source in the catheter 100, the light output intensity, and the diffuse reflection and specular reflection of the inner wall of the target organ (such as the bronchial wall), and adopting algorithms such as the finite element method, the lighting model can be obtained. According to the obtained lighting model, the light radiation change at each position on the inner wall of the three-dimensional model of the target organ (such as the bronchus) within the field of view of the image acquisition device 110 can be calculated. The light radiation change is used to represent the light energy attenuation caused by factors such as the optical path transmission direction and light ray superposition within the field of view of the image acquisition device 110. Thus, according to the virtual imaging parameter information and the lighting model, rendering can be performed at each position of the three-dimensional model of the target organ (such as the bronchus) and in different directions to obtain a sequence of virtual endoscopic images corresponding to the three-dimensional model of the target organ.

[0126] Since in the actual operation process, the catheter 100 generally moves along the central line direction of the target organ (such as the bronchus), thus, when rendering the three-dimensional model of the target organ (for the specific rendering process, please refer to: https: / / en-m.jinzhao.wiki / wiki / Physically_based_rendering, https: / / en-m.jinzhao.wiki / wiki /

[0127] When performing global illumination, it is preferred to select the positions of each pixel point on the center line of the three-dimensional model of the target organ (i.e., the target movement path) along the direction of the center line, and perform rendering in different directions, so as to obtain a virtual endoscopy image sequence from the perspective of the image acquisition device 110. Preferably, in order to further expand the feature set, when rendering the three-dimensional model, the positions of each pixel point in the vicinity of the center line of the three-dimensional model of the target organ are also selected and rendered in different directions to obtain virtual endoscopy images in more poses. It should be noted that as can be understood by those skilled in the art, the pose corresponding to the virtual endoscopy image is represented by the corresponding rendering position and direction. Please refer to Figure 9 , which schematically shows a schematic diagram of the selection of the rendering position in a specific example of the present invention. As Figure 9 shown, for the same position A, rendering can be performed in different directions D1 and D2, so that different virtual endoscopy images can be obtained.

[0128] Correspondingly, the feature vector set is obtained through the following steps:

[0129] Count the frequency of each feature type in the feature dictionary that appears in the virtual endoscopy image;

[0130] According to the frequency of each feature type that appears in the virtual endoscopy image, obtain the feature histogram of the virtual endoscopy image;

[0131] According to the feature histogram of the virtual endoscopy image, obtain the feature vector of the virtual endoscopy image;

[0132] According to the feature vectors of all the virtual endoscopy images, obtain the feature vector set.

[0133] Furthermore, the clustering of the feature points of the virtual endoscopy image and the generation of the feature dictionary according to the clustering result include:

[0134] Divide the pre-obtained three-dimensional model of the target organ into multiple organ regions;

[0135] Cluster the feature points of all the virtual endoscopy images corresponding to each organ region, and generate the feature dictionary corresponding to the organ region according to the clustering result.

[0136] Correspondingly, the obtaining of the feature vector set according to the feature vectors of all the virtual endoscopy images includes:

[0137] According to the feature vectors of all the virtual endoscopy images corresponding to the organ region, obtain the feature vector set corresponding to the organ region.

[0138] Correspondingly, obtaining the feature vector of the current frame of real endoscopic image according to the feature points and the pre-acquired feature dictionary includes:

[0139] Obtaining the feature vector of the current frame of real endoscopic image according to the feature points and the pre-acquired feature dictionary of the corresponding organ region.

[0140] Correspondingly, finding the feature vector that matches the feature vector of the current frame of real endoscopic image in the pre-acquired set of feature vectors includes:

[0141] Finding the feature vector that matches the feature vector of the current frame of real endoscopic image in the pre-acquired set of feature vectors of the corresponding organ region.

[0142] Thus, by dividing the three-dimensional model of the target organ into multiple organ regions, and for each of the organ regions, clustering the feature points of all the virtual endoscopic images corresponding to the organ region (for example, using the K-means algorithm for clustering) to generate the feature dictionary corresponding to the organ region, according to the feature dictionary corresponding to the organ region, the feature vectors of all the virtual endoscopic images corresponding to the organ region can be obtained, so as to obtain the set of feature vectors corresponding to the organ region, that is, different organ regions correspond to different feature dictionaries and sets of feature vectors. Furthermore, during specific positioning, the feature dictionary corresponding to the corresponding organ region is selected, the feature vector of the current frame of real endoscopic image is obtained, and the feature vector that matches the feature vector of the current frame of real endoscopic image is found in the set of feature vectors corresponding to the corresponding organ region, which can not only reduce the amount of matching and improve the matching speed, but also effectively avoid false matching and improve the accuracy of matching, thereby further improving the positioning accuracy of the catheter positioning method provided by the present invention. It should be noted that, as can be understood by those skilled in the art, the catheter 100 is generally inserted into the target organ along a pre-planned path. Thus, according to the path and in combination with the previous position of the end of the catheter 100, the organ region where the end of the catheter 100 is currently located can be determined.

[0143] In an exemplary embodiment, when the target organ is the bronchus, the dividing the three-dimensional model of the pre-acquired target organ into multiple organ regions includes:

[0144] Dividing the pre-acquired three-dimensional model of the bronchus into multiple airway regions according to the pre-acquired bronchial tree topological structure.

[0145] Thus, according to the pre-acquired bronchial tree topological structure, the bronchial three-dimensional model can be accurately divided into multiple airway regions. Furthermore, based on the position information of each airway region and the poses corresponding to each virtual endoscopic image, it can be known to which airway region each virtual endoscopic image in the virtual endoscopic image sequence belongs. Thus, according to all the virtual endoscopic images corresponding to each airway region, the feature dictionary and the set of feature vectors corresponding to this airway region can be obtained.

[0146] Further, please refer to Figure 10 , which schematically shows a schematic diagram of obtaining the bronchial tree topological structure provided by an embodiment of the present invention. As Figure 10 shown, the bronchial tree topological structure is obtained through the following process:

[0147] Obtain a bronchial mask image;

[0148] Perform thinning processing on the bronchial mask image to extract the skeleton;

[0149] Traverse each pixel point on the skeleton, and mark the number of the airway where each pixel point is located and / or the layer where the airway is located and / or the number of the parent airway of the airway where the pixel point is located and / or the number of the sub-airway of the airway where the pixel point is located, so as to obtain the bronchial tree topological structure.

[0150] Specifically, an image segmentation method, such as a threshold segmentation method or a region generation method, etc., can be used to segment the pre-acquired pulmonary medical image to obtain a preliminary bronchial mask image, and then a morphological method is used to fill the holes in the preliminary bronchial mask image, and a clear and complete bronchial mask image can be obtained. By performing thinning processing on the bronchial mask image, the skeleton of the bronchus (i.e., the center line of the bronchus) can be extracted.

[0151] In an exemplary embodiment, the performing thinning processing on the bronchial mask image to extract the skeleton includes:

[0152] Perform thinning processing on the bronchial mask image to extract a preliminary skeleton;

[0153] Perform de-ringing and pruning operations on the preliminary skeleton to extract the final skeleton.

[0154] Thus, by first performing thinning processing on the bronchial mask image to extract a preliminary skeleton, and then performing de-ringing and pruning operations on the preliminary skeleton, interference regions can be effectively removed, and the accuracy of the extracted skeleton can be further ensured.

[0155] Please refer to Figure 11, which schematically shows a schematic diagram of the extracted bronchial skeleton in a specific example of the present invention, where the black lines represent the extracted skeleton. As Figure 11 shown, after performing thinning, de-ringing, and pruning operations on the bronchial mask image, a complete skeleton can be extracted.

[0156] After extracting the skeleton, traverse each pixel point on the skeleton to mark the number of the airway where each pixel point is located and / or the layer where the airway is located and / or the number of the parent airway of the airway where the pixel point is located and / or the number of the sub-airway of the airway where the pixel point is located, and the bronchial tree topological structure can be obtained. It should be noted that, as can be understood by those skilled in the art, in some other embodiments, other methods can also be used to obtain the bronchial tree topological structure, and the present invention does not limit this.

[0157] Please refer to Figure 12 , which schematically shows a partial schematic diagram of the bronchial tree topological structure in a specific example of the present invention. As Figure 12 shown, in the figure, the airway numbered B0 is the first-level airway, the airways numbered (also known as ID) B0-0 and B0-1 are the second-level airways, the airways numbered B0-0-0, B0-0-1, B0-1-0, and B0-1-1 are the third-level airways, and the airways numbered B0-0-0-0 and B0-0-0-1 are the fourth-level airways. Among them, the airways numbered B0-0 and B0-1 are the sub-airways of the airway numbered B0, the airways numbered B0-0-0 and B0-0-1 are the sub-airways of the airway numbered B0-0, the airways numbered B0-1-0 and B0-1-1 are the sub-airways of the airway numbered B0-1, the airways numbered B0-0-0-0 and B0-0-0-1 are the sub-airways of the airway numbered B0-0-0, and the airways numbered B0-1-1-0 and B0-1-1-1 are the sub-airways of the airway numbered B0-1-1. That is, the airway numbered B0 is the parent airway of the airways numbered B0-0 and B0-1, the airway numbered B0-0 is the parent airway of the airways numbered B0-0-0 and B0-0-1, the airway numbered B0-1 is the parent airway of the airways numbered B0-1-0 and B0-1-1, the airway numbered B0-0-0 is the parent airway of the airways numbered B0-0-0-0 and B0-0-0-1, and the airway numbered B0-1-1 is the parent airway of the airways numbered B0-1-1-0 and B0-1-1-1. It should be noted that according to the position coordinates of the starting point and the ending point of each airway, the layer where each of the airways is located can be obtained. In addition, it should be noted that, as can be understood by those skilled in the art, the numbering setting rule in the figure is only for exemplary illustration, and those skilled in the art can set different types of numbers according to actual needs, and the present invention does not limit this.

[0158] Preferably, the feature dictionaries and the sets of feature vectors corresponding to all airway regions are stored in a tree data structure according to the bronchial tree topology structure. Among them, the feature dictionaries and the sets of feature vectors corresponding to the same airway region are stored in the same node of the tree data structure, and each airway segment in the bronchial tree topology structure forms a one-to-one correspondence with each node in the tree data structure. Please refer to Figure 13 , which schematically shows a schematic diagram of the mapping relationship between the bronchial tree topology structure and the tree data structure in a specific example of the present invention. As Figure 13 shown, the feature dictionary and the set of feature vectors corresponding to the airway numbered B0 are stored in the first-layer node (root node) b0 of the tree data structure, the feature dictionary and the set of feature vectors corresponding to the airway numbered B0-1 are stored in the second-layer node b0-1 of the tree data structure, the feature dictionary and the set of feature vectors corresponding to the airway numbered B0-1-1 are stored in the third-layer node b0-1-1 of the tree data structure, and the feature dictionary and the set of feature vectors corresponding to the airway numbered B0-1-1-1 are stored in the fourth-layer node b0-1-1-1 of the tree data structure. Thus, by storing the feature dictionaries and the sets of feature vectors corresponding to each airway in a tree data structure according to the bronchial tree topology structure, it is more convenient to select the feature dictionaries and the sets of feature vectors of the corresponding airway region for matching, thereby further reducing mis-matching and improving the matching speed and accuracy.

[0159] Furthermore, the following one or more pieces of information are also stored in the node: the pose information of the virtual endoscopic image corresponding to each feature vector in the set of feature vectors, as well as the number information of each feature vector, the number information of the airway where it is located, the number information of the sub-airway of the airway where it is located, the information of the parent node of the node where it is located, and the information of the sub-node of the node where it is located.

[0160] Specifically, please refer to Table 1 (Airway Feature Vector Set Storage Model). As shown in Table 1, the parent airway ID represents the number of the parent airway of the current airway, the current airway ID represents the number of the current airway, the sub-airway ID represents the number of the sub-airway of the current airway, the feature ID represents the serial number of the virtual endoscopic image corresponding to the feature vector, the position represents the rendering position of the virtual endoscopic image corresponding to the feature vector, and the pose represents the rendering direction of the virtual endoscopic image corresponding to the feature vector.

[0161] Table 1 Airway Feature Vector Set Storage Model

[0162]

[0163]

[0164] In an exemplary embodiment, after obtaining the current pose of the end of the catheter, the catheter positioning method further includes:

[0165] According to the current pose of the end of the catheter, determine whether the end of the catheter is located at the end position of the current airway. If so, obtain the feature dictionary and the set of feature vectors corresponding to all the sub-airways of the current airway.

[0166] Thus, when the end of the catheter 100 reaches the end position of a certain airway, by loading the feature dictionary and the set of feature vectors of all the sub-airways (i.e., the next-level airways) of this airway, dynamic import and export of the feature dictionary and the set of feature vectors can be realized, thereby effectively preventing mis-matching, improving the matching speed and the matching accuracy rate, and further improving the positioning accuracy. It should be noted that, as can be understood by those skilled in the art, at the initial stage (i.e., before the end of the catheter 100 enters the main airway), first import the feature dictionary and the set of feature vectors corresponding to the main airway, that is, first import the data in the root node of the tree-shaped data structure. When the end of the catheter 100 reaches the end position of the main airway, then import the feature dictionary and the set of feature vectors corresponding to all the sub-airways of the main airway, that is, import the data of all the child nodes of the root node of the tree-shaped data structure.

[0167] Preferably, the step of searching for the feature vector that matches the feature vector of the current frame real endoscopic image in the set of feature vectors of the pre-acquired corresponding organ region includes:

[0168] Calculate the total sum of the matching scores between the nearest n endoscopic images and each of the sub-airways respectively, where n≥2;

[0169] Select the sub-airway with the highest matching score as the current airway where the end of the catheter is located;

[0170] Search for the feature vector that matches the feature vector of the current frame real endoscopic image in the feature set corresponding to the current airway.

[0171] Specifically, please refer to Figure 14 and Figure 15 , where Figure 14 schematically shows the specific flow diagram of the positioning provided by an embodiment of the present invention, Figure 15 schematically shows the loading flow diagram of the feature dictionary and the set of feature vectors in a specific example of the present invention. As Figure 14 and Figure 15As shown, initially, the data stored in the root node b0 of the tree-shaped data structure is loaded first, that is, the feature dictionary and the set of feature vectors corresponding to the main airway are obtained. After the catheter 100 enters the current airway, if the current airway is the main airway, the current frame image collected by the image acquisition device 110 is obtained, and feature extraction is performed on the current frame real endoscopic image. Then, according to the feature dictionary corresponding to the main airway (current airway), the feature vector of the current frame real endoscopic image is calculated, and in the set of feature vectors corresponding to the main airway (current airway), the feature vector that matches the feature vector of the current frame real endoscopic image is searched for to obtain the current pose of the end of the catheter 100. When the catheter 100 reaches the end position of the main airway (current airway), all the sub-airways at the next level of the main airway (current airway) are used as candidate airways for the next current airway (i.e., the current candidate airways), and the data stored in the two sub-nodes b0-0 and b0-1 of the root node is loaded, that is, the feature dictionaries and the sets of feature vectors corresponding to all the sub-airways of the main airway (all the sub-airways of the main airway may be the next current airway) are obtained. After the image acquisition device 110 enters the new airway, the nearest n frames (for example, 10 frames) of endoscopic images collected by the image acquisition device 110 are obtained and feature extraction is performed. Then, according to the feature dictionaries corresponding to all the sub-airways of the main airway, the feature vectors of the nearest n frames of endoscopic images under each feature dictionary are obtained, and in the corresponding sets of feature vectors, the feature vectors that match them most are searched for respectively. Finally, the matching degrees between the feature vectors of each frame of endoscopic image under the same feature dictionary and the feature vectors that match them most in the corresponding sets of feature vectors are added up to obtain the total matching score between the nearest n frames of endoscopic images and the sub-airway corresponding to the feature dictionary. Similarly, the total matching scores between the nearest n frames of endoscopic images and the sub-airways corresponding to other feature dictionaries are obtained. Finally, the sub-airway with the highest matching score is selected as the current airway where the end of the catheter 100 is located, the feature dictionary and the set of feature vectors corresponding to the current airway are retained, and the feature dictionaries and the sets of feature vectors corresponding to other candidate airways at the same level are excluded. Thus, according to the feature dictionary corresponding to the current airway, the feature vector of the current frame real endoscopic image collected by the image acquisition device 110 can be obtained, and by searching in the set of feature vectors corresponding to the current airway for the feature vector that matches the feature vector of the current frame real endoscopic image, the current pose of the end of the catheter 100 can be obtained. And so on, repeating the above steps, through the feature dictionary and the set of feature vectors of the current airway, the current pose of the end of the catheter 100 can be accurately obtained. Please refer to Figure 16, which schematically shows a matching schematic diagram between the current frame real endoscopic image and the virtual endoscopic image in a specific example of the present invention, where feature point 1 is a feature point on the current frame real endoscopic image; feature point 2 is a feature point on the virtual endoscopic image. As Figure 16 shown, by dynamically loading the feature dictionaries and feature vector sets corresponding to each airway, the virtual endoscopic image that matches the current frame real endoscopic image can be accurately found, and then the current pose of the end of the catheter 100 can be obtained according to the pose of the matching virtual endoscopic image.

[0172] Based on the same inventive concept, the present invention also provides a method for controlling the movement of a catheter. Please continue to refer to Figure 17 , which schematically shows a flowchart of the method for controlling the movement of a catheter provided by an embodiment of the present invention. As Figure 17 shown, the control method includes the following steps:

[0173] Step S210, obtaining the current frame real endoscopic image collected by the image acquisition device in the space provided by the target organ and the current frame virtual endoscopic image corresponding to the current frame real endoscopic image;

[0174] Step S220, matching the current frame real endoscopic image and the current frame virtual endoscopic image to obtain the current pose deviation information of the end of the catheter;

[0175] Step S230, controlling the catheter to move along the target movement path according to the current pose deviation information and the target movement path planned in advance according to the three-dimensional model of the target area.

[0176] Specifically, according to the current pose of the end of the catheter 100 (the current pose in the three-dimensional model coordinate system) and the target movement path planned in advance according to the three-dimensional model of the target organ, the path point closest to the end of the catheter can be found on the target movement path, and rendering can be performed at the corresponding position and corresponding direction of the three-dimensional model of the target organ according to the pre-acquired virtual imaging parameter information and illumination model, so as to obtain the current frame virtual endoscopic image corresponding to the current frame real endoscopic image (it is also possible to directly find the current frame virtual endoscopic image in the pre-acquired virtual endoscopic image sequence according to the found closest path point and the forward direction of the target movement path). Thus, the present invention can realize the automatic control of the movement of the catheter 100 during the operation without manual intervention, thereby effectively reducing the surgical difficulty. At the same time, the present invention can realize the precise control of the movement of the catheter 100, effectively reduce the harm to the patient, and improve the surgical safety.

[0177] It should be noted that, as can be understood by those skilled in the art, in some other embodiments, the current target pose of the end of the catheter (i.e., the pose that the catheter should reach currently when moving along the target motion path 200) can also be directly obtained according to the current pose of the end of the catheter in the three-dimensional model coordinate system and the target motion path planned for the target organ. Then, according to the current pose of the end of the catheter in the world coordinate system and the current target pose of the end of the catheter in the world coordinate system, the current pose deviation information of the end of the catheter is calculated. Then, according to the current pose deviation information and the target motion path pre-obtained according to the three-dimensional model of the target area, the catheter is controlled to move along the target motion path.

[0178] Please continue to refer to Figure 18 , which schematically shows a partial structural view of the catheter provided by another embodiment of the present invention. As Figure 18 shown, in this embodiment, a position sensor 120 is further installed at the end of the catheter 100. Thus, in this embodiment, the obtaining of the current frame virtual endoscopy image corresponding to the current frame real endoscopy image includes:

[0179] Obtaining the current position information of the end of the catheter;

[0180] Rendering the three-dimensional model of the target organ according to the current position information and the target motion path to obtain the current frame virtual endoscopy image corresponding to the current frame real endoscopy image.

[0181] Specifically, according to the position sensor 120, the current position information of the end of the catheter 100 can be collected in real time (since there is a rigid assembly between the image acquisition device 110 and the catheter 100 and there is no relative movement, the current position of the end of the catheter 100 is the current position of the image acquisition device 110). Then, according to the current position information and the target motion path, rendering is performed at the corresponding position and in the corresponding direction of the three-dimensional model of the target organ according to the pre-obtained virtual imaging parameter information and the lighting model, and thus the current frame virtual endoscopy image corresponding to the current frame real endoscopy image can be obtained.

[0182] Please continue to refer to Figure 19 , which schematically shows a schematic diagram of obtaining a virtual endoscopy image provided by another embodiment of the present invention. As Figure 19As shown, based on the current position information of the end of the catheter 100 (position information in the world coordinate system) and the mapping relationship between the three-dimensional model coordinate system and the world coordinate system obtained in advance, the position information of the end of the catheter 100 in the three-dimensional model coordinate system can be obtained. According to the position information of the end of the catheter 100 in the three-dimensional model coordinate system, the current path point 210 closest to the current position can be found on the target motion path 200. According to the position of the current path point 210 and the path direction corresponding to the current path point 210 (i.e., the direction of advancing along the target motion path 200), the three-dimensional model of the target organ can be rendered, and the current frame virtual endoscope image 400 corresponding to the current frame real endoscope image can be obtained.

[0183] Preferably, the sampling frequency of the image acquisition device 110 is the same as that of the position sensor 120. Since the sampling frequency of the image acquisition device 110 is consistent with that of the position sensor 120, it can be ensured that the acquisition frequencies of the real endoscope image and the virtual endoscope image are consistent, so that the current frame real endoscope image and the current frame virtual endoscope image can always maintain a one-to-one correspondence relationship, ensuring the timeliness of the motion control of the catheter 100 and further improving the control accuracy of the motion of the catheter 100.

[0184] More preferably, the sampling intervals of both the image acquisition device 110 and the position sensor 120 are less than the unit motion time interval of the catheter 100 (i.e., the motion time interval for the catheter 100 to perform a single-step forward or bend). Thus, this setting can ensure that at least one image acquisition and position acquisition action is performed within the unit motion time interval of the catheter 100, thereby further ensuring the timeliness of the motion control of the catheter 100 and further improving the control accuracy of the motion of the catheter 100.

[0185] Please continue to refer to Figure 20 , which schematically shows a specific flowchart of obtaining the current pose deviation information of the end of the catheter provided by an embodiment of the present invention. As Figure 20 shown, the matching of the current frame real endoscope image and the current frame virtual endoscope image to obtain the current pose deviation information of the end of the catheter 100 includes:

[0186] Performing feature extraction on the current frame real endoscope image to obtain a first feature point set;

[0187] Performing feature extraction on the current frame virtual endoscope image to obtain a second feature point set;

[0188] Match the first set of feature points with the second set of feature points to obtain the spatial mapping relationship between the current-frame real endoscopic image and the current-frame virtual endoscopic image;

[0189] Obtain the current pose deviation information of the end of the catheter according to the spatial mapping relationship.

[0190] Thus, by respectively performing feature extraction on the current-frame real endoscopic image and the current-frame virtual endoscopic image, a first set of feature points and a second set of feature points can be obtained. By matching the first set of feature points with the second set of feature points, the spatial mapping relationship between the current-frame real endoscopic image and the current-frame virtual endoscopic image can be obtained. According to the mapping relationship, the pose deviation information between the current position of the end of the catheter 100 and the current path point 210 can be obtained, that is, the current pose deviation information of the end of the catheter 100. It should be noted that although Figure 20 the example is described by first obtaining the first set of feature points and then obtaining the second set of feature points, as can be understood by those skilled in the art, in some other embodiments, the second set of feature points can also be obtained first and then the first set of feature points, or the first set of feature points and the second set of feature points can be obtained simultaneously. The present invention does not limit this.

[0191] In an exemplary embodiment, the performing feature extraction on the current-frame real endoscopic image to obtain the first set of feature points includes:

[0192] Perform feature extraction on the current-frame real endoscopic image by using the ORB algorithm to obtain the first set of feature points.

[0193] The performing feature extraction on the current-frame virtual endoscopic image to obtain the second set of feature points includes:

[0194] Perform feature extraction on the current-frame virtual endoscopic image by using the ORB algorithm to obtain the second set of feature points.

[0195] Thus, by using the ORB algorithm to extract the first and second feature points, not only is the extraction speed extremely fast, but also it is not affected by noise and image transformation to a certain extent, such as the influence of rotation and scaling transformation, etc. Therefore, in the subsequent matching process, the rotation error along the movement direction of the catheter 100 during the movement of the catheter 100 can be eliminated. Please refer to Figure 21 and Figure 22 , where Figure 21 schematically shows a schematic diagram of the first set of feature points extracted in a specific example of the present invention; Figure 22 schematically shows a schematic diagram of the second set of feature points extracted in a specific example of the present invention. As Figure 21As shown, by using the ORB algorithm to extract features from the current frame of the real endoscopic image 300, a first feature point set composed of multiple first feature points 310 can be accurately and quickly extracted. As Figure 22 shown, by using the ORB algorithm to extract features from the current frame of the virtual endoscopic image 400, a second feature point set composed of multiple second feature points 410 can be accurately and quickly extracted.

[0196] It should be noted that although this article takes the extraction of the first feature point set and the second feature point set using the ORB algorithm as an example for illustration, as those skilled in the art can understand, in some other embodiments, other feature point extraction methods in the prior art can also be used to extract the first feature point set and the second feature point set, such as the harris corner detection method, the SIFT algorithm, the SURF algorithm, etc.

[0197] Preferably, before using the ORB algorithm to extract features from the current frame of the real endoscopic image, the current frame of the real endoscopic image is first grayscale processed to convert the current frame of the real endoscopic image into a grayscale image. Thus, by first converting the current frame of the real endoscopic image into a grayscale image and then extracting features from the corresponding grayscale image of the current frame of the real endoscopic image, the computational amount can be effectively reduced and the extraction rate of the first feature point set can be improved. Similarly, before using the ORB algorithm to extract features from the current frame of the virtual endoscopic image, the current frame of the virtual endoscopic image is first grayscale processed to convert the current frame of the virtual endoscopic image into a grayscale image. Thus, by first converting the current frame of the virtual endoscopic image into a grayscale image and then extracting features from the corresponding grayscale image of the current frame of the virtual endoscopic image, the computational amount can be effectively reduced and the extraction rate of the second feature point set can be improved.

[0198] Please continue to refer to Figure 23 , which schematically shows a schematic diagram of the matching process of the first feature point set and the second feature point set provided by an embodiment of the present invention. As Figure 23 shown, the matching of the first feature point set and the second feature point set to obtain the spatial mapping relationship between the current frame of the real endoscopic image and the current frame of the virtual endoscopic image includes:

[0199] Matching the first feature point set and the second feature point set to determine the matching first feature point and second feature point;

[0200] According to the pixel coordinate information of the second feature point and the pose information corresponding to the current frame of the virtual endoscopic image in the world coordinate system, obtain the position information of the second feature point in the world coordinate system;

[0201] Obtain the pose information corresponding to the current frame of real endoscopic image in the world coordinate system according to the pixel coordinate information of the first feature point and the position information of the second feature point matching therewith in the world coordinate system;

[0202] Obtain the spatial mapping relationship between the current frame of real endoscopic image and the current frame of virtual endoscopic image according to the pose information corresponding to the current frame of real endoscopic image in the world coordinate system and the pose information corresponding to the current frame of virtual endoscopic image in the world coordinate system.

[0203] Please continue to refer to Figure 24 , which schematically shows a schematic diagram of the matching result of the first feature point set and the second feature point set in a specific example of the present invention. As Figure 24 shown, by matching the first feature point set and the second feature point set, multiple groups of matching first feature points 310 and second feature points 410 can be found. Since the pose corresponding to the current frame of virtual endoscopic image in the three-dimensional model coordinate system is known (according to the rendering position and direction corresponding to the current frame of virtual endoscopic image, the pose corresponding to the current frame of virtual endoscopic image in the three-dimensional model coordinate system can be obtained), thus according to the pose information corresponding to the current frame of virtual endoscopic image in the three-dimensional model coordinate system and the pre-obtained mapping relationship between the three-dimensional model coordinate system and the world coordinate system, the pose information corresponding to the current frame of virtual endoscopic image in the world coordinate system can be obtained.

[0204] Please continue to refer to Figure 25 , which schematically shows a schematic diagram of the process of obtaining the mapping relationship between the three-dimensional model coordinate system and the world coordinate system provided by an embodiment of the present invention. As Figure 25 shown, the mapping relationship between the three-dimensional model coordinate system and the world coordinate system is obtained through the following process:

[0205] Determine a plurality of target marker points in the three-dimensional model of the target organ, and obtain the position information of the target marker points in the three-dimensional model coordinate system;

[0206] Control the end of the catheter to move to the target position corresponding to the target marker point in the target organ, and obtain the position information of the target position in the world coordinate system;

[0207] Obtain the mapping relationship between the three-dimensional model coordinate system and the world coordinate system according to the position information of the target marker point in the three-dimensional model coordinate system and the position information of the target position in the world coordinate system.

[0208] Please continue to refer to Figure 26, which schematically shows a registration scenario diagram provided by a specific example of the present invention. As Figure 26 shown, the target marker point 510 is preferably a point that is relatively easy to distinguish in the three-dimensional model 500. Thus, by selecting a point that is relatively easy to distinguish as the target marker point 510, it is possible to more easily determine whether the end of the catheter 100 has moved to the target position corresponding to the target marker point 510 within the target organ 600, thereby ensuring the accuracy of the mapping relationship between the three-dimensional model coordinate system and the world coordinate system. It should be noted that, as can be understood by those skilled in the art, the coordinates of the target position in the world coordinate system can be measured by the position sensor 120 installed at the end of the catheter 100. Thus, based on the position information of the target marker point 510 in the three-dimensional model coordinate system and the position information of the target position in the world coordinate system, the mapping relationship between the three-dimensional model coordinate system and the world coordinate system can be obtained. In addition, it should be noted that, as can be understood by those skilled in the art, the number of the target marker points 510 needs to be sufficient to establish the mapping relationship between the three-dimensional model coordinate system and the world coordinate system.

[0209] Furthermore, according to the following formula, obtain the position information of each of the matched second feature points in the world coordinate system:

[0210]

[0211] In the formula, (u vi , v vi ) is the pixel coordinate of the second feature point vi in the pixel coordinate system of the current frame of virtual endoscopic image, a is a scaling factor, Mc is the internal parameter matrix of the image acquisition device 110, Mv is the pose (a 3×4 matrix composed of position and attitude) of the current frame of endoscopic image in the world coordinate system, (x vi , y vi , z vi ) is the coordinate of the second feature point vi in the world coordinate system, where the a and M C are both obtained by pre-calibrating the image acquisition device 110. The specific calibration process can refer to the prior art and will not be elaborated here.

[0212] Similarly, the pixel coordinates (u ri , v ri ) of the first feature point ri in the pixel coordinate system of the current frame of real endoscopic image and its coordinates (x ri , y ri , z ri ) in the world coordinate system satisfy the following relational formula:

[0213]

[0214] Since the coordinates of the first feature point ri and the second feature point vi matched with it in the world coordinate system are the same, thus according to the pixel coordinates (u ri , v ri ) of each group of the matched first feature points ri and the coordinates (x vi , y vi , z vi ) of the second feature point vi in the world coordinate system, calculating according to formula (2), the pose M corresponding to the current frame real endoscopic image in the world coordinate system can be obtained r .

[0215] Finally, according to the following formula (3), the spatial mapping relationship between the current frame real endoscopic image and the current frame virtual endoscopic image can be obtained:

[0216] M r→v =(M r ) -1 *M v (3)

[0217] The spatial mapping relationship between the current frame real endoscopic image and the current frame virtual endoscopic image represents the current pose deviation information of the end of the catheter 100. Thus, according to the current pose deviation information and based on the inverse kinematic equation of the catheter 100, the corresponding adjustment movement instruction can be obtained, so as to control the catheter 100 to adjust to the position corresponding to the current frame virtual endoscopic image, so that the catheter 100 can continue to move forward along the target movement path 200 until reaching the termination position (i.e., the end point of the target movement path 200). It should be noted that, as can be understood by those skilled in the art, the position of the current path point 210 described above is the rendering position corresponding to the current frame virtual endoscopic image, and the path direction corresponding to the current path point 210 described above is the rendering direction corresponding to the current frame virtual endoscopic image.

[0218] Further, controlling the catheter to move along the target movement path according to the current pose deviation information and the target movement path planned in advance according to the three-dimensional model of the target organ includes:

[0219] Judging whether the catheter currently deviates from the target movement path according to the current pose deviation information;

[0220] If so, calculate the adjustment motion parameters of the catheter according to the current pose deviation information, and control the catheter to perform corresponding bending motions according to the adjustment motion parameters so that the catheter can move along the target motion path;

[0221] If not, control the catheter to continue moving along the target motion path.

[0222] Please continue to refer to Figure 27 , which schematically shows the motion state diagram of the catheter initialization in a specific example of the present invention. As Figure 27 shown, before performing the motion control work of the catheter 100 (i.e., initially), the doctor needs to place the catheter 100 at the starting position of the target motion path 200. Specifically, with the assistance of the display of the image acquisition device 110, the doctor can control the catheter 100 to move to the starting position of the target motion path 200.

[0223] Please continue to refer to Figure 28 , which schematically shows the motion state diagram of the catheter deviating from the target motion path 200 in a specific example of the present invention. As Figure 28 shown, when it is determined that the catheter 100 currently deviates from the target motion path 200, according to the current pose deviation information and based on the inverse kinematic equation of the catheter 100, the adjustment motion parameters of the catheter 100 can be calculated to obtain corresponding adjustment motion instructions. Thus, according to this adjustment motion instruction, the catheter 100 can be controlled to bend so that the catheter 100 can move along the target motion path 200. Preferably, when executing the adjustment motion instruction, reduce the forward speed of the catheter 100 and gradually adjust the end pose of the catheter 100 according to a unit angle.

[0224] Please continue to refer to Figure 29 , which schematically shows the motion state diagram of the catheter not deviating from the target motion path in a specific example of the present invention. As Figure 29 shown, when it is determined that the catheter 100 currently does not deviate from the target motion path 200, control the catheter 100 to continue moving forward along the target motion path 200. Preferably, when the catheter 100 does not deviate from the target motion path 200, control the catheter 100 to continue moving along the target motion path 200 at a higher speed.

[0225] Based on the same inventive concept, the present invention also provides an interventional surgery system. Please refer to Figure 30 , which schematically shows the application scenario diagram of the interventional surgery system provided by an embodiment of the present invention. As Figure 30As shown, the interventional surgery system includes a robot 10 and a controller 20 connected in communication, the robot 10 includes a trolley 11 and a mechanical arm 12 installed on the trolley 11, the end of the mechanical arm 12 is used to install a catheter 100, the end of the catheter 100 is installed with an image acquisition device 110, and the controller 20 is configured to implement the catheter positioning method and / or the catheter motion control method described above. Since the controller 20 included in the interventional surgery system provided by the present invention can implement the catheter positioning method described above, the interventional surgery system provided by the present invention does not need to set other additional positioning devices to achieve the positioning of the catheter 100 in the human body, thereby effectively reducing costs and simplifying the structure of the entire interventional surgery system, and also effectively reducing the constraints of the interventional surgery system on the environment and patients, and expanding the scope of application of the interventional surgery system. Since the controller 20 included in the interventional surgery system provided by the present invention can implement the catheter motion control method described above, the interventional surgery system provided by the present invention can realize the automatic control of the movement of the catheter 100 during the operation without manual intervention, thereby effectively reducing the difficulty of the operation. At the same time, the present invention can achieve precise control of the movement of the catheter 100, effectively reduce harm to the patient, and improve surgical safety.

[0226] Please continue to refer to Figure 31 , which schematically shows a schematic diagram of the connection relationship between the catheter and the robotic arm provided in one embodiment of the present invention. Figure 31 As shown, at least one guide wire 130 is inserted into the catheter 100, and the proximal end of the guide wire 130 is connected to a first drive device 30 (preferably a motor), the first drive device 30 is installed at the end of the mechanical arm 12 and is in communication with the controller 20, and the distal end of the guide wire 130 is connected to the end of the catheter 100. Under the action of the first drive device 30, the guide wire 130 can be extended and shortened, so that the end of the catheter 100 can be bent in at least one direction. Thus, the controller 20 can calculate the adjustment motion parameters of the catheter 100 (i.e., the extension or shortening amount of the guide wire 130) based on the current posture deviation information of the end of the catheter 100 and the inverse kinematics equation of the catheter 100, and according to the adjustment motion parameters of the catheter 100, the first drive device 30 can be controlled to perform corresponding movements, so that the end of the catheter 100 can move along the target motion path 200.

[0227] Furthermore, if Figure 30 and Figure 31As shown, a mounting plate 40 is provided at the end of the robotic arm 12, and the first driving device 30 is mounted on the mounting plate 40. The first driving device 30 can reciprocate on the mounting plate 40 (i.e., move along the arrow direction in the figure). Thus, when the first driving device 30 moves towards the distal end of the mounting plate 40, the end of the catheter 100 can advance along the target movement path 200, and when the first driving device 30 moves towards the proximal end of the mounting plate 40, the catheter 100 can be withdrawn from the target organ.

[0228] In an exemplary embodiment, a second driving device (not shown in the figure) connected to the first driving device 30 and communicatively connected to the controller 20 is further provided at the end of the robotic arm 12. The second driving device is used to drive the first driving device 30 to reciprocate on the mounting plate 40. Thus, by providing the second driving device, automatic control of the movement of the catheter 100 can be further achieved, and the control accuracy of the movement of the catheter 100 can be further improved.

[0229] As Figure 31 shown, a fixing seat 50 for fixing the catheter 100 is provided at the distal end of the mounting plate 40, and the end of the catheter 100 can approach and move away from the fixing seat 50. Thus, by providing the fixing seat 50 on the mounting plate 40, it is not only more convenient to place the catheter 100, but also the control accuracy of the movement of the catheter 100 can be further improved.

[0230] Furthermore, as Figure 30As shown, the interventional surgical system further includes a display device 60 communicatively connected to the controller 20. The display device 60 is configured to display the current pose of the end of the catheter 100 and / or to display a reconstructed endoscopic image based on the current pose of the end of the catheter 100 and / or to display a real endoscopic image acquired by the image acquisition device 110 and / or a three-dimensional model of the target organ. Since the catheter positioning method provided by the present invention can obtain the pose of the catheter 100 in the three-dimensional model coordinate system, thereby, the display device 60 can display the real-time pose of the catheter 100 on the three-dimensional model, so that intraoperative navigation can be realized. By displaying the reconstructed endoscopic image based on the current pose of the end of the catheter 100, it is more convenient for the operator to adjust the actual path of the catheter 100 in real time according to the reconstructed endoscopic image and in combination with the three-dimensional model, so as to better perform the surgery. It should be noted that, as can be understood by those skilled in the art, based on the current pose of the end of the catheter 100, rendering can be performed at the corresponding position and direction of the three-dimensional model, and then the reconstructed endoscopic image can be obtained. In addition, by displaying the real endoscopic image acquired by the image acquisition device 110 and the three-dimensional model of the target organ, it is more convenient for the doctor to understand the real situation during the operation, so as to further improve the safety during the operation.

[0231] Please continue to refer to Figure 32 , which schematically shows an application scenario diagram of an interventional surgical system provided by another embodiment of the present invention. As Figure 32 shown, in this embodiment, the surgical system further includes a magnetic field generator 70. A position sensor 130 (i.e., a magnetic sensor) is installed at the end of the catheter 100. The magnetic field generator 70 is configured to generate a magnetic field passing through the target organ. The position sensor 130 is configured to collect magnetic field intensity information in the magnetic field. The controller 20 is configured to obtain the position information of the end of the catheter 100 according to the magnetic induction intensity information collected by the position sensor 130. Thus, by using the magnetic induction method to obtain the position information of the end of the catheter 100, the accuracy of the obtained position information of the end of the catheter 100 can be ensured, and further the control accuracy of the movement of the catheter 100 can be ensured.

[0232] Based on the same inventive concept, the present invention also provides an electronic device. Please refer to Figure 33 , which schematically shows a block structure diagram of an electronic device provided by an embodiment of the present invention. As Figure 33As shown, the electronic device includes a processor 101 and a memory 103. A computer program is stored on the memory 103. When the computer program is executed by the processor 101, the catheter positioning method and / or the catheter movement control method described above are implemented. Since the electronic device provided by the present invention and the catheter positioning method and / or the catheter movement control method provided by the present invention belong to the same inventive concept, it has all the advantages of the catheter positioning method and / or the catheter movement control method described above, and thus will not be elaborated herein.

[0233] As Figure 33 shown, the electronic device further includes a communication interface 102 and a communication bus 104. Among them, the processor 101, the communication interface 102, and the memory 103 complete mutual communication through the communication bus 104. The communication bus 104 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus 104 can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 102 is used for communication between the above-mentioned electronic device and other devices.

[0234] The processor 101 referred to in the present invention may be a Central Processing Unit (CPU), or may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The processor 101 is the control center of the electronic device, and connects all parts of the entire electronic device through various interfaces and lines.

[0235] The memory 103 can be used to store the computer program. The processor 101 realizes various functions of the electronic device by running or executing the computer program stored in the memory 103 and calling the data stored in the memory 103.

[0236] The memory 103 may include non-volatile and / or volatile memory. The non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. The volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0237] The present invention also provides a readable storage medium storing a computer program therein, and when the computer program is executed by a processor, the catheter positioning method and / or the catheter movement control method described above can be implemented. Since the readable storage medium provided by the present invention and the catheter positioning method and / or the catheter movement control method provided by the present invention belong to the same inventive concept, it has all the advantages of the catheter positioning method and / or the catheter movement control method described above, and thus will not be elaborated herein.

[0238] The readable storage medium according to the embodiment of the present invention may adopt any combination of one or more computer-readable media. The readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer hard disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, apparatus, or device.

[0239] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0240] The computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., connected through the Internet using an Internet service provider).

[0241] In summary, compared with the prior art, the catheter positioning method, interventional surgery system, electronic device, and storage medium provided by the present invention have the following advantages: The present invention obtains the current frame real endoscopic image collected by the image acquisition device installed at the end of the catheter in the space provided by the target organ, and obtains the feature vector of the current frame real endoscopic image; then searches for the feature vector that matches the feature vector of the current frame real endoscopic image in the pre-acquired feature vector set; finally, according to the pose corresponding to the virtual endoscopic image corresponding to the matching feature vector in the feature vector set, obtains the current pose of the end of the catheter. It can be seen that the catheter positioning method provided by the present invention is a vision-based catheter positioning method and does not need to rely on other positioning devices, such as an electromagnetic positioning system. Therefore, it can effectively reduce the constraints on the environment and patients of the interventional surgery system and expand the applicable range of the interventional surgery system. In addition, since the present invention does not need to use additional positioning devices, the structure of the entire interventional surgery system is more simplified, the user operation is more convenient, the surgical speed is effectively increased, and the patient's surgical time is reduced.

[0242] It should be noted that the devices and methods disclosed in the embodiments of this article can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions, and operations of devices, methods, and computer program products according to multiple embodiments of this article. In this regard, each block in the flowchart or block diagram may represent a module, program, or part of code, and the module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0243] In addition, the functional modules in each embodiment of this article can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

[0244] The above description is only a description of the preferred embodiments of the present invention and does not limit the scope of the present invention in any way. Any changes and modifications made by those of ordinary skill in the art of the present invention based on the above disclosure belong to the protection scope of the present invention. Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations fall within the scope of the present invention and its equivalent technologies, the present invention is also intended to include these modifications and variations.

Claims

1. A catheter positioning method, characterized in that, An image acquisition device is installed at the end of the catheter, and the positioning method includes: Obtaining the current frame real endoscopic image acquired by the image acquisition device in the space provided by the target organ; Obtaining the feature vector of the current frame real endoscopic image; Searching for the feature vector in the pre-acquired feature vector set that matches the feature vector of the current frame real endoscopic image; Obtaining the current pose of the end of the catheter according to the pose corresponding to the virtual endoscopic image corresponding to the matching feature vector in the feature vector set; The obtaining the feature vector of the current frame real endoscopic image includes: Performing feature extraction on the current frame real endoscopic image to obtain the feature points of the current frame real endoscopic image; Obtaining the feature vector of the current frame real endoscopic image according to the feature points and the pre-acquired feature dictionary; The feature dictionary is obtained through the following steps: Obtaining the virtual endoscopic image sequence of the target organ; Performing feature extraction on each of the virtual endoscopic images to obtain the feature points of the virtual endoscopic image; Clustering the feature points of all the virtual endoscopic images, and generating a feature dictionary according to the clustering result; Wherein, the virtual endoscopic image sequence is obtained by rendering the three-dimensional model of the target organ along a preset path and in different directions according to the pre-acquired virtual imaging parameter information and illumination model.

2. The catheter positioning method according to claim 1, characterized in that, The preset path is set along the center line direction of the three-dimensional model of the target organ.

3. The catheter positioning method according to claim 1, characterized in that, The performing feature extraction on the current frame real endoscopic image to obtain the feature points of the current frame real endoscopic image includes: Using the ORB algorithm to perform feature extraction on the current frame real endoscopic image to obtain the feature points of the current frame real endoscopic image.

4. The catheter positioning method according to claim 1, characterized in that, The obtaining the feature vector of the current frame real endoscopic image according to the feature points and the pre-acquired feature dictionary includes: Counting the frequency of occurrence of each feature type in the feature dictionary in the current frame real endoscopic image according to the feature points and the pre-acquired feature dictionary; Obtaining the feature histogram of the current frame real endoscopic image according to the frequency of occurrence of each feature type in the current frame real endoscopic image; Obtaining the feature vector of the current frame real endoscopic image according to the feature histogram of the current frame real endoscopic image.

5. The catheter positioning method according to claim 1, characterized in that, The clustering the feature points of all the virtual endoscopic images and generating a feature dictionary according to the clustering result includes: Dividing the pre-acquired three-dimensional model of the target organ into multiple organ regions; Clustering the feature points of all the virtual endoscopic images corresponding to each organ region, and generating a feature dictionary corresponding to the organ region according to the clustering result.

6. The catheter positioning method according to claim 1, characterized in that, The feature vector set is obtained through the following steps: Counting the frequency of occurrence of each feature type in the feature dictionary in the virtual endoscopic image; Obtaining the feature histogram of the virtual endoscopic image according to the frequency of occurrence of each feature type in the virtual endoscopic image; Obtaining the feature vector of the virtual endoscopic image according to the feature histogram of the virtual endoscopic image; Obtain a set of feature vectors based on the feature vectors of all virtual endoscopy images.

7. The catheter positioning method according to claim 1, characterized in that, The target organ is the bronchus, and the positioning method further includes: Divide the pre-acquired three-dimensional bronchial model into multiple airway regions according to the pre-acquired bronchial tree topology.

8. The catheter positioning method according to claim 7, characterized in that, The set of feature vectors corresponding to all airway regions is stored in a tree-shaped data structure according to the bronchial tree topology, wherein the set of feature vectors corresponding to the same airway region is stored in the same node of the tree-shaped data structure.

9. The catheter positioning method according to claim 8, wherein, The following one or more pieces of information are also stored in the node: the pose information of the virtual endoscopy image corresponding to each feature vector in the set of feature vectors, the number information of each feature vector, the number information of the airway where it is located, the number information of the sub-airways of the airway where it is located, the information of the parent node of the node where it is located, and the information of the sub-nodes of the node where it is located.

10. The catheter positioning method according to claim 7, wherein, After obtaining the current pose of the end of the catheter, the catheter positioning method further includes: According to the current pose of the end of the catheter, determine whether the end of the catheter is located at the end position of the current airway. If so, obtain the set of feature vectors corresponding to all sub-airways of the current airway.

11. The catheter positioning method according to claim 10, wherein, The step of searching for the feature vector that matches the feature vector of the current frame of real endoscopy image in the pre-acquired set of feature vectors corresponding to the organ region includes: Calculate the total matching score between the nearest n frames of endoscopy images and each of the sub-airways, where n≥2; Select the sub-airway with the highest matching score as the current airway where the end of the catheter is located; Search for the feature vector that matches the feature vector of the current frame of real endoscopy image in the feature set corresponding to the current airway.

12. The catheter positioning method according to claim 1, wherein, The step of obtaining the current pose of the end of the catheter according to the pose of the virtual endoscopy image corresponding to the matching feature vector in the set of feature vectors includes: Obtain the pose of the end of the catheter in the three-dimensional model coordinate system according to the pose of the virtual endoscopy image corresponding to the matching feature vector in the set of feature vectors; According to the pre-acquired mapping relationship between the three-dimensional model coordinate system and the world coordinate system, convert the current pose of the end of the catheter in the three-dimensional model coordinate system into the pose in the world coordinate system.

13. An interventional surgery system, wherein, It includes a robot and a controller connected by communication. The robot includes a trolley and a robotic arm mounted on the trolley. The end of the robotic arm is used to mount a catheter, and an image acquisition device is mounted at the end of the catheter. The controller is configured to implement the catheter positioning method according to any one of claims 1 to 12.

14. The interventional surgery system according to claim 13, wherein, The interventional surgery system further includes a display device communicatively connected to the controller. The display device is used to display the current pose of the end of the catheter and / or display the endoscopy image reconstructed according to the current pose of the end of the catheter.

15. An electronic device, wherein, It includes a processor and a memory. A computer program is stored on the memory. When the computer program is executed by the processor, the catheter positioning method according to any one of claims 1 to 12 is implemented.

16. A readable storage medium, wherein, The readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

Images