Bronchoscope control method, device, automatic registration method, and computer device

Abstract

The application relates to a bronchoscope control method and device, an automatic registration method and computer equipment. The method comprises the following steps: acquiring a pre-constructed bronchial model, wherein the bronchial model comprises a plurality of registration points; moving the bronchoscope to each registration point according to a preset path, including: collecting real endoscopic images during the moving process; and controlling the movement of the bronchoscope according to the real endoscopic images and virtual endoscopic images corresponding to the current registration point. According to the real endoscopic images and the virtual endoscopic images corresponding to the current registration point, the bronchoscope can be automatically moved to each registration point according to the preset path, an accurate pose can be provided for the bronchial model, and the accuracy of virtual endoscopic rendering is improved.

Description

Technical Field

[0001] This application relates to the field of medical device technology, and in particular to a bronchoscope control method, device, automatic registration method and computer equipment. Background Technology

[0002] With the development of science and technology, virtual bronchoscope systems based on three-dimensional imaging technology provide users with reliable auxiliary navigation paths, helping users to conveniently and accurately insert into the human bronchus and observe the lesions of the bronchus, which plays an important role in clinical observation and diagnosis.

[0003] In traditional technology, virtual bronchoscope systems use electromagnetic sensors to manually acquire a sufficient number of poses of the bronchoscope and register multiple poses to the bronchial model. Based on the correspondence of multiple poses, a mapping relationship is constructed between the bronchial model and all poses on the electromagnetic sensors, and a virtual endoscope is rendered in the bronchial model.

[0004] However, the current registration process requires manual operation by personnel familiar with the bronchial model and with certain robot operation experience, which places high demands on the operators. At the same time, the operators need to judge whether the designated registration point has been reached, which leads to certain errors in the pose mapping relationship constructed later, affecting the accuracy of subsequent virtual endoscopy rendering. Summary of the Invention

[0005] Therefore, it is necessary to provide a bronchoscope control method, device, automatic registration method, and computer equipment to address the above-mentioned technical problems. This is to solve the problem that the registration process of virtual bronchoscopes in the prior art is manually operated, which leads to certain errors in the subsequently constructed pose mapping relationship and affects the accuracy of subsequent virtual endoscopy rendering.

[0006] Firstly, this application provides a method for controlling a bronchoscope. The method includes:

[0007] Obtain a pre-constructed bronchial model, which includes multiple registration points;

[0008] Following a preset path, the bronchoscope is moved sequentially to each of the registration points, including:

[0009] Acquire real endoscopic images during movement;

[0010] The movement of the bronchoscope is controlled based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point.

[0011] In one embodiment, acquiring real endoscopic images during the movement includes:

[0012] During the movement, the bronchoscope is positionally compensated, and the current phase of the subject's breathing is detected;

[0013] When the current phase is the target phase, the pose-compensated real endoscopic image of the bronchoscope is acquired. In one embodiment, the pose compensation of the bronchoscope includes:

[0014] Obtain the current virtual coordinates of the bronchoscope;

[0015] Obtain multiple virtual endoscopic images of the current virtual coordinates at a preset angle;

[0016] Traverse all the virtual endoscopic images to determine the optimal virtual endoscopic image, wherein the optimal virtual endoscopic image has the highest first similarity to the real endoscopic image at the current virtual coordinates;

[0017] Obtain the virtual camera pose parameters corresponding to the optimal virtual endoscopic image;

[0018] The bronchoscope is subjected to pose compensation based on the virtual camera pose parameters.

[0019] In one embodiment, obtaining the current virtual coordinates of the bronchoscope in the bronchial model includes:

[0020] The current spatial coordinates of the bronchoscope are obtained based on the positioning module of the bronchoscope.

[0021] Obtain the automatic registration matrix;

[0022] Based on the automatic registration matrix, the current spatial coordinates of the bronchoscope are converted into the current virtual coordinates in the bronchial model.

[0023] In one embodiment, the bronchial model further includes multiple bifurcation points;

[0024] The process of obtaining the automatic registration matrix includes:

[0025] A predetermined number of bifurcation points are selected from the bronchial model as marker points;

[0026] Acquire real endoscopic images of each of the aforementioned marker points in advance;

[0027] Based on the spatial coordinates and virtual coordinates of the marked points, the rotation and translation matrix is ​​obtained;

[0028] Calculate the similarity between the virtual and real endoscopic images corresponding to all the marked points for each rotation and translation matrix, and select the rotation and translation matrix whose similarity satisfies the preset conditions as the automatic registration matrix.

[0029] In one embodiment, controlling the movement of the bronchoscope based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point includes:

[0030] Obtain the current virtual coordinates of the bronchoscope;

[0031] Obtain the optimal virtual image at the current virtual coordinates;

[0032] Calculate the second similarity between the optimal virtual image and the virtual endoscopic image at the current registration point;

[0033] If the second similarity is greater than or equal to a preset threshold, the region within the first preset range of the current virtual coordinates is recorded as the region to be registered, and the movement of the bronchoscope is controlled within the region to be registered according to the first movement strategy.

[0034] If the second similarity is less than a preset threshold, the region within the second preset range of the current virtual coordinates is recorded as a non-registered region, and the movement of the bronchoscope is controlled within the non-registered region according to the second movement strategy.

[0035] In one embodiment, the first movement strategy includes:

[0036] The bronchoscope is moved according to the first direction vector by the first step length, where the first direction vector is the direction vector of the bronchial segment where the bronchoscope is currently located.

[0037] Once the bronchoscope reaches the current registration point, the rotation angle is calculated based on the current registration point and the next registration point.

[0038] Based on the stated rotation angle, rotate the tip of the bronchoscope to the next registration point.

[0039] In one embodiment, the method for determining that the bronchoscope has reached the current registration point includes:

[0040] During the movement, the second similarity is calculated. If the next second similarity is less than the previous second similarity, it is considered that the bronchoscope has reached the current registration point.

[0041] In one embodiment, calculating the rotation angle based on the current registration point and the next registration point includes:

[0042] The first direction vector is calculated based on the positioning module of the bronchoscope;

[0043] Calculate the direction vector to be executed based on the current registration point and the next registration point;

[0044] The rotation angle is calculated based on the first direction vector and the direction vector to be executed;

[0045] Based on the rotation angle, rotate the tip of the bronchoscope catheter to the next registration point.

[0046] In one embodiment, the second movement strategy includes:

[0047] The bronchoscope is moved according to the first direction vector and the second step size.

[0048] Secondly, this application also provides a bronchoscopy control device. The device includes:

[0049] The acquisition module is used to acquire a pre-built bronchial model, which includes multiple registration points;

[0050] The acquisition module is used to acquire real endoscopic images of the bronchi during movement;

[0051] The processing module is used to generate movement commands based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point;

[0052] The motion module is used to control the bronchoscope to move sequentially to each of the registration points along a preset path according to the movement command.

[0053] In one embodiment, the device further includes:

[0054] The compensation module is used to perform position compensation for the bronchoscope during movement.

[0055] The breathing phase recognition module is used to detect the current phase of the breathing of the subject under test;

[0056] The acquisition module is used to acquire the pose-compensated real endoscopic image of the bronchoscope when the current phase is the target phase.

[0057] Thirdly, this application also provides an automatic registration method for a bronchoscope. The method includes:

[0058] Obtain a pre-constructed bronchial model, which includes multiple bifurcation points, and select a number of registration points from the bifurcation points;

[0059] Using the bronchoscope control method described in any one of the first aspects, the bronchoscope is moved sequentially to each of the registration points, and pose sampling is performed at each of the registration points to obtain sampling data;

[0060] Based on the sampled data, a mapping relationship between the bronchus model and the bronchus is constructed to complete the automatic registration of the bronchoscope.

[0061] Fourthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method steps of the bronchoscope control method in the first aspect and the automatic registration method for the bronchoscope in the third aspect.

[0062] Fifthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the method steps of the bronchoscope control method in the first aspect and the automatic registration method for the bronchoscope in the third aspect.

[0063] Sixthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the method steps of the bronchoscope control method in the first aspect and the automatic registration method of the bronchoscope in the third aspect.

[0064] The above-mentioned bronchoscopy control method, device, automatic registration method, and computer equipment have at least the following advantages:

[0065] This application acquires real endoscopic images of the bronchus during the movement of the bronchoscope. Based on the real endoscopic images and the virtual endoscopic images corresponding to the current registration points, the bronchoscope is controlled to automatically move to each registration point along a preset path, providing accurate pose for the bronchial model and improving the accuracy of virtual endoscopic rendering. Attached Figure Description

[0066] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is an application environment diagram of the bronchoscopy control method and automatic registration method in one embodiment;

[0069] Figure 2 This is a flowchart illustrating a bronchoscopy control method in one embodiment;

[0070] Figure 3 This is a schematic diagram of the structure of a human bronchus in one embodiment;

[0071] Figure 4 This is a schematic diagram of catheter movement in one embodiment;

[0072] Figure 5 This is a schematic diagram of the pose compensation process in one embodiment;

[0073] Figure 6 This is a schematic diagram illustrating the correspondence between spatial coordinates and virtual coordinates in one embodiment;

[0074] Figure 7 This is a diagram illustrating the effect of pose compensation in one embodiment;

[0075] Figure 8 This is a schematic diagram of the process for controlling the movement of the bronchoscope in one embodiment;

[0076] Figure 9 This is a schematic diagram of a bronchoscope moving according to a first moving strategy in one embodiment;

[0077] Figure 10 This is a schematic diagram illustrating the movement of the bronchoscope according to a first movement strategy in another embodiment;

[0078] Figure 11 This is a structural block diagram of a bronchoscope control device in one embodiment;

[0079] Figure 12 This is a schematic diagram of the installation of an electromagnetic sensor in one embodiment;

[0080] Figure 13 This is a schematic diagram of the bending of the catheter tip in one embodiment;

[0081] Figure 14 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0082] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0083] Some exemplary embodiments of the invention have been described for illustrative purposes. It should be understood that the invention may be implemented in other ways not specifically shown in the accompanying drawings.

[0084] The bronchoscope control method and automatic bronchoscope registration method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server.

[0085] Specifically, terminal 102 can send pre-captured lung scan images to server 104, enabling server 104 to process the lung scan images, construct a bronchial model, and sequentially move the bronchoscope to the registration points according to a preset path, performing pose sampling at each registration point to obtain sampling data; based on the sampling data, constructing a mapping relationship between the bronchial model and the bronchus, completing the automatic registration of the bronchoscope, and server 104 feeding back the registration results to terminal 102.

[0086] The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0087] Please see Figure 2 In one feasible embodiment, this application provides a bronchoscopy control method, specifically including:

[0088] Step S202: Obtain the pre-constructed bronchial model.

[0089] Specifically, the bronchial model is derived from pre-captured lung scan images, which can be computed tomography (CT) images, magnetic resonance imaging (MRI) images, etc. In this embodiment, the lung scan image is a CT image. A binary image of the bronchus is extracted from the CT image. A bronchial centerline is generated from the binary bronchial image, and then a centerline tree structure is constructed based on the bronchial centerline to obtain the corresponding 3D bronchial model. The tree structure stores the coordinates of points contained in each branch, the parent branch, and the child branches.

[0090] Please see Figure 3 , Figure 3The diagram shows the structure of the human bronchus. The human bronchus resembles an inverted branch and can be divided into multiple bronchial segments according to its direction of extension. Figure 3 The dashed line in the diagram represents the central line of the human bronchus. This central line has multiple nodes, forming a nodal cloud. Each node is further divided into several levels based on its branch. For example... Figure 3 The point on the centerline of the main airway is the root node, and the points on the centerlines of the left and right bronchial segments connected to the main airway are the secondary nodes, and so on.

[0091] Optionally, a bronchial model may be constructed, including:

[0092] The process involves acquiring pre-captured lung scan images, segmenting them, and extracting the bronchial centerline images. Bifurcation points are marked within these images, resulting in multiple bifurcation points. Based on these bifurcation points, the bronchial centerline images are cropped to obtain the centerlines of all bronchial segments. A tree structure of the bronchi is then constructed based on these centerlines, resulting in a bronchial model. Simultaneously with cropping the bronchial centerline images, the node cloud on the centerline is stored. The bronchial topology is reconstructed based on the connections between bronchial segments. The root node stores the centerline point cloud of the main airway, and the child nodes of the root node store the point clouds of the left and right secondary branches, and so on.

[0093] The tree structure of the bronchial centerline includes: a binary tree with the main airway as the root node, or an n-ary tree with the main airway as the root node. The specific structure can be set as needed.

[0094] Furthermore, after extracting the bronchial centerline image, the process further includes: smoothing or deburring the centerline of the bronchial centerline image.

[0095] By utilizing the reconstructed bronchial tree structure, the bronchial region where the catheter is currently located can be quickly located during automatic registration, and prior knowledge can be provided for subsequent automatic catheter movement, thereby improving the accuracy of automatic registration.

[0096] Furthermore, the bronchial model includes multiple registration points located on the centerline. The position and number of registration points can be preset as needed. Generally, registration points are selected from the bifurcation points.

[0097] It should be understood that the registration points and bifurcation points mentioned above are all nodes on the centerline.

[0098] Step S204: Move the bronchoscope to each registration point sequentially according to the preset path.

[0099] Specifically, the bronchoscope's movement path is pre-set based on the registration point's location, and the steps for moving the bronchoscope include:

[0100] During movement, real endoscopic images are acquired. The movement of the bronchoscope is controlled based on these real endoscopic images and the virtual endoscopic image corresponding to the current registration point. It should be understood that the bronchoscope's catheter tip is actively bendable and equipped with a monocular endoscope. Once the catheter is inserted into the bronchus, real endoscopic images of the bronchus can be acquired in real time through the monocular endoscope. The virtual endoscopic images are acquired using a virtual camera based on a bronchial model. In the bronchial model, the focal length of the virtual camera is calculated based on the node preceding the registration point in the current bronchial segment. The coordinates of the registration point are used as the camera position to acquire the virtual endoscopic image of the registration point. It should be understood that, following the above steps, virtual endoscopic images corresponding to all nodes on the centerline can be acquired. Furthermore, the virtual endoscopic images store corresponding virtual camera pose parameters, including the virtual coordinates and focal length of the virtual camera.

[0101] Please see Figure 4 For example, registration points A1-A5 are pre-set, and the movement path is from registration point A1 to registration point A5 sequentially. First, the bronchoscope tube is inserted into the bronchus. The rotation angle of the tube tip is controlled, and the tube is moved along the centerline to the current registration point A1. During the movement of the tube, a real endoscopic image of the bronchus is acquired, and simultaneously, a virtual endoscopic image corresponding to registration point A1 is retrieved. Based on the real endoscopic image and the virtual endoscopic image corresponding to registration point A1, it is determined whether registration point A1 has been reached. The frequency of acquiring real endoscopic images can be set as needed; generally, a higher frequency results in higher accuracy in subsequent judgments.

[0102] Upon reaching registration point A1, the bronchoscope is rotated to move the catheter tip to registration points A2 and A3 sequentially. The same method used to check registration points A2 and A3 is employed to confirm their arrival. Upon reaching registration point A3, the catheter tip is rotated to exit the current bronchial segment, passing through registration point A2 back to registration point A1. The process is then repeated to registration points A4 and A5, thus reaching each registration point.

[0103] The above-mentioned bronchoscopy control method acquires real endoscopic images of the bronchus during the movement of the bronchoscope. Based on the real endoscopic images and the virtual endoscopic images corresponding to the current registration points, the bronchoscope is controlled to automatically move to each registration point along a preset path, providing accurate pose for the bronchial model and improving the accuracy of virtual endoscopic rendering.

[0104] In one feasible embodiment, acquiring real endoscopic images of the bronchus during movement includes:

[0105] The current phase of the subject's breathing is detected, and when the current phase is detected to be the target phase, a real endoscopic image is acquired.

[0106] Specifically, during the automatic movement of the bronchoscope, coordinate shifts are easily caused by human respiration, resulting in sampling points falling outside the bronchial model. Consequently, the rendered virtual endoscopic image also falls outside the bronchus, affecting navigation performance. Therefore, this embodiment detects the respiration of the subject, generates corresponding waveforms, and performs phase analysis based on the waveforms. In this embodiment, the target phase is set as the phase corresponding to the end of inspiration. When the current phase is detected to be below a preset threshold, it is considered that the target phase has been reached. At this time, a real endoscopic image is acquired using a monocular endoscope at the tip of the catheter.

[0107] By adopting the above scheme and setting the sampling time to the end of inhalation, the deviation of the bronchoscope caused by human breathing can be avoided, making the collected real endoscopic images more accurate and further improving the accuracy of virtual endoscopic rendering.

[0108] In one feasible embodiment, before acquiring real endoscopic images of the bronchus, the method further includes: performing pose compensation on the bronchoscope during movement.

[0109] Specifically, in order to further improve the accuracy of virtual endoscopy rendering, the pose of the bronchoscope is adjusted before sampling so that it has a more similar pose to the coordinate point cloud of the center line where the current registration point is located, thereby completing the compensation of respiratory motion.

[0110] Please see Figure 5 Posture compensation for bronchoscopy includes:

[0111] Step S502: Obtain the current virtual coordinates of the bronchoscope.

[0112] Specifically, the bronchoscope has spatial coordinates and virtual coordinates. The spatial coordinates are the actual coordinates of the bronchoscope, obtained from the bronchoscope's positioning module. The virtual coordinates are the coordinates of the bronchoscope within the bronchial model. For any node on the center line of the bronchial model, to maintain coordinate consistency, there is an automatic registration matrix corresponding to the spatial and virtual coordinates. It should be noted that the positioning module is located at the tip of the bronchoscope's catheter; therefore, in this embodiment, the spatial coordinates of the bronchoscope essentially refer to the spatial coordinates of the tip of the bronchoscope's catheter.

[0113] Please see Figure 6 Obtain the automatic registration matrix, including:

[0114] A predetermined number of bifurcation points are selected as marker points from the bronchial model. It should be understood that the more marker points selected, the higher the accuracy of the automatic registration matrix. For example... Figure 6Five bifurcation points were selected, namely markers B1-B5; the spatial coordinates, virtual coordinates, pre-acquired real endoscopic images, and virtual endoscopic images of these five marker points were obtained.

[0115] For each marker point, calculate the rotation and translation matrix that transforms its spatial coordinates and virtual coordinates between the two coordinate systems, resulting in five rotation and translation matrices. For example, for marker points B1-B5, calculate the rotation and translation matrices T1-T5 respectively.

[0116] The similarity between the virtual and real endoscopic images corresponding to all marked points is calculated for each rotation-translation matrix. The rotation-translation matrix whose similarity satisfies a preset condition is selected as the automatic registration matrix. For example, the spatial coordinates of marked points B1-B5 are multiplied by the rotation-translation matrix T1 to obtain five virtual coordinates. The similarity between the virtual and real endoscopic images corresponding to each virtual coordinate is then calculated. In this embodiment, the preset condition for satisfying the similarity is set to the maximum sum of similarities, meaning that the five virtual coordinates correspond to five similarities, and the sum of these five similarities is calculated. Similarly, the spatial coordinates of marked points B1-B5 are multiplied by the rotation-translation matrix T2 to obtain a sum of similarities. This process is repeated for rotation-translation matrices T3-T5. The rotation-translation matrix with the maximum sum of similarities, such as that for marked point B1, is selected as the automatic registration matrix. This automatic registration matrix is ​​a rigid transformation matrix that does not change during the entire movement of the bronchoscope.

[0117] Multiply the current spatial coordinates of the bronchoscope by the automatic registration matrix to convert them into the current virtual coordinates in the bronchial model.

[0118] Step S504: Obtain the virtual endoscopic image corresponding to the current virtual coordinates.

[0119] Specifically, during the automatic movement of the bronchoscope, virtual endoscopic images corresponding to nodes on each centerline can be acquired according to preset time intervals or preset step sizes. In order to make the acquired virtual endoscopic images closer to the real endoscopic images, this embodiment adjusts the lens orientation of the virtual camera within a preset angle at each node to obtain virtual endoscopic images from multiple angles.

[0120] Step S506: Determine the optimal virtual endoscopic image and obtain the virtual camera pose parameters corresponding to the optimal virtual endoscopic image.

[0121] Specifically, the catheter is first placed at the starting point of the main airway centerline. As the bronchoscope moves automatically along the centerline according to the preset path, the spatial coordinates of the bronchoscope are acquired in real time. Based on the automatic registration matrix, the spatial coordinates are converted into corresponding virtual coordinates, thereby obtaining the current pose parameters of the bronchoscope catheter. These pose parameters include the catheter's direction vector, spatial coordinates, and virtual coordinates.

[0122] For a node on the centerline traversed by the bronchoscope, virtual endoscopic images from multiple angles are acquired at that node. These virtual images are then iterated through to determine the optimal one. The optimal virtual endoscopic image is determined by having the highest similarity to the real endoscopic image at the current virtual coordinates. It should be understood that the above optimization algorithm can be implemented in various ways, such as using the Powell algorithm for image optimization, setting the objective function as the similarity between the real and virtual endoscopic images, and iteratively optimizing to obtain the virtual endoscopic image most similar to the real endoscopic image. Using the method of calculating the similarity between virtual and real endoscopic images, rather than the similarity between real and real endoscopic images, can address the impact of brightness variations in the catheter within the human bronchus, improving the accuracy of automatic registration.

[0123] Obtain the virtual camera pose parameters corresponding to the optimal virtual endoscopic image.

[0124] Step S508: Perform pose compensation based on the virtual camera pose parameters.

[0125] Please see Figure 7 , Figure 7 The effect of pose compensation. Figure 7 As can be seen, before compensation, the sampling point was located outside the bronchial model due to the offset caused by respiration. After compensation, the bronchoscope was adjusted to the virtual coordinate position corresponding to the virtual camera of the current node, keeping the sampling point inside the bronchus. This achieved dynamic pose compensation, ensuring consistency between the virtual and real endoscopes and improving the accuracy of the movement path. Furthermore, due to the complex bronchial tree structure, optimizing multiple virtual endoscopic images of a node, rather than performing image matching on all centerlines, significantly reduced the computational load and met the requirements of real-time computing.

[0126] Please see Figure 8 In one feasible embodiment, controlling the movement of the bronchoscope based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point includes:

[0127] Step S802: Obtain the current virtual coordinates of the bronchoscope and the optimal virtual image of the current virtual coordinates. The steps of obtaining the virtual coordinates and the optimal virtual image are the same as those in steps S502-S506, and will not be repeated here for the sake of brevity.

[0128] Step S804: Calculate the second similarity, where the second similarity is obtained by comparing the optimal virtual image and the virtual endoscopic image of the current registration point.

[0129] Step S806: Determine the registration area and control the movement of the bronchoscope according to the first movement strategy.

[0130] Step S808: Determine the non-registration area and control the movement of the bronchoscope according to the second movement strategy.

[0131] Specifically, the second similarity is compared with a preset threshold. If the second similarity is greater than or equal to the preset threshold, the region within the first preset range of the current virtual coordinates is recorded as the region to be registered; otherwise, the region within the second preset range of the current virtual coordinates is recorded as the non-registered region.

[0132] It should be noted that the distinction between the registration area and the non-registration area is to facilitate control of the bronchoscope's movement. In the registration area, it is necessary to determine the rotation angle of the bronchoscope's tube and whether it has reached the registration point; therefore, more precise control is required. In the non-registration area, the bronchoscope only needs to be moved forward or backward according to the direction vector entering the bronchial segment. The movement process will be explained in detail below.

[0133] In one feasible embodiment, the first movement strategy includes:

[0134] Based on the first direction vector, the bronchoscope is moved one step at a time, where the first direction vector is the direction vector of the bronchial segment where the bronchoscope is currently located. Once the bronchoscope reaches the current registration point, the rotation angle is calculated based on the current registration point and the next registration point. Based on the rotation angle, the tip of the bronchoscope's catheter is rotated to proceed to the next registration point.

[0135] Please see Figure 9 and Figure 10 , Figure 9 This is a schematic diagram illustrating the movement of the bronchoscope according to the first movement strategy. For bronchial segments C1 and C2, where registration point C1 is located, the tip of the bronchoscope is rotated until it is parallel to the first direction vector corresponding to bronchial segment C1. Based on this first direction vector, the bronchoscope moves forward within bronchial segment C1 by a first step length. It should be understood that at this point, the bronchoscope is close to registration point C1, and the first step length can be set relatively small, for example, to 0.1 mm. Once the bronchoscope has reached registration point C1, the registration area corresponding to registration point C1 becomes invalid. At this point, the bronchoscope needs to be moved towards registration point C2.

[0136] To further clarify, determining that the bronchoscope has reached the registration point includes:

[0137] During the movement, based on the length of the first step, after moving to the next node, the second similarity corresponding to that node is calculated. If the next second similarity is less than the previous second similarity, it is considered that the bronchoscope has reached the current registration point C1.

[0138] It should be noted that during the automatic movement of the bronchoscope, end-inspiratory sampling and bronchoscope pose compensation are continuously performed to counteract the offset caused by respiration. Simultaneously, the optimization parameters are updated after pose compensation to ensure a more accurate optimal virtual image. These optimization parameters include the optical center coordinates, focal length vector, orientation vector, duct spatial coordinates, and virtual coordinates of the virtual camera at each moment.

[0139] Generally speaking, the closer the bronchoscope is to the registration point C1, the greater the second similarity. Therefore, if the second similarity decreases at the next moment, it is considered that the bronchoscope has reached the registration point C1.

[0140] To further explain, the rotation angle is calculated based on the current registration point and the next registration point, including:

[0141] Based on the bronchoscope's positioning module, the spatial and virtual coordinates of the bronchoscope are acquired in real time, and the first direction vector is calculated. Based on the terminal node of bronchial segment C1 and the head node of bronchial segment C2, the direction vector to be executed is calculated. For example... Figure 10 As shown, the first direction vector is the direction indicated by the arrow in bronchial segment C1, and the direction vector to be executed is the direction indicated by the arrow in bronchial segment C2.

[0142] Based on the first direction vector and the direction vector to be executed, the rotation angle is calculated. The bronchoscope's catheter tip is then rotated according to this rotation angle until it is parallel to the direction vector to be executed, and then proceeds to the next registration point C2. To ensure that the catheter tip is positioned as close as possible to the centerline of bronchial segment C2 after rotation, in this embodiment, when the catheter reaches registration point C1, it continues to advance along the first direction vector by a preset distance, for example, 2 mm, before rotating the catheter tip again and entering bronchial segment C2 according to the direction vector to be executed.

[0143] Please see Figure 10 If the next registration point after registration point C2 is registration point C3, then after reaching registration point C2, follow the above steps to first calculate the next direction vector to be executed, then move backward according to the direction vector of bronchial segment C2, exit bronchial segment C2, return to bronchial segment C1, and proceed to registration point C3.

[0144] In one feasible embodiment, the second movement strategy includes:

[0145] The bronchoscope is moved according to the first direction vector and the second step size. It should be understood that the portion of each bronchial segment outside the registration area is the non-registration area. Within the non-registration area, the rotation direction of the catheter tip does not need to be considered; it only needs to move forward or backward according to the first direction vector. Therefore, the value of the second step size can be greater than the first step size. In this embodiment, the second step size is set to 1 mm. Simultaneously, to reduce computational load, in the non-registration area, the virtual endoscopic image corresponding to the node is acquired, and then the virtual camera optical center of that node is traversed in the virtual position of the bronchial model, without considering focal length or orientation. Furthermore, the moving speed in the non-registration area can be slightly greater than the moving speed in the registration area; for example, moving quickly when far from the registration point and moving slowly when approaching the registration point.

[0146] The aforementioned bronchoscopy control method detects the spatial coordinates of the bronchoscope in real time, calculates the direction vector of the bronchial segment where the bronchoscope is currently located, and rotates the tip of the bronchoscope according to the direction vector. Following a preset path, it automatically performs forward and backward movements along the centerline to reach each registration point sequentially. Simultaneously, during movement, it performs end-inspiratory sampling and bronchoscope pose compensation to offset deviations caused by respiration. After pose compensation, it updates the optimization parameters to ensure control accuracy during movement.

[0147] In order to enable those skilled in the art to fully understand this application, the steps of the bronchoscopy control method of this application are described in detail below:

[0148] Before performing bronchoscopy, this application first acquires lung scan images of the subject, segments the lung scan images, extracts the bronchial centerline images, marks the bifurcation parts of the bronchial centerline images to obtain multiple bifurcation points, and based on each bifurcation point, crops the bronchial centerline images to obtain the centerlines of all bronchial segments. Based on the centerlines of each bronchial segment, a tree structure of the bronchi is constructed to obtain a bronchial model.

[0149] After obtaining the bronchial model, multiple nodes are selected from the nodes along the bronchial centerline as registration points, and a movement path is pre-defined based on the location of the registration points. A virtual camera is used to acquire virtual endoscopic images of each node along the centerline.

[0150] After obtaining the bronchial model, multiple bifurcation points are selected as marker points, and real endoscopic images of each marker point are pre-acquired. Based on the spatial coordinates and virtual coordinates of the marker points, the rotation and translation matrix is ​​obtained, and the rotation and translation matrix that meets the preset conditions is selected as the automatic registration matrix.

[0151] During bronchoscopy, the bronchoscope tube is first placed at the starting point of the main airway centerline and then automatically moves along the bronchial centerline according to a preset path. During this movement, real endoscopic images are acquired at the end of inspiration, and the bronchoscope's current spatial coordinates are also acquired in real time. Based on an automatic registration matrix, these spatial coordinates are converted to current virtual coordinates. Multiple virtual endoscopic images at preset angles are obtained from these virtual coordinates. These virtual images are then traversed, and the virtual endoscopic image with the highest similarity to the real endoscopic image at the current virtual coordinates is selected as the optimal virtual endoscopic image. The virtual camera pose parameters corresponding to the optimal virtual endoscopic image are then obtained. Based on these virtual camera pose parameters, the bronchoscope's pose is compensated to ensure it remains on the bronchial centerline at all times.

[0152] Simultaneously, during the automatic movement of the bronchoscope, the first direction vector of the current bronchial segment and the direction vector to be executed for moving to the next registration point are calculated based on the real-time acquired virtual coordinates. Based on the first direction vector and the direction vector to be executed, the rotation angle of the bronchoscope's catheter tip is controlled to allow the bronchoscope to enter or exit each bronchial segment.

[0153] Simultaneously, during the automatic movement of the bronchoscope, a second similarity is calculated between the optimal virtual endoscope image corresponding to the real-time acquired virtual coordinates and the virtual endoscope image at the current registration point. If the second similarity is less than a preset threshold, the bronchoscope is considered to be in a non-registration area. In this case, the bronchoscope's movement speed can be appropriately increased, and it can move forward or backward within the bronchial segment according to the first direction vector. If the second similarity is greater than or equal to the preset threshold, the bronchoscope is considered to have entered the registration area. In this case, the bronchoscope's movement speed should be reduced, and the optimization parameters should be updated after pose compensation to ensure a more accurate optimal virtual image. During the forward movement, if the next second similarity is less than the previous second similarity, the bronchoscope is considered to have reached the current registration point. At this time, the current registration area is canceled, and the rotation angle of the bronchoscope's catheter tip is controlled according to the first direction vector and the direction vector to be executed, moving to the next registration point.

[0154] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0155] Based on the same inventive concept, this application also provides a bronchoscopy control device for implementing the bronchoscopy control method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more bronchoscopy control device embodiments provided below can be found in the limitations of the bronchoscopy control method described above, and will not be repeated here.

[0156] Please see Figure 11 In one feasible embodiment, this application provides a bronchoscopy control device, including: an acquisition module, a data collection module, a processing module, and a motion module, wherein:

[0157] The acquisition module is used to acquire a pre-built bronchial model, which includes multiple registration points.

[0158] The acquisition module is used to acquire real endoscopic images of the bronchi during movement.

[0159] Specifically, the acquisition module is located inside the bronchoscope's catheter, and in this embodiment, a monocular endoscope is used to acquire image information.

[0160] The processing module is used to generate movement commands based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point.

[0161] The motion module is used to control the bronchoscope to move sequentially to each of the preset registration points according to the movement commands. In this embodiment, the motion module is a bronchoscope robot.

[0162] In one feasible embodiment, the bronchoscopy control device of this application further includes:

[0163] The breathing phase recognition module is used to detect the current phase of the breathing of the subject under test.

[0164] The processing module is also used to determine the target phase based on the current phase and acquire real endoscopic images at the target phase.

[0165] In one feasible embodiment, the bronchoscopy control device of this application further includes:

[0166] The compensation module is used to compensate for the position of the bronchoscope during movement.

[0167] The positioning module is used to obtain the spatial coordinates of the bronchoscope.

[0168] Specifically, in this embodiment, the positioning module includes a magnetic field generator and an electromagnetic sensor mounted on the bronchoscope's catheter. The magnetic field generator is positioned on the outside of the object to be measured. When the bronchoscope is inserted into the human bronchus, the magnetic field generator generates a magnetic field. By collecting data from the electromagnetic sensor, the real pose of the electromagnetic sensor in the electromagnetic coordinate system can be obtained in real time, thereby determining the real pose of the catheter. This real pose can be used for selecting marker coordinates and rendering virtual endoscopy.

[0169] Please see Figure 12 and Figure 13 The tip of the catheter is a flexible section, such as... Figure 12 As shown, in this embodiment, there are four electromagnetic sensors, symmetrically arranged at both ends of the conduit. Figure 13 As shown, the length of the catheter tip is known, denoted as 'a'. From this, an arc of length 'a' passing through the center points of both ends of the catheter can be fitted. Let the end of the catheter closest to the bronchus be the catheter head. Then, the tangent direction of this arc at the center point of the catheter head is taken as the direction of the bronchoscope, and the coordinates of the center point of the catheter head are taken as the spatial coordinates of the bronchoscope. The pose of the catheter and the monocular endoscope in the electromagnetic coordinate system can be calculated using the above method.

[0170] The processing module is also used to calculate the automatic registration matrix and convert the spatial coordinates into corresponding virtual coordinates based on the automatic registration matrix.

[0171] In one feasible embodiment, the bronchoscopy control device of this application further includes:

[0172] The model building module is used to construct a bronchial model based on pre-captured lung scan images.

[0173] The display module, connected to the processing module, is used to display real and virtual endoscopic images in real time for the operator to view.

[0174] The aforementioned bronchoscope control device detects the spatial coordinates of the bronchoscope in real time, calculates the direction vector of the bronchial segment where the bronchoscope is currently located, and rotates the front end of the bronchoscope according to the direction vector. Following a preset path, it automatically performs forward and backward movements along the centerline to reach each registration point sequentially. Simultaneously, during movement, it performs end-inspiratory sampling and bronchoscope pose compensation to offset deviations caused by respiration. After pose compensation, it updates the optimization parameters, ensuring precise control of the movement.

[0175] Each module in the aforementioned bronchoscopy control device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0176] Based on the same inventive concept, embodiments of this application also provide an automatic registration method for a bronchoscope, comprising:

[0177] Obtain a pre-built bronchial model, which includes multiple bifurcation points, and select a number of registration points from the bifurcation points;

[0178] Using the bronchoscope control method described in the above embodiments, the bronchoscope is moved sequentially to each registration point, and pose sampling is performed at each registration point to obtain sampling data;

[0179] Based on the sampled data, a mapping relationship between the bronchial model and the bronchus is constructed to complete the automatic registration of the bronchoscope.

[0180] The above-mentioned automatic registration method for bronchoscopes can automatically move to each registration point according to a preset path. During the movement, it performs end-inspiratory sampling and bronchoscope posture compensation to offset the offset caused by breathing. After posture compensation, it updates the optimization parameters to ensure the accuracy of bronchoscope movement, thereby obtaining more accurate sampling data and improving the accuracy of automatic registration.

[0181] In one feasible embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 14As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a bronchoscopy control method and an automatic registration method for the bronchoscopy. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0182] Those skilled in the art will understand that Figure 14 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0183] In one feasible embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps in the above-described bronchoscopy control method and the automatic registration method of the bronchoscopy.

[0184] In one feasible embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method steps in the above-described bronchoscopy control method and the automatic registration method for bronchoscopy.

[0185] In one feasible embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the method steps in the above-described bronchoscopy control method and the automatic registration method for bronchoscopy.

[0186] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0187] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0188] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A bronchoscopy control device, characterized in that, The device includes: The acquisition module is used to acquire a pre-built bronchial model, which includes multiple registration points; The acquisition module is used to acquire real endoscopic images of the bronchi during movement; The processing module is used to generate movement commands based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point; The motion module is used to control the bronchoscope to move sequentially to each of the registration points along a preset path according to the movement command. The compensation module is used to obtain the current virtual coordinates of the bronchoscope; obtain multiple virtual endoscopic images of the current virtual coordinates at a preset angle; traverse each of the virtual endoscopic images to determine the optimal virtual endoscopic image, wherein the optimal virtual endoscopic image has the highest first similarity to the real endoscopic image of the current virtual coordinates; obtain the virtual camera pose parameters corresponding to the optimal virtual endoscopic image; and perform pose compensation on the bronchoscope according to the virtual camera pose parameters.

2. The apparatus according to claim 1, characterized in that, The device further includes: The compensation module is used to perform position compensation for the bronchoscope during movement. The breathing phase recognition module is used to detect the current phase of the breathing of the subject under test; The acquisition module is used to acquire the pose-compensated real endoscopic image of the bronchoscope when the current phase is the target phase.

3. The apparatus according to claim 2, characterized in that, The compensation module is further configured to obtain the current spatial coordinates of the bronchoscope according to the positioning module of the bronchoscope; obtain an automatic registration matrix; and convert the current spatial coordinates of the bronchoscope into the current virtual coordinates in the bronchial model according to the automatic registration matrix.

4. The apparatus according to claim 3, characterized in that, The bronchial model also includes multiple bifurcation points; The compensation module is also used to select a preset number of bifurcation points from the bronchial model as marker points; and to acquire real endoscopic images of each marker point in advance. Based on the spatial coordinates and virtual coordinates of the marker points, a rotation and translation matrix is ​​obtained; the similarity between the virtual and real endoscopic images corresponding to all the marker points is calculated for each rotation and translation matrix, and the rotation and translation matrix whose similarity satisfies the preset conditions is selected as the automatic registration matrix.

5. The apparatus according to claim 1 or 2, characterized in that, The device is also used to obtain the current virtual coordinates of the bronchoscope; obtain the optimal virtual image of the current virtual coordinates; and calculate the second similarity between the optimal virtual image and the virtual endoscopic image of the current registration point. If the second similarity is greater than or equal to a preset threshold, the region within the first preset range of the current virtual coordinates is recorded as the region to be registered, and the movement of the bronchoscope is controlled within the region to be registered according to the first movement strategy; if the second similarity is less than the preset threshold, the region within the second preset range of the current virtual coordinates is recorded as the non-registered region, and the movement of the bronchoscope is controlled within the non-registered region according to the second movement strategy.

6. The apparatus according to claim 5, characterized in that, The first movement strategy includes: moving the bronchoscope by a step length according to a first direction vector, wherein the first direction vector is the direction vector of the bronchial segment where the bronchoscope is currently located; after the bronchoscope reaches the current registration point, calculating the rotation angle according to the current registration point and the next registration point; and rotating the front end of the bronchoscope according to the rotation angle to move to the next registration point.

7. The apparatus according to claim 6, characterized in that, The device is also used to calculate the second similarity during the movement process, and if the next second similarity is less than the previous second similarity, it is considered that the bronchoscope has reached the current registration point.

8. The apparatus according to claim 6, characterized in that, The device is further configured to calculate the first direction vector based on the positioning module of the bronchoscope; calculate the direction vector to be executed based on the current registration point and the next registration point; and calculate the rotation angle based on the first direction vector and the direction vector to be executed. Based on the rotation angle, rotate the tip of the bronchoscope catheter to the next registration point.

9. The apparatus according to claim 8, characterized in that, The second movement strategy includes: The bronchoscope is moved according to the first direction vector and the second step size.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it performs the following steps: Obtain a pre-constructed bronchial model, which includes multiple registration points; Following a preset path, the bronchoscope is moved sequentially to each of the aforementioned registration points, including: Acquire real endoscopic images during movement; The movement of the bronchoscope is controlled based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point; When the processor executes the computer program, it also performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain multiple virtual endoscopic images of the current virtual coordinates at a preset angle; Traverse all the virtual endoscopic images to determine the optimal virtual endoscopic image, wherein the optimal virtual endoscopic image has the highest first similarity to the real endoscopic image at the current virtual coordinates; Obtain the virtual camera pose parameters corresponding to the optimal virtual endoscopic image; The bronchoscope is subjected to pose compensation based on the virtual camera pose parameters.

11. The computer device according to claim 10, characterized in that, When the processor executes the computer program, it also performs the following steps: During the movement, the bronchoscope is positionally compensated, and the current phase of the subject's breathing is detected; When the current phase is the target phase, the actual endoscopic image of the bronchoscope after pose compensation is acquired.

12. The computer device according to claim 11, characterized in that, When the processor executes the computer program, it also performs the following steps: The current spatial coordinates of the bronchoscope are obtained based on the positioning module of the bronchoscope. Obtain the automatic registration matrix; Based on the automatic registration matrix, the current spatial coordinates of the bronchoscope are converted into the current virtual coordinates in the bronchial model.

13. The computer device according to claim 12, characterized in that, The bronchial model also includes multiple bifurcation points; When the processor executes the computer program, it also performs the following steps: A predetermined number of bifurcation points are selected from the bronchial model as marker points; Acquire real endoscopic images of each of the aforementioned marker points in advance; Based on the spatial coordinates and virtual coordinates of the marked points, the rotation and translation matrix is ​​obtained; Calculate the similarity between the virtual and real endoscopic images corresponding to all the marked points for each rotation and translation matrix, and select the rotation and translation matrix whose similarity satisfies the preset conditions as the automatic registration matrix.

14. The computer device according to claim 10 or 11, characterized in that, When the processor executes the computer program, it also performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain the optimal virtual image at the current virtual coordinates; Calculate the second similarity between the optimal virtual image and the virtual endoscopic image at the current registration point; If the second similarity is greater than or equal to a preset threshold, the region within the first preset range of the current virtual coordinates is recorded as the region to be registered, and the movement of the bronchoscope is controlled within the region to be registered according to the first movement strategy. If the second similarity is less than a preset threshold, the region within the second preset range of the current virtual coordinates is recorded as a non-registered region, and the movement of the bronchoscope is controlled within the non-registered region according to the second movement strategy.

15. The computer device according to claim 14, characterized in that, When the processor executes the computer program, it also performs the following steps: The bronchoscope is moved according to the first direction vector by the first step length, where the first direction vector is the direction vector of the bronchial segment where the bronchoscope is currently located. Once the bronchoscope reaches the current registration point, the rotation angle is calculated based on the current registration point and the next registration point. Based on the stated rotation angle, rotate the tip of the bronchoscope to the next registration point.

16. The computer device according to claim 15, characterized in that, When the processor executes the computer program, it also performs the following steps: During the movement, the second similarity is calculated. If the next second similarity is less than the previous second similarity, it is considered that the bronchoscope has reached the current registration point.

17. The computer device according to claim 15, characterized in that, When the processor executes the computer program, it also performs the following steps: The first direction vector is calculated based on the positioning module of the bronchoscope; Calculate the direction vector to be executed based on the current registration point and the next registration point; The rotation angle is calculated based on the first direction vector and the direction vector to be executed; Based on the rotation angle, rotate the tip of the bronchoscope catheter to the next registration point.

18. The computer device according to claim 17, characterized in that, When the processor executes the computer program, it also performs the following steps: The bronchoscope is moved according to the first direction vector and the second step size.

19. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it performs the following steps: Obtain a pre-constructed bronchial model, which includes multiple bifurcation points, and select a number of registration points from the bifurcation points; Obtain a pre-constructed bronchial model, which includes multiple registration points; Following a preset path, the bronchoscope is moved sequentially to each of the aforementioned registration points, including: Acquire real endoscopic images during movement; Based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point, the bronchoscope is controlled to move sequentially to each registration point, and pose sampling is performed at each registration point to obtain sampling data; Based on the sampled data, a mapping relationship between the bronchus model and the bronchus is constructed to complete the automatic registration of the bronchoscope; When the processor executes the computer program, it also performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain multiple virtual endoscopic images of the current virtual coordinates at a preset angle; Traverse all the virtual endoscopic images to determine the optimal virtual endoscopic image, wherein the optimal virtual endoscopic image has the highest first similarity to the real endoscopic image at the current virtual coordinates; Obtain the virtual camera pose parameters corresponding to the optimal virtual endoscopic image; The bronchoscope is subjected to pose compensation based on the virtual camera pose parameters.

20. The computer device according to claim 19, characterized in that, When the processor executes the computer program, it also performs the following steps: During the movement, the bronchoscope is positionally compensated, and the current phase of the subject's breathing is detected; When the current phase is the target phase, the actual endoscopic image of the bronchoscope after pose compensation is acquired.

21. The computer device according to claim 20, characterized in that, When the processor executes the computer program, it also performs the following steps: The current spatial coordinates of the bronchoscope are obtained based on the positioning module of the bronchoscope. Obtain the automatic registration matrix; Based on the automatic registration matrix, the current spatial coordinates of the bronchoscope are converted into the current virtual coordinates in the bronchial model.

22. The computer device according to claim 21, characterized in that, The bronchial model also includes multiple bifurcation points; When the processor executes the computer program, it also performs the following steps: A predetermined number of bifurcation points are selected from the bronchial model as marker points; Acquire real endoscopic images of each of the aforementioned marker points in advance; Based on the spatial coordinates and virtual coordinates of the marked points, the rotation and translation matrix is ​​obtained; Calculate the similarity between the virtual and real endoscopic images corresponding to all the marked points for each rotation and translation matrix, and select the rotation and translation matrix whose similarity satisfies the preset conditions as the automatic registration matrix.

23. The computer device according to claim 19 or 20, characterized in that, When the processor executes the computer program, it also performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain the optimal virtual image at the current virtual coordinates; Calculate the second similarity between the optimal virtual image and the virtual endoscopic image at the current registration point; If the second similarity is greater than or equal to a preset threshold, the region within the first preset range of the current virtual coordinates is recorded as the region to be registered, and the movement of the bronchoscope is controlled within the region to be registered according to the first movement strategy. If the second similarity is less than a preset threshold, the region within the second preset range of the current virtual coordinates is recorded as a non-registered region, and the movement of the bronchoscope is controlled within the non-registered region according to the second movement strategy.

24. The computer device according to claim 23, characterized in that, When the processor executes the computer program, it also performs the following steps: The bronchoscope is moved according to the first direction vector by the first step length, where the first direction vector is the direction vector of the bronchial segment where the bronchoscope is currently located. Once the bronchoscope reaches the current registration point, the rotation angle is calculated based on the current registration point and the next registration point. Based on the stated rotation angle, rotate the tip of the bronchoscope to the next registration point.

25. The computer device according to claim 24, characterized in that, When the processor executes the computer program, it also performs the following steps: During the movement, the second similarity is calculated. If the next second similarity is less than the previous second similarity, it is considered that the bronchoscope has reached the current registration point.

26. The computer device according to claim 24, characterized in that, When the processor executes the computer program, it also performs the following steps: The first direction vector is calculated based on the positioning module of the bronchoscope; Calculate the direction vector to be executed based on the current registration point and the next registration point; The rotation angle is calculated based on the first direction vector and the direction vector to be executed; Based on the rotation angle, rotate the tip of the bronchoscope catheter to the next registration point.

27. The computer device according to claim 26, characterized in that, When the processor executes the computer program, it also performs the following steps: The bronchoscope is moved according to the first direction vector and the second step size.

28. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it performs the following steps: Obtain a pre-constructed bronchial model, which includes multiple registration points; Following a preset path, the bronchoscope is moved sequentially to each of the aforementioned registration points, including: Acquire real endoscopic images during movement; The movement of the bronchoscope is controlled based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point; When the computer program is executed by the processor, it performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain multiple virtual endoscopic images of the current virtual coordinates at a preset angle; Traverse all the virtual endoscopic images to determine the optimal virtual endoscopic image, wherein the optimal virtual endoscopic image has the highest first similarity to the real endoscopic image at the current virtual coordinates; Obtain the virtual camera pose parameters corresponding to the optimal virtual endoscopic image; The bronchoscope is subjected to pose compensation based on the virtual camera pose parameters.

29. The computer-readable storage medium according to claim 28, characterized in that, When the computer program is executed by the processor, it performs the following steps: During the movement, the bronchoscope is positionally compensated, and the current phase of the subject's breathing is detected; When the current phase is the target phase, the actual endoscopic image of the bronchoscope after pose compensation is acquired.

30. The computer-readable storage medium according to claim 28, characterized in that, When the computer program is executed by the processor, it performs the following steps: The current spatial coordinates of the bronchoscope are obtained based on the positioning module of the bronchoscope. Obtain the automatic registration matrix; Based on the automatic registration matrix, the current spatial coordinates of the bronchoscope are converted into the current virtual coordinates in the bronchial model.

31. The computer-readable storage medium according to claim 30, characterized in that, The bronchial model also includes multiple bifurcation points; When the computer program is executed by the processor, it performs the following steps: A predetermined number of bifurcation points are selected from the bronchial model as marker points; Acquire real endoscopic images of each of the aforementioned marker points in advance; Based on the spatial coordinates and virtual coordinates of the marked points, the rotation and translation matrix is ​​obtained; Calculate the similarity between the virtual and real endoscopic images corresponding to all the marked points for each rotation and translation matrix, and select the rotation and translation matrix whose similarity satisfies the preset conditions as the automatic registration matrix.

32. The computer-readable storage medium according to claim 28 or 29, characterized in that, When the computer program is executed by the processor, it performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain the optimal virtual image at the current virtual coordinates; Calculate the second similarity between the optimal virtual image and the virtual endoscopic image at the current registration point; If the second similarity is greater than or equal to a preset threshold, the region within the first preset range of the current virtual coordinates is recorded as the region to be registered, and the movement of the bronchoscope is controlled within the region to be registered according to the first movement strategy. If the second similarity is less than a preset threshold, the region within the second preset range of the current virtual coordinates is recorded as a non-registered region, and the movement of the bronchoscope is controlled within the non-registered region according to the second movement strategy.

33. The computer-readable storage medium according to claim 32, characterized in that, When the computer program is executed by the processor, it performs the following steps: The bronchoscope is moved according to the first direction vector by the first step length, where the first direction vector is the direction vector of the bronchial segment where the bronchoscope is currently located. Once the bronchoscope reaches the current registration point, the rotation angle is calculated based on the current registration point and the next registration point. Based on the stated rotation angle, rotate the tip of the bronchoscope to the next registration point.

34. The computer-readable storage medium according to claim 33, characterized in that, When the computer program is executed by the processor, it performs the following steps: During the movement, the second similarity is calculated. If the next second similarity is less than the previous second similarity, it is considered that the bronchoscope has reached the current registration point.

35. The computer-readable storage medium according to claim 33, characterized in that, When the computer program is executed by the processor, it performs the following steps: The first direction vector is calculated based on the positioning module of the bronchoscope; Calculate the direction vector to be executed based on the current registration point and the next registration point; The rotation angle is calculated based on the first direction vector and the direction vector to be executed; Based on the rotation angle, rotate the tip of the bronchoscope catheter to the next registration point.

36. The computer-readable storage medium according to claim 35, characterized in that, When the computer program is executed by the processor, it performs the following steps: The bronchoscope is moved according to the first direction vector and the second step size.

37. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it performs the following steps: Obtain a pre-constructed bronchial model, which includes multiple bifurcation points, and select a number of registration points from the bifurcation points; Obtain a pre-constructed bronchial model, which includes multiple registration points; Following a preset path, the bronchoscope is moved sequentially to each of the aforementioned registration points, including: Acquire real endoscopic images during movement; Based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point, the bronchoscope is controlled to move sequentially to each registration point, and pose sampling is performed at each registration point to obtain sampling data; Based on the sampled data, a mapping relationship between the bronchus model and the bronchus is constructed to complete the automatic registration of the bronchoscope; When the computer program is executed by the processor, it performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain multiple virtual endoscopic images of the current virtual coordinates at a preset angle; Traverse all the virtual endoscopic images to determine the optimal virtual endoscopic image, wherein the optimal virtual endoscopic image has the highest first similarity to the real endoscopic image at the current virtual coordinates; Obtain the virtual camera pose parameters corresponding to the optimal virtual endoscopic image; The bronchoscope is subjected to pose compensation based on the virtual camera pose parameters.

38. The computer-readable storage medium according to claim 37, characterized in that, When the computer program is executed by the processor, it performs the following steps: During the movement, the bronchoscope is positionally compensated, and the current phase of the subject's breathing is detected; When the current phase is the target phase, the actual endoscopic image of the bronchoscope after pose compensation is acquired.

39. The computer-readable storage medium according to claim 37, characterized in that, When the computer program is executed by the processor, it performs the following steps: The current spatial coordinates of the bronchoscope are obtained based on the positioning module of the bronchoscope. Obtain the automatic registration matrix; Based on the automatic registration matrix, the current spatial coordinates of the bronchoscope are converted into the current virtual coordinates in the bronchial model.

40. The computer-readable storage medium according to claim 39, characterized in that, The bronchial model also includes multiple bifurcation points; When the computer program is executed by the processor, it performs the following steps: A predetermined number of bifurcation points are selected from the bronchial model as marker points; Acquire real endoscopic images of each of the aforementioned marker points in advance; Based on the spatial coordinates and virtual coordinates of the marked points, the rotation and translation matrix is ​​obtained; Calculate the similarity between the virtual and real endoscopic images corresponding to all the marked points for each rotation and translation matrix, and select the rotation and translation matrix whose similarity satisfies the preset conditions as the automatic registration matrix.

41. The computer-readable storage medium according to claim 37 or 38, characterized in that, When the computer program is executed by the processor, it performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain the optimal virtual image at the current virtual coordinates; Calculate the second similarity between the optimal virtual image and the virtual endoscopic image at the current registration point; If the second similarity is greater than or equal to a preset threshold, the region within the first preset range of the current virtual coordinates is recorded as the region to be registered, and the movement of the bronchoscope is controlled within the region to be registered according to the first movement strategy. If the second similarity is less than a preset threshold, the region within the second preset range of the current virtual coordinates is recorded as a non-registered region, and the movement of the bronchoscope is controlled within the non-registered region according to the second movement strategy.

42. The computer-readable storage medium according to claim 41, characterized in that, When the computer program is executed by the processor, it performs the following steps: The bronchoscope is moved according to the first direction vector by the first step length, where the first direction vector is the direction vector of the bronchial segment where the bronchoscope is currently located. Once the bronchoscope reaches the current registration point, the rotation angle is calculated based on the current registration point and the next registration point. Based on the stated rotation angle, rotate the tip of the bronchoscope to the next registration point.

43. The computer-readable storage medium according to claim 42, characterized in that, When the computer program is executed by the processor, it performs the following steps: During the movement, the second similarity is calculated. If the next second similarity is less than the previous second similarity, it is considered that the bronchoscope has reached the current registration point.

44. The computer-readable storage medium according to claim 43, characterized in that, When the computer program is executed by the processor, it performs the following steps: The first direction vector is calculated based on the positioning module of the bronchoscope; Calculate the direction vector to be executed based on the current registration point and the next registration point; The rotation angle is calculated based on the first direction vector and the direction vector to be executed; Based on the rotation angle, rotate the tip of the bronchoscope catheter to the next registration point.

45. The computer-readable storage medium according to claim 44, characterized in that, When the computer program is executed by the processor, it performs the following steps: The bronchoscope is moved according to the first direction vector and the second step size.

46. ​​A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it performs the following steps: Obtain a pre-constructed bronchial model, which includes multiple registration points; Following a preset path, the bronchoscope is moved sequentially to each of the aforementioned registration points, including: Acquire real endoscopic images during movement; The movement of the bronchoscope is controlled based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point; When the processor executes the computer program, it also performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain multiple virtual endoscopic images of the current virtual coordinates at a preset angle; Traverse all the virtual endoscopic images to determine the optimal virtual endoscopic image, wherein the optimal virtual endoscopic image has the highest first similarity to the real endoscopic image at the current virtual coordinates; Obtain the virtual camera pose parameters corresponding to the optimal virtual endoscopic image; The bronchoscope is subjected to pose compensation based on the virtual camera pose parameters.

47. The computer program product according to claim 46, characterized in that, When the processor executes the computer program, it also performs the following steps: During the movement, the bronchoscope is positionally compensated, and the current phase of the subject's breathing is detected; When the current phase is the target phase, the actual endoscopic image of the bronchoscope after pose compensation is acquired.

48. The computer program product according to claim 47, characterized in that, When the processor executes the computer program, it also performs the following steps: The current spatial coordinates of the bronchoscope are obtained based on the positioning module of the bronchoscope. Obtain the automatic registration matrix; Based on the automatic registration matrix, the current spatial coordinates of the bronchoscope are converted into the current virtual coordinates in the bronchial model.

49. The computer program product according to claim 48, characterized in that, The bronchial model also includes multiple bifurcation points; When the processor executes the computer program, it also performs the following steps: A predetermined number of bifurcation points are selected from the bronchial model as marker points; Acquire real endoscopic images of each of the aforementioned marker points in advance; Based on the spatial coordinates and virtual coordinates of the marked points, the rotation and translation matrix is ​​obtained; Calculate the similarity between the virtual and real endoscopic images corresponding to all the marked points for each rotation and translation matrix, and select the rotation and translation matrix whose similarity satisfies the preset conditions as the automatic registration matrix.

50. The computer program product according to claim 46 or 47, characterized in that, When the processor executes the computer program, it also performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain the optimal virtual image at the current virtual coordinates; Calculate the second similarity between the optimal virtual image and the virtual endoscopic image at the current registration point; If the second similarity is greater than or equal to a preset threshold, the region within the first preset range of the current virtual coordinates is recorded as the region to be registered, and the movement of the bronchoscope is controlled within the region to be registered according to the first movement strategy. If the second similarity is less than a preset threshold, the region within the second preset range of the current virtual coordinates is recorded as a non-registered region, and the movement of the bronchoscope is controlled within the non-registered region according to the second movement strategy.

51. The computer program product according to claim 50, characterized in that, When the processor executes the computer program, it also performs the following steps: The bronchoscope is moved according to the first direction vector by the first step length, where the first direction vector is the direction vector of the bronchial segment where the bronchoscope is currently located. Once the bronchoscope reaches the current registration point, the rotation angle is calculated based on the current registration point and the next registration point. Based on the stated rotation angle, rotate the tip of the bronchoscope to the next registration point.

52. The computer program product according to claim 51, characterized in that, When the processor executes the computer program, it also performs the following steps: During the movement, the second similarity is calculated. If the next second similarity is less than the previous second similarity, it is considered that the bronchoscope has reached the current registration point.

53. The computer program product according to claim 51, characterized in that, When the processor executes the computer program, it also performs the following steps: The first direction vector is calculated based on the positioning module of the bronchoscope; Calculate the direction vector to be executed based on the current registration point and the next registration point; The rotation angle is calculated based on the first direction vector and the direction vector to be executed; Based on the rotation angle, rotate the tip of the bronchoscope catheter to the next registration point.

54. The computer program product according to claim 53, characterized in that, When the processor executes the computer program, it also performs the following steps: The bronchoscope is moved according to the first direction vector and the second step size.

55. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it performs the following steps: Obtain a pre-constructed bronchial model, which includes multiple bifurcation points, and select a number of registration points from the bifurcation points; Obtain a pre-constructed bronchial model, which includes multiple registration points; Following a preset path, the bronchoscope is moved sequentially to each of the aforementioned registration points, including: Acquire real endoscopic images during movement; Based on the real endoscopic image and the virtual endoscopic image corresponding to the current registration point, the bronchoscope is controlled to move sequentially to each registration point, and pose sampling is performed at each registration point to obtain sampling data; Based on the sampled data, a mapping relationship between the bronchus model and the bronchus is constructed to complete the automatic registration of the bronchoscope; When the processor executes the computer program, it also performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain multiple virtual endoscopic images of the current virtual coordinates at a preset angle; Traverse all the virtual endoscopic images to determine the optimal virtual endoscopic image, wherein the optimal virtual endoscopic image has the highest first similarity to the real endoscopic image at the current virtual coordinates; Obtain the virtual camera pose parameters corresponding to the optimal virtual endoscopic image; The bronchoscope is subjected to pose compensation based on the virtual camera pose parameters.

56. The computer program product according to claim 55, characterized in that, When the processor executes the computer program, it also performs the following steps: During the movement, the bronchoscope is positionally compensated, and the current phase of the subject's breathing is detected; When the current phase is the target phase, the actual endoscopic image of the bronchoscope after pose compensation is acquired.

57. The computer program product according to claim 56, characterized in that, When the processor executes the computer program, it also performs the following steps: The current spatial coordinates of the bronchoscope are obtained based on the positioning module of the bronchoscope. Obtain the automatic registration matrix; Based on the automatic registration matrix, the current spatial coordinates of the bronchoscope are converted into the current virtual coordinates in the bronchial model.

58. The computer program product according to claim 57, characterized in that, The bronchial model also includes multiple bifurcation points; When the processor executes the computer program, it also performs the following steps: A predetermined number of bifurcation points are selected from the bronchial model as marker points; Acquire real endoscopic images of each of the aforementioned marker points in advance; Based on the spatial coordinates and virtual coordinates of the marked points, the rotation and translation matrix is ​​obtained; Calculate the similarity between the virtual and real endoscopic images corresponding to all the marked points for each rotation and translation matrix, and select the rotation and translation matrix whose similarity satisfies the preset conditions as the automatic registration matrix.

59. The computer program product according to claim 55 or 56, characterized in that, When the processor executes the computer program, it also performs the following steps: Obtain the current virtual coordinates of the bronchoscope; Obtain the optimal virtual image at the current virtual coordinates; Calculate the second similarity between the optimal virtual image and the virtual endoscopic image at the current registration point; If the second similarity is greater than or equal to a preset threshold, the region within the first preset range of the current virtual coordinates is recorded as the region to be registered, and the movement of the bronchoscope is controlled within the region to be registered according to the first movement strategy. If the second similarity is less than a preset threshold, the region within the second preset range of the current virtual coordinates is recorded as a non-registered region, and the movement of the bronchoscope is controlled within the non-registered region according to the second movement strategy.

60. The computer program product according to claim 59, characterized in that, When the processor executes the computer program, it also performs the following steps: The bronchoscope is moved according to the first direction vector by the first step length, where the first direction vector is the direction vector of the bronchial segment where the bronchoscope is currently located. Once the bronchoscope reaches the current registration point, the rotation angle is calculated based on the current registration point and the next registration point. Based on the stated rotation angle, rotate the tip of the bronchoscope to the next registration point.

61. The computer program product according to claim 60, characterized in that, When the processor executes the computer program, it also performs the following steps: During the movement, the second similarity is calculated. If the next second similarity is less than the previous second similarity, it is considered that the bronchoscope has reached the current registration point.

62. The computer program product according to claim 60, characterized in that, When the processor executes the computer program, it also performs the following steps: The first direction vector is calculated based on the positioning module of the bronchoscope; Calculate the direction vector to be executed based on the current registration point and the next registration point; The rotation angle is calculated based on the first direction vector and the direction vector to be executed; Based on the rotation angle, rotate the tip of the bronchoscope catheter to the next registration point.

63. The computer program product according to claim 62, characterized in that, When the processor executes the computer program, it also performs the following steps: The bronchoscope is moved according to the first direction vector and the second step size.

Images