Compensation method, device, system and computer device for respiratory motion data

Abstract

The application relates to a respiratory motion data compensation method, device, system, computer equipment, storage medium and computer program product. The method comprises the following steps: acquiring multiple bronchus images and multiple electromagnetic sensor data of a measured object; rendering a virtual endoscopy image corresponding to a breathing phase based on the electromagnetic sensor data and the bronchus image corresponding to the same breathing phase; constructing a bronchus three-dimensional model according to the bronchus scanning data, extracting a bronchus centerline tree based on the bronchus three-dimensional model; calculating a position change trend of the virtual endoscopy image in a breathing cycle, determining a bronchus centerline search range; searching at least one virtual endoscopy image in the bronchus centerline search range; matching an actual endoscopy image with the at least one virtual endoscopy image in the bronchus centerline search range, obtaining and outputting a motion-compensated virtual endoscopy image. The method can improve the bronchus model registration efficiency.

Description

Technical Field

[0001] This application relates to the field of intelligent medical technology, and in particular to a method, apparatus, system, computer equipment, storage medium, and computer program product for compensating respiratory motion data. Background Technology

[0002] With the development of computer technology and medical imaging technology, surgical navigation systems have been increasingly widely used. Surgical navigation systems generate a 3D model based on the patient's preoperative static medical images and register this 3D model with the positional information collected intraoperatively by sensors within the patient's body to establish a mapping relationship between the intraoperative patient coordinate system and the preoperative 3D model coordinate system. During the surgery, the sensors then acquire the real-time position, reflecting this position back into the 3D model, thus achieving intraoperative navigation. However, for organs affected by respiration, such as the bronchi, the data acquired by sensors intraoperatively is dynamic data influenced by respiration. Therefore, minimizing the impact of respiratory motion on navigation accuracy is crucial in these types of navigation surgeries.

[0003] In traditional techniques, preoperative CT (Computerized Tomography) is typically used to obtain the position and orientation of the patient's bronchi, and this position and orientation are then registered to a bronchial model to complete respiratory motion compensation.

[0004] However, the traditional method requires multiple shots to obtain the bronchial position of the patient at multiple respiratory phases. Multiple shots expose the patient to more radiation and are time-consuming, resulting in low efficiency of bronchial model registration. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, device, system, computer equipment, computer-readable storage medium, and computer program product for compensating respiratory motion data that can improve the registration efficiency of bronchial models, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a method for compensating respiratory motion data. The method includes:

[0007] Acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested object;

[0008] Based on the electromagnetic sensing data and bronchial images corresponding to the same respiratory phase, a virtual endoscopic image of the corresponding respiratory phase is rendered.

[0009] Based on the bronchial scan data taken before the operation, a three-dimensional model of the bronchus of the subject was constructed, and the bronchial centerline tree was extracted based on the three-dimensional bronchial model.

[0010] Calculate the positional change trend of the virtual endoscopic image within a respiratory cycle, determine the search range of the bronchial centerline, and search for at least one virtual endoscopic image within the search range of the bronchial centerline based on the search range of the bronchial centerline.

[0011] Match the actual endoscopic image with at least one virtual endoscopic image within the search range of the bronchial centerline. Use the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image and output the motion-compensated virtual endoscopic image.

[0012] In one embodiment, the division of respiratory phases includes:

[0013] Respiratory phases are defined based on the motion displacement of the breathing patch;

[0014] Alternatively, respiratory phases can be determined based on the time and displacement recorded by the respiratory patch;

[0015] Alternatively, respiratory phases can be determined based on sensor data acquired by the sensors.

[0016] In one embodiment, based on electromagnetic sensing data and bronchial images corresponding to the same respiratory phase, a virtual endoscopic image corresponding to the respiratory phase is rendered, including:

[0017] Acquire electromagnetic sensing data and bronchial images corresponding to the same respiratory phase, and perform position matching on the electromagnetic sensing data and bronchial images corresponding to the same respiratory phase.

[0018] When the location is successfully matched, the electromagnetic sensor data and bronchial image that were successfully matched are rendered to obtain a virtual endoscopic image of the corresponding respiratory phase.

[0019] In one embodiment, electromagnetic sensing data and bronchial images corresponding to the same respiratory phase are acquired, and the electromagnetic sensing data and bronchial images corresponding to the same respiratory phase are matched for location, including:

[0020] Obtain the first coordinate system corresponding to the electromagnetic sensing data and the first coordinate of the electromagnetic induction data in the first coordinate system;

[0021] Obtain the second coordinate system corresponding to the bronchial image and the second coordinate of the bronchial image in the second coordinate system;

[0022] Based on the mapping relationship between the first coordinate and the second coordinate, the first coordinate system and the second coordinate system are registered.

[0023] In one embodiment, a three-dimensional model of the bronchus of the subject is constructed based on preoperative bronchial scan data, including:

[0024] The bronchial scan data of the subject was acquired. The bronchial scan data included bronchial images of the subject at multiple time points. The span of multiple time points covered at least one respiratory cycle of the subject when the bronchial scan data was acquired.

[0025] Based on bronchial scan data, a three-dimensional model of the bronchus is constructed in the second coordinate system.

[0026] In one embodiment, the extraction of the bronchial centerline tree based on the bronchial 3D model includes:

[0027] The bronchial centerline is obtained by segmenting the bronchial 3D model.

[0028] Based on the bronchial centerline and bronchial tree structure, the main airway of the bronchus corresponding to the bronchial centerline is obtained.

[0029] An ordered tree is constructed with the main airway of the bronchus as the root node, which serves as the bronchial centerline tree of the bronchus.

[0030] In one embodiment, the actual endoscopic image is matched with multiple virtual endoscopic images within the bronchial centerline search range, and the virtual endoscopic image that successfully matches the actual endoscopic image is used as the motion-compensated virtual endoscopic image, including:

[0031] The actual endoscopic image was matched with multiple virtual endoscopic images within the search range of the bronchial centerline to obtain the similarity between the actual endoscopic image and the multiple virtual endoscopic images within the search range of the bronchial centerline.

[0032] Based on multiple similarity scores, the virtual endoscopic image with the highest similarity score is used as the motion-compensated virtual endoscopic image.

[0033] In one embodiment, the actual endoscopic image is matched with multiple virtual endoscopic images within the bronchial centerline search range, and the virtual endoscopic image that successfully matches the actual endoscopic image is used as the motion-compensated virtual endoscopic image, including:

[0034] Extract image features from actual endoscopic images and use them as actual features;

[0035] The image features of each intermediate image are extracted and used as the corresponding virtual features; where the intermediate image is a virtual endoscopic image within the search range of the bronchial centerline;

[0036] The actual features are matched with multiple virtual features to obtain feature similarity scores. The intermediate image corresponding to the feature with the highest similarity score is used as the motion-compensated virtual endoscopic image.

[0037] Secondly, this application also provides a device for compensating respiratory motion data. The device includes:

[0038] The data acquisition module is used to acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested object;

[0039] The rendering module is used to render a virtual endoscopic image of the corresponding respiratory phase based on electromagnetic sensor data and bronchial images of the same respiratory phase.

[0040] The centerline tree construction module is used to construct a three-dimensional model of the bronchus of the tested object based on the bronchial scan data taken before the operation, and to extract the bronchial centerline tree based on the three-dimensional model of the bronchus.

[0041] The virtual endoscopic image acquisition module is used to calculate the positional change trend of the virtual endoscopic image within a respiratory cycle, determine the search range of the bronchial centerline, and search for at least one virtual endoscopic image within the search range of the bronchial centerline based on the search range of the bronchial centerline.

[0042] The matching module is used to match the actual endoscopic image with at least one virtual endoscopic image within the search range of the bronchial centerline, and output the motion-compensated virtual endoscopic image as the virtual endoscopic image that successfully matches the actual endoscopic image.

[0043] Thirdly, this application also provides a system for compensating respiratory motion data. The system includes:

[0044] Electromagnetic positioning equipment is used to acquire the positional movement and temporal changes of a respirator patch as the subject breathes.

[0045] The breathing phase division end is used to divide the breathing phase based on the change in the position and movement of the breathing patch over time.

[0046] Data acquisition equipment is used to acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested object;

[0047] The computing device is used to render virtual endoscopic images corresponding to the same respiratory phase based on electromagnetic sensor data and bronchial images; construct a three-dimensional model of the bronchus of the subject based on preoperative bronchial scan data, and extract the bronchial centerline tree based on the bronchial three-dimensional model; calculate the positional change trend of the virtual endoscopic images within a respiratory cycle, determine the bronchial centerline search range, and search for at least one virtual endoscopic image within the bronchial centerline search range based on the bronchial centerline search range; match the actual endoscopic image with at least one virtual endoscopic image within the bronchial centerline search range, and use the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image, and output the motion-compensated virtual endoscopic image.

[0048] The display device is used to display actual bronchial images in natural cavities acquired by a bronchoscope, display virtual endoscopic images corresponding to the actual bronchial images, and display navigation information for bronchial lesion areas.

[0049] Fourthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described in any of the above embodiments.

[0050] Fifthly, this application also provides a computer device readable storage medium. The computer device readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the above embodiments.

[0051] Sixthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the methods described in any of the above embodiments.

[0052] The aforementioned respiratory motion data compensation method, device, system, computer equipment, storage medium, and computer program product first acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the subject. Then, based on the electromagnetic sensor data and bronchial images corresponding to the same respiratory phase, a virtual endoscopic image of the corresponding respiratory phase is rendered. Based on preoperative bronchial scan data, a three-dimensional bronchial model of the subject is constructed, and a bronchial centerline tree is extracted based on the bronchial three-dimensional model. Further, the positional change trend of the virtual endoscopic images within a respiratory cycle is calculated to determine the bronchial centerline search range. Based on the bronchial centerline search range, multiple virtual endoscopic images within the bronchial centerline search range are obtained by searching the bronchial centerline tree. Finally, the actual endoscopic image is matched with the multiple virtual endoscopic images within the bronchial centerline search range. The virtual endoscopic image that successfully matches the actual endoscopic image is used as the motion-compensated virtual endoscopic image, and the motion-compensated virtual endoscopic image is output. By outputting motion-compensated virtual endoscopic images from bronchial images and electromagnetic sensing data in a single respiratory phase, the problem of inaccurate registration caused by respiratory motion can be solved, thereby improving the registration efficiency of bronchial models. Attached Figure Description

[0053] Figure 1 This is a diagram illustrating the application environment of a respiratory motion data compensation method in one embodiment.

[0054] Figure 2 This is a flowchart illustrating a method for compensating respiratory motion data in one embodiment;

[0055] Figure 3 This is a schematic diagram of the reconstruction of the bronchial centerline tree structure in one embodiment;

[0056] Figure 4 This is a schematic diagram illustrating the change in the range of motion of a virtual endoscopic image in one embodiment.

[0057] Figure 5 This is a schematic diagram of the search range of the bronchial centerline in one embodiment;

[0058] Figure 6 This is a schematic diagram illustrating the selection of the nearest point on the centerline in one embodiment;

[0059] Figure 7 This is a schematic diagram of the respiratory motion compensation results in one embodiment;

[0060] Figure 8 This is a schematic diagram of the respiratory patch installation in one embodiment;

[0061] Figure 9 This is a schematic diagram of respiratory phase division in one embodiment;

[0062] Figure 10 This is a schematic diagram illustrating the adjustment of the virtual endoscope in one embodiment;

[0063] Figure 11 This is a schematic diagram illustrating the matching of virtual endoscopic images and actual endoscopic images in one embodiment;

[0064] Figure 12 This is a schematic diagram of the endoscopic image matching results in one embodiment;

[0065] Figure 13 This is a schematic diagram of rigid registration between a virtual endoscopic image and an actual endoscopic image in one embodiment;

[0066] Figure 14 This is a schematic diagram of the rigid registration result between a virtual endoscopic image and an actual endoscopic image in one embodiment;

[0067] Figure 15 This is a structural block diagram of a respiratory motion data compensation device in one embodiment;

[0068] Figure 16 This is a hardware block diagram of a respiratory motion data compensation system in one embodiment;

[0069] Figure 17 This is a schematic diagram of a bronchial navigation scenario in one embodiment;

[0070] Figure 18 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0072] The respiratory motion data compensation method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 communicates and interacts with terminal 102 to enter the respiratory motion data compensation environment. First, server 104 can acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested subject from terminal 102. Then, server 104 can render a virtual endoscopic image corresponding to the respiratory phase based on the electromagnetic sensor data and bronchial images corresponding to the same respiratory phase; based on the bronchial scan data taken before surgery, a three-dimensional model of the bronchus of the tested subject is constructed, and a bronchial centerline tree is extracted based on the three-dimensional bronchial model. Furthermore, server 104 can calculate the positional change trend of the virtual endoscopic image within a respiratory cycle, determine the bronchial centerline search range, and search for at least one virtual endoscopic image within the bronchial centerline search range. Finally, server 104 can match the actual endoscopic image with at least one virtual endoscopic image within the bronchial centerline search range, and output the motion-compensated virtual endoscopic image as the virtual endoscopic image that successfully matches the actual endoscopic image. Terminal 102 can be various medical diagnostic devices, such as computed tomography (CT) imaging equipment, and server 104 can be a standalone server or a server cluster consisting of multiple servers. Server 104 can also be a general-purpose computer device.

[0073] The respiratory motion data compensation method provided in this application embodiment can be applied to a single terminal or server, or it can be applied to a system including a terminal and a server, and implemented through the interaction between the cluster and the server.

[0074] In one embodiment, such as Figure 2 As shown, a method for compensating respiratory motion data is provided. Taking the application of this method to a single-side server implementation as an example, the method includes the following steps 202 to 210.

[0075] Step 202: Acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested object.

[0076] In this embodiment, each frame of bronchial image and the electromagnetic sensing data are respectively associated with the respiratory phase.

[0077] In this embodiment, the breathing phase is defined based on the periodic changes in the position of the breathing patch over time.

[0078] In this embodiment, the respiratory phase may include, but is not limited to: end of expiration, inspiration, end of inspiration, and expiration.

[0079] In this embodiment, electromagnetic sensing data can be acquired using devices such as a respiratory patch and a bronchoscope. The bronchoscope may include a catheter, an endoscope, and an electromagnetic patch.

[0080] In this embodiment, the breathing patch may contain four electromagnetic patches for acquiring coordinate information of different respiratory phases of the patient's respiratory movements. The catheter may contain one monocular endoscope and four electromagnetic patches, wherein the monocular endoscope is used to acquire multi-frame bronchial images of the subject, and the electromagnetic patches are used to acquire coordinate information of the catheter relative to the magnetic field generator, i.e., electromagnetic sensing data.

[0081] Step 204: Based on the electromagnetic sensing data and bronchial images corresponding to the same respiratory phase, render a virtual endoscopic image of the corresponding respiratory phase.

[0082] In this embodiment, the server can obtain the coordinate information and the corresponding time information obtained through the electromagnetic patch in real time. According to the spatial and temporal changes, the electromagnetic sensing data is divided into different breathing phases within a breathing cycle to obtain the electromagnetic sensing data corresponding to each breathing phase in multiple breathing phases.

[0083] Step 206: Based on the bronchial scan data taken before the operation, construct a three-dimensional model of the bronchus of the subject, and extract the bronchial centerline tree based on the three-dimensional bronchial model.

[0084] In this embodiment, bronchial scan data can be acquired using a computed tomography (CT) imaging device.

[0085] In this embodiment, as Figure 3 As shown, the server can construct a three-dimensional model of the bronchus of the tested object based on bronchial scan data, extract the centerline of the three-dimensional bronchial model, and obtain the bronchial centerline tree based on the centerline of the three-dimensional bronchial model.

[0086] Step 208: Calculate the positional change trend of the virtual endoscopic image within one respiratory cycle, determine the search range of the bronchial centerline, and search for at least one virtual endoscopic image within the search range of the bronchial centerline based on the search range of the bronchial centerline.

[0087] In this embodiment, the server can first acquire the spatial coordinates (coordinate information) of the electromagnetic patch in real time, calculate the point cloud closest to these spatial coordinates on the center line, and use it as the target point cloud. Further, based on the respiratory phase segmentation results, the server can obtain the spatial position of the target point cloud corresponding to each respiratory phase, generate a virtual endoscopic image of the target point cloud corresponding to each respiratory phase, and calculate the depth information of the target point cloud in the virtual endoscopic image to determine the motion range of the virtual endoscopic image within a respiratory cycle.

[0088] In this embodiment, as Figure 4 As shown, for each of multiple respiratory phases, the server can acquire the centerline and electromagnetic sensing information (coordinate information corresponding to the current respiratory phase time) of the same respiratory phase, and obtain the coordinates of the point cloud closest to the target position for that respiratory phase. Similarly, the server can obtain the coordinates of the point cloud closest to the target position for each respiratory phase. Furthermore, based on the coordinates of the point cloud closest to the target position for each respiratory phase, the server can generate virtual endoscopic images corresponding to different respiratory phases. Furthermore, the server can also use the VTK (visualization toolkit) algorithm to calculate the depth information of the target point cloud in the virtual endoscopic image. Based on the coordinates of the point cloud closest to the target position for each respiratory phase, and the depth information of the target point cloud in the virtual endoscopic image, the motion range of the virtual endoscopic image within one respiratory cycle can be obtained.

[0089] In this embodiment, as Figure 5 As shown, the range of motion of the virtual endoscope image within a respiratory cycle is linearly related to the displacement of the respiratory patch. Therefore, the server can construct a linear model based on the amplitude and time proportion of each respiratory cycle and the changes in the range of motion of the virtual endoscope image. For example, during the expiratory phase, the position of the virtual endoscope image tends to extend forward along the centerline to match this phase; during the inspiratory phase, the position of the virtual endoscope image tends to extend backward along the centerline to match this phase. Based on the different phases, a strategy for determining the search range of the bronchial centerline can improve the speed of subsequent image matching and ensure the real-time performance of surgical navigation.

[0090] Step 210: Match the actual endoscopic image with at least one virtual endoscopic image within the search range of the bronchial centerline, and use the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image, and output the motion-compensated virtual endoscopic image.

[0091] In this embodiment, as Figure 6 , Figure 7 As shown, due to the influence of respiratory movements, the virtual endoscopic image may drift out of the bronchus compared to the actual endoscopic image. Therefore, the server can use the electromagnetic sensing information (coordinate information corresponding to the current time) as the target position, and then calculate the coordinates of the point cloud closest to the target position on the center line. This coordinate is used for image matching region selection, narrowing the range of image matching and reducing the probability of the image drifting out of the bronchus.

[0092] In this embodiment, the server can render virtual endoscopic time-frequency images based on motion-compensated virtual endoscopic images to provide navigation information for doctors during surgery.

[0093] The above-mentioned method for compensating respiratory motion data first acquires multiple frames of bronchial images and multiple electromagnetic sensor data of the subject. Then, based on the electromagnetic sensor data and bronchial images corresponding to the same respiratory phase, a virtual endoscopic image of the corresponding respiratory phase is rendered. A three-dimensional bronchial model of the subject is constructed based on preoperative bronchial scan data, and a bronchial centerline tree is extracted from the bronchial three-dimensional model. Further, the positional change trend of the virtual endoscopic image within a respiratory cycle is calculated to determine the bronchial centerline search range. Based on the bronchial centerline search range, at least one virtual endoscopic image within the search range is obtained. Finally, the actual endoscopic image is matched with at least one virtual endoscopic image within the bronchial centerline search range. The virtual endoscopic image that successfully matches the actual endoscopic image is used as the motion-compensated virtual endoscopic image, and the motion-compensated virtual endoscopic image is output. By outputting a motion-compensated virtual endoscopic image from bronchial images and electromagnetic sensor data of a single respiratory phase, the problem of inaccurate registration caused by respiratory motion can be solved, improving the registration efficiency of the bronchial model.

[0094] In some embodiments, the division of respiratory phases may include: dividing the respiratory phase based on the motion displacement of the respiratory patch; or dividing the respiratory phase based on the time and displacement recorded by the respiratory patch; or dividing the respiratory phase based on sensor data acquired by the sensor.

[0095] Specifically, such as Figure 8 As shown, two electromagnetic patches (electromagnetic sensors) are installed on each of the horizontal sides of the head and tail of the breathing patch, namely S1, S3 and S2, S4. The spatial coordinates of S1, S3 and S2, S4 are obtained. Based on these spatial coordinates, a structure can be constructed as follows: Figure 8 The XYZ spatial coordinate system is shown. Further, the amplitude of the Y-axis change in the XYZ spatial coordinate system corresponding to one respiratory cycle and the respiratory motion time recorded by the sensor are obtained. Based on the amplitude change of the Y-axis in the XYZ spatial coordinate system corresponding to one respiratory cycle and the respiratory motion time recorded by the sensor, the respiratory phase is divided.

[0096] In this embodiment, as Figure 9As shown, based on the amplitude change of the Y-axis in the XYZ spatial coordinate system corresponding to a respiratory cycle and the respiratory motion time recorded by the sensor, the respiratory phase is divided. This can include: converting the sensor data from sensors S1, S2, S3, and S4 into amplitude changes of the Y-axis in the XYZ spatial coordinate system, and obtaining smoother amplitude change data through Gaussian filtering and standardization. Based on the respiratory motion time recorded by the sensor, periodic respiratory motion is determined. According to the amplitude (range)-time (time) change curve, the inflection point from inhalation to the end of inhalation, the inflection point from the end of inhalation to exhalation, and the inflection point from exhalation to the end of exhalation are divided, thus dividing the respiratory motion cycle into four different respiratory phases: inhalation, end of inhalation, exhalation, and end of exhalation.

[0097] In some embodiments, rendering a virtual endoscopic image of a corresponding respiratory phase based on electromagnetic sensing data and bronchial images corresponding to the same respiratory phase may include: acquiring electromagnetic sensing data and bronchial images corresponding to the same respiratory phase; performing position matching on the electromagnetic sensing data and bronchial images corresponding to the same respiratory phase; and rendering the electromagnetic sensing data and bronchial images that have successfully matched the positions when the position matching is successful, to obtain a virtual endoscopic image of the corresponding respiratory phase.

[0098] In this embodiment, the server can select electromagnetic sensor data of a single respiratory phase that is the same as the respiratory patch time for position matching based on the respiratory phase division results (four different respiratory phases: inhalation, end of inhalation, exhalation, and end of exhalation).

[0099] In some embodiments, acquiring electromagnetic sensing data and bronchial images corresponding to the same respiratory phase, and performing position matching on the electromagnetic sensing data and bronchial images corresponding to the same respiratory phase, may include: acquiring a first coordinate system corresponding to the electromagnetic sensing data and a first coordinate of the electromagnetic sensing data in the first coordinate system; acquiring a second coordinate system corresponding to the bronchial image and a second coordinate of the bronchial image in the second coordinate system; and registering the first coordinate system and the second coordinate system based on the mapping relationship between the first coordinate and the second coordinate.

[0100] In this embodiment, the first coordinate system corresponding to the electromagnetic sensing data is the aforementioned XYZ spatial coordinate system, and the first coordinate of the electromagnetic sensing data in the first coordinate system is the spatial coordinate of the sensor in the breathing patch.

[0101] In this embodiment, the second coordinate system corresponding to the bronchial image is the aforementioned CT coordinate system, and the second coordinate of the bronchial image in the second coordinate system is the spatial coordinate of each pixel point in the bronchial image in the CT coordinate system.

[0102] In this embodiment, since the XYZ spatial coordinate system corresponding to the electromagnetic sensing data is different from the CT coordinate system corresponding to the bronchial image, global registration is required before position matching to unify the coordinates of each point in the XYZ spatial coordinate system and the coordinates of each point in the CT coordinate system into the same coordinate system. This is to reduce the distance between the electromagnetic sampling point in the electromagnetic sensing data and the center line of the bronchial three-dimensional model, and reduce the problem of virtual endoscopic image drift caused by respiratory motion.

[0103] In some embodiments, constructing a three-dimensional model of the bronchus of the subject based on preoperative bronchial scan data may include: acquiring bronchial scan data of the subject, the bronchial scan data including bronchial images of the subject at multiple times; the span of the multiple times covering at least one respiratory cycle of the subject when the bronchial scan data was acquired; and constructing a three-dimensional model of the bronchus in a second coordinate system based on the bronchial scan data.

[0104] In this embodiment, the bronchial scan data may include, but is not limited to: the bronchial image of the subject being tested, the spatial coordinates of each pixel in the bronchial image in the CT coordinate system, etc.

[0105] In this embodiment, a respiratory cycle includes at least four different respiratory phases: inhalation, end of inhalation, exhalation, and end of exhalation.

[0106] In some embodiments, extracting the bronchial centerline tree based on the bronchial three-dimensional model may include: segmenting the bronchial three-dimensional model to obtain the bronchial centerline corresponding to the bronchus; obtaining the main airway of the bronchus corresponding to the bronchial centerline based on the bronchial centerline and the bronchial tree structure; and constructing an ordered tree with the main airway of the bronchus as the root node, which serves as the bronchial centerline tree of the bronchus.

[0107] In this embodiment, the server can trim the extracted centerlines according to the bifurcation points, store the centerline point cloud of each airway as a node of the bronchial tree, and construct the topology of the bronchial centerline tree according to the connection relationship between the airways.

[0108] In this embodiment, as Figure 3 As shown, the root node stores the point cloud of the main airway centerline, and the child nodes of the root node store the point clouds of the left and right secondary branches, and so on, to obtain multiple nodes and their spatial coordinates corresponding to the 3D model of the bronchus. Based on the multiple nodes and their spatial coordinates of the bronchial centerline tree, the server can build a 3D ordered tree to obtain the bronchial centerline tree of the bronchus.

[0109] In some embodiments, matching an actual endoscopic image with multiple virtual endoscopic images within the bronchial centerline search range, and using the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image, may include: performing similarity matching between the actual endoscopic image and multiple virtual endoscopic images within the bronchial centerline search range to obtain the similarity between the actual endoscopic image and the multiple virtual endoscopic images within the bronchial centerline search range; and based on the multiple similarities, using the virtual endoscopic image with the highest similarity value as the motion-compensated virtual endoscopic image.

[0110] In this embodiment, the server generates a virtual endoscopic image within the bronchial centerline search range and adjusts the orientation of the virtual endoscope within a certain angle to find the virtual endoscopic image that best matches the actual endoscopic image. Then, rigid registration is performed between the virtual and actual endoscopic images to make them even closer. This virtual endoscopic image is then output and displayed for real-time navigation during bronchoscopy. First, determining the bronchial centerline search range, and then generating a virtual endoscopic image within that range for image matching, reduces computational load. By adjusting the angle and using rigid registration, the accuracy of image matching can be improved.

[0111] In this embodiment, as Figure 10 As shown, the server can adjust the orientation of the virtual endoscope at a certain angle within the bronchial centerline search range, and generate a series of virtual endoscopic images within this bronchial centerline search range for matching with actual endoscopic images, which can improve the accuracy of image matching.

[0112] In some embodiments, matching an actual endoscopic image with multiple virtual endoscopic images within the bronchial centerline search range, and using the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image, may include: extracting image features from the actual endoscopic image as actual features; extracting image features from each intermediate image as corresponding virtual features; wherein the intermediate image is a virtual endoscopic image within the bronchial centerline search range; performing similarity matching between the actual features and multiple virtual features respectively to obtain feature similarity, and using the intermediate image corresponding to the feature similarity with the highest value as the motion-compensated virtual endoscopic image.

[0113] In this embodiment, as Figure 11 As shown, image features may include, but are not limited to, bronchial structure information, image grayscale information, and image brightness information.

[0114] In this embodiment, as Figure 12As shown, the server can use the SSIM (structural similarity index measure) image similarity index as a standard to perform similarity matching between actual features and multiple virtual features to obtain feature similarity. The intermediate image corresponding to the feature similarity with the highest value (i.e., the single virtual endoscopic image with the highest SSIM index) is used as the motion-compensated virtual endoscopic image.

[0115] In this embodiment, as Figure 13 , Figure 14 As shown, when the feature similarity between the virtual endoscopic image and the actual endoscopic image is greater than or equal to a preset similarity threshold, the server can also perform rigid registration operations such as rotation, translation, and scaling on multiple virtual endoscopic images (original virtual endoscopic images) based on the bronchial morphology features in the actual endoscopic image to obtain a registered virtual endoscopic image. This can further improve the similarity between multiple virtual endoscopic images and the actual endoscopic image. Then, the single virtual endoscopic image with the highest feature similarity value is used as the motion-compensated virtual endoscopic image. Furthermore, based on the server's rigid registration operations such as rotation, translation, and scaling on the virtual endoscopic images, a rigid registration matrix corresponding to the motion-compensated virtual endoscopic image can be generated for subsequent surgical navigation.

[0116] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0117] Based on the same inventive concept, this application also provides a respiratory motion data compensation device for implementing the respiratory motion data compensation method described above. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the respiratory motion data compensation device provided below can be found in the limitations of the respiratory motion data compensation method described above, and will not be repeated here.

[0118] In one embodiment, such as Figure 15As shown, a device for compensating respiratory motion data is provided, comprising: a data acquisition module 1502, a rendering module 1504, a centerline tree construction module 1506, a virtual endoscopic image acquisition module 1508, and a matching module 1510, wherein:

[0119] The data acquisition module 1502 is used to acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested object.

[0120] The rendering module 1504 is used to render a virtual endoscopic image corresponding to the respiratory phase based on electromagnetic sensing data and bronchial images corresponding to the same respiratory phase.

[0121] The centerline tree construction module 1506 is used to construct a three-dimensional model of the bronchus of the tested object based on the bronchial scan data taken before the operation, and extract the bronchial centerline tree based on the three-dimensional model of the bronchus.

[0122] The virtual endoscopic image acquisition module 1508 is used to calculate the positional change trend of the virtual endoscopic image within a respiratory cycle, determine the search range of the bronchial centerline, and search for at least one virtual endoscopic image within the search range of the bronchial centerline based on the search range of the bronchial centerline.

[0123] The matching module 1510 is used to match the actual endoscopic image with at least one virtual endoscopic image within the search range of the bronchial centerline, and to output the motion-compensated virtual endoscopic image as the virtual endoscopic image that is successfully matched with the actual endoscopic image.

[0124] In one embodiment, the data acquisition module 1502 may include:

[0125] The breathing phase segmentation submodule is used to segment the breathing phase based on the motion displacement of the breathing patch; or, based on the time and displacement recorded by the breathing patch; or, based on the sensor data acquired by the sensor.

[0126] In one embodiment, the rendering module 1504 may include:

[0127] The location matching submodule is used to acquire electromagnetic sensor data and bronchial images corresponding to the same respiratory phase, and to perform location matching on the electromagnetic sensor data and bronchial images corresponding to the same respiratory phase.

[0128] The virtual endoscopic image generation submodule is used to render the electromagnetic sensor data and bronchial image that have been successfully matched when the position is matched, so as to obtain a virtual endoscopic image of the corresponding respiratory phase.

[0129] In one embodiment, the location matching submodule may include:

[0130] The first position acquisition unit is used to acquire the first coordinate system corresponding to the electromagnetic sensing data and the first coordinate of the electromagnetic induction data in the first coordinate system.

[0131] The second position acquisition unit is used to acquire the second coordinate system corresponding to the bronchial image and the second coordinates of the bronchial image in the second coordinate system.

[0132] The coordinate system matching unit is used to register the first coordinate system and the second coordinate system based on the mapping relationship between the first coordinate and the second coordinate.

[0133] In one embodiment, the centerline tree construction module 1506 may include:

[0134] The bronchial image acquisition submodule is used to acquire bronchial scan data of the subject. The bronchial scan data includes bronchial images of the subject at multiple time points. The span of multiple time points covers at least one respiratory cycle of the subject when the bronchial scan data is acquired.

[0135] The model building submodule is used to construct a three-dimensional model of the bronchus in a second coordinate system based on bronchial scan data.

[0136] In one embodiment, the centerline tree construction module 1506 may further include:

[0137] The bronchial centerline acquisition submodule is used to segment the bronchus in the 3D model of the bronchus and obtain the bronchial centerline corresponding to the bronchus.

[0138] The bronchial main airway acquisition unit is used to acquire the bronchial main airway corresponding to the bronchial centerline based on the bronchial centerline and bronchial tree structure.

[0139] The ordered tree construction unit is used to construct an ordered tree with the main airway of the bronchus as the root node, which serves as the bronchial centerline tree of the bronchus.

[0140] In one embodiment, the matching module 1510 may include:

[0141] The similarity acquisition submodule is used to perform similarity matching between the actual endoscope image and multiple virtual endoscope images within the bronchial centerline search range to obtain the similarity between the actual endoscope image and the multiple virtual endoscope images within the bronchial centerline search range.

[0142] The virtual endoscopic image determination submodule is used to select the virtual endoscopic image with the highest similarity value as the motion-compensated virtual endoscopic image based on multiple similarity scores.

[0143] In one embodiment, the matching module 1510 may further include:

[0144] The actual feature acquisition submodule is used to extract image features from actual endoscopic images as actual features.

[0145] The virtual feature extraction submodule is used to extract the image features of each intermediate image as the corresponding virtual features; where the intermediate image is a virtual endoscopic image within the search range of the bronchial centerline.

[0146] The feature matching submodule is used to perform similarity matching between actual features and multiple virtual features to obtain feature similarity. The intermediate image corresponding to the feature with the highest numerical similarity is used as the motion-compensated virtual endoscopic image.

[0147] The modules in the aforementioned respiratory motion data compensation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0148] Based on the same inventive concept, this application also provides a respiratory motion data compensation system for implementing the respiratory motion data compensation method described above. The solution provided by this system is similar to the implementation described in the above method; therefore, the specific limitations of one or more respiratory motion data compensation system embodiments provided below can be found in the limitations of the respiratory motion data compensation method described above, and will not be repeated here.

[0149] In one embodiment, such as Figure 16 As shown, a respiratory motion data compensation system is provided, comprising: an electromagnetic positioning device for acquiring the positional movement and temporal changes of a respiratory patch as the subject breathes; a respiratory phase segmentation device for segmenting respiratory phases based on the positional movement of the respiratory patch over time; a data acquisition device for acquiring multiple frames of bronchial images and multiple electromagnetic sensor data of the subject; and a computing device for rendering a virtual endoscopic image of the corresponding respiratory phase based on the electromagnetic sensor data and bronchial images corresponding to the same respiratory phase; and constructing a data acquisition system for the subject based on preoperative bronchial scan data. A 3D model of the bronchus is generated, and a bronchial centerline tree is extracted based on the 3D model. The positional change trend of the virtual endoscopic image within one respiratory cycle is calculated to determine the search range of the bronchial centerline. Based on the search range, at least one virtual endoscopic image within the search range is obtained. The actual endoscopic image is matched with at least one virtual endoscopic image within the search range. The virtual endoscopic image that successfully matches the actual endoscopic image is used as the motion-compensated virtual endoscopic image, and the motion-compensated virtual endoscopic image is output. Display device, such as... Figure 17 As shown, it is used to display the actual bronchial image in the natural cavity acquired by the bronchoscope, the virtual endoscopic image corresponding to the actual bronchial image, and the navigation information of the bronchial lesion area.

[0150] The electromagnetic positioning device may include, but is not limited to, a breathing patch and a magnetic field generator. The magnetic field generator is used to generate a magnetic field.

[0151] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 18 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as bronchial images of the tested object, electromagnetic sensor data, image features of actual endoscopic images, and image features of virtual endoscopic images. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for compensating respiratory motion data.

[0152] Those skilled in the art will understand that Figure 18 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0153] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0154] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0155] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0156] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0157] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0158] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A device for compensating respiratory motion data, characterized in that, The device includes: The data acquisition module is used to acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested object; The rendering module is used to render a virtual endoscopic image corresponding to the respiratory phase based on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase. The centerline tree construction module is used to construct a three-dimensional model of the bronchus of the tested object based on the bronchial scan data taken before the operation, and to extract the bronchial centerline tree based on the three-dimensional model of the bronchus. A virtual endoscopic image acquisition module is used to calculate the positional change trend of the virtual endoscopic image within a respiratory cycle, determine the bronchial centerline search range, and search for at least one virtual endoscopic image within the bronchial centerline search range based on the bronchial centerline search range. The method for determining the positional change trend includes: determining the point cloud closest to the electromagnetic sensing data on the bronchial centerline as the target point cloud; calculating the depth information of the target point cloud in the virtual endoscopic image; and determining the motion range of the virtual endoscopic image within a respiratory cycle as the positional change trend based on the coordinates of the point cloud closest to the electromagnetic sensing data corresponding to each respiratory phase and the depth information of the target point cloud in the virtual endoscopic image. The matching module is used to match the actual endoscopic image with at least one of the virtual endoscopic images within the search range of the bronchial centerline, and to output the motion-compensated virtual endoscopic image as the virtual endoscopic image that successfully matches the actual endoscopic image.

2. The apparatus according to claim 1, characterized in that, The data acquisition module includes: The breathing phase segmentation submodule is used to segment the breathing phase based on the motion displacement of the breathing patch; or, to segment the breathing phase based on the time and displacement recorded by the breathing patch; or, to segment the breathing phase based on sensor data acquired by the sensor.

3. The apparatus according to claim 1, characterized in that, The rendering module includes: The location matching submodule is used to acquire the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, and to perform location matching on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase. The virtual endoscopic image generation submodule is used to render the electromagnetic sensing data and the bronchial image that have been successfully matched at the location when the location is successfully matched, so as to obtain a virtual endoscopic image corresponding to the respiratory phase.

4. The apparatus according to claim 3, characterized in that, The location matching submodule includes: The first position acquisition unit is used to acquire the first coordinate system corresponding to the electromagnetic sensing data and the first coordinate of the electromagnetic sensing data in the first coordinate system. The second position acquisition unit is used to acquire the second coordinate system corresponding to the bronchial image and the second coordinates of the bronchial image in the second coordinate system; The coordinate system matching unit is used to register the first coordinate system and the second coordinate system based on the mapping relationship between the first coordinate and the second coordinate.

5. The apparatus according to claim 4, characterized in that, The centerline tree construction module includes: A bronchial image acquisition submodule is used to acquire bronchial scan data of the subject, the bronchial scan data including bronchial images of the subject at multiple times; the span of the multiple times covers at least one respiratory cycle of the subject when the bronchial scan data is acquired. The model building submodule is used to construct a three-dimensional model of the bronchus in the second coordinate system based on the bronchus scan data.

6. The apparatus according to claim 1, characterized in that, The centerline tree construction module includes: The bronchial centerline acquisition submodule is used to segment the bronchial three-dimensional model to obtain the bronchial centerline corresponding to the bronchial. The bronchial main airway acquisition unit is used to acquire the bronchial main airway corresponding to the bronchial centerline based on the bronchial centerline and the bronchial tree structure. An ordered tree construction unit is used to construct an ordered tree with the main airway of the bronchus as the root node, which serves as the bronchial centerline tree of the bronchus.

7. The apparatus according to claim 1, characterized in that, The matching module includes: The similarity acquisition submodule is used to perform similarity matching between the actual endoscope image and multiple virtual endoscope images within the search range of the bronchial centerline, respectively, to obtain the similarity between the actual endoscope image and the multiple virtual endoscope images within the search range of the bronchial centerline; The virtual endoscopic image determination submodule is used to select the virtual endoscopic image with the highest similarity value as the motion-compensated virtual endoscopic image based on multiple similarity scores.

8. The apparatus according to claim 1, characterized in that, The matching module includes: The actual feature acquisition submodule is used to extract the image features of the actual endoscope image as actual features; The virtual feature extraction submodule is used to extract the image features of each intermediate image as the corresponding virtual features; wherein, the intermediate image is the virtual endoscopic image within the search range of the bronchial centerline; The feature matching submodule is used to perform similarity matching between the actual features and multiple virtual features to obtain feature similarity, and to take the intermediate image corresponding to the feature similarity with the largest value as the motion-compensated virtual endoscopic image.

9. A system for compensating respiratory motion data, characterized in that, The system includes: Electromagnetic positioning equipment is used to acquire the positional movement and temporal changes of a respirator patch as the subject breathes. The breathing phase division end is used to divide the breathing phase according to the change of the position movement of the breathing patch over time; Data acquisition equipment is used to acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested object; A computing device is used to render a virtual endoscopic image corresponding to the respiratory phase based on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase; construct a three-dimensional model of the bronchus of the subject based on the preoperative bronchial scan data, and extract a bronchial centerline tree based on the bronchial three-dimensional model; calculate the positional change trend of the virtual endoscopic image within a respiratory cycle, determine the bronchial centerline search range, and search for at least one virtual endoscopic image within the bronchial centerline search range based on the bronchial centerline search range; and compare the actual endoscopic image with at least one image within the bronchial centerline search range. The virtual endoscopic image is matched, and the virtual endoscopic image that successfully matches the actual endoscopic image is used as the motion-compensated virtual endoscopic image. The motion-compensated virtual endoscopic image is then output. The method for determining the position change trend includes: determining the point cloud on the bronchial centerline that is closest to the electromagnetic sensing data as the target point cloud; calculating the depth information of the target point cloud in the virtual endoscopic image; and determining the motion range of the virtual endoscopic image within a respiratory cycle as the position change trend based on the coordinates of the point cloud closest to the electromagnetic sensing data corresponding to each respiratory phase and the depth information of the target point cloud in the virtual endoscopic image. The display device is used to display actual bronchial images in natural cavities acquired by a bronchoscope, display virtual endoscopic images corresponding to the actual bronchial images, and display navigation information for bronchial lesion areas.

10. The system according to claim 9, characterized in that, The computing device is also used to divide the respiratory phase based on the motion displacement of the breathing patch; or, to divide the respiratory phase based on the time and displacement recorded by the breathing patch; or, to divide the respiratory phase based on sensor data acquired by the sensor.

11. The system according to claim 9, characterized in that, The computing device is also used for: Acquire the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, and perform position matching on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase; When the location is successfully matched, the electromagnetic sensing data and the bronchial image of the successfully matched location are rendered to obtain a virtual endoscopic image corresponding to the respiratory phase.

12. The system according to claim 11, characterized in that, The computing device is also used for: Obtain the first coordinate system corresponding to the electromagnetic sensing data and the first coordinate of the electromagnetic sensing data in the first coordinate system; Obtain the second coordinate system corresponding to the bronchial image and the second coordinates of the bronchial image in the second coordinate system; Based on the mapping relationship between the first coordinate and the second coordinate, the first coordinate system and the second coordinate system are registered.

13. The system according to claim 12, characterized in that, The computing device is also used for: The bronchial scan data of the subject is acquired, and the bronchial scan data includes bronchial images of the subject at multiple time points; the span of the multiple time points covers at least one respiratory cycle of the subject when the bronchial scan data is acquired. Based on the bronchial scan data, a three-dimensional model of the bronchus is constructed in the second coordinate system.

14. The system according to claim 9, characterized in that, The computing device is also used for: The bronchus three-dimensional model is segmented to obtain the bronchus centerline corresponding to the bronchus; Based on the bronchial centerline and bronchial tree structure, the main airway of the bronchus corresponding to the bronchial centerline is obtained. An ordered tree is constructed with the main airway of the bronchus as the root node, which serves as the bronchial centerline tree of the bronchus.

15. The system according to claim 9, characterized in that, The computing device is also used for: The actual endoscopic image is matched with a plurality of virtual endoscopic images within the search range of the bronchial centerline to obtain the similarity between the actual endoscopic image and the plurality of virtual endoscopic images within the search range of the bronchial centerline. Based on multiple similarity scores, the virtual endoscopic image corresponding to the largest similarity score is taken as the motion-compensated virtual endoscopic image.

16. The system according to claim 9, characterized in that, The computing device is also used for: Extract the image features of the actual endoscope image as the actual features; Image features of each intermediate image are extracted as corresponding virtual features; wherein, the intermediate image is the virtual endoscopic image within the search range of the bronchial centerline; The actual features are matched with multiple virtual features to obtain feature similarity scores. The intermediate image corresponding to the feature similarity score with the highest value is used as the motion-compensated virtual endoscopic image.

17. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it performs the following steps: Acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested object; Based on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, a virtual endoscopic image corresponding to the respiratory phase is rendered. Based on the bronchial scan data taken before the operation, a three-dimensional model of the bronchus of the subject was constructed, and the bronchial centerline tree was extracted based on the three-dimensional bronchial model. The positional change trend of the virtual endoscopic image within one respiratory cycle is calculated, the bronchial centerline search range is determined, and at least one virtual endoscopic image within the bronchial centerline search range is obtained based on the bronchial centerline search range. The method for determining the positional change trend includes: identifying the point cloud closest to the electromagnetic sensing data on the bronchial centerline as the target point cloud; calculating the depth information of the target point cloud in the virtual endoscopic image; and determining the motion range of the virtual endoscopic image within one respiratory cycle as the positional change trend based on the coordinates of the point cloud closest to the electromagnetic sensing data corresponding to each respiratory phase and the depth information of the target point cloud in the virtual endoscopic image. The actual endoscopic image is matched with at least one of the virtual endoscopic images within the search range of the bronchial centerline. The virtual endoscopic image that successfully matches the actual endoscopic image is used as the motion-compensated virtual endoscopic image, and the motion-compensated virtual endoscopic image is output.

18. The computer device according to claim 17, characterized in that, The division of the respiratory phase implemented by the processor when executing the computer program includes: Respiratory phases are defined based on the motion displacement of the breathing patch; Alternatively, the respiratory phase can be divided based on the time and displacement recorded by the respiratory patch; Alternatively, respiratory phases can be determined based on sensor data acquired by the sensors.

19. The computer device according to claim 17, characterized in that, The processor, when executing the computer program, renders a virtual endoscopic image corresponding to the respiratory phase based on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, including: Acquire the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, and perform position matching on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase; When the location is successfully matched, the electromagnetic sensing data and the bronchial image of the successfully matched location are rendered to obtain a virtual endoscopic image corresponding to the respiratory phase.

20. The computer device according to claim 19, characterized in that, The process implemented by the processor when executing the computer program, which involves acquiring the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, and performing position matching on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, includes: Obtain the first coordinate system corresponding to the electromagnetic sensing data and the first coordinate of the electromagnetic sensing data in the first coordinate system; Obtain the second coordinate system corresponding to the bronchial image and the second coordinates of the bronchial image in the second coordinate system; Based on the mapping relationship between the first coordinate and the second coordinate, the first coordinate system and the second coordinate system are registered.

21. The computer device according to claim 20, characterized in that, The process implemented by the processor when executing the computer program, which involves constructing a three-dimensional bronchial model of the subject based on preoperative bronchial scan data, includes: The bronchial scan data of the subject is acquired, and the bronchial scan data includes bronchial images of the subject at multiple time points; the span of the multiple time points covers at least one respiratory cycle of the subject when the bronchial scan data is acquired. Based on the bronchial scan data, a three-dimensional model of the bronchus is constructed in the second coordinate system.

22. The computer device according to claim 17, characterized in that, The extraction of the bronchial centerline tree based on the bronchial three-dimensional model, implemented by the processor when executing the computer program, includes: The bronchus three-dimensional model is segmented to obtain the bronchus centerline corresponding to the bronchus; Based on the bronchial centerline and bronchial tree structure, the main airway of the bronchus corresponding to the bronchial centerline is obtained. An ordered tree is constructed with the main airway of the bronchus as the root node, which serves as the bronchial centerline tree of the bronchus.

23. The computer device according to claim 17, characterized in that, The process implemented by the processor when executing the computer program, which involves matching the actual endoscopic image with multiple virtual endoscopic images within the search range of the bronchial centerline, and using the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image, includes: The actual endoscopic image is matched with a plurality of virtual endoscopic images within the search range of the bronchial centerline to obtain the similarity between the actual endoscopic image and the plurality of virtual endoscopic images within the search range of the bronchial centerline. Based on multiple similarity scores, the virtual endoscopic image corresponding to the largest similarity score is taken as the motion-compensated virtual endoscopic image.

24. The computer device according to claim 17, characterized in that, The process implemented by the processor when executing the computer program, which involves matching the actual endoscopic image with multiple virtual endoscopic images within the search range of the bronchial centerline, and using the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image, includes: Extract the image features of the actual endoscope image as the actual features; Image features of each intermediate image are extracted as corresponding virtual features; wherein, the intermediate image is the virtual endoscopic image within the search range of the bronchial centerline; The actual features are matched with multiple virtual features to obtain feature similarity scores. The intermediate image corresponding to the feature similarity score with the highest value is used as the motion-compensated virtual endoscopic image.

25. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it performs the following steps: Acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested object; Based on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, a virtual endoscopic image corresponding to the respiratory phase is rendered. Based on the bronchial scan data taken before the operation, a three-dimensional model of the bronchus of the subject was constructed, and the bronchial centerline tree was extracted based on the three-dimensional bronchial model. The positional change trend of the virtual endoscopic image within one respiratory cycle is calculated, the bronchial centerline search range is determined, and at least one virtual endoscopic image within the bronchial centerline search range is obtained based on the bronchial centerline search range. The method for determining the positional change trend includes: identifying the point cloud closest to the electromagnetic sensing data on the bronchial centerline as the target point cloud; calculating the depth information of the target point cloud in the virtual endoscopic image; and determining the motion range of the virtual endoscopic image within one respiratory cycle as the positional change trend based on the coordinates of the point cloud closest to the electromagnetic sensing data corresponding to each respiratory phase and the depth information of the target point cloud in the virtual endoscopic image. The actual endoscopic image is matched with at least one of the virtual endoscopic images within the search range of the bronchial centerline. The virtual endoscopic image that successfully matches the actual endoscopic image is used as the motion-compensated virtual endoscopic image, and the motion-compensated virtual endoscopic image is output.

26. The computer-readable storage medium according to claim 25, characterized in that, The division of the respiratory phases implemented by the computer program when executed by the processor includes: Respiratory phases are defined based on the motion displacement of the breathing patch; Alternatively, the respiratory phase can be divided based on the time and displacement recorded by the respiratory patch; Alternatively, respiratory phases can be determined based on sensor data acquired by the sensors.

27. The computer-readable storage medium according to claim 25, characterized in that, When the computer program is executed by the processor, it implements the rendering of a virtual endoscopic image corresponding to the respiratory phase based on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, including: Acquire the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, and perform position matching on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase; When the location is successfully matched, the electromagnetic sensing data and the bronchial image of the successfully matched location are rendered to obtain a virtual endoscopic image corresponding to the respiratory phase.

28. The computer-readable storage medium according to claim 27, characterized in that, When the computer program is executed by the processor, the following steps are implemented: acquiring the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, and performing position matching on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase: Obtain the first coordinate system corresponding to the electromagnetic sensing data and the first coordinate of the electromagnetic sensing data in the first coordinate system; Obtain the second coordinate system corresponding to the bronchial image and the second coordinates of the bronchial image in the second coordinate system; Based on the mapping relationship between the first coordinate and the second coordinate, the first coordinate system and the second coordinate system are registered.

29. The computer-readable storage medium according to claim 28, characterized in that, When the computer program is executed by the processor, the process of constructing a three-dimensional bronchial model of the tested object based on preoperative bronchial scan data includes: The bronchial scan data of the subject is acquired, and the bronchial scan data includes bronchial images of the subject at multiple time points; the span of the multiple time points covers at least one respiratory cycle of the subject when the bronchial scan data is acquired. Based on the bronchial scan data, a three-dimensional model of the bronchus is constructed in the second coordinate system.

30. The computer-readable storage medium according to claim 25, characterized in that, When the computer program is executed by the processor, the extraction of the bronchial centerline tree based on the bronchial three-dimensional model includes: The bronchus three-dimensional model is segmented to obtain the bronchus centerline corresponding to the bronchus; Based on the bronchial centerline and bronchial tree structure, the main airway of the bronchus corresponding to the bronchial centerline is obtained. An ordered tree is constructed with the main airway of the bronchus as the root node, which serves as the bronchial centerline tree of the bronchus.

31. The computer-readable storage medium according to claim 25, characterized in that, When the computer program is executed by the processor, the following steps are implemented: matching the actual endoscopic image with multiple virtual endoscopic images within the search range of the bronchial centerline, and using the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image, including: The actual endoscopic image is matched with a plurality of virtual endoscopic images within the search range of the bronchial centerline to obtain the similarity between the actual endoscopic image and the plurality of virtual endoscopic images within the search range of the bronchial centerline. Based on multiple similarity scores, the virtual endoscopic image corresponding to the largest similarity score is taken as the motion-compensated virtual endoscopic image.

32. The computer-readable storage medium according to claim 25, characterized in that, When the computer program is executed by the processor, the following steps are implemented: matching the actual endoscopic image with multiple virtual endoscopic images within the search range of the bronchial centerline, and using the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image, including: Extract the image features of the actual endoscope image as the actual features; Image features of each intermediate image are extracted as corresponding virtual features; wherein, the intermediate image is the virtual endoscopic image within the search range of the bronchial centerline; The actual features are matched with multiple virtual features to obtain feature similarity scores. The intermediate image corresponding to the feature similarity score with the highest value is used as the motion-compensated virtual endoscopic image.

33. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it performs the following steps: Acquire multiple frames of bronchial images and multiple electromagnetic sensor data of the tested object; Based on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, a virtual endoscopic image corresponding to the respiratory phase is rendered. Based on the bronchial scan data taken before the operation, a three-dimensional model of the bronchus of the subject was constructed, and the bronchial centerline tree was extracted based on the three-dimensional bronchial model. The positional change trend of the virtual endoscopic image within one respiratory cycle is calculated, the bronchial centerline search range is determined, and at least one virtual endoscopic image within the bronchial centerline search range is obtained based on the bronchial centerline search range. The method for determining the positional change trend includes: identifying the point cloud closest to the electromagnetic sensing data on the bronchial centerline as the target point cloud; calculating the depth information of the target point cloud in the virtual endoscopic image; and determining the motion range of the virtual endoscopic image within one respiratory cycle as the positional change trend based on the coordinates of the point cloud closest to the electromagnetic sensing data corresponding to each respiratory phase and the depth information of the target point cloud in the virtual endoscopic image. The actual endoscopic image is matched with at least one of the virtual endoscopic images within the search range of the bronchial centerline. The virtual endoscopic image that successfully matches the actual endoscopic image is used as the motion-compensated virtual endoscopic image, and the motion-compensated virtual endoscopic image is output.

34. The computer program product according to claim 33, characterized in that, The division of the respiratory phases implemented by the computer program when executed by the processor includes: Respiratory phases are defined based on the motion displacement of the breathing patch; Alternatively, the respiratory phase can be divided based on the time and displacement recorded by the respiratory patch; Alternatively, respiratory phases can be determined based on sensor data acquired by the sensors.

35. The computer program product according to claim 33, characterized in that, When the computer program is executed by the processor, it implements the rendering of a virtual endoscopic image corresponding to the respiratory phase based on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, including: Acquire the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, and perform position matching on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase; When the location is successfully matched, the electromagnetic sensing data and the bronchial image of the successfully matched location are rendered to obtain a virtual endoscopic image corresponding to the respiratory phase.

36. The computer program product according to claim 35, characterized in that, When the computer program is executed by the processor, the following steps are implemented: acquiring the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase, and performing position matching on the electromagnetic sensing data and the bronchial image corresponding to the same respiratory phase: Obtain the first coordinate system corresponding to the electromagnetic sensing data and the first coordinate of the electromagnetic sensing data in the first coordinate system; Obtain the second coordinate system corresponding to the bronchial image and the second coordinates of the bronchial image in the second coordinate system; Based on the mapping relationship between the first coordinate and the second coordinate, the first coordinate system and the second coordinate system are registered.

37. The computer program product according to claim 36, characterized in that, When the computer program is executed by the processor, the process of constructing a three-dimensional bronchial model of the tested object based on preoperative bronchial scan data includes: The bronchial scan data of the subject is acquired, and the bronchial scan data includes bronchial images of the subject at multiple time points; the span of the multiple time points covers at least one respiratory cycle of the subject when the bronchial scan data is acquired. Based on the bronchial scan data, a three-dimensional model of the bronchus is constructed in the second coordinate system.

38. The computer program product according to claim 33, characterized in that, When the computer program is executed by the processor, the extraction of the bronchial centerline tree based on the bronchial three-dimensional model includes: The bronchus three-dimensional model is segmented to obtain the bronchus centerline corresponding to the bronchus; Based on the bronchial centerline and bronchial tree structure, the main airway of the bronchus corresponding to the bronchial centerline is obtained. An ordered tree is constructed with the main airway of the bronchus as the root node, which serves as the bronchial centerline tree of the bronchus.

39. The computer program product according to claim 33, characterized in that, When the computer program is executed by the processor, the following steps are implemented: matching the actual endoscopic image with multiple virtual endoscopic images within the search range of the bronchial centerline, and using the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image, including: The actual endoscopic image is matched with a plurality of virtual endoscopic images within the search range of the bronchial centerline to obtain the similarity between the actual endoscopic image and the plurality of virtual endoscopic images within the search range of the bronchial centerline. Based on multiple similarity scores, the virtual endoscopic image corresponding to the largest similarity score is taken as the motion-compensated virtual endoscopic image.

40. The computer program product according to claim 33, characterized in that, When the computer program is executed by the processor, the following steps are implemented: matching the actual endoscopic image with multiple virtual endoscopic images within the search range of the bronchial centerline, and using the virtual endoscopic image that successfully matches the actual endoscopic image as the motion-compensated virtual endoscopic image, including: Extract the image features of the actual endoscope image as the actual features; Image features of each intermediate image are extracted as corresponding virtual features; wherein, the intermediate image is the virtual endoscopic image within the search range of the bronchial centerline; The actual features are matched with multiple virtual features to obtain feature similarity scores. The intermediate image corresponding to the feature similarity score with the highest value is used as the motion-compensated virtual endoscopic image.

Images