Information feedback, path planning, model establishing method and device

Abstract

This invention provides a method and apparatus for information feedback, path planning, and model building. The information feedback method includes: acquiring a three-dimensional model of the bronchial tract, wherein each bronchial segment has a unique identifier; the three-dimensional model is configured with a topology map, which includes the unique identifiers of each bronchial segment and the connection relationships between the bronchial segments represented by the unique identifiers; and providing feedback through at least one of the following methods: displaying the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path via image; playing the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path via voice; and adjusting the resistance on the target object manipulator according to the distance between the target object and the bronchial wall. This solution can reduce the experience requirements for bronchial surgery and reduce the probability of judgment errors.

Description

Technical Field

[0001] This application relates to the field of medical device technology, and in particular to information feedback, path planning, model building methods and devices. Background Technology

[0002] Bronchial surgery planning and preoperative simulation can greatly improve the proficiency and success rate of surgeons of all experience levels in actual bronchial surgery. Current surgical methods, after obtaining the planned path, typically require the surgeon to assess the condition of the target patient within the bronchus based on endoscopic images and a lung bronchial model, and then determine the appropriate control method based on the assessment results.

[0003] This shows that current technology requires a high level of experience from doctors and is prone to errors. Summary of the Invention

[0004] The purpose of this application is to provide information feedback, path planning, and model building methods and devices to address the problems of existing bronchial surgeries that require high skill from doctors and are prone to errors.

[0005] To address the aforementioned technical problems, this specification provides an information feedback method, comprising: acquiring a three-dimensional model of the bronchial tract, wherein each bronchial segment in the three-dimensional model has a unique identifier, and the three-dimensional model of the bronchial tract is further configured with a topology map, the topology map including the unique identifiers of each bronchial segment and the connection relationships between the bronchial segments represented by the unique identifiers; acquiring in real time the unique identifier of the bronchial segment where the target object is located and the unique identifier of the next bronchial segment based on the three-dimensional model and the topology map of the bronchial tract; and feeding back information through at least one of the following methods: presenting the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through an image, or playing the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through voice.

[0006] In some embodiments, the unique identifier of a bronchial segment contains bronchial segment level information, wherein a lower-level bronchial segment is a branch of a higher-level bronchial segment.

[0007] In some embodiments, presenting the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment through an image includes: presenting a two-dimensional top view image of the bifurcation point at the front end of the bronchial segment where the target object is located, wherein the two-dimensional top view image contains the unique identifier of each bronchial segment associated with the bifurcation point; and / or, presenting a virtual image inside the bronchial segment where the target object is located, wherein the virtual image inside the bronchial segment contains the unique identifier of each bronchial segment associated with the bifurcation point.

[0008] In some embodiments, the feedback information may further include: determining a resistance level based on a unique identifier of the bronchial segment where the target object is located; the resistance level being used to represent the range of resistance values ​​fed back to the target object manipulator; and controlling the resistance value fed back to the target object manipulator based on the resistance level.

[0009] In some embodiments, controlling the resistance value fed back to the target object manipulator according to the resistance level includes: obtaining the distance between the target object and the bronchial wall; when the distance between the target object and the bronchial wall is less than a first threshold, adjusting the resistance fed back to the target object manipulator within the range corresponding to the resistance level according to the inverse relationship between resistance and distance; the first threshold is determined according to the resistance level.

[0010] In some embodiments, controlling the resistance value fed back to the target manipulator according to the resistance level further includes: maintaining the resistance on the target manipulator at a first resistance value when the distance between the target object and the bronchial wall is less than a second threshold, wherein the second threshold is less than the first threshold, and each resistance value corresponding to the distance between the first threshold and the second threshold is less than the first resistance value; the second threshold is determined according to the resistance level.

[0011] In some embodiments, the feedback information may further include: controlling the target manipulator to vibrate when the distance between the target object and the bronchial wall is less than a third threshold; the third threshold is less than or equal to the second threshold, and the third threshold is determined based on the resistance level.

[0012] A second aspect of this specification provides a path planning method, comprising: obtaining a three-dimensional model of the bronchus of the lungs, wherein each bronchus segment in the three-dimensional model of the bronchus has a unique identifier, and the three-dimensional model of the bronchus of the lungs is further configured with a topology map, wherein the topology map includes the unique identifier of each bronchus segment and the connection relationship between the bronchus segments represented by the unique identifier; determining the unique identifier of the target bronchus segment closest to the target pulmonary nodule; and, based on the topology map, determining the unique identifiers of each bronchus segment that the path from the main trachea to the target bronchus passes through in sequence, thereby obtaining the planned path represented by the unique identifier of each bronchus segment.

[0013] In some embodiments, the method further includes: replacing the bronchus with the centerline of each bronchus segment in the three-dimensional model of the lung bronchus to obtain a skeleton model of the lung bronchus; after obtaining the planned path represented by the unique identifier of each bronchus segment, the method further includes: determining the centerline of the bronchus corresponding to the unique identifier of each bronchus segment in the planned path in the skeleton model of the lung bronchus to obtain the planned path composed of the centerlines of each bronchus segment.

[0014] The third aspect of this specification provides a method for establishing a lung bronchus model, comprising: acquiring multi-slice scan images of the lung; inputting the multi-slice scan images of the lung into a pre-trained network model to obtain multi-slice scan images labeled with unique identifiers of each bronchial segment; and reconstructing a three-dimensional image of the lung bronchus from the multi-slice scan images labeled with unique identifiers of each bronchial segment, wherein the three-dimensional image is configured with a topology map.

[0015] In some embodiments, before inputting the multi-layer scan images of the lungs into a pre-trained network model, the method further includes: training the network model by: acquiring multiple training data, one of which includes multi-layer scan images of the lungs, each layer of the scan images being labeled with a unique identifier for each bronchus segment; the unique identifier for each bronchus carrying bronchial level information; and training the network model using the multiple training data, wherein the multi-layer scan images in each training data are used as input to the network model, and the unique identifiers of each bronchus segment labeled in each layer of the image are used as output to the network model.

[0016] In some embodiments, the unique identifier of each bronchus segment in each layer of scanned images in the training data is marked by the following method: identifying each bronchus segment and assigning a unique identifier to each bronchus segment; performing the following operation for each bronchus segment: identifying multiple pixels on the scanned image that make up the current bronchus, and marking each of the multiple pixels as the unique identifier of the current bronchus.

[0017] In some embodiments, the method further includes: identifying each lung nodule during annotation and assigning a unique identifier to each lung nodule; and performing the following operations for each lung nodule: identifying a plurality of pixels on the scanned image that make up the current lung nodule and marking each of the plurality of pixels as a unique identifier for the current lung nodule.

[0018] This specification provides an information feedback device in a fourth aspect, comprising: a first acquisition unit for acquiring a three-dimensional model of the bronchus of the lungs, wherein each bronchus segment in the three-dimensional model of the bronchus has a unique identifier, and the three-dimensional model of the bronchus of the lungs is also configured with a topology map, the topology map including the unique identifier of each bronchus segment and the connection relationship between the bronchus segments represented by the unique identifier; a second acquisition unit for acquiring, in real time, the unique identifier of the bronchus segment where the target object is located and the unique identifier of the next bronchus segment based on the three-dimensional model of the bronchus of the lungs and the topology map; and a feedback unit for feeding back information in at least one of the following ways: presenting the unique identifier of the bronchus segment where the target object is located and / or the unique identifier of the next bronchus segment in the planned path through an image, or playing the unique identifier of the bronchus segment where the target object is located and / or the unique identifier of the next bronchus segment in the planned path through voice.

[0019] In some embodiments, the unique identifier of a bronchial segment contains bronchial segment level information, wherein a lower-level bronchial segment is a branch of a higher-level bronchial segment.

[0020] In some embodiments, the feedback unit includes: a first presentation subunit, configured to present a two-dimensional top-view image of the bifurcation point at the front end of the bronchial segment where the target object is located, the two-dimensional top-view image containing a unique identifier of each bronchial segment associated with the bifurcation point; and / or, a second presentation subunit, configured to present a virtual image inside the bronchial segment where the target object is located, the virtual image inside the bronchial segment containing a unique identifier of each bronchial segment associated with the bifurcation point.

[0021] In some embodiments, the feedback unit includes: a determining subunit, configured to determine a resistance level based on a unique identifier of the bronchial segment where the target object is located; the resistance level is used to represent the range of resistance values ​​fed back to the target object manipulator; and a first resistance feedback subunit, configured to control the resistance value fed back to the target object manipulator based on the resistance level.

[0022] In some embodiments, the first resistance feedback subunit includes: an acquisition subunit for acquiring the distance between the target object and the bronchial wall; and a second resistance feedback subunit for adjusting the resistance fed back to the target object manipulator within a range corresponding to the resistance level, according to the inverse relationship between resistance and distance, when the distance between the target object and the bronchial wall is less than a first threshold; the first threshold is determined based on the resistance level.

[0023] In some embodiments, the first resistance feedback subunit includes: a third resistance feedback subunit, configured to maintain the resistance on the target object manipulator at a first resistance value when the distance between the target object and the bronchial wall is less than a second threshold, wherein the second threshold is less than the first threshold, and each resistance value corresponding to the distance between the first threshold and the second threshold is less than the first resistance value; the second threshold is determined according to the resistance level.

[0024] In some embodiments, the feedback unit further includes: a vibration feedback subunit, used to control the target object manipulator to vibrate when the distance between the target object and the bronchial wall is less than a third threshold; the third threshold is less than or equal to the second threshold, and the third threshold is determined according to the resistance level.

[0025] This specification provides a path planning device in a fifth aspect, comprising: a first acquisition unit for acquiring a three-dimensional model of a lung bronchus, wherein each bronchus segment in the three-dimensional model of the lung bronchus has a unique identifier, and the three-dimensional model of the lung bronchus is further configured with a topology map, the topology map including the unique identifiers of each bronchus segment and the connection relationships between the bronchus segments represented by the unique identifiers; a first determination unit for determining the unique identifier of the target bronchus segment closest to the target lung nodule; and a second determination unit for determining, based on the topology map, the unique identifiers of each bronchus segment sequentially traversed by the path from the main trachea to the target bronchus, thereby obtaining the planned path represented by the unique identifier of each bronchus segment.

[0026] In some embodiments, the apparatus further includes: a skeleton extraction unit, configured to replace the bronchus with the centerline of each bronchus segment in a three-dimensional model of the lung bronchus to obtain a skeleton model of the lung bronchus; and a third determination unit, configured to, after obtaining the planned path represented by the unique identifier of each bronchus segment, determine the centerline of the bronchus corresponding to the unique identifier of each bronchus segment in the skeleton model of the lung bronchus to obtain the planned path composed of the centerlines of each bronchus segment.

[0027] The sixth aspect of this specification provides an apparatus for establishing a lung bronchus model, comprising: a third acquisition unit for acquiring multi-slice scan images of the lung; an identification unit for inputting the multi-slice scan images of the lung into a pre-trained network model to obtain multi-slice scan images labeled with unique identifiers for each bronchial segment; and a reconstruction unit for reconstructing a three-dimensional image of the lung bronchus from the multi-slice scan images labeled with unique identifiers for each bronchial segment, wherein the three-dimensional image is configured with a topology map.

[0028] In some embodiments, before inputting the multi-layer scan images of the lungs into a pre-trained network model, the method further includes: training the network model by: acquiring multiple training data, one of which includes multi-layer scan images of the lungs, each layer of the scan images being labeled with a unique identifier for each bronchus segment; the unique identifier for each bronchus carrying bronchial level information; and training the network model using the multiple training data, wherein the multi-layer scan images in each training data are used as input to the network model, and the unique identifiers of each bronchus segment labeled in each layer of the image are used as output to the network model.

[0029] In some embodiments, the unique identifier of each bronchus segment in each layer of scanned images in the training data is marked by the following method: identifying each bronchus segment and assigning a unique identifier to each bronchus segment; performing the following operation for each bronchus segment: identifying multiple pixels on the scanned image that make up the current bronchus, and marking each of the multiple pixels as the unique identifier of the current bronchus.

[0030] In some embodiments, during annotation, each lung nodule is identified and a unique identifier is assigned to each lung nodule; for each lung nodule, the following operations are performed: multiple pixels on the scanned image that make up the current lung nodule are identified, and each of the multiple pixels is marked as a unique identifier for the current lung nodule.

[0031] This specification provides a bronchial surgery system in a seventh aspect, comprising: a surgical robot including a base, a robotic arm, a track, a power unit, and a catheter; a first end of the robotic arm is mounted on the base, and a second end is suspended; the track is mounted on the second end of the robotic arm; the power unit is mounted on the track and can move along the extension direction of the track; a first end of the catheter is mounted on the power unit, and a second end can extend into the patient's bronchus; the second end of the catheter is equipped with an image acquisition component or instrument; a manipulator for manipulating the surgical robot to move the second end of the catheter along a specified path within the pulmonary bronchus; and a controller for acquiring a three-dimensional model of the pulmonary bronchus. Each bronchus segment in the model has a unique identifier, and the three-dimensional model of the lung bronchi is also equipped with a topology map, which includes the unique identifier of each bronchus segment and the connection relationship between the bronchus segments represented by the unique identifier; the unique identifier of the bronchus segment where the target object is located and the unique identifier of the next bronchus segment are obtained in real time based on the three-dimensional model of the lung bronchi and the topology map; a presentation device is used to provide feedback to the doctor in at least one of the following ways: presenting the unique identifier of the bronchus segment where the target object is located and / or the unique identifier of the next level bronchus segment in the planned path through an image, or playing the unique identifier of the bronchus segment where the target object is located and / or the unique identifier of the next level bronchus segment in the planned path through voice.

[0032] The eighth aspect of this specification provides a computer storage medium storing computer program instructions that, when executed by a processor, implement the steps of the method described in any one of the first, second, third, and fourth aspects.

[0033] The information feedback, path planning, model building methods and devices provided in this manual are based on the three-dimensional model and topology of the lung bronchi. They can provide real-time feedback to the surgeon on the condition of the target object in the bronchi through multiple modes such as vision, hearing, and touch, based on the unique identifier of each bronchial segment. This helps the surgeon understand the condition of the target object in the bronchi, thereby reducing the experience required for bronchial surgery and reducing the probability of judgment errors. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0035] Figure 1 A flowchart of the information feedback method provided in this specification is shown;

[0036] Figure 2 A schematic diagram of a three-dimensional model of the lung bronchi is shown;

[0037] Figure 3 A schematic diagram of the topology configured for a three-dimensional model of the lung bronchi is shown.

[0038] Figure 4 A schematic diagram showing a two-dimensional top view of the bifurcation point at the front end of the bronchus where the target object is located;

[0039] Figure 5 This is a schematic diagram showing a virtual image inside the bronchus at the bifurcation of the front end where the target object is located;

[0040] Figure 6 A schematic diagram showing a virtual image within the imaging tube of VR glasses is shown;

[0041] Figure 7 A schematic diagram illustrates a method for adjusting the resistance on a target manipulator based on the distance between the target object and the bronchial wall;

[0042] Figure 8 A schematic diagram illustrates an embodiment of providing feedback to surgeons using a multimodal interaction method;

[0043] Figure 9 A flowchart of a bronchial surgery pathway planning method provided in this specification is shown;

[0044] Figure 10 A schematic diagram of a skeletal model of the lung bronchi is shown;

[0045] Figure 11 A flowchart illustrating a method for establishing a lung bronchus model provided in this specification is shown;

[0046] Figure 12 A schematic diagram of a single-layer CT image is shown;

[0047] Figure 13 A schematic diagram showing a bronchus solidly labeled is shown;

[0048] Figure 14 A schematic diagram showing a lung nodule marked with solid lines is shown;

[0049] Figure 15 A schematic diagram of a bronchial surgery system is shown;

[0050] Figure 16 A schematic diagram of the surgical robot in a bronchial surgery system is shown.

[0051] Figure 17 A schematic diagram of the structure of a controller provided in this specification is shown. Detailed Implementation

[0052] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0053] This specification provides an information feedback method that, based on a three-dimensional model and topology map of the lung bronchi, can determine the bronchial segment where the target object is located in real time, and based on the determination result, provides real-time feedback to the surgeon on the condition of the target object in the bronchus, thereby reducing the requirements for the doctor's experience and reducing the probability of judgment errors.

[0054] In the 3D model of the lung bronchi provided in this specification, each bronchial segment has a unique identifier. In existing technologies, 3D models of the lung bronchi typically only have a three-dimensional structure and do not identify each bronchial segment or assign a unique identifier to each segment. Because the number and complexity of bronchial segments are numerous, manually marking the bronchial segments in the 3D model is impractical. Therefore, existing technologies, when performing bronchial surgery based on a 3D model of the lung bronchi, do not display the unique identifier of the bronchial segment where the target patient is located, the unique identifier of the next-level bronchial segment, or other information to assist the surgeon in assessing the patient's condition within the bronchus.

[0055] like Figure 1 As shown, the information feedback method provided in this manual includes the following steps:

[0056] S110: Obtain a three-dimensional model of the lung bronchi. Each bronchus segment in the three-dimensional model of the lung bronchi has a unique identifier. The three-dimensional model of the lung bronchi is also configured with a topology map, which includes the unique identifier of each bronchus segment and the connection relationship between the bronchus represented by the unique identifier.

[0057] A three-dimensional model of the lung bronchi can be like Figure 2 As shown, each bronchus segment in the three-dimensional model of the lung bronchi has a unique identifier.

[0058] In this manual, "a section of bronchus" and "a bronchial segment" have the same meaning, both referring to a section of bronchus between the bifurcation points of two adjacent bronchi.

[0059] The unique identifiers for each bronchus segment in the three-dimensional model of the lung bronchi can be manually labeled, assigned by a computer based on the tree-like structural features of the bronchi, or determined using the lung bronchi model establishment method provided below. This method can be referred to the following description.

[0060] The three-dimensional model of the lung bronchi described in this specification is a three-dimensional image model. Each pixel constituting a bronchial segment in this three-dimensional image is labeled with a unique identifier for that bronchial segment. To illustrate the connections between bronchial segments, the three-dimensional model of the lung bronchi also includes a topology map. This topology map includes the unique identifiers of each bronchial segment and the connections between the bronchial segments identified by those unique identifiers. For example, if bronchial segments B and C are the next-level bronchi after bronchial segment A, and bronchimeters D and E are the next-level bronchi after bronchial segment B, then this can be represented as follows: Figure 3 The topology diagram shown indicates that A, B, C, D, and E represent unique identifiers for bronchial segments. Figure 3 The lines in the diagram represent the connection between two bronchial segments.

[0061] In some embodiments, the unique identifier of a bronchial segment contains bronchial segment level information, wherein a lower-level bronchial segment is a branch of a higher-level bronchial segment.

[0062] In some cases, the bronchial segment levels can be represented numerically during labeling, using a format of "level + label value" to label each bronchial segment. For example, level 1 corresponds to all main tracheae, level 2 to all lobar tracheae, level 3 to the superior bronchus, lingular bronchus, and so on. Level 3 belongs to level 2 (i.e., level 3 is a branch of level 2), and level 2 belongs to level 1 (i.e., level 2 is a branch of level 1). The labeling results can be: 3-5, 3-6, 2-4. Bronchial segments 3-5 and 3-6 are the next level bronchial segments after bronchial segment 2-4. The number to the left of the "-" indicates the level, and the number to the right of the "-" indicates the label value.

[0063] In some cases, since the name of the bronchus itself also contains hierarchical information, the bronchial segment can be labeled using the method of "bronchial name (such as main airway, left main bronchus, right main bronchus, right upper lobe bronchus, etc.) + label value", for example, right posterior segment bronchus (5), right anterior segment bronchus (6), right upper lobe bronchus (4).

[0064] The above examples illustrate two methods for labeling bronchial segments. As can be seen from these examples, the labeling content (i.e., unique identifier) ​​of a bronchial segment includes information about its bronchial level. Inspired by these two examples, those skilled in the art can conceive of many other labeling methods, which will not be listed here.

[0065] When the unique identifier of the bronchial segment containing the aforementioned hierarchical information is fed back to the surgeon, the surgeon can more quickly and intuitively perceive the global position of the target object in the three-dimensional model of the lung bronchi.

[0066] S120: Based on the three-dimensional model and topology of the lung bronchi, obtain in real time the unique identifier of the bronchial segment where the target object is located and the unique identifier of the next bronchial segment.

[0067] For example, combining Figure 2 The three-dimensional model of the lung bronchi shown Figure 3 As shown in the topology diagram, if the current target object is located in bronchial segment B, then during the process of controlling the target bronchus to move forward, if a bronchial branch is encountered, the next level bronchial segment of the branch can be determined to be D or E. Combining the orientation information of the next level bronchus in the 3D model and the control direction of the target object, it can be determined whether the target object is about to or actually passes through bronchial segment D or E.

[0068] The target objects in this instruction manual can be surgical instruments such as endoscopes or biopsy instruments.

[0069] S130: Provide surgical information to the doctor in at least one of the following ways: present the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through an image; or play the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through voice.

[0070] In some embodiments, presenting the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through an image can be: presenting a two-dimensional top view image of the bifurcation point at the front end of the bronchial segment where the target object is located, wherein the two-dimensional top view image contains the unique identifiers of each bronchial segment associated with the bifurcation point.

[0071] Figure 4This diagram illustrates a two-dimensional top-view image of the bronchial bifurcation at the front of the bronchus where the target object is located. The two-dimensional top-view image is an image of the bronchial bifurcation viewed from a plane perpendicular to the plane containing the bronchial segments associated with the bifurcation. The bronchial walls are outlined with lines in this two-dimensional top-view image. In some cases, a planned path can also be presented in the two-dimensional top-view image, for example... Figure 4 The line M in the diagram represents the planned path.

[0072] In some embodiments, presenting the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through an image can be: presenting a virtual image of the bifurcation point at the front end of the bronchial segment where the target object is located, wherein the virtual image contains the unique identifiers of each bronchial segment associated with the bifurcation point.

[0073] Figure 5 This diagram illustrates a virtual image of the bronchial bifurcation at the front of the bronchus where the target object is located. The virtual image corresponds to the virtual image of the bronchus as observed from the endoscopic viewpoint on the target object. This virtual image can be a three-dimensional image rendered based on information from the bronchial bifurcation. In some cases, a planned path can also be presented in the virtual image, for example... Figure 5 The line N in the diagram represents the planned path.

[0074] The virtual images within the tube can be displayed on a monitor, for example... Figure 5 As shown; it can also be presented through VR glasses, for example. Figure 6 As shown; it can also be presented through other methods such as holographic projection.

[0075] In some embodiments, surgical information can also be fed back to the doctor in the following ways: the resistance level is determined based on the unique identifier of the bronchial segment where the target object is located, and the resistance value fed back to the target object manipulator is controlled according to the resistance level.

[0076] The resistance level is used to represent the range of resistance values ​​fed back to the target object manipulator, and the resistance value ranges of adjacent resistance levels may overlap.

[0077] Typically, the unique identifier of a bronchial segment contains information about its level. Since different levels of bronchial segments vary in diameter, the required level of focus during manipulation differs. Therefore, different resistance levels are set for different levels of bronchial segments. When the surgeon controls the movement of the target object within different levels of bronchial segments, different levels of resistance are fed back to the target object manipulator based on the bronchial segment level. This allows the surgeon to sense the target object entering different bronchial segments, and a higher resistance value enables the surgeon to operate more precisely. This allows for faster manipulation in thicker-walled bronchial segments and slower, more precise manipulation in thinner-walled bronchial segments, improving both the accuracy and efficiency of bronchial surgery.

[0078] The resistance value fed back to the target object manipulator based on the aforementioned resistance level can be selected from the resistance value range corresponding to the resistance level and fed back to the target object manipulator, or the resistance value in the resistance value range corresponding to the resistance level can be processed and the processed resistance value can be used as the resistance value fed back to the target object manipulator.

[0079] In some embodiments, after processing the resistance value within the resistance value range corresponding to the resistance level, the processed resistance value is used as the resistance value fed back to the target object manipulator. This can be achieved by: obtaining the distance between the target object and the bronchial wall; when the distance between the target object and the bronchial wall is less than a first threshold, adjusting the resistance fed back to the target object manipulator within the value range corresponding to the resistance level according to the inverse relationship between resistance and distance.

[0080] The distance between the target object and the bronchial wall can be the minimum distance between the target object and the surrounding bronchial walls. This distance can be obtained by setting sensors on the target object, or it can be obtained based on the centerline and radius of the bronchus in the 3D model of the lung bronchi, combined with the actual movement path of the target object.

[0081] The first threshold is determined based on the resistance level. For example, if a range of values ​​for the pipe wall radius corresponding to the resistance level is pre-set, then the first threshold can be the product of the pipe wall radius corresponding to the resistance level and a coefficient (e.g., 0.3, 0.2, 0.1).

[0082] This inverse relationship means that the smaller the distance, the greater the resistance. This inverse relationship can be a reciprocal function curve, a straight line with a negative slope, or similar.

[0083] The smaller the distance between the target object and the bronchial wall, the easier it is for the surgeon to cause the target object to collide with the bronchial wall, thus causing pain to the patient. Therefore, increasing the resistance felt by the surgeon when operating the manipulator allows the surgeon to perceive the distance between the target object and the bronchial wall tactilely, enabling timely adjustments to the control method and preventing the target object from colliding with the bronchial wall.

[0084] Furthermore, when the distance between the target object and the bronchial wall is less than the second threshold, the resistance on the target object manipulator is maintained at the first resistance value, wherein the second threshold is less than the first threshold, and each resistance value corresponding to the distance between the first threshold and the second threshold is less than the first resistance value.

[0085] The second threshold is determined based on the resistance level. For example, if a range of values ​​for the pipe wall radius corresponding to the resistance level is preset, then the second threshold can be the product of the pipe wall radius corresponding to the resistance level and a coefficient (e.g., 0.3, 0.2, 0.1), and the coefficient corresponding to the first threshold is less than the coefficient corresponding to the first threshold.

[0086] Furthermore, surgical information can be fed back to the doctor in the following way: when the distance between the target object and the bronchial wall is less than the third threshold, the target object manipulator is controlled to vibrate.

[0087] The third threshold is determined based on the resistance level. For example, if a range of pipe wall radii corresponding to the resistance level is pre-set, then the third threshold can be the product of the pipe wall radius corresponding to the resistance level and a coefficient (e.g., 0.3, 0.2, 0.1), and the coefficient corresponding to the third threshold is less than or equal to the coefficient corresponding to the second threshold.

[0088] Compared to resistance, vibration is more likely to attract the surgeon's attention and alertness. Vibration can be generated when the target object is very close to the bronchial wall, alerting the surgeon that the distance between them is extremely short. Furthermore, the vibration frequency can be adjusted based on the inverse relationship between vibration frequency and distance; that is, the closer the target object is to the bronchus, the higher the vibration frequency.

[0089] Figure 7This diagram illustrates a method for adjusting the resistance on a target object manipulator based on the distance between the target object and the bronchial wall. The horizontal axis represents the distance between the target object and the bronchial wall, and the vertical axis represents the resistance fed back to the target object manipulator. When the distance is less than a resistance threshold (i.e., a first threshold) and greater than a vibration threshold (i.e., a second threshold), the resistance fed back to the target object manipulator is adjusted according to the inverse relationship between resistance and distance. When the distance is between 0 and the vibration threshold (i.e., the second threshold), vibration is maintained, and the resistance on the target object is kept at the first resistance value. Figure 7 In the embodiment shown, the second threshold and the third threshold have the same value.

[0090] The information feedback method provided in this manual is based on a three-dimensional model and topological map of the lung bronchi. It can provide real-time feedback to the surgeon on the condition of the target object in the bronchi through multiple modes such as vision, hearing, and touch, based on the unique identifier of each bronchial segment. This helps the surgeon understand the condition of the target object in the bronchi, thereby reducing the experience required for bronchial surgery and reducing the probability of misjudgment.

[0091] For example, Figure 8 The diagram illustrates an embodiment of providing feedback to surgeons using a multi-modal interactive approach. This embodiment can simultaneously provide feedback to surgeons through multiple modes, including visual, auditory, and tactile feedback. Visual feedback includes displaying image information using VR glasses, which includes the aforementioned two-dimensional top-view image, virtual image inside the tube, and three-dimensional model image of the lung and bronchus model. Auditory feedback includes the aforementioned information played back via voice. Tactile feedback includes the aforementioned adjustment feedback of resistance, vibration, etc., on the manipulators applied to the target object.

[0092] In step S120 above, when controlling the movement of the target object within the bronchus, the surgeon typically controls the movement according to a pre-determined planned path. Based on the three-dimensional model of the lungs and bronchi provided in this specification, this specification provides a path planning method for bronchial surgery, which can be used in electronic devices with computing capabilities. For example... Figure 9 As shown, the path planning method includes the following steps:

[0093] S210: Obtain a three-dimensional model of the lung bronchi. Each bronchus segment in the three-dimensional model of the lung bronchi has a unique identifier. The three-dimensional model of the lung bronchi is also configured with a topology map, which includes the unique identifier of each bronchus segment and the connection relationship between the identifiers of each bronchus segment.

[0094] In some embodiments, the three-dimensional model of the bronchus may further include a target pulmonary nodule sub-model, which can be used to determine the location of the target pulmonary nodule in the three-dimensional model of the bronchus. In some embodiments, the three-dimensional model of the bronchus may not include the target pulmonary nodule, but instead, the relative positional relationship between the pulmonary nodule and at least two points in the lung is determined by other means, and the location of the target pulmonary nodule is marked in the three-dimensional model of the bronchus based on these relative positional relationships.

[0095] For further descriptions of the 3D model of the lung bronchi, please refer to [link to relevant documentation]. Figure 1 The embodiments shown are not described in detail here.

[0096] S220: A unique identifier for determining the bronchial segment closest to the target lung nodule.

[0097] After determining the location of the target lung nodule in the three-dimensional model of the lung bronchi, the nearest bronchial segment can be searched based on the location of the target lung nodule, and the unique identifier of that bronchial segment can be obtained. Specific search methods can employ existing technologies, which will not be elaborated upon in this specification.

[0098] S230: Based on the topology map, determine the unique identifiers of each bronchial segment that the path from the main bronchus to the target bronchus passes through in sequence, and obtain the planned path represented by the unique identifier of each bronchial segment.

[0099] When the unique identifier of a bronchial segment contains level information, the path planning result can be generated using the following method, including the following steps:

[0100] A1. Based on the centerline extraction results, the apex of the bronchus is selected as the initial point and the point on the centerline closest to the lung nodule is selected as the target point.

[0101] A2. Based on the bronchial level identification results, obtain the label value of the bronchus or level to which all points belong on the central line, and record the label values ​​of the initial point and the target point.

[0102] A3. If there is a neighboring point with a value smaller than the current point's mark, it means that the next higher level bronchus has been found. The previous point is a valid point and needs to be recorded. The mark should be cleared to prevent accidental release later.

[0103] A4. If there is a neighboring point with a value greater than the current point's mark, it means that the next level bronchus has been found, but the direction is incorrect. In this case, stop the calculation and release the previously saved point of the current bronchus.

[0104] A5. If there are multiple neighboring points with the same mark value as the current point, it is uncertain which direction is correct. Therefore, it is necessary to traverse them separately and mark them for release.

[0105] In some embodiments, after obtaining the planned path represented by the unique identifier of each bronchial segment, the target object can be controlled to move from the main trachea toward the target lung nodule according to the path. During the motion control process, the next position of the target object on the bronchial cross-section is determined in real time so that the target object is far away from the surrounding bronchial walls and is less likely to collide with the bronchial walls.

[0106] In some embodiments, the centerline of each bronchus segment in the three-dimensional model of the lung bronchi can be used to replace the bronchus, thus obtaining a skeletal model of the lung bronchi. Accordingly, after obtaining the planned path represented by the unique identifier of each bronchial segment, it is also possible to: determine the centerline of the bronchus corresponding to the unique identifier of each bronchial segment in the planned path in the skeletal model of the lung bronchi, thus obtaining the planned path composed of the centerlines of each bronchial segment.

[0107] For example, according to Figure 2 The skeletal model of the lung bronchi obtained from the three-dimensional model of the lung bronchi shown can be as follows: Figure 10 As shown. Since the skeletal model of the lung bronchi is obtained by extracting the centerline from its 3D model, and the points on the centerline are the center points of the cross-section of the bronchus in the 3D model, therefore: 1. Each pixel in the skeletal model that constitutes the centerline still has a unique identifier for the bronchial segment; 2. The skeletal model of the lung bronchi and the 3D model are in the same coordinate system. The operation of extracting the centerline can use 3D thinning algorithms.

[0108] This specification also provides a method for establishing a lung bronchus model, which can be used in electronic devices with computing capabilities.

[0109] like Figure 11 As shown, it includes the following steps:

[0110] S310: Acquire multi-slice scan images of the lungs.

[0111] Lung scans are typically CT (Computed Tomography) images. This method uses precisely collimated X-ray beams, gamma rays, ultrasound, etc., along with highly sensitive detectors, to perform a series of cross-sectional scans around a specific part of the body.

[0112] Figure 12 This diagram illustrates a single-layer CT image. The curve X represents the bed on which the patient lay during the CT scan, the white area represents a cross-section of the patient's body, and the black area Y represents the lungs. Multiple lung CT scans are typically taken. Figure 12 The images shown are all multi-layer images, each corresponding to a layer, meaning that multi-layer scan images are cross-sectional images of the lungs taken from multiple different locations.

[0113] S320: Input the multi-slice scan image of the lungs into a pre-trained network model to obtain a multi-slice scan image labeled with a unique identifier for each bronchial segment.

[0114] The input to this network model is a multi-slice scan image of the lungs, and the output is a multi-slice scan image labeled with a unique identifier for each bronchial segment. In other words, the role of the network model is to label the unique identifier of each bronchial segment in the multi-slice scan image.

[0115] S330: Reconstruct a three-dimensional image of the lung bronchi from multi-slice scan images labeled with unique identifiers for each bronchial segment, wherein the three-dimensional image is configured with a topological map.

[0116] Reconstructing three-dimensional images of the lung bronchi from multi-slice scan images is something that can be done by those skilled in the art, and will not be described in detail here.

[0117] The topology map can be determined during the reconstruction process based on the branching relationship between bronchial segments in the reconstructed 3D image; or it can be determined when the scanned image is annotated in step S320. That is, the topology map can also be output from the network model.

[0118] The method for reconstructing three-dimensional images of lung bronchi provided in this manual uses a pre-trained network model to annotate the scanned images, which can automatically identify the names of each bronchial segment in the scanned images with high recognition efficiency and accuracy.

[0119] In some embodiments, the above network model can be trained using the following methods:

[0120] S410: Acquire multiple training data, one of which includes multi-layer scan images of the lungs, each layer of the scan image being marked with a unique identifier for each segment of the bronchus; the unique identifier of the bronchus carries bronchial level information.

[0121] S420: The network model is trained using the multiple training data, wherein the multi-layer scan images in each training data are used as the input of the network model, and the unique identifier of each bronchus segment marked in each layer image is used as the output of the network model.

[0122] The network model trained above can be any kind of deep learning network model, and this specification does not limit it.

[0123] In some embodiments, the unique identifier of each bronchial segment in each layer of scanned images in the training data is labeled through the following steps S430 and S440:

[0124] S430: Identifies each bronchial segment in the scanned image and assigns a label to each bronchial segment.

[0125] This step can be done manually.

[0126] When assigning bronchial segment identifiers, unique identifiers containing level information can be used. This makes it easier for annotators to understand the assigned identifiers, and the predictable patterns in the unique identifiers of each bronchial segment allow the network model to converge faster during training, resulting in higher recognition accuracy. For details on the level information of unique identifiers, please refer to [link to relevant documentation]. Figure 1 The descriptions in the illustrated embodiments will not be repeated here.

[0127] S440: For each bronchial segment, perform the following operation: identify multiple pixels on the scanned image that make up the current bronchus, and mark each of the multiple pixels as the identifier of the current bronchus.

[0128] Figure 13 The figure below illustrates a schematic diagram of step S440, which involves labeling bronchial segments in a scanned image. The white areas represent pixels that make up multiple bronchial segments. During labeling, the labeler can identify the bronchial segment and select all pixels that make up that bronchus, marking each pixel as a unique identifier for that bronchus. This labeling method is also known as solid labeling.

[0129] When annotating each bronchial segment, the annotator can determine which bronchial segment to annotate based on the usual location of the pulmonary nodules (e.g., which bronchial segment they are usually close to).

[0130] In some embodiments, it may be necessary to know the location of the pulmonary nodules in the three-dimensional model of the lung bronchi. In this regard, when annotating the scanned images, the annotator can also identify the pulmonary nodules in the scanned images, assign a unique identifier to the pulmonary nodules, and mark each of the multiple pixels that make up the pulmonary nodule image as the unique identifier of the pulmonary nodule.

[0131] Figure 14 The image below illustrates a diagram of labeling a lung nodule. The white areas represent the pixels that make up the nodule. During labeling, the labeler can identify the nodule and select all the pixels that make it up, marking each pixel as a unique identifier for the nodule. This labeling method is also known as solid labeling.

[0132] Since lung nodules are labeled in the training data, the network model trained using this data can also automatically identify and label lung nodules in scanned images. Accordingly, the three-dimensional model of the lung bronchi can include a lung nodule sub-model.

[0133] This specification provides an information feedback device, including a first acquisition unit, a second acquisition unit, and a feedback unit.

[0134] The first acquisition unit is used to acquire a three-dimensional model of the bronchial tubes of the lungs. Each bronchial segment in the three-dimensional model of the lungs has a unique identifier. The three-dimensional model of the lungs also includes a topology map, which includes the unique identifiers of each bronchial segment and the connection relationships between the bronchial segments represented by the unique identifiers. The second acquisition unit is used to acquire, in real time, the unique identifier of the bronchial segment where the target object is located and the unique identifier of the next bronchial segment based on the three-dimensional model of the lungs and the topology map. The feedback unit is used to provide surgical information to the doctor in at least one of the following ways: presenting the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through an image; or playing the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through voice.

[0135] In some embodiments, the unique identifier of a bronchial segment contains bronchial segment level information, wherein a lower-level bronchial segment is a branch of a higher-level bronchial segment.

[0136] In some embodiments, the feedback unit includes a first presentation subunit and / or a second presentation subunit. The first presentation subunit is used to present a two-dimensional top-view image of the bifurcation point at the front of the bronchial segment where the target object is located, the two-dimensional top-view image containing unique identifiers of each bronchial segment associated with the bifurcation point. The second presentation subunit is used to present a virtual image of the bronchial segment at the bifurcation point at the front of the bronchial segment where the target object is located, the virtual image of the bronchial segment containing unique identifiers of each bronchial segment associated with the bifurcation point.

[0137] In some embodiments, the feedback unit includes: a determining subunit, configured to determine a resistance level based on a unique identifier of the bronchial segment where the target object is located; the resistance level is used to represent the range of resistance values ​​fed back to the target object manipulator; and a first resistance feedback subunit, configured to control the resistance value fed back to the target object manipulator based on the resistance level.

[0138] In some embodiments, the first resistance feedback subunit includes: an acquisition subunit for acquiring the distance between the target object and the bronchial wall; and a second resistance feedback subunit for adjusting the resistance fed back to the target object manipulator within a range corresponding to the resistance level, according to the inverse relationship between resistance and distance, when the distance between the target object and the bronchial wall is less than a first threshold; the first threshold is determined based on the resistance level.

[0139] In some embodiments, the first resistance feedback subunit includes: a third resistance feedback subunit, configured to maintain the resistance on the target object manipulator at a first resistance value when the distance between the target object and the bronchial wall is less than a second threshold, wherein the second threshold is less than the first threshold, and each resistance value corresponding to the distance between the first threshold and the second threshold is less than the first resistance value; the second threshold is determined according to the resistance level.

[0140] In some embodiments, the feedback unit further includes: a vibration feedback subunit, used to control the target object manipulator to vibrate when the distance between the target object and the bronchial wall is less than a third threshold; the third threshold is less than or equal to the second threshold, and the third threshold is determined according to the resistance level.

[0141] This specification provides a path planning device for bronchial surgery, including a first acquisition unit, a first determination unit, and a second determination unit.

[0142] The first acquisition unit is used to acquire a three-dimensional model of the lung bronchi. Each bronchial segment in the three-dimensional model has a unique identifier. The model also includes a topology map, which contains the unique identifiers of each bronchial segment and the connection relationships between the segments represented by those identifiers. The first determination unit is used to determine the unique identifier of the target bronchial segment closest to the target lung nodule. The second determination unit is used to determine, based on the topology map, the unique identifiers of each bronchial segment sequentially traversed by the path from the main trachea to the target bronchus, thus obtaining the planned path represented by the unique identifier of each bronchial segment.

[0143] In some embodiments, the apparatus further includes a skeleton extraction unit and a third determination unit.

[0144] The skeleton extraction unit is used to replace the bronchi with the centerlines of each bronchial segment in the three-dimensional model of the lung bronchi to obtain a skeleton model of the lung bronchi. The third determination unit is used to determine the centerlines of the bronchi corresponding to the unique identifiers of each bronchial segment in the skeleton model of the lung bronchi after obtaining the planned path represented by the unique identifiers of each bronchial segment, thereby obtaining the planned path composed of the centerlines of each bronchial segment.

[0145] This specification provides a device for establishing a lung bronchus model, including a third acquisition unit, an identification unit, and a reconstruction unit.

[0146] The third acquisition unit is used to acquire multi-slice scan images of the lungs. The recognition unit is used to input the multi-slice scan images of the lungs into a pre-trained network model to obtain multi-slice scan images labeled with unique identifiers for each bronchial segment. The reconstruction unit is used to reconstruct three-dimensional images of the lung bronchi from the multi-slice scan images labeled with unique identifiers for each bronchial segment, wherein the three-dimensional images are configured with a topological map.

[0147] In some embodiments, before inputting the multi-layer scan images of the lungs into a pre-trained network model, the method further includes: training the network model by: acquiring multiple training data, one of which includes multi-layer scan images of the lungs, each layer of the scan images being labeled with a unique identifier for each bronchus segment; the unique identifier for each bronchus carrying bronchial level information; and training the network model using the multiple training data, wherein the multi-layer scan images in each training data are used as input to the network model, and the unique identifiers of each bronchus segment labeled in each layer of the image are used as output to the network model.

[0148] In some embodiments, the unique identifier of each bronchus segment in each layer of scanned images in the training data is marked by the following method: identifying each bronchus segment and assigning a unique identifier to each bronchus segment; performing the following operation for each bronchus segment: identifying multiple pixels on the scanned image that make up the current bronchus, and marking each of the multiple pixels as the unique identifier of the current bronchus.

[0149] In some embodiments, during annotation, each lung nodule is identified and a unique identifier is assigned to each lung nodule; for each lung nodule, the following operations are performed: multiple pixels on the scanned image that make up the current lung nodule are identified, and each of the multiple pixels is marked as a unique identifier for the current lung nodule.

[0150] The descriptions and beneficial effects of the above-mentioned devices can be found in the descriptions and beneficial effects of the corresponding methods, and will not be repeated here.

[0151] The information feedback method provided in this manual can be used in bronchial surgery, and surgical robot systems used to perform bronchial surgical procedures, such as... Figure 15 and Figure 16 As shown, it includes a surgical robot 100, a manipulator 200, a controller 300 (not shown in the figure), and a presentation device 400.

[0152] The surgical robot 100 includes a base 110, a robotic arm 120, a track 130, a power unit 140, and a catheter 150. The first end of the robotic arm 120 is mounted on the base 110, and the second end of the robotic arm 120 is suspended. The track 130 is mounted on the second end of the robotic arm 120. The power unit 140 is mounted on the track 130 and can move along the extension direction of the track 130. The first end of the catheter 150 is mounted on the power unit 140, and the second end can extend into the patient's bronchus. The second end of the catheter 150 is equipped with an image acquisition component or surgical instruments.

[0153] The manipulator 200 is used by the surgeon to manipulate the surgical robot 100 so that the second end of the catheter 150 moves within the bronchus of the lung along a designated path. This designated path may be determined using the path planning method for bronchial surgery provided in this instruction manual.

[0154] The controller 300 is used to acquire a three-dimensional model of the lung bronchi. Each bronchus segment in the three-dimensional model of the lung bronchi has a unique identifier. The three-dimensional model of the lung bronchi is also configured with a topology map, which includes the unique identifier of each bronchus segment and the connection relationship between the bronchus segments represented by the unique identifier. Based on the three-dimensional model of the lung bronchi and the topology map, the unique identifier of the bronchus segment where the target object is located and the unique identifier of the next bronchus segment are acquired in real time.

[0155] The presentation device 400 is used to provide surgical information to the physician in at least one of the following ways: presenting the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through an image; or playing the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through voice.

[0156] This invention also provides an electronic device, such as... Figure 17 As shown, the electronic device may include a processor 1701 and a memory 1702, wherein the processor 1701 and the memory 1702 may be connected via a bus or other means. Figure 17 Taking the example of a connection between China and Israel via a bus.

[0157] Processor 1701 can be a Central Processing Unit (CPU). Processor 1701 can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0158] The memory 1702, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the acetabular cup prosthesis coverage prediction method in this embodiment of the invention. The processor 1701 executes various processor functions and data classification by running the non-transitory software programs, instructions, and modules stored in the memory 1702, thereby implementing any of the following methods in the above method embodiments: information feedback method, bronchial surgery path planning method, and lung bronchial model establishment method.

[0159] Memory 1702 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by processor 1701, etc. Furthermore, memory 1702 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 1702 may optionally include memory remotely located relative to processor 1701, and these remote memories may be connected to processor 1701 via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0160] The one or more modules are stored in the memory 1702. When executed by the processor 1701, they execute any of the following methods in the above method embodiments: information feedback method, bronchial surgery path planning method, and lung bronchial model establishment method.

[0161] For details regarding the electronic device described above, please refer to the relevant descriptions and effects in the corresponding method embodiments; they will not be repeated here.

[0162] This specification provides a computer storage medium storing computer program instructions that, when executed by a processor, implement the steps of any of the following methods in the above-described method embodiments: an information feedback method, a bronchial surgery path planning method, and a method for establishing a lung bronchus model.

[0163] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0164] The various embodiments in this specification are described in a progressive manner. For the same or similar parts between the various embodiments, please refer to each other. The focus of each embodiment is to describe the differences from other embodiments.

[0165] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions.

[0166] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0167] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute certain parts of the methods of various embodiments of this application.

[0168] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0169] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0170] Although this application has been described through embodiments, those skilled in the art will know that this application has many modifications and variations without departing from the spirit of this application, and it is intended that the appended claims cover such modifications and variations without departing from the spirit of this application.

Claims

1. An information feedback device, characterized in that, include: The first acquisition unit is used to acquire a three-dimensional model of the lung bronchi. Each bronchus segment in the three-dimensional model of the lung bronchi has a unique identifier. The three-dimensional model of the lung bronchi is also configured with a topology map, which includes the unique identifier of each bronchus segment and the connection relationship between the bronchus segments represented by the unique identifier. The second acquisition unit is used to acquire, in real time, the unique identifier of the bronchial segment where the target object is located and the unique identifier of the next bronchial segment based on the three-dimensional model and topology map of the lung bronchi. The feedback unit is used to provide feedback information in at least one of the following ways so that the surgeon can intuitively perceive the global position of the target object in the three-dimensional model of the lung bronchi: presenting the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through an image; or playing the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through voice. The unique identifier of the bronchial segment contains the level information of the bronchial segment, where the next-level bronchial segment is a branch of the previous-level bronchial segment, and the bronchial segments of different levels have different thicknesses. The feedback unit further includes: The determination subunit is used to determine the resistance level based on the unique identifier of the bronchial segment where the target object is located, and different resistance levels are set for bronchial segments of different levels; the resistance level is used to represent the range of resistance values ​​fed back to the target object manipulator; The first resistance feedback subunit is used to control the resistance value fed back to the target manipulator according to the resistance level, so that the surgeon can feel that the target object has entered different bronchial segments based on the resistance value of the target manipulator.

2. The apparatus according to claim 1, characterized in that, The feedback unit includes: The first presentation subunit is used to present a two-dimensional top view image of the bifurcation point at the front end of the bronchial segment where the target object is located. The two-dimensional top view image contains a unique identifier for each bronchial segment associated with the bifurcation point. And / or, The second presentation subunit is used to present a virtual image inside the bronchial segment where the target object is located, wherein the virtual image inside the bronchial segment contains a unique identifier for each bronchial segment associated with the bifurcation.

3. The apparatus according to claim 1, characterized in that, The first resistance feedback subunit includes: The acquisition sub-unit is used to obtain the distance between the target object and the bronchial wall; The second resistance feedback subunit is used to adjust the resistance fed back to the target object manipulator within the range corresponding to the resistance level, according to the inverse relationship between resistance and distance, when the distance between the target object and the bronchial wall is less than the first threshold; the first threshold is determined based on the resistance level.

4. The apparatus according to claim 3, characterized in that, The first resistance feedback subunit includes: The third resistance feedback subunit is used to maintain the resistance on the target object manipulator at a first resistance value when the distance between the target object and the bronchial wall is less than a second threshold. The second threshold is less than the first threshold, and each resistance value corresponding to the distance between the first threshold and the second threshold is less than the first resistance value. The second threshold is determined according to the resistance level.

5. The apparatus according to claim 4, characterized in that, The feedback unit further includes: A vibration feedback subunit is used to control the vibration of the target object manipulator when the distance between the target object and the bronchial wall is less than a third threshold; the third threshold is less than or equal to the second threshold, and the third threshold is determined according to the resistance level.

6. A path planning device, characterized in that, include: The first acquisition unit is used to acquire a three-dimensional model of the lung bronchi. Each bronchus segment in the three-dimensional model of the lung bronchi has a unique identifier. The three-dimensional model of the lung bronchi is also configured with a topology map. The topology map includes the unique identifier of each bronchus segment and the connection relationship between the bronchus segments represented by the unique identifier. The unique identifier of the bronchus segment contains the level information of the bronchus segment, wherein the next level bronchus segment is a branch of the previous level bronchus segment, and the bronchus segments of different levels have different thicknesses. The first identifying unit is used to determine the unique identifier of the target bronchial segment closest to the target lung nodule; The second determining unit is used to determine, according to the topology map, the unique identifiers of each bronchial segment that the path from the main trachea to the target bronchus passes through in sequence, and obtain the planned path represented by the unique identifier of each bronchial segment; the planned path is the basis for the surgeon to control the movement of the target object in the bronchus. When the surgeon controls the movement of the target subject within the bronchus, the following feedback method is implemented: Based on the three-dimensional model and topology of the lung bronchi, the unique identifier of the bronchial segment where the target object is located and the unique identifier of the next bronchial segment are obtained in real time. Information can be fed back in at least one of the following ways so that the surgeon can intuitively feel the global position of the target object in the three-dimensional model of the lung bronchi: presenting the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next level bronchial segment in the planned path through images; or playing the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next level bronchial segment in the planned path through voice. The feedback methods also include: determining the resistance level based on the unique identifier of the bronchial segment where the target object is located, and setting different resistance levels for bronchial segments of different levels; the resistance level is used to represent the range of resistance values ​​fed back to the target object manipulator; and controlling the resistance value fed back to the target object manipulator according to the resistance level, so that the surgeon can sense that the target object has entered different bronchial segments based on the resistance value on the target object manipulator.

7. The apparatus according to claim 6, characterized in that, The device further includes: The skeleton extraction unit is used to replace the bronchus with the centerline of each bronchus segment in the three-dimensional model of the lung bronchus to obtain the skeleton model of the lung bronchus. The third determining unit is used to determine the centerline of the bronchus corresponding to the unique identifier of each bronchial segment in the skeletal model of the lung bronchus after obtaining the planned path represented by the unique identifier of each bronchial segment, so as to obtain the planned path composed of the centerlines of each bronchial segment.

8. A device for establishing a lung bronchus model, characterized in that, include: The third acquisition unit is used to acquire multi-slice scan images of the lungs; The identification unit is used to input the multi-layer scan image of the lungs into a pre-trained network model to obtain a multi-layer scan image labeled with a unique identifier for each bronchial segment. The reconstruction unit is used to reconstruct a three-dimensional image of the lung bronchi from multi-slice scan images labeled with unique identifiers for each bronchial segment, wherein the three-dimensional image is configured with a topological map. The lung bronchial model is used by surgeons to execute the following feedback method when moving within the bronchi according to the planned path: A three-dimensional model of the lung bronchi is obtained. Each bronchus segment in the three-dimensional model of the lung bronchi has a unique identifier. The three-dimensional model of the lung bronchi is also configured with a topology map, which includes the unique identifier of each bronchus segment and the connection relationship between the bronchus segments represented by the unique identifier. Based on the three-dimensional model and topology of the lung bronchi, the unique identifier of the bronchial segment where the target object is located and the unique identifier of the next bronchial segment are obtained in real time. Information can be fed back in at least one of the following ways to enable surgeons to intuitively perceive the global location of the target object in the 3D model of the lung bronchi: presenting the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through images; or playing the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through voice. The unique identifier of the bronchial segment contains the level information of the bronchial segment, where the next-level bronchial segment is a branch of the previous-level bronchial segment, and the bronchial segments of different levels have different thicknesses. The feedback methods also include: determining the resistance level based on the unique identifier of the bronchial segment where the target object is located, and setting different resistance levels for bronchial segments of different levels; the resistance level is used to represent the range of resistance values ​​fed back to the target object manipulator; and controlling the resistance value fed back to the target object manipulator according to the resistance level, so that the surgeon can sense that the target object has entered different bronchial segments based on the resistance value on the target object manipulator.

9. The apparatus according to claim 8, characterized in that, Before inputting the multi-layer scan images of the lungs into the pre-trained network model, the method further includes training the network model according to the following method: Multiple training data sets are acquired, one of which includes multi-layer scan images of the lungs, with each layer of scan image labeled with a unique identifier for each bronchus segment; the unique identifier of the bronchus carries bronchial level information. The network model is trained using the multiple training data, wherein the multi-layer scan images in each training data are used as the input of the network model, and the unique identifier of each bronchial segment marked in each layer image is used as the output of the network model.

10. The apparatus according to claim 8, characterized in that, The unique identifiers of each bronchus segment in each layer of the scanned image in the training data are labeled by the following method: each bronchus segment is identified and a unique identifier is assigned to each bronchus segment; for each bronchus segment, the following operation is performed: multiple pixels that make up the current bronchus on the scanned image are identified, and each of the multiple pixels is marked as the unique identifier of the current bronchus.

11. The apparatus according to claim 8, characterized in that, During annotation, each lung nodule is identified and assigned a unique identifier. For each lung nodule, the following operations are performed: multiple pixels that make up the current lung nodule on the scanned image are identified, and each of the multiple pixels is marked as a unique identifier for the current lung nodule.

12. A bronchial surgical system, characterized in that, include: Surgical robots include a base, robotic arms, tracks, a power unit, and catheters; The first end of the robotic arm is mounted on the base, and the second end is suspended in the air; The track is located at the second end of the robotic arm; the power box is located on the track and can move along the extension direction of the track; the first end of the catheter is located on the power box, and the second end can extend into the patient's bronchus; the second end of the catheter is equipped with an image acquisition component or instrument. Manipulator, used to control the surgical robot so that the second end of the catheter moves along a specified path in the bronchus of the lung; The controller is used to acquire a three-dimensional model of the lung bronchi. Each bronchus segment in the three-dimensional model of the lung bronchi has a unique identifier. The three-dimensional model of the lung bronchi is also configured with a topology map, which includes the unique identifier of each bronchus segment and the connection relationship between the bronchus segments represented by the unique identifier. Based on the three-dimensional model of the lung bronchi and the topology map, the controller can acquire in real time the unique identifier of the bronchus segment where the target object is located and the unique identifier of the next bronchus segment. The presentation device is used to provide feedback to the surgeon in at least one of the following ways, so that the surgeon can intuitively perceive the global position of the target object in the three-dimensional model of the lung bronchi: presenting the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through an image; or playing the unique identifier of the bronchial segment where the target object is located and / or the unique identifier of the next-level bronchial segment in the planned path through voice; the unique identifier of the bronchial segment contains the level information of the bronchial segment, wherein the next-level bronchial segment is a branch of the previous-level bronchial segment, and the bronchial segments of different levels have different thicknesses; The feedback methods also include: determining the resistance level based on the unique identifier of the bronchial segment where the target object is located, and setting different resistance levels for bronchial segments of different levels; the resistance level is used to represent the range of resistance values ​​fed back to the target object manipulator; and controlling the resistance value fed back to the target object manipulator according to the resistance level, so that the surgeon can sense that the target object has entered different bronchial segments based on the resistance value on the target object manipulator.

13. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, which, when executed by a processor, perform the functions of the apparatus according to any one of claims 1 to 11.

Images