A bronchoscope robot autonomous navigation method and system based on a transformer behavior cloning

Abstract

The application provides a bronchoscope robot autonomous navigation method and system based on a Transformer behavior cloning, comprising: acquiring multi-modal state data of a bronchoscope robot running in an airway; converting the multi-modal state data into a state vector; inputting a time sequence formed by the state vectors of the most recent continuous multiple time steps into a Transformer behavior cloning model to obtain a three-degree-of-freedom control vector output by the Transformer behavior cloning model; and controlling the robot to run in the airway based on the three-degree-of-freedom control vector. The application uses a Transformer behavior cloning model to learn the action of an expert control robot, and realizes autonomous navigation control of the bronchoscope robot.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an autonomous navigation method and system for a bronchoscopic robot based on Transformer behavior cloning. Background Technology

[0002] The diagnosis and treatment of respiratory diseases such as lung tumors often require bronchoscopy to access deep into the patient's lungs for sampling or treatment. Traditional bronchoscopy relies entirely on manual manipulation by the surgeon. Due to the complex structure and numerous branches of the human bronchial tree, navigating within the narrow airways is extremely challenging for the surgeon. This operating mode is not only physically demanding but also prone to causing surgeon fatigue. It also heavily depends on the surgeon's personal experience and spatial reasoning, resulting in a long learning curve and limiting the efficiency and widespread adoption of complex surgeries.

[0003] To assist surgeons in performing operations, existing electromagnetic navigation bronchoscopy technology and robot-assisted surgical systems have emerged. These technologies typically utilize preoperative chest computed tomography (CT) data to construct a three-dimensional virtual airway map and plan the path to the lesion. During the procedure, the system tracks the position of the bronchoscope in real time using electromagnetic sensors and displays navigation path guidance on the screen, or uses a robotic system to provide positioning feedback, thereby helping surgeons determine the relationship between the current position and the target path, improving the accuracy of reaching peripheral lung lesions.

[0004] However, the existing navigation systems mentioned above primarily provide path guidance, visual assistance, and positioning feedback. The actual movement of the bronchoscope within the airway still requires manual operation of the handle or joystick by the physician. This approach does not change the fundamental "human decision-making, human execution" model. During prolonged surgeries, the vibrations, fatigue, and delayed response to navigation information caused by manual operation can still lead to the accumulation of navigation errors, and true automated control cannot be achieved. Therefore, how to achieve autonomous navigation control of bronchoscope robots has become an urgent problem to be solved in this field. Summary of the Invention

[0005] This invention provides an autonomous navigation method and system for a bronchoscopy robot based on Transformer behavior cloning, which solves the technical problem of how to achieve autonomous navigation control of a bronchoscopy robot.

[0006] This invention provides an autonomous navigation method for a bronchoscopy robot based on Transformer behavior cloning, comprising: Acquire multimodal state data of the bronchoscopic robot during its operation within the airway; The multimodal state data is converted into a state vector; Input the time sequence of the state vectors from the most recent consecutive time steps into the Transformer behavioral cloning model to obtain the three-degree-of-freedom control vector output by the Transformer behavioral cloning model. The robot is controlled to operate within the airway based on the three-degree-of-freedom control vector; The training set of the Transformer behavior clone model includes samples of the time sequence and corresponding labels, wherein the labels include the robot's actual actions.

[0007] According to the present invention, an autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning is provided, wherein the multimodal state data includes endoscope images, pose data of the bronchoscope tip, joint angles of the robot, and path deviation vector. The path deviation vector is the deviation vector between the current coordinates of the bronchoscope tip and the target coordinate point. The target coordinate point is the next coordinate point in the navigation reference path that is closest to the current coordinates. The starting point of the navigation reference path is the entrance of the airway, and the ending point is the location of the lesion.

[0008] According to the present invention, an autonomous navigation method for a bronchoscopy robot based on Transformer behavior cloning is provided, wherein obtaining the path deviation vector includes: Establish a three-dimensional model of the airway; Based on the three-dimensional model, path planning is performed between the airway inlet and the location of the lesion to obtain the navigation reference path; Determine the target coordinate point in the navigation reference path; The path deviation vector is determined based on the current coordinates and the target coordinate point.

[0009] According to the present invention, an autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning is provided, wherein the step of converting the multimodal state data into a state vector includes: Feature extraction is performed on the endoscopic image to obtain a visual feature vector; The visual feature vector, the pose data, the joint angle, and the path deviation vector are concatenated to obtain the state vector. The state vector is then encoded using time-position encoding.

[0010] According to the present invention, an autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning is provided, wherein the tag further includes a path direction truth vector, and the path direction truth vector is a unit vector of the current coordinates toward the target coordinate point; Before inputting the time-series sequence of the state vectors from the most recent consecutive time steps into the Transformer behavioral cloning model to obtain the three-degree-of-freedom control vector output by the Transformer behavioral cloning model, the method further includes: The time series samples are input into the Transformer behavioral cloning model to obtain the control prediction vector and path direction prediction vector output by the Transformer behavioral cloning model. A first loss is determined based on the true action and the control prediction vector corresponding to the samples of each of the time series, and a second loss is determined based on the path direction truth vector and the path direction prediction vector corresponding to the samples of each of the time series. The parameters of the Transformer behavior clone model are updated based on the first loss and the second loss.

[0011] According to the present invention, an autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning is provided, wherein the endoscopic image is an image that conforms to the style of a real endoscope; Before inputting the time-series sequence of the state vectors from the most recent consecutive time steps into the Transformer behavioral cloning model to obtain the three-degree-of-freedom control vector output by the Transformer behavioral cloning model, the method further includes: After data augmentation and random perturbation of the training set, the Transformer behavior clone model is augmented and trained using the training set.

[0012] This invention also provides an autonomous navigation system for a bronchoscopy robot based on Transformer behavior cloning, comprising: The acquisition module is used to acquire multimodal state data of the bronchoscopic robot during its operation within the airway; The conversion module is used to convert the multimodal state data into a state vector; The input module is used to input the time sequence composed of the state vectors of the most recent consecutive time steps into the Transformer behavioral clone model to obtain the three-degree-of-freedom control vector output by the Transformer behavioral clone model. The control module is used to control the robot to operate within the airway based on the three-degree-of-freedom control vector; The training set of the Transformer behavior clone model includes samples of the time sequence and corresponding labels, wherein the labels include the robot's actual actions.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning as described above.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the autonomous navigation method for a bronchoscope robot based on Transformer behavior cloning as described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the autonomous navigation method for a bronchoscope robot based on Transformer behavior cloning as described above.

[0016] The present invention provides an autonomous navigation method and system for a bronchoscope robot based on Transformer behavior cloning. By acquiring multimodal state data and using the Transformer behavior cloning model for decision-making, it can leverage the advantages of Transformer in modeling long sequence data to output a three-degree-of-freedom control vector to directly drive the robot, realizing autonomous navigation of the bronchoscope without the need for continuous manual operation by the doctor. This reduces the doctor's workload and fatigue, and solves the problems of low efficiency and error-proneness caused by reliance on manual operation in existing technologies. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning provided by the present invention.

[0019] Figure 2 This is a schematic diagram of the structure of the bronchoscopic robot autonomous navigation system based on Transformer behavior cloning provided by the present invention.

[0020] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0022] The following is combined with Figures 1 to 3 This invention describes the autonomous navigation method and system for a bronchoscopic robot based on Transformer behavior cloning.

[0023] Figure 1 This is a flowchart illustrating the autonomous navigation method for a bronchoscopy robot based on Transformer behavior cloning provided by the present invention, as shown below. Figure 1 As shown, the method includes, but is not limited to, steps S1, S2, S3 and S4.

[0024] Step S1: Obtain multimodal state data of the bronchoscope robot during operation in the airway.

[0025] Multimodal state data is a comprehensive dataset that integrates the robot's visual, kinematic, and path planning information within the airway. The multimodal state data at time t can be denoted as... Multimodal state data is used to provide comprehensive, real-time environmental perception and self-state information for the robot's decision-making system. Multimodal state data can be collected through various sensors.

[0026] Step S2: Convert the multimodal state data into a state vector.

[0027] A state vector is a unified mathematical representation formed by concatenating features and numerical processing of multimodal state data. The state vector at time t can be denoted as: Transforming multimodal state data into state vectors can unify heterogeneous multi-source sensor data into a mathematical format that can be processed by deep learning models, thereby achieving feature fusion.

[0028] Step S3: Input the time sequence consisting of the state vectors of the most recent consecutive time steps into the Transformer behavior clone model to obtain the three-degree-of-freedom control vector output by the Transformer behavior clone model. The training set of the Transformer behavior clone model includes time sequence samples and corresponding labels, where the labels include the robot's actual actions.

[0029] A time series is a set of state vectors of a fixed length (e.g., L time steps) that trace back from the current time step, and can be denoted as... .

[0030] The Transformer behavioral cloning model is a deep neural network based on the self-attention mechanism. It consists of stacked multi-layer self-attention encoders, each of which performs the following transformation on the temporal sequence of the state vector: First, calculate the self-attention weights and then apply the output from the previous layer. Projection yields the query matrix Key matrix Sum matrix And calculate attention output. T represents the matrix transpose operation, and d represents the dimension of each column of features in the key matrix. Then, a feedforward network is used to... Perform nonlinear transformation and residual connection to obtain Multi-head attention performs the above process in parallel for multiple sets to broaden the feature dimensions the model focuses on, and then merges them. After multiple layers of encoding, the output is a sequence of hidden representations of the same length as the temporal sequence of the state vector. Take the last moment As a representation of the current state that incorporates historical information. Finally, for Perform a fully connected mapping to predict the three-degree-of-freedom control vector at the current time step. This includes the control of the bronchoscope's bending, steering, and propulsion.

[0031] By leveraging the powerful sequence modeling capabilities of Transformer, the best action to be performed can be predicted based on historical observation information, thus achieving a mapping from perception to decision-making.

[0032] The training set includes samples of state vector time series and labels of the robot's real actions. The robot's real actions are those of the robot under the manual control of the expert. The model has learned the expert's operating strategy when facing a specific airway environment through supervised learning, and has the ability to clone behavior and imitate the expert's operating strategy.

[0033] Step S4: The robot operates within the airway using three-degree-of-freedom control vector control.

[0034] Based on the three-degree-of-freedom control vector The target values ​​corresponding to each motor are calculated separately, and then the control commands are sent to each drive motor of the bronchoscope robot. After the motors execute the commands, the bronchoscope's posture changes. During this process, the robot continuously compares its own position with the navigation reference path. When a slight deviation occurs, the robot corrects its direction through control fine-tuning output from the Transformer policy network, ensuring that the bronchoscope always moves towards the next navigation coordinate point. If unforeseen circumstances are encountered (such as low image quality due to mucus obstruction, loss of pose data, abnormal predictive control, etc.), the robot can slow down or pause, and alert the physician to take over control to ensure surgical safety.

[0035] As described above, this invention acquires multimodal state data and uses the Transformer behavior cloning model for decision-making. It can leverage the advantages of Transformer in modeling long sequence data to output a three-degree-of-freedom control vector to directly drive the robot, thus realizing autonomous navigation control of the bronchoscope. This eliminates the need for continuous manual operation by doctors, thereby reducing their workload and fatigue. It also solves the problems of low efficiency and error-proneness caused by reliance on manual operation in existing technologies.

[0036] In one embodiment, the multimodal state data of the present invention may include endoscopic images, bronchoscope tip pose data, robot joint angles, and path deviation vectors. The path deviation vector is the deviation vector between the current coordinates of the bronchoscope tip and the target coordinate point. The target coordinate point is the next coordinate point in the navigation reference path that is closest to the current coordinates. The starting point of the navigation reference path is the airway entrance, and the ending point is the location of the lesion.

[0037] The endoscopic image is an image of the airway interior. Pose data may include the position coordinates and Euler angles of the bronchoscope tip in the patient's thoracic coordinate system. Joint angles are the current rotation angles of the robot's joint motors. The path deviation vector is a geometric vector representing the deviation of the bronchoscope tip's current coordinates from the navigation reference path.

[0038] Real-time video frames within the bronchial lumen can be captured via an endoscopic camera, typically at a frame rate of approximately 30fps, yielding endoscopic images. The pose data of the bronchoscope tip can be obtained using an electromagnetic sensor (6-DoF sensor) mounted at the end of the bronchoscope. This pose data is then converted to the patient's thoracic coordinate system through pre-registration in the CT coordinate system, providing a unified benchmark for subsequently associating the lens position with the navigation reference path. The joint angles of each drive motor can be obtained using joint sensors on the robot body.

[0039] The navigation reference path is represented by a series of continuous three-dimensional coordinate points, i.e. ,in It is the entrance to the airway. This indicates the location of the lesion. Each pair of adjacent points... arrive This constitutes a segment of the path. Path deviation vector. It can be represented as , For the target coordinate point, The current coordinates of the bronchoscope tip are shown. The path deviation vector reflects both the navigation progress (how far along the path) and the approximate direction of the next turn. It provides a clear local guidance target for autonomous navigation, enabling the robot not only to maintain its movement style through imitation learning but also to constantly perceive its relative position with respect to the globally optimal path, thereby ensuring continuous advancement towards the lesion.

[0040] Then the multimodal state data at time t can be represented as: ,in This is an endoscopic image. For pose data, For joint angle, This is the path deviation vector.

[0041] This invention incorporates a well-defined path deviation vector as part of the state input, integrating prior knowledge of global path planning into the decision input of the behavior cloning model. This allows the robot to not only rely on visual texture to determine direction during autonomous navigation, but also to make corrections based on geometric position deviations. This significantly improves the accuracy and robustness of navigation in complex branching structures and reduces the risk of getting lost or entering the wrong branch.

[0042] In one embodiment, the method of obtaining the path deviation vector may further include: Establish a three-dimensional model of the airway; Based on the 3D model, path planning is performed between the airway entrance and the location of the lesion to obtain a navigation reference path; Determine the target coordinates in the navigation reference path; The path deviation vector is determined based on the current coordinates and the target coordinates.

[0043] Before surgery, a three-dimensional bronchial tree model of the lungs can be created using the patient's chest CT images, and the airway centerline can be extracted to form the main airway topology. After the surgeon selects the location of the lesion on the 3D model, the shortest or optimal path from the airway inlet to the lesion location is automatically calculated based on the 3D model, i.e., the navigation reference path. Path planning can be based on Dijkstra's algorithm or A... The algorithm searches for the optimal path on the grid model of the airway tree, and the cost function can take into account factors such as path length and curvature to improve navigation safety.

[0044] Next, the distances between the current coordinates of the bronchoscope tip and all coordinate points in the navigation reference path are calculated, and the coordinate point P with the smallest distance is found. i The next coordinate point P is determined by incrementing the index by one. i+1 As the target coordinate point.

[0045] Finally, the path deviation vector from the current coordinates of the bronchoscope tip to the target coordinates is calculated using vector subtraction.

[0046] This invention ensures the medical rationality and accuracy of the navigation reference path through preoperative CT reconstruction and path planning. This path deviation vector, generated based on global map planning, guides the robot to consistently follow the optimal path to the lesion within complex bronchial trees, thus improving the efficiency of autonomous surgical navigation.

[0047] In one embodiment, step S2 may further include: Feature extraction is performed on the endoscopic images to obtain visual feature vectors; The visual feature vector, pose data, joint angles, and path deviation vector are concatenated to obtain the state vector. Perform temporal position encoding on the state vector.

[0048] Convolutional neural networks can be used to process endoscopic images. Feature extraction is performed to obtain visual feature vectors. This compresses high-dimensional pixel information into a low-dimensional feature representation that includes key information such as airway texture, lighting, and structure. This visual feature vector... With pose data Joint angle Path deviation vector The features are concatenated to form a fixed-length real-valued vector, i.e., the state vector. This allows for the construction of a full-state description that includes both environmental visual information and robot motion information.

[0049] Each Each sequence is encoded with a corresponding time position code to preserve sequence continuity. Sine wave position coding can be used for each sequence. Add corresponding temporal position encoding. Since the Transformer model's self-attention mechanism itself does not have the ability to process sequence order, temporal position encoding allows the model to distinguish between past and present states, thereby correctly understanding the causal relationship of actions.

[0050] This invention constructs a high-quality state representation suitable for Transformer processing through feature extraction, multimodal concatenation, and temporal position encoding. In particular, the introduction of temporal position encoding enables the model to effectively capture the temporal logic of bronchoscopy operations (e.g., adjusting the angle first, then advancing), thereby more accurately predicting coherent and smooth control actions.

[0051] In one embodiment, the sample label of the present invention may include the robot's actual action and path direction truth vector, wherein the path direction truth vector is a unit vector of the current coordinates toward the target coordinate point; Before step S3, the method of the present invention may further include: Input the time series samples into the Transformer behavioral clone model to obtain the control prediction vector and path direction prediction vector output by the Transformer behavioral clone model; The first loss is determined based on the true action and control prediction vectors corresponding to the samples of each time series, and the second loss is determined based on the true path direction vector and the predicted path direction vector corresponding to the samples of each time series. The parameters of the Transformer behavioral clone model are updated based on the first loss and the second loss.

[0052] Before using the Transformer Behavioral Cloning Model to predict three-degree-of-freedom control vectors, the model needs to be trained using a training set. The training set for the Transformer Behavioral Cloning Model primarily consists of multimodal teaching samples collected by physicians operating on a real bronchoscopic robot platform within a bronchial model. During training set construction, physicians control the movement of the bronchoscopic robot within the bronchial model (including forward / backward movement, bending, and turning) using a joystick, simultaneously acquiring endoscopic images, pose data, joint angles, and path deviation vectors. These data are then aligned using timestamps to obtain samples of multimodal state data. '. Samples of multimodal state data' 'Samples transformed into state vectors' ', combines samples from multiple most recent consecutive state vectors into a sample of the state vector time series sequence. .

[0053] The Transformer behavioral cloning model of this invention has a dual-branch output head structure, where the main branch outputs a three-degree-of-freedom control prediction vector through a fully connected layer. ', used to estimate the doctor's condition The following operations; the auxiliary branch outputs the path direction prediction vector through the fully connected layer. The auxiliary output is used to estimate the direction from the current position to the next navigation reference point. It provides an explicit prediction of the navigation target, helping the network maintain focus on the path planning objective while learning and imitating control, thus reducing cumulative deviations in long-range navigation.

[0054] During model training, this invention can optimize network parameters by using a combination of supervised learning-based behavioral cloning loss and auxiliary task loss. The behavioral cloning loss is the first loss, and the auxiliary task loss is the second loss.

[0055] Define the real actions demonstrated by the expert as The truth vector of the path direction is , Behavioral cloning loss For control prediction vectors 'Real Actions with Experts' The mean squared error (L2 norm loss) of '': ,in Let be the total number of samples. The first loss directly measures the consistency between the robot's actions and those of human experts, and is also the main optimization objective of imitation learning.

[0056] Path direction auxiliary loss It can be: That is, it requires the model to predict the direction of the next path. 'As close as possible to the actual direction obtained from the navigation reference path' By minimizing and The difference can enhance the sensitivity of the implicit representation within the model to the navigation direction, thus making it more inclined to adjust towards the target direction when controlling the output.

[0057] The total training loss is the weighted sum of primary and secondary losses: .in This is a weighting factor used to adjust the weight of auxiliary tasks relative to the main task. A balance between the decrease in loss of both can be achieved through experimentation on the validation set.

[0058] The parameters of the Transformer network can be iteratively updated using the Adam optimization algorithm by minimizing the total loss mentioned above until performance converges on the validation set. It should be noted that the loss in this invention is not limited to mean squared error loss; the auxiliary loss can be replaced with cosine similarity loss or other forms as needed to directly maximize the cosine value of the angle between the predicted direction and the true direction.

[0059] Traditional reinforcement learning methods indirectly guide the policy towards the center path through reward functions, such as designing a penalty term for deviation from the centerline to encourage the robot to stay centered. This invention, however, introduces explicit reference path direction supervision, directly constraining the model's output towards the predetermined path during the learning process. This largely avoids the yaw accumulation error that can occur in pure behavioral cloning due to a lack of long-term planning. This multi-objective learning paradigm effectively combines the requirements of imitating expert behavior and following the planned path, enabling the trained policy to both realistically reproduce the expert's operating style and ensure that it moves along the optimal path planned beforehand.

[0060] In one embodiment, the endoscopic image of the present invention can be an image that conforms to the style of a real endoscope; Before step S3, the method of the present invention may further include: After data augmentation and random perturbation of the training set, the Transformer behavioral clone model is augmented and trained using the training set.

[0061] Training samples can be perturbed with pose, brightness, texture, and noise injection to improve the model's robustness to changes in the clinical environment. For example, the pose angle of the endoscope camera can be randomly adjusted (e.g., randomly rotated around the axis of travel at different angles); the brightness of the endoscope light source can be randomly changed; small random noise can be added to the expert demonstration when the model is near the airway wall; and the texture details and tone of the images can be randomly altered. These measures can effectively broaden the range of image distributions seen by the model, prompting it to learn insensitivity to non-critical visual changes.

[0062] This invention enhances the model's generalization ability and robustness through data augmentation and random perturbation during training. This enables the model trained using this method to maintain stable control performance even in real surgical scenarios with different patient airway characteristics, varying lighting conditions, or sensor noise, effectively solving the problem of performance degradation of deep learning models in practical deployment.

[0063] In summary, this invention addresses the problems of high dependence on manual operation, long learning curves, and heavy reliance on physician experience in existing bronchoscopy navigation systems. By integrating preoperative three-dimensional path planning, intraoperative multimodal data acquisition, and sequence decision model training, it achieves autonomous navigation control of the bronchoscope within the lung airways, reducing the physician's workload while improving the accuracy and reliability of navigation. This invention leverages the powerful temporal modeling capabilities of Transformer to learn expert control strategies, and combines this with auxiliary path direction prediction to improve the stability and accuracy of travel along the centerline. Simultaneously, through data augmentation and random perturbation, the generalization performance of the strategy across different patient anatomy and imaging conditions is significantly enhanced. This enables automated navigation control of the bronchoscope while reducing the workload of physicians, which is of great significance for improving the efficiency and safety of minimally invasive pulmonary diagnosis and treatment.

[0064] The following describes the autonomous navigation system for a bronchoscope robot based on Transformer behavior cloning provided by this invention. The autonomous navigation system for a bronchoscope robot based on Transformer behavior cloning described below can be referred to in correspondence with the autonomous navigation method for a bronchoscope robot based on Transformer behavior cloning described above.

[0065] like Figure 2 As shown, the bronchoscopy robot autonomous navigation system based on Transformer behavior cloning provided by the present invention includes: The acquisition module is used to acquire multimodal state data of the bronchoscopic robot during its operation within the airway; The conversion module is used to convert multimodal state data into state vectors. The input module is used to input the time sequence consisting of the state vectors of the most recent consecutive time steps into the Transformer behavioral clone model to obtain the three-degree-of-freedom control vector output by the Transformer behavioral clone model. The control module is used to control the robot's operation within the airway based on a three-degree-of-freedom control vector. The training set for the Transformer behavior clone model includes time-series samples and corresponding labels, with the labels including the robot's actual actions.

[0066] Figure 3 A schematic diagram of the physical structure of an electronic device is provided. This device may include a processor, a communications interface, memory, and a communication bus. The processor, communications interface, and memory communicate with each other via the communication bus. The processor can invoke logical instructions from the memory to execute an autonomous navigation method for a bronchoscopy robot based on Transformer behavior cloning.

[0067] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0068] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the autonomous navigation method for a bronchoscope robot based on Transformer behavior cloning provided by the above methods.

[0069] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the autonomous navigation method for a bronchoscope robot based on Transformer behavior cloning provided by the methods described above.

[0070] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for autonomous navigation of a bronchoscopic robot based on Transformer behavior cloning, characterized in that, include: Acquire multimodal state data of the bronchoscopic robot during its operation within the airway; The multimodal state data is converted into a state vector; Input the time sequence of the state vectors from the most recent consecutive time steps into the Transformer behavioral cloning model to obtain the three-degree-of-freedom control vector output by the Transformer behavioral cloning model. The robot is controlled to operate within the airway based on the three-degree-of-freedom control vector; The training set of the Transformer behavior clone model includes samples of the time sequence and corresponding labels, wherein the labels include the robot's actual actions.

2. The autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning according to claim 1, characterized in that, The multimodal state data includes endoscopic images, bronchoscope tip pose data, and the robot's joint angles and path deviation vectors. The path deviation vector is the deviation vector between the current coordinates of the bronchoscope tip and the target coordinate point. The target coordinate point is the next coordinate point in the navigation reference path that is closest to the current coordinates. The starting point of the navigation reference path is the entrance of the airway, and the ending point is the location of the lesion.

3. The method for autonomous navigation of a bronchoscopic robot based on Transformer behavior cloning according to claim 2, characterized in that, Obtaining the path deviation vector includes: Establish a three-dimensional model of the airway; Based on the three-dimensional model, path planning is performed between the airway inlet and the location of the lesion to obtain the navigation reference path; Determine the target coordinate point in the navigation reference path; The path deviation vector is determined based on the current coordinates and the target coordinate point.

4. The autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning according to claim 2, characterized in that, The step of converting the multimodal state data into a state vector includes: Feature extraction is performed on the endoscopic image to obtain a visual feature vector; The visual feature vector, the pose data, the joint angle, and the path deviation vector are concatenated to obtain the state vector. The state vector is then encoded using time-position encoding.

5. The autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning according to claim 2, characterized in that, The label also includes a path direction truth vector, which is a unit vector from the current coordinates toward the target coordinate point; Before inputting the time-series sequence of the state vectors from the most recent consecutive time steps into the Transformer behavioral cloning model to obtain the three-degree-of-freedom control vector output by the Transformer behavioral cloning model, the method further includes: The time series samples are input into the Transformer behavioral cloning model to obtain the control prediction vector and path direction prediction vector output by the Transformer behavioral cloning model. A first loss is determined based on the true action and the control prediction vector corresponding to the samples of each of the time series, and a second loss is determined based on the path direction truth vector and the path direction prediction vector corresponding to the samples of each of the time series. The parameters of the Transformer behavior clone model are updated based on the first loss and the second loss.

6. The autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning according to claim 2, characterized in that, The endoscopic images are images that conform to the style of a real endoscope; Before inputting the time-series sequence of the state vectors from the most recent consecutive time steps into the Transformer behavioral cloning model to obtain the three-degree-of-freedom control vector output by the Transformer behavioral cloning model, the method further includes: After data augmentation and random perturbation of the training set, the Transformer behavior clone model is augmented and trained using the training set.

7. An autonomous navigation system for a bronchoscopy robot based on Transformer behavior cloning, characterized in that, include: The acquisition module is used to acquire multimodal state data of the bronchoscopic robot during its operation within the airway; The conversion module is used to convert the multimodal state data into a state vector; The input module is used to input the time sequence composed of the state vectors of the most recent consecutive time steps into the Transformer behavioral clone model to obtain the three-degree-of-freedom control vector output by the Transformer behavioral clone model. The control module is used to control the robot to operate within the airway based on the three-degree-of-freedom control vector; The training set of the Transformer behavior clone model includes samples of the time sequence and corresponding labels, wherein the labels include the robot's actual actions.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the autonomous navigation method for a bronchoscopic robot based on Transformer behavior cloning as described in any one of claims 1 to 6.

Images