An apparatus positioning navigation method, device, system and storage medium

Abstract

This application belongs to the field of navigation technology, specifically relating to a method, device, system, and storage medium for instrument positioning and navigation. The method includes: acquiring real-time images of an endoscope and preoperative images of the target object; determining a first position transformation model of the endoscope relative to the target object based on the real-time and preoperative images; acquiring a second position transformation model of a fixed base point in the instrument arm relative to the endoscope; determining a third position transformation model of the fixed base point relative to the target object based on the first and second position transformation models; and obtaining the position of the moving node relative to the target object based on the positional relationship between the moving node and the fixed base point in the instrument arm and the third position transformation model. This application improves robot navigation accuracy and avoids safety risks in robot operation by determining the position of the instrument arm relative to the target based on endoscope images and a robot kinematic model, and then performing precise navigation based on the determined position.

Description

Technical Field

[0001] This invention relates to the field of surgical robot technology, and in particular to an instrument positioning and navigation method, device, system and storage medium. Background Technology

[0002] In the medical field, flexible robots can serve as surgical robots, especially suitable for complex and confined workspaces, providing more precise control and operation for surgery. Unlike traditional rigid robots, the instrument arm of a flexible robot is usually composed of multiple bendable continuums. During surgical operations, the instrument arm can have various configurations, and the position of each point on the instrument arm relative to the end of the instrument arm can change at any time.

[0003] Due to the high compliance and redundant degrees of freedom of flexible robot structures, traditional path navigation methods for rigid robots are no longer suitable for flexible robots. Furthermore, some current navigation methods for flexible robots suffer from inaccurate and unclear path planning, thereby increasing the safety risks and operational difficulty of robot manipulation. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides a method, apparatus, system, and storage medium for instrument positioning and navigation. This application determines the position of the instrument arm relative to a target based on endoscopic images and a robot kinematic model, and performs precise navigation based on the determined position, thereby improving robot navigation accuracy and avoiding safety risks during robot operation.

[0005] On one hand, this application provides a device positioning and navigation method applied to a flexible robot including an endoscope and a robotic arm, the method comprising:

[0006] Acquire real-time endoscopic images and preoperative images of the identified subject;

[0007] A first positional transformation model of the endoscope relative to the identified object is determined based on real-time endoscopic images and preoperative images.

[0008] Obtain the second position transformation model of the fixed base point in the instrument arm relative to the endoscope;

[0009] A third position transformation model is determined based on the first and second position transformation models, relative to the object being identified, and the fixed base point is then determined.

[0010] The position of the moving node relative to the object being identified is obtained based on the positional relationship between the moving node and the fixed base point in the robotic arm and the third position transformation model.

[0011] Optionally, the number of moving nodes can be multiple; the position of the moving node relative to the object to be identified is obtained based on the positional relationship between the moving node and the fixed base point in the instrument arm and the third position transformation model, including: determining the positional relationship between each moving node and the fixed base point in the instrument arm; and determining the position of each moving node relative to the object to be identified based on the positional relationship between each moving node and the fixed base point and the third position transformation model.

[0012] Optionally, the instrument arm includes interconnected rigid rods and flexible tubes. Determining the positional relationship between each moving node and a fixed base point in the instrument arm includes: acquiring the position of the connection point relative to the fixed base point; the connection point is the connection point between the rigid rod and the flexible tube of the instrument arm; each moving node is located on the flexible tube; determining the bending angle and rotation angle of the flexible tube by driving sensors; and determining the positional relationship between each moving node and the fixed base point in the instrument arm based on the positional relationship of the connection point relative to the fixed base point, the length of the flexible tube, the bending angle, and the rotation angle.

[0013] Optionally, after obtaining the position of the moving node relative to the object to be identified based on the positional relationship between the moving node and the fixed base point in the instrument arm and the third position transformation model, the method further includes: controlling the end node in the moving node to move to a preset position of the object to be identified based on the position of the moving node relative to the object; the end node is the moving node located at the end of the instrument arm.

[0014] Optionally, the preset position is the position corresponding to the target object in the recognition object; based on the position of the moving node relative to the recognition object, controlling the end node in the moving node to move to the preset position of the recognition object includes: determining a first distance between the end node and the preset position based on the position of the end node relative to the recognition object; when the first distance is greater than a first preset distance threshold, controlling the robotic arm to move along a straight path until the first distance is less than or equal to the first preset distance threshold, so that the end node moves to the preset position of the recognition object; the straight path is the line connecting the end node and the preset position.

[0015] Optionally, after determining the first distance between the end node and the preset position based on the positional relationship of each mobile node relative to the identified object, the method further includes: determining a second distance between the robotic arm and the non-target object based on the positional relationship of each mobile node relative to the identified object; the second distance characterizes the preset distance between the mobile nodes on the robotic arm and the non-target object; the non-target object is an object other than the target object among the identified objects; when the first distance is greater than a first preset distance threshold and the second distance is less than a second preset distance threshold, the robotic arm is controlled to move along an obstacle avoidance path; when the robotic arm moves to the end of the obstacle avoidance path and the first distance is greater than the first preset distance threshold, the robotic arm is controlled to move along a straight path until the first distance is less than or equal to the first preset distance threshold, so that the end node moves to the preset position of the identified object.

[0016] Optionally, when the first distance is greater than the first preset distance threshold and the second distance is less than the second preset distance threshold, controlling the robotic arm to move along the obstacle avoidance path includes: when the first distance is greater than the first preset distance threshold and the second distance is less than the second preset distance threshold, controlling the robotic arm to stop moving; and responding to the control command, controlling the robotic arm to move along the obstacle avoidance path.

[0017] Optionally, controlling the robotic arm to move along the obstacle avoidance path includes: determining the obstacle avoidance path based on the outer contour of the non-target object; the distance between the obstacle avoidance path and the outer contour of the non-target object is a preset obstacle avoidance distance; and controlling the robotic arm to move along the obstacle avoidance path.

[0018] Optionally, determining the obstacle avoidance path based on the outer contour of the non-target object includes: determining the first and second intersection points of the straight path and the outer contour of the non-target object; the distance between the first intersection point and the end node is less than the distance between the second intersection point and the end node; determining the starting point of the obstacle avoidance path on the straight path based on the first intersection point; the distance between the starting point and the first intersection point is a preset obstacle avoidance distance; determining the ending point of the obstacle avoidance path on the straight path based on the second intersection point; the distance between the ending point and the second intersection point is a preset obstacle avoidance distance; and determining the obstacle avoidance path based on the starting point and the ending point of the obstacle avoidance path.

[0019] Optionally, the method further includes: determining the instrument activity boundary based on the edge of the identified object; determining the positional relationship of each moving node relative to the instrument activity boundary based on the positional relationship of each moving node relative to the identified object; determining a third distance between the instrument arm and the instrument activity boundary based on the positional relationship of each moving node relative to the instrument activity boundary; the third distance characterizes the shortest distance between the moving node on the instrument arm and the instrument activity boundary; and controlling the instrument arm to stop moving when the third distance is less than a third preset distance threshold.

[0020] Optionally, after determining the third distance between the instrument arm and the instrument's active boundary based on the positional relationship of each moving node relative to the instrument's active boundary, the method further includes: applying a repulsive force to the handle of the flexible robot when the third distance is less than a fourth preset distance threshold; the fourth preset distance threshold is greater than the third preset distance threshold.

[0021] Optionally, when the third distance is less than the fourth preset distance threshold, a repulsive force is applied to the handle of the flexible robot, including: when the third distance is less than the fourth preset distance threshold, identifying the moving node whose distance to the boundary of the moving node is the third distance as an alarm node; determining the repulsive force based on the speed of the alarm node, the third distance and the preset safety threshold; and applying the repulsive force to the handle of the flexible robot.

[0022] On the other hand, this application provides a medical device positioning and navigation device for use in a flexible robot including an endoscope and a medical arm. The device includes:

[0023] The acquisition module is used to acquire real-time images from the endoscope and preoperative images of the identified object;

[0024] The first conversion module is used to determine a first position conversion model of the endoscope relative to the identified object based on real-time endoscope images and preoperative images;

[0025] The second conversion module is used to obtain the second position conversion model of the fixed base point in the instrument arm relative to the endoscope;

[0026] The third conversion module is used to determine the third position conversion model of the fixed base point relative to the object to be identified based on the first position conversion model and the second position conversion model.

[0027] The position module is used to obtain the position of the moving node relative to the identified object based on the positional relationship between the moving node and the fixed base point in the robotic arm and the third position transformation model.

[0028] Optionally, the number of moving nodes can be multiple; the position module is used to: obtain the position of the moving node relative to the object to be identified based on the positional relationship between the moving node and the fixed base point in the instrument arm and the third position transformation model, including: determining the positional relationship between each moving node and the fixed base point in the instrument arm; and determining the position of each moving node relative to the object to be identified based on the positional relationship between each moving node and the fixed base point and the third position transformation model.

[0029] Optionally, the position module is used to obtain the position of the connection point relative to the fixed base point during the process of determining the positional relationship between each moving node and the fixed base point in the instrument arm; the connection point is the connection point between the rigid rod and the flexible tube of the instrument arm; each moving node is located on the flexible tube; the bending angle and rotation angle of the flexible tube are determined by driving sensors; based on the positional relationship between the connection point and the fixed base point, the length of the flexible tube, the bending angle and the rotation angle, the positional relationship between each moving node and the fixed base point in the instrument arm is determined.

[0030] Optionally, the device also includes a navigation module for controlling the end node of the mobile node to move to a preset position of the object to be identified based on the position of the mobile node relative to the object being identified; the end node is the mobile node located at the end of the robotic arm.

[0031] Optionally, the preset position is the position corresponding to the target object in the recognition object; the position module is used to determine the first distance between the end node and the preset position based on the position of the end node relative to the recognition object during the process of controlling the end node in the mobile node to move to the preset position of the recognition object based on the position of the mobile node relative to the recognition object; when the first distance is greater than the first preset distance threshold, the control arm is controlled to move along a straight path until the first distance is less than or equal to the first preset distance threshold, so that the end node moves to the preset position of the recognition object; the straight path is the line connecting the end node and the preset position.

[0032] Optionally, the position module is used to, after determining the first distance between the end node and the preset position based on the positional relationship of each mobile node relative to the identified object, perform the following: determining the second distance between the robotic arm and the non-target object based on the positional relationship of each mobile node relative to the identified object; the second distance characterizes the preset distance between the mobile node on the robotic arm and the non-target object; the non-target object is an object other than the target object among the identified objects; when the first distance is greater than the first preset distance threshold and the second distance is less than the second preset distance threshold, control the robotic arm to move along the obstacle avoidance path; when the robotic arm moves to the end of the obstacle avoidance path and the first distance is greater than the first preset distance threshold, control the robotic arm to move along a straight path until the first distance is less than or equal to the first preset distance threshold, so that the end node moves to the preset position of the identified object.

[0033] Optionally, the position module is used to control the robotic arm to move along the obstacle avoidance path when the first distance is greater than the first preset distance threshold and the second distance is less than the second preset distance threshold: when the first distance is greater than the first preset distance threshold and the second distance is less than the second preset distance threshold, control the robotic arm to stop moving; and in response to the control command, control the robotic arm to move along the obstacle avoidance path.

[0034] Optionally, the position module is used to: determine the obstacle avoidance path based on the outer contour of the non-target object during the movement of the control arm along the obstacle avoidance path; the distance between the obstacle avoidance path and the outer contour of the non-target object is a preset obstacle avoidance distance; and control the control arm to move along the obstacle avoidance path.

[0035] Optionally, the location module is used in the process of determining the obstacle avoidance path based on the outer contour of the non-target object: determining the first intersection point and the second intersection point of the straight path and the outer contour of the non-target object; the distance between the first intersection point and the end node is less than the distance between the second intersection point and the end node; determining the starting point of the obstacle avoidance path on the straight path based on the first intersection point; the distance between the starting point and the first intersection point is a preset obstacle avoidance distance; determining the ending point of the obstacle avoidance path on the straight path based on the second intersection point; the distance between the ending point and the second intersection point is a preset obstacle avoidance distance; and determining the obstacle avoidance path based on the starting point and the ending point of the obstacle avoidance path.

[0036] Optionally, the position module is also used to: determine the instrument activity boundary based on the edge of the identified object; determine the positional relationship of each moving node relative to the instrument activity boundary based on the positional relationship of each moving node relative to the identified object; determine the third distance between the instrument arm and the instrument activity boundary based on the positional relationship of each moving node relative to the instrument activity boundary; the third distance characterizes the shortest distance between the moving node on the instrument arm and the instrument activity boundary; when the third distance is less than a third preset distance threshold, control the instrument arm to stop moving.

[0037] Optionally, the position module is used to determine the third distance between the instrument arm and the instrument activity boundary based on the positional relationship of each moving node relative to the instrument activity boundary, and then perform the following: when the third distance is less than a fourth preset distance threshold, apply a repulsive force to the handle of the flexible robot; when the fourth preset distance threshold is greater than the third preset distance threshold.

[0038] Optionally, the position module is used to: when the third distance is less than the fourth preset distance threshold, during the process of applying a repulsive force to the handle of the flexible robot, identify the moving node whose distance to the boundary of the moving node is the third distance as an alarm node; determine the repulsive force based on the speed of the alarm node, the third distance and the preset safety threshold; and apply the repulsive force to the handle of the flexible robot.

[0039] On the other hand, this application provides a device positioning and navigation system, comprising:

[0040] The robotic arm consists of a moving node and a fixed base point.

[0041] An endoscope is used to acquire real-time images of the object being identified.

[0042] The processor and memory, wherein the memory stores at least one instruction, at least one program, code set or instruction set, and the processor loads and executes at least one instruction, at least one program, code set or instruction set to implement the above-described instrument positioning and navigation method.

[0043] On the other hand, this application provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the above-described instrument positioning and navigation method.

[0044] This application proposes a device, system, and storage medium for instrument positioning and navigation. It determines the relative position of a robot arm within the target by combining the 3D scene of the target and real-time images from the endoscope. Then, based on a robot kinematic model, it obtains the relative positions of the robot arm and the endoscope, and performs calculations to obtain a position transformation model. By calculating the positions of multiple moving nodes on the robot arm relative to a fixed base point and inputting the calculated positions into the position transformation model, the position of the robot arm relative to the target can be accurately determined, and precise navigation can be performed based on the determined position, thereby improving the accuracy of real-time navigation. Furthermore, when the robot arm is too close to the target, obstacle avoidance can be achieved by controlling the robot arm through handle-based manipulation commands or path planning, enabling safe navigation without touching the target. Moreover, by setting a boundary for the robot's movement, when the distance between the robot arm and the boundary is close, a repulsive force is applied to the robot handle or a forced stop is taken, thereby controlling the robot arm's movement area to prevent the robot from accidentally injuring unrelated targets. Attached Figure Description

[0045] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram illustrating the implementation environment of an instrument positioning and navigation method according to an exemplary embodiment;

[0047] Figure 2 This is a first flowchart illustrating a device positioning and navigation method according to an exemplary embodiment;

[0048] Figure 3 This is a schematic diagram of the second process of a device positioning and navigation method according to an exemplary embodiment;

[0049] Figure 4 This is a schematic diagram of the structure of an instrument arm in an instrument positioning and navigation method according to an exemplary embodiment;

[0050] Figure 5 This is a schematic diagram of the third process of an instrument positioning and navigation method according to an exemplary embodiment;

[0051] Figure 6 This is a schematic diagram of the fourth process of an instrument positioning and navigation method according to an exemplary embodiment;

[0052] Figure 7 This is a fifth flowchart illustrating a device positioning and navigation method according to an exemplary embodiment;

[0053] Figure 8 This is a schematic diagram illustrating an implementation environment of a method for determining a second distance according to an exemplary embodiment;

[0054] Figure 9 This is a sixth flowchart illustrating a device positioning and navigation method according to an exemplary embodiment;

[0055] Figure 10 This is a seventh flowchart illustrating a device positioning and navigation method according to an exemplary embodiment;

[0056] Figure 11 This is an eighth flowchart illustrating a device positioning and navigation method according to an exemplary embodiment;

[0057] Figure 12 This is a schematic diagram illustrating the implementation environment of a method for determining an obstacle avoidance path according to an exemplary embodiment;

[0058] Figure 13 This is a ninth flowchart illustrating a device positioning and navigation method according to an exemplary embodiment;

[0059] Figure 14 This is a schematic diagram illustrating the implementation environment of a method for controlling a robotic arm based on a device interaction boundary, according to an exemplary embodiment.

[0060] Figure 15 This is a schematic diagram of the tenth process of a device positioning and navigation method according to an exemplary embodiment;

[0061] Figure 16 This is a block diagram illustrating a device positioning and navigation apparatus according to an exemplary embodiment.

[0062] Figure 17 This is a hardware structure block diagram of a server for a device positioning and navigation method according to an exemplary embodiment. Detailed Implementation

[0063] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0064] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the present application described herein can be implemented in orders other than those illustrated or described herein. The terms "first" and "second" should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments, unless otherwise stated, "a plurality of" means two or more. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0065] To make the objectives, technical solutions, and advantages disclosed in the embodiments of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this application and are not intended to limit the embodiments of this application.

[0066] In the field of robot navigation, technologies such as automatic path navigation and virtual boundary protection exist, but these technologies are mainly based on rigid robots, not flexible robots. For flexible robots, existing navigation methods for maneuvering them to a target location suffer from unclear path planning and an inability to automatically identify obstacles, potentially leading to collisions between the robot and the target object, preventing the robot from reaching the target location efficiently and safely.

[0067] In view of this, embodiments of this application provide a method, apparatus, system, and storage medium for instrument positioning and navigation. By combining a three-dimensional scene within the human body and real-time images from the endoscope, the relative position of the endoscope tip to the target (i.e., the surgical area within the human body, such as the tissue to be operated on) is determined. Furthermore, the relative position of the robotic arm and the endoscope is obtained by combining a robot kinematic model. This allows for precise determination of the arm's position relative to the target, and accurate navigation based on the determined position. Moreover, by judging the relative distance based on the relative position, the robotic arm is controlled to avoid obstacles in a timely manner when the distance is within the obstacle avoidance range, thereby achieving safe navigation without touching the target.

[0068] Figure 1 This is a schematic diagram illustrating the implementation environment of an instrument positioning and navigation method according to an exemplary embodiment. For example... Figure 1 As shown, the implementation environment may include robot 01. Specifically, robot 01 may include an endoscope and a robotic arm. The endoscope can be used to acquire images of a target object, and the robotic arm can be used to move close to the target object. Optionally, robot 01 may include a flexible robot, which can be a continuum robot. As a novel type of biomimetic robot, the flexible robot includes a continuously bending structure or an elastic skeleton, and the structural deformation of the robotic arm is generated by bending motion driven by a nickel-titanium alloy wire or pneumatic drive. Optionally, robot 01 may also include software running in the physical device, such as an application program.

[0069] It should be noted that, Figure 1 This is just one example. Other implementation environments may also be included in other scenarios.

[0070] Figure 2 This is a first flowchart illustrating an instrument positioning and navigation method according to an exemplary embodiment. This specification provides the operational steps of the instrument positioning and navigation method as described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown in the flowcharts... Figure 2 As shown, the method may include:

[0071] S101: Acquire real-time endoscopic images and preoperative images of the identified subject.

[0072] In step S101, real-time endoscopic images can be acquired via the endoscope. As an example, the object to be identified can be target human tissue, and the preoperative medical image can include a three-dimensional scene inside the human body, specifically a three-dimensional scene including the object to be identified. Preoperative medical images can be obtained by three-dimensional reconstruction of preoperative CT or MRI images, but are not limited to the methods described above.

[0073] S102: Determine the first positional transformation model of the endoscope relative to the identified object based on real-time endoscopic images and preoperative images.

[0074] In step S102, a first position transformation model of the endoscope relative to the identified object can be determined based on real-time endoscopic images combined with preoperative medical images. Based on the three-dimensional scene inside the human body and the image of the identified object acquired by the endoscope, the position of the endoscope tip relative to the identified object can be calculated, thereby further determining the first position transformation model of the endoscope relative to the identified object.

[0075] As an example, the first position transformation model can be a coordinate transformation matrix from the endoscope coordinate system to the object being identified, which can be represented as follows:

[0076] S103: Obtain the second position transformation model of the fixed base point in the instrument arm relative to the endoscope.

[0077] In step S103, a second position transformation model of the fixed base point in the instrument arm relative to the endoscope can be obtained. Optionally, the fixed base point can be a fixed point in the instrument arm. Since the endoscope and the instrument arm both belong to the robot system, the positional relationship between the endoscope and the instrument arm can be obtained, and the second position transformation model can be determined based on this positional relationship using the robot kinematics model.

[0078] As an example, the second position transformation model can be a coordinate transformation matrix from the robot's base coordinate system to the endoscope coordinate system, which can be represented as:

[0079] S104: Determine the third position transformation model from the fixed base point to the object to be identified based on the first position transformation model and the second position transformation model.

[0080] In step S104, a third position transformation model from the fixed base point to the identified object can be determined based on the first position transformation model and the second position transformation model. In an optional embodiment, the first position transformation model and the second position transformation model can be multiplied to obtain the third position transformation model.

[0081] As an example, the first and second position transformation models can be coordinate transformation matrices, and are respectively represented as... The third position transformation model can be a coordinate transformation matrix from the robot's base coordinate system to the object's coordinate system, represented as follows: Optionally, the third position transformation model can be calculated based on the following formula:

[0082]

[0083] In this embodiment, a coordinate transformation model is determined based on the image to identify the object, endoscope, and instrument arm, enabling precise relative positioning. This accurate positional information can be used to guide the instrument arm's navigation, thereby improving navigation accuracy.

[0084] S105: The position of the moving node relative to the identified object is obtained based on the positional relationship between the moving node and the fixed base point in the robotic arm and the third position transformation model.

[0085] In step S105, the position of the moving node relative to the identified object is obtained based on the positional relationship between the moving node and the fixed base point in the robotic arm and the third position transformation model.

[0086] In some alternative implementations, the number of moving nodes can be multiple, with the moving node located at the end of the robotic arm being the end node. The following is based on... Figure 3 A further exemplary process for step S105 is described. Please refer to... Figure 3 , Figure 3 This is a schematic diagram of the second process of a device positioning and navigation method according to an exemplary embodiment. Specifically, as shown below... Figure 3 As illustrated, based on the positional relationship between the moving node and the fixed base point in the robotic arm and the third position transformation model, the position of the moving node relative to the identified object can be obtained, which may include:

[0087] Step S1051: Determine the positional relationship between each moving node and the fixed base point in the robotic arm.

[0088] In step S1051, the positional relationship between each of the moving nodes and the fixed base point in the entire robotic arm can be determined.

[0089] As an example, position sensors can be used to determine the positional relationship between each moving node and the fixed base point in a robotic arm.

[0090] The following is based on Figure 4 and Figure 5 An exemplary process for step S1051 is further described.

[0091] Please refer to Figure 4 , Figure 4 This is a schematic diagram illustrating a surgical arm structure according to an exemplary embodiment. Figure 4 The illustration shows a specific embodiment of a surgical arm structure, such as... Figure 4 As illustrated, the robotic arm can be composed of a rigid rod and a flexible tube. In one optional embodiment, the two ends of the rigid rod can be a fixed base point and a connection point A, respectively; in another optional embodiment, the connection point A can coincide with the fixed base point. The two ends of the flexible tube are the connection point A and the end point C, respectively. Multiple continuous sections can be arranged from position A to position C, where the center point of each continuous section can be a moving node, and the multiple center points and end nodes of the multiple continuous sections constitute multiple moving nodes in this embodiment.

[0092] As an example, the position of any point on the robotic arm relative to the robot's base coordinate system can be obtained by using a position sensor to determine the robot's joint positions, and then substituting these joint positions into the robot's forward kinematics. Please refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram of the third process of a device positioning and navigation method according to an exemplary embodiment. Specifically, the exemplary process of step S1051 is as follows: Figure 5 As illustrated, determining the positional relationship between each moving node and fixed base point in the robotic arm can include:

[0093] S10511: Obtain the position of the connection point relative to the fixed base point.

[0094] In one alternative implementation, the connection point is the connection between the rigid rod and the flexible tube of the robotic arm. The rigid rod is a fixed structure, while the flexible tube contains multiple movable nodes. As an example, in... Figure 4 In the illustrated instrument arm structure, the connection point is point A, and the two ends of the flexible tube are connection point A and end point C, respectively.

[0095] In step S10511, the positional relationship of the connection point relative to the fixed base point can be determined based on the length of the rigid rod. In one optional embodiment, the fixed base point can be located at the end of the rigid rod furthest from the flexible tube. In this embodiment, the distance between the connection point and the fixed base point is the length of the rigid rod, and the positional relationship of the connection point relative to the fixed base point can be determined based on the length of the rigid rod. In other optional embodiments, the fixed base point can also be located at any position on the rigid rod, including the connection point between the rigid rod and the flexible tube. In this embodiment, the positional relationship of the connection point relative to the fixed base point can be determined based on the setting position of the fixed base point.

[0096] S10512: Determines the bending angle and rotation angle of the flexible tube by driving the sensor.

[0097] In one optional implementation, the bending angle and rotation angle of the flexible tube can be calculated by sensors at the drive end. Specifically, an encoder can be used to measure the rotation angle of the gear at the drive end, and then the gear rotation angle can be converted into the bending and rotation angles of the flexible segment. In the formulas and expressions of this application embodiment, as an example, the bending angle can be expressed as θ, and the rotation angle can be expressed as...

[0098] S10513: Based on the positional relationship of the connection point relative to the fixed base point, the length of the flexible tube, the bending angle, and the rotation angle, determine the positional relationship between each moving node and the fixed base point in the instrument arm.

[0099] In one alternative implementation, the positional relationship between each moving node and the fixed base point in the robotic arm can be determined by calculation formulas based on the positional relationship of the connection point relative to the fixed base point, the length of the flexible tube, the bending angle, and the rotation angle.

[0100] As an example, suppose the flexible tube consists of n continuous sections, then the flexible tube contains n moving nodes. The position of the i-th moving node relative to the connection point A can be calculated using the following formula.

[0101]

[0102]

[0103]

[0104] Where l represents the length of the flexible tube; θ represents the bending angle of the flexible tube; Characterizes the rotation angle of the flexible tube.

[0105] Optionally, the position of the i-th moving node relative to A among the n moving nodes can be calculated using the formula above. This yields the position of each mobile node relative to its connection point. Then, based on the positional relationship between the connection point and the fixed base point, and the positional relationship of each mobile node relative to its connection point, the positional relationship between each mobile node and the fixed base point can be determined. In this embodiment, only a drive sensor needs to be set up to calculate the positional relationship between the mobile node and the identified object, allowing for convenient sensor setup and reduced costs. Furthermore, since the position of the connection point between the flexible tube and the rigid rod is fixed relative to the fixed base point, and all mobile nodes are on the flexible tube, this embodiment allows for more accurate determination of the position of the mobile nodes relative to the robot's basic coordinate system by calculating the position of the mobile nodes relative to the fixed base point in real time based on the connection point position when the flexible tube of the flexible robot rotates or bends during operation. This results in more accurate calculation results.

[0106] The following continues based on Figure 3 To elaborate:

[0107] Step S1052: Based on the positional relationship between each mobile node and the fixed base point, and the third position transformation model, determine the position of each mobile node relative to the identified object.

[0108] In one alternative implementation, the fixed base point can be the origin of the robot's base coordinate system, and the position of each moving node relative to the fixed base point is the position of each moving node relative to the robot's base coordinate system.

[0109] In step S1052, the positional relationship of each mobile node relative to the object being identified can be determined using a calculation formula based on the position of each mobile node relative to the robot's base coordinate system and the third position transformation model. Specifically, the position of each mobile node relative to the robot's base coordinate system can be input into the third position transformation model to obtain the position of each mobile node relative to the object being identified.

[0110] As an example, the position of each mobile node relative to the identified object can be calculated using the following formula.

[0111]

[0112] in, This indicates the position of the moving node relative to the robot's base coordinate system;

[0113] Indicates the position of the moving node relative to the coordinate system of the object being identified;

[0114] This represents the coordinate transformation matrix from the robot's base coordinate system to the object being identified.

[0115] In this embodiment, the position of the moving node relative to the robot's basic coordinate system can be calculated based on the coordinate transformation matrix from the robot's basic coordinate system to the object being identified, thereby obtaining the position of the moving node relative to the object being identified. This allows for convenient and accurate determination of the position of the moving node relative to the object being identified, while reducing the complexity of the algorithm.

[0116] The following continues based on Figure 2 To elaborate:

[0117] S106: Based on the position of the mobile node relative to the identified object, control the end node in the mobile node to move to the preset position of the identified object.

[0118] In step S106, the end node can be controlled to move to a preset position of the identified object based on the position of each mobile node relative to the identified object. Optionally, the identified object may include a target object and a non-target object, wherein the preset position is the position corresponding to the target object among the identified objects. As an example, the identified object may be human tissue, the target object may be a lesion in the human tissue, and the non-target object may be other parts of the human tissue other than the lesion. In this embodiment, the preset position may be a position on the lesion or a surgical area near the lesion.

[0119] The following is based on Figure 6 The specific implementation method of step S1053 is further described.

[0120] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the fourth process of a device positioning and navigation method according to an exemplary embodiment. Specifically, as shown below... Figure 6 As illustrated, controlling the end node to move to a preset position on the object being identified, based on the positional relationship of each mobile node relative to the object being identified, includes:

[0121] S10531: Determine the first distance between the end node and the preset position based on the position of the end node relative to the identified object.

[0122] In step S10531, the first distance between the end node and the preset position can be determined based on the position of the end node in the mobile node relative to the identified object and the position of the preset position in the identified object.

[0123] S10533: When the first distance does not meet the first preset distance condition, the control arm moves along a straight path until the first distance meets the first preset distance condition, so that the end node moves to the preset position of the object to be identified.

[0124] Optionally, the straight path is the line connecting the current position of the end node and the preset position. Moving along the straight path means moving in the direction of the preset position.

[0125] Optionally, the first preset distance condition can be that the first distance is less than or equal to the first preset distance threshold. The first preset distance threshold can be set according to the actual operation scenario. This application embodiment does not limit the first preset distance threshold.

[0126] In step S10533, when the first distance does not meet the first preset distance condition, it means that the instrument arm is not currently within the area of ​​the preset position. This can control the instrument arm to move along a straight path so that the instrument arm moves along the shortest path to the preset position of the object to be identified.

[0127] The following is based on Figure 7 Another specific implementation of step S1053 is further described.

[0128] Please refer to Figure 7 , Figure 7 This is a schematic diagram of the fifth step of a device positioning and navigation method according to an exemplary embodiment. Specifically, as shown below... Figure 7 As illustrated, after determining the first distance between the end node and the preset position based on the positional relationship of each mobile node relative to the identified object, the method may include:

[0129] S10532: Determine a second distance between the robotic arm and the non-target object based on the positional relationship of each moving node relative to the identified object.

[0130] Optionally, the second distance can characterize a preset distance between the nearest moving node and the non-target object among all moving nodes on the robotic arm.

[0131] In step S10532, based on the position of each mobile node relative to the identified object, the shortest distance between each mobile node and the non-target object can be determined as the candidate distance for each mobile node, and the shortest candidate distance among all candidate distances of all mobile nodes can be determined as the second distance.

[0132] In one alternative implementation, a second distance between the robotic arm and the non-target object can be determined based on the position of a preset mobile node relative to the object being identified. Specifically, multiple target points (i.e., preset mobile nodes) are selected from multiple mobile nodes, and the second distance between the robotic arm and the non-target object is determined based on the position of the target points relative to the object being identified, thereby reducing computational load and improving algorithm execution efficiency.

[0133] Please refer to Figure 8 , Figure 8As an example, the second distance can characterize the shortest distance between the robotic arm and the non-target object. To efficiently calculate the second distance, target points 1, 2, and 3 can be set on the flexible tube. The second distance is determined based on the positional relationship between these three target points and the non-target object, thus ensuring that the algorithm can be executed within a specified time. Of course, the number of target points is not limited to the three in the example above. To ensure both accuracy and efficiency in the calculation, the number of target points can preferably be 3-30.

[0134] S10534: When the first distance does not meet the first preset distance condition and the second distance does not meet the second preset distance condition, control the robotic arm to move along the obstacle avoidance path.

[0135] Optionally, the obstacle avoidance path can be a path that does not intersect with non-target objects.

[0136] Optionally, the second preset distance condition can be that the second distance is greater than or equal to the second preset distance threshold. The second preset distance threshold can be set according to the actual operation scenario. This application embodiment does not limit the second preset distance threshold.

[0137] In step S10534, when the first distance does not meet the first preset distance condition and the second distance does not meet the second preset distance condition, it indicates that the robotic arm has not yet reached the preset position and the robotic arm is too close to the non-target object, which poses a risk of collision. Therefore, the robotic arm can be controlled to move along the obstacle avoidance path to avoid collision between the robotic arm and the non-target object.

[0138] The following are based on Figure 9 , Figure 10 Two specific implementation methods for controlling the movement of the control arm along the obstacle avoidance path in step S10534 are introduced respectively.

[0139] The first type:

[0140] Please refer to Figure 9 , Figure 9 This is a sixth flowchart illustrating a device positioning and navigation method according to an exemplary embodiment.

[0141] Specifically, such as Figure 9 As illustrated, when the first distance does not meet the first preset distance condition and the second distance does not meet the second preset distance condition, controlling the robotic arm to move along the obstacle avoidance path may include:

[0142] S201: When the first distance does not meet the first preset distance condition and the second distance does not meet the second preset distance condition, control the robotic arm to stop moving.

[0143] In step S201, when the first distance does not meet the first preset distance condition and the second distance does not meet the second preset distance condition, it indicates that the robotic arm has not yet reached the preset position and the robotic arm is too close to the non-target object, posing a risk of collision. At this time, the robotic arm can be controlled to stop moving.

[0144] S202: In response to control commands, control the robotic arm to move along the obstacle avoidance path.

[0145] In step S202, after the robotic arm has stopped moving, it can be moved along an obstacle avoidance path in response to a manipulation command. As an example, the manipulation command can be generated based on the user's operation of the robot handle.

[0146] In one alternative implementation, the specific instruction content, including the movement direction and speed, can be parsed from the manipulation instructions. The robotic arm can then be controlled to move along the obstacle avoidance path based on the movement direction and speed information, making the robot's obstacle avoidance path more flexible.

[0147] The second type:

[0148] Please refer to Figure 10 , Figure 10 This is a seventh flowchart illustrating a device positioning and navigation method according to an exemplary embodiment.

[0149] Specifically, such as Figure 10 As illustrated, controlling the movement of the robotic arm along the obstacle avoidance path may include:

[0150] S301: Determine the obstacle avoidance path based on the outer contour of the non-target object.

[0151] When there is a risk of collision when the robotic arm moves along a straight path, it can be changed to move along an arc path. Optionally, the distance between the obstacle avoidance path (i.e., the arc path) and the outer contour of the non-target object can be a preset obstacle avoidance distance. It should be noted that the preset obstacle avoidance distance can be equal to, less than, or greater than the second preset distance threshold; the relationship between the two is not limited here.

[0152] When the second preset distance threshold equals the preset obstacle avoidance distance, specifically, the node with the shortest distance to the non-target object among all moving nodes is the nearest node. For this nearest node, the distance between the nearest node and the non-target object can remain at the preset obstacle avoidance distance as the robotic arm moves along the obstacle avoidance path. It should be noted that during the movement of the robotic arm, the nearest node is not a fixed moving node, but rather a new nearest node will appear as the movement progresses. The preset obstacle avoidance distance is always the distance between the current nearest node and the non-target object.

[0153] When the second preset distance threshold is greater than or less than the preset obstacle avoidance distance, specifically, the node with the shortest distance to the non-target object among all moving nodes is the nearest node. For this nearest node, if the distance between the nearest node and the non-target object reaches the second preset distance threshold during the movement of the robotic arm along the obstacle avoidance path, the robotic arm first moves from its current position to the starting point of the arc path, and then moves along the arc path to the ending point. Here, the distance between the current position of the nearest node and the non-target object can be the second preset distance threshold, and the distance between the starting point of the arc path and the non-target object can be the preset obstacle avoidance distance.

[0154] In some other alternative implementations, the shortest distance between the obstacle avoidance path and the outer contour of the non-target object can be a preset obstacle avoidance distance, and the longest distance between the obstacle avoidance path and the outer contour of the non-target object can be greater than the preset obstacle avoidance distance.

[0155] The following is based on Figure 11 and Figure 12 The specific implementation method of step S301 will be further described.

[0156] Please refer to Figure 11 , Figure 11 This is an eighth flowchart illustrating a device positioning and navigation method according to an exemplary embodiment.

[0157] Specifically, such as Figure 11 The diagram illustrates the process of determining an obstacle avoidance path based on the outer contour of a non-target object, including:

[0158] S3011: Determine the first and second intersection points of the straight path and the outer contour of the non-target object.

[0159] Optionally, the distance between the first intersection point and the end node is less than the distance between the second intersection point and the end node.

[0160] S3012: Determine the starting point of the obstacle avoidance path on the straight path based on the first intersection point.

[0161] Optionally, the distance between the starting point and the first intersection point is a preset obstacle avoidance distance.

[0162] S3013: Determine the endpoint of the obstacle avoidance path on the straight path based on the second intersection point.

[0163] Optionally, the distance between the endpoint and the second intersection point is a preset obstacle avoidance distance;

[0164] S3014: Determine the obstacle avoidance path based on the starting point and ending point of the obstacle avoidance path.

[0165] Optionally, the starting point and the ending point of the obstacle avoidance path can be connected by an arc, so that the distance between the arc and the non-target object is equal to the preset obstacle avoidance distance, and the arc is determined as the obstacle avoidance path.

[0166] Please refer to Figure 12 , Figure 12 This is a schematic diagram illustrating an implementation environment for a method of determining an obstacle avoidance path according to an exemplary embodiment. The first intersection point and the second intersection point determined in step S3011 can be points A and B in the diagram, respectively. Then, steps S3012-S3014 can be executed to accurately determine the obstacle avoidance path shown in the diagram, enabling the robotic arm to move to the target lesion without colliding with human tissue.

[0167] In an optional implementation, in steps S3011-S3014, the preset obstacle avoidance distance can be consistent with the second preset distance threshold in step S10532.

[0168] In another optional implementation, the preset obstacle avoidance distance may be different from the second preset distance threshold in step S10532. Therefore, when the robotic arm detects an obstacle, its current position is not on the obstacle avoidance path. In this implementation, the starting point determined in step S3012 may be the starting point of the second sub-path of the obstacle avoidance path. Step S3014 may include: connecting the starting point of the second sub-path and the ending point of the obstacle avoidance path with an arc, so that the distance between the arc and the non-target object is equal to the preset obstacle avoidance distance, and determining the arc as the second sub-path of the obstacle avoidance path; determining the straight line connecting the current robotic arm and the starting point of the second sub-path as the first sub-path; and connecting the first sub-path and the second sub-path to obtain the obstacle avoidance path.

[0169] The following continues based on Figure 10 To elaborate:

[0170] S302: Control the movement of the control arm along the obstacle avoidance path.

[0171] In step S302, the robotic arm can be controlled to move along the obstacle avoidance path to prevent accidents caused by handle operation errors, thereby ensuring that obstacle avoidance is foolproof during the movement.

[0172] The following continues based on Figure 7 To elaborate:

[0173] S10535: When the robotic arm moves to the end of the obstacle avoidance path and the first distance does not meet the first preset distance condition, control the robotic arm to move along a straight path until the first distance meets the first preset distance condition, so that the end node moves to the preset position of the identified object.

[0174] In step S10535, when the robotic arm moves to the end of the obstacle avoidance path and the first distance does not meet the first preset distance condition, it indicates that the robotic arm has escaped the collision risk but has not yet reached the preset position. At this time, the robotic arm can be controlled to move along a straight path until the first distance meets the first preset distance condition, so that the end node moves to the preset position of the identified object.

[0175] In some scenarios, the robotic arm can not only navigate automatically based on path planning under automatic control, but also be manually controlled by the user in manual or semi-automatic control modes. Regardless of whether it's automatic navigation or manual control, the robotic arm can easily injure objects other than the target during movement, such as non-surgical organs, cavities, and blood vessels. By setting boundaries, the robotic arm's movement path can be pre-programmed, and control can be based on these boundaries. The following is based on... Figure 13 This application further describes a device positioning and navigation method provided by an embodiment. As an example, a virtual boundary can be set around the identified object. This virtual boundary can be a device activity boundary, meaning that the robot arm of the robot system is only allowed to move within the virtual boundary; the virtual boundary can also be a device restricted area boundary, meaning that the robot arm is not allowed to move within the virtual boundary.

[0176] Please refer to Figure 13 , Figure 13 This is a ninth flowchart illustrating a device positioning and navigation method according to an exemplary embodiment. Figure 13 As illustrated, a device positioning and navigation method may further include:

[0177] S401: Determine the boundary of instrument activity based on the edge of the identified object.

[0178] In step S401, the instrument activity boundary can be determined based on the edge of the identified object. This instrument activity boundary can be the boundary of the instrument activity area where the instrument arm is allowed to move. Optionally, the instrument activity boundary can include one or more.

[0179] In one alternative implementation, a three-dimensional scene inside the human body can be created using preoperative medical imaging, and the instrument's range of motion can be established based on the surgical procedure. Areas within the instrument's range of motion are where the instrument is permitted to move, while other areas outside the range are where the instrument cannot enter. Optionally, the instrument's range of motion may include a target object; that is, the boundary of the instrument's movement may be located outside the edge of the target object.

[0180] S402: Based on the positional relationship of each moving node relative to the identified object, determine the positional relationship of each moving node relative to the instrument's activity boundary.

[0181] In step S402, the position of the instrument's active boundary relative to the identified object can be obtained, and the positional relationship of each moving node relative to the instrument's active boundary can be determined by combining the position of each moving node relative to the identified object.

[0182] In one alternative implementation, multiple target points can be determined from the moving nodes. Step S402 may include: determining the positions of the multiple target points relative to the identified object based on the position of each moving node relative to the identified object; and determining the positional relationship between each moving node and the instrument's moving boundary relative to the identified object based on the position of the instrument's moving boundary and the positions of the multiple target points relative to the identified object.

[0183] S403: Determine the third distance between the instrument arm and the instrument's active boundary based on the positional relationship of each moving node relative to the instrument's active boundary.

[0184] Optionally, the third distance can characterize the shortest distance between all moving nodes on the instrument arm and the instrument's active boundary.

[0185] In step S403, based on the positional relationship of each moving node relative to the instrument's active boundary, the shortest distance between each moving node and the instrument's active boundary can be determined as a candidate distance, thus obtaining all candidate distances corresponding to all moving nodes, and the shortest candidate distance among all candidate distances is determined as the third distance.

[0186] S404: When the third distance does not meet the third preset distance condition, control the robotic arm to stop moving.

[0187] Optionally, the third distance threshold condition can be that the third distance is greater than or equal to a third preset distance threshold.

[0188] In step S404, when the third distance does not meet the third preset distance condition, it indicates that the instrument arm is too close to the instrument activity boundary. At this time, the instrument arm can be controlled to stop moving in order to avoid the instrument arm from contacting other objects besides the identified object, thereby causing damage to other objects.

[0189] The following is combined Figure 14 Further details are provided on steps S401-S404 above.

[0190] Please refer to Figure 14 , Figure 14 This is a schematic diagram illustrating the implementation environment of a method for controlling an instrument arm based on the instrument's activity boundary, according to an exemplary embodiment.

[0191] according to Figure 14As shown in the figure, the virtual boundary indicated by the dashed line can be the instrument activity boundary determined in step S401, and the shortest distance d can be the third distance determined in steps S402-S403. By executing step S404, the instrument arm can be made to stop moving when it gets too close to the virtual boundary, thereby avoiding contact between the instrument arm and the tissues in the human body.

[0192] As an example, after step S403, the method may further include: applying a repulsive force to the handle of the flexible robot when the third distance does not meet the fourth preset distance condition. The fourth preset distance condition may be that the third distance is greater than or equal to a fourth preset distance threshold, which may be greater than the third preset distance threshold described in step S404. This application does not limit the specific values ​​of the third preset distance threshold and the fourth preset distance threshold. The following is in conjunction with... Figure 15 This implementation method will be described in detail.

[0193] Please refer to Figure 15 , Figure 15 This is a schematic diagram of the tenth process of a device positioning and navigation method according to an exemplary embodiment.

[0194] like Figure 15 As illustrated, when the third distance does not meet the fourth preset distance condition, a repulsive force is applied to the handle of the flexible robot, including:

[0195] S4051: When the third distance does not meet the fourth preset distance condition, the moving node whose distance to the device's moving boundary is the third distance is determined as the alarm node.

[0196] In step S4051, when the third distance does not meet the fourth preset distance condition, it indicates that the instrument arm is too close to the instrument's active boundary. At this time, the point among all moving nodes that is closest to the instrument's active boundary can be determined as an alarm node. That is, the moving node among all moving nodes whose distance to the instrument's active boundary is the third distance is determined as an alarm node.

[0197] S4052: Determine the repulsion force based on the speed of the alarm node, the third distance, and the preset safety threshold.

[0198] In step S4052, the repulsive force can be determined based on the speed of the alarm node, the third distance, and the preset safety threshold.

[0199] As an example, the repulsive force can be determined using a calculation formula based on the alarm node's velocity, third distance, and a preset safety threshold. Specifically, the repulsive force F can be calculated using the following formula: c :

[0200]

[0201] Where d represents the third distance; threshold represents the preset safety threshold; e is a natural number; and v represents the speed of the alarm node.

[0202] S4053: Apply a repulsive force to the handle of the flexible robot.

[0203] In step S4053, a repulsive force can be applied to the handle of the flexible robot based on the repulsive force determined in step S4052, thereby guiding the user to operate the handle so that the robotic arm no longer moves toward the boundary of the robotic movement.

[0204] In some alternative implementations, step S4053 may further include generating an audio prompt and generating an image prompt. The audio and image prompts can be used to issue an alarm, reminding the user to stop controlling the robotic arm to move towards the boundary of the robotic arm's movement.

[0205] This application also provides an instrument positioning and navigation device for use in flexible robots including endoscopes and instrument arms. Figure 16 This is a block diagram illustrating a device positioning and navigation system according to an exemplary embodiment. Figure 16 As shown, the instrument positioning and navigation device 200 may include at least:

[0206] The acquisition module 201 is used to acquire real-time images of the endoscope and preoperative images of the object to be identified;

[0207] The first conversion module 202 is used to determine a first position conversion model of the endoscope relative to the identified object based on the real-time endoscope image and the preoperative image;

[0208] The second conversion module 203 is used to obtain a second position conversion model of the fixed base point in the instrument arm relative to the endoscope.

[0209] The third conversion module 204 is used to determine the third position conversion model of the fixed base point relative to the object to be identified based on the first position conversion model and the second position conversion model.

[0210] Position module 205 is used to obtain the position of the moving node relative to the identified object based on the positional relationship between the moving node and the fixed base point in the instrument arm and the third position transformation model.

[0211] Optionally, the number of moving nodes can be multiple; the position module 205 is used to: obtain the position of the moving node relative to the object to be identified based on the positional relationship between the moving node and the fixed base point in the instrument arm and the third position transformation model, including: determining the positional relationship between each moving node and the fixed base point in the instrument arm; and determining the position of each moving node relative to the object to be identified based on the positional relationship between each moving node and the fixed base point and the third position transformation model.

[0212] Optionally, the position module 205 is used to obtain the position of the connection point relative to the fixed base point during the process of determining the positional relationship between each moving node and the fixed base point in the instrument arm; the connection point is the connection point between the rigid rod and the flexible tube of the instrument arm; each moving node is located on the flexible tube; the bending angle and rotation angle of the flexible tube are determined by driving sensors; based on the positional relationship between the connection point and the fixed base point, the length of the flexible tube, the bending angle and the rotation angle, the positional relationship between each moving node and the fixed base point in the instrument arm is determined.

[0213] Optionally, the device also includes a navigation module for controlling the end node of the mobile node to move to a preset position of the object to be identified based on the position of the mobile node relative to the object being identified; the end node is the mobile node located at the end of the robotic arm.

[0214] Optionally, the preset position is the position corresponding to the target object in the recognition object; the position module 205 is used to determine a first distance between the end node and the preset position based on the position of the end node relative to the recognition object during the process of controlling the end node in the mobile node to move to the preset position of the recognition object based on the position of the mobile node relative to the recognition object; when the first distance is greater than the first preset distance threshold, the control arm is controlled to move along a straight path until the first distance is less than or equal to the first preset distance threshold, so that the end node moves to the preset position of the recognition object; the straight path is the line connecting the end node and the preset position.

[0215] Optionally, the position module 205 is used to, after determining the first distance between the end node and the preset position based on the positional relationship of each mobile node relative to the identified object, perform the following: determining the second distance between the robotic arm and the non-target object based on the positional relationship of each mobile node relative to the identified object; the second distance represents the preset distance between the mobile node on the robotic arm and the non-target object; the non-target object is an object other than the target object among the identified objects; when the first distance is greater than the first preset distance threshold and the second distance is less than the second preset distance threshold, controlling the robotic arm to move along the obstacle avoidance path; when the robotic arm moves to the end of the obstacle avoidance path and the first distance is greater than the first preset distance threshold, controlling the robotic arm to move along a straight path until the first distance is less than or equal to the first preset distance threshold, so that the end node moves to the preset position of the identified object.

[0216] Optionally, the position module 205 is used to control the robotic arm to move along the obstacle avoidance path when the first distance is greater than the first preset distance threshold and the second distance is less than the second preset distance threshold: when the first distance is greater than the first preset distance threshold and the second distance is less than the second preset distance threshold, control the robotic arm to stop moving; and in response to the control command, control the robotic arm to move along the obstacle avoidance path.

[0217] Optionally, the position module 205 is used to: determine the obstacle avoidance path based on the outer contour of the non-target object during the process of controlling the mechanical arm to move along the obstacle avoidance path; the distance between the obstacle avoidance path and the outer contour of the non-target object is a preset obstacle avoidance distance; and control the mechanical arm to move along the obstacle avoidance path.

[0218] Optionally, the position module 205 is used in the process of determining the obstacle avoidance path based on the outer contour of the non-target object: determining the first intersection point and the second intersection point of the straight path and the outer contour of the non-target object; the distance between the first intersection point and the end node is less than the distance between the second intersection point and the end node; determining the starting point of the obstacle avoidance path on the straight path based on the first intersection point; the distance between the starting point and the first intersection point is a preset obstacle avoidance distance; determining the ending point of the obstacle avoidance path on the straight path based on the second intersection point; the distance between the ending point and the second intersection point is a preset obstacle avoidance distance; and determining the obstacle avoidance path based on the starting point and the ending point of the obstacle avoidance path.

[0219] Optionally, the position module 205 is further configured to: determine the instrument activity boundary based on the edge of the identified object; determine the positional relationship of each moving node relative to the instrument activity boundary based on the positional relationship of each moving node relative to the identified object; determine a third distance between the instrument arm and the instrument activity boundary based on the positional relationship of each moving node relative to the instrument activity boundary; the third distance characterizes the shortest distance between the moving node on the instrument arm and the instrument activity boundary; and control the instrument arm to stop moving when the third distance is less than a third preset distance threshold.

[0220] Optionally, the position module 205 is used to determine the third distance between the instrument arm and the instrument activity boundary based on the positional relationship of each moving node relative to the instrument activity boundary, and then perform the following: when the third distance is less than a fourth preset distance threshold, apply a repulsive force to the handle of the flexible robot; when the fourth preset distance threshold is greater than the third preset distance threshold.

[0221] Optionally, the position module 205 is used to: when the third distance is less than the fourth preset distance threshold, during the process of applying a repulsive force to the handle of the flexible robot, identify the moving node whose distance to the boundary of the moving node is the third distance as an alarm node; determine the repulsive force based on the speed of the alarm node, the third distance and the preset safety threshold; and apply the repulsive force to the handle of the flexible robot.

[0222] It should be noted that the instrument positioning and navigation device embodiments provided in this application are based on the same inventive concept as the above-described instrument positioning and navigation method embodiments.

[0223] This application also provides an instrument positioning and navigation system, including: an instrument arm; the instrument arm includes a moving node and a fixed base point; an endoscope for acquiring real-time images of the object to be identified; a processor and a memory, the memory storing at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by the processor to implement the above-described instrument positioning and navigation method.

[0224] Embodiments of this application also provide a computer-readable storage medium that can be disposed in a terminal to store at least one instruction or at least one program for implementing an instrument positioning and navigation method as described in the method embodiments. The at least one instruction or at least one program is loaded and executed by a processor to implement the instrument positioning and navigation method as described in the above method embodiments.

[0225] Optionally, in the embodiments of this specification, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0226] The memory described in this specification can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for functions, etc.; the data storage area may store data created based on the use of the device, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory may also include a memory controller to provide the processor with access to the memory.

[0227] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the instrument positioning and navigation method provided in the above-described method embodiments.

[0228] The device positioning and navigation method embodiments provided in this application can be executed in a terminal, computer terminal, server, or similar computing device. Taking running on a server as an example, Figure 17 This is a hardware structure block diagram of a server for a device positioning and navigation method according to an exemplary embodiment. For example... Figure 17 As shown, the server 300 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 310 (CPUs 310 may include, but are not limited to, microprocessors (MCUs) or programmable logic devices (FPGAs), a memory 330 for storing data, and one or more storage media 320 (e.g., one or more mass storage devices) for storing application programs 323 or data 322. The memory 330 and storage media 320 may be temporary or persistent storage. The program stored in the storage media 320 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 310 may be configured to communicate with the storage media 320 and execute the series of instruction operations stored in the storage media 320 on the server 300. Server 300 may also include one or more power supplies 360, one or more wired or wireless network interfaces 350, one or more input / output interfaces 340, and / or one or more operating systems 321, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0229] The input / output interface 340 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 300. In one example, the input / output interface 340 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 340 may be a radio frequency (RF) module for wireless communication with the Internet.

[0230] Those skilled in the art will understand that Figure 3 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 300 may also include... Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown.

[0231] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0232] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and server embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0233] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0234] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A device for locating and navigating machinery, characterized in that, Flexible robots used in applications including endoscopes and robotic arms include: The acquisition module is used to acquire real-time images from the endoscope and preoperative images of the identified object; The first conversion module is used to determine a first position conversion model of the endoscope relative to the identified object based on the real-time endoscope image and the preoperative image; The second conversion module is used to obtain a second position conversion model of the fixed base point in the instrument arm relative to the endoscope; The third conversion module is used to determine a third position conversion model of the fixed base point relative to the identified object based on the first position conversion model and the second position conversion model. The position module is used to obtain the position of the moving node relative to the identified object based on the positional relationship between the moving node and the fixed base point in the robotic arm and the third position transformation model.

2. The instrument positioning and navigation device according to claim 1, characterized in that, The number of mobile nodes is multiple; The process of obtaining the position of the moving node relative to the identified object based on the positional relationship between the moving node and the fixed base point in the robotic arm and the third position transformation model includes: Determine the positional relationship between each moving node in the robotic arm and the fixed base point; Based on the positional relationship between each mobile node and the fixed base point, and the third position transformation model, the position of each mobile node relative to the identified object is determined.

3. The instrument positioning and navigation device according to claim 2, wherein the instrument arm comprises a rigid rod and a flexible tube connected to each other, characterized in that, Determining the positional relationship between each moving node in the robotic arm and the fixed base point includes: The position of the connection point relative to the fixed base point is obtained; the connection point is the connection point between the rigid rod and the flexible tube; each moving node is located on the flexible tube. The bending angle and rotation angle of the flexible tube are determined by driving sensors; Based on the positional relationship of the connection point relative to the fixed base point, the length, bending angle, and rotation angle of the flexible tube, the positional relationship between each moving node and the fixed base point in the instrument arm is determined.

4. The instrument positioning and navigation device according to claim 1, characterized in that, The device is further configured to, after obtaining the position of the moving node relative to the identified object based on the positional relationship between the moving node and the fixed base point in the robotic arm and the third position transformation model, Based on the position of the mobile node relative to the identified object, the end node of the mobile node is controlled to move to a preset position of the identified object; the end node is the mobile node located at the end of the robotic arm.

5. The instrument positioning and navigation device according to claim 4, characterized in that, The preset position is the position corresponding to the target object in the identification object; The step of controlling the end node of the mobile node to move to a preset position of the identified object based on the position of the mobile node relative to the identified object includes: Based on the position of the end node relative to the identified object, a first distance between the end node and the preset position is determined; When the first distance is greater than the first preset distance threshold, the control arm moves along a straight path until the first distance is less than or equal to the first preset distance threshold, so that the end node moves to the preset position of the identified object; the straight path is the line connecting the end node and the preset position.

6. The instrument positioning and navigation device according to claim 5, characterized in that, The device is further configured to, after determining the first distance between the end node and the preset position based on the position of the end node relative to the identified object, Based on the positional relationship of each mobile node relative to the identified object, a second distance between the robotic arm and the non-target object is determined; the second distance characterizes a preset distance between the mobile nodes on the robotic arm and the non-target object; the non-target object is any object among the identified objects other than the target object. When the first distance is greater than the first preset distance threshold and the second distance is less than the second preset distance threshold, the control arm moves along the obstacle avoidance path; When the robotic arm moves to the end of the obstacle avoidance path and the first distance is greater than the first preset distance threshold, the robotic arm is controlled to move along the straight path until the first distance is less than or equal to the first preset distance threshold, so that the end node moves to the preset position of the identified object.

7. The instrument positioning and navigation device according to claim 6, characterized in that, When the first distance is greater than a first preset distance threshold and the second distance is less than a second preset distance threshold, controlling the robotic arm to move along the obstacle avoidance path includes: When the first distance is greater than the first preset distance threshold and the second distance is less than the second preset distance threshold, the control arm is to stop moving; In response to a control command, the robotic arm is controlled to move along the obstacle avoidance path.

8. The instrument positioning and navigation device according to claim 6, characterized in that, The control of the robotic arm to move along the obstacle avoidance path includes: The obstacle avoidance path is determined based on the outer contour of the non-target object; the distance between the obstacle avoidance path and the outer contour of the non-target object is a preset obstacle avoidance distance; Control the movement of the robotic arm along the obstacle avoidance path.

9. The instrument positioning and navigation device according to claim 8, characterized in that, Determining the obstacle avoidance path based on the outer contour of the non-target object includes: Determine the first and second intersection points of the straight path and the outer contour of the non-target object; the distance between the first intersection point and the end node is less than the distance between the second intersection point and the end node; The starting point of the obstacle avoidance path is determined on the straight path based on the first intersection point; the distance between the starting point and the first intersection point is the preset obstacle avoidance distance; The endpoint of the obstacle avoidance path is determined on the straight path based on the second intersection point; the distance between the endpoint and the second intersection point is the preset obstacle avoidance distance; The obstacle avoidance path is determined based on the starting point and the ending point of the obstacle avoidance path.

10. A device for positioning and navigation of an instrument according to claim 3, characterized in that, The device is also used for: The boundaries of the instrument's activity are determined based on the edges of the identified object; Based on the positional relationship of each mobile node relative to the identified object, the positional relationship of each mobile node relative to the instrument's activity boundary is determined; Based on the positional relationship of each moving node relative to the instrument's active boundary, a third distance is determined between the instrument arm and the instrument's active boundary; the third distance characterizes the shortest distance between the moving node on the instrument arm and the instrument's active boundary. When the third distance is less than a third preset distance threshold, the control arm stops moving.

11. A device for positioning and navigation of an instrument according to claim 10, characterized in that, The device is further configured to, after determining a third distance between the instrument arm and the instrument's active boundary based on the positional relationship of each moving node relative to the instrument's active boundary, When the third distance is less than the fourth preset distance threshold, a repulsive force is applied to the handle of the flexible robot; the fourth preset distance threshold is greater than the third preset distance threshold.

12. The instrument positioning and navigation device according to claim 11, characterized in that, When the third distance is less than a fourth preset distance threshold, applying a repulsive force to the handle of the flexible robot includes: When the third distance is less than the fourth preset distance threshold, the moving node whose distance to the device activity boundary is the third distance is determined as the alarm node. The repulsive force is determined based on the speed of the alarm node, the third distance, and the preset safety threshold. The repulsive force is applied to the handle of the flexible robot.

13. A device positioning and navigation system, characterized in that, include: Instrument arm; The robotic arm includes a moving node and a fixed base point; An endoscope is used to acquire real-time images of the object being identified. A processor and a memory, wherein the memory stores at least one instruction, at least one program, code set, or instruction set, and the processor loads and executes at least one instruction, at least one program, code set, or instruction set to implement the operation process corresponding to the instrument positioning and navigation device of any one of claims 1-12.

14. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, at least one program, code set, or instruction set, and the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor to realize the operation process corresponding to the instrument positioning and navigation system as described in claim 13.

Images