Simulation-based navigation method and device, surgical robot and electronic equipment

Abstract

The invention discloses a navigation method and device based on simulation, a surgical robot and electronic equipment. The method comprises the following steps: acquiring a target three-dimensional image and a target path, wherein the target three-dimensional image comprises a first object; a first three-dimensional model of the first object and a second three-dimensional model of the introducer are obtained, wherein the second three-dimensional model comprises a placement hole model. And obtaining a first pose of the first three-dimensional model in the first coordinate system based on the target three-dimensional image. Based on the first three-dimensional model, the second three-dimensional model and the first pose, a second pose of the second three-dimensional model in the first coordinate system is obtained, and under the condition that the pose of the second three-dimensional model is the second pose, the axis of the placement hole model is aligned with the target path, and the first three-dimensional model does not intersect with the second three-dimensional model. Simulation based on the method can obtain the pose of the introducer meeting the following conditions: the axis of the placement hole in the introducer is aligned with the target path, and the introducer does not collide with the first object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical devices, in particular to a simulation-based navigation method and device, a surgical robot, and an electronic device. BACKGROUND

[0002] In the medical field, after a path from a skin region of a first object to a body of the first object is obtained through planning, a guide can be navigated based on the path to align an axis of a placement hole of the guide with the path, and then a second object can be moved along the axis of the placement hole to move the second object from the skin region of the first object to the body of the first object. Therefore, when the guide is navigated based on the path, how to determine the pose of the guide based on the path is of great significance. SUMMARY

[0003] The present application provides a simulation-based navigation method and device, a surgical robot, and an electronic device to obtain a pose of a guide that satisfies the following conditions through simulation: an axis of a placement hole in the guide is aligned with a target path, and the guide does not collide with a first object.

[0004] In a first aspect, a simulation-based navigation method is provided, and the method includes: obtaining a target three-dimensional image and a target path, the target three-dimensional image including a first object, and the target path being a path for moving from a skin region of the first object to a target point in a body of the first object; obtaining a first three-dimensional model of the first object and a second three-dimensional model of a guide, the second three-dimensional model including a placement hole sub-model, the placement hole sub-model being a model corresponding to a placement hole in the guide, and a second object being movable along an axis of the placement hole when the second object is placed in the placement hole, the second object being an object for moving from the skin region of the first object to the target point in the body of the first object; obtaining a first pose of the first three-dimensional model in a first coordinate system based on the target three-dimensional image, the first coordinate system being an image coordinate system of the target three-dimensional image; obtaining a second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model, and the first pose, the axis of the placement hole sub-model being aligned with the target path when the second pose of the second three-dimensional model is the second pose, and the first three-dimensional model not intersecting the second three-dimensional model.

[0005] According to any one of the embodiments of the present application, the first structure of the guide is connected to the mechanical arm; Before the obtaining the second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model and the first pose, the method further comprises: obtaining a third pose of a third three-dimensional model of the robot arm in the first coordinate system and a fourth pose of the second three-dimensional model in the first coordinate system, wherein the third pose of the third three-dimensional model and the fourth pose of the second three-dimensional model are obtained when the third three-dimensional model does not start to move; obtaining a first orientation relationship based on the third pose and the fourth pose, wherein the first orientation relationship indicates whether a first sub-model in the second three-dimensional model is oriented to a second sub-model in the third three-dimensional model, the first sub-model corresponds to the first structure, and the second sub-model corresponds to a base of the robot arm; The obtaining the second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model and the first pose comprises: obtaining n first candidate poses based on the first three-dimensional model, the second three-dimensional model and the first pose, wherein n is a positive integer, the axis of the placement hole sub-model is aligned with the target path, and the first three-dimensional model and the second three-dimensional model do not intersect when the pose of the second three-dimensional model is the first candidate pose; determining the second pose from the n first candidate poses based on the first orientation relationship, wherein a second orientation relationship indicates whether the first sub-model is oriented to the second sub-model when the pose of the second three-dimensional model is the second pose, and the first orientation relationship is the same as the second orientation relationship.

[0006] In combination with any one of the embodiments of the present application, before the determining the second pose from the n first candidate poses based on the first orientation relationship, the method further comprises: obtaining a first motion range in the first coordinate system, wherein the first motion range is a motion range of the third three-dimensional model; obtaining m second candidate poses based on the first motion range, wherein m is a positive integer, and the pose of the second three-dimensional model is the second candidate pose when the third three-dimensional model moves in the first motion range; The determining the second pose from the n first candidate poses based on the first orientation relationship comprises: determining x third candidate poses from the m second candidate poses based on the first orientation relationship, wherein x is a positive integer less than or equal to m; The second pose is obtained based on the x third candidate poses and the n first candidate poses, and the x third candidate poses and the n first candidate poses all include the second pose.

[0007] In combination with any of the embodiments of the present application, the second pose is determined from the n first candidate poses based on the first orientation relationship, including: y fourth candidate poses are determined from the n first candidate poses based on the first orientation relationship, y is a positive integer less than or equal to n, a third orientation relationship indicates whether the first sub-model is oriented to the second sub-model when the pose of the second three-dimensional model is the fourth candidate pose, and the first orientation relationship is the same as the third orientation relationship; r first motion paths are obtained based on the y fourth candidate poses, the second three-dimensional model moves from the fourth pose to the fourth candidate pose when the third three-dimensional model moves based on the first motion path, and r is a positive integer less than or equal to y; The shortest path in the r first motion paths is determined as a second motion path; The second pose is obtained based on the pose corresponding to the second motion path in the y fourth candidate poses.

[0008] In combination with any of the embodiments of the present application, the r first motion paths are obtained based on the y fourth candidate poses, including: y third motion paths are obtained based on the y fourth candidate poses, the third motion path corresponds to the fourth candidate pose one by one, and the second three-dimensional model moves from the fourth pose to the fourth candidate pose when the third three-dimensional model moves based on the third motion path; y first error tolerance ranges are obtained based on the y third motion paths, the first error tolerance range corresponds to the third motion path one by one, and the first error tolerance range is a range in which the distance to the third motion path is less than or equal to a first threshold value; The number of the third motion paths in the y first error tolerance ranges is respectively determined to obtain y path numbers; The range corresponding to the y path numbers in the y first error tolerance ranges is determined as r second error tolerance ranges, where r is a positive integer less than or equal to y; The paths corresponding to the r second error tolerance ranges in the y third motion paths are determined as the r first motion paths.

[0009] In combination with any of the embodiments of the present application, the r first motion paths are obtained based on the y fourth candidate poses, including: Based on the y fourth candidate poses, y third motion paths are obtained. The third motion paths correspond one-to-one with the fourth candidate poses. When the third 3D model moves based on the third motion path, the second 3D model moves from the fourth pose to the fourth candidate pose. Determine the maximum curvature of the y third motion paths to obtain the y first curvatures; Based on the r curvatures that are the smallest among the y first curvatures, r second curvatures are obtained; Based on the paths corresponding to the r second curvatures among the y third motion paths, the r first motion paths are obtained.

[0010] In any embodiment of this application, the guide is connected to the robotic arm, and the method further includes: Based on the second pose and the first transformation relationship, the third pose of the guide in the second coordinate system is obtained, where the second coordinate system is a coordinate system constructed based on the base of the robotic arm; Based on the third pose, the movement of the robotic arm is controlled so that the pose of the guide in the second coordinate system is the third pose.

[0011] Secondly, a simulation-based navigation device is provided, the simulation-based navigation device comprising: An acquisition unit is used to acquire a target 3D image and a target path. The target 3D image includes a first object, and the target path is a path that moves from the skin region of the first object toward a target point within the first object. The acquisition unit is further configured to acquire a first three-dimensional model of the first object and a second three-dimensional model of the guide. The second three-dimensional model includes a placement hole model, which is a model corresponding to the placement hole in the guide. When the second object is placed in the placement hole, the second object can move along the axis of the placement hole. The second object is an object that moves from the skin area of ​​the first object toward a target point inside the first object. The processing unit is configured to obtain the first pose of the first three-dimensional model in a first coordinate system based on the target three-dimensional image, wherein the first coordinate system is the image coordinate system of the target three-dimensional image; The processing unit is further configured to obtain a second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model, and the first pose. When the pose of the second three-dimensional model is the second pose, the axis of the model for placing the hole is aligned with the target path, and the first three-dimensional model and the second three-dimensional model do not intersect.

[0012] In combination with any of the embodiments of the present application, the first structure of the guide is connected to the mechanical arm; The acquisition unit is further configured to acquire a third pose of a third three-dimensional model of the mechanical arm in the first coordinate system and a fourth pose of the second three-dimensional model in the first coordinate system, wherein the third pose of the third three-dimensional model and the fourth pose of the second three-dimensional model are acquired when the third three-dimensional model is not moving; The processing unit is further configured to obtain a first orientation relationship based on the third pose and the fourth pose, wherein the first orientation relationship indicates whether a first sub-model in the second three-dimensional model is oriented toward a second sub-model in the third three-dimensional model, the first sub-model corresponds to the first structure, and the second sub-model corresponds to a base of the mechanical arm; The processing unit is further configured to obtain n first candidate poses based on the first three-dimensional model, the second three-dimensional model, and the first pose, wherein n is a positive integer, and when the second three-dimensional model is in the first candidate pose, an axis of the placement hole sub-model is aligned with the target path, and the first three-dimensional model does not intersect with the second three-dimensional model; The processing unit is further configured to determine the second pose from the n first candidate poses based on the first orientation relationship, wherein a second orientation relationship indicates whether the first sub-model is oriented toward the second sub-model when the second three-dimensional model is in the second pose, and the first orientation relationship is the same as the second orientation relationship.

[0013] In combination with any of the embodiments of the present application, the acquisition unit is further configured to acquire a first motion range in the first coordinate system, wherein the first motion range is a motion range of the third three-dimensional model; The processing unit is further configured to obtain m second candidate poses based on the first motion range, wherein m is a positive integer, and when the third three-dimensional model moves in the first motion range, the second three-dimensional model is in the second candidate pose; The processing unit is further configured to determine x third candidate poses from the m second candidate poses based on the first orientation relationship, wherein x is a positive integer less than or equal to m; The processing unit is further configured to obtain the second pose based on the x third candidate poses and the n first candidate poses, wherein the x third candidate poses and the n first candidate poses all include the second pose.

[0014] With any of the embodiments of the present application, the processing unit is further configured to determine, based on the first orientation relationship, y fourth candidate poses from the n first candidate poses, y being a positive integer less than or equal to n, a third orientation relationship indicating whether the first sub-model is oriented towards the second sub-model when the second three-dimensional model is in the fourth candidate pose, the first orientation relationship being the same as the third orientation relationship; The processing unit is further configured to obtain, based on the y fourth candidate poses, r first motion paths, the second three-dimensional model moving from the fourth pose to the fourth candidate pose when the third three-dimensional model moves based on the first motion paths, r being a positive integer less than or equal to y; The processing unit is further configured to determine that a shortest path in the r first motion paths is a second motion path; The processing unit is further configured to obtain, based on a pose corresponding to the second motion path from the y fourth candidate poses, the second pose.

[0015] With any of the embodiments of the present application, the processing unit is further configured to: obtain, based on the y fourth candidate poses, y third motion paths, the third motion paths corresponding one-to-one to the fourth candidate poses, the second three-dimensional model moving from the fourth pose to the fourth candidate pose when the third three-dimensional model moves based on the third motion paths; obtain, based on the y third motion paths, y first error tolerance ranges, the first error tolerance ranges corresponding one-to-one to the third motion paths, the first error tolerance range being a range in which a distance to the third motion path is less than or equal to a first threshold value; determine, respectively, a number of the third motion paths in the y first error tolerance ranges, to obtain y path numbers; determine, from the y first error tolerance ranges, r second error tolerance ranges corresponding to r largest numbers from the y path numbers; determine, from the y third motion paths, the r first motion paths corresponding to the r second error tolerance ranges.

[0016] With any of the embodiments of the present application, the processing unit is further configured to: obtain, based on the y fourth candidate poses, y third motion paths, the third motion paths corresponding one-to-one to the fourth candidate poses, the second three-dimensional model moving from the fourth pose to the fourth candidate pose when the third three-dimensional model moves based on the third motion paths; determine a maximum curvature of the y third motion paths, to obtain y first curvatures; obtain r second curvatures based on the minimum r curvatures of the y first curvatures; obtain the r first motion paths based on paths corresponding to the r second curvatures in the y third motion paths.

[0017] With any of the embodiments of the present application, the guide is connected with the mechanical arm, and the processing unit is further configured to: obtain a third pose of the guide in a second coordinate system based on the second pose and the first conversion relationship, the second coordinate system being a coordinate system constructed based on a base of the mechanical arm; control the mechanical arm to move based on the third pose, so that the pose of the guide in the second coordinate system is the third pose.

[0018] In a third aspect, a surgical robot is provided, comprising the simulation-based navigation device of the second aspect. In the third aspect, the surgical robot can perform the simulation-based navigation method by the simulation-based navigation device, and the pose of the guide satisfying the conditions that the axis of the placement hole in the guide is aligned with the target path and the guide does not collide with the first object can be obtained by simulation.

[0019] In a fourth aspect, an electronic device is provided, comprising a processor and a memory, the memory being configured to store computer program code, the computer program code comprising computer instructions, and the electronic device being configured to perform the method of the first aspect and any possible implementation manner thereof when the processor executes the computer instructions.

[0020] In a fifth aspect, another electronic device is provided, comprising a processor, a sending device, an input device, an output device and a memory, the memory being configured to store computer program code, the computer program code comprising computer instructions, and the electronic device being configured to perform the method of the first aspect and any possible implementation manner thereof when the processor executes the computer instructions.

[0021] In a sixth aspect, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, the computer program comprising program instructions, and the program instructions causing a processor to perform the method of the first aspect and any possible implementation manner thereof when the program instructions are executed by the processor.

[0022] In a seventh aspect, a computer program product is provided, the computer program product comprising a computer program or instructions, and the computer program or instructions causing a computer to perform the method of the first aspect and any possible implementation manner thereof when the computer program or instructions are executed on the computer.

[0023] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application, as claimed.

[0024] In the embodiment of the present application, after obtaining the target three-dimensional image and the first three-dimensional model, the navigation device can obtain the first pose of the first three-dimensional model in the first coordinate system based on the target three-dimensional image. Then, the second pose of the second three-dimensional model in the first coordinate system can be obtained based on the first three-dimensional model, the second three-dimensional model and the first pose, wherein, in the case that the pose of the second three-dimensional model is the second pose, the axis of the placed Confucius model is aligned with the target path, and the first three-dimensional model does not intersect with the second three-dimensional model. In this way, the pose of the guide device that meets the following conditions can be obtained in a simulation manner: the axis of the placed hole in the guide device is aligned with the target path, and the guide device does not collide with the first object. BRIEF DESCRIPTION OF DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the embodiments of the present application will be described below.

[0026] The drawings incorporated into the specification and forming a part thereof illustrate embodiments consistent with the present application and, together with the specification, serve to explain the technical solutions of the present application.

[0027] Figure 1 A flowchart of a simulation-based navigation method provided by the embodiment of the present application; Figure 2 A structural diagram of a guide device provided by the embodiment of the present application; Figure 3 A diagram showing that the guide device is connected to the mechanical arm provided by the embodiment of the present application; Figure 4 A diagram showing the first three-dimensional model and the second three-dimensional model provided by the embodiment of the present application; Figure 5 Another diagram showing the first three-dimensional model and the second three-dimensional model provided by the embodiment of the present application; Figure 6 Still another diagram showing the first three-dimensional model and the second three-dimensional model provided by the embodiment of the present application; Figure 7 An interface diagram of simulation navigation provided by the embodiment of the present application; Figure 8 A diagram showing that the first three-dimensional model and the second three-dimensional model collide provided by the embodiment of the present application; Figure 9 A diagram showing that the first three-dimensional model and the second three-dimensional model do not collide provided by the embodiment of the present application; Figure 10Another schematic diagram provided by the embodiment of the present application shows that the first three-dimensional model and the second three-dimensional model do not collide; Figure 11 A schematic diagram of a navigation device based on simulation provided by the embodiment of the present application; Figure 12 A schematic diagram of a hardware structure of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the personnel in the technical field better understand the scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by the person skilled in the art without creative labor are within the scope of protection of the present application.

[0029] The terms "first", "second", and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish different objects, not to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device.

[0030] In this document, the reference to "embodiment" means that the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment is referred to, nor does it mean that the embodiments are mutually exclusive or alternative to each other. The person skilled in the art explicitly and implicitly understands that the embodiments described herein can be combined with other embodiments. It should be understood that in the present application, "at least one" means one or more, "more" means two or more, and "at least two" means two or more.

[0031] The execution subject of the embodiment of the present application is a navigation device based on simulation (hereinafter referred to as navigation device), wherein the navigation device can be any kind of electronic device that can execute the technical scheme disclosed in the method embodiment of the present application. Optionally, the navigation device can be one of the following: computer, flat server.

[0032] It should be understood that the method embodiment of the present application can also be realized by the processor executing the computer program code. The embodiments of the present application will be described below in combination with the drawings in the embodiments of the present application. Please refer toFigure 1 Figure 1 A flowchart of a navigation method based on simulation provided in an embodiment of the present application.

[0033] 101, obtain a target three-dimensional image, a target path.

[0034] In an embodiment of the present application, the target three-dimensional image can be a medical image. Optionally, the target three-dimensional image is one of the following: a three-dimensional ultrasound image, a three-dimensional computed tomography (CT) image.

[0035] The target three-dimensional image includes a first object. Optionally, the first object is a human. The target path is a path from a skin region of the first object to a target point in the first object. In other words, the starting point of the target path is a point in the skin region of the first object, and the end point of the target path is the target point. Exemplarily, the target point is a target point in the first object.

[0036] Optionally, the target path is a path in a first coordinate system, wherein the first coordinate system is an image coordinate system of the target three-dimensional image.

[0037] In one implementation manner of obtaining the target three-dimensional image, the navigation device controls a CT scanning device to scan the first object to obtain the target three-dimensional image. In another implementation manner of obtaining the target three-dimensional image, the navigation device obtains the target three-dimensional image by receiving a target three-dimensional image input by a user through an input component, wherein the input component includes at least one of the following: a mouse, a keyboard, and a touch screen.

[0038] 102, obtain a first three-dimensional model of the first object, and a second three-dimensional model of the introducer.

[0039] In an embodiment of the present application, the three-dimensional model (including the above-mentioned first three-dimensional model, the above-mentioned second three-dimensional model, and the third three-dimensional model and the fourth three-dimensional model to be mentioned below) can be one of the following: a triangular mesh model, a computer aided design (CAD) three-dimensional model, and a point cloud three-dimensional model.

[0040] The second three-dimensional model includes a placement hole sub-model, wherein the placement hole sub-model is a model corresponding to a placement hole in the introducer. In the case that a second object is placed in the placement hole, the second object can move along an axis of the placement hole, wherein the second object is an object moving from a skin region of the first object to a target point in the first object. Exemplarily, the second object is a needle. In the case that the axis of the placement hole is aligned with the target path, the needle is placed in the placement hole, and the needle can be moved from the skin region of the first object to the target point in the first object.

[0041] ​Optionally, the axis of the placement hole is a center line of the placement hole, and the second object is movable along the center line of the placement hole in a case where the second object is placed in the placement hole.

[0042] Exemplarily, Figure 2 A structure diagram of a guide device is provided in an embodiment of the present application, as shown in Figure 2 The guide device includes four optical markers and a placement hole. The optical markers have the ability to reflect light, and an optical tracking device can locate the optical markers based on the light reflected by the optical markers. Figure 2 The axis of the placement hole (i.e., the dashed line in Figure 2 The second object coincides with the axis of the placement hole when the second object is placed in the placement hole.

[0043] 103. Obtain, based on the target three-dimensional image, a first pose of the first three-dimensional model in the first coordinate system.

[0044] Because the target three-dimensional model includes the first object, the pose of the first object in the first coordinate system can be obtained based on the target three-dimensional image, and the pose can be taken as the pose of the first three-dimensional model in the first coordinate system (i.e., the first pose described above).

[0045] In a possible implementation manner, the navigation device performs semantic segmentation on the target three-dimensional image, determines the first object in the target three-dimensional image, and then determines the pose of the first object in the target three-dimensional image, so as to obtain the first pose based on the pose.

[0046] 104. Obtain, based on the first three-dimensional model, the second three-dimensional model, and the first pose, a second pose of the second three-dimensional model in the first coordinate system.

[0047] In an embodiment of the present application, the axis of the placement hole sub-model is aligned with the target path in a case where the pose of the second three-dimensional model is the second pose. That is, the axis of the placement hole sub-model in the second three-dimensional model is aligned with the target path in a case where the pose of the second three-dimensional model in the first coordinate system is the second pose. At this time, if the three-dimensional model corresponding to the second object is placed in the placement hole sub-model, the three-dimensional model corresponding to the second object can be aligned with the target path.

[0048] Optionally, after obtaining the second pose, the navigation device can align the axis of the placement hole sub-model of the second three-dimensional model with the target path by adjusting the pose of the second three-dimensional model to the second pose.

[0049] In some embodiments, the guide is connected to the robot arm. In some embodiments, the guide is moved by moving the robot arm. In some embodiments, the guide is moved by moving the robot arm based on information in a second coordinate system. In some embodiments, the second coordinate system is a coordinate system constructed based on a base of the robot arm. In some embodiments, the guide is moved based on a third pose of the guide in the second coordinate system. In some embodiments, the third pose of the guide in the second coordinate system is determined based on the second pose of the second three-dimensional model in the first coordinate system. In some embodiments, the third pose of the guide in the second coordinate system is determined based on the second pose of the second three-dimensional model in the first coordinate system and the first pose of the first three-dimensional model in the first coordinate system. In some embodiments, the guide is moved based on the third pose of the guide in the second coordinate system to align the axis of the placement hole in the guide with the target path. In some embodiments, the guide is moved based on the third pose of the guide in the second coordinate system to align the axis of the placement hole in the guide with the target path, and the second object is placed in the placement hole in the guide when the axis of the placement hole in the guide is aligned with the target path. In some embodiments, the second object is moved along the target path by moving the second object along the axis of the placement hole in the guide.

[0050] In some embodiments, the first three-dimensional model and the second three-dimensional model do not intersect when the second three-dimensional model has the second pose. In some embodiments, the first three-dimensional model and the second three-dimensional model do not collide when the second three-dimensional model has the second pose. In some embodiments, the guide and the first object do not collide when the axis of the placement hole in the guide is aligned with the target path.

[0051] In some embodiments, the distance between the second three-dimensional model and the start point of the target path is greater than or equal to a distance threshold when the second three-dimensional model has the second pose. In some embodiments, the distance between the second three-dimensional model and the start point of the target path is greater than or equal to a distance threshold when the second three-dimensional model has the second pose, and the guide and the first object do not collide when the axis of the placement hole in the guide is aligned with the target path.

[0052] In some embodiments, the guide is moved based on the third pose of the guide in the second coordinate system to align the axis of the placement hole in the guide with the target path. In some embodiments, the guide is moved based on the third pose of the guide in the second coordinate system to align the axis of the placement hole in the guide with the target path, and the second object is placed in the placement hole in the guide when the axis of the placement hole in the guide is aligned with the target path. In some embodiments, the second object is moved along the target path by moving the second object along the axis of the placement hole in the guide. Figure 1 In some embodiments, the navigation method comprises obtaining a target three-dimensional image and a first three-dimensional model, obtaining a first pose of the first three-dimensional model in a first coordinate system based on the target three-dimensional image, obtaining a second pose of a second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model, the first pose, wherein the axis of the placement hole in the guide is aligned with the target path and the first three-dimensional model and the second three-dimensional model do not intersect when the second three-dimensional model has the second pose. In some embodiments, the guide is moved based on the third pose of the guide in the second coordinate system to align the axis of the placement hole in the guide with the target path. In some embodiments, the guide is moved based on the third pose of the guide in the second coordinate system to align the axis of the placement hole in the guide with the target path, and the second object is placed in the placement hole in the guide when the axis of the placement hole in the guide is aligned with the target path. In some embodiments, the second object is moved along the target path by moving the second object along the axis of the placement hole in the guide.

[0053] In some scenarios, the pose of the guide is uncertain when the axis of the placement hole in the guide is aligned with the target path, specifically, the guide can rotate around the axis of the placement hole when the axis of the placement hole in the guide is aligned with the target path. Therefore, if the pose of the guide is adjusted based on the target path after the target path is planned, so that the axis of the placement hole in the guide is aligned with the target path, the guide may collide with the first object, which may cause the first object to be damaged. In the method provided in the present application, the second pose is obtained by simulation after the target path is obtained, and then the pose of the guide is adjusted based on the second pose, which can reduce the probability of collision between the guide and the first object, thereby improving the safety. Figure 1

[0054] As an optional implementation, the first structure of the guide is connected with the robot arm. Optionally, the first structure is a flange. Before the navigation device performs the step of obtaining the second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model and the first pose, the navigation device further performs the following steps: obtaining a third pose of a third three-dimensional model of the robot arm in the first coordinate system and a fourth pose of the second three-dimensional model in the first coordinate system, wherein the pose of the second three-dimensional model is the fourth pose when the third three-dimensional model does not start to move. Based on the third pose and the fourth pose, a first orientation relationship is obtained, wherein the first orientation relationship indicates whether a first sub-model in the second three-dimensional model is oriented towards a second sub-model in the third three-dimensional model, the first sub-model is a model corresponding to the first structure, and the second sub-model is a model corresponding to the base of the robot arm. At this time, the step of obtaining the second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model and the first pose includes the following steps: obtaining n first candidate poses based on the first three-dimensional model, the second three-dimensional model and the first pose, wherein n is a positive integer, the axis of the placement hole sub-model is aligned with the target path when the pose of the second three-dimensional model is the first candidate pose, and the first three-dimensional model does not intersect with the second three-dimensional model. The second pose is determined from the n first candidate poses based on the first orientation relationship, wherein the second orientation relationship indicates whether the first sub-model is oriented towards the second sub-model when the pose of the second three-dimensional model is the second pose, and the first orientation relationship is the same as the second orientation relationship.

[0055] An exemplary Figure 3 guide connected with a robot arm is provided in the present application. As shown in Figure 3 , the guide is connected with the robot arm through a first structure. Figure 3 The structure of the guide in the present application is the same as that of the guide shown in Figure 2 , and the method provided in the present application is based on Figure 2 ​It can be seen that one end of the guide is a structure including a placement hole, and the other end of the guide is an end face of a cylinder, and in Figure 3 , the first structure is an end face of a cylinder. Moreover, in Figure 3 , the first structure is towards the base of the mechanical arm.

[0056] The third pose is a pose of the mechanical arm when it is returned to zero, that is, each movement of the mechanical arm starts from the third pose. In other words, in the case where the third three-dimensional model does not start to move, the pose of the third three-dimensional model is the third pose. In addition, in the case where the third three-dimensional model does not start to move, the pose of the second three-dimensional model is the fourth pose, that is, the pose of the second three-dimensional model is the fourth pose when the mechanical arm is returned to zero.

[0057] In a possible implementation, after obtaining the first candidate pose, the navigation device can adjust the pose of the second three-dimensional model based on the first candidate pose, and display the second three-dimensional model after the pose is adjusted and the first three-dimensional model. In this way, the user can determine whether the orientations of the first sub-model and the second sub-model are the same as the first orientation relationship. For example, the user can determine whether the orientations of the first sub-model and the second sub-model are the same as the first orientation relationship when the guide is on the left side of the first object based on the actual positional relationship between the mechanical arm and the guide. Or it can be determined whether the orientations of the first sub-model and the second sub-model are the same as the first orientation relationship when the guide is on the right side of the first object. Further, the pose of the second three-dimensional model can be adjusted so that the orientations of the first sub-model and the second sub-model are the same as the first orientation relationship. For example, Figure 4 is a schematic diagram of a first three-dimensional model and a second three-dimensional model provided by an embodiment of the present application, wherein Figure 4 , the pose of the second three-dimensional model in the first coordinate system is the first candidate pose. Figure 5 is another schematic diagram of a first three-dimensional model and a second three-dimensional model provided by an embodiment of the present application, Figure 6 is still another schematic diagram of a first three-dimensional model and a second three-dimensional model provided by an embodiment of the present application. Wherein Figure 5 , the second three-dimensional model in Figure 6 , the second three-dimensional model in Figure 4 , the second three-dimensional model is obtained by adjusting the pose of the second three-dimensional model in

[0058] As described above, by controlling the movement of the mechanical arm, the guide can be moved, and the pose of the guide can be adjusted. If the movement amplitude when the orientation relationship between the first structure and the base changes is referred to as the first movement difficulty, and the movement amplitude when the orientation relationship between the first structure and the base does not change is referred to as the second movement difficulty, then based on Figure 3It can be seen that the first motion difficulty is greater than the second motion difficulty. Specifically, because the orientation relationship between the first structure and the base is changed, the motion of the robot arm needs to satisfy at least one of the following: a greater motion amplitude, a greater pose transformation amount, and a longer motion path, so the first motion difficulty is greater than the second motion difficulty.

[0059] Therefore, during the simulation, the navigation device can obtain the second pose of the second three-dimensional model according to whether the orientation relationship between the model corresponding to the first structure and the model corresponding to the base changes, so that the motion difficulty of the robot arm can be reduced. In this way, the difficulty and accuracy of controlling the motion of the robot arm based on the second pose can be improved.

[0060] In this embodiment, the first orientation relationship is the orientation relationship between the model corresponding to the first structure (i.e., the first sub-model) and the model corresponding to the base (i.e., the second sub-model) when the robot arm is zeroed. The second orientation relationship is the orientation relationship between the first sub-model and the second sub-model when the pose of the second three-dimensional model is the second pose. If the first orientation relationship is the same as the second orientation relationship, the motion difficulty of the third three-dimensional model from the third pose to the terminal pose can be reduced, where the pose of the second three-dimensional model is the second pose in the case that the pose of the third three-dimensional model is the terminal pose. Therefore, the navigation device can first obtain n first candidate poses that can align the axis of the placement hole sub-model with the target path based on the first three-dimensional model, the second three-dimensional model, and the first pose. Then, the navigation device can select, from the n first candidate poses, a pose that makes the first orientation relationship and the second orientation relationship the same as the second pose.

[0061] Optionally, the first orientation relationship and the second orientation relationship being the same means that both the first orientation relationship and the second orientation relationship indicate that the first sub-model is oriented toward the second sub-model, or both the first orientation relationship and the second orientation relationship indicate that the first sub-model is not oriented toward the second sub-model.

[0062] Optionally, the first structure is an end face of the guide, and the first sub-model is a model corresponding to the end face. The first sub-model is oriented toward the second sub-model in the case that the angle between the normal vector of the first sub-model and the reference vector is less than or equal to the second threshold, and the first sub-model is not oriented toward the second sub-model in the case that the angle between the normal vector of the first sub-model and the reference vector is greater than the second threshold, where the reference vector is perpendicular to the bottom surface of the base of the robot arm, the starting point of the reference vector is the center of the bottom surface of the base, and the reference vector points to the base. The robot arm is fixed to the ground, and the bottom surface of the base is in contact with the ground.

[0063] As an optional implementation, before the navigation device performs the step "determine the second pose from n first candidate poses based on the first orientation relationship", the following steps are also performed: Obtaining a first motion range in a first coordinate system, wherein the first motion range is the motion range of the third 3D model. Based on the first motion range, obtaining m second candidate poses, where m is a positive integer, and the pose of the second 3D model is a second candidate pose when the third 3D model moves within the first motion range. Given m second candidate poses, determining the second pose from n first candidate poses based on the first orientation relationship includes the following steps: Based on the first orientation relationship, determining x third candidate poses from the m second candidate poses, where x is a positive integer less than or equal to m. Based on the x third candidate poses and n first candidate poses, obtaining the second pose, wherein both the x third candidate poses and the n first candidate poses include the second pose.

[0064] In this implementation, the first range of motion is determined based on the movable range of the robotic arm, which is determined based on the movable range of each joint of the robotic arm. Since the robotic arm cannot exceed its movable range during actual movement, after acquiring the first range of motion, the navigation device first obtains m second candidate poses based on the first range of motion. When the pose of the third 3D model is any one of the m second candidate poses, the third 3D model is within the first range of motion. After obtaining the m second candidate poses, based on the first orientation relationship, x third candidate poses that satisfy the first orientation relationship are selected from the m second candidate poses. At this point, when the pose of the third 3D model is any one of the x third candidate poses, the third 3D model is within the first range of motion, and the orientation relationship between the first sub-model and the second sub-model is the same as the first orientation relationship. Finally, based on the intersection of x third candidate poses and n first candidate poses, the second pose can be obtained. Thus, the pose of the third 3D model can satisfy the following conditions when it is the second pose: the third 3D model is within the first motion range, the orientation relationship between the first sub-model and the second sub-model is the same as the first orientation relationship, the axis of the placed hole model is aligned with the target path, and the first 3D model and the second 3D model do not intersect.

[0065] As an optional implementation, the second pose is determined from the n first candidate poses based on the first orientation relationship, including the following steps: y fourth candidate poses are determined from the n first candidate poses based on the first orientation relationship, where y is a positive integer less than or equal to n, the third orientation relationship indicates whether the first sub-model is oriented to the second sub-model when the pose of the second three-dimensional model is the fourth candidate pose, and the first orientation relationship is the same as the third orientation relationship. Based on the y fourth candidate poses, r first motion paths are obtained, where the second three-dimensional model moves from the fourth pose to the fourth candidate pose when the third three-dimensional model moves based on the first motion paths, and r is a positive integer less than or equal to y. The shortest path in the r first motion paths is determined as the second motion path. The second pose is obtained based on the pose corresponding to the second motion path in the y fourth candidate poses.

[0066] In this implementation, the navigation device first screens y fourth candidate poses from the n first candidate poses that satisfy the first orientation relationship, and then obtains r first motion paths based on the y fourth candidate poses, where the third three-dimensional model moves based on the first motion paths, so that the pose of the third three-dimensional model moves from the third pose to the fourth candidate pose. Optionally, the first motion path is a structure connected to the first structure in the robot arm, and an example is that the first motion path is the motion path of the flange. Then the shortest path in the r first motion paths is determined as the second motion path, and the second pose is obtained based on the pose corresponding to the second motion path in the y fourth candidate poses. In this way, the robot arm is controlled based on the second pose, which can reduce the motion distance of the robot arm, thereby reducing the energy consumption of the robot arm and reducing the difficulty of controlling the robot arm.

[0067] Optionally, the first object body is pasted with an optical marker, and correspondingly, the first three-dimensional model includes an optical marker sub-model. The navigation device also detects whether the optical marker sub-model is blocked during the movement of the second three-dimensional model and the third three-dimensional model during the simulation, where the light reflected by the optical marker sub-model cannot be detected by the optical tracking three-dimensional model when the optical marker sub-model is blocked. The optical tracking three-dimensional model is a three-dimensional model of an optical tracking device. When the detection of whether the optical marker sub-model is blocked is performed, the second motion path can be determined based on the detection result, so that the third three-dimensional model will not cause the optical marker sub-model to be blocked when moving based on the second motion path.

[0068] For example, Figure 7 An interface diagram for simulating navigation is provided for the embodiments of the present application. As shown in Figure 7As shown, the interface for simulating navigation includes a first three-dimensional model, a second three-dimensional model, a third three-dimensional model, and an optical tracking three-dimensional model, wherein the first three-dimensional model includes an optical marker sub-model. During the simulation, the second three-dimensional model can be moved by controlling the movement of the third three-dimensional model, and then the second three-dimensional model can be moved to the vicinity of the first three-dimensional model. During this process, the navigation device can detect whether the optical marker sub-model is blocked.

[0069] In an alternative embodiment, the navigation device can obtain the second pose in combination with the previous two embodiments. Specifically, the navigation device can perform the following steps before determining the second pose from the n first candidate poses based on the first orientation relationship: obtaining a first motion range in the first coordinate system. Based on the first motion range, m second candidate poses are obtained. Then, during the execution of the step of "determining the second pose from the n first candidate poses based on the first orientation relationship", the following steps are performed: determining x third candidate poses from the m second candidate poses based on the first orientation relationship. Based on the x third candidate poses, s first motion paths are obtained, wherein s is a positive integer less than or equal to x. The shortest path among the s first motion paths is determined as a fourth motion path. Based on the pose corresponding to the fourth motion path among the x third candidate poses, the second pose is obtained.

[0070] As an alternative embodiment, based on the y fourth candidate poses, r first motion paths are obtained, including the following steps: based on the y fourth candidate poses, y third motion paths are obtained, wherein the third motion path corresponds to the fourth candidate pose one by one, and under the condition that the third three-dimensional model moves based on the third motion path, the second three-dimensional model moves from the fourth pose to the fourth candidate pose. Based on the y third motion paths, y first tolerance ranges are obtained, wherein the first tolerance range corresponds to the third motion path one by one, and the first tolerance range is the range whose distance to the third motion path is less than or equal to the first threshold. The number of third motion paths in each first tolerance range is determined respectively to obtain y path numbers. The range corresponding to the r largest numbers in the y path numbers among the y first tolerance ranges is determined as r second tolerance ranges. The paths corresponding to the r second tolerance ranges among the y third motion paths are determined as the r first motion paths.

[0071] On the one hand, because the y third motion paths are obtained based on the fourth candidate pose, the third three-dimensional model can meet the following conditions when moving based on any third motion path: the third three-dimensional model is within the first motion range, the orientation relationship between the first sub-model and the second sub-model is the same as the first orientation relationship, the axis of the placement hole sub-model is aligned with the target path, and the first three-dimensional model and the second three-dimensional model do not intersect. In other words, any third motion path is a reasonable motion path. On the other hand, when the third three-dimensional model moves based on the third motion path, there can be a situation where the actual motion path of the third three-dimensional model deviates from the third motion path. In the case where any third motion path is a reasonable motion path, for any motion path in the y third motion paths, the more the number of other third motion paths around it, the greater the probability that the actual motion path is still a reasonable motion path when the third three-dimensional model moves based on the third motion path, even if the actual motion path deviates from the third motion path. In other words, the more the number of other third motion paths around the third motion path, the larger the fault tolerance space of the third three-dimensional model moving based on the third motion path.

[0072] Based on the above two aspects, in this embodiment, the navigation device first obtains y first fault tolerance ranges based on the y third motion paths. For example, the y third motion paths include third motion path p1 and third motion path p2, and the y first fault tolerance ranges include first fault tolerance range f1 and first fault tolerance range f2, where the first fault tolerance range f1 is a range within 1 centimeter of the third motion path p1, and the first fault tolerance range f2 is a range within 1 centimeter of the third motion path p2. That is, the first fault tolerance range f1 corresponds to the third motion path p1, and the first fault tolerance range f2 corresponds to the third motion path p2. Optionally, the first threshold value is 1 centimeter, that is, the first fault tolerance range is a range within 1 centimeter of the third motion path.

[0073] After obtaining the y first fault-tolerant ranges, the number of third motion paths in each first fault-tolerant range is determined respectively to obtain y path numbers, wherein the greater the path number is, the stronger the fault tolerance of the first fault-tolerant range is, and the greater the fault tolerance space of the third three-dimensional model based on the third motion path corresponding to the first fault-tolerant range is. For example, the y third motion paths include a third motion path p1, a third motion path p2, and a third motion path p3, and the y first fault-tolerant ranges include a first fault-tolerant range f1, a first fault-tolerant range f2, and a first fault-tolerant range f3, wherein the first fault-tolerant range f1 corresponds to the third motion path p1, the first fault-tolerant range f2 corresponds to the third motion path p2, and the first fault-tolerant range f3 corresponds to the third motion path p3. If the first fault-tolerant range f1 and the first fault-tolerant range f2 both include the third motion path p1 and the third motion path p2, and the first fault-tolerant range f3 includes the third motion path p3, then the path number of the first fault-tolerant range f1 and the path number of the first fault-tolerant range f2 are both 2, and the path number of the first fault-tolerant range f3 is 1.

[0074] After obtaining the y path numbers, the navigation device determines the ranges corresponding to the maximum r numbers in the y path numbers in the y first fault-tolerant ranges as r second fault-tolerant ranges, that is, the r second fault-tolerant ranges are the r ranges with the strongest fault tolerance in the y first fault-tolerant ranges. Then, the paths corresponding to the r second fault-tolerant ranges in the y third motion paths are determined as r first motion paths, which can improve the fault tolerance of the r first motion paths, that is, the probability that the actual motion path of the third three-dimensional model is a reasonable motion path when the third three-dimensional model moves based on any first motion path in the r first motion paths.

[0075] As an optional implementation, based on the y fourth candidate poses, the r first motion paths are obtained by the following steps: based on the y fourth candidate poses, y third motion paths are obtained, wherein the third motion path corresponds to the fourth candidate pose one by one, and the second three-dimensional model moves from the fourth pose to the fourth candidate pose in the case that the third three-dimensional model moves based on the third motion path. The maximum curvature of the y third motion paths is determined to obtain y first curvatures. Based on the r smallest curvatures in the y first curvatures, r second curvatures are obtained. Based on the paths corresponding to the r second curvatures in the y first motion paths, the r first motion paths are obtained.

[0076] Because the greater the maximum curvature of the motion path of the third three-dimensional model is, the greater the difficulty of the motion of the third three-dimensional model is, that is, the greater the difficulty of the motion of the mechanical arm is, therefore, in this implementation, after the navigation device obtains the r second curvatures based on the r smallest curvatures in the y first curvatures, the r first motion paths are obtained based on the r second curvatures, which can reduce the difficulty of the motion of the third three-dimensional model based on the first motion path.

[0077] As an optional implementation, the navigation device obtains the r first motion paths based on the previous two implementations. Specifically, the navigation device performs the following steps in the process of performing the step of "obtaining r first motion paths based on the y fourth candidate poses": obtaining y third motion paths based on the y fourth candidate poses. Obtaining y first fault-tolerant ranges based on the y third motion paths. Determining the number of third motion paths in each of the y first fault-tolerant ranges to obtain y path numbers. Determining the ranges corresponding to the t largest numbers in the y path numbers in the y first fault-tolerant ranges as t second fault-tolerant ranges, where t is a positive integer, t is less than or equal to y, and t is greater than or equal to r. Determining the paths corresponding to the t second fault-tolerant ranges in the y third motion paths as t fifth motion paths. Determining the maximum curvatures of the t fifth motion paths to obtain t third curvatures. Obtaining r second curvatures based on the r smallest curvatures in the t third curvatures. Obtaining the r first motion paths based on the paths corresponding to the r second curvatures in the t fifth motion paths.

[0078] As an optional implementation, the guide is connected to the robot arm, and the navigation device further performs the following steps: obtaining a third pose of the guide in a second coordinate system based on the second pose and the first conversion relationship, where the second coordinate system is a coordinate system constructed based on the base of the robot arm. Controlling the robot arm to move based on the third pose, so that the pose of the guide in the second coordinate system is the third pose.

[0079] Since the robot arm moves based on the pose information in the second coordinate system, in this implementation, after obtaining the second pose, the navigation device first converts the second pose to the third pose in the second coordinate system based on the first conversion relationship, and then controls the robot arm to move based on the third pose, so that the pose of the guide in the second coordinate system is the third pose. Thus, the robot arm can be controlled to move based on the second pose.

[0080] In some schemes, the navigation device converts the second pose to a sixth pose in the optical coordinate system based on the second conversion relationship. Optionally, TTargetTracer2CT represents the second pose, TTargetTracer2OTS represents the sixth pose, and registrationMatrix represents the second conversion relationship. Then, TTargetTracer2CT, TTargetTracer2OTS, and registrationMatrix satisfy the following formula: TTargetTracer2OTS = registrationMatrix × TTargetTracer2CT.

[0081] The navigation device obtains a seventh pose of the guide in the optical coordinate system, wherein the seventh pose is obtained by locating the optical marker in the guide by the optical tracking device. Optionally, the poses obtained by the optical tracking device are all poses in the optical coordinate system. Then, based on the sixth pose and the seventh pose, a first transformation matrix is obtained, wherein the first transformation matrix is used to transform the seventh pose into the sixth pose or to transform the sixth pose into the seventh pose. Optionally, the first transformation matrix is denoted as TTargetTracer2CurrentTracer, the sixth pose is denoted as TTargetTracer2OTS, and the seventh pose is denoted as TCurrentTracer2OTS, then TTargetTracer2CurrentTracer, TTargetTracer2OTS, and TCurrentTracer2OTS satisfy the following formula: TTargetTracer2CurrentTracer = TCurrentTracer2OTS - TTargetTracer2OTS. After obtaining the first transformation matrix, the first transformation matrix is converted into the target pose in the second coordinate system based on the relative positional relationship between the guide and the base of the robot arm.

[0082] Optionally, before performing the step of "obtaining the second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model, and the first pose", the navigation device also obtains an eighth pose of a fourth three-dimensional model in the first coordinate system, wherein the fourth three-dimensional model is a three-dimensional model of a bed on which the first object lies. At this time, obtaining the second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model, and the first pose includes: obtaining the second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model, the fourth three-dimensional model, the first pose, and the eighth pose, wherein in the case that the pose of the second three-dimensional model is the second pose, the axis of the placement hole sub-model is aligned with the target path, and the first three-dimensional model does not intersect with the second three-dimensional model and the fourth three-dimensional model. In this way, if the pose of the guide is adjusted based on the second pose so that the axis of the placement hole in the guide is aligned with the target path, then in the case that the axis of the placement hole in the guide is aligned with the target path, the guide does not collide with the first object and the bed.

[0083] Optionally, in the case that the pose of the second three-dimensional model is the second pose, the angle between the axis of the placement hole sub-model and the start point normal vector is less than or equal to a third threshold value, wherein the start point normal vector is the normal vector at the start point of the target path.

[0084] Optionally, the navigation device can also receive a user inputted reference pose, wherein the reference pose is a pose of the second three-dimensional model in the first coordinate system. The navigation device simulates a movement of the third three-dimensional model from the third pose to a pose of the second three-dimensional model in the reference pose based on the first three-dimensional model, the second three-dimensional model, the third three-dimensional model, the first pose, the reference pose, and the third pose, and detects whether a collision occurs and whether the optical marker sub-model is blocked during the movement. The detection result can be displayed to the user.

[0085] Optionally, in the case that the detection result indicates that a collision occurs and / or there is a blocking, the collision and / or the blocking can be highlighted, such as highlighted display or display of text describing the collision and / or the blocking, so as to facilitate the user to adjust the reference pose.

[0086] For example, Figure 8 A schematic diagram of a first three-dimensional model colliding with a second three-dimensional model is provided for the embodiments of the present application. As shown in Figure 8 For example, the first three-dimensional model intersects with the second three-dimensional model, i.e., the first three-dimensional model collides with the second three-dimensional model. At this time, the user can adjust the reference pose so that the first three-dimensional model does not collide with the second three-dimensional model. For example, Figure 9 A schematic diagram of a first three-dimensional model not colliding with a second three-dimensional model is provided for the embodiments of the present application, Figure 10 A schematic diagram of another first three-dimensional model not colliding with a second three-dimensional model is provided for the embodiments of the present application. Wherein, Figure 9 and Figure 10 are results obtained based on the pose of the user adjusting the reference pose. As shown in Figure 9 and Figure 10 For example, the first three-dimensional model does not intersect with the second three-dimensional model, i.e., the first three-dimensional model does not collide with the second three-dimensional model.

[0087] Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0088] If the technical solution of the application involves personal information, the product applying the technical solution of the application has clearly informed the personal information processing rules before processing the personal information and has obtained the personal independent consent. If the technical solution of the application involves sensitive personal information, the product applying the technical solution of the application has obtained the personal independent consent before processing the sensitive personal information and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent sign is set to inform that the personal information collection range has been entered and the personal information will be collected, and if the individual voluntarily enters the collection range, it is regarded as consent to collect the personal information; or in the case of using obvious signs / information to inform the personal information processing rules on the personal information processing device, the personal authorization is obtained through the pop-up information or the individual uploads the personal information by himself / herself; wherein, the personal information processing can include personal information processor, personal information processing purpose, processing method and personal information type and the like.

[0089] The above describes the method of the embodiment of the application in detail, and the device of the embodiment of the application is provided below.

[0090] Please refer to Figure 11 , Figure 11 A structure schematic diagram of a navigation device based on simulation is provided for the embodiment of the application. The navigation device based on simulation 1 comprises an acquisition unit 11 and a processing unit 12, wherein: The acquisition unit 11 is configured to acquire a target three-dimensional image and a target path, the target three-dimensional image comprising a first object, and the target path being a path for moving from a skin region of the first object to a target point in the first object. The acquisition unit 11 is further configured to acquire a first three-dimensional model of the first object and a second three-dimensional model of a guide, the second three-dimensional model comprising a placement hole sub-model, the placement hole sub-model being a model corresponding to a placement hole in the guide, and the second object being movable along an axis of the placement hole when the second object is placed in the placement hole, the second object being an object for moving from the skin region of the first object to the target point in the first object. The processing unit 12 is configured to obtain a first pose of the first three-dimensional model in a first coordinate system based on the target three-dimensional image, the first coordinate system being an image coordinate system of the target three-dimensional image. The processing unit 12 is further configured to obtain a second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model and the first pose, and the axis of the placement hole sub-model is aligned with the target path and the first three-dimensional model does not intersect with the second three-dimensional model when the pose of the second three-dimensional model is the second pose.

[0091] In any of the embodiments of the present application, the first structure of the guide is connected to the mechanical arm; The acquisition unit 11 is further configured to acquire a third pose of a third three-dimensional model of the mechanical arm in the first coordinate system and a fourth pose of the second three-dimensional model in the first coordinate system, wherein the pose of the third three-dimensional model is the third pose and the pose of the second three-dimensional model is the fourth pose when the third three-dimensional model does not start moving; The processing unit 12 is further configured to obtain a first orientation relationship based on the third pose and the fourth pose, wherein the first orientation relationship indicates whether a first sub-model in the second three-dimensional model is oriented to a second sub-model in the third three-dimensional model, the first sub-model is a model corresponding to the first structure, and the second sub-model is a model corresponding to a base of the mechanical arm; The processing unit 12 is further configured to obtain n first candidate poses based on the first three-dimensional model, the second three-dimensional model and the first pose, wherein n is a positive integer, the axis of the placement hole sub-model is aligned with the target path, and the first three-dimensional model does not intersect with the second three-dimensional model when the pose of the second three-dimensional model is the first candidate pose; The processing unit 12 is further configured to determine the second pose from the n first candidate poses based on the first orientation relationship, wherein a second orientation relationship indicates whether the first sub-model is oriented to the second sub-model when the pose of the second three-dimensional model is the second pose, and the first orientation relationship is the same as the second orientation relationship.

[0092] In any of the embodiments of the present application, the acquisition unit 11 is further configured to acquire a first motion range in the first coordinate system, wherein the first motion range is a motion range of the third three-dimensional model; The processing unit 12 is further configured to obtain m second candidate poses based on the first motion range, wherein m is a positive integer, and the pose of the second three-dimensional model is the second candidate pose when the third three-dimensional model moves in the first motion range; The processing unit 12 is further configured to determine x third candidate poses from the m second candidate poses based on the first orientation relationship, wherein x is a positive integer less than or equal to m; The processing unit 12 is further configured to obtain the second pose based on the x third candidate poses and the n first candidate poses, wherein the x third candidate poses and the n first candidate poses all include the second pose.

[0093] In combination with any of the embodiments of the present application, the processing unit 12 is further configured to determine, based on the first orientation relationship, y fourth candidate poses from the n first candidate poses, y being a positive integer less than or equal to n, a third orientation relationship indicating whether the first sub-model is oriented towards the second sub-model when the pose of the second three-dimensional model is the fourth candidate pose, the first orientation relationship being the same as the third orientation relationship; The processing unit 12 is further configured to obtain, based on the y fourth candidate poses, r first motion paths, the second three-dimensional model moving from the fourth pose to the fourth candidate pose when the third three-dimensional model moves based on the first motion paths, r being a positive integer less than or equal to y; The processing unit 12 is further configured to determine that the shortest path in the r first motion paths is a second motion path; The processing unit 12 is further configured to obtain, based on the pose corresponding to the second motion path in the y fourth candidate poses, the second pose.

[0094] In combination with any of the embodiments of the present application, the processing unit 12 is further configured to: obtain, based on the y fourth candidate poses, y third motion paths, the third motion paths corresponding one-to-one to the fourth candidate poses, the second three-dimensional model moving from the fourth pose to the fourth candidate pose when the third three-dimensional model moves based on the third motion paths; obtain, based on the y third motion paths, y first error tolerance ranges, the first error tolerance ranges corresponding one-to-one to the third motion paths, the first error tolerance range being a range in which the distance to the third motion path is less than or equal to a first threshold value; determine the number of the third motion paths in the y first error tolerance ranges respectively, to obtain y path numbers; determine that the ranges corresponding to the r largest numbers in the y path numbers in the y first error tolerance ranges are r second error tolerance ranges; determine that the paths corresponding to the r second error tolerance ranges in the y third motion paths are the r first motion paths.

[0095] In combination with any of the embodiments of the present application, the processing unit 12 is further configured to: obtain, based on the y fourth candidate poses, y third motion paths, the third motion paths corresponding one-to-one to the fourth candidate poses, the second three-dimensional model moving from the fourth pose to the fourth candidate pose when the third three-dimensional model moves based on the third motion paths; determine the maximum curvature of the y third motion paths to obtain y first curvatures; Based on the r curvatures that are the smallest among the y first curvatures, r second curvatures are obtained; Based on the paths corresponding to the r second curvatures among the y third motion paths, the r first motion paths are obtained.

[0096] In any embodiment of this application, the guide is connected to the robotic arm, and the processing unit 12 is further configured to: Based on the second pose and the first transformation relationship, the third pose of the guide in the second coordinate system is obtained, where the second coordinate system is a coordinate system constructed based on the base of the robotic arm; Based on the third pose, the movement of the robotic arm is controlled so that the pose of the guide in the second coordinate system is the third pose.

[0097] In this embodiment, after acquiring a target 3D image and a first 3D model, the navigation device can obtain a first pose of the first 3D model in a first coordinate system based on the target 3D image. Then, based on the first 3D model, a second 3D model, and the first pose, a second pose of the second 3D model in the first coordinate system can be obtained. When the pose of the second 3D model is the second pose, the axis of the hole placement model is aligned with the target path, and the first 3D model and the second 3D model do not intersect. Thus, through simulation, the pose of the guide that satisfies the following conditions can be obtained: the axis of the hole placement in the guide is aligned with the target path, and the guide does not collide with the first object.

[0098] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0099] Figure 12 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled together via connectors, which include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection in a specific way, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0100] The processor 21 can be one or more graphics processing units (GPUs), which can be single-core GPUs or multi-core GPUs. Alternatively, the processor 21 can be a processor group composed of multiple GPUs, and the multiple processors are coupled with each other through one or more buses. Alternatively, the processor can also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0101] The memory 22 can be used to store computer program instructions, and various computer program codes for executing the scheme of the present application. Alternatively, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0102] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices, or can be an integral device.

[0103] It can be understood that, in the embodiments of the present application, the memory 22 can be used not only to store related instructions, but also to store related data, and the embodiments of the present application do not limit the data stored in the memory.

[0104] It can be understood that, Figure 12 Only a simplified design of an electronic device is shown. In actual applications, the electronic device can also include other necessary elements, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0105] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0106] Those skilled in the art can clearly understand the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments for the convenience and brevity of description, which will not be repeated here. Those skilled in the art can also clearly understand that each embodiment of the present application describes each focus, and for the convenience and brevity of description, the same or similar parts can not be described in different embodiments. Therefore, the parts not described or not described in detail in an embodiment can be referred to the description of other embodiments.

[0107] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0108] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0109] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0110] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable apparatus. The computer instructions can be stored in or transmitted by a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through a wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that includes one or more available media sets. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a digital versatile disc (DVD)), or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0111] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be instructed by a computer program to complete the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments. The storage medium includes a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

Claims

1. A simulation-based navigation method, characterized in that, The method includes: Acquire a target 3D image and a target path, wherein the target 3D image includes a first object and the target path is a path that moves from the skin region of the first object toward a target point within the first object; Obtain a first three-dimensional model of the first object and a second three-dimensional model of the guide. The second three-dimensional model includes a placement hole model, which is a model corresponding to the placement hole in the guide. When the second object is placed in the placement hole, the second object can move along the axis of the placement hole. The second object is an object that moves from the skin area of ​​the first object toward a target point inside the first object. Based on the target 3D image, the first pose of the first 3D model in the first coordinate system is obtained, where the first coordinate system is the image coordinate system of the target 3D image; Based on the first 3D model, the second 3D model, and the first pose, the second pose of the second 3D model in the first coordinate system is obtained. When the pose of the second 3D model is the second pose, the axis of the model where the hole is placed is aligned with the target path, and the first 3D model and the second 3D model do not intersect.

2. The method according to claim 1, characterized in that, The first structure of the guide is connected to the robotic arm; Before obtaining the second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model, and the first pose, the method further includes: The third pose of the third 3D model of the robotic arm in the first coordinate system and the fourth pose of the second 3D model in the first coordinate system are obtained. When the third 3D model has not started to move, the pose of the third 3D model is the third pose and the pose of the second 3D model is the fourth pose. Based on the third pose and the fourth pose, a first orientation relationship is obtained. The first orientation relationship indicates whether the first sub-model in the second three-dimensional model is facing the second sub-model in the third three-dimensional model. The first sub-model is the model corresponding to the first structure, and the second sub-model is the model corresponding to the base of the robotic arm. The step of obtaining the second pose of the second 3D model in the first coordinate system based on the first 3D model, the second 3D model, and the first pose includes: Based on the first 3D model, the second 3D model, and the first pose, n first candidate poses are obtained, where n is a positive integer. When the pose of the second 3D model is the first candidate pose, the axis of the placement of the Confucius model is aligned with the target path, and the first 3D model and the second 3D model do not intersect. Based on the first orientation relationship, the second pose is determined from the n first candidate poses. The second orientation relationship indicates whether the first sub-model is facing the second sub-model when the pose of the second 3D model is the second pose. The first orientation relationship is the same as the second orientation relationship.

3. The method according to claim 2, characterized in that, Before determining the second pose from the n first candidate poses based on the first orientation relationship, the method further includes: Obtain the first range of motion in the first coordinate system, where the first range of motion is the range of motion of the third three-dimensional model; Based on the first motion range, m second candidate poses are obtained, where m is a positive integer. When the third 3D model moves within the first motion range, the pose of the second 3D model is the second candidate pose. The step of determining the second pose from the n first candidate poses based on the first orientation relationship includes: Based on the first orientation relationship, x third candidate poses are determined from the m second candidate poses, where x is a positive integer less than or equal to m; The second pose is obtained based on the x third candidate poses and the n first candidate poses, wherein the x third candidate poses and the n first candidate poses all include the second pose.

4. The method according to claim 2, characterized in that, The step of determining the second pose from the n first candidate poses based on the first orientation relationship includes: Based on the first orientation relationship, y fourth candidate poses are determined from the n first candidate poses, where y is a positive integer less than or equal to n. The third orientation relationship indicates whether the first sub-model is oriented towards the second sub-model when the pose of the second 3D model is the fourth candidate pose. The first orientation relationship is the same as the third orientation relationship. Based on the y fourth candidate poses, r first motion paths are obtained. When the third 3D model moves based on the first motion paths, the second 3D model moves from the fourth pose to the fourth candidate pose, where r is a positive integer less than or equal to y. The shortest path among the r first motion paths is determined as the second motion path; The second pose is obtained based on the pose corresponding to the second motion path among the y fourth candidate poses.

5. The method according to claim 4, characterized in that, The process of obtaining r first motion paths based on the y fourth candidate poses includes: Based on the y fourth candidate poses, y third motion paths are obtained. The third motion paths correspond one-to-one with the fourth candidate poses. When the third 3D model moves based on the third motion path, the second 3D model moves from the fourth pose to the fourth candidate pose. Based on the y third motion paths, y first fault tolerance ranges are obtained. The first fault tolerance range corresponds one-to-one with the third motion path. The first fault tolerance range is the range where the distance to the third motion path is less than or equal to a first threshold. The number of the third motion paths within each of the y first fault tolerance ranges is determined to obtain the number of y paths; The ranges corresponding to the r largest quantities among the y first fault tolerance ranges are determined as r second fault tolerance ranges; The path that corresponds to the r second fault tolerance range among the y third motion paths is determined as the r first motion paths.

6. The method according to claim 4, characterized in that, The process of obtaining r first motion paths based on the y fourth candidate poses includes: Based on the y fourth candidate poses, y third motion paths are obtained. The third motion paths correspond one-to-one with the fourth candidate poses. When the third 3D model moves based on the third motion path, the second 3D model moves from the fourth pose to the fourth candidate pose. Determine the maximum curvature of the y third motion paths to obtain the y first curvatures; Based on the r curvatures that are the smallest among the y first curvatures, r second curvatures are obtained; Based on the paths corresponding to the r second curvatures among the y third motion paths, the r first motion paths are obtained.

7. The method according to any one of claims 1 to 6, characterized in that, The guide is connected to the robotic arm, and the method further includes: Based on the second pose and the first transformation relationship, the third pose of the guide in the second coordinate system is obtained, where the second coordinate system is a coordinate system constructed based on the base of the robotic arm; Based on the third pose, the movement of the robotic arm is controlled so that the pose of the guide in the second coordinate system is the third pose.

8. A simulation-based navigation device, characterized in that, The simulation-based navigation device includes: An acquisition unit is used to acquire a target 3D image and a target path. The target 3D image includes a first object, and the target path is a path that moves from the skin region of the first object toward a target point within the first object. The acquisition unit is further configured to acquire a first three-dimensional model of the first object and a second three-dimensional model of the guide. The second three-dimensional model includes a placement hole model, which is a model corresponding to the placement hole in the guide. When the second object is placed in the placement hole, the second object can move along the axis of the placement hole. The second object is an object that moves from the skin area of ​​the first object toward a target point inside the first object. The processing unit is configured to obtain the first pose of the first three-dimensional model in a first coordinate system based on the target three-dimensional image, wherein the first coordinate system is the image coordinate system of the target three-dimensional image; The processing unit is further configured to obtain a second pose of the second three-dimensional model in the first coordinate system based on the first three-dimensional model, the second three-dimensional model, and the first pose. When the pose of the second three-dimensional model is the second pose, the axis of the model for placing the hole is aligned with the target path, and the first three-dimensional model and the second three-dimensional model do not intersect.

9. A surgical robot, characterized in that, Including the simulation-based navigation device as described in claim 8.

10. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 7.

Images