Positioning method and apparatus, electronic device, and computer-readable storage medium

Abstract

The present disclosure provides a positioning method, device, electronic equipment and computer readable storage medium, relating to the technical field of surgical robots, which comprises: acquiring a current image feature of a surgical tool at a current time, calculating an expected image feature of the surgical tool at a terminal point of a navigation path of the surgical tool under a world coordinate system and a navigation path coordinate system established by the terminal point, calculating a pose control amount of a surgical robot at a next time based on the current image feature, adjusting the pose of the surgical robot through the pose control amount at the next time, acquiring a next-time image feature of the surgical tool, and determining that the positioning of the surgical tool is completed when an actual image feature error between the next-time image feature and the expected image feature is less than a preset error threshold, otherwise, recalculating the expected image feature until the actual image feature error is less than the preset error threshold. The present disclosure improves the positioning speed of the surgical robot.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of surgical robots, and particularly relates to a positioning method and device, electronic equipment and a computer readable storage medium. BACKGROUND

[0002] Surgical robotics is a new cross-discipline subject integrating clinical medicine, biomedical engineering, computer, robotics and other disciplines. The image features designed and used for the positioning of the existing surgical robots mainly include the three-dimensional coordinates of the marker ball on the surgical tool in the optical positioning system, without considering the direction of the surgical tool, thereby leading to slow convergence speed of the calculated image feature error and reducing the positioning speed of the surgical robot.

[0003] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0004] The present disclosure provides a positioning method, device, electronic equipment and computer readable storage medium, which can at least improve the positioning speed of the surgical robot.

[0005] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a positioning method is provided, applied to a surgical robot system, the surgical robot system comprising a surgical robot, a surgical tool and an optical positioning system, the surgical tool being installed on the surgical robot, the positioning method comprising:

[0007] obtaining a current image feature of the surgical tool at a first position at a current time;

[0008] determining an expected image feature of the surgical tool reaching an end point of a navigation path of the surgical tool according to the navigation path and a first coordinate system established by the end point, wherein the navigation path is in a second coordinate system of the optical positioning system;

[0009] determining a pose control amount of the surgical robot at a next time based on the current image feature, and adjusting the pose of the surgical robot according to the pose control amount at the next time, so as to move the surgical tool to a second position;

[0010] obtaining a next time image feature of the surgical tool at the second position;

[0011] The method further comprises: determining a desired image feature of the surgical tool at the end point of the navigation path of the surgical tool according to a first coordinate system established by the end point of the navigation path of the surgical tool; and determining that the positioning of the surgical tool is completed until an actual image feature error between the image feature of the surgical tool at the next time point and the desired image feature of the surgical tool at the end point of the navigation path of the surgical tool is less than a preset error threshold.

[0012] Optionally, the step of determining the desired image feature of the surgical tool at the end point of the navigation path of the surgical tool according to the first coordinate system established by the end point of the navigation path of the surgical tool comprises: obtaining position coordinates of the end point and the start point of the navigation path of the surgical tool in the second coordinate system; and determining the desired image feature according to the origin of the first coordinate system and the position coordinates.

[0013] Optionally, before the step of obtaining the current image feature of the surgical tool at the first position at the current time point, the positioning method further comprises: constructing an initial image Jacobian matrix of the surgical tool; and rearranging the initial image Jacobian matrix into a column vector to obtain an initial system state vector of the surgical robot system.

[0014] Optionally, the step of constructing the initial image Jacobian matrix of the surgical tool comprises: controlling the surgical robot to reach an initial pose corresponding to initial set pose parameters, and obtaining an initial image feature of the surgical tool at the initial pose; wherein the initial set pose parameters comprise i elements, and i is greater than or equal to 6; adding a preset offset to each element in the initial set pose parameters to obtain i offset pose parameters; controlling the surgical robot to reach an offset pose corresponding to the i-th offset pose parameter; obtaining an i-th intermediate image feature of the surgical tool based on the offset pose corresponding to the i-th offset pose parameter; determining an i-th feature difference value between the i-th intermediate image feature and the initial image feature to obtain i feature difference values; and constructing the initial image Jacobian matrix of the surgical tool according to the i feature difference values and the preset offset.

[0015] Optionally, the step of determining the pose control amount of the surgical robot at the next moment based on the current image features includes: determining the difference between the current image features and the expected image features to obtain the current image feature error corresponding to the surgical tool; determining the current pose and pose change of the surgical robot based on the current moment; obtaining the image Jacobian matrix of the surgical tool at the previous moment; determining the image Jacobian matrix of the surgical tool at the current moment based on the initial system state vector; and determining the pose control amount at the next moment based on the pose change, the image Jacobian matrix at the previous moment, the image Jacobian matrix at the current moment, the current image feature error, and the current pose.

[0016] Optionally, the step of determining the image Jacobian matrix of the surgical tool at the current moment based on the initial system state vector includes: determining the current system observation matrix of the surgical robot system according to the pose change; rearranging the image Jacobian matrix of the previous moment into a column vector according to row order to obtain the system state vector of the surgical robot system at the previous moment; obtaining the system observations, process noise covariance matrix, measurement noise covariance matrix, the first square root of the error covariance matrix of the previous moment, and the second square root of the error covariance matrix of the current moment of the surgical robot system; determining the system state vector of the surgical robot system at the current moment based on the square root capacitive Kalman filter, the initial system state vector, the system state vector of the previous moment, the first square root, the current system observation matrix, the system observations, the process noise covariance matrix, the measurement noise covariance matrix, and the second square root; and rearranging the system state vector at the current moment to obtain the image Jacobian matrix at the current moment.

[0017] Optionally, the step of obtaining the image features of the surgical tool at the second position at the next moment includes: constructing a fourth coordinate system based on the needle tip position of the surgical tool; obtaining the needle tip coordinates of the surgical tool in the second coordinate system; determining the unit coordinates of unit points on each coordinate axis of the fourth coordinate system, and transforming each unit coordinate to obtain the transformed coordinates of each unit point in the second coordinate system; and determining the image features at the next moment based on the needle tip coordinates and the obtained transformed coordinates.

[0018] Optionally, the step of constructing a fourth coordinate system based on the needle tip position of the surgical tool includes: taking the needle tip position of the surgical tool as the origin of the fourth coordinate system; taking the direction of the needle tip of the surgical tool as the X-axis of the fourth coordinate system; taking the normal vector of the plane of the surgical tool as the Z-axis of the fourth coordinate system; and taking the cross product of the X-axis and the Z-axis as the Y-axis of the fourth coordinate system.

[0019] According to another aspect of this disclosure, a positioning device is provided, configured in a surgical robot system, the surgical robot system including a surgical robot, surgical instruments, and an optical positioning system, wherein the surgical instruments are mounted on the surgical robot; the positioning device includes:

[0020] The first calculation module is used to obtain the current image features of the surgical tool at the first position at the current moment;

[0021] The second calculation module is used to determine the desired image features of the surgical tool at the endpoint based on the navigation path of the surgical tool and a first coordinate system established by the endpoint of the navigation path; wherein the navigation path is in the second coordinate system where the optical positioning system is located.

[0022] The pose adjustment module is used to determine the pose control amount of the surgical robot at the next moment based on the current image features, and adjust the pose of the surgical robot according to the pose control amount at the next moment so that the surgical tool moves to the second position.

[0023] The third calculation module is used to obtain the image features of the surgical tool at the second position at the next moment.

[0024] The error comparison module is used to return to the step of determining the expected image features of the surgical tool at the endpoint based on the navigation path of the surgical tool and the first coordinate system established by the endpoint of the navigation path, until the actual image feature error between the image features at the next moment and the expected image features is less than a preset error threshold, and then determine that the positioning of the surgical tool is completed.

[0025] According to another aspect of this disclosure, an electronic device is provided, the electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the positioning method as described in the above embodiments.

[0026] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the positioning method as described in the above embodiments.

[0027] The positioning method, apparatus, electronic device, and computer-readable storage medium provided in the embodiments of this disclosure have the following technical effects:

[0028] This disclosure employs a technical solution that involves acquiring the current image features of the surgical tool at a first position, determining the desired image features of the surgical tool at the destination based on the navigation path of the surgical tool in the second coordinate system of the optical positioning system and the first coordinate system established by the endpoint of the navigation path, determining the pose control quantity of the surgical robot at the next moment based on the current image features, adjusting the pose of the surgical robot according to the pose control quantity at the next moment to move the surgical tool to a second position, acquiring the image features of the surgical tool at the next moment in the second position, returning to execute the determination of the desired image features of the surgical tool at the destination based on the navigation path of the surgical tool in the second coordinate system of the optical positioning system and the first coordinate system established by the endpoint of the navigation path, until the actual image feature error between the image features at the next moment and the desired image features is less than a preset error threshold, thus determining that the positioning of the surgical tool is complete. By judging whether the surgical tool has reached the endpoint of the navigation path through image feature error, the convergence speed of image feature error is accelerated, making the positioning of the surgical robot more accurate and less time-consuming, and the motion trajectory during positioning is more ideal, close to a straight line, reducing randomness.

[0029] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0030] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0031] Figure 1 A surgical robot system provided in an embodiment of this disclosure is shown;

[0032] Figure 2 A flowchart illustrating an exemplary embodiment of the positioning method of this disclosure is shown;

[0033] Figure 3 A schematic diagram of the surgical instruments and navigation path is shown;

[0034] Figure 4 An exemplary flowchart of S120 in the positioning method provided in this embodiment of the present disclosure is shown;

[0035] Figure 5An exemplary flowchart illustrating the calculation of surgical robot pose control quantities in the positioning method provided in this embodiment of the present disclosure is shown.

[0036] Figure 6 An exemplary flowchart corresponding to S140 in the positioning method of this disclosure is shown;

[0037] Figure 7 A schematic diagram of the positioning device provided in an embodiment of this disclosure is shown;

[0038] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this disclosure clearer, the embodiments of this disclosure will be described in further detail below with reference to the accompanying drawings.

[0040] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0041] It should be noted that the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0042] Surgical robotics is an emerging interdisciplinary field integrating clinical medicine, biomedical engineering, computer science, and robotics. Generally, vision-based robotic systems are divided into two categories: "eye-in-hand" (EIH) systems and "eye-to-hand" (ETH) systems. Optically guided surgical robots typically employ the ETH model, acquiring global visual information about the robot and its working environment. Currently, there are two main types of optically guided robot control: one is the hand-eye calibration method, where obtaining the spatial transformation relationship between the optical positioning system and the robotic arm is a prerequisite for hand-eye coordination. Through hand-eye calibration, the spatial relationships between the robot base and the optical positioning system, and between the robot's end effector and the surgical instruments, can be obtained. The accuracy of target point localization in surgical robots mainly depends on the accuracy of hand-eye calibration. The other is the calibration-free visual servoing method, which does not require hand-eye calibration. Target localization is achieved by controlling the robot to move in a direction that approximates and converges to visual features.

[0043] The existing technology of using surgical robots to achieve target localization of surgical tools has the following problems:

[0044] 1. The image features designed and used in the existing technology are the three-dimensional coordinates of four marker balls on the surgical tool in the optical positioning system. Since these four points are on the same plane, the feature stability is not high, which may cause the online estimation algorithm to fail to converge due to image feature errors. As a result, the surgical robot deviates from the target and the positioning fails.

[0045] 2. The convergence speed of image feature errors is slow, resulting in slow target localization speed and relatively strong randomness of motion trajectory in actual surgical robots.

[0046] 3. The CKF (Culture Kalman Filter) algorithm is used to analyze the Jacobian matrix of the image. However, the state error covariance matrix in CKF needs to be symmetric and positive definite. The variables in the state filtering problem of nonlinear dynamic systems may be ill-conditioned. Each iteration of CKF may destroy the symmetry and positive definiteness of the covariance matrix, and the loss of positive definiteness will cause the CKF algorithm to terminate.

[0047] In view of the relevant technical problems existing in the prior art, this disclosure provides a positioning method, device, electronic device and computer-readable storage medium to solve the aforementioned technical problems.

[0048] This localization method, on the one hand, calculates the current image features and desired image features along the path of the surgical tool. Compared with existing technologies, this localization method has better stability and faster convergence of image feature errors. On the other hand, when calculating the features on the surgical tool and the path, the orientation constraint of the surgical tool is considered to ensure that the marker ball on the tool is always facing the optical localization system, thereby improving system safety.

[0049] On the other hand, by introducing square root filtering into CKF, we obtain the Square-root Cubature Kalman Filter (SCKF). SCKF uses the least squares method to calculate the Kalman gain and updates the covariance by decomposing or triangulating the matrix, avoiding explicit matrix inversion. The algorithm loop always ensures the symmetry and non-negativity of the covariance, thus avoiding situations that may eventually lead to algorithm instability or even divergence.

[0050] The following is an embodiment of the positioning method provided in this disclosure.

[0051] This disclosure provides a positioning method according to an embodiment, which is applied to a surgical robot system. For example... Figure 1 As shown, Figure 1A surgical robot system according to an embodiment of this disclosure is illustrated. The surgical robot system includes a surgical robot 100, a tool holder 200, surgical instruments 300, several marker balls 201, and an optical positioning system 400. The surgical robot 100 is a six-degree-of-freedom robotic arm used to hold and move surgical instruments, replacing the human hand in performing surgical operations. The marker balls 201 are small spheres with a special reflective layer on their surface, capable of reflecting near-infrared light, and are typically mounted on surgical instruments or targets for optical positioning. The optical positioning system 400 is a binocular camera capable of emitting and receiving near-infrared light, working in conjunction with the marker balls for target positioning. The tool holder 200 has four non-collinear marker balls for gripping the surgical instruments 300. The surgical instruments 300 are mounted on the surgical robot 100, i.e., the tool holder 200 is mounted at the end of the six-degree-of-freedom robotic arm. The surgical robot 100 grips the surgical instruments 300 via the tool holder 200. The surgical instruments 300 are, for example, surgical navigation tools.

[0052] The above-mentioned positioning method is based on image-based uncalibrated visual servoing (IBVS). Image-based uncalibrated visual servoing does not require precise camera calibration parameters or robot kinematic models. The control law is based on image feature errors, which gradually approach zero as the surgical robot moves, and finally completes the visual servoing task.

[0053] In image-based uncalibrated visual servoing, the hand-eye relationship can be described by the image Jacobian matrix:

[0054]

[0055]

[0056] In the above formula, f represents the image features, p represents the end-effector pose in the robotic arm's task space, which refers to the spatial coordinate system where the base of the six-DOF robotic arm is located, or the spatial coordinate system where the surgical robot's base is located. J p (p) is the image Jacobian matrix. The image Jacobian matrix reflects the difference mapping relationship from the robot task space to the image feature space, and online estimation of the image Jacobian matrix is ​​a key problem in IBVS (visual servoing) systems.

[0057] The selection and extraction of image features are particularly important, significantly impacting the performance of visual servoing systems. Most visual servoing systems use the geometric features of the target object, such as points, lines, contraction angles, circles, or combinations thereof. Some also use global features, such as Fourier descriptors and image moments. Compared to traditional visual servoing systems, the surgical robot system disclosed herein is primarily applied in surgical scenarios, and with the aid of an optical positioning system and a marker ball, it can obtain target position information more accurately and effectively. Based on this, the image features used in the surgical robot system of this disclosure are as follows: A coordinate system for the surgical tool and a coordinate system for the navigation path are established separately. The spatial positions of four points on the coordinate system are used as actual or desired image features. These four points include the three coordinate axes and the tip of the surgical tool, f∈R. 12 .

[0058] like Figure 2 As shown, Figure 2 A flowchart illustrating an exemplary embodiment of the positioning method of this disclosure is shown. The positioning method includes the following scheme:

[0059] S110: Obtain the current image features of the surgical tool at the first position at the current moment.

[0060] In one exemplary embodiment, the current image features of the surgical tool at the first position at the current moment are obtained, denoted as f. k The first position refers to the location of the surgical instrument at the current moment, and the current image feature refers to the image features at the current location of the surgical instrument. The calculation process of the current image feature is the same as the calculation process of the image feature at the next moment, which is described below. Please refer to the calculation process of the image feature at the next moment for details.

[0061] S120: Based on the navigation path of the surgical tool and a first coordinate system established by the endpoint of the navigation path, determine the desired image features of the surgical tool at the endpoint; wherein the navigation path is in the second coordinate system of the optical positioning system.

[0062] like Figure 3 As shown, Figure 3 A schematic diagram of the surgical instruments and navigation path is shown. The navigation path of the surgical instruments is in the second coordinate system where the optical positioning system is located. The second coordinate system refers to the real-world coordinate system, denoted as S. w The navigation path is determined by the starting point and the destination, and points from the starting point to the destination. In other words, the navigation path is a path from the starting point to the destination. The starting point is a certain location outside the human body, which can be understood as a safety point and can be selected according to the actual situation. The destination can be understood as the target point, which is a certain location on the human body to be reached.

[0063] A first coordinate system is established with the endpoint (target point) of the navigation path as the origin. This first coordinate system can be understood as the navigation path coordinate system. After the first coordinate system is constructed, the desired image features at the endpoint of the navigation path are calculated based on the first coordinate system and the navigation path. The desired image features are represented as f. * The desired image features can be understood as the image features of the surgical tool when the tip of the surgical tool coincides with the endpoint of the navigation path. Figure 3 In the diagram, 'e' represents the tail of the surgical instrument, and 't' represents the tip of the surgical instrument.

[0064] S130: Based on the current image features, determine the pose control amount of the surgical robot at the next moment, and adjust the pose of the surgical robot according to the pose control amount at the next moment, so that the surgical tool moves to the second position.

[0065] After obtaining the current image features of the surgical tools, the pose control quantity of the surgical robot at the next moment is calculated based on the current image features. The pose control quantity at the next moment is represented as u. k+1 The pose control variable is used to control the surgical robot to adjust its pose, thereby enabling the movement of the surgical instruments and achieving surgical instrument navigation. For example, at the current moment, the surgical robot is holding the surgical instruments at position P outside the human body. The navigation path is a vertical path from top to bottom, with position P as the starting point and the starting point as the entry point of the navigation path. After calculating the pose control variable of the surgical robot at the next moment based on the current image features, the pose of the surgical robot is adjusted according to the pose control variable at the next moment. After that, the surgical instruments will move to a position below the starting point, for example, position W. At this time, position W where the surgical instruments are located is the second position.

[0066] S140: Obtain the image features of the surgical tool at the second position at the next moment.

[0067] After the surgical instrument moves to the second position, the image features of the surgical instrument at the next moment in the second position are obtained. The image features at the next moment are represented as f. k+1 The second position differs from the first position; it may or may not be the endpoint. Whether the second position is the endpoint needs to be determined by the actual image feature error between the image features at the next time step and the expected image features.

[0068] S150: Determine whether the actual image feature error between the image feature at the next time step and the expected image feature is less than a preset error threshold; if the actual image feature error is less than the preset error threshold, then execute S160; if the actual image feature error is greater than or equal to the preset error threshold, then return to execute S120.

[0069] S160: The above-mentioned surgical tools have been positioned.

[0070] After obtaining the desired image features and the image features at the next time step, the difference between the image features at the next time step and the desired image features is calculated. This difference yields the actual image feature error between the image features at the next time step and the desired image features. The actual image feature error = f k+1 -f * Then, it is determined whether the actual image feature error is less than a preset error threshold. If the actual image feature error is less than the preset error threshold, the second position where the surgical tool is located is considered the endpoint, the surgical tool has reached the endpoint, and the navigation task has been completed, that is, the positioning of the surgical tool is determined to be complete. If the actual image feature error is greater than or equal to the preset error threshold, the second position where the surgical tool is located is considered not the endpoint, the surgical tool has not yet completed the navigation task, and then the process returns to execute S120 until the calculated actual image feature error is less than the preset error threshold.

[0071] This embodiment, based on the above technical solution, utilizes the following approach: First, it acquires the current image features of the surgical tool at its first position. Then, based on the navigation path of the surgical tool in the second coordinate system of the optical positioning system and the first coordinate system established by the endpoint of the navigation path, it determines the desired image features of the surgical tool at the endpoint. Based on the current image features, it determines the pose control quantity of the surgical robot at the next moment and adjusts the pose of the surgical robot according to the pose control quantity at the next moment, causing the surgical tool to move to the second position. Next, it acquires the image features of the surgical tool at the second position at the next moment. Then, it returns to execute the determination of the desired image features of the surgical tool at the endpoint based on the navigation path of the surgical tool in the second coordinate system of the optical positioning system and the first coordinate system established by the endpoint of the navigation path. This process continues until the actual image feature error between the next moment's image features and the desired image features is less than a preset error threshold, at which point the surgical tool's positioning is considered complete. By using image feature error to determine whether the surgical tool has reached the endpoint of the navigation path, the convergence speed of image feature error is accelerated, making the positioning of the surgical robot more accurate and faster. The movement trajectory during positioning is more ideal, closer to a straight line, and randomness is reduced.

[0072] like Figure 4 As shown, Figure 4 An exemplary flowchart of step S120 in the positioning method provided in this embodiment is shown. Optionally, based on the above method embodiment, step S120 includes the following:

[0073] S122: Obtain the position coordinates of the endpoint and the starting point of the navigation path in the second coordinate system.

[0074] S124: Determine the desired image features based on the origin of the first coordinate system and the position coordinates.

[0075] In an exemplary embodiment, the second coordinate system S w Below, the coordinates of the starting point of the navigation path are represented as follows: The coordinates of the destination of the navigation path are represented as follows: After constructing the first coordinate system, obtain the coordinates of its origin, i.e., the coordinates of the origin and the endpoint are the same. Then, based on the coordinates of the origin, the starting point, and the ending point, calculate the coordinates of a point on the X-axis of the first coordinate system that is one unit distance from the origin, represented as... The coordinates of a point on the Y-axis in the first coordinate system, located one unit distance from the origin, are denoted as: And the coordinates of a point on the Z-axis in the first coordinate system that is a unit distance from the origin, denoted as: in, and The calculation formula is:

[0076]

[0077] get and Afterwards, according to as well as Calculate the expected image features f of the surgical instrument's location when it reaches the endpoint. * ,Right now:

[0078]

[0079] By using the origin of the navigation path coordinate system and the position coordinates of the start and end points of the navigation path as the basis for calculating the image features of the surgical tool, the image features are made more stable. Since the image features include the position of the needle tip of the surgical tool, that is, the orientation of the surgical tool is taken into account in the construction method of the image features, it can avoid the loss of guidance information of the surgical tool due to the marker ball being obstructed during surgical navigation, thereby avoiding the occurrence of dangerous situations in surgical navigation.

[0080] Optionally, based on the above method embodiments, before S110, the positioning method further includes the following scheme:

[0081] Construct the initial image Jacobian matrix of the aforementioned surgical tools;

[0082] The initial image Jacobian matrix is ​​rearranged into column vectors to obtain the initial system state vector of the surgical robot system.

[0083] In an exemplary embodiment, when the surgical tool is located at the starting point, an initial image Jacobian matrix of the surgical tool at the starting point is constructed, i.e., the process of initializing the image Jacobian matrix. The resulting initial image Jacobian matrix is ​​denoted as J0. After obtaining the initial image Jacobian matrix, it is rearranged into column vectors to obtain the initial system state vector of the surgical robot system, denoted as X0. For example, if the initial image Jacobian matrix is ​​a 12×6 matrix, rearranging it into column vectors yields a 72×1 matrix, which is the initial system state vector of the surgical robot system.

[0084] Optionally, based on the above method embodiments, constructing the initial image Jacobian matrix of the surgical tool includes the following schemes:

[0085] The surgical robot is controlled to reach the initial pose corresponding to the initial set pose parameters, and the initial image features of the surgical tool under the initial pose are obtained; wherein the initial set pose parameters include i elements, i is greater than or equal to 6.

[0086] Add a preset offset to each element in the initial pose parameters mentioned above to obtain i offset pose parameters;

[0087] Control the surgical robot to reach the offset pose corresponding to the i-th offset pose parameter;

[0088] Based on the offset pose corresponding to the i-th offset pose parameter, the i-th intermediate image feature of the surgical tool is obtained.

[0089] Determine the i-th feature difference between the i-th intermediate image feature and the i-th initial image feature to obtain the i-th feature difference;

[0090] The initial image Jacobian matrix of the surgical tool is constructed based on the i feature differences and the preset offset.

[0091] In an exemplary embodiment, a pose parameter is set, represented by an initial pose parameter. This initial pose parameter is used to control the surgical robot, thereby adjusting its pose. The adjusted position of the surgical robot is called the initial pose. Then, the initial image features of the surgical tool under the initial pose are obtained, denoted as f0. The initial pose parameter includes i elements, where i is greater than or equal to 6. This embodiment uses i = 6 as an example. The initial pose parameter is represented as p0, for example, p0 = [X0, Y0, Z0, Rx0, Ry0, Rz0]. T .

[0092] Each element in the initial pose parameters is incremented by a preset offset Δp, resulting in six linearly independent offset pose parameters. This can also be understood as incrementing an offset Δp in one element of p0 each time, while keeping the values ​​of all other elements unchanged, to obtain six linearly independent offset poses. The i-th offset pose parameter is denoted as p. i ,Right now:

[0093]

[0094] in, is the i-th column of a 6th-order identity matrix.

[0095] Based on the initial pose, the surgical robot is controlled to reach the offset pose corresponding to the i-th offset pose parameter. That is, the pose of the surgical robot is adjusted sequentially from p1 to p6. The surgical robot first reaches the offset pose corresponding to p1, then the offset pose corresponding to p2, then the offset pose corresponding to p3, ..., and finally the offset pose corresponding to p6.

[0096] When the surgical robot reaches the offset pose corresponding to the i-th offset pose parameter, the i-th intermediate image feature of the surgical tool is obtained, denoted as f. i Since there are 6 offset poses, namely the offset poses corresponding to p1 to p6, there are 6 intermediate image features, namely f1 to f6.

[0097] Calculate the difference between the i-th intermediate image feature and the initial image feature, denoted as Δf. i , Δf i =f i -f0, thus obtaining i feature differences. After obtaining i feature differences, the initial image Jacobian matrix of the surgical tool is calculated based on the i feature differences and the preset offset Δp, that is:

[0098]

[0099] When i = 6, six feature differences can be obtained, namely Δf1 to Δf6. The initial image Jacobian matrix of the surgical tool is:

[0100]

[0101] Optionally, after obtaining the initial system state vector of the surgical robot system, the system-related parameters of the surgical robot system are also initialized, that is, the error covariance matrix P0 is initialized and its initial square root S0 is calculated; the process noise covariance matrix Q and the measurement noise covariance matrix R are set, the current time is set, the current time is represented as k, k=1; and the image feature error threshold e0 is set, which is the preset error threshold mentioned above.

[0102] like Figure 5 As shown, Figure 5 An exemplary flowchart illustrating the calculation of surgical robot pose control quantities in the positioning method provided in this embodiment is shown. Optionally, based on the above method embodiment, determining the pose control quantities of the surgical robot includes the following schemes:

[0103] S131: Determine the difference between the current image features and the desired image features to obtain the current image feature error corresponding to the surgical tool;

[0104] S132: Based on the current moment, determine the current pose and pose change of the surgical robot.

[0105] S133: Obtain the image Jacobian matrix of the surgical tool at the previous time step;

[0106] S134: Determine the image Jacobian matrix of the surgical tool at the current moment based on the initial system state vector described above;

[0107] S135: Based on the above pose change, the above image Jacobian matrix of the previous moment, the above image Jacobian matrix of the current moment, the above current image feature error, and the above current pose, determine the above pose control quantity for the next moment.

[0108] In an exemplary embodiment, the current time is denoted as k, and the current image feature error corresponding to the surgical tool is denoted as e. k e k =f k -f * f k This represents the current image features. The current pose of the surgical robot refers to its current pose in the task space, denoted as u. k The task space refers to the coordinate system in which the surgical robot's base is located, i.e., the base coordinate system; the pose change is represented by Δu. k ,Δu k =u k -u k-1 u k-1 This represents the pose of the surgical robot in the task space at the previous moment. The Jacobian matrix of the surgical tool's image at the previous moment is represented by J. k-1 The Jacobian matrix of the surgical instrument at the current moment is represented by J. k The pose control quantity u of the surgical robot at the next moment. k+1 The calculation formula is as follows:

[0109]

[0110]

[0111] k←k+1;

[0112] Among them, T k Kp represents the controller parameter, which is the pose control quantity of the surgical robot at the next moment, and also the pose change quantity of the surgical robot at the next moment.

[0113] Optionally, based on the above embodiments, the determination of the image Jacobian matrix of the surgical tool at the current moment based on the initial system state vector includes the following scheme:

[0114] The current system observation matrix of the surgical robot system is determined based on the aforementioned pose changes.

[0115] The image Jacobian matrix of the previous time step is rearranged into a column vector according to the row order to obtain the system state vector of the surgical robot system at the previous time step.

[0116] Obtain the system observations, process noise covariance matrix, measurement noise covariance matrix, the first square root of the error covariance matrix at the previous time step, and the second square root of the error covariance matrix at the current time step of the above surgical robot system.

[0117] Based on the square root commutative Kalman filter, the initial system state vector, the system state vector of the previous time step, the first square root, the current system observation matrix, the system observations, the process noise covariance matrix, the measurement noise covariance matrix, and the second square root, the system state vector of the surgical robot system at the current time step is determined.

[0118] The system state vector at the current moment is rearranged to obtain the image Jacobian matrix at the current moment.

[0119] In an exemplary embodiment, the current system observation matrix of the surgical robot system refers to the system observation matrix at the current moment, denoted as C. k ,Right now:

[0120]

[0121] Where n represents the dimension of the surgical robot's task space, which corresponds to the surgical robot's base coordinate system; m represents the dimension of the image feature space, which corresponds to the second coordinate system (the real-world coordinate system).

[0122] The Jacobian matrix of the surgical instrument in the previous time step is J. k-1 J in row order k-1Rearranged into column vectors, we obtain the system state vector of the surgical robot system at the previous time step, denoted as X. k-1 .

[0123] The square root capacitive Kalman filter is denoted as SCKF, and the system observation of the surgical robot system is denoted as Y. k Y k =f k -f k-1 f k-1 The image features representing the location of the surgical instrument at the previous moment are denoted as the image features at the previous moment; the process noise covariance matrix is ​​denoted as Q, the measurement noise covariance matrix is ​​denoted as R, and the error covariance matrix at the previous moment is denoted as P. k-1 The first square root of the error covariance matrix at the previous time step is denoted as S. k-1 The error covariance matrix at the current time is represented as P. k The second square root of the error covariance matrix at the current time is represented as S. k The current state vector of the surgical robot system is represented as X. k ,Right now:

[0124]

[0125] Where, when k=1, S k-1 =S0,S k-1 To initialize the square root S0 of the error covariance matrix P0; X k-1 =X0, X k-1 This represents the initial system state vector. Since the current system state vector of the surgical robot system is estimated online using the SCKF method, the positive definiteness and symmetry of the matrix during the calculation process are not required, thus enhancing the robustness of the uncalibrated robot target localization method.

[0126] After obtaining the system state vector of the surgical robot system at the current moment, the system state vector at the current moment is rearranged, that is: J k ←X k The Jacobian matrix of the surgical tool image at the current time is obtained, and the Jacobian matrix of the surgical tool image at the current time is J. k .

[0127] like Figure 6 As shown, Figure 6 An exemplary flowchart corresponding to S140 in the positioning method of this disclosure is shown. Optionally, based on the above embodiments, S140 includes the following:

[0128] S142: Construct a fourth coordinate system based on the needle tip position of the aforementioned surgical tools;

[0129] S144: Obtain the needle tip coordinates of the above surgical tool in the above second coordinate system;

[0130] S146: Determine the unit coordinates of each unit point on each coordinate axis of the fourth coordinate system, and transform each unit coordinate to obtain the transformed coordinates of each unit point in the second coordinate system.

[0131] S148: Based on the above-mentioned needle tip coordinates and the obtained above-mentioned transformed coordinates, determine the above-mentioned image features at the next moment.

[0132] In one exemplary embodiment, such as Figure 3 As shown, for the construction of image features, a corresponding coordinate system is established on the surgical tool through tool registration. This coordinate system can be understood as an intermediate coordinate system, denoted as S. t Simultaneously, the needle tip and needle tail of the surgical instrument can be obtained at S. t The needle tip position t and the needle tail position e are shown in the diagram.

[0133] A fourth coordinate system is constructed based on the needle tip position t of the surgical tool. This fourth coordinate system is denoted as S. n The fourth coordinate system can be understood as a needle tip coordinate system. The construction of the fourth coordinate system based on the needle tip position 't' of the surgical tool includes the following schemes:

[0134] The position of the needle tip of the aforementioned surgical instrument is taken as the origin of the aforementioned fourth coordinate system;

[0135] The direction of the needle tip of the aforementioned surgical tool is taken as the X-axis of the aforementioned fourth coordinate system;

[0136] The normal vector of the plane of the aforementioned surgical tool is taken as the Z-axis of the aforementioned fourth coordinate system;

[0137] The cross product of the X-axis and the Z-axis is taken as the Y-axis of the fourth coordinate system.

[0138] It should be understood that, in constructing the fourth coordinate system, the position of the needle tip 't' is taken as the origin of the fourth coordinate system, and the direction of the needle tip is the X-axis of the fourth coordinate system. This can also be understood as the line pointing towards the needle tip being the X-axis of the fourth coordinate system, and the normal vector of the plane containing the surgical tool being the X-axis. The Z-axis is the fourth coordinate system. The cross product of the X-axis and the Z-axis is the Y-axis of the fourth coordinate system, which means the fourth coordinate system is now complete.

[0139] After the fourth coordinate system is constructed, obtain the transformation matrix from the second coordinate system to the intermediate coordinate system, as well as the needle tip coordinates of the surgical tool in the second coordinate system. The transformation matrix includes the rotation matrix and the translation matrix. The rotation matrix is ​​represented as R. wt , represented as translation matrix T wtThe coordinates of the needle tip are represented as t. w Then, determine the unit coordinates of the unit point on the Y-axis of the fourth coordinate system, denoted as x. t The unit coordinates of a unit point on the X-axis of the fourth coordinate system are represented as y. t The unit coordinates of a unit point on the Z-axis of the fourth coordinate system are denoted as z. t Among them, x t y t and z t The calculation formula is as follows:

[0140]

[0141] x is manipulated by rotation and translation matrices. t y t and z t Perform the transformation to obtain x t y t and z t The transformed coordinates of each in the second coordinate system, i.e., x t The corresponding transformed coordinates in the second coordinate system are x w y t The corresponding transformed coordinates in the second coordinate system are y w , z t The corresponding transformed coordinates in the second coordinate system are z w Among them, x w y w and z w The conversion formula is as follows:

[0142]

[0143] Furthermore, through t w x w y w and z w Together, they form a j-dimensional column vector, for example, j=12. This j-dimensional column vector is then used as the image feature for the next time step, denoted as f.

[0144] f = [t w ,x w ,y w ,z w ] T .

[0145] The following is another embodiment of the positioning method provided in this disclosure.

[0146] This disclosure provides a positioning method according to an embodiment, applied to a surgical robot system. The positioning method includes the following steps:

[0147] Step A: Construct the initial image Jacobian matrix J0 of the surgical tool, which can also be understood as initializing the image Jacobian matrix of the surgical tool to obtain the initial image Jacobian matrix J0.

[0148] Step B: Rearrange the initial image Jacobian matrix J0 into column vectors to obtain the initial system state vector X0 of the surgical robot system.

[0149] Step C: Initialize the relevant parameters of the surgical robot system, i.e., initialize the error covariance matrix P0 and calculate its square root S0; set the process noise covariance matrix Q and the measurement noise covariance matrix R, set the current time k=1; set the image feature error threshold e0.

[0150] Step D: Calculate the current image features f of the surgical instrument at the first position at the current moment. k .

[0151] Step E: Establish a first coordinate system based on the endpoint of the navigation path, and calculate the expected image features f of the surgical tool reaching the endpoint location based on the coordinates of the origin of the first coordinate system, the position coordinates of the starting point and the position coordinates of the endpoint of the navigation path. * .

[0152] Step F: Calculate the system observation Y of the surgical robot system. k The pose change Δu of the surgical robot in the task space k The current image feature error of the surgical tool e k ,Right now:

[0153] Y k =f k -f k-1 ;

[0154] Δu k =u k -u k-1 ;

[0155] e k =f k -f * ;

[0156] Step G: Calculate the current system observation matrix C of the surgical robot system. k And the system state vector X of the surgical robot system at the previous moment. k-1 ,Right now:

[0157]

[0158] Rearrange the Jacobian matrix J of the surgical instruments in row order from the previous time step. k-1 This yields a column vector, which is the system state vector X.k-1 。

[0159] Step H: Calculate the image Jacobian matrix J of the surgical tool at the current moment k , from the system state vector X of the surgical robot system at the previous moment k-1 Calculate the system state vector X of the surgical robot system at the current moment through the SCKF k , re-arrange the system state vector X k to obtain the image Jacobian matrix J k , that is:

[0160]

[0161] J k ←X k ;

[0162] Step I: Calculate the pose control quantity u of the surgical robot at the next moment k+1 , that is:

[0163]

[0164]

[0165] k←k + 1;

[0166] Step J: Compare the current image feature error e k with the preset error threshold e0. If e k <e0, the positioning of the surgical tool is completed and the algorithm exits; if e k ≥e0, return to Step E and continue to execute until e k <e0.

[0167] This embodiment accelerates the convergence speed of the image feature error, making the positioning of the surgical robot more accurate and time-consuming shorter, with a more ideal motion trajectory during positioning, approaching a straight line, and reducing randomness. In the construction method of the image feature, the orientation problem of the surgical tool is considered, which can avoid the loss of the guiding information of the surgical tool due to the occlusion of the marker ball during the surgical navigation process, thus preventing the occurrence of surgical navigation risks. In addition, the system state vector of the surgical robot system at the current moment is obtained by online estimation using the SCKF method, without requiring the positive definiteness and symmetry of the matrix during the operation process, enhancing the robustness of the calibration-free robot target positioning method.

[0168] The following is an embodiment of the apparatus of the present disclosure, which can be used to execute the method embodiment of the present disclosure. For details not disclosed in the embodiment of the apparatus of the present disclosure, please refer to the method embodiment of the present disclosure.

[0169] Among them, Figure 7A schematic diagram of a positioning device applicable to an embodiment of this disclosure is shown. Please refer to... Figure 7 The positioning device shown in the figure can be implemented as all or part of an electronic device through software, hardware, or a combination of both, or it can be integrated as an independent module into an electronic device or server.

[0170] The positioning device 700 in this embodiment is configured in a surgical robot system, which includes a surgical robot, surgical instruments, and an optical positioning system. The surgical instruments are mounted on the surgical robot. The positioning device 700 includes:

[0171] The first calculation module 710 is used to obtain the current image features of the surgical tool at the first position at the current moment;

[0172] The second calculation module 720 is used to determine the desired image features of the surgical tool at the endpoint based on the navigation path of the surgical tool and a first coordinate system established by the endpoint of the navigation path; wherein the navigation path is in the second coordinate system where the optical positioning system is located.

[0173] The pose adjustment module 730 is used to determine the pose control amount of the surgical robot at the next moment based on the current image features, and adjust the pose of the surgical robot according to the pose control amount at the next moment so that the surgical tool moves to the second position.

[0174] The third calculation module 740 is used to obtain the image features of the surgical tool at the second position at the next moment.

[0175] Error comparison module 750 is used to return to the step of determining the expected image features of the surgical tool at the endpoint based on the navigation path of the surgical tool and the first coordinate system established by the endpoint of the navigation path, until the actual image feature error between the image features at the next moment and the expected image features is less than a preset error threshold, and then determine that the positioning of the surgical tool is completed.

[0176] In an exemplary embodiment, based on the foregoing scheme, the second computing module 720 includes:

[0177] The position coordinate acquisition unit is used to acquire the position coordinates of the endpoint and the starting point of the navigation path in the second coordinate system.

[0178] The expected feature calculation unit is used to determine the expected image features based on the origin of the first coordinate system and the position coordinates.

[0179] In an exemplary embodiment, based on the foregoing solution, the positioning device 700 further includes:

[0180] A matrix initialization unit is used to construct the initial image Jacobian matrix of the aforementioned surgical tools;

[0181] The state vector calculation unit is used to rearrange the initial image Jacobian matrix into column vectors to obtain the initial system state vector of the surgical robot system.

[0182] In an exemplary embodiment, based on the foregoing scheme, the matrix initialization unit includes:

[0183] The first control subunit is used to control the surgical robot to reach the initial pose corresponding to the initial set pose parameters, and to acquire the initial image features of the surgical tool under the initial pose; wherein the initial set pose parameters include i elements, i is greater than or equal to 6.

[0184] The parameter adjustment subunit is used to add a preset offset to each element in the above-mentioned initial pose parameters to obtain i offset pose parameters.

[0185] The second control subunit is used to control the surgical robot to reach the offset pose corresponding to the i-th offset pose parameter.

[0186] The intermediate feature acquisition subunit is used to acquire the i-th intermediate image feature of the surgical tool based on the offset pose corresponding to the i-th offset pose parameter.

[0187] The difference calculation subunit is used to determine the i-th feature difference between the i-th intermediate image feature and the i-th initial image feature, and to obtain the i-th feature difference.

[0188] The initial matrix calculation sub-unit is used to construct the initial image Jacobian matrix of the surgical tool based on the above i feature differences and the above preset offset.

[0189] In an exemplary embodiment, based on the foregoing scheme, the pose adjustment module 740 includes the following steps in determining the pose control amount of the surgical robot at the next moment based on the current image features:

[0190] The feature error calculation unit is used to determine the difference between the current image feature and the expected image feature to obtain the current image feature error corresponding to the surgical tool.

[0191] The pose data calculation unit is used to determine the current pose and pose change of the surgical robot based on the current moment.

[0192] The first matrix acquisition unit is used to acquire the image Jacobian matrix of the surgical tool at the previous time step.

[0193] The second matrix acquisition unit is used to determine the image Jacobian matrix of the surgical tool at the current moment based on the initial system state vector.

[0194] The control quantity calculation unit is used to determine the pose control quantity for the next moment based on the pose change, the image Jacobian matrix of the previous moment, the image Jacobian matrix of the current moment, the current image feature error, and the current pose.

[0195] In an exemplary embodiment, based on the foregoing scheme, the second matrix acquisition unit includes:

[0196] The observation matrix calculation subunit is used to determine the current system observation matrix of the surgical robot system based on the aforementioned pose change.

[0197] The first vector calculation subunit is used to rearrange the image Jacobian matrix of the previous time step into a column vector according to the row order, so as to obtain the system state vector of the surgical robot system at the previous time step.

[0198] The relevant data acquisition subunit is used to acquire the system observations, process noise covariance matrix, measurement noise covariance matrix, the first square root of the error covariance matrix at the previous time step, and the second square root of the error covariance matrix at the current time step of the surgical robot system.

[0199] The second vector calculation subunit is used to determine the system state vector of the surgical robot system at the current moment based on the square root commutative Kalman filter, the initial system state vector, the system state vector at the previous moment, the first square root, the current system observation matrix, the system observations, the process noise covariance matrix, the measurement noise covariance matrix, and the second square root.

[0200] The vector rearrangement subunit is used to rearrange the system state vector at the current time to obtain the image Jacobian matrix at the current time.

[0201] In an exemplary embodiment, based on the foregoing scheme, the third computing module 740 includes:

[0202] A coordinate system construction unit is used to construct a fourth coordinate system based on the needle tip position of the aforementioned surgical tool;

[0203] The coordinate data acquisition unit is used to acquire the needle tip coordinates of the surgical tool in the second coordinate system.

[0204] The transformation unit is used to determine the unit coordinates of the unit points on each coordinate axis of the fourth coordinate system and transform each unit coordinate to obtain the transformed coordinates of each unit point in the second coordinate system.

[0205] The image feature calculation unit is used to determine the image features at the next moment based on the aforementioned needle tip coordinates and the obtained transformed coordinates.

[0206] In an exemplary embodiment, based on the foregoing scheme, the coordinate system construction unit includes:

[0207] The origin determination unit is used to take the position of the needle tip of the surgical tool as the origin of the fourth coordinate system.

[0208] The coordinate axis determination unit is used to take the direction of the needle tip of the surgical tool as the X-axis of the fourth coordinate system; take the normal vector of the plane of the surgical tool as the Z-axis of the fourth coordinate system; and take the cross product of the X-axis and the Z-axis as the Y-axis of the fourth coordinate system.

[0209] It should be noted that the positioning device provided in the above embodiments is only illustrated by the division of the above functional modules when executing the positioning method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the positioning device and positioning method embodiments provided in the above embodiments belong to the same concept. Therefore, for details not disclosed in the device embodiments of this disclosure, please refer to the above-described embodiments of the positioning method of this disclosure, which will not be repeated here.

[0210] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0211] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods described in the foregoing embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0212] This disclosure also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of any of the methods described above.

[0213] Figure 8 A schematic diagram of the electronic device is shown. Please refer to [link / reference]. Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802.

[0214] In this embodiment, the processor 801 is the control center of the computer system and can be a processor of a physical machine or a processor of a virtual machine. The processor 801 may include one or more processing cores, such as a 4-core processor or an 8-core processor. The processor 801 can be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0215] In this embodiment of the disclosure, the processor 801 is specifically configured to: acquire the current image features of the surgical tool at a first position at the current moment; determine the desired image features of the surgical tool reaching the endpoint based on the navigation path of the surgical tool and a first coordinate system established by the endpoint of the navigation path; wherein the navigation path is in a second coordinate system where the optical positioning system is located; determine the pose control amount of the surgical robot at the next moment based on the current image features, and adjust the pose of the surgical robot according to the pose control amount at the next moment so that the surgical tool moves to a second position; acquire the image features of the surgical tool at the next moment at the second position; return to execute the step of determining the desired image features of the surgical tool reaching the endpoint based on the navigation path of the surgical tool and the first coordinate system established by the endpoint of the navigation path, until the actual image feature error between the image features at the next moment and the desired image features is less than a preset error threshold, and determine that the positioning of the surgical tool is completed.

[0216] Furthermore, the processor 801 is also configured to: obtain the position coordinates of the endpoint and the starting point of the navigation path in the second coordinate system; and determine the desired image features based on the origin of the first coordinate system and the position coordinates.

[0217] Furthermore, the processor 801 is also used to: construct an initial image Jacobian matrix of the surgical tool; rearrange the initial image Jacobian matrix into column vectors to obtain the initial system state vector of the surgical robot system.

[0218] Furthermore, the processor 801 is also configured to: control the surgical robot to reach the initial pose corresponding to the initial set pose parameters, and acquire the initial image features of the surgical tool under the initial pose; wherein the initial set pose parameters include i elements, i being greater than or equal to 6; add a preset offset to each element in the initial set pose parameters to obtain i offset pose parameters; control the surgical robot to reach the offset pose corresponding to the i-th offset pose parameter; acquire the i-th intermediate image feature of the surgical tool based on the offset pose corresponding to the i-th offset pose parameter; determine the i-th feature difference between the i-th intermediate image feature and the initial image feature to obtain i feature differences; and construct the initial image Jacobian matrix of the surgical tool according to the i feature differences and the preset offset.

[0219] Furthermore, the processor 801 is also configured to: determine the difference between the current image features and the desired image features to obtain the current image feature error corresponding to the surgical tool; determine the current pose and pose change of the surgical robot based on the current moment; obtain the image Jacobian matrix of the surgical tool at the previous moment; determine the image Jacobian matrix of the surgical tool at the current moment based on the initial system state vector; and determine the pose control quantity at the next moment based on the pose change, the image Jacobian matrix at the previous moment, the image Jacobian matrix at the current moment, the current image feature error, and the current pose.

[0220] Furthermore, the processor 801 is also configured to: determine the current system observation matrix of the surgical robot system based on the pose change; rearrange the image Jacobian matrix of the previous time step into a column vector according to row order to obtain the system state vector of the surgical robot system at the previous time step; acquire the system observations, process noise covariance matrix, measurement noise covariance matrix, the first square root of the error covariance matrix of the previous time step, and the second square root of the error covariance matrix of the current time step of the surgical robot system; determine the system state vector of the surgical robot system at the current time step based on the square root capacitive Kalman filter, the initial system state vector, the system state vector of the previous time step, the first square root, the current system observation matrix, the system observations, the process noise covariance matrix, the measurement noise covariance matrix, and the second square root; and rearrange the system state vector at the current time step to obtain the image Jacobian matrix at the current time step.

[0221] Furthermore, the processor 801 is also configured to: construct a fourth coordinate system based on the needle tip position of the surgical tool; obtain the needle tip coordinates of the surgical tool in the second coordinate system; determine the unit coordinates of unit points on each coordinate axis of the fourth coordinate system, and transform each unit coordinate to obtain the transformed coordinates of each unit point in the second coordinate system; and determine the image features at the next moment based on the needle tip coordinates and the obtained transformed coordinates.

[0222] Furthermore, the processor 801 is also configured to: use the position of the needle tip of the surgical tool as the origin of the fourth coordinate system; use the direction of the needle tip of the surgical tool as the X-axis of the fourth coordinate system; use the normal vector of the plane of the surgical tool as the Z-axis of the fourth coordinate system; and use the cross product of the X-axis and the Z-axis as the Y-axis of the fourth coordinate system.

[0223] Memory 802 may include one or more computer-readable storage media, which may be non-transitory. Memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this disclosure, the non-transitory computer-readable storage media in memory 802 is used to store at least one instruction, which is executed by processor 801 to implement the methods in the embodiments of this disclosure.

[0224] In some embodiments, the electronic device 800 further includes a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 are connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 803 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of a display screen 804, a camera 805, and an audio circuit 806.

[0225] Peripheral device interface 803 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 801 and memory 802. In some embodiments of this disclosure, processor 801, memory 802, and peripheral device interface 803 are integrated on the same chip or circuit board; in other embodiments of this disclosure, any one or two of processor 801, memory 802, and peripheral device interface 803 can be implemented on separate chips or circuit boards. This disclosure does not specifically limit the scope of the embodiments.

[0226] Display screen 804 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 804 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 801 for processing. In this case, display screen 804 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments of this disclosure, there may be one display screen 804, which serves as the front panel of the electronic device 800; in other embodiments, there may be at least two display screens 804, respectively disposed on different surfaces of the electronic device 800 or in a folded design; in still other embodiments, display screen 804 may be a flexible display screen, disposed on a curved or folded surface of the electronic device 800. Furthermore, display screen 804 may be configured as a non-rectangular irregular shape, i.e., a non-rectangular screen. Display screen 804 may be made of materials such as LCD (Liquid Crystal Display) or OLED (Organic Light-Emitting Diode).

[0227] Camera 805 is used to acquire images or videos. Optionally, camera 805 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the electronic device 800, and the rear-facing camera is located on the back of the electronic device 800. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments of this disclosure, camera 805 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0228] The audio circuit 806 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input to the processor 801 for processing. For stereo sound acquisition or noise reduction purposes, there may be multiple microphones, each located in a different part of the electronic device 800. The microphone may also be an array microphone or an omnidirectional microphone.

[0229] Power supply 807 is used to supply power to various components in electronic device 800. Power supply 807 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 807 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0230] The block diagram of the electronic device 800 shown in the embodiments of this disclosure does not constitute a limitation on the electronic device 800. The electronic device 800 may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0231] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in the embodiments of this specification are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the object characteristics, interactive behavior characteristics, and user information involved in this specification were all obtained under full authorization.

[0232] In the description of this disclosure, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this disclosure based on the specific circumstances. Furthermore, in the description of this disclosure, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0233] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. Therefore, equivalent variations made in accordance with the claims of this disclosure are still within the scope of this disclosure.

Claims

1. A positioning method, characterized in that, An application in a surgical robot system, the surgical robot system comprising a surgical robot, surgical tools, and an optical positioning system, wherein the surgical tools are mounted on the surgical robot; The positioning method includes: Obtain the current image features of the surgical tool at the first position at the current moment; Based on the navigation path of the surgical tool and a first coordinate system established by the endpoint of the navigation path, the desired image features of the surgical tool at the endpoint are determined; wherein the navigation path is in a second coordinate system where the optical positioning system is located; Based on the current image features, the pose control amount of the surgical robot at the next moment is determined, and the pose of the surgical robot is adjusted according to the pose control amount at the next moment so that the surgical tool moves to the second position. A fourth coordinate system is constructed based on the needle tip position of the surgical tool; Obtain the needle tip coordinates of the surgical tool in the second coordinate system; Determine the unit coordinates of each unit point on each coordinate axis of the fourth coordinate system, and transform each unit coordinate to obtain the transformed coordinates of each unit point in the second coordinate system; Based on the needle tip coordinates and the obtained transformed coordinates, determine the next moment image features of the surgical tool at the second position; Return to the step of determining the desired image features of the surgical tool at the endpoint based on the navigation path of the surgical tool and the first coordinate system established by the endpoint of the navigation path, until the actual image feature error between the image features and the desired image features at the next moment is less than a preset error threshold, and determine that the positioning of the surgical tool is complete; The step of constructing a fourth coordinate system based on the needle tip position of the surgical tool includes: The position of the needle tip of the surgical tool is taken as the origin of the fourth coordinate system; The direction of the needle tip of the surgical tool is taken as the X-axis of the fourth coordinate system; The normal vector of the plane of the surgical tool is taken as the Z-axis of the fourth coordinate system; The cross product of the X-axis and the Z-axis is used as the Y-axis of the fourth coordinate system.

2. The positioning method as described in claim 1, characterized in that, The step of determining the desired image features at the endpoint of the surgical tool based on the navigation path of the surgical tool and a first coordinate system established by the endpoint of the navigation path includes: Obtain the position coordinates of the endpoint and the starting point of the navigation path in the second coordinate system; The desired image features are determined based on the origin of the first coordinate system and the position coordinates.

3. The positioning method as described in claim 1, characterized in that, Before the step of acquiring the current image features of the surgical tool at the first position at the current moment, the localization method further includes: Construct the initial image Jacobian matrix of the surgical tool; The initial image Jacobian matrix is ​​rearranged into column vectors to obtain the initial system state vector of the surgical robot system.

4. The positioning method as described in claim 3, characterized in that, The step of constructing the initial image Jacobian matrix of the surgical tool includes: The surgical robot is controlled to reach the initial pose corresponding to the initial set pose parameters, and the initial image features of the surgical tool under the initial pose are obtained; wherein, the initial set pose parameters include i elements, i is greater than or equal to 6; Each element in the initial pose parameter is added with a preset offset to obtain i offset pose parameters. Control the surgical robot to reach the offset pose corresponding to the i-th offset pose parameter; Based on the offset pose corresponding to the i-th offset pose parameter, the i-th intermediate image feature of the surgical tool is obtained; Determine the i-th feature difference between the i-th intermediate image feature and the initial image feature to obtain the i-th feature difference; The initial image Jacobian matrix of the surgical tool is constructed based on the i feature differences and the preset offset.

5. The positioning method as described in claim 3, characterized in that, The step of determining the pose control parameters of the surgical robot at the next moment based on the current image features includes: The difference between the current image features and the desired image features is determined to obtain the current image feature error corresponding to the surgical tool; Based on the current moment, determine the current pose and pose change of the surgical robot; Obtain the Jacobian matrix of the surgical tool at the previous time step; The image Jacobian matrix of the surgical tool at the current moment is determined based on the initial system state vector; The pose control amount for the next moment is determined based on the pose change, the image Jacobian matrix of the previous moment, the image Jacobian matrix of the current moment, the current image feature error, and the current pose.

6. The positioning method as described in claim 5, characterized in that, The step of determining the image Jacobian matrix of the surgical tool at the current time based on the initial system state vector includes: The current system observation matrix of the surgical robot system is determined based on the pose change. The image Jacobian matrix of the previous time step is rearranged into a column vector according to row order to obtain the system state vector of the surgical robot system at the previous time step; Obtain the system observations, process noise covariance matrix, measurement noise covariance matrix, the first square root of the error covariance matrix at the previous time step, and the second square root of the error covariance matrix at the current time step of the surgical robot system. The system state vector of the surgical robot system at the current moment is determined based on the square root commutative Kalman filter, the initial system state vector, the system state vector at the previous moment, the first square root, the current system observation matrix, the system observations, the process noise covariance matrix, the measurement noise covariance matrix, and the second square root. The system state vector at the current moment is rearranged to obtain the image Jacobian matrix at the current moment.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the positioning method as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the positioning method as described in any one of claims 1 to 6.

Images