Navigation method and related products based on displacement prediction

Abstract

The present application discloses a navigation method based on displacement prediction and related products. The method includes: obtaining a target CT image of a target object at a first moment, determining a first coordinate of a target point in a CT coordinate system, where the CT coordinate system is the coordinate system of the target CT image; determining a second coordinate of a target marker in a CT coordinate system; obtaining a third coordinate of the target marker in an optical coordinate system, where the third coordinate is the coordinate of the target marker at a second moment, where the optical coordinate system is the coordinate system of an optical tracking system; based on a first conversion relationship, converting the third coordinate to a CT coordinate system to obtain a fourth coordinate, where the first conversion relationship is a conversion relationship between an optical coordinate system and a CT coordinate system; based on the second coordinate and the fourth coordinate, determining a reference displacement of the target marker from the first moment to the second moment; inputting the first coordinate and the reference displacement into a target model to obtain a target displacement of the target point.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a navigation method based on displacement prediction and related products. Background Art

[0002] During the puncture surgery on the patient, the puncture path is usually planned based on the computed tomography (CT) images taken in advance, and the patient is punctured according to the planned puncture path. However, the patient's target movement will cause the movement of the organ to which the puncture target belongs, and then cause the movement of the target, where the target movement includes breathing, heartbeat, etc. The movement of the target may cause the planned puncture path to fail to pass through the target, resulting in the failure of the puncture surgery. Therefore, how to determine the displacement of the patient's target caused by target movement is a very important issue. Summary of the invention

[0003] The present application provides a navigation method based on displacement prediction and related products, wherein the related products include a navigation device based on displacement prediction, an electronic device, a computer-readable storage medium and a computer program product.

[0004] In a first aspect, a navigation method based on displacement prediction is provided, characterized in that the method is used to determine a target displacement of a target point in the target object from the first moment to the second moment caused by a target motion of the target object from a first moment to a second moment, and the method comprises:

[0005] Acquire a target CT image of the target object at the first moment, wherein the target CT image includes the target point and a target marker on the surface of the target object;

[0006] Determine a first coordinate of the target point in a CT coordinate system of the target CT image, where the CT coordinate system is a coordinate system of the target CT image;

[0007] Determining a second coordinate of the target marker in the CT coordinate system;

[0008] Acquire a third coordinate of the target marker in the optical coordinate system, where the third coordinate is the coordinate of the target marker at a second moment, and the optical coordinate system is a coordinate system of the optical tracking system;

[0009] Based on a first conversion relationship, convert the third coordinate into a CT coordinate system to obtain a fourth coordinate, wherein the first conversion relationship is a conversion relationship between the optical coordinate system and the CT coordinate system;

[0010] Determining a reference displacement of the target marker from the first moment to the second moment based on the second coordinate and the fourth coordinate;

[0011] The first coordinate and the reference displacement are input into a target model to obtain the target displacement of the target point. The target model is obtained by updating the parameters of the model to be trained based on the target loss. The target loss is obtained according to the displacement difference of the training target point in the training object from the third moment to the fourth moment. The displacement difference is the difference between the actual displacement and the predicted displacement. The predicted displacement is the displacement of the training target point predicted by the model to be trained from the third moment to the fourth moment. The model to be trained obtains the predicted displacement based on the marker displacement and the fifth coordinate. The marker displacement is the displacement of the training marker on the surface of the training object from the third moment to the fourth moment. The fifth coordinate is the coordinate of the training target point at the third moment in the CT coordinate system.

[0012] In combination with any embodiment of the present application, the step of inputting the first coordinate and the reference displacement into a target model to obtain the target displacement of the target point includes:

[0013] Inputting the reference displacement, the first coordinate and the second coordinate into the target model, so that the target model determines the relative position relationship between the target point and the target marker according to the first coordinate and the second coordinate;

[0014] The target displacement of the target point is obtained by using the target model according to the reference displacement, the first coordinate and the relative position relationship.

[0015] In combination with any embodiment of the present application, the step of inputting the first coordinate and the reference displacement into a target model to obtain the target displacement of the target point includes:

[0016] Acquire a target region in the CT coordinate system, wherein the target region represents a region of the organ to which the target point belongs;

[0017] The target area, the reference displacement and the first coordinate are input into the target model to obtain the target displacement of the target point. According to the first coordinate and the target displacement of the target point at the first moment, the coordinate of the target point at the second moment is within the target area.

[0018] In combination with any embodiment of the present application, after obtaining the target displacement of the target point, the method further includes:

[0019] Determine the coordinates of the target point at the second moment in the CT coordinate system according to the first coordinates and the target displacement to obtain a sixth coordinate;

[0020] Determine the target posture of the robotic arm according to the sixth coordinate, and when the posture of the robotic arm is the target posture, a puncture path determined based on the puncture guide rail of the robotic arm passes through the target target point;

[0021] The robot arm is controlled to move so that the posture of the robot arm is the target posture.

[0022] In combination with any embodiment of the present application, the obtaining the third coordinate of the target marker in the optical coordinate system includes:

[0023] Acquire the third coordinate of the target marker in the optical coordinate system every N milliseconds;

[0024] The controlling the movement of the mechanical arm so that the posture of the mechanical arm is the target posture comprises:

[0025] After determining the target posture, the robot arm is controlled to move so that the posture of the robot arm is the target posture, and the movement of the robot arm is not affected by the movement instruction before determining the target posture, and the movement instruction is an instruction for controlling the movement of the robot arm.

[0026] In combination with any embodiment of the present application, the target CT image includes m candidate CT images; the candidate CT images are two-dimensional images obtained by performing continuous CT scans on the target object;

[0027] The step of inputting the first coordinate and the reference displacement into a target model to obtain the target displacement of the target point includes:

[0028] Merging the m candidate CT images to obtain a CT three-dimensional image;

[0029] According to a second conversion relationship, the first coordinate is converted into an organ coordinate system to obtain a seventh coordinate, wherein the second conversion relationship is a conversion relationship between the CT coordinate system and the organ coordinate system, and the organ coordinate system is a coordinate system where the CT three-dimensional image is located;

[0030] According to the second conversion relationship, the reference displacement is converted into the organ coordinate system to obtain a candidate displacement;

[0031] The seventh coordinate and the candidate displacement are input into the target model to obtain the target displacement of the target point.

[0032] In a second aspect, a navigation device based on displacement prediction is provided, and the navigation device based on displacement prediction includes:

[0033] an acquisition unit, configured to acquire a target CT image of the target object at the first moment, wherein the target CT image includes the target point and a target marker on the surface of the target object;

[0034] A determination unit, configured to determine a first coordinate of the target point in a CT coordinate system of the target CT image, wherein the CT coordinate system is a coordinate system of the target CT image;

[0035] The determination unit is further used to determine a second coordinate of the target marker in the CT coordinate system;

[0036] an acquisition unit, further used to acquire a third coordinate of the target marker in the optical coordinate system, wherein the third coordinate is a coordinate of the target marker at a second moment, and the optical coordinate system is a coordinate system of the optical tracking system;

[0037] a conversion unit, configured to convert the third coordinate into a CT coordinate system to obtain a fourth coordinate based on a first conversion relationship, wherein the first conversion relationship is a conversion relationship between the optical coordinate system and the CT coordinate system;

[0038] a determining unit, further used to determine a reference displacement of the target marker from the first moment to the second moment based on the second coordinate and the fourth coordinate;

[0039] A displacement prediction unit is used to input the first coordinate and the reference displacement into a target model to obtain the target displacement of the target point, wherein the target model is obtained by updating the parameters of the model to be trained based on the target loss, wherein the target loss is obtained according to the displacement difference of the training target point in the training object from the third moment to the fourth moment, wherein the displacement difference is the difference between the actual displacement and the predicted displacement, wherein the predicted displacement is the displacement of the training target point predicted by the model to be trained from the third moment to the fourth moment, wherein the model to be trained obtains the predicted displacement according to the marker displacement and the fifth coordinate, wherein the marker displacement is the displacement of the training marker on the surface of the training object from the third moment to the fourth moment, and wherein the fifth coordinate is the coordinate of the training target point at the third moment in the CT coordinate system.

[0040] In combination with any embodiment of the present application, the displacement prediction unit is further used to:

[0041] Inputting the reference displacement, the first coordinate and the second coordinate into the target model, so that the target model determines the relative position relationship between the target point and the target marker according to the first coordinate and the second coordinate;

[0042] The target displacement of the target point is obtained by using the target model according to the reference displacement, the first coordinate and the relative position relationship.

[0043] In combination with any embodiment of the present application, the displacement prediction unit is further used to:

[0044] Acquire a target region in the CT coordinate system, wherein the target region represents a region of the organ to which the target point belongs;

[0045] The target area, the reference displacement and the first coordinate are input into the target model to obtain the target displacement of the target point. According to the first coordinate and the target displacement of the target point at the first moment, the coordinate of the target point at the second moment is within the target area.

[0046] In combination with any embodiment of the present application, the mechanical arm is moved according to the target displacement, and the puncture guide rail on the mechanical arm is driven to move by moving the mechanical arm;

[0047] The navigation device based on displacement prediction also includes a moving unit, which is used to:

[0048] Determine the coordinates of the target point at the second moment in the CT coordinate system according to the first coordinates and the target displacement to obtain a sixth coordinate;

[0049] Determine the target posture of the robotic arm according to the sixth coordinate, and when the posture of the robotic arm is the target posture, a puncture path determined based on the puncture guide rail of the robotic arm passes through the target target point;

[0050] The robot arm is controlled to move so that the posture of the robot arm is the target posture.

[0051] In conjunction with any embodiment of the present application, the acquisition unit is specifically used to:

[0052] Acquire the third coordinate of the target marker in the optical coordinate system every N milliseconds;

[0053] The above mobile unit is specifically used for:

[0054] After determining the target posture, clearing the movement instruction of the robotic arm, wherein the movement instruction is an instruction for controlling the movement of the robotic arm;

[0055] The robot arm is controlled to move so that the posture of the robot arm is the target posture.

[0056] In combination with any embodiment of the present application, the target CT image includes m candidate CT images; the candidate CT images are two-dimensional images obtained by performing continuous CT scans on the target object;

[0057] The above displacement prediction unit is specifically used for:

[0058] Merging the m candidate CT images to obtain a CT three-dimensional image;

[0059] According to a second conversion relationship, the first coordinate is converted into an organ coordinate system to obtain a seventh coordinate, wherein the second conversion relationship is a conversion relationship between the CT coordinate system and the organ coordinate system, and the organ coordinate system is a coordinate system where the CT three-dimensional image is located;

[0060] According to the second conversion relationship, the reference displacement is converted into the organ coordinate system to obtain a candidate displacement;

[0061] The seventh coordinate and the candidate displacement are input into the target model to obtain the target displacement of the target point.

[0062] According to a third aspect, an electronic device is provided, comprising: a processor and a memory, wherein the memory is used to store computer program code, wherein the computer program code comprises computer instructions, and when the processor executes the computer instructions, the electronic device executes the first aspect and any one of its embodiments described above.

[0063] In a fourth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the first aspect and any one of its embodiments.

[0064] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions. When the program instructions are executed by a processor, the processor is caused to execute the first aspect and any one of its embodiments.

[0065] In a sixth aspect, a computer program product is provided, wherein the computer program product comprises a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the above-mentioned first aspect and any embodiment thereof.

[0066] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0067] In the embodiment of the present application, after receiving the target CT image of the target object at the first moment, the navigation device determines the first coordinate of the target point and the second coordinate of the target marker in the CT coordinate system from the target CT image, and then uses the optical tracking system to determine the third coordinate of the target marker at the second moment in the optical coordinate system, and uses the first conversion relationship to convert the third coordinate to the CT coordinate system to obtain the coordinate of the target marker at the second moment in the CT coordinate system as the fourth coordinate. When the second coordinate of the target marker at the first moment and the fourth coordinate at the second moment in the CT coordinate system are determined, the reference displacement of the target marker from the first moment to the second moment is determined. The first coordinate and the reference displacement are used as inputs of the target model, wherein the target model is a model obtained by updating the parameters of the model to be trained through the target loss, the target loss is obtained according to the displacement difference of the training target point in the training object from the third moment to the fourth moment, the displacement difference is the difference between the actual displacement and the predicted displacement, the predicted displacement is the displacement of the training target point from the third moment to the fourth moment predicted by the model to be trained, the model to be trained obtains the predicted displacement according to the marker displacement and the fifth coordinate, the marker displacement is the displacement of the training marker on the surface of the training object from the third moment to the fourth moment, and the fifth coordinate is the coordinate of the training target point at the third moment in the CT coordinate system. Since the target model is obtained through the marker displacement and the fifth coordinate training, the target model has the ability to predict the displacement of the target target point in the CT coordinate system from the first moment to the second moment based on the reference displacement and the first coordinate. Therefore, after the first coordinate and the reference displacement are input into the target model, the target model can determine the target displacement of the target target point in the target object caused by the target movement from the first moment to the second moment based on the reference displacement and the first coordinate. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0069] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0070] Figure 1 A schematic diagram of a flow chart of a navigation method based on displacement prediction provided in an embodiment of the present application;

[0071] Figure 2 A schematic diagram of a target marker in a target CT image provided in an embodiment of the present application;

[0072] Figure 3 A schematic diagram of a CT three-dimensional image provided in an embodiment of the present application;

[0073] Figure 4 A schematic diagram of a training process of a target model provided in an embodiment of the present application;

[0074] Figure 5 A schematic diagram of the structure of a navigation device based on displacement prediction provided in an embodiment of the present application;

[0075] Figure 6 A schematic diagram of planning a preset path based on a target CT image provided in an embodiment of the present application;

[0076] Figure 7 A schematic diagram of performing a puncture operation on a target object based on a needle insertion path provided in an embodiment of the present application;

[0077] Figure 8 A schematic diagram of rigid registration provided in an embodiment of the present application;

[0078] Fig. 9 A workflow diagram of a navigation device based on displacement prediction provided in an embodiment of the present application;

[0079] Fig.10 A schematic diagram of the structure of a navigation device based on displacement prediction provided in an embodiment of the present application;

[0080] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0082] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0083] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0084] The execution subject of the embodiment of the present application is a navigation device based on displacement prediction, wherein the navigation device based on displacement prediction can be any electronic device that can execute the technical solution disclosed in the method embodiment of the present application, hereinafter referred to as a navigation device. Optionally, the navigation device can be one of the following: a computer, a server.

[0085] In an embodiment of the present application, the navigation device is used to determine the target displacement of a target point in the target object from the first moment to the second moment due to the target movement of the target object from the first moment to the second moment during the puncture surgery, wherein the target movement is the movement of the target object during the surgery, for example, the target movement includes breathing. For another example, the target movement includes heartbeat. For another example, the target movement includes muscle reflex.

[0086] It should be understood that the method embodiment of the present application can also be implemented by a processor executing a computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 A flowchart of a navigation method based on displacement prediction provided in an embodiment of the present application.

[0087] 101. Acquire a target CT image of a target object at a first moment.

[0088] In the embodiment of the present application, the target CT image is an image obtained by scanning the target object. Since there is a target point in the target object and there is a target marker on the surface of the target object, the target CT image obtained by CT scanning the target object includes the target point and the target marker on the surface of the target object.

[0089] In an implementation of acquiring a target CT image, the navigation device receives the target CT image input by an input component. The input component includes at least one of the following: a keyboard, a mouse, a touch screen, a touch pad, and an audio input device.

[0090] In another implementation of acquiring a target CT image, the navigation device receives the target CT image sent by a terminal. The terminal includes at least one of the following: a mobile phone, a computer, a tablet computer, and a server.

[0091] 102. Determine a first coordinate of the target point in a CT coordinate system of the target CT image.

[0092] In the embodiment of the present application, after the target CT image is acquired, a CT coordinate system is established in the target CT image, and the coordinates of the target point are determined in the CT coordinate system to obtain the first coordinates.

[0093] 103. Determine a second coordinate of the target marker in the CT coordinate system.

[0094] In the embodiment of the present application, when a CT coordinate system is established, the coordinates of the target marker are determined in the CT coordinate system to obtain the second coordinates. Optionally, there are three or more target markers. When the number of target markers is greater than or equal to 3, the number of coordinates of the second coordinates of the target marker determined in the CT coordinate system is the same as the number of target markers, and the second coordinates correspond to the target markers one by one. For example, Figure 2 is a schematic diagram of target markers in a target CT image. Specifically, the number of target markers is 5, and Ma represents the positions of the above 5 target markers in the CT coordinate system.

[0095] 104. Obtain a third coordinate of the target marker in the optical coordinate system.

[0096] In the embodiment of the present application, the optical coordinate system is a coordinate system of an optical tracking system, wherein the optical tracking system is used to track a target marker to obtain the coordinates of the target marker in the optical coordinate system. In a possible implementation, the optical tracking system takes the center of the optical tracking system as the origin and defines three mutually perpendicular axes to establish an optical coordinate system, and tracks the target marker in real time through a camera in the optical tracking system, and determines the coordinates of the target marker in the established optical coordinate system as the third coordinate.

[0097] Optionally, there are three or more target markers on the surface of the target object. When the number of target markers is greater than or equal to 3, the number of coordinates of the third coordinates of the target markers determined in the optical coordinate system is the same as the number of target markers, and the third coordinates correspond to the target markers one by one.

[0098] 105. Based on the first conversion relationship, convert the third coordinate into a CT coordinate system to obtain a fourth coordinate.

[0099] In the embodiment of the present application, the first conversion relationship is the conversion relationship between the optical coordinate system and the CT coordinate system, and the coordinates in the optical coordinate system can be converted to the CT coordinate system through the first conversion relationship. Based on the first conversion relationship, the third coordinate in the optical coordinate system is converted to the CT coordinate system to obtain the coordinate of the target marker at the second moment in the CT coordinate system, that is, the fourth coordinate.

[0100] Optionally, there are three or more target markers on the surface of the target object, the number of coordinates of the third coordinate is the same as the number of the target markers, and the third coordinate corresponds to the target markers one-to-one. The number of coordinates of the fourth coordinate obtained by transforming the third coordinate through the first transformation relationship is the same as the number of the target markers, and the fourth coordinate corresponds to the target markers one-to-one.

[0101] 106. Based on the second coordinate and the fourth coordinate, determine a reference displacement of the target marker from the first moment to the second moment.

[0102] In this embodiment of the present application, since the second coordinates of the target marker at the first moment and the fourth coordinates at the second moment in the CT coordinate system are known, the reference displacement of the target marker from the first moment to the second moment in the CT coordinate system can be obtained according to the difference between the fourth coordinate and the second coordinate.

[0103] Optionally, there are three or more target markers on the surface of the target object. When the number of target markers is greater than or equal to 3, the number of coordinates of the second coordinate is the same as the number of target markers, the number of coordinates of the fourth coordinate is the same as the number of target markers, and the second coordinate corresponds to the target marker one-to-one, and the fourth coordinate corresponds to the target marker one-to-one. Based on the three or more target markers, the difference between the fourth coordinate corresponding to any target marker and the second coordinate is calculated to obtain three or more reference displacements, and the reference displacements correspond to the target markers one-to-one.

[0104] 107. Input the first coordinate and the reference displacement into the target model to obtain the target displacement of the target point.

[0105] In an embodiment of the present application, the target model is a model obtained by updating the parameters of the model to be trained using the target loss. The target loss is obtained based on the displacement difference of the training target in the training object from the third moment to the fourth moment, and the training difference represents the difference between the actual displacement and the predicted displacement of the training target. The actual displacement represents the displacement of the training target from the third moment to the fourth moment, and the predicted displacement represents the displacement of the training target predicted by the model to be trained from the third moment to the fourth moment. Specifically, the model to be trained obtains the predicted displacement based on the marked displacement of the training marker and the fifth coordinate prediction of the training target, and the marked displacement represents the displacement of the training marker from the third moment to the fourth moment in the CT coordinate system. The fifth coordinate represents the coordinate of the training target at the third moment in the CT coordinate system.

[0106] In an embodiment of the present application, the navigation device uses a target model to predict the displacement of a target point at a first coordinate at a first moment based on a reference displacement of a target marker in a CT coordinate system from a first moment to a second moment, as the target displacement, which is the displacement from the first moment to the second moment.

[0107] Optionally, there are three or more target markers on the surface of the target object. When the number of target markers is greater than or equal to 3, the number of displacements in the reference displacement is also greater than or equal to 3, and the reference displacement corresponds to the target marker one by one. By taking three or more reference displacements of three or more target markers in the CT coordinate system as input, the target model can obtain more abundant motion information of the target markers, thereby improving the accuracy of the target model in predicting the target displacement.

[0108] Optionally, the target CT image includes m candidate CT images, and the candidate CT images are two-dimensional images obtained by continuous CT scanning of the target object. By merging the m candidate CT images, a CT three-dimensional image is obtained, wherein the CT three-dimensional image is a three-dimensional image containing the organ to which the target target point belongs and the target marker. According to the second conversion relationship, the first coordinate and the reference displacement are converted to the organ coordinate system to obtain the seventh coordinate of the target target point at the first moment in the organ coordinate system and the candidate displacement of the target marker from the first moment to the second moment in the organ coordinate system, wherein the organ coordinate system is the coordinate system of the CT three-dimensional image. The seventh coordinate and the candidate displacement are used as inputs of the target model to obtain the target displacement of the target target point from the first moment to the second moment in the organ coordinate system. By constructing a CT three-dimensional image, more abundant information about the organ to which the target target point belongs can be provided, and the preset path of the puncture surgery can be determined from the CT three-dimensional image, reducing the possibility of unreasonable preset path planning due to insufficient information, thereby leading to the failure of the puncture surgery. For example, in the case where the target target point is a point in the lungs of the target object, Figure 3 It is a schematic diagram of a CT three-dimensional image, wherein the x-axis in the CT three-dimensional image represents the x-axis of the organ coordinate system, the y-axis in the CT three-dimensional image represents the y-axis of the organ coordinate system, and the z-axis in the CT three-dimensional image represents the z-axis of the organ coordinate system.

[0109] In the embodiment of the present application, after receiving the target CT image of the target object at the first moment, the navigation device determines the first coordinate of the target point and the second coordinate of the target marker in the CT coordinate system from the target CT image, and then uses the optical tracking system to determine the third coordinate of the target marker at the second moment in the optical coordinate system, and uses the first conversion relationship to convert the third coordinate to the CT coordinate system to obtain the coordinate of the target marker at the second moment in the CT coordinate system as the fourth coordinate. When the second coordinate of the target marker at the first moment and the fourth coordinate at the second moment in the CT coordinate system are determined, the reference displacement of the target marker from the first moment to the second moment is determined. The first coordinate and the reference displacement are used as inputs of the target model, wherein the target model is a model obtained by updating the parameters of the model to be trained through the target loss, the target loss is obtained according to the displacement difference of the training target point in the training object from the third moment to the fourth moment, the displacement difference is the difference between the actual displacement and the predicted displacement, the predicted displacement is the displacement of the training target point from the third moment to the fourth moment predicted by the model to be trained, the model to be trained obtains the predicted displacement according to the marker displacement and the fifth coordinate, the marker displacement is the displacement of the training marker on the surface of the training object from the third moment to the fourth moment, and the fifth coordinate is the coordinate of the training target point at the third moment in the CT coordinate system. Since the target model is obtained through the marker displacement and the fifth coordinate training, the target model has the ability to predict the displacement of the target target point in the CT coordinate system from the first moment to the second moment based on the reference displacement and the first coordinate. Therefore, after the first coordinate and the reference displacement are input into the target model, the target model can determine the target displacement of the target target point in the target object caused by the target movement from the first moment to the second moment based on the reference displacement and the first coordinate.

[0110] In a possible scenario, after determining the target displacement of the target point from the first moment to the second moment in the CT coordinate system, the coordinates of the target point at the first moment and the target displacement in the CT coordinate system can be used to determine the coordinates of the target point at the second moment. When the displacement of the target point at the second moment is determined, the movable robotic arm can realize real-time tracking of the target point at the second moment, and then the robotic arm can be used to perform a puncture operation on the target point, thereby reducing the influence of the displacement of the target point in the target object on the accuracy of the puncture operation.

[0111] As an optional implementation, the navigation device performs the following steps during the process of executing step 107:

[0112] 201. Input the reference displacement, the first coordinate and the second coordinate into the target model, so that the target model determines the relative position relationship between the target point and the target marker according to the first coordinate and the second coordinate.

[0113] In an embodiment of the present application, the first coordinate of the target point at the first moment in the CT coordinate system, the second coordinate of the target marker at the first moment, and the reference displacement of the target marker from the first moment to the second moment are used as inputs of the target model, so that the target model determines the relative position relationship between the target point and the target marker at the first moment in the CT coordinate system based on the first coordinate and the second coordinate.

[0114] In this embodiment, the target model is a model obtained by updating the parameters of the model to be trained using the target loss. The target loss is obtained based on the displacement difference of the training target in the training object from the third moment to the fourth moment, the training difference represents the difference between the actual displacement and the predicted displacement of the training target, the actual displacement represents the displacement of the training target from the third moment to the fourth moment, and the predicted displacement represents the displacement of the training target predicted by the model to be trained from the third moment to the fourth moment. Specifically, the model to be trained obtains the predicted displacement based on the marked displacement of the training marker, the fifth coordinate of the training target and the marker coordinate prediction, and the marked displacement represents the displacement of the training marker from the third moment to the fourth moment in the CT coordinate system. The fifth coordinate represents the coordinate of the training target at the third moment in the CT coordinate system, and the marker coordinate represents the coordinate of the target marker at the third moment in the CT coordinate system.

[0115] 202. Obtain the target displacement of the target point using the target model according to the reference displacement, the first coordinate and the relative position relationship.

[0116] In the embodiment of the present application, the navigation device uses the target model to determine the target displacement of the target point according to the reference displacement, the first coordinate and the relative position relationship.

[0117] In this embodiment, the navigation device determines the coordinates of the target target and the target marker at the first moment in the CT coordinate system, that is, the first coordinate and the second coordinate, so the relative position of the target target and the target marker at the first moment can be determined according to the first coordinate and the second coordinate. By determining the relative position relationship between the target target and the target marker, more abundant position information can be provided for the target model to predict the target displacement of the target target, thereby improving the accuracy of the target model to predict the target displacement. For example, at the first moment, the coordinate difference between the first coordinate and the second coordinate is less than the first threshold value, the distance between the target target and the target marker at the first moment is relatively close, and the target displacement of the target target from the first moment to the second moment is similar to the reference displacement of the target marker from the first moment to the second moment, so the weight of the target model affected by the reference displacement in the process of determining the target displacement is relatively high. For another example, at the first moment, the coordinate difference between the first coordinate and the second coordinate is greater than the second threshold value, the distance between the target target and the target marker at the second moment is relatively far, and the displacement of the target target from the first moment to the second moment is relatively different from the reference displacement of the target marker, so the weight of the target model affected by the reference displacement in the process of determining the target displacement is relatively low.

[0118] As an optional implementation, the navigation device performs the following steps during the process of executing step 107:

[0119] 301. Obtain a target area in the above CT coordinate system.

[0120] In the embodiment of the present application, the target region is the region of the organ to which the target point belongs in the CT coordinate system. For example, if the target point is a point in the lung of the target object, the target region is the region of the lung of the target object in the CT coordinate system. For another example, if the target point is a point in the liver of the target object, the target region is the region of the liver of the target object in the CT coordinate system.

[0121] 302. Input the target area, the reference displacement and the first coordinate into the target model to obtain the target displacement of the target point.

[0122] In an embodiment of the present application, the target area of ​​the organ to which the target target point belongs in the CT coordinate system, the first coordinates of the target target point at the first moment, and the reference displacement of the target marker from the first moment to the second moment are used as inputs of the target model to predict the target displacement of the target target point from the first moment to the second moment.

[0123] In this embodiment, the target model is a model obtained after updating the parameters of the model to be trained using the target loss. The target loss is obtained based on the displacement difference of the training target in the training object from the third moment to the fourth moment, the training difference represents the difference between the actual displacement and the predicted displacement of the training target, the actual displacement represents the displacement of the training target from the third moment to the fourth moment, and the predicted displacement represents the displacement of the training target predicted by the model to be trained from the third moment to the fourth moment. Specifically, the model to be trained obtains the predicted displacement based on the marked displacement of the training marker, the fifth coordinate of the training target and the training area prediction, and the marked displacement represents the displacement of the training marker from the third moment to the fourth moment in the CT coordinate system. The fifth coordinate represents the coordinate of the training target at the third moment in the CT coordinate system, and the training area represents the region of the organ to which the training target belongs in the CT coordinate system.

[0124] In this embodiment, when determining the target area of ​​the organ to which the target target belongs in the CT coordinate system, the navigation device determines the target displacement of the target target from the first moment to the second moment using the target model according to the reference displacement of the target marker from the first moment to the second moment in the CT coordinate system, the first coordinate of the target target at the first moment and the target area, and the coordinates of the target target at the second moment determined according to the first coordinate of the target target at the first moment and the target displacement of the target target from the first moment to the second moment are within the target area. By limiting the target displacement predicted by the navigation device using the target area, the possibility that the coordinates of the target target at the second moment predicted by the navigation device using the target model are not within the target area is reduced, thereby improving the accuracy of the target displacement predicted by the navigation device.

[0125] In a possible implementation, the navigation device uses the first coordinate, the second coordinate, the reference displacement and the target area as inputs of the target model. The navigation device uses the target model to determine the relative positional relationship between the target target and the target marker at the first moment according to the first coordinate of the target target at the first moment and the second coordinate of the target marker at the first moment in the CT coordinate system, and predicts the target displacement of the target target from the first moment to the second moment according to the relative positional relationship between the target target and the target marker and the reference displacement of the target marker from the first moment to the second moment, and determines that the coordinates of the target target at the second moment are within the target area according to the first coordinate of the target target at the first moment and the target displacement of the target target from the first moment to the second moment.

[0126] In this embodiment, the target model is a model obtained after updating the parameters of the model to be trained using the target loss. The target loss is obtained based on the displacement difference of the training target in the training object from the third moment to the fourth moment, the training difference represents the difference between the actual displacement and the predicted displacement of the training target, the actual displacement represents the displacement of the training target from the third moment to the fourth moment, and the predicted displacement represents the displacement of the training target predicted by the model to be trained from the third moment to the fourth moment. Specifically, the model to be trained obtains the predicted displacement based on the marker displacement of the training marker, the marker coordinates of the training marker, the fifth coordinate of the training target, and the training area prediction. The marker displacement represents the displacement of the training marker from the third moment to the fourth moment in the CT coordinate system, the fifth coordinate represents the coordinates of the training target at the third moment in the CT coordinate system, the training area represents the region of the organ to which the training target belongs in the CT coordinate system, and the marker coordinates represent the coordinates of the target marker at the third moment in the CT coordinate system.

[0127] In this implementation, by determining the relative positional relationship between the target point and the target marker, more abundant positional information can be provided for the target model to predict the target displacement of the target point, thereby improving the accuracy of the target model predicting the target displacement. By using the target area to limit the target point predicted by the navigation device, the possibility of the navigation device using the target model to predict the displacement that will cause the target point to move to an organ that does not belong to the target point is reduced, thereby improving the accuracy of the target displacement predicted by the navigation device.

[0128] For example, Figure 4The figure is a training flow diagram of a target model in this implementation. Specifically, the first CT image of the training object at the third moment and the second CT image at the fourth moment are taken as a set of target motion phase images, and the coordinates of 10,000 points are uniformly sampled from the first CT image to obtain the relative coordinates of the sampling points, which are the training targets for this training. The coordinates of the training markers at the third moment are obtained from the first CT image, wherein the training markers include marker ball 1, marker ball 2, marker ball 3, and marker ball 4. The coordinates of marker ball 1 in the first CT image are used as the relative position of marker ball 1. The coordinates of marker ball 2 in the first CT image are used as the relative position of marker ball 2. The coordinates of marker ball 3 in the first CT image are used as the relative position of marker ball 3. The coordinates of marker ball 4 in the first CT image are used as the relative position of marker ball 4. The coordinates of the training marker at the fourth moment in the CT coordinate system are obtained from the second CT image. Specifically, the coordinates of the marker ball 1 at the fourth moment in the CT coordinate system are the first training coordinates, the coordinates of the marker ball 2 at the fourth moment in the CT coordinate system are the second training coordinates, the coordinates of the marker ball 3 at the fourth moment in the CT coordinate system are the third training coordinates, and the coordinates of the marker ball 4 at the fourth moment in the CT coordinate system are the fourth training coordinates. According to the relative position of the marker ball 1 at the third moment in the CT coordinate system and the first training coordinates at the fourth moment, the coordinate displacement of the marker ball 1 from the third moment to the fourth moment in the CT coordinate system is obtained. According to the relative position of the marker ball 2 at the third moment in the CT coordinate system and the second training coordinates at the fourth moment, the coordinate displacement of the marker ball 2 from the third moment to the fourth moment in the CT coordinate system is obtained. According to the relative position of the marker ball 3 at the third moment in the CT coordinate system and the third training coordinates at the fourth moment, the coordinate displacement of the marker ball 3 from the third moment to the fourth moment in the CT coordinate system is obtained. According to the relative position of the marker ball 4 at the third moment in the CT coordinate system and the fourth training coordinate at the fourth moment, the coordinate displacement of the marker ball 4 from the third moment to the fourth moment in the CT coordinate system is obtained. The region of the organ to which the training target belongs is determined from the first CT image to obtain the training region. The relative coordinates of the sampling point, the coordinate displacement of the marker ball 1, the coordinate displacement of the marker ball 2, the coordinate displacement of the marker ball 3, the coordinate displacement of the marker ball 4, the relative position of the marker ball 1, the relative position of the marker ball 2, the relative position of the marker ball 3, the relative position of the marker ball 4, and the training region are used as inputs of the model to be trained, so that it outputs the predicted displacement of the training target from the third moment to the fourth moment. After determining the predicted displacement of the training target from the third moment to the fourth moment, the actual displacement of the training target from the third moment to the fourth moment can be determined according to the second CT image and the first CT image.After determining the predicted displacement and the actual displacement, the target loss of the model to be trained can be determined according to the difference between the predicted displacement and the actual displacement, and the parameters of the model to be trained can be updated according to the target loss to obtain the target model. Optionally, the training object includes one or more objects. When the number of objects in the training object is greater than 1, the more objects in the training object, the more data is used to train the model to be trained. The more data is used to train the model to be trained, the higher the accuracy of the target model obtained by training.

[0129] As an optional implementation, the navigation device performs the following steps after executing step 107:

[0130] 401. Determine the coordinates of the target point at the second moment in the CT coordinate system according to the first coordinates and the target displacement to obtain a sixth coordinate.

[0131] In the embodiment of the present application, the sixth coordinate of the target point at the second moment can be determined according to the first coordinate of the target point at the first moment and the target displacement of the target point from the first moment to the second moment in the CT coordinate system.

[0132] 402. Determine a target position and posture of the robotic arm according to the sixth coordinate.

[0133] In the embodiment of the present application, the posture represents the position and posture of the robot arm, and the puncture guide rail on the robot arm can be moved by adjusting the posture of the robot arm. Specifically, the preset path of the puncture surgery in the CT coordinate system is determined by the sixth coordinate, and the target posture is then determined by the preset path, so that after adjusting the posture of the robot arm to the target posture, the puncture path in the puncture guide rail coincides with the needle insertion path, wherein the needle insertion path is the path converted from the preset path to the optical coordinate system.

[0134] Optional, Figure 5 Schematic diagram of the structure of a navigation device provided in an embodiment of the present application. Figure 5 The puncture path following device includes a mechanical arm and an optical tracking system, wherein the mechanical arm includes a mechanical arm joint, a base, a flange, a tracer, a puncture guide rail, and a puncture needle. The flange is used to fix the base and the tracer. There are m reference markers on the tracer, and the optical tracking system tracks the tracer by tracking the m reference markers on the tracer. The puncture guide rail is fixed on the tracer, and the puncture guide rail includes an introducer, which is used to clamp the puncture needle, wherein the introducer includes a tip and an end, the tip of the guide is the position where the puncture needle comes out of the needle, and the end of the guide is the position where the puncture needle enters the needle.

[0135] The coordinate system constructed based on the base is the base coordinate system. The coordinate system constructed based on the flange is the flange coordinate system. The coordinate system constructed based on the tracer is the tracer coordinate system, wherein the origin of the tracer coordinate system can be any reference marker on the tracer. The coordinate system constructed based on the tip reference point is the tip coordinate system, wherein the origin of the tip coordinate system is the tip reference point, the tip reference point is the point of the tip of the introducer in the puncture guide rail, and the vertical axis of the tip coordinate system is parallel to the axis of the introducer. The coordinate system constructed based on the needle insertion point is the needle insertion point coordinate system, wherein the needle insertion point is the point where the needle insertion path intersects with the body surface of the target object, and the origin of the needle insertion point coordinate system is the needle insertion point. The transformed needle insertion point coordinate system is the coordinate system obtained by translating and rotating the needle insertion point coordinate system outward along the needle insertion path.

[0136] The specific steps of the puncture path following device determining the target posture using the above preset path are as follows:

[0137] Obtain the transformation matrix T0 from the flange coordinate system to the base coordinate system, the transformation matrix T1 from the tracer coordinate system to the flange coordinate system, and the transformation matrix T2 from the tip coordinate system to the tracer coordinate system;

[0138] According to the transformation matrix T1 from the tracer coordinate system to the flange coordinate system and the transformation matrix T2 from the tip coordinate system to the tracer coordinate system, the transformation matrix T3 from the tip coordinate system to the flange coordinate system can be obtained;

[0139] Determine the preset path v0 in the CT coordinate system according to the sixth coordinate, for example, Figure 6 It is a schematic diagram of planning a preset path based on a target CT image. Specifically, the preset path passes through a needle entry point on the surface of a target object in the target CT image and a target point in the body. Figure 7 It is a schematic diagram of performing a puncture operation on a target object based on a needle insertion path. Specifically, a preset path is converted from a CT coordinate system to an optical coordinate system to obtain a needle insertion path, and then the target object is punctured according to the needle insertion path, and the needle insertion point on the surface of the target object and the target point in the body are passed through during the puncture process;

[0140] The target CT image and the target marker in the optical coordinate system are registered to obtain the transformation matrix M1 from the CT coordinate system to the optical coordinate system, and then the rotation matrix R1 and the translation matrix P1 from the CT coordinate system to the optical coordinate system are determined according to the transformation matrix M1. For example, Figure 8is a schematic diagram of rigid registration. Specifically, the target CT image includes 5 target markers, and the 5 target markers in the target CT image correspond to the 5 target markers attached to the surface of the target object, and the coordinate group Ma of the 5 target markers in the CT coordinate system is determined. The coordinate group Mb of the target marker in the optical coordinate system is determined, and the transformation matrix M1 is obtained by rigidly registering Ma and Mb, wherein the CT marker represents the target marker in the target CT image, and the marker tracked by the optical system is the target marker tracked by the optical tracking system;

[0141] The optical tracking system is used to track the reference markers on the tracer in real time, determine the transformation matrix M2 from the tracer coordinate system to the optical coordinate system, and then Determine the rotation matrix R2 and translation matrix P2 from the tracer coordinate system to the optical coordinate system;

[0142] The transformation matrix M3 from the tip coordinate system to the optical coordinate system is determined based on the transformation matrix M2 and the transformation matrix T2, wherein the transformation matrix T2 is the transformation matrix from the tip coordinate system to the tracer coordinate system. Determine the rotation matrix R3 and translation matrix P3 from the tip coordinate system to the optical coordinate system;

[0143] The direction vector v1 of the tip reference point in the optical coordinate system is obtained, wherein the direction of v1 is parallel to the direction of the vertical axis root of the tip coordinate system. Therefore, the direction vector v1 can be obtained by rotating the direction of the tip reference point pointing to the target in the tip coordinate system. The tip coordinate system is a coordinate system constructed based on the tip reference point. The specific formula is as follows:

[0144] …Formula (1)

[0145] Where R3 is the rotation matrix from the tip coordinate system to the optical coordinate system. is the unit vector of the tip reference point in the positive direction of the Z axis. v1 is the direction vector of the tip reference point in the optical coordinate system.

[0146] The direction vector v2 of the puncture path in the optical coordinate system is determined according to the preset path v0 in the CT coordinate system and the rotation matrix R1, where the direction of the direction vector v2 is the direction from the needle insertion point to the target point. The specific formula is as follows:

[0147] …Formula (2)

[0148] Among them, R1 is the rotation matrix from the CT coordinate system to the optical coordinate system, v0 is the preset path in the CT coordinate system, and v2 is the direction vector of the puncture path in the optical coordinate system.

[0149] Determine the normal vector v3 based on the direction vector v1 and the direction vector v2. The specific formula is as follows:

[0150] …Formula (3)

[0151] Among them, v1 is the direction vector of the tip reference point in the optical coordinate system, v2 is the direction vector of the puncture path in the optical coordinate system, and v3 is the normal vector of the direction vector v1 and the direction vector v2. To calculate the normal vector of vector A and vector B.

[0152] The normal vector v3 is converted to the tip coordinate system to obtain the normal vector v4 in the tip coordinate system. The specific formula is as follows:

[0153] …Formula (4)

[0154] Among them, v3 is the normal vector of direction vector v1 and direction vector v2, is the inverse of the rotation matrix from the tip coordinate system to the optical coordinate system, and v4 is the normal vector in the tip coordinate system.

[0155] According to the angle between the normal vector v4 and the direction vector v1 and the direction vector v2 in the tip coordinate system , determine the rotation matrix R4 from the needle entry point to the tip reference point, where the needle entry point is the point where the puncture needle penetrates the body surface of the target object during the puncture surgery. The specific formula is as follows:

[0156]

[0157]

[0158] Among them, v4 represents the normal vector in the tip coordinate system. Represents the angle between direction vector v1 and direction vector v2. R4 represents the rotation matrix from the needle insertion point to the tip reference point. Rodrigues(C,D) represents the calculation of C and D using the Rodrigues formula. angle(E,F) represents the angle between E and F.

[0159] Obtain the direction vector v5 from the tip to the needle insertion point in the optical coordinate system;

[0160] According to the direction vector v5 converted to the tip coordinate system, the translation matrix P4 from the needle insertion point coordinate system to the tip reference point is obtained. The specific formula is as follows:

[0161] …Formula (7)

[0162] in, It is represented as the inverse of the rotation matrix from the tip coordinate system to the optical coordinate system, v5 represents the direction vector from the tip to the needle insertion point in the optical coordinate system, and P4 represents the translation matrix from the needle insertion point coordinate system to the tip reference point.

[0163] The transformation matrix T4 from the needle insertion point to the tip reference point can be determined based on the rotation matrix R4 and the translation matrix P4. The specific formula is:

[0164] …Formula (8)

[0165] Among them, R4 represents the rotation matrix from the needle entry point to the tip reference point, and P4 represents the translation matrix from the needle entry point coordinate system to the tip reference point.

[0166] The needle insertion point coordinate system is translated outward along the needle insertion path by a distance d to keep the tip and the needle insertion point at a safe distance, and the transformed needle insertion point coordinate system and the transformation matrix T5 of the needle insertion point coordinate system are obtained. The formula of the transformation matrix T5 is as follows:

[0167]

[0168] After determining the transformation matrix T5 of the needle entry point coordinate system after translation and the needle entry point coordinate system before translation, the transformation matrix T4 from the needle entry point to the tip reference point, the transformation matrix T3 from the tip coordinate system to the flange coordinate system, and the transformation matrix T0 from the flange coordinate system to the base coordinate system, the target position T of the robot arm can be calculated according to the following formula.

[0169] …Formula (10)

[0170] Among them, T0 represents the conversion matrix from the flange coordinate system to the base coordinate system; T3 represents the conversion matrix from the tip coordinate system to the flange coordinate system; T4 represents the conversion matrix from the needle entry point to the tip reference point; T5 represents the conversion matrix from the needle entry point coordinate system after translation to the needle entry point coordinate system before translation; The inverse matrix that represents the transformation matrix from the tip coordinate system to the flange coordinate system.

[0171] 403. Control the movement of the robotic arm so that the posture of the robotic arm is the target posture.

[0172] In an embodiment of the present application, the navigation device moves the mechanical arm according to the requirements of the above-mentioned target posture, and moves the puncture guide rail on the mechanical arm by moving the mechanical arm. Specifically, the mechanical arm includes a mechanical arm joint, and the joint angle of the mechanical arm joint is determined by using the target posture. The movement of the puncture guide rail on the mechanical arm is achieved by adjusting the mechanical arm joint. When the angle of the mechanical arm joint is adjusted to the above-mentioned joint angle, the puncture path in the puncture guide rail coincides with the needle insertion path.

[0173] In a possible implementation, the target posture T is converted into a translation vector and a rotation vector to obtain [Tx, Ty, Tz, Rx, Ry, Rz], where Tx, Ty, Tz represent the translation vectors on the x-axis, y-axis, and z-axis respectively, and Rx, Ry, Rz represent the rotation vectors on the x-axis, y-axis, and z-axis respectively. And it is converted into the joint angles [j1, j2, j3, j4, j5, j6] of the manipulator joints through inverse kinematics. After determining the joint angles, the navigation device adjusts the manipulator joints according to the joint angles.

[0174] Optionally, the optical tracking system obtains the third coordinate of the target marker in the optical coordinate system every N milliseconds, and then determines the target posture of the robotic arm through steps 101 to 107 and steps 401 to 403. After determining the target posture, the robotic arm is controlled to move so that the posture of the robotic arm is in the target posture, and the puncture guide rail on the robotic arm is driven to move. In the process of controlling the movement of the robotic arm, it will not be affected by the movement instructions before the target posture is determined. When the posture of the robotic arm is in the target posture, the puncture path in the puncture guide rail on the robotic arm passes through the target target point. For example, after the optical tracking system obtains the third coordinate of the target marker in the optical coordinate system at the fifth moment, it determines the target posture of the robotic arm at the fifth moment. There is a movement instruction generated at the sixth moment before the fifth moment. After determining the target position of the robot arm at the fifth moment, the robot arm is controlled to move so that the position of the robot arm is at the target position, and the robot arm is not affected by the above movement command during the process of controlling the movement of the robot arm. When the position of the robot arm is at the target position, the puncture path in the puncture guide rail on the robot arm passes through the target target point, and the puncture path is tracked in real time at the fifth moment. When the position of the robot arm is controlled to move to the target position, it is not affected by the movement command before the target position is determined. Therefore, it is possible to avoid the robot arm from conflicting with the target position due to the movement command at a non-current moment, resulting in the puncture path in the puncture guide rail failing to pass through the target target point, thereby failing to achieve real-time tracking of the puncture path and the target target point.

[0175] In one possible implementation, the target posture is converted into the joint angle of the robot arm joint, and the joint angle is sent to the robot arm with a preset fixed sending period using online joint position command control (Joint Space Servoing, ServoJ), and the angle of the robot arm joint is adjusted to the above joint angle in real time to achieve real-time tracking of the target.

[0176] In another possible implementation, the target posture is converted into the spatial position of the target point in the optical coordinate system, and the spatial position is sent to the robot arm with a preset fixed sending period using online Cartesian spatial position command control (Linear Servoing, ServoL), and the robot arm is controlled to track the above spatial position to achieve real-time tracking of the target point.

[0177] In the above two implementations, the control of the robot arm is realized by using servo control instructions (the instructions include ServoJ and ServoL), which can improve the accuracy of the movement of the robot arm and speed up the response speed when controlling the robot arm. Specifically, the navigation device uses feedback elements such as encoders and sensors in the robot arm to monitor the posture, speed and acceleration of the robot arm in real time, and uses the target posture, the posture, speed and acceleration of the robot arm to determine the servo control instructions for controlling the movement of the robot arm, so as to control the posture of the robot arm to the target posture, wherein the real-time monitoring of the posture of the robot arm enables the navigation device to accurately determine the posture of the robot arm, and avoid the posture error generated when controlling the movement of the robot arm, which causes the posture of the robot arm to be not in the target posture, thereby improving the accuracy of the movement of the robot arm. By real-time monitoring of the speed and acceleration of the robot arm, the navigation device can accurately determine the motion state of the robot arm, and after generating the target posture, the motion state of the robot arm can be adjusted to control the robot arm to move to the target posture, thereby improving the response speed of the robot arm. In summary, using servo control instructions to control the robotic arm can improve the movement accuracy of the robotic arm and increase the response speed when controlling the robotic arm.

[0178] In this embodiment, after determining the first coordinate of the target target at the first moment and the target displacement from the first moment to the second moment in the CT coordinate system, the navigation device determines the sixth coordinate of the target target at the second moment, and determines the target posture of the robotic arm according to the sixth coordinate, wherein the target posture is used to move the robotic arm, thereby realizing the movement of the puncture guide rail on the robotic arm. By moving the puncture guide rail, the puncture path in the puncture guide rail passes through the target target. After determining the target posture, the robotic arm is moved according to the requirements of the target posture, and then the puncture guide rail is moved, and the puncture path in the puncture guide rail passes through the target target. After determining the coordinates of the target target at the second moment, the navigation device can move the robotic arm so that the puncture path in the puncture guide rail passes through the target target, thereby realizing the tracking of the target target, and reducing the influence of the target movement of the target object from the first moment to the second moment on the accuracy of the puncture surgery caused by the movement of the target target in the above-mentioned target object.

[0179] In one possible implementation, Fig. 9The present invention is a workflow diagram of a navigation device based on displacement prediction. The specific workflow is as follows: start the workflow of the navigation device; use the optical tracking system to track the target marker and the tracer on the surface of the target object, specifically, there is a reference marker on the tracer, and the navigation device uses the optical tracking system to track the target marker on the surface of the target object and the reference marker on the tracer at the second moment; obtain the current position of the manipulator at the second moment; calculate the target position of the manipulator at the second moment according to steps 101 to 107, specifically, determine the target displacement of the target point from the first moment to the second moment in the CT coordinate system, determine the second coordinate of the target point at the second moment according to the first coordinate and the target displacement of the target point at the first moment in the CT coordinate system, and plan a preset path in the target CT image, and the preset path passes through the above-mentioned second coordinate. The target position of the robot arm at the second moment is calculated according to the preset path; when the target position is determined, ServoJ is used to send the target position to the robot arm to enable the robot arm to track the target point at the second moment. For specific implementation steps, please refer to step 403; after the target point is tracked at the second moment, it is determined whether the target point needs to be tracked. If the target point needs to be tracked, the above steps are repeated. If the target point does not need to be tracked, the above workflow ends.

[0180] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0181] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0182] See also Fig.10 , Fig.10 The present invention provides a structural diagram of a navigation device based on displacement prediction according to an embodiment of the present invention. The navigation device based on displacement prediction 1 comprises: an acquisition unit 11, a determination unit 12, a conversion unit 13, and a displacement prediction unit 14. Among them:

[0183] An acquisition unit 11 is used to acquire a target CT image of the target object at the first moment, wherein the target CT image includes the target point and a target marker on the surface of the target object;

[0184] A determination unit 12, configured to determine a first coordinate of the target point in a CT coordinate system of the target CT image, wherein the CT coordinate system is a coordinate system of the target CT image;

[0185] The determination unit 12 is further configured to determine a second coordinate of the target marker in the CT coordinate system;

[0186] The acquisition unit 11 is further used to acquire a third coordinate of the target marker in the optical coordinate system, where the third coordinate is the coordinate of the target marker at the second moment, and the optical coordinate system is a coordinate system of the optical tracking system;

[0187] A conversion unit 13, configured to convert the third coordinate into a CT coordinate system to obtain a fourth coordinate based on a first conversion relationship, wherein the first conversion relationship is a conversion relationship between the optical coordinate system and the CT coordinate system;

[0188] The determination unit 12 is further configured to determine a reference displacement of the target marker from the first moment to the second moment based on the second coordinate and the fourth coordinate;

[0189] The displacement prediction unit 14 is used to input the first coordinate and the reference displacement into the target model to obtain the target displacement of the target point, wherein the target model is obtained by updating the parameters of the model to be trained based on the target loss, wherein the target loss is obtained according to the displacement difference of the training target point in the training object from the third moment to the fourth moment, wherein the displacement difference is the difference between the actual displacement and the predicted displacement, wherein the predicted displacement is the displacement of the training target point predicted by the model to be trained from the third moment to the fourth moment, wherein the model to be trained obtains the predicted displacement according to the marker displacement and the fifth coordinate, wherein the marker displacement is the displacement of the training marker on the surface of the training object from the third moment to the fourth moment, and the fifth coordinate is the coordinate of the training target point at the third moment in the CT coordinate system.

[0190] In combination with any embodiment of the present application, the displacement prediction unit 14 is further configured to:

[0191] Inputting the reference displacement, the first coordinate and the second coordinate into the target model, so that the target model determines the relative position relationship between the target point and the target marker according to the first coordinate and the second coordinate;

[0192] The target displacement of the target point is obtained by using the target model according to the reference displacement, the first coordinate and the relative position relationship.

[0193] In combination with any embodiment of the present application, the displacement prediction unit 14 is further configured to:

[0194] Acquire a target region in the CT coordinate system, wherein the target region represents a region of the organ to which the target point belongs;

[0195] The target area, the reference displacement and the first coordinate are input into the target model to obtain the target displacement of the target point. According to the first coordinate and the target displacement of the target point at the first moment, the coordinate of the target point at the second moment is within the target area.

[0196] In combination with any embodiment of the present application, the mechanical arm is moved according to the target displacement, and the puncture guide rail on the mechanical arm is driven to move by moving the mechanical arm;

[0197] The navigation device 1 based on displacement prediction further includes a moving unit 15, which is used to:

[0198] Determine the coordinates of the target point at the second moment in the CT coordinate system according to the first coordinates and the target displacement to obtain a sixth coordinate;

[0199] Determine the target posture of the robotic arm according to the sixth coordinate, and when the posture of the robotic arm is the target posture, a puncture path determined based on the puncture guide rail of the robotic arm passes through the target target point;

[0200] The robot arm is controlled to move so that the posture of the robot arm is the target posture.

[0201] In combination with any embodiment of the present application, the acquisition unit 11 is specifically used for:

[0202] Acquire the third coordinate of the target marker in the optical coordinate system every N milliseconds;

[0203] The mobile unit 15 is specifically used for:

[0204] After determining the target posture, the robot arm is controlled to move so that the posture of the robot arm is the target posture, and the movement of the robot arm is not affected by the movement instruction before determining the target posture, and the movement instruction is an instruction for controlling the movement of the robot arm.

[0205] In combination with any embodiment of the present application, the target CT image includes m candidate CT images; the candidate CT images are two-dimensional images obtained by performing continuous CT scans on the target object;

[0206] The displacement prediction unit 14 is specifically used for:

[0207] Merging the m candidate CT images to obtain a CT three-dimensional image;

[0208] According to a second conversion relationship, the first coordinate is converted into an organ coordinate system to obtain a seventh coordinate, wherein the second conversion relationship is a conversion relationship between the CT coordinate system and the organ coordinate system, and the organ coordinate system is a coordinate system where the CT three-dimensional image is located;

[0209] According to the second conversion relationship, the reference displacement is converted into the organ coordinate system to obtain a candidate displacement;

[0210] The seventh coordinate and the candidate displacement are input into the target model to obtain the target displacement of the target point.

[0211] In the embodiment of the present application, after receiving the target CT image of the target object at the first moment, the navigation device determines the first coordinate of the target point and the second coordinate of the target marker in the CT coordinate system from the target CT image, and then uses the optical tracking system to determine the third coordinate of the target marker at the second moment in the optical coordinate system, and uses the first conversion relationship to convert the third coordinate to the CT coordinate system to obtain the coordinate of the target marker at the second moment in the CT coordinate system as the fourth coordinate. When the second coordinate of the target marker at the first moment and the fourth coordinate at the second moment in the CT coordinate system are determined, the reference displacement of the target marker from the first moment to the second moment is determined. The first coordinate and the reference displacement are used as inputs of the target model, wherein the target model is a model obtained by updating the parameters of the model to be trained through the target loss, the target loss is obtained according to the displacement difference of the training target point in the training object from the third moment to the fourth moment, the displacement difference is the difference between the actual displacement and the predicted displacement, the predicted displacement is the displacement of the training target point from the third moment to the fourth moment predicted by the model to be trained, the model to be trained obtains the predicted displacement according to the marker displacement and the fifth coordinate, the marker displacement is the displacement of the training marker on the surface of the training object from the third moment to the fourth moment, and the fifth coordinate is the coordinate of the training target point at the third moment in the CT coordinate system. Since the target model is obtained through the marker displacement and the fifth coordinate training, the target model has the ability to predict the displacement of the target target point in the CT coordinate system from the first moment to the second moment based on the reference displacement and the first coordinate. Therefore, after the first coordinate and the reference displacement are input into the target model, the target model can determine the target displacement of the target target point in the target object caused by the target movement from the first moment to the second moment based on the reference displacement and the first coordinate.

[0212] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0213] Fig.11A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 2 includes a processor 21 and a memory 22. Optionally, the electronic device 2 also includes an input device 23 and an output device 24. The processor 21, the memory 22, the input device 23 and the output device 24 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, it can be connected through various interfaces, transmission lines, buses, etc.

[0214] The processor 21 may include one or more processors, for example, one or more central processing units (CPUs). When the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Optionally, the processor 21 may be a processor group consisting of multiple CPUs, and the multiple processors are coupled to each other through one or more buses. Optionally, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.

[0215] The memory 22 can be used to store computer program instructions and various computer program codes including the program code for executing the program code of the present application. Optionally, the memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM), or portable read only memory (CD-ROM), which is used for related instructions and data.

[0216] The input device 23 is used to input data and / or signals, and the output device 24 is used to output data and / or signals. The input device 23 and the output device 24 can be independent devices or an integrated device.

[0217] It can be understood that in the embodiment of the present application, the memory 22 can be used not only to store relevant instructions, but also to store relevant data. For example, the memory 22 can be used to store a target CT image obtained through the input device 23, or the memory 22 can also be used to store a target displacement obtained by the processor 21, etc. The embodiment of the present application does not limit the specific data stored in the memory.

[0218] Understandably, Fig.11Only a simplified design of an electronic device is shown. In practical applications, the electronic device may also include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the protection scope of the present application.

[0219] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0220] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. Those skilled in the art can also clearly understand that the descriptions of the various embodiments of the present application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, refer to the records of other embodiments.

[0221] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0222] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0223] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0224] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0225] A person skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by a computer program to instruct the relevant hardware, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above-mentioned method embodiments. The aforementioned storage medium includes: a read-only memory (ROM) or a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

Claims

1. A navigation method based on displacement prediction, characterized in that: The method is used to determine a target displacement of a target point in the target object from the first moment to the second moment caused by a target movement of the target object from the first moment to the second moment, and the method comprises: Acquire a target CT image of the target object at the first moment, the target CT image including the target target point and a target marker on the surface of the target object; the target CT image includes m candidate CT images; the candidate CT images are two-dimensional images obtained by performing continuous CT scanning on the target object; Determine a first coordinate of the target point in a CT coordinate system of the target CT image, where the CT coordinate system is a coordinate system of the target CT image; Determining a second coordinate of the target marker in the CT coordinate system; Acquire a third coordinate of the target marker in an optical coordinate system, where the third coordinate is a coordinate of the target marker at a second moment, and the optical coordinate system is a coordinate system of an optical tracking system; Based on a first conversion relationship, convert the third coordinate into a CT coordinate system to obtain a fourth coordinate, wherein the first conversion relationship is a conversion relationship between the optical coordinate system and the CT coordinate system; Determining a reference displacement of the target marker from the first moment to the second moment based on the second coordinate and the fourth coordinate; Inputting the first coordinate and the reference displacement into a target model to obtain the target displacement of the target point, wherein the target model is obtained by updating the parameters of the model to be trained based on a target loss, wherein the target loss is obtained according to the displacement difference of the training target point in the training object from the third moment to the fourth moment, wherein the displacement difference is the difference between the actual displacement and the predicted displacement, wherein the predicted displacement is the displacement of the training target point predicted by the model to be trained from the third moment to the fourth moment, wherein the model to be trained obtains the predicted displacement according to a marker displacement and a fifth coordinate, wherein the marker displacement is the displacement of a training marker on the surface of the training object from the third moment to the fourth moment, and wherein the fifth coordinate is the coordinate of the training target point at the third moment in the CT coordinate system; The step of inputting the first coordinate and the reference displacement into a target model to obtain the target displacement of the target point includes: merging the m candidate CT images to obtain a CT three-dimensional image; converting the first coordinate into an organ coordinate system according to a second conversion relationship to obtain a seventh coordinate, wherein the second conversion relationship is a conversion relationship between the CT coordinate system and the organ coordinate system, and the organ coordinate system is a coordinate system in which the CT three-dimensional image is located; converting the reference displacement into the organ coordinate system according to the second conversion relationship to obtain a candidate displacement; and inputting the seventh coordinate and the candidate displacement into a target model to obtain the target displacement of the target point; Determine the coordinates of the target point at the second moment in the CT coordinate system according to the first coordinates and the target displacement to obtain a sixth coordinate; Determine the target posture of the robotic arm according to the sixth coordinate, and when the posture of the robotic arm is the target posture, a puncture path determined based on the puncture guide rail of the robotic arm passes through the target target point; The robot arm is controlled to move so that the posture of the robot arm is the target posture.

2. The method according to claim 1, characterized in that: The step of inputting the first coordinate and the reference displacement into a target model to obtain the target displacement of the target point includes: Inputting the reference displacement, the first coordinate and the second coordinate into the target model, so that the target model determines the relative position relationship between the target point and the target marker according to the first coordinate and the second coordinate; The target displacement of the target point is obtained by using the target model according to the reference displacement, the first coordinate and the relative position relationship.

3. The method according to claim 1, characterized in that The step of inputting the first coordinate and the reference displacement into a target model to obtain the target displacement of the target point includes: Acquire a target region in the CT coordinate system, wherein the target region represents a region of the organ to which the target point belongs; The target area, the reference displacement and the first coordinate are input into the target model to obtain the target displacement of the target point. According to the first coordinate and the target displacement of the target point at the first moment, the coordinate of the target point at the second moment is within the target area.

4. The method according to claim 1, characterized in that The step of acquiring the third coordinate of the target marker in the optical coordinate system comprises: Acquire the third coordinate of the target marker in the optical coordinate system every N milliseconds; The controlling the movement of the mechanical arm so that the posture of the mechanical arm is the target posture comprises: After determining the target posture, the robot arm is controlled to move so that the posture of the robot arm is the target posture, and the movement of the robot arm is not affected by the movement instruction before determining the target posture, and the movement instruction is an instruction for controlling the movement of the robot arm.

5. A navigation device based on displacement prediction, characterized in that: The navigation device based on displacement prediction is used to determine the target displacement of a target point in the target object from the first moment to the second moment caused by the target movement of the target object from the first moment to the second moment, and the navigation device based on displacement prediction includes: an acquisition unit, configured to acquire a target CT image of the target object at the first moment, wherein the target CT image includes the target target point and a target marker on the surface of the target object; the target CT image includes m candidate CT images; the candidate CT images are two-dimensional images obtained by performing continuous CT scanning on the target object; A determination unit, configured to determine a first coordinate of the target point in a CT coordinate system of the target CT image, wherein the CT coordinate system is a coordinate system of the target CT image; The determining unit is further used to determine a second coordinate of the target marker in the CT coordinate system; The acquisition unit is further used to acquire a third coordinate of the target marker in the optical coordinate system, where the third coordinate is the coordinate of the target marker at the second moment, and the optical coordinate system is the coordinate system of the optical tracking system; a conversion unit, configured to convert the third coordinate into a CT coordinate system to obtain a fourth coordinate based on a first conversion relationship, wherein the first conversion relationship is a conversion relationship between the optical coordinate system and the CT coordinate system; The determining unit is further used to determine a reference displacement of the target marker from the first moment to the second moment based on the second coordinate and the fourth coordinate; a displacement prediction unit, used for inputting the first coordinate and the reference displacement into a target model to obtain the target displacement of the target point, wherein the target model is obtained by updating the parameters of the model to be trained based on a target loss, wherein the target loss is obtained according to a displacement difference of the training target point in the training object from a third moment to a fourth moment, wherein the displacement difference is a difference between an actual displacement and a predicted displacement, wherein the predicted displacement is the displacement of the training target point predicted by the model to be trained from the third moment to the fourth moment, wherein the model to be trained obtains the predicted displacement according to a marker displacement and a fifth coordinate, wherein the marker displacement is the displacement of a training marker on the surface of the training object from the third moment to the fourth moment, and wherein the fifth coordinate is the coordinate of the training target point at the third moment in the CT coordinate system; The step of inputting the first coordinate and the reference displacement into a target model to obtain the target displacement of the target point includes: merging the m candidate CT images to obtain a CT three-dimensional image; converting the first coordinate into an organ coordinate system according to a second conversion relationship to obtain a seventh coordinate, wherein the second conversion relationship is a conversion relationship between the CT coordinate system and the organ coordinate system, and the organ coordinate system is a coordinate system in which the CT three-dimensional image is located; converting the reference displacement into the organ coordinate system according to the second conversion relationship to obtain a candidate displacement; and inputting the seventh coordinate and the candidate displacement into a target model to obtain the target displacement of the target point; The determining unit is further used to determine the coordinates of the target point at the second moment in the CT coordinate system according to the first coordinates and the target displacement, so as to obtain a sixth coordinate; The determination unit is further configured to determine a target posture of the robotic arm according to the sixth coordinate, and when the posture of the robotic arm is the target posture, a puncture path determined based on the puncture guide rail of the robotic arm passes through the target target point; The control unit is also used to control the movement of the robotic arm so that the posture of the robotic arm is the target posture.

6. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 4.

8. A computer program product, characterized in that The computer program product comprises a computer program or instructions; when the computer program or instructions are run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 4.

Images