Positioning and navigation method, positioning and navigation device, electronic device and readable storage medium

Abstract

The application discloses a positioning navigation method, a positioning navigation device, an electronic device and a readable storage medium, and belongs to the field of medical equipment. The positioning navigation method comprises the following steps: determining first conversion data for mutual conversion between depth data of a camera and space data of a base of a visualized mechanical arm model; determining second conversion data for mutual conversion between a visualized medical device model and space data of a tail end of the visualized mechanical arm model; determining third conversion data for mutual conversion between the depth data of the camera and space data of a visualized target object model; determining fourth conversion data for mutual conversion between the base of the visualized mechanical arm model and the tail end of the visualized mechanical arm model; obtaining a relative position relationship between the visualized target object model and the visualized medical device model based on the first conversion data, the second conversion data, the third conversion data and the fourth conversion data; and navigating according to the relative position relationship.

Description

Technical Field

[0001] This invention relates to the field of medical devices, and in particular to a positioning and navigation method, a positioning and navigation device, an electronic device, and a readable storage medium. Background Technology

[0002] With the rapid development of medical imaging and medical image processing technologies, image-guided operating systems have emerged. Utilizing three-dimensional reconstruction models of medical images, operators can intuitively and accurately analyze the subject's structure and surrounding tissues, thereby facilitating the corresponding procedures.

[0003] In the aforementioned process, optical-assisted navigation is an effective medical navigation system. It utilizes an optical tracking device in conjunction with, for example, a six-axis robot to form a robotic operating system. The system uses the robot to hold the medical device, the optical tracking device is fixed by a support, and markers are attached to the target object (e.g., the subject, specifically the subject's head) to allow the optical tracking device to locate, for example, the coordinates of the head.

[0004] However, some navigation systems are relatively cumbersome to operate. For example, a registration process is required before processing can begin. This not only increases the number of steps but also may lead to re-registration issues if the camera is accidentally touched, thus limiting the applicability of the process, for example, it is not suitable for autistic patients.

[0005] In addition, the medical equipment needs to be kept stable throughout the entire process. Because medical equipment is heavy and the processing time is long, holding the equipment by hand increases the instability of the process. Summary of the Invention

[0006] To address at least one of the aforementioned problems and defects in the prior art, embodiments of the present invention provide a positioning and navigation method, a positioning and navigation device, an electronic device, and a readable storage medium to solve the problem of cumbersome operation of existing navigation systems.

[0007] One object of the present invention is to provide a positioning and navigation method.

[0008] Another object of the present invention is to provide an electronic device.

[0009] Another object of the present invention is to provide a readable storage medium.

[0010] Another object of the present invention is to provide a positioning and navigation device.

[0011] According to one aspect of the present invention, a positioning and navigation method is provided, comprising:

[0012] First transformation data is determined for the conversion between depth data used for the camera and spatial data of the visualization robotic arm model base;

[0013] Determine the second conversion data used for mutual conversion between spatial data of the end effector of the visualized medical device model and the visualized robotic arm model;

[0014] A third transformation data is determined to convert between depth data used for the camera and spatial data of the visualized target model;

[0015] Determine the fourth transformation data used for the mutual conversion between the visualization robotic arm model base and the visualization robotic arm model end effector;

[0016] The relative positional relationship between the visualized target model and the visualized medical device model is obtained based on the first, second, third, and fourth transformation data, and navigation is performed according to the relative positional relationship.

[0017] In some embodiments, the first conversion data for determining the conversion between depth data for the camera and spatial data of the visualization robotic arm model base includes:

[0018] A marker plate is set on the end of the robotic arm so that the robotic arm moves according to preset information;

[0019] The first conversion data is determined based on the robotic arm motion information collected by the marker plate and the camera.

[0020] In some embodiments, the marking patterns on the marking board include at least one of a checkerboard pattern, a dot matrix pattern, a ChAruco pattern, and a deep learning training graph.

[0021] The preset information includes a preset posture.

[0022] In some embodiments, the second conversion data for converting between spatial data of the end effector of a visualized medical device model and a visualized robotic arm model includes:

[0023] The medical device is mounted on the end effector of the robotic arm, and the positional relationship between the medical device and the end effector of the robotic arm is determined.

[0024] The second conversion data is determined based on the positional relationship between the medical device and the end effector of the robotic arm.

[0025] In some embodiments, the third transformation data used to determine the conversion between depth data for the camera and spatial data of the visualized object model includes:

[0026] The first data for the target part of the visualized target object model is obtained by using RGB data and depth data acquired by the camera.

[0027] After matching the first data with the second data of the target object, the third transformed data is determined.

[0028] In some embodiments, obtaining first data for the target portion of a visualized object model using RGB data and depth data acquired by a camera includes:

[0029] Extract the third data of the target portion from the RGB data;

[0030] The depth data is converted to the plane containing the intrinsic parameters of the RGB data to obtain the fourth data.

[0031] After removing the non-target portion of the fifth data from the fourth data based on the third data of the target portion, the first data of the target portion is obtained.

[0032] After matching the first data with the second data of the target object, the second converted data is determined to include:

[0033] The first data of the target portion is coarsely matched with the second data of the target object;

[0034] After performing precise matching on the coarsely matched data, the third transformed data is determined.

[0035] In some embodiments, the depth data includes environmental point cloud data;

[0036] The first data of the target portion includes the point cloud data of the target portion;

[0037] The second data includes point cloud data of the target object.

[0038] In some embodiments, navigation based on the relative positional relationship includes:

[0039] Provide a desired visual perspective based on the relative positional relationship, and perform navigation based on the desired visual perspective.

[0040] The desired visualization perspective includes at least one of the following: a visualization perspective of the medical device moving relative to the target object, a visualization perspective of the target object moving relative to the medical device, and a visualization perspective in which both the medical device and the target object are moving relative to the camera.

[0041] In some embodiments, the visualization perspective for showing the motion of a medical device relative to a target object includes:

[0042] Based on the second transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model;

[0043] Based on the fourth transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed to the coordinate system of the visualized robotic arm model base.

[0044] Based on the first transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is transformed to the depth coordinate system of the camera;

[0045] Based on the third transformation data, the data of the visualized medical device model in the depth coordinate system is transformed to the coordinate system of the visualized target object model to show the visualization perspective of the medical device's motion relative to the target object.

[0046] In some embodiments, the first transformation data includes a first transformation matrix;

[0047] The second transformation data includes the second transformation matrix;

[0048] The third transformation data includes the third transformation matrix;

[0049] The fourth transformation data includes the fourth transformation matrix.

[0050] Visual perspectives showcasing the movement of medical devices relative to a target object include:

[0051] By multiplying the data of the visualized medical device model in the coordinate system of the visualized medical device model by the second transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed into the coordinate system of the end effector of the visualized robotic arm model.

[0052] By multiplying the data of the visualized medical device model in the coordinate system of the visualized robotic arm model's end effector into the coordinate system of the visualized robotic arm model's base by the fourth transformation matrix;

[0053] By multiplying the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base by the first transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is transformed into the depth coordinate system of the camera.

[0054] By multiplying the data of the visualized medical device model in the depth coordinate system by the third transformation matrix, the data of the visualized medical device model in the depth coordinate system is transformed into the coordinate system of the visualized target object model, so as to show the visual perspective of the movement of the medical device relative to the target object.

[0055] In some embodiments, the visualization perspective for showing the motion of the target object relative to the medical device includes:

[0056] Based on the third transformation data, the data of the visualized target model in the coordinate system is transformed to the depth coordinate system of the camera;

[0057] Based on the first transformation data, the data of the visualized target model in the depth coordinate system is transformed to the coordinate system of the visualized robotic arm model base;

[0058] Based on the fourth transformation data, the data of the visualized target model in the coordinate system of the visualized robotic arm model base is transformed to the coordinate system of the visualized robotic arm model end;

[0059] Based on the second transformation data, the data of the visualized target object model in the coordinate system of the visualized robotic arm end effector model is transformed to the coordinate system of the visualized medical device model, so as to show the visualization perspective of the target object's motion relative to the medical device.

[0060] In some embodiments, the first transformation data includes a first transformation matrix;

[0061] The second transformation data includes the second transformation matrix;

[0062] The third transformation data includes the third transformation matrix;

[0063] The fourth transformation data includes the fourth transformation matrix.

[0064] Visual perspectives showing the motion of the target object relative to the medical device include:

[0065] By multiplying the data of the visualized target model in the coordinate system of the visualized target model by the inverse of the third transformation matrix, the data of the visualized target model in the coordinate system of the visualized target model is transformed into the depth coordinate system of the camera.

[0066] By multiplying the data of the visualized target model in the depth coordinate system by the inverse of the first transformation matrix, the data of the visualized target model in the depth coordinate system is transformed into the coordinate system of the visualized robotic arm model base.

[0067] By multiplying the data of the visualized target model in the coordinate system of the visualized robotic arm model base by the inverse of the fourth transformation matrix, the data of the visualized target model in the coordinate system of the visualized robotic arm model base is transformed into the coordinate system of the visualized robotic arm model end.

[0068] By multiplying the data of the visualized target model in the coordinate system of the visualized robotic arm end effector model by the inverse of the second transformation matrix, the data of the visualized target model in the coordinate system of the visualized medical device model is transformed into the coordinate system of the visualized medical device model, so as to show the visual perspective of the target object's motion relative to the medical device.

[0069] In some embodiments, the visualization perspective showing that both the medical device and the target are moving relative to the camera includes:

[0070] Based on the second transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model;

[0071] Based on the fourth transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed to the coordinate system of the visualized robotic arm model base.

[0072] Based on the first transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is transformed to the depth coordinate system of the camera;

[0073] Based on the third transformation data, the data of the visualized target model in the coordinate system of the visualized target model is transformed to the depth coordinate system of the camera to show the visualization perspective of the medical device and the target object moving relative to the camera.

[0074] In some embodiments, the first transformation data includes a first transformation matrix;

[0075] The second transformation data includes the second transformation matrix;

[0076] The third transformation data includes the third transformation matrix;

[0077] The fourth transformation data includes the fourth transformation matrix.

[0078] Visual perspectives showing the movement of medical equipment and objects relative to the camera include:

[0079] By multiplying the data of the visualized medical device model in the coordinate system of the visualized medical device model by the second transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed into the coordinate system of the end effector of the visualized robotic arm model.

[0080] By multiplying the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector by the inverse matrix based on the fourth transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed into the coordinate system of the visualized robotic arm model base.

[0081] By multiplying the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base by the first transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is transformed into the depth coordinate system of the camera.

[0082] By multiplying the data of the visualized target model in the coordinate system of the visualized target model by the inverse of the third transformation matrix, the data of the visualized target model in the coordinate system of the visualized target model is transformed into the depth coordinate system of the camera, so as to show the visualization perspective of the medical device and the target object moving relative to the camera.

[0083] In some embodiments, navigation based on the relative positional relationship includes:

[0084] The robotic arm is controlled to place the medical device at the target point based on the relative positional relationship.

[0085] In some embodiments, controlling the robotic arm to place the medical device at the target point according to the relative positional relationship includes:

[0086] Based on the third transformation data, the data of the target points on the target object in the coordinate system of the visualized target object model are transformed to the depth coordinate system of the camera;

[0087] Based on the first transformation data, the target point data in the depth coordinate system is transformed to the coordinate system of the visual robotic arm model base;

[0088] Based on the second transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model;

[0089] Based on the fourth transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed to the coordinate system of the visualized robotic arm model base.

[0090] Based on the data of the target point in the coordinate system of the visualized robotic arm model base and the data of the visualized medical device model, the robotic arm is controlled to place the medical device on the target point.

[0091] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0092] A memory and a processor, wherein the memory stores a program, and the processor, when executing the program in the memory, implements the positioning and navigation method according to any of the foregoing embodiments.

[0093] According to another aspect of the present invention, a readable storage medium is provided, wherein a computer-readable program or instructions are stored therein, which, when executed by a processor, implement the positioning and navigation method according to any of the foregoing embodiments.

[0094] According to another aspect of the present invention, a positioning and navigation device is provided, the positioning and navigation device comprising:

[0095] The first determining module is configured to determine first conversion data for mutual conversion between depth data of the camera and spatial data of the visualization robotic arm model base;

[0096] The second determining module is configured to determine second conversion data for mutual conversion between spatial data of the end effector of the visualized medical device model and the visualized robotic arm model.

[0097] The third determining module is configured to determine third conversion data for the mutual conversion between the depth data of the camera and the spatial data of the visualized target model;

[0098] The fourth determining module is configured to determine fourth conversion data for mutual conversion between the visual robotic arm model base and the visual robotic arm model end effector;

[0099] A control module is communicatively connected to a first determining module, a second determining module, a third determining module, and a fourth determining module. The control module is configured to obtain the relative positional relationship between the visualized target model and the visualized medical device model based on the first conversion data, the second conversion data, the third conversion data, and the fourth conversion data, and to perform navigation based on the relative positional relationship.

[0100] The positioning and navigation method, positioning and navigation device, electronic device, and readable storage medium according to the present invention have at least one of the following advantages:

[0101] (1) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention can eliminate the registration process before processing and simplify the navigation operation process;

[0102] (2) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention allow the target object (e.g., subject) to not wear a rigid body model, thus expanding the applicability of the processing;

[0103] (3) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention eliminate the necessity of hand-held medical devices by holding medical devices with a robotic arm, which increases system stability and reduces labor costs;

[0104] (4) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention can automatically follow and control, thereby providing a more accurate and convenient processing method;

[0105] (5) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention can provide navigation from multiple dimensions, increasing the flexibility of the navigation process. Attached Figure Description

[0106] These and / or other aspects and advantages of the present invention will become apparent and readily understood from the following description of preferred embodiments taken in conjunction with the accompanying drawings, in which:

[0107] Figure 1 A positioning and navigation method according to an embodiment of the present invention is shown;

[0108] Figure 2 A flowchart illustrating the determination of first conversion data according to an embodiment of the present invention is shown;

[0109] Figure 3 A flowchart illustrating the determination of second transformed data according to an embodiment of the present invention is shown;

[0110] Figure 4 A flowchart illustrating the determination of third-transformation data according to an embodiment of the present invention is shown;

[0111] Figure 5 A positioning and navigation device according to an embodiment of the present invention is shown. Detailed Implementation

[0112] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings. In this specification, the same or similar reference numerals indicate the same or similar components. The following description of the embodiments of the present invention with reference to the accompanying drawings is intended to explain the overall inventive concept of the present invention and should not be construed as a limitation thereof.

[0113] In an embodiment of the present invention, a positioning and navigation method is provided, which can determine the relative positional relationship between a medical device and a target object (or, a visualized medical device model and a visualized target object model) through a corresponding transformation relationship, and perform navigation based on the relative positional relationship.

[0114] Medical devices include devices that manipulate a target object. For example, medical devices can include magnetic stimulation devices, electrical stimulation devices, light stimulation devices, and mechanical wave stimulation devices.

[0115] The target object includes the object to be treated, such as a body part of the subject. For example, a body part of the subject could be the head, feet, etc. The target point includes the point on the target object that is to be treated, such as a target point.

[0116] like Figure 1 As shown, the positioning and navigation method includes:

[0117] First transformation data is determined for the conversion between depth data used for the camera and spatial data of the visualization robotic arm model base;

[0118] Determine the second transformation data for mutual conversion between spatial data used to visualize medical device models and spatial data at the end effector of visualized robotic arm models;

[0119] A third conversion data is determined for the mutual conversion between the depth data used by the camera and the spatial data of the visualized target model;

[0120] Determine the fourth transformation data used for the mutual conversion between the visualization robotic arm model base and the visualization robotic arm model end effector;

[0121] The relative positional relationship between the visualized target model and the visualized medical device model is obtained based on the first, second, third, and fourth conversion data, and navigation is performed according to the relative positional relationship.

[0122] The embodiments of the present invention obtain the relative positional relationship between the visualized target model and the visualized medical device model based on the first conversion data, the second conversion data, the third conversion data and the fourth conversion data, eliminating the steps of registering medical devices and target objects, simplifying the navigation operation process, and solving the problem of re-registration caused by misoperation.

[0123] The embodiments of the present invention eliminate the need for handheld medical devices by using a robotic arm to hold the medical devices, thereby increasing system stability and reducing labor costs.

[0124] In embodiments of the present invention, the camera refers to a camera capable of providing both depth data and color data (RGB data), and the conversion relationship between the depth data and color data acquired by the camera is known. For example, the camera may be an RGBD camera. In one example, the camera may be a single camera integrating an RGB camera and a depth camera. Alternatively, the camera may consist of a separate RGB camera and a separate depth camera.

[0125] Specifically, the first transformation data can determine the conversion relationship between the camera's depth data and the spatial data of the visualized robotic arm model base. For example... Figure 2 As shown, determining the first conversion data includes: setting a marker plate on the end of the robotic arm so that the robotic arm moves according to preset information; and determining the first conversion data based on the marker plate and the robotic arm motion information collected by the camera.

[0126] The marking plate is fixed to the side wall of the robotic arm's end effector by pasting, imprinting, or binding. The marking plate has a marking pattern. The marking pattern includes at least one of a checkerboard pattern, a dot matrix pattern, a ChAruco pattern, and a deep learning training image. Embodiments of this disclosure do not limit the specific location of the marking plate at the robotic arm's end effector, as long as the marking pattern on the marking plate can be captured by a camera.

[0127] The camera is placed in a preset position so that it can capture the posture of the robotic arm. Furthermore, the robotic arm moves according to preset information. For example, the preset information includes preset postures, which the robotic arm moves sequentially according to.

[0128] Using the marking patterns on the marking board, the camera can capture information about each movement of the robotic arm. Then, through algorithms such as hand-eye calibration, it can obtain transformation data between the robotic arm base coordinate system and the depth coordinate system. That is, this transformation data can convert the spatial data of the robotic arm base in the depth coordinate system to the spatial data of the robotic arm base in the visual robotic arm model base coordinate system. Since the relationship between the robotic arm base and the visual robotic arm model base is rigid, the transformation data between the robotic arm base coordinate system and the depth coordinate system, as well as the relationship between the robotic arm base and the visual robotic arm model base, can correspondingly obtain the first transformation data between the visual robotic arm model base coordinate system and the depth coordinate system. The first transformation data can be a first transformation matrix. The second transformation data can determine the mutual transformation relationship between the spatial data of the visual medical device model and the end effector of the visual robotic arm model. For example... Figure 3 As shown, determining the second conversion data includes: setting the medical device on the end effector of the robotic arm and determining the positional relationship between the medical device and the end effector of the robotic arm; and determining the second conversion data based on the positional relationship between the medical device and the end effector of the robotic arm.

[0129] The medical device is mounted on the end effector of a robotic arm, for example, allowing the end effector to grip the medical device. The conversion data between the medical device and the end effector can then be determined.

[0130] Since the transformation relationship between the medical device and the visualized medical device model is rigid, and the transformation relationship between the robotic arm end effector and the visualized robotic arm model end effector is also rigid, a second transformation data between the spatial data of the visualized medical device model and the spatial data of the visualized robotic arm model end effector can be obtained through the transformation data between the medical device and the robotic arm end effector. This second transformation data can be a second transformation matrix.

[0131] The third transformation data can determine the conversion relationship between the camera's depth data and the spatial data of the visualized target model. For example... Figure 4 As shown, determining the third transformation data includes: obtaining first data of the target portion of the visualized target model through RGB data and depth data obtained from the camera; and determining the third transformation data by matching the first data with the second data of the target object. The third transformation data can be a third transformation matrix.

[0132] In acquiring the initial data, the target object needs to be brought into the camera's field of view. The RGB data acquired by the camera represents the RGB data of the real-world environment, while the depth data acquired is environmental point cloud data, which includes point cloud data of the target object and surrounding objects. The target portion is the region of interest in navigation. For example, if the target object is the subject's head, the target portion could be the face.

[0133] Furthermore, the first data for obtaining the target portion of the visualized target model through the RGB data and depth data acquired by the camera includes:

[0134] The third data of the target portion is extracted from the RGB data. Existing methods can be used to extract the target portion and form the third data. For example, if the target portion is a face, a face recognition algorithm can be used for face extraction.

[0135] The depth data is converted to the plane containing the intrinsic parameters of the RGB data to obtain the fourth data. The collected environmental point cloud data can be converted to the plane containing the intrinsic parameters of the RGB data to obtain the fourth data. For example, the fourth data can be obtained using the D2C algorithm.

[0136] After removing the non-target portion of the fifth data from the fourth data based on the third data of the target portion, the first data of the target portion is obtained. Depth data includes both target and non-target portions, so after converting the depth data to the plane containing the intrinsic parameters of the RGB data, the non-target portion needs to be removed. The remaining portion then constitutes the first data of the target portion.

[0137] Further, after matching the first data with the second data of the target object, determining the second converted data includes: performing a coarse match between the first data of the target portion and the second data of the target object; and performing a precise match between the coarsely matched data to determine the third converted data.

[0138] The second set of data for the target object includes its point cloud data. For example, if the target object is a head, head MRI (Magnetic Resonance Imaging) data can be converted into MRI point cloud data, which then constitutes the target object's point cloud data. Point cloud data refers to the set of points obtained after acquiring the spatial coordinates of each sampling point on the object's surface. The RANSAC (Random Sample Consensus) algorithm can be used to perform a coarse matching between the point cloud data of the target part and the point cloud data of the target object. Coarse matching refers to a relatively rough registration when the transformation between the two point clouds is completely unknown, and its main purpose is to provide better initial transformation values ​​for fine registration.

[0139] The ICP (Iterative Closest Point) algorithm can be used to perform precise matching on data that has undergone coarse matching. Precise matching refers to further optimization to obtain a more accurate transformation given an initial transformation.

[0140] The embodiments of the present invention obtain the relationship between the visual head model and the depth data of the camera through a point cloud matching algorithm, which reduces the steps of the target object (e.g., the subject) wearing a rigid body model and registering it, expands the applicability of the positioning and navigation method, and can reduce the difficulty of operation.

[0141] The first transformation data determines the transformation relationship between the camera's depth data and the spatial data of the visualized robotic arm model base. The second transformation data determines the transformation relationship between the spatial data of the visualized medical device model and the spatial data of the visualized robotic arm model's end effector. The third transformation data determines the transformation relationship between the camera's depth data and the spatial data of the visualized target object model. To obtain the transformation relationship between the visualized target object model and the visualized medical device model, a fourth transformation data is needed to determine the mutual transformation between the visualized robotic arm model base and the visualized robotic arm model's end effector. The fourth transformation data can be a fourth transformation matrix.

[0142] The conversion relationship between the robotic arm base and the robotic arm end effector can be obtained in real time. The relationship between the robotic arm base and the visual robotic arm model base is rigid; the relationship between the robotic arm end effector and the visual robotic arm model end effector is also rigid. Therefore, the fourth conversion data between the visual robotic arm model base and the visual robotic arm model end effector can be determined accordingly based on the conversion relationship between the robotic arm base and the robotic arm end effector.

[0143] After determining the existence of the first, second, third, and fourth transformation data, the relative positional relationship between the visualized medical device model and the visualized target object model can be determined, and navigation can be performed based on the relative positional relationship.

[0144] In one example, navigation based on the relative positional relationship includes: providing a desired visual perspective based on the relative positional relationship, and navigating based on the desired visual perspective. For example, multiple perspective navigation can be implemented to guide the operator to find the most desired action location (e.g., a target point) from their desired perspective.

[0145] The desired visualization perspective includes at least one of the following: a visualization perspective of the medical device moving relative to the target object, a visualization perspective of the target object moving relative to the medical device, and a visualization perspective in which both the medical device and the target object are moving relative to the camera.

[0146] In one example, the visualization perspectives showing the motion of a medical device relative to a target object include:

[0147] Based on the second transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model;

[0148] Based on the fourth transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed to the coordinate system of the visualized robotic arm model base.

[0149] Based on the first transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is transformed to the depth coordinate system of the camera;

[0150] Based on the third transformation data, the data of the visualized medical device model in the depth coordinate system is transformed to the coordinate system of the visualized target object model to show the visualization perspective of the medical device's motion relative to the target object.

[0151] For example, the first transformation data includes the first transformation matrix M. c2b The second transformation data includes the second transformation matrix M. c2g The third transformation data includes the third transformation matrix M. d2h The fourth transformation data includes the fourth transformation matrix M. g2b Visual perspectives showcasing the movement of medical devices relative to a target object include:

[0152] The data of the visualized medical device model in the coordinate system of the visualized medical device model is multiplied by the second transformation matrix M. c2g The data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model.

[0153] The data of the visualized medical device model in the coordinate system of the visualized robotic arm end effector is multiplied by the fourth transformation matrix M. g2b The data of the visualized medical device model in the coordinate system of the end effector of the visualized robotic arm model is transformed to the coordinate system of the base of the visualized robotic arm model.

[0154] The data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is multiplied by the first transformation matrix M. c2b The data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is converted to the depth coordinate system of the camera;

[0155] The data of the visualized medical device model in the depth coordinate system is multiplied by the third transformation matrix M. d2hThe data of the visualized medical device model in the depth coordinate system is converted to the coordinate system of the visualized target object model to show the visual perspective of the movement of the medical device relative to the target object.

[0156] In one example, the visualization perspectives showing the motion of the target object relative to the medical device include:

[0157] Based on the third transformation data, the data of the visualized target model in the coordinate system is transformed to the depth coordinate system of the camera;

[0158] Based on the first transformation data, the data of the visualized target model in the depth coordinate system is transformed to the coordinate system of the visualized robotic arm model base;

[0159] Based on the fourth transformation data, the data of the visualized target model in the coordinate system of the visualized robotic arm model base is transformed to the coordinate system of the visualized robotic arm model end;

[0160] Based on the second transformation data, the data of the visualized target object model in the coordinate system of the visualized robotic arm end effector model is transformed to the coordinate system of the visualized medical device model, so as to show the visualization perspective of the target object's motion relative to the medical device.

[0161] For example, the first transformation data includes the first transformation matrix M. c2b The second transformation data includes the second transformation matrix M. c2g The third transformation data includes the third transformation matrix M. d2h The fourth transformation data includes the fourth transformation matrix M. g2b Visual perspectives showing the movement of the target object relative to the medical device include:

[0162] The data of the visualized target model in the coordinate system of the visualized target model is multiplied by the inverse matrix M of the third transformation matrix. d2h -1 This transforms the data of the visualized target model in the coordinate system to the depth coordinate system of the camera;

[0163] The data of the visualized target model in the depth coordinate system is multiplied by the inverse matrix M of the first transformation matrix. c2b -1 The data of the visualized target model in the depth coordinate system is converted to the coordinate system of the visualized robotic arm model base;

[0164] The data of the visualized target model in the coordinate system of the visualized robotic arm model base is multiplied by the inverse matrix M of the fourth transformation matrix. g2b -1The data of the visualized target model in the coordinate system of the base of the visualized robotic arm model is transformed to the coordinate system of the end of the visualized robotic arm model.

[0165] The data of the visualized target model in the coordinate system of the visualized robotic arm end effector is multiplied by the inverse matrix M of the second transformation matrix. c2g -1 The data of the visualized target object model in the coordinate system of the visualized robotic arm end effector model is transformed into the coordinate system of the visualized medical device model to show the visualized perspective of the target object's motion relative to the medical device.

[0166] In one example, a visualization showing that both the medical device and the target are moving relative to the camera includes:

[0167] Based on the second transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model;

[0168] Based on the fourth transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed to the coordinate system of the visualized robotic arm model base.

[0169] Based on the first transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is transformed to the depth coordinate system of the camera;

[0170] Based on the third transformation data, the data of the visualized target model in the coordinate system of the visualized target model is transformed to the depth coordinate system of the camera to show the visualization perspective of the medical device and the target object moving relative to the camera.

[0171] For example, the first transformation data includes the first transformation matrix M. c2b The second transformation data includes the second transformation matrix M. c2g The third transformation data includes the third transformation matrix M. d2h The fourth transformation data includes the fourth transformation matrix M. g2b Visual perspectives showing the movement of both medical equipment and the target object relative to the camera include:

[0172] The data of the visualized medical device model in the coordinate system of the visualized medical device model is multiplied by the second transformation matrix M. c2g The data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model.

[0173] The data of the visualized medical device model in the coordinate system of the visualized robotic arm end effector is multiplied by the inverse matrix M based on the fourth transformation matrix. g2b -1The data of the visualized medical device model in the coordinate system of the end effector of the visualized robotic arm model is transformed to the coordinate system of the base of the visualized robotic arm model.

[0174] The data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is multiplied by the first transformation matrix M. c2b The data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is converted to the depth coordinate system of the camera;

[0175] The data of the visualized target model in the coordinate system of the visualized target model is multiplied by the inverse matrix M of the third transformation matrix. d2h -1 The data of the visualized target model in the coordinate system is transformed to the depth coordinate system of the camera to show the visualization perspective of the medical device and the target object moving relative to the camera.

[0176] The embodiments of the present invention can display multiple navigation perspectives through the above four conversion data, allowing operators to select the navigation perspective as needed during processing, thereby enabling more effective tracking of target points (e.g., target points). This approach avoids the inconvenience of a single perspective and increases ease of use. The embodiments of this disclosure are not limited to the above three navigation perspectives and may also provide other required navigation perspectives based on the above four conversion data.

[0177] In one example, navigation based on the relative positional relationship includes controlling a robotic arm to place a medical device at a target point based on the relative positional relationship. Thus, embodiments of the present invention enable automatic following control, thereby improving the precision and convenience of the processing method.

[0178] Specifically, controlling the robotic arm to place the medical device onto the target point based on the relative positional relationship includes:

[0179] Based on the third transformation data, the data of the target points on the target object in the coordinate system of the visualized target object model are transformed to the depth coordinate system of the camera;

[0180] Based on the first transformation data, the target point data in the depth coordinate system is transformed to the coordinate system of the visual robotic arm model base;

[0181] Based on the second transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model;

[0182] Based on the fourth transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed to the coordinate system of the visualized robotic arm model base.

[0183] Based on the data of the target point in the coordinate system of the visualized robotic arm model base and the data of the visualized medical device model, the robotic arm is controlled to place the medical device on the target point.

[0184] For example, the first transformation data includes the first transformation matrix M. c2b The second transformation data includes the second transformation matrix M. c2g The third transformation data includes the third transformation matrix M. d2h The fourth transformation data includes the fourth transformation matrix M. g2b Controlling the robotic arm to place the medical device at the target point based on the relative positional relationship includes:

[0185] The data of the target points on the target object in the coordinate system of the visualized target object model are multiplied by the inverse matrix M of the third transformation matrix. d2h -1 The target point data is converted to the camera's depth coordinate system;

[0186] The target point data in the camera's depth coordinate system is multiplied by the inverse matrix M of the first transformation matrix. c2b -1 The data of the target point is transformed into the coordinate system of the visual robotic arm model base;

[0187] The data of the visualized medical device model in the coordinate system of the visualized medical device model is multiplied by the second transformation matrix M. c2g The data from the visualized medical device model is converted to the coordinate system of the end effector of the visualized robotic arm model;

[0188] The data of the visualized medical device model in the coordinate system of the visualized robotic arm end effector is multiplied by the fourth transformation matrix M. g2b The data of the visualized medical equipment model is converted to the coordinate system of the visualized robotic arm model base;

[0189] Based on the data of the target point in the coordinate system of the visualized robotic arm model base and the data of the visualized medical device model, the robotic arm is controlled to place the medical device on the target point.

[0190] It should be noted that the embodiments of the present invention are not limited to the navigation process described above. For example, one step involves multiplying the data of the visualized medical device model in the coordinate system of the visualized robotic arm model's end effector by the fourth transformation matrix M. g2b The data of the visualized medical device model is transformed into the coordinate system of the visualized robotic arm model base; this step can be adjusted accordingly by changing the fourth transformation matrix.

[0191] The embodiments of the present invention are based on a 1:1 relationship between the model and the real object; for example, the visualized medical device model and the medical device are in a 1:1 relationship. However, those skilled in the art will understand that the embodiments of the present invention are not limited to a specific ratio between the model and the real object, and other ratios can be set, such as 1:2, 1:4, etc. As long as other ratios are converted to a 1:1 relationship during navigation, the positioning and navigation method of the embodiments of the present invention can be obtained.

[0192] In embodiments of the present invention, a positioning and navigation device is also provided. For example... Figure 5 As shown, the positioning and navigation device 100 includes a first determining module 10, a second determining module 20, a third determining module 30, a fourth determining module 40, and a control module 50.

[0193] The first determining module 10 is configured to determine first conversion data for the mutual conversion between depth data for the camera and spatial data of the visualization robotic arm model base. The first determining module 10 may determine the first conversion data using the method described in any of the above embodiments.

[0194] The second determining module 20 is configured to determine second conversion data for converting between spatial data of the visualized medical device model and the end effector of the visualized robotic arm model. The second determining module 20 may determine the second conversion data using the method described in any of the above embodiments.

[0195] The third determining module 30 is configured to determine third conversion data for the mutual conversion between the depth data of the camera and the spatial data of the visualized target model. The third determining module 30 may determine the third conversion data using the method described in any of the above embodiments.

[0196] The fourth determining module 40 is configured to determine fourth conversion data for the mutual conversion between the visualization robotic arm model base and the visualization robotic arm model end effector. The fourth determining module 40 may determine the fourth conversion data using the method described in any of the above embodiments.

[0197] The control module 50 is communicatively connected to the first determining module 10, the second determining module 20, the third determining module 30, and the fourth determining module 40, respectively. The control module 50 is configured to obtain the relative positional relationship between the visualized target model and the visualized medical device model based on the first, second, third, and fourth conversion data, and to perform navigation according to the relative positional relationship. The control module 50 can determine the relative positional relationship using the method described in any of the above embodiments to control the navigation process.

[0198] In embodiments of the present invention, a readable storage medium is provided. The readable storage medium stores a program or instructions that, when executed by a processor, implement the positioning and navigation method described in any of the above embodiments.

[0199] In embodiments of the present invention, "readable storage medium" refers to any medium that participates in providing a program or instructions to a processor for execution. The medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage devices. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including conductors containing buses. Transmission media can also take the form of acoustic or optical waves, such as acoustic or optical waves generated during radio frequency (RF) and infrared (IR) data communications. Common forms of readable storage media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, DVDs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves as described below, or any other medium from which a computer can read.

[0200] In an embodiment of the present invention, an electronic device is also provided. The electronic device (not shown) includes a processor (not shown) and a memory (not shown). A program is stored in the memory, which, when executed by the processor, can implement the positioning and navigation method of any of the above examples.

[0201] In one example, the processor can be a microprocessor, such as a general-purpose processor like a graphics processing unit (GPU), a central processing unit (CPU), or a digital signal processor (DSP). In another example, the processor can also be a microprocessor core implemented through hardware circuitry, such as a microprocessor core implemented in hardware logic components using reconfigurable logic, including field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), and systems-on-a-chip (SoCs).

[0202] In one example, the processor can also be a virtual processor, which can be a virtual processor with Intel x86 processor features or a virtual processor with PowerPC processor features. Preferably, the processor is a graphics processor. In one example, the processor can be a single-core processor or a multi-core processor.

[0203] In one example, the memory includes volatile memory (i.e., random access memory) and non-volatile memory. Volatile memory includes main memory, cache, etc., while non-volatile memory includes auxiliary memory, etc. In one example, the memory can be configured as remote memory, which can be connected to the processor via a network (wired or wireless network). The network includes, but is not limited to, wide area networks (WANs), local area networks (LANs), metropolitan area networks (MANs), personal area networks (PANs), the Internet, satellite communication networks, and any combination thereof.

[0204] In one example, the processor creates a corresponding task thread based on a program retrieved from memory and executes the thread. In another example, the processor retrieves a program from secondary storage based on a read instruction from memory to create a corresponding task thread and executes the thread. The above program is used to implement a positioning and navigation method.

[0205] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that it can also be implemented in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the method steps described in conjunction with any of the examples herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Implementation in hardware or software depends primarily on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods for each specific application to implement the described functionality, but such implementation should not be considered beyond the scope of this application.

[0206] When the method steps are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Therefore, the technical solution of this invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various examples of this invention.

[0207] The positioning and navigation method, positioning and navigation device, electronic device, and readable storage medium according to the present invention have at least one of the following advantages:

[0208] (1) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention can eliminate the registration process before processing and simplify the navigation operation process;

[0209] (2) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention allow the target object (e.g., subject) to not wear a rigid body model, thus expanding the applicability of the processing;

[0210] (3) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention eliminate the necessity of hand-held medical devices by using a robotic arm to hold medical devices, thereby increasing system stability and reducing labor costs;

[0211] (4) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention can automatically follow and control, thereby providing a more accurate and convenient processing method;

[0212] (5) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention can provide navigation from multiple dimensions, increasing the flexibility of the navigation process.

[0213] While some embodiments of the present general inventive concept have been shown and described, those skilled in the art will understand that changes may be made to these embodiments without departing from the principles and spirit of the present general inventive concept, the scope of which is defined by the claims and their equivalents.

Claims

1. A positioning and navigation method, comprising: First transformation data is determined for the conversion between depth data used for the camera and spatial data of the visualization robotic arm model base; Determine the second conversion data used for mutual conversion between spatial data of the end effector of the visualized medical device model and the visualized robotic arm model; A third conversion data is determined for the mutual conversion between the depth data used by the camera and the spatial data of the visualized target model; Determine the fourth transformation data used for the mutual conversion between the visualization robotic arm model base and the visualization robotic arm model end effector; Based on the first, second, third, and fourth transformation data, the relative positional relationship between the visualized target model and the visualized medical device model is obtained. The desired visualization perspective is provided according to the relative positional relationship, and navigation is performed based on the desired visualization perspective.

2. The positioning and navigation method according to claim 1, wherein, The first transformation data used to determine the conversion between depth data for the camera and spatial data for the visualization of the robotic arm model base includes: A marker plate is set on the end of the robotic arm so that the robotic arm moves according to preset information; The first conversion data is determined based on the robotic arm motion information collected by the marker plate and the camera.

3. The positioning and navigation method according to claim 2, wherein, The marking patterns on the marking board include at least one of the following: checkerboard pattern, dot matrix pattern, chAruco pattern, and deep learning training graph. The preset information includes a preset posture.

4. The positioning and navigation method according to claim 3, wherein, The second transformation data, used for converting between spatial data of the visualized medical device model and the visualized robotic arm model's end effector, includes: The medical device is mounted on the end effector of the robotic arm, and the positional relationship between the medical device and the end effector of the robotic arm is determined. The second conversion data is determined based on the positional relationship between the medical device and the end effector of the robotic arm.

5. The positioning and navigation method according to claim 4, wherein, The third transformation data used to determine the conversion between depth data for the camera and spatial data for visualizing the target model includes: The first data for the target part of the visualized target object model is obtained by using RGB data and depth data acquired by the camera. After matching the first data with the second data of the target object, the third transformed data is determined.

6. The positioning and navigation method according to claim 5, wherein, The first data obtained from the RGB and depth data acquired by the camera to visualize the target part of the target object model includes: Extract the third data of the target portion from the RGB data; The depth data is converted to the plane containing the intrinsic parameters of the RGB data to obtain the fourth data. After removing the non-target portion of the fifth data from the fourth data based on the third data of the target portion, the first data of the target portion is obtained. After matching the first data with the second data of the target object, the second converted data is determined to include: The first data of the target portion is coarsely matched with the second data of the target object; After performing precise matching on the coarsely matched data, the third transformed data is determined.

7. The positioning and navigation method according to claim 6, wherein, The depth data includes environmental point cloud data; The first data of the target portion includes the point cloud data of the target portion; The second data includes point cloud data of the target object.

8. The positioning and navigation method according to any one of claims 1-7, wherein, The desired visualization perspective includes at least one of the following: a visualization perspective of the medical device moving relative to the target object, a visualization perspective of the target object moving relative to the medical device, and a visualization perspective in which both the medical device and the target object are moving relative to the camera.

9. The positioning and navigation method according to claim 8, wherein, Visual perspectives showcasing the movement of medical devices relative to a target object include: Based on the second transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model; Based on the fourth transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed to the coordinate system of the visualized robotic arm model base. Based on the first transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is transformed to the depth coordinate system of the camera; Based on the third transformation data, the data of the visualized medical device model in the depth coordinate system is transformed to the coordinate system of the visualized target object model to show the visualization perspective of the medical device's motion relative to the target object.

10. The positioning and navigation method according to claim 9, wherein, The first transformation data includes the first transformation matrix; The second transformation data includes the second transformation matrix; The third transformation data includes the third transformation matrix; The fourth transformation data includes the fourth transformation matrix. Visual perspectives showcasing the movement of medical devices relative to a target object include: By multiplying the data of the visualized medical device model in the coordinate system of the visualized medical device model by the second transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed into the coordinate system of the end effector of the visualized robotic arm model. By multiplying the data of the visualized medical device model in the coordinate system of the visualized robotic arm model's end effector into the coordinate system of the visualized robotic arm model's base by the fourth transformation matrix; By multiplying the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base by the first transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is transformed into the depth coordinate system of the camera. By multiplying the data of the visualized medical device model in the depth coordinate system by the third transformation matrix, the data of the visualized medical device model in the depth coordinate system is transformed into the coordinate system of the visualized target object model, so as to show the visual perspective of the movement of the medical device relative to the target object.

11. The positioning and navigation method according to claim 8, wherein, Visual perspectives showing the motion of the target object relative to the medical device include: Based on the third transformation data, the data of the visualized target model in the coordinate system is transformed to the depth coordinate system of the camera; Based on the first transformation data, the data of the visualized target model in the depth coordinate system is transformed to the coordinate system of the visualized robotic arm model base; Based on the fourth transformation data, the data of the visualized target model in the coordinate system of the visualized robotic arm model base is transformed to the coordinate system of the visualized robotic arm model end; Based on the second transformation data, the data of the visualized target object model in the coordinate system of the visualized robotic arm end effector model is transformed to the coordinate system of the visualized medical device model, so as to show the visualization perspective of the target object's motion relative to the medical device.

12. The positioning and navigation method according to claim 11, wherein, The first transformation data includes the first transformation matrix; The second transformation data includes the second transformation matrix; The third transformation data includes the third transformation matrix; The fourth transformation data includes the fourth transformation matrix. Visual perspectives showing the motion of the target object relative to the medical device include: By multiplying the data of the visualized target model in the coordinate system of the visualized target model by the inverse of the third transformation matrix, the data of the visualized target model in the coordinate system of the visualized target model is transformed into the depth coordinate system of the camera. By multiplying the data of the visualized target model in the depth coordinate system by the inverse of the first transformation matrix, the data of the visualized target model in the depth coordinate system is transformed into the coordinate system of the visualized robotic arm model base. By multiplying the data of the visualized target model in the coordinate system of the visualized robotic arm model base by the inverse of the fourth transformation matrix, the data of the visualized target model in the coordinate system of the visualized robotic arm model base is transformed into the coordinate system of the visualized robotic arm model end. By multiplying the data of the visualized target model in the coordinate system of the visualized robotic arm end effector model by the inverse of the second transformation matrix, the data of the visualized target model in the coordinate system of the visualized medical device model is transformed into the coordinate system of the visualized medical device model, so as to show the visual perspective of the target object's motion relative to the medical device.

13. The positioning and navigation method according to claim 8, wherein, Visual perspectives showing the movement of medical equipment and objects relative to the camera include: Based on the second transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model; Based on the fourth transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed to the coordinate system of the visualized robotic arm model base. Based on the first transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is transformed to the depth coordinate system of the camera; Based on the third transformation data, the data of the visualized target model in the coordinate system of the visualized target model is transformed to the depth coordinate system of the camera to show the visualization perspective of the medical device and the target object moving relative to the camera.

14. The positioning and navigation method according to claim 13, wherein, The first transformation data includes the first transformation matrix; The second transformation data includes the second transformation matrix; The third transformation data includes the third transformation matrix; The fourth transformation data includes the fourth transformation matrix. Visual perspectives showing the movement of medical equipment and objects relative to the camera include: By multiplying the data of the visualized medical device model in the coordinate system of the visualized medical device model by the second transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed into the coordinate system of the end effector of the visualized robotic arm model. By multiplying the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector by the inverse matrix based on the fourth transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed into the coordinate system of the visualized robotic arm model base. By multiplying the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base by the first transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model base is transformed into the depth coordinate system of the camera. By multiplying the data of the visualized target model in the coordinate system of the visualized target model by the inverse of the third transformation matrix, the data of the visualized target model in the coordinate system of the visualized target model is transformed into the depth coordinate system of the camera, so as to show the visualization perspective of the medical device and the target object moving relative to the camera.

15. The positioning and navigation method according to any one of claims 1-7, wherein, Navigation based on the relative positional relationship includes: The robotic arm is controlled to place the medical device at the target point based on the relative positional relationship.

16. The positioning and navigation method according to claim 15, wherein, Controlling the robotic arm to place the medical device at the target point based on the relative positional relationship includes: Based on the third transformation data, the data of the target points on the target object in the coordinate system of the visualized target object model are transformed to the depth coordinate system of the camera; Based on the first transformation data, the target point data in the depth coordinate system is transformed to the coordinate system of the visual robotic arm model base; Based on the second transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the coordinate system of the end effector of the visualized robotic arm model; Based on the fourth transformation data, the data of the visualized medical device model in the coordinate system of the visualized robotic arm model end effector is transformed to the coordinate system of the visualized robotic arm model base. Based on the data of the target point in the coordinate system of the visualized robotic arm model base and the data of the visualized medical device model, the robotic arm is controlled to place the medical device on the target point.

17. An electronic device, characterized in that, The electronic device includes: A memory and a processor, wherein the memory stores a program, and the processor, when executing the program in the memory, implements the positioning and navigation method according to any one of claims 1-16.

18. A readable storage medium, characterized in that, The readable storage medium stores a computer-readable program or instructions, which, when executed by a processor, implement the positioning and navigation method according to any one of claims 1-16.

19. A positioning and navigation device, characterized in that, The positioning and navigation device includes: The first determining module is configured to determine first conversion data for mutual conversion between depth data of the camera and spatial data of the visualization robotic arm model base; The second determining module is configured to determine second conversion data for mutual conversion between spatial data of the end effector of the visualized medical device model and the visualized robotic arm model. The third determining module is configured to determine third conversion data for the mutual conversion between the depth data of the camera and the spatial data of the visualized target model; The fourth determining module is configured to determine fourth conversion data for mutual conversion between the visual robotic arm model base and the visual robotic arm model end effector; A control module is communicatively connected to a first determining module, a second determining module, a third determining module, and a fourth determining module. The control module is configured to obtain the relative positional relationship between the visualized target model and the visualized medical device model based on the first conversion data, the second conversion data, the third conversion data, and the fourth conversion data, and to perform navigation based on the relative positional relationship.

Images