Positioning and navigation methods, positioning and navigation devices, electronic devices and readable storage media

Abstract

This invention discloses a positioning and navigation method, a positioning and navigation device, an electronic device, and a readable storage medium, belonging to the field of medical devices. The positioning and navigation method includes: determining first conversion data for mutual conversion between RGB data used by a camera and spatial data of a visualized medical device model; determining second conversion data for mutual conversion between depth data used by a camera and spatial data of a visualized target object model; determining third conversion data for mutual conversion between RGB data used by the camera and depth data; determining the relative positional relationship between the visualized medical device model and the visualized target object model based on the first, second, and third conversion data, and displaying the desired visualization perspective. Embodiments of this invention determine the relative positional relationship between the visualized medical device model and the visualized target object model through the aforementioned conversion data, at least to some extent eliminating the necessity of registration during the positioning and navigation process.

Description

Technical Field

[0001] This invention relates to the field of medical devices, and in particular to a positioning and navigation method, a positioning and navigation device, an electronic device, and a readable storage medium. Background Technology

[0002] With the rapid development of medical imaging and medical image processing technologies, image-guided operating systems have emerged. Utilizing three-dimensional reconstruction models of medical images, operators can intuitively and accurately analyze the subject's structure and surrounding tissues, thereby facilitating the corresponding procedures.

[0003] In the aforementioned process, optical-assisted navigation is an effective medical navigation system. It utilizes an optical tracking device in conjunction with, for example, a six-axis robot to form a robotic operating system. The system uses the robot to hold the medical device, the optical tracking device is fixed by a support, and markers are attached to the target object (e.g., the subject, specifically the subject's head) to allow the optical tracking device to locate, for example, the coordinates of the head.

[0004] Optical positioning and navigation systems enable visualization during transcranial magnetic stimulation (TMS) treatment, improving the accuracy of medical device positioning to some extent. However, current navigation methods are relatively cumbersome to operate. For example, a registration process is required before starting the procedure, including registering the medical device and the target (e.g., the subject). If the camera is accidentally touched during registration, it can lead to re-registration issues. This makes these positioning and navigation methods unsuitable for patients with conditions such as autism. Summary of the Invention

[0005] To address at least one of the aforementioned problems and defects in the prior art, embodiments of the present invention provide a positioning and navigation method, a positioning and navigation device, an electronic device, and a readable storage medium to solve the problem of needing to re-register due to misoperation, thereby at least to some extent eliminating the necessity of registration during the positioning and navigation process.

[0006] One object of the present invention is to provide a positioning and navigation method.

[0007] Another object of the present invention is to provide an electronic device.

[0008] Another object of the present invention is to provide a readable storage medium.

[0009] Another object of the present invention is to provide a positioning and navigation device.

[0010] According to one aspect of the present invention, a positioning and navigation method is provided, comprising:

[0011] First transformation data is determined for the conversion between RGB data used in the camera and spatial data of the visualized medical device model;

[0012] Determine the second transformation data for converting between depth data used for the camera and spatial data for visualizing the target object model;

[0013] A third conversion data is determined for the mutual conversion between RGB data and depth data of the camera;

[0014] Based on the first, second, and third transformation data, the relative positional relationship between the visualized medical device model and the visualized target object model is determined, and the desired visualization perspective is displayed.

[0015] In some embodiments, the first conversion data for determining the conversion between RGB data for a camera and spatial data for visualizing a medical device model includes:

[0016] A marker pattern is set on the medical device, and the positional relationship between the medical device and the marker pattern is determined;

[0017] The spatial data of the marker pattern in the camera's RGB coordinate system is determined using RGB data obtained from the camera.

[0018] Based on the positional relationship between the medical device and the marker pattern and the spatial data of the marker pattern in the RGB coordinate system, determine the spatial data of the visualized medical device model in the RGB coordinate system;

[0019] The first transformation data is determined based on the spatial data of the visualized medical device model in the RGB coordinate system.

[0020] In some embodiments, the marking pattern includes at least one of a checkerboard pattern, a dot matrix pattern, a chAruco pattern, and a deep learning training graph.

[0021] In some embodiments, the second transformation data used to determine the conversion between depth data for the camera and spatial data for visualizing the target model includes:

[0022] The first data for the target part of the visualized target object model is obtained by using RGB data and depth data acquired by the camera.

[0023] After matching the first data with the second data of the target object, the second converted data is determined.

[0024] In some embodiments, obtaining first data for the target portion of a visualized object model using RGB data and depth data acquired by a camera includes:

[0025] Extract the third data of the target portion from the RGB data;

[0026] The depth data is converted to the plane containing the intrinsic parameters of the RGB data to obtain the fourth data.

[0027] After removing the fifth data (which is not part of the target data) from the fourth data based on the third data of the target part, the first data of the target part is obtained.

[0028] In some embodiments, determining the second converted data after matching the first data with the second data of the target object includes:

[0029] The first data of the target portion is coarsely matched with the second data of the target object;

[0030] After performing precise matching on the coarsely matched data, the second transformed data is determined.

[0031] In some embodiments, the depth data includes environmental point cloud data;

[0032] The first data of the target portion includes the point cloud data of the target portion;

[0033] The second data includes point cloud data of the target object.

[0034] In some embodiments, the desired visualization perspective includes at least one of the following: a visualization perspective of the medical device moving relative to the target object, a visualization perspective of the target object moving relative to the medical device, and a visualization perspective of both the medical device and the target object moving relative to the camera.

[0035] In some embodiments, the visualization perspective for showing the motion of a medical device relative to a target object includes:

[0036] Based on the first transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the RGB coordinate system of the camera;

[0037] Based on the third transformation data, the data of the visualized medical device model in the RGB coordinate system is transformed to the depth coordinate system of the camera;

[0038] Based on the second transformation data, the data of the visualized medical device model in the depth coordinate system is transformed to the coordinate system of the visualized target object model to show the visualization perspective of the medical device's motion relative to the target object.

[0039] In some embodiments, the first transformation data includes a first transformation matrix;

[0040] The second transformation data includes the second transformation matrix;

[0041] The third transformation data includes the third transformation matrix.

[0042] Visual perspectives showcasing the movement of medical devices relative to a target object include:

[0043] By multiplying the data of the visualized medical device model in the coordinate system of the visualized medical device model by the inverse of the first transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed into the RGB coordinate system of the camera.

[0044] The data of the visualized medical device model in the RGB coordinate system is transformed into the depth coordinate system of the camera by multiplying the data in the RGB coordinate system by the third transformation matrix.

[0045] By multiplying the data of the visualized medical device model in the depth coordinate system by the second transformation matrix, the data of the visualized medical device model in the depth coordinate system is transformed into the coordinate system of the visualized target object model, so as to show the visualization perspective of the movement of the medical device relative to the target object.

[0046] In some embodiments, the visualization perspective for showing the motion of the target object relative to the medical device includes:

[0047] Based on the second transformation data, the data of the visualized target model in the coordinate system of the visualized target model is transformed to the depth coordinate system of the camera;

[0048] Based on the third transformation data, the data of the visualized target model in the depth coordinate system is transformed to the RGB coordinate system of the camera;

[0049] Based on the first transformation data, the data of the visualized target object model in the RGB coordinate system is transformed into the coordinate system of the visualized medical device model to show the visual perspective of the target object's motion relative to the medical device.

[0050] In some embodiments, the first transformation data includes a first transformation matrix;

[0051] The second transformation data includes the second transformation matrix;

[0052] The third transformation data includes the third transformation matrix.

[0053] Visual perspectives showing the motion of the target object relative to the medical device include:

[0054] By multiplying the data of the visualized target model in the coordinate system of the visualized target model by the inverse of the second transformation matrix, the data of the visualized target model in the coordinate system of the visualized target model is transformed into the depth coordinate system of the camera.

[0055] The data of the visualized target model in the depth coordinate system is transformed into the camera's RGB coordinate system by multiplying the data of the visualized target model in the depth coordinate system by the inverse of the third transformation matrix.

[0056] By multiplying the data of the visualized target model in the RGB coordinate system by the first transformation matrix, the data of the visualized target model in the RGB coordinate system is transformed into the coordinate system of the visualized medical device model, so as to show the visual perspective of the target object's movement relative to the medical device.

[0057] In some embodiments, the visualization perspective showing that both the medical device and the target are moving relative to the camera includes:

[0058] Based on the first transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the RGB coordinate system of the camera;

[0059] Based on the third transformation data, the data of the visualized medical device model in the RGB coordinate system is transformed to the depth coordinate system of the camera;

[0060] Based on the second transformation data, the data of the visualized target model in the coordinate system of the visualized target model is transformed to the depth coordinate system of the camera to show the visualization perspective of the medical device and the target object moving relative to the camera.

[0061] In some embodiments, the first transformation data includes a first transformation matrix;

[0062] The second transformation data includes the second transformation matrix;

[0063] The third transformation data includes the third transformation matrix.

[0064] Visual perspectives showing the movement of medical equipment and objects relative to the camera include:

[0065] By multiplying the data of the visualized medical device model in the coordinate system of the visualized medical device model by the inverse of the first transformation matrix, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed into the RGB coordinate system of the camera.

[0066] The data of the visualized medical device model in the RGB coordinate system is transformed into the depth coordinate system of the camera by multiplying the data in the RGB coordinate system by the third transformation matrix.

[0067] By multiplying the data of the visualized target model in the coordinate system of the visualized target model by the inverse of the second transformation matrix, the data of the visualized target model in the coordinate system of the visualized target model is transformed into the depth coordinate system of the camera, so as to show the visualization perspective of the medical device and the target object moving relative to the camera.

[0068] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0069] A memory and a processor, wherein the memory stores a program, and the processor, when executing the program in the memory, implements the positioning and navigation method according to any of the foregoing embodiments.

[0070] According to another aspect of the present invention, a readable storage medium is provided, wherein a computer-readable program or instructions are stored therein, which, when executed by a processor, implement the positioning and navigation method according to any of the foregoing embodiments.

[0071] According to another aspect of the present invention, a positioning and navigation device is provided, the positioning and navigation device comprising:

[0072] A first determining module is configured to determine first conversion data for mutual conversion between RGB data of a camera and spatial data of a visualized medical device model.

[0073] The second determining module is configured to determine second transformation data for the mutual conversion between depth data of the camera and spatial data of the visualized target model;

[0074] The third determining module is configured to determine third conversion data for the mutual conversion between RGB data and depth data of the camera;

[0075] A control module is communicatively connected to a first determining module, a second determining module, and a third determining module. The control module is configured to determine the relative positional relationship between the visualized medical device model and the visualized target object model based on the first conversion data, the second conversion data, and the third conversion data, and to display the desired visualization perspective.

[0076] The positioning and navigation method, positioning and navigation device, electronic device, and readable storage medium according to the present invention have at least one of the following advantages:

[0077] (1) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention can obtain the relative positional relationship between the visualization coil model and the visualization target model based on the corresponding conversion data, thereby eliminating the need for the registration process;

[0078] (2) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention solve the problem of needing to re-register due to misoperation;

[0079] (3) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention simplify the operation process and improve the processing efficiency and processing effect;

[0080] (4) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention can provide more accurate and more convenient navigation guidance;

[0081] (5) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention enable the target object (e.g., subject) to not wear a rigid body model, thereby expanding the applicability of the positioning and navigation method. Attached Figure Description

[0082] These and / or other aspects and advantages of the present invention will become apparent and readily understood from the following description of preferred embodiments taken in conjunction with the accompanying drawings, in which:

[0083] Figure 1 A positioning and navigation method according to an embodiment of the present invention is shown;

[0084] Figure 2 A flowchart illustrating the determination of first conversion data according to an embodiment of the present invention is shown;

[0085] Figure 3 A flowchart illustrating the determination of second transformed data according to an embodiment of the present invention is shown;

[0086] Figure 4 A positioning and navigation device according to an embodiment of the present invention is shown. Detailed Implementation

[0087] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings. In this specification, the same or similar reference numerals indicate the same or similar components. The following description of the embodiments of the present invention with reference to the accompanying drawings is intended to explain the overall inventive concept of the present invention and should not be construed as a limitation thereof.

[0088] In an embodiment of the present invention, a positioning and navigation method is provided, which can determine the relative positional relationship between a medical device and a target object (or, a visualized medical device model and a visualized target object model) through a corresponding transformation relationship, and display the desired visualization perspective to achieve positioning and navigation.

[0089] Medical devices include devices that manipulate a target object. For example, medical devices can include magnetic stimulation devices, electrical stimulation devices, light stimulation devices, and mechanical wave stimulation devices.

[0090] The target object includes the object to be treated, such as a body part of the subject. For example, a body part of the subject could be the head, feet, etc. The target point includes the point on the target object that is to be treated, such as a target point.

[0091] like Figure 1 As shown, the positioning and navigation method includes:

[0092] First transformation data is determined for the conversion between RGB data used in the camera and spatial data of the visualized medical device model;

[0093] Determine the second transformation data for converting between depth data used for the camera and spatial data for visualizing the target object model;

[0094] A third conversion data is determined for the mutual conversion between RGB data and depth data of the camera;

[0095] Based on the first, second, and third transformation data, the relative positional relationship between the visualized medical device model and the visualized target object model is determined, and the desired visualization perspective is displayed.

[0096] The embodiments of the present invention determine the relative positional relationship between the visualized medical device model and the visualized target object model based on the first conversion data, the second conversion data and the third conversion data, thereby eliminating the need to register the medical device and the target object and solving the problem of re-registration caused by misoperation.

[0097] Moreover, embodiments of the present invention eliminate the registration process, thereby simplifying the operation process and improving processing efficiency and results.

[0098] In embodiments of the present invention, the camera refers to a camera capable of providing both depth data and color data (RGB data), and the conversion relationship between the depth data and color data acquired by the camera is known. For example, the camera may be an RGBD camera. In one example, the camera may be a single camera integrating an RGB camera and a depth camera. Alternatively, the camera may consist of a separate RGB camera and a separate depth camera.

[0099] Determining the first transformation data can reduce the steps involved in registering medical devices. Specifically, such as... Figure 2 As shown, the first transformation data used to determine the conversion between RGB data for the camera and spatial data for visualizing the medical device model includes:

[0100] A marker pattern is set on a medical device, and the positional relationship between the medical device and the marker pattern is determined. The marker pattern includes at least one of a checkerboard pattern, a dot matrix pattern, a ChAruco pattern, and a deep learning training graph. The marker pattern can be fixed to the medical device by means of pasting, imprinting, or binding. Embodiments of this disclosure do not limit the size and position of the marker pattern, as long as it can be captured by a camera. The positional relationship can be a transformation matrix.

[0101] The spatial data of the marker pattern in the camera's RGB coordinate system is determined using RGB data acquired by the camera. During this process, the medical device needs to be within the camera's field of view. Determining the spatial data of the marker pattern in the camera's RGB coordinate system enables pose recognition of the marker pattern.

[0102] Based on the positional relationship between the medical device and the marker pattern, and the spatial data of the marker pattern in the RGB coordinate system, the spatial data of the visualized medical device model in the RGB coordinate system is determined. The positional relationship between the medical device and the marker pattern is rigid; therefore, the positional relationship between the medical device and the marker pattern is the same as the positional relationship between the visualized medical device model and the marker pattern in the RGB coordinate system. Determining the spatial data of the visualized medical device model in the RGB coordinate system based on this positional relationship and the spatial data of the marker pattern in the RGB coordinate system, i.e., determining the orientation of the visualized medical device model in the RGB coordinate system, is equivalent to determining the orientation of the visualized medical device model in the RGB coordinate system.

[0103] The first transformation data is determined based on the spatial data of the visualized medical device model in the RGB coordinate system. The first transformation data may be a first transformation matrix.

[0104] Since the RGB data of the camera cannot identify the posture of the medical device, the embodiments of the present invention indirectly obtain the posture of the medical device by setting a marking pattern on the medical device and recognizing the posture of the marking pattern. This reduces the steps of manually registering the medical device (e.g., a magnetic stimulation device, specifically a magnetic stimulation coil) and lowers the difficulty of operation.

[0105] The process of registering targets (such as the subject's head) can be reduced by identifying second-transformation data. Specifically, such as Figure 3 As shown, the second transformation data used to determine the conversion between depth data for the camera and spatial data for visualizing the target model includes:

[0106] The first data for the target portion of the visualized object model is obtained by using RGB and depth data acquired by the camera. During this process, the target object needs to be within the camera's field of view. The RGB data acquired by the camera represents the RGB data of the real-world environment, and the depth data represents environmental point cloud data, which includes point cloud data of the target object and surrounding objects. The target portion is the region of interest in navigation. For example, if the target object is the subject's head, the target portion could be the face.

[0107] Furthermore, the first data for obtaining the target portion of the visualized target model through the RGB data and depth data acquired by the camera includes:

[0108] The third data of the target portion is extracted from the RGB data. Existing methods can be used to extract the target portion and form the third data. For example, if the target portion is a face, a face recognition algorithm can be used for face extraction.

[0109] The depth data is converted to the plane containing the intrinsic parameters of the RGB data to obtain the fourth data. The collected environmental point cloud data can be converted to the plane containing the intrinsic parameters of the RGB data to obtain the fourth data. For example, the fourth data can be obtained using the D2C algorithm.

[0110] After removing the non-target portion of the fifth data from the fourth data based on the third data of the target portion, the first data of the target portion is obtained. Depth data includes both target and non-target portions, so after converting the depth data to the plane containing the intrinsic parameters of the RGB data, the non-target portion needs to be removed. The remaining portion then constitutes the first data of the target portion.

[0111] After matching the first data with the second data of the target object, the second transformed data is determined. The second transformed data can be a second transformed matrix.

[0112] Further, after matching the first data with the second data of the target object, determining the second converted data includes: performing a coarse match between the first data of the target portion and the second data of the target object; and performing a precise match between the coarsely matched data to determine the second converted data.

[0113] The second set of data for the target object includes its point cloud data. For example, if the target object is a head, head MRI (Magnetic Resonance Imaging) data can be converted into MRI point cloud data, which then constitutes the target object's point cloud data. Point cloud data refers to the set of points obtained after acquiring the spatial coordinates of each sampling point on the object's surface. The RANSAC (Random Sample Consensus) algorithm can be used to perform a coarse matching between the point cloud data of the target portion and the point cloud data of the target object. Coarse matching refers to a relatively rough registration when the transformation between the two point clouds is completely unknown, and its main purpose is to provide better initial transformation values ​​for fine registration.

[0114] The ICP (Iterative Closest Point) algorithm can be used to perform precise matching on data that has undergone coarse matching. Precise matching refers to further optimization to obtain a more accurate transformation given an initial transformation.

[0115] The embodiments of the present invention obtain the relationship between the visual head model and the depth data of the camera through a point cloud matching algorithm, which reduces the steps of the target object (e.g., the subject) wearing a rigid body model and registering it, expands the applicability of the positioning and navigation method, and can reduce the difficulty of operation.

[0116] The first conversion data determines the conversion relationship between the visualized medical device model and the camera's RGB data. The second conversion data determines the conversion relationship between the visualized target object model and the camera's depth data. To obtain the conversion relationship between the visualized medical device model and the visualized target object model, it is also necessary to determine the conversion relationship between the camera's RGB data and depth data. That is, a third conversion data is determined for the mutual conversion between the camera's RGB data and depth data. The third conversion data can be a third conversion matrix. The third conversion matrix can be determined using a camera calibration algorithm (e.g., Zhang's calibration method).

[0117] After determining the first, second, and third transformation data, the relative positional relationship between the visualized medical device model and the visualized target object model can be established, and the desired visualization perspective can be displayed. For example, multiple perspective navigation can be implemented to guide the operator to find the most desired action location (e.g., the target point) from their desired perspective.

[0118] The desired visualization perspective includes at least one of the following: a visualization perspective of the medical device moving relative to the target object, a visualization perspective of the target object moving relative to the medical device, and a visualization perspective in which both the medical device and the target object are moving relative to the camera.

[0119] In one example, the visualization perspectives showing the motion of a medical device relative to a target object include:

[0120] Based on the first transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the RGB coordinate system of the camera;

[0121] Based on the third transformation data, the data of the visualized medical device model in the RGB coordinate system is transformed to the depth coordinate system of the camera;

[0122] Based on the second transformation data, the data of the visualized medical device model in the depth coordinate system is transformed to the coordinate system of the visualized target object model to show the visualization perspective of the medical device's motion relative to the target object.

[0123] For example, the first transformation data includes the first transformation matrix M. rgb The second transformation data includes the second transformation matrix M. d2b The third transformation data includes the third transformation matrix M. r2d Visual perspectives showcasing the movement of medical devices relative to a target object include:

[0124] The data of the visualized medical device model in the coordinate system of the visualized medical device model is multiplied by the inverse matrix M of the first transformation matrix. rgb -1 The data of the visualized medical device model in the coordinate system of the visualized medical device model is converted to the RGB coordinate system of the camera;

[0125] The data of the visualized medical device model in the RGB coordinate system is multiplied by the third transformation matrix M. r2d The data of the visualized medical device model in the RGB coordinate system is converted to the depth coordinate system of the camera;

[0126] The data of the visualized medical device model in the depth coordinate system is multiplied by the second transformation matrix M. d2b The data of the visualized medical device model in the depth coordinate system is transformed to the coordinate system of the visualized target object model to show the visual perspective of the medical device's motion relative to the target object.

[0127] In one example, the visualization perspectives showing the motion of the target object relative to the medical device include:

[0128] Based on the second transformation data, the data of the visualized target model in the coordinate system of the visualized target model is transformed to the depth coordinate system of the camera;

[0129] Based on the third transformation data, the data of the visualized target model in the depth coordinate system is transformed to the RGB coordinate system of the camera;

[0130] Based on the first transformation data, the data of the visualized target object model in the RGB coordinate system is transformed into the coordinate system of the visualized medical device model to show the visual perspective of the target object's motion relative to the medical device.

[0131] For example, the first transformation data includes the first transformation matrix M. rgb The second transformation data includes the second transformation matrix M. d2b The third transformation data includes the third transformation matrix M. r2d Visual perspectives showing the movement of the target object relative to the medical device include:

[0132] The data of the visualized target model in the coordinate system of the visualized target model is multiplied by the inverse matrix M of the second transformation matrix. d2b -1 This transforms the data of the visualized target model in the coordinate system to the depth coordinate system of the camera;

[0133] The data of the visualized target model in the depth coordinate system is multiplied by the inverse matrix M of the third transformation matrix. r2d -1 The data of the visualized target model in the depth coordinate system is converted to the RGB coordinate system of the camera.

[0134] The data of the visualized target model in the RGB coordinate system is multiplied by the first transformation matrix M. rgb The data of the visualized target object model in the RGB coordinate system is converted to the coordinate system of the visualized medical device model to show the visual perspective of the target object's motion relative to the medical device.

[0135] In one example, a visualization showing that both the medical device and the target are moving relative to the camera includes:

[0136] Based on the first transformation data, the data of the visualized medical device model in the coordinate system of the visualized medical device model is transformed to the RGB coordinate system of the camera;

[0137] Based on the third transformation data, the data of the visualized medical device model in the RGB coordinate system is transformed to the depth coordinate system of the camera;

[0138] Based on the second transformation data, the data of the visualized target model in the coordinate system of the visualized target model is transformed to the depth coordinate system of the camera to show the visualization perspective of the medical device and the target object moving relative to the camera.

[0139] For example, the first transformation data includes the first transformation matrix M. rgb The second transformation data includes the second transformation matrix M. d2bThe third transformation data includes the third transformation matrix M. r2d Visual perspectives showing the movement of both medical equipment and the target object relative to the camera include:

[0140] The data of the visualized medical device model in the coordinate system of the visualized medical device model is multiplied by the inverse matrix M of the first transformation matrix. rgb -1 The data of the visualized medical device model in the coordinate system of the visualized medical device model is converted to the RGB coordinate system of the camera;

[0141] The data of the visualized medical device model in the RGB coordinate system is multiplied by the third transformation matrix M. r2d The data of the visualized medical device model in the RGB coordinate system is converted to the depth coordinate system of the camera;

[0142] The data of the visualized target model in the coordinate system of the visualized target model is multiplied by the inverse matrix M of the second transformation matrix. d2b -1 The data of the visualized target model in the coordinate system of the visualized target model is transformed into the depth coordinate system of the camera to show the visualization perspective of the medical device and the target object moving relative to the camera.

[0143] The embodiments of the present invention can display multiple navigation perspectives through the above three conversion data, allowing operators to select the navigation perspective as needed during processing, thereby enabling more effective tracking of target points (e.g., target points). This approach avoids the inconvenience of a single perspective and increases ease of use. The embodiments of this disclosure are not limited to the above three navigation perspectives and may also provide other required navigation perspectives based on the above three conversion data.

[0144] It should be noted that the embodiments of the present invention are not limited to the navigation process described above. For example, one step is to multiply the data of the visualized medical device model in the RGB coordinate system by the third transformation matrix M. r2d The data of the visualized medical device model in the RGB coordinate system is converted to the depth coordinate system of the camera; this step can be adjusted accordingly by changing the third transformation matrix.

[0145] The embodiments of the present invention are based on a 1:1 relationship between the model and the real object; for example, the visualized medical device model and the medical device are in a 1:1 relationship. However, those skilled in the art will understand that the embodiments of the present invention are not limited to a specific ratio between the model and the real object, and other ratios can be set, such as 1:2, 1:4, etc. As long as other ratios are converted to a 1:1 relationship during navigation, the positioning and navigation method of the embodiments of the present invention can be obtained.

[0146] In embodiments of the present invention, a positioning and navigation device is also provided. For example... Figure 4 As shown, the positioning and navigation device 100 includes a first determining module 10, a second determining module 20, a third determining module 30, and a control module 40.

[0147] The first determining module 10 is configured to determine first conversion data for the mutual conversion between RGB data of the camera and spatial data of the visualized medical device model. The first determining module 10 may determine the first conversion data using the method described in any of the above embodiments.

[0148] The second determining module 20 is configured to determine second conversion data for the mutual conversion between depth data of the camera and spatial data of the visualized target model. The second determining module 20 may determine the second conversion data using the method described in any of the above embodiments.

[0149] The third determining module 30 is configured to determine third conversion data for the mutual conversion between RGB data and depth data of the camera. The third determining module 30 may determine the third conversion data using the method described in any of the above embodiments.

[0150] The control module 40 is communicatively connected to the first determining module 10, the second determining module 20, and the third determining module 30, respectively. The control module 40 is configured to determine the relative positional relationship between the visualized medical device model and the visualized target object model based on the first, second, and third conversion data, and to display the desired visualization perspective. The control module 40 can use the method described in any of the above embodiments to determine the relative positional relationship in order to control the navigation process.

[0151] In embodiments of the present invention, a readable storage medium is provided. The readable storage medium stores a program or instructions that, when executed by a processor, implement the positioning and navigation method described in any of the above embodiments.

[0152] In embodiments of the present invention, "readable storage medium" refers to any medium that participates in providing a program or instructions to a processor for execution. The medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage devices. Volatile media include dynamic memory, such as main memory. Transmission media include coaxial cables, copper wires, and optical fibers, including conductors containing buses. Transmission media can also take the form of acoustic or optical waves, such as acoustic or optical waves generated during radio frequency (RF) and infrared (IR) data communications. Common forms of readable storage media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROMs, DVDs, any other optical media, punched cards, paper tape, any other physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, carrier waves as described below, or any other medium from which a computer can read.

[0153] In an embodiment of the present invention, an electronic device is also provided. The electronic device (not shown) includes a processor (not shown) and a memory (not shown). A program is stored in the memory, which, when executed by the processor, can implement the positioning and navigation method of any of the above examples.

[0154] In one example, the processor can be a microprocessor, such as a general-purpose processor like a graphics processing unit (GPU), a central processing unit (CPU), or a digital signal processor (DSP). In another example, the processor can also be a microprocessor core implemented through hardware circuitry, such as a microprocessor core implemented in hardware logic components using reconfigurable logic, including field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), and systems-on-a-chip (SoCs).

[0155] In one example, the processor can also be a virtual processor, which can be a virtual processor with Intel x86 processor features or a virtual processor with PowerPC processor features. Preferably, the processor is a graphics processor. In one example, the processor can be a single-core processor or a multi-core processor.

[0156] In one example, the memory includes volatile memory (i.e., random access memory) and non-volatile memory. Volatile memory includes main memory, cache, etc., while non-volatile memory includes auxiliary memory, etc. In one example, the memory can be configured as remote memory, which can be connected to the processor via a network (wired or wireless network). The network includes, but is not limited to, wide area networks (WANs), local area networks (LANs), metropolitan area networks (MANs), personal area networks (PANs), the Internet, satellite communication networks, and any combination thereof.

[0157] In one example, the processor creates a corresponding task thread based on a program retrieved from memory and executes the thread. In another example, the processor retrieves a program from secondary storage based on a read instruction from memory to create a corresponding task thread and executes the thread. The above program is used to implement a positioning and navigation method.

[0158] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that it can also be implemented in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the method steps described in conjunction with any of the examples herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Implementation in hardware or software depends primarily on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods for each specific application to implement the described functionality, but such implementation should not be considered beyond the scope of this application.

[0159] When the method steps are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Therefore, the technical solution of this invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various examples of this invention.

[0160] The positioning and navigation method, positioning and navigation device, electronic device, and readable storage medium according to the present invention have at least one of the following advantages:

[0161] (1) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention can obtain the relative positional relationship between the visualization coil model and the visualization target model based on the corresponding conversion data, thereby eliminating the need for the registration process;

[0162] (2) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention solve the problem of needing to re-register due to misoperation;

[0163] (3) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention simplify the operation process and improve the processing efficiency and processing effect;

[0164] (4) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention can provide more accurate and more convenient navigation guidance;

[0165] (5) The positioning and navigation method, positioning and navigation device, electronic device and readable storage medium of the present invention enable the target object (e.g., subject) to not wear a rigid body model, thereby expanding the applicability of the positioning and navigation method.

[0166] While some embodiments of the present general inventive concept have been shown and described, those skilled in the art will understand that changes may be made to these embodiments without departing from the principles and spirit of the present general inventive concept, the scope of which is defined by the claims and their equivalents.

Claims

1. A positioning and navigation method, comprising: determining first conversion data for mutual conversion between RGB data of a camera and spatial data of a visualized medical device model, wherein a marker pattern is set on the medical device and pose recognition is performed on the marker pattern in a RGB coordinate system of the camera; determining second conversion data for mutual conversion between depth data of the camera and spatial data of a visualized target object model, comprising first data of a target portion of the visualized target object model obtained by RGB data and depth data obtained by the camera, and determining the second conversion data after matching the first data with second data of the target object; determining third conversion data for mutual conversion between the RGB data and the depth data of the camera; determining a relative position relationship between the visualized medical device model and the visualized target object model based on the first conversion data, the second conversion data and the third conversion data, and displaying a desired visualized view.

2. The positioning and navigation method of claim 1, wherein determining the first conversion data for mutual conversion between the RGB data of the camera and the spatial data of the visualized medical device model comprises: setting a marker pattern on the medical device and determining a position relationship between the medical device and the marker pattern; determining spatial data of the marker pattern in a RGB coordinate system of the camera by RGB data obtained by the camera; determining spatial data of the visualized medical device model in the RGB coordinate system based on the position relationship between the medical device and the marker pattern and the spatial data of the marker pattern in the RGB coordinate system; determining the first conversion data based on the spatial data of the visualized medical device model in the RGB coordinate system.

3. The positioning and navigation method of claim 2, wherein the marker pattern comprises at least one of a checkerboard pattern, a dot matrix pattern, a chAruco pattern and a deep learning training pattern.

4. The positioning and navigation method of claim 3, wherein obtaining the first data of the target portion of the visualized target object model by the RGB data and the depth data obtained by the camera comprises: extracting third data of the target portion in the RGB data; converting the depth data into a plane where an intrinsic parameter of the RGB data is located and obtaining fourth data; obtaining the first data of the target portion after removing fifth data of non-target portions in the fourth data based on the third data of the target portion.

5. The positioning and navigation method of claim 4, wherein determining the second conversion data after matching the first data with the second data of the target object comprises: coarsely matching the first data of the target portion with the second data of the target object; determining the second conversion data after accurately matching the coarsely matched data.

6. The positioning and navigation method of claim 5, wherein the depth data comprises environmental point cloud data; the first data of the target portion comprises point cloud data of the target portion; the second data comprises target object point cloud data.

7. The positioning and navigation method of any one of claims 1-6, wherein The desired visualization perspective includes at least one of a visualization perspective of the medical device movement relative to the target, a visualization perspective of the target movement relative to the medical device, and a visualization perspective of both the medical device and the target movement relative to the camera.

8. The positioning and navigation method of claim 7, wherein, displaying the visualization perspective of the medical device movement relative to the target includes: converting data of the visualized medical device model in a coordinate system of the visualized medical device model to an RGB coordinate system of the camera based on first conversion data; converting data of the visualized medical device model in the RGB coordinate system to a depth coordinate system of the camera based on third conversion data; converting data of the visualized medical device model in the depth coordinate system to a coordinate system of the visualized target model based on the second conversion data to display the visualization perspective of the medical device movement relative to the target.

9. The positioning and navigation method of claim 8, wherein, the first conversion data includes a first conversion matrix; the second conversion data includes a second conversion matrix; the third conversion data includes a third conversion matrix, displaying the visualization perspective of the medical device movement relative to the target includes: converting data of the visualized medical device model in a coordinate system of the visualized medical device model to an RGB coordinate system of the camera by multiplying the data of the visualized medical device model by an inverse of the first conversion matrix; converting data of the visualized medical device model in the RGB coordinate system to a depth coordinate system of the camera by multiplying the data of the visualized medical device model in the RGB coordinate system by the third conversion matrix; converting data of the visualized medical device model in the depth coordinate system to a coordinate system of the visualized target model by multiplying the data of the visualized medical device model in the depth coordinate system by the second conversion matrix to display the visualization perspective of the medical device movement relative to the target.

10. The positioning and navigation method of claim 7, wherein, displaying the visualization perspective of the target movement relative to the medical device includes: converting data of the visualized target model in a coordinate system of the visualized target model to a depth coordinate system of the camera based on second conversion data; converting data of the visualized target model in the depth coordinate system to an RGB coordinate system of the camera based on third conversion data; converting data of the visualized target model in the RGB coordinate system to a coordinate system of the visualized medical device model based on first conversion data to display the visualization perspective of the target movement relative to the medical device.

11. The positioning and navigation method of claim 10, wherein, the first conversion data includes a first conversion matrix; the second conversion data includes a second conversion matrix; the third conversion data includes a third conversion matrix, displaying the visualization perspective of the target movement relative to the medical device includes: converting data of the visualized target model in a coordinate system of the visualized target model to an RGB coordinate system of the camera by multiplying the data of the visualized target model by an inverse of the first conversion matrix; converting data of the visualized target model in the RGB coordinate system to a depth coordinate system of the camera by multiplying the data of the visualized target model in the RGB coordinate system by the third conversion matrix; converting data of the visualized target model in the depth coordinate system to a coordinate system of the visualized medical device model by multiplying the data of the visualized target model in the depth coordinate system by the second conversion matrix to display the visualization perspective of the target movement relative to the medical device. The data of the visualized target object model in the coordinate system of the visualized target object model is converted to the depth coordinate system of the camera by multiplying the data of the visualized target object model in the coordinate system of the visualized target object model by an inverse matrix of the second conversion matrix; The data of the visualized target object model in the depth coordinate system is converted to the RGB coordinate system of the camera by multiplying the data of the visualized target object model in the depth coordinate system by an inverse matrix of the third conversion matrix; The data of the visualized target object model in the RGB coordinate system is converted to the coordinate system of the visualized medical device model by multiplying the data of the visualized target object model in the RGB coordinate system by the first conversion matrix, to show the visualized perspective of the target object relative to the movement of the medical device.

12. The positioning and navigation method of claim 7, wherein showing the visualized perspective of the medical device and the target object both relative to the movement of the camera comprises: converting the data of the visualized medical device model in the coordinate system of the visualized medical device model to the RGB coordinate system of the camera based on the first conversion data; converting the data of the visualized medical device model in the RGB coordinate system to the depth coordinate system of the camera based on the third conversion data; converting the data of the visualized target object model in the coordinate system of the visualized target object model to the depth coordinate system of the camera based on the second conversion data, to show the visualized perspective of the medical device and the target object both relative to the movement of the camera.

13. The positioning and navigation method of claim 12, wherein the first conversion data comprises a first conversion matrix; the second conversion data comprises a second conversion matrix; the third conversion data comprises a third conversion matrix, showing the visualized perspective of the medical device and the target object both relative to the movement of the camera comprises: converting the data of the visualized medical device model in the coordinate system of the visualized medical device model to the RGB coordinate system of the camera by multiplying the data of the visualized medical device model in the coordinate system of the visualized medical device model by an inverse matrix of the first conversion matrix; converting the data of the visualized medical device model in the RGB coordinate system to the depth coordinate system of the camera by multiplying the data of the visualized medical device model in the RGB coordinate system by the third conversion matrix; converting the data of the visualized target object model in the coordinate system of the visualized target object model to the depth coordinate system of the camera by multiplying the data of the visualized target object model in the coordinate system of the visualized target object model by an inverse matrix of the second conversion matrix, to show the visualized perspective of the medical device and the target object both relative to the movement of the camera.

14. An electronic device, comprising: The electronic device comprises: a memory and a processor, wherein the memory has a program stored thereon, and the processor implements the positioning and navigation method according to any one of claims 1-13 when executing the program on the memory.

15. A readable storage medium, wherein the readable storage medium has a computer readable program or instruction stored thereon, and the computer readable program or instruction is executed by a processor to implement the positioning and navigation method according to any one of claims 1-13.

16. A positioning navigation device, characterized by The positioning and navigation device comprises: a first determining module configured to determine first conversion data for mutual conversion between RGB data of a camera and spatial data of a visualized medical device model, wherein a marker pattern is set on the medical device and pose recognition is performed on the marker pattern in a RGB coordinate system of the camera; a second determining module configured to determine second conversion data for mutual conversion between depth data of the camera and spatial data of a visualized target object model, comprising first data of a target part of the visualized target object model obtained by RGB data and depth data obtained by the camera, and the second conversion data is determined after matching the first data with second data of the target object; a third determining module configured to determine third conversion data for mutual conversion between the RGB data and the depth data of the camera; a control module in communication connection with the first determining module, the second determining module and the third determining module respectively, and configured to determine a relative position relationship between the visualized medical device model and the visualized target object model based on the first conversion data, the second conversion data and the third conversion data, and to display a desired visualized view.

Images