Medical navigation method, medical navigation system, and computer program

Abstract

The present invention provides a medical navigation method for guiding a medical instrument to an appropriate position on a body surface in accordance with individual differences in body type.　This medical navigation method comprises: a step for holding three-dimensional point cloud information of a subject as body model data of the subject (S102), holding N pieces of reference model data and medical instrument arrangement data in each piece of the reference model data, and determining criterion model data exhibiting the highest similarity to the body model data of the subject (S106); a step for using the body model data of the subject as a target to apply, to the criterion model data, registration using a non-rigid ICP algorithm (S108); and a step for using a coordinate transformation formula for the three-dimensional point cloud information between the body model data of the subject and the criterion model data obtained by registration to transform arrangement data in the criterion model data into the medical instrument arrangement data in the body model data of the subject (S110).

Description

Medical navigation method, medical navigation system, and computer program

[0001] The present invention relates to a medical navigation method, a medical navigation system, and a computer program executed in a medical robot device.

[0002] When a clinical expert such as a doctor remotely controls a medical instrument such as an ultrasound probe or a stethoscope via a medical robot or other device, the robot must be guided to determine where on the body surface the instrument should be placed in order to perform the examination efficiently. Furthermore, when a doctor provides remote medical care, it may be necessary for a non-medical person, such as the patient or a close relative of the patient, to place the medical instrument such as a stethoscope in an appropriate position on the patient's body surface instead of the doctor. However, the position where the medical instrument should be placed, i.e., the position of the patient's organs to be examined, varies depending on individual differences in the patient's body shape, making it important to estimate the position taking individual differences into account.

[0003] For example, Patent Documents 1 and 2 and Non-Patent Document 1 disclose a method for mapping functional areas of the brain by applying a standard model of the human brain to an image of a patient's brain to perform alignment and deformation (registration).

[0004] US Patent No. 8,311,359 JP 2019-535472 A

[0005] Brian B Avants, Nicholas J Tustison, Michael Stauffer, Gang Song, Baohua Wu, and James C Gee, “The Insight ToolKit image registration framework”, Front Neuroinform, Frontiers Research Foundation, Switzerland, April 28,2014

[0006] However, the above-mentioned conventional techniques deal with brain images that have a uniform shape compared to the patient's body shape, and registration requires medical imaging modalities such as CT (Computed Tomography) and MRI (Magnetic Resonance Imaging), which require time-consuming imaging.

[0007] In view of the above problems, the present invention aims to provide a medical navigation method, a medical navigation system, and a computer program that can guide medical instruments such as ultrasound probes and stethoscopes to appropriate positions on the body surface in accordance with individual differences in body shape using an easily usable RGB-D (image / depth) camera.

[0008] In order to solve the above-mentioned problems, one aspect of the present invention is a medical navigation method executed by a computer system for positioning a medical instrument on a subject, wherein the computer system holds three-dimensional point cloud information of the subject as body model data of the subject, and holds N pieces of reference model data representing the shape of a human body and placement data representing placement positions of the medical instrument in each of the reference model data, the medical navigation method including the steps of: determining, as reference model data, one of the N pieces of reference model data that is most similar to the body model data of the subject; applying registration to the reference model data using a non-rigid ICP (interactive closest point) algorithm, with the body model data of the subject as a target; and converting the placement data in the reference model data into placement data of the medical instrument in the body model data of the subject, using a coordinate transformation formula for the three-dimensional point cloud information between the body model data of the subject obtained by the registration and the reference model data.

[0009] Another aspect of the present invention is a medical navigation system for positioning a medical instrument on a subject, the medical navigation system storing three-dimensional point cloud information of the subject as body model data of the subject, storing N pieces of reference model data representing the shape of a human body and placement data representing the placement position of the medical instrument in each piece of reference model data, determining, as reference model data, the reference model data of the N pieces of reference model data that is most similar to the body model data of the subject, using the body model data of the subject as a target, applying registration to the reference model data using a non-rigid ICP (interactive closest point) algorithm, and converting the placement data in the reference model data into placement data of the medical instrument in the body model data of the subject using a coordinate transformation formula for the three-dimensional point cloud information between the body model data of the subject obtained by the registration and the reference model data.

[0010] Another aspect of the present invention is a computer program for causing a computer system to execute the above-described medical navigation method. Another aspect of the present invention is a computer-readable recording medium storing a computer program for causing a computer system to execute the above-described medical navigation method.

[0011] FIG. 1 is a diagram showing an example of the configuration of a medical robot device (medical navigation system) according to an embodiment of the present invention. FIG. 2 is a diagram showing an overview of a medical navigation method according to an embodiment of the present invention. FIG. 3 is a diagram showing a human body model created in a simulation. FIG. 4 is a diagram showing all results of registration errors according to chamfer distance. FIG. 5 is a diagram showing an example of the processing flow of a medical navigation method according to an embodiment of the present invention.

[0012] Embodiments of the present invention will be described in detail below with reference to the drawings. The medical navigation method and medical navigation system according to this embodiment are a method and system for appropriately estimating the position of a medical instrument, such as an ultrasound probe or a stethoscope, on the body surface of a subject when a medical procedure is performed on the subject using a medical robotic device. The following description will be given, as an example, of a case in which the medical instrument is a stethoscope and auscultates a specific location on the patient's body (subject). (Configuration of the Medical Robotic Device) FIG. 1 is a diagram showing an example of the configuration of a medical robotic device (medical navigation system) that executes the medical navigation method according to this embodiment. The medical robotic device 1 according to this embodiment includes a robotic arm 10, a constant-force passive scanning mechanism (end effector) 20, and an RGB-D camera 30. The robotic arm 10 is configured to be able to move around a patient (subject) 5, on whom a medical procedure is to be performed. (Note that FIG. 1 is a diagram for illustrative purposes only, and a mannequin is used to represent the patient 5 (specifically, the patient's body; hereinafter, the same may be referred to as the "patient's body 5").) The tip of the robot arm 10 is provided with a constant-force passive scanning mechanism (end effector) 20, either integrally or detachably. The constant-force passive scanning mechanism 20 holds the medical instrument 40 and allows the medical instrument 40 to be pressed against the body surface of the patient 5 while maintaining a constant load. Here, the medical instrument may refer to any instrument necessary for medical procedures performed on the patient 5, such as an ultrasound probe or a stethoscope. More specifically, an example of a medical instrument that can be used with the medical robot device 1 of this embodiment is an electronic stethoscope, such as the JPES-01 electronic stethoscope (digital stethoscope) manufactured by Mitrica Co., Ltd. Sound collected by the electronic stethoscope when it comes into contact with the patient 5 can be transmitted as data to a computer device (such as the client computer device 60 described below). Similarly, other medical instruments may also be used as long as sound and video data acquired by the medical instrument when it comes into contact with the patient 5 can be transmitted to a computer device.The robot arm 10 can be equipped with an RGB-D camera 30 either integrally or detachably at the tip of the arm. The RGB-D camera 30 is a camera that can simultaneously capture a color image (RGB information) and a distance image (distance information).

[0013] The medical robot device 1 may also include a client computer device 60. The client computer device 60 is directly or indirectly connected to each of the robot arm 10, the constant-force passive scanning mechanism (end effector) 20, and the RGB-D camera 30 of the medical robot device 1, and transmits and receives data to and from these components. For example, the client computer device 60 transmits control signals for controlling each component and data necessary for operating each component. The client computer device 60 also receives data acquired or generated by each component from each component. The client computer device 60 can be realized by a hardware configuration similar to that of a general computer device. The client computer device 60 may include, for example, a processor, RAM (Random Access Memory), ROM (Read Only Memory), an internal hard disk drive, an external hard disk drive, removable memory such as a CD, DVD, USB memory, memory stick, or SD card, an input / output user interface (display, keyboard, mouse, touch panel, speaker, microphone, LED, etc.), and a wired / wireless communication interface capable of communicating with each component of the medical robot device 1 and other computers. The processor of the client computer device 60 can, for example, read a computer program for causing the client computer device 60 to execute the medical navigation method according to this embodiment, which has been stored in advance in the hard disk drive, ROM, or removable memory, into a memory such as RAM, and execute the necessary data while reading it as needed from the hard disk drive, ROM, removable memory, etc. Such operations of the client computer device 60 realize various processes in the medical robot device 1 described in this embodiment. In addition, various data used in each process described in this embodiment is stored in a storage device or storage medium such as a hard disk drive, RAM, or removable memory, and is read and used by the processor as needed during processing.

[0014] The medical robotic device 1 shown in FIG. 1 is a medical robotic device actually constructed by the inventors of the present application. A six-axis, 6-DOF collaborative robotic arm (UR5e, Universal Robot, Denmark) was used as the robot arm 10. A LiDAR camera (L515 RealSense, Intel, USA) was used as the RGB-D camera 30, capturing the three-dimensional contour of the body surface of the patient 5 as point cloud data. A well-known computer device (Dell Precision 5380, Dell, USA) was used as the client computer device 60. The medical robotic device 1 in this example was constructed, for example, to perform auscultation using a stethoscope. Note that, because FIG. 1 illustrates a medical robotic device constructed experimentally, a mannequin is used as the patient 5. As described below, the nipples and navel, which are landmarks of the patient 5 in this example, are unclear on the mannequin, so the inventors added markers to these positions. 1 is merely an example, and it goes without saying that the configuration of the medical robot device 1 according to this embodiment is not limited to this. The same applies throughout this specification.

[0015] In the configuration example shown in FIG. 1 , a spring-based constant-force passive scanning mechanism is mounted on the end effector 20 of the robot arm 10 to adaptively grasp the stethoscope (medical instrument) 40 on the body surface of the patient 5 from the perspective of safety during diagnosis of the patient 5. A six-axis force / torque sensor 25 (Axia-80-M20, ATI Industrial Automation, USA) is attached to the base of the constant-force passive scanning mechanism (end effector) 20, configured to measure the contact force when placing the stethoscope 40 on the body surface of the patient 5. Furthermore, a client computer 60 that controls the medical robot device 1 and an external server computer (not shown) are directly connected to a network (speed: 1 GB / sec (gigabytes per second)), and the data transmission protocol is TCP / IP. In this example, the RGB-D camera 30 is provided within the constant-force passive scanning mechanism (end effector) 20, but this is not limiting. The RGB-D camera 30 is fixed so that its relative position with respect to the robot arm 10 does not change, and may be installed in a location different from that of the constant load passive scanning mechanism (end effector) 20 .

[0016] The RGB-D camera 30 acquires depth and color information of the shape of the patient 5 as point cloud data, and the point cloud data is used for coordinate registration (alignment) using the positional relationship between the robot arm 10 and the patient 5. Because the RGB-D camera 30 is attached to the end effector 20 of the robot arm 10, the positional relationship between the RGB-D camera 30 and the robot arm 10 remains kinematically fixed even when the robot arm 10 moves around the patient 5. Therefore, the position of the acquired point cloud data of the patient 5 is linked to the coordinates of the robot arm 10.

[0017] The robot arm 10 can be controlled by URScript (Universal Robots A / S), a programming language used in the medical robot device 1. The client computer 60 controlling the medical robot device 1 can send URScript commands to an external server computer (not shown) via socket communication. Point cloud data was acquired using Intel RealSense SDK 2.0. A customized software system based on Python programming in Visual Studio Code can synchronize control of the robot arm 10 with the reading of the point cloud data. (Overview of the Medical Navigation Method) In this embodiment, the medical navigation method uses body surface registration between the patient's body 5 and a reference body model previously registered in the medical navigation system to localize the auscultation area using a stethoscope (medical instrument) while compensating for individual differences in patient body shape. Here, the "reference model" refers to a human body model that serves as the basis for body surface registration. The auscultation position where the stethoscope should be placed (including a region (auscultation region) with a certain degree of extent; the same applies below) is determined in advance for the reference model based on general medical knowledge of clinical experts, etc., and placement data indicating the auscultation position is stored in the medical navigation system. In the body surface registration of this embodiment, a correspondence relationship between the surface of the patient's body 5 and the reference model is calculated. Based on the correspondence relationship calculated by body surface registration, the auscultation region identified in the reference model can be projected onto the patient's body.

[0018] One commonly known registration technique is the interactive closest point (ICP) algorithm. This algorithm applies input point cloud data of the body surface using an affine transformation to fit a reference model to the patient's body surface, and as a result, the point cloud is considered to be rigid. However, since the body surface is individual-dependent, rigid registration using ICP cannot find an accurate correspondence between the surface of the patient's body 5 and the reference model, and the input point cloud data must be deformed to fit the body surface of the target (patient's body 5).

[0019] Therefore, the medical navigation method and medical navigation system according to this embodiment employ an improved ICP algorithm, non-rigid ICP, which can non-rigidly fit point cloud data to the patient's body 5 using feature points and deformation constraints. In this embodiment, the auscultation position on the patient's body 5 is identified by applying body surface registration between the patient 5 and a reference model. More specifically, several types of human body models (reference models) that can serve as reference models are prepared in advance, and a reference model that is close to the patient's body is selected, thereby improving the accuracy of localizing the auscultation area. For each reference model, the auscultation position (including a relatively large area (auscultation area) (the same applies below) where the stethoscope should be placed is determined in advance based on general medical knowledge of clinical experts, etc., and placement data indicating the auscultation position is stored in the medical navigation system.

[0020] In this embodiment, as an example, a case will be described in which the positions of four valves (aortic valve, pulmonary valve, tricuspid valve, and mitral valve) necessary for auscultation of a patient's (subject's) heart are estimated. The tricuspid valve, mitral valve, pulmonary valve, and aortic valve are generally located on the left side of the lower sternum near the fifth intercostal space, at the left fifth intercostal cusp on the midclavicular line (approximately 10 cm from the midline), on the inner edge of the second left intercostal space, and in the third right intercostal space, respectively. However, it is difficult for non-medical personnel to correctly identify these valves. The medical navigation method and medical navigation system according to this embodiment apply non-rigid registration between the patient's body 5 and a reference model of the human body (registered in advance in the medical navigation system) to estimate the positions corresponding to each valve on the patient's body surface (the positions where the stethoscope should be placed) from the reference model.

[0021] FIG. 2 is a diagram illustrating an example of an overview of a medical navigation method according to this embodiment. As shown in FIG. 2, the medical navigation method according to this embodiment includes two processing steps: step (A) and step (B). In step (A), a reference model that is most similar to the patient's body data (patient body model) 50 obtained by capturing an image of the patient's body 5 with an RGB-D camera 30 is selected as the reference model 52 from among multiple reference models 52-1, 52-2, ..., 52-N (N is an integer greater than or equal to 2) that are the basis of the reference model and that are pre-registered in the medical navigation system. In step (B), body surface registration using non-rigid ICP is performed on the reference model 52 selected in step (A), and the reference model 52 is deformed to fit the patient's body 5 (patient body model 50). As a result, the positions (reference positions) 55 of each valve are projected from the reference model 52 onto the patient's body 5 (patient body model 50), thereby estimating the positions (estimated positions) 56 of each valve on the body surface of the patient's body 5. (Step (A): Selection of Reference Model Most Similar to Patient's Body) An example of the processing in step (A) will be described in detail below. In order to select a reference model that is closest to the patient's body 5, the point cloud data of the patient's body model 50 and the point cloud data of each reference model 52-k (k: an integer from 1 to N) are superimposed, and the degree of superimposition is calculated to measure the similarity between the patient's body model 50 and each reference model 52-k. The chamfer distance is widely used to measure the similarity between two point sets, and is defined as follows:

[0022]

[0023] Here, S １ and S ２ is a subset of the point cloud data. x and y are １ and S ２ The point data included in d ＣＤ is S １ and S ２ represents the chamfer distance between the ＣＤcan be calculated by summing the squares of the distances between the nearest neighboring points of the two point clouds. If the shape deviation between the two point clouds is large, the distance between the nearest neighboring points of the two point clouds will be large, and therefore the chamfer distance will be large. By calculating the chamfer distance between the patient body model 50 and each reference model 52-k, the reference model 52-k (reference model 52) that is most similar to the patient body model 50 is identified.

[0024] To calculate the chamfer distance, point cloud subsets with different shapes and arrangements are first roughly overlapped. Then, using ICP registration, the two subsets are roughly registered based on local features of the point clouds. For each subset, Fast Point Feature Histogram (FPFH) features are calculated, and the correspondence between the features of the two subsets can be found using the Random Sample Consensus (RANSAC) algorithm.

[0025] For every reference model 52-k, the chamfer distance is calculated as the similarity to the patient body model 50 after applying ICP registration. The most similar reference model 52-k (reference model 52) is used in the non-rigid ICP process described below. (Step (B): Non-rigid ICP for Body Surface Registration) Non-rigid ICP is used to deform the reference model 52 identified in step (A) to fit the patient body model 50. In this embodiment, non-rigid ICP is applied to mesh data converted from point cloud data. The source mesh converted from the point cloud data of the reference model 52 is given as a set V of n vertices (n is a natural number equal to or greater than 3) and a set ε of m edges (m is a natural number equal to or greater than 3). Registration involves detecting a detection parameter X representing a set V(X) of displaced source vertices, which is a deformed mesh relative to the body surface of the patient 5. To determine the parameter X, an objective function E was defined as follows:

[0026]

[0027] Here, E ｄ , E ｓ , E ｌ represent the distance objective function, stiffness objective function, and sensitivity mark distance objective function, respectively. α and β represent stiffness and landmark parameters. Corresponding source vertices and target vertices, which are vertices in the target mesh transformed from the point cloud data of the patient body model 50, are denoted by (v ｉ , u ｉ ), then the distance objective function is defined as:

[0028]

[0029] Here, the reliability of the match is W ｉ If there is no corresponding vertex, the weight is set to zero. If a corresponding vertex is found, the weight is set to one. A stiffness objective function is used to normalize the deformation by penalizing the weighted difference in the transformations of adjacent vertices. The stiffness objective function is the Frobenius norm

[0030]

[0031] and a weighting matrix G, which is defined as follows:

[0032]

[0033] where G was used to weight the difference between the rotational and distortion parts of the deformation relative to the translational part of the deformation. Finally, a landmark distance objective function was used for registration initialization and guidance, and was defined as follows:

[0034]

[0035] where:

[0036]

[0037] represents a set of landmarks that map source vertices to target vertices. In this embodiment, the nipples and navel were used as landmarks to accurately match the reference model 52 to the body surface of the upper body of the patient 5. The positions of these landmarks were manually determined on the body surfaces of both the reference model 52 and the patient body model 50 before applying nonrigid ICP registration. The total objective function E can be solved by the method disclosed in Amberg B, Romdhani S, Vetter T (2007) Optimal Step Nonrigid ICP Algorithms for Surface Registration. 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8. (Simulation) The inventors of the medical navigation method and medical navigation system according to this embodiment conducted simulations to verify that the accuracy of registration is improved by selecting a reference model that is close to the patient's body 5 (patient body model 50). For the simulation, the inventors created several types of human body models using the digital human platform software "DhaibaWorks" (National Institute of Advanced Industrial Science and Technology, https: / / www.dhaibaworks.com / ). This software allows human body models of various shapes to be generated as mesh data. As shown in FIG. 3 , the inventors used this software to create nine types of human body models in advance, ranging from short models (#1, #2) to tall models (#3, #4) and from light models (#5, #6) to heavy models (#7, #8), compared to a model (#0) with standard height and weight.

[0038] In this simulation, the reference model 52 (source mesh) was fixed to the standard model (#0 in Figure 3), and the patient body model 50 (target mesh) was set to each of the other models (#1 to #8 in Figure 3). A non-rigid ICP was then applied to each pair of the source mesh and each target mesh. The registration accuracy was evaluated by the error between the projected position of each valve in the registration and the actual position of each valve determined by the medical knowledge of a clinical expert. Furthermore, the chamfer distance was calculated for each pair of the source mesh and each target mesh.

[0039] Table 1 shows the non-rigid ICP registration error results and chamfer distances for each valve position in the simulated human body model. Auscultation areas I to IV in Table 1 refer to the auscultation areas for the aortic valve, pulmonary valve, tricuspid valve, and mitral valve, respectively. The results suggest that the greater the deviation in body shape between the target model (#1 to #8 in Figure 3 ) and the standard model (#0 in Figure 3 ), the greater the registration error. The chamfer distance indicates the degree of shape deviation, which corresponds to the registration error results. The simulation results show that selecting a reference model 52 that is close to the patient's body 5 improves the accuracy of registration.

[0040]

[0041] (Experiment) Next, the inventors of the medical navigation method and medical navigation system according to this embodiment conducted an experiment on the medical navigation method according to this embodiment using an actual human body. For the experiment, eight body surface data sets were obtained from eight healthy male volunteers. The male volunteers were selected taking into consideration the diversity of their body shapes. The average height, weight, and body mass index (BMI) of the volunteers were 1.73±0.06 m, 69.1±5.87 kg, and 23.2±1.98 kg / m, respectively. ２ Table 2 shows detailed information about the recruited volunteers.

[0042]

[0043] This experiment was conducted using the medical robot device 1 shown in Figure 1. Point cloud data of the body surface was acquired using an RGB-D camera 30 (L515 RealSense, Intel, USA). The volunteers lay on a bed, and the RGB-D camera 30 was placed approximately 80 cm above the surface of the bed. As with the simulation described above, the actual auscultation positions of each valve on each volunteer's body surface were determined by a clinical expert. In this experiment, the registration error and chamfer distance were calculated, and all combinations of the body surfaces of eight volunteers were compared. The body data of one volunteer was fixed as a source model (reference model), and the body data of the other seven volunteers were used as reference models (i.e., N = 7). Registration was performed between the source model and each reference model, and this process was repeated while fixing the body data of each of the eight volunteers to the source model.

[0044] Table 3 shows the results of non-rigid ICP registration performed between the physical data of each volunteer. Auscultation regions I to IV in Table 3 refer to the auscultation regions of the aortic valve, pulmonary valve, tricuspid valve, and mitral valve, respectively. In Table 3, the minimum and maximum values ​​of the average values ​​of the chamfer distance and registration error calculated for each source model are shown in bold (shaded). The results show that in approximately 7 / 8 of the source model conditions, the minimum chamfer distance corresponds to the minimum registration error, and the maximum chamfer distance corresponds to the maximum registration error.

[0045]

[0046] Figure 4 shows the overall results of registration error as a function of chamfer distance. For the results shown in Figure 4, a linear regression analysis was performed to examine the strength of the association between the accuracy of the non-rigid ICP registration and the similarity of the compared models. The linear coefficient of determination, R ２was approximately 0.66, indicating some relationship between the chamfer distance and registration error. (Conclusion) As explained above, the non-rigid ICP registration according to this example is capable of estimating the auscultation area with an average error of 5 to 19 mm, providing a more accurate auscultation area that takes into account individual differences in body shape. Errors of less than 20 mm are considered to be within a range that does not qualitatively affect auscultation. Furthermore, the above-mentioned simulations and experiments confirmed that registration accuracy may depend on the similarity between the body surface of the patient's body 5 (patient body model 50) and the body surface of the reference model 52. Statistical results indicated a correlation between registration accuracy and the similarity of the models used and the equivalent chamfer distance.

[0047] Furthermore, in the process of non-rigid ICP registration, the nipples and navel were used as landmarks. Instead of or in addition to these, other easily identifiable locations on the body surface, such as rib boundaries, can be used as landmarks. By using additional landmarks, the registration accuracy can be further improved. Furthermore, in the above example, seven reference models (data) were used, but this is not a limitation. For example, if body type is considered to be approximately equal to BMI, body types can be classified into six categories according to the obesity assessment criteria (for both men and women) established by the Japan Society for the Study of Obesity, as shown in the table below. Based on the above classification, the number of reference models N may be an integer of 6 or greater, and more preferably, an integer of 12 or greater (6 for each gender). The above-mentioned criteria of the Japan Society for the Study of Obesity are merely an example, and the number of reference models may be determined based on similar medical evidence. (Processing Flow) Figure 5 shows an example of the processing flow of the medical navigation method according to this embodiment. The following steps S102 to S106 correspond to step (A) (selection of the reference model most similar to the patient's body) described in Figure 2, and steps S108 to S110 correspond to step (B) (non-rigid ICP for body surface registration) described in Figure 2.

[0048] In step S102, the RGB-D camera 30 acquires three-dimensional point cloud information of the patient's body 5. The method for acquiring the three-dimensional point cloud information of the patient's body 5 is, for example, as follows. Specifically, the RGB-D camera 30 moves to a predetermined initial position. The RGB-D camera 30 acquires three-dimensional point cloud information (acquired three-dimensional point cloud information) at each imaging location while moving within the imaging area (in this embodiment, the entire chest of the subject 50). At this time, the RGB-D camera 30 also acquires two-dimensional color image information at each imaging location and position and angle information (camera position coordinate information) of the RGB-D camera 30 at each imaging position. This process is repeated until the entire imaging area has been moved. The above process is executed by controlling the robot arm 10 and the RGB-D camera 30 using control signals from the client computer device 60. The body surface shape of the subject 50 is reconstructed using the acquired three-dimensional point cloud information and the camera position coordinate information acquired at the multiple imaging positions. Specifically, for example, the acquired multiple pieces of acquired 3D point cloud information are converted from local coordinates to global coordinates based on the position and angle information (position coordinate information) of the RGB-D camera 30 at each imaging position. Correspondences between overlapping points in each piece of acquired 3D point cloud information are searched for. The acquired multiple pieces of acquired 3D point cloud information are integrated into one piece of 3D point cloud information based on the correspondences between the searched overlapping points. The acquired 3D point cloud information is stored in a storage area, such as a hard disk drive, of the client computer device 60 as data for the patient body model 50.

[0049] In step S104, the client computer device 60 calculates the chamfer distance between the N-body reference model 52-k, which has been stored in advance on a hard disk drive or the like of the client computer device 60, and the body model of the patient 5 acquired in step S102. In step S106, the client computer device 60 determines the reference model with the smallest chamfer distance as the reference model 52.

[0050] In step S108, the client computer device 60 applies registration using a non-rigid ICP to the reference model 52, targeting the patient body model 50. In step S110, the client computer device 60 converts the reference position 55, which is the position (position where the stethoscope should be placed) of the diagnostic site (aortic valve, pulmonary valve, tricuspid valve, mitral valve) previously stored in the medical navigation system, into a position on the patient body model 50 using a coordinate transformation formula for the 3D point cloud between the patient body model 50 and the reference model 52 obtained by registration, and determines this as the estimated position 55 of the auscultation position. The determined estimated position is displayed on, for example, a remote computer device operated by a doctor and connected to the client computer device 60 via a network. The doctor can then operate the medical robot device 1 (robot arm 10) while visually checking the displayed estimated position. Alternatively, the estimated position determined in step S110 is displayed on the display of the client computer device 60, allowing the patient or a patient's close relative to place the stethoscope at the displayed estimated position. Furthermore, the medical robot 1 can automatically place the stethoscope 40 at the estimated position determined in step S110 to obtain auscultation data of the patient 5 and provide it to a clinical expert such as a doctor.

[0051] Although one embodiment of the present invention has been described above, it goes without saying that the present invention is not limited to the above embodiment and may be embodied in various different forms within the scope of the technical concept thereof.

[0052] For example, in the above-described embodiment, a stethoscope is used as a medical instrument, but this is not limiting. The present invention can be used in various situations, such as robotic diagnosis and treatment navigation, or preoperative planning, which require the placement of other medical equipment on the patient's body 5. For example, in an ultrasound examination using an ultrasound probe by an autonomous robot, the medical navigation method and medical navigation system according to this embodiment can be used to estimate the scanning path or scanning area on the body surface of the patient 5 that should be scanned by the ultrasound probe, taking into account individual differences in body shape.

[0053] In the above-described embodiment, the nipples and navel are used as landmarks on the body surface of the subject 5, but any externally conspicuous part of the human body can be used as a landmark. For example, it is assumed that the shoulders, collarbones, or pelvis can be used as landmarks.

[0054] The scope of the present invention is not limited to the exemplary embodiments shown and described, but includes all embodiments that achieve equivalent effects to those intended by the present invention. Furthermore, the scope of the present invention is not limited to the combination of inventive features defined by each claim, but may be defined by any desired combination of specific features from among all the respective disclosed features.

[0055] The following configurations also fall within the technical scope of the present invention: (1) A medical navigation method executed by a computer system for positioning a medical instrument on a subject, wherein the computer system holds three-dimensional point cloud information of the subject as body model data of the subject, and holds N pieces of reference model data representing the shape of a human body and placement data representing placement positions of the medical instrument in each piece of reference model data, the medical navigation method comprising the steps of: determining, as reference model data, one of the N pieces of reference model data that is most similar to the body model data of the subject; applying registration to the reference model data using a non-rigid ICP (interactive closest point) algorithm, with the body model data of the subject as a target; and converting the placement data in the reference model data into placement data of the medical instrument in the body model data of the subject, using a coordinate transformation formula for the three-dimensional point cloud information between the body model data of the subject obtained by the registration and the reference model data. (2) The medical navigation method of (1) above, wherein the step of determining, as the reference model data, the reference model data that has the highest similarity to the body model data of the subject, calculates a chamfer distance between each of the N reference model data and the body model data of the subject, and determines, as the reference model data, the reference model data with the smallest chamfer distance. (3) The medical navigation method of (1) or (2) above, wherein the three-dimensional point cloud information of the subject is acquired by imaging the subject from a plurality of imaging positions. (4) The medical navigation method of any of (1) to (3) above, further comprising the step of outputting the position data of the medical instrument in the body model data of the subject to a remote computer device.(5) A medical navigation system for positioning a medical instrument on a subject, comprising: storing three-dimensional point cloud information of the subject as body model data of the subject; storing N pieces of reference model data representing the shape of a human body and placement data representing the placement position of the medical instrument in each reference model data; determining, among the N pieces of reference model data, the reference model data that is most similar to the body model data of the subject as reference model data; applying registration to the reference model data using a non-rigid ICP (interactive closest point) algorithm with the body model data of the subject as a target; and converting the placement data in the reference model data into placement data of the medical instrument in the body model data of the subject using a coordinate transformation formula for the three-dimensional point cloud information between the body model data of the subject obtained by the registration and the reference model data. (6) The medical navigation system according to (5) above, wherein the medical navigation system includes an RGB-D camera, and the RGB-D camera acquires the three-dimensional point cloud information of the subject by capturing images of the subject from a plurality of imaging positions. (7) A computer program causing a computer system to execute any one of the medical navigation methods of (1) to (4) above. (8) A computer-readable recording medium storing a computer program causing a computer system to execute any one of the medical navigation methods of (1) to (4) above.

[0056] DESCRIPTION OF SYMBOLS 1... Medical robot device (medical navigation system) 5... Patient (subject) 10... Robot arm 20... Constant load passive scanning mechanism (end effector) 25... Force / torque sensor 30... LiDAR camera 40... Medical instrument (stethoscope) 50... Patient body model 55... Reference position 56... Estimated position 60... Client computer device

Claims

1. 1. A medical navigation method executed by a computer system for placing a medical instrument in a subject, comprising: the computer system holds three-dimensional point cloud information of the subject as body model data of the subject, holds N pieces of reference model data (N is an integer of 2 or more) indicating a shape of a human body, and position data indicating a position of the medical instrument in each reference model data; The medical navigation method includes: determining, from among the N pieces of reference model data, the reference model data having the highest similarity to the subject's body model data as a reference model data; applying a registration to the reference model data using a non-rigid interactive closest point (ICP) algorithm with the subject's body model data as a target; converting the arrangement data in the reference model data into arrangement data of the medical instrument in the body model data of the subject, using a coordinate conversion equation of the three-dimensional point cloud information between the body model data of the subject obtained by the registration and the reference model data; Including, The step of determining the reference model data most similar to the subject's body model data as the reference model data includes calculating a chamfer distance between the subject's body model data and each of the N pieces of reference model data to which registration using an ICP algorithm targeting the subject's body model data has been applied, and determining the reference model data with the smallest chamfer distance as the reference model data.

2. The medical navigation method according to claim 1 , wherein the three-dimensional point cloud information of the subject is acquired by imaging the subject from a plurality of imaging positions.

3. The medical navigation method according to claim 1 , further comprising the step of outputting the position data of the medical instrument in the body model data of the subject to a remote computing device.

4. The medical navigation method according to claim 1 , wherein the medical instrument is a stethoscope or an ultrasound probe.

5. 1. A medical navigation system for positioning a medical instrument on a subject, comprising: storing three-dimensional point cloud information of the subject as body model data of the subject, and storing N pieces of reference model data indicating a shape of a human body and arrangement data indicating an arrangement position of the medical instrument in each of the reference model data; Among the N pieces of reference model data, the reference model data having the highest similarity to the subject's body model data is determined as a reference model data; Applying registration to the reference model data using a non-rigid ICP (interactive closest point) algorithm with the subject's body model data as a target; converting the arrangement data in the reference model data into arrangement data of the medical instrument in the body model data of the subject using a coordinate conversion equation of the three-dimensional point cloud information between the body model data of the subject obtained by the registration and the reference model data; When determining the reference model data having the highest similarity to the subject's body model data as the reference model data, a chamfer distance between each of the N reference model data to which registration using an ICP algorithm targeting the subject's body model data is applied and the subject's body model data is calculated, and the reference model data having the smallest chamfer distance is determined as the reference model data. Medical navigation system.

6. the medical navigation system includes a robot arm and an RGB-D camera attached to a tip of the robot arm; The medical navigation system of claim 5, wherein the robot arm moves and the RGB-D camera images the subject from multiple imaging positions to obtain multiple acquired three-dimensional point cloud information that is multiple three-dimensional point cloud information of the subject, and the three-dimensional point cloud information of the subject is obtained based on the correspondence between overlapping points in the multiple acquired three-dimensional point cloud information.

7. A computer program product for causing a computer system to execute the medical navigation method according to any one of claims 1 to 4.