Medical support methods, medical support robotic devices, and computer programs

Abstract

Provided is a medical support method capable of autonomously bringing a medical tool such as an ultrasonic probe or a stethoscope into contact with a proper position on a specimen.　This medical support method is executed by a medical support robot device carrying out a medical practice on a specimen using a medical tool. The medical support method comprises: (1) a step for imaging a specimen from a plurality of imaging positions and thereby acquiring pieces of information pertaining to each imaging position including three-dimensional point group acquisition information on the specimen, two-dimensional image information, and coordinate information on the imaging position; (2) a step for generating one piece of integrated three-dimensional point group information on the specimen using a plurality of pieces of the three-dimensional point group acquisition information and a plurality of pieces of the coordinate information on the imaging positions; (3) a step for determining the position of a specific portion on the specimen set in advance in a plurality of pieces of the two-dimensional image information; and (4) a step for estimating, using statistical anatomic information and the position of the specific portion on the specimen, the position in the integrated three-dimensional point group information on a diagnosed portion on the specimen subjected to a medical practice.

Description

[Technical Field]

[0001] The present invention relates to a medical support method performed by a medical support robot device. [Background technology]

[0002] In the medical field, doctors perform medical procedures such as examinations on patients' bodies (specimens) as needed. However, many developed countries are experiencing declining birth rates and an aging population, leading to a decrease in the proportion of doctors in the working-age population. Under these circumstances, it is expected that maintaining the current quality of medical services will become difficult in the near future. On the other hand, research and development in the field of robotics has advanced remarkably, and robotic devices are being used in various fields in recent years.

[0003] Therefore, in the medical field, relatively safe procedures are increasingly being entrusted to robotic devices. For example, Patent Document 1 discloses technology for improving the efficiency of ultrasonic probe operation using a robotic device. Patent Document 2 also discloses technology for flexibly bringing medical instruments into contact with specimens without using force sensors. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-157058 [Patent Document 2] Japanese Patent Publication No. 2014-100377 [Overview of the project] [Problems that the invention aims to solve]

[0005] However, Patent Documents 1 and 2 only disclose techniques related to the operation of ultrasonic probes and the like by a robotic device. They do not disclose a method for a robotic device to autonomously determine where on a specimen the medical instrument, such as an ultrasonic probe, should be applied when performing an ultrasound diagnosis on the specimen.

[0006] In other words, while conventional technologies have partially automated the scanning of medical instruments such as ultrasound probes using robotic devices, no useful proposals have been made for the comprehensive automation of scanning, from when the medical instrument is separated from the specimen to when the robotic device determines the scanning position on the body surface and moves the medical instrument on the body surface.

[0007] In view of the above problems, the present invention aims to provide a medical support robot device that can autonomously bring medical instruments such as ultrasound probes and stethoscopes into contact with the appropriate position on a specimen. [Means for solving the problem]

[0008] To solve the above problems, one aspect of the present invention is a medical support method performed by a medical support robot device that performs medical procedures on a specimen using medical instruments, comprising the steps of: acquiring acquired 3D point cloud information, which is 3D point cloud information of the specimen, 2D image information, and coordinate information of the imaging position, by imaging the specimen from a plurality of imaging positions; generating integrated 3D point cloud information, which is one 3D point cloud information of the specimen, using the plurality of acquired 3D point cloud information and the plurality of coordinate information of the imaging positions; determining the position of a predetermined specific part on the specimen in the plurality of 2D image information; and estimating the position of the diagnostic site on the specimen in the integrated 3D point cloud information, which is the medical procedure to be performed, using anatomical statistical information and the position of the specific part on the specimen.

[0009] Another aspect of the present invention is a medical support robot device that performs medical procedures on a specimen using medical instruments, the medical support robot device comprising an imaging sensor and a computer device, wherein the imaging sensor acquires acquired 3D point cloud information, which is 3D point cloud information of the specimen, 2D image information, and coordinate information of the imaging position by imaging the specimen from a plurality of imaging positions, the computer device generates integrated 3D point cloud information, which is one 3D point cloud information of the specimen, using the plurality of acquired 3D point cloud information and the plurality of coordinate information of the imaging positions, determines the position of a predetermined specific part on the specimen in the plurality of 2D image information, and estimates the position of the diagnostic site on the specimen in the one integrated 3D point cloud information using anatomical statistical information and the position of the specific part on the specimen, the medical support robot device.

[0010] Another aspect of the present invention is a computer program that causes a medical support robot device to perform any of the above-described medical support methods. [Brief explanation of the drawing]

[0011] [Figure 1] This figure shows an example of the configuration of a medical support robot device according to one embodiment of the present invention. [Figure 2] This figure illustrates coordinate transformation in a medical support robot device according to one embodiment of the present invention. [Figure 3] This figure shows an example of a processing flow for a medical support robot device according to one embodiment of the present invention to autonomously perform auscultation on a specimen. [Figure 4] This figure shows an example of an anatomical map illustrating the positional location of each part of a specimen on the body surface in two-dimensional space. [Figure 5] This figure shows an example of a procedure for estimating the placement position of a medical support robot device according to one embodiment of the present invention. [Figure 6] This figure shows an example of the configuration of the end effector of a medical support robot device according to one embodiment of the present invention. [Figure 7] It is a diagram showing an example of a flowchart of a process for reconstructing the body surface shape in a medical support robot device according to an embodiment of the present invention. [Figure 8] It is a diagram showing an example of a flowchart of a process for reconstructing the body surface shape in a medical support robot device according to an embodiment of the present invention. [Figure 9] It is a diagram showing an example of a flowchart of a process for estimating the placement position in a medical support robot device according to an embodiment of the present invention. [Figure 10] It is a diagram showing an example of a flowchart of a process for estimating the placement position in a medical support robot device according to an embodiment of the present invention. [Figure 11] It is a diagram showing an example of a flowchart of a constant load passive scanning in a medical support robot device according to an embodiment of the present invention. [Figure 12A] It is a diagram showing the position of the target on the mannequin in a multi-way registration experiment. [Figure 12B] It is a diagram showing the positions of the target on the mannequin and four diagnostic sites in a multi-way registration experiment. [Figure 13] It is a diagram showing the results of multi-way registration with or without position feedback of the LiDAR camera. <000009​​​​​​​​​​​​​​​​​​​​(Configuration of a medical support robot system) Figure 1 shows an example of the configuration of a medical support robot device according to this embodiment. The medical support robot device 1 according to this embodiment comprises a robot arm 10, a constant-load passive scanning mechanism (end effector) 20, and an RGB-D camera 30. The robot arm 10 is configured to move around a specimen 50, which is the target of medical procedures, such as the patient's body (Figure 1 is an illustrative diagram, and the specimen 50 is substituted with a mannequin here). The tip of the robot arm 10 is also equipped with a constant-load passive scanning mechanism (end effector) 20, which can be attached integrally or detachably. The constant-load passive scanning mechanism 20 is a mechanism that holds a medical instrument 40 and enables the medical instrument 40 to be pressed against the surface of the specimen 50 while maintaining a constant load and moving it. Here, medical instruments can include, for example, ultrasound probes, stethoscopes, and all other instruments necessary for medical procedures performed on the specimen 50. More specifically, an example of a medical instrument usable in the medical support robot device 1 of this embodiment is an electronic stethoscope (digital stethoscope) such as the JPES-01 electronic stethoscope (digital stethoscope) manufactured by Mitrika Co., Ltd., disclosed at https: / / www.milas.co.jp / product_stethoscope.html. The sound collected when the electronic stethoscope comes into contact with the specimen 50 can be transmitted as data to a computer device (such as the client computer device 60 described later). Similarly, other medical instruments only need to be able to transmit sound and video data acquired when the medical instrument comes into contact with the specimen 50 to a computer device. In addition, the robot arm 10 can be equipped with an RGB-D camera 30 integrally or detachably at its tip. The RGB-D camera 30 is an example of an imaging sensor (distance measuring sensor) that can simultaneously acquire a color image (RGB information) and a distance image (distance information, depth information). An image sensor comprises a light emitter that outputs laser light such as ultraviolet light, visible light, or infrared light, and a light receiver that receives the reflected light when the laser light is reflected by the object being measured. Distance measurement methods in image sensors include, for example, the FoT (Time of Flight) method and the pattern irradiation method.The FoT method is a distance measurement method that measures the distance from the image sensor to the object by the time it takes for the laser beam to be emitted from the emitter, reflected from the object, and received by the receiver. The pattern irradiation method is a distance measurement method that irradiates the object with a laser beam having a specific pattern, and measures the distance from the image sensor to the object by the distortion of the pattern of the reflected light from the object. In this embodiment, the image sensor is a LiDAR (Laser Imaging Detection and Ranging) camera. However, this is merely an example, and image sensors using other methods may be used.

[0013] Furthermore, the medical support robot device 1 may include a client computer device 60. The client computer device 60 is directly or indirectly connected to each of the medical support robot device 1's robot arm 10, constant-load passive scanning mechanism (end effector) 20, and RGB-D camera 30, and transmits and receives data to and from these components. For example, the client computer device 60 transmits control signals to control each component and transmits data necessary to operate each component. The client computer device 60 also receives data acquired or generated by each component from each component. The client computer device 60 can be implemented with a hardware configuration similar to that of a general computer device. The client computer device 60 may include, for example, a processor that can be implemented by electronic circuits such as a CPU (Central Processing Unit) or a microprocessor, RAM (Random Access Memory), ROM (Read Only Memory), an internal hard disk drive, an external hard disk drive, removable memory such as a CD, DVD, USB memory, memory stick, or SD card, an input / output user interface (display, keyboard, mouse, touch panel, speaker, microphone, LED, etc.), and wired / wireless communication interfaces that can communicate with the various components of the medical support robot device 1 and other computer devices. For example, the processor of the client computer device 60 can read computer programs pre-stored in the hard disk drive, ROM, or removable memory into a memory area such as RAM, and execute them while appropriately reading necessary data from the hard disk drive, ROM, removable memory, etc. Through such operation of the client computer device 60, various processes in the medical support robot device 1, which are detailed below, are realized. Furthermore, the various types of data used in each process described in this embodiment are stored in a storage device or storage medium such as a hard disk drive, RAM, or removable memory, and are read into a memory area such as RAM and used as needed when the processor executes a computer program.

[0014] Furthermore, the program executed in the client computer device 60 according to this embodiment can be partially or entirely stored and delivered on a computer-readable medium, or downloaded via a wired or wireless communication network.

[0015] Furthermore, the medical support robot device 1 shown in Figure 1 is a medical support robot device actually constructed by the inventors of this application. It uses a 6-axis collaborative 6-DOF robot arm (UR5e, Universal Robot, Denmark) as the robot arm 10, a LiDAR camera (Intel RealSense L515, Intel, USA) as the RGB-D camera 30 to acquire the three-dimensional contour of the surface of the specimen 50 as point cloud data, and a client computer device 60 (Dell Precision 5380, Dell, USA) to acquire point cloud data while synchronizing the robot arm 10. In addition, the medical support robot device 1 in this example was constructed assuming the use of a stethoscope for auscultation. Note that since Figure 1 is an experimentally constructed medical support robot device, the specimen 50 is a mannequin here, and as will be described later, the nipple and navel, which were used as landmarks for the specimen 50 in this example, are unclear on the mannequin, so the inventors placed markers at these positions. Furthermore, the configuration shown in Figure 1 is merely an example, and it goes without saying that the configuration of the medical support robot device 1 according to this embodiment is not limited thereto. The same applies throughout this specification.

[0016] Furthermore, from the standpoint of ensuring the safety of diagnosing specimen 50, a constant-load passive scanning mechanism using a spring was implemented on the end effector 20 of the robot arm 10 to adaptively grasp the stethoscope (medical instrument) 40 against the body surface of specimen 50. A 6-axis force / torque sensor 25 (Axia-80-M20, ATI Industrial Automation, USA) was attached to the base of the constant-load passive scanning mechanism (end effector) 20 to measure the contact force when the stethoscope 40 was placed on the body surface of specimen 50. In addition, the client computer device 60 that controls the medical support robot device 1 and an external server computer device (not shown) were directly connected to a network (speed 1 GB / sec (gigabytes per second)), and the data transmission protocol was TCP / IP. In this embodiment, the LiDAR camera 30 is provided within the constant-load passive scanning mechanism (end effector) 20, but is not limited to this. The LiDAR camera 30 only needs to be fixed so that its relative position to the robot arm 10 does not change, and it may be installed in a different location from the constant-load passive scanning mechanism (end effector) 20.

[0017] The LiDAR camera 30 acquires depth and color information of the sample 50 as point cloud data, and this point cloud data is used for coordinate registration (alignment) using the positional relationship between the robot arm 10 and the sample 50. Since the LiDAR camera 30 is attached to the end effector 20 of the robot arm 10, the positional relationship between the LiDAR camera 30 and the robot arm 10 remains kinematically fixed even when the robot arm 10 moves around the sample 50. Therefore, the position of the acquired point cloud data of the sample 50 is linked to the coordinates of the robot arm 10. Figure 2 is a diagram illustrating the coordinate transformation in the medical support robot device 1 of this example, which consists of the robot arm 10, the LiDAR camera 30, and the stethoscope (medical instrument) 40. The coordinate space of the LiDAR camera 30 (P L The scanning point in ) is in the coordinate space (P) of the base 15 of the robot arm 10. B ) is converted to the coordinate space (P) of the stethoscope 40. S The contact point with the sample 50 in the coordinate space of the LiDAR camera 30 (PL It can be converted to (Equation (1)).

[0018]

number

[0019]

number

[0020]

number

[0021]

number

[0022]

number

[0023]

number

[0024]

number

[0025] The robotic arm 10 can be controlled by URScript (Universal Robots A / S), a programming language used in the medical support robot device 1. The client computer device 60 that controls the medical support robot device 1 can send URScript commands to an external server computer device (not shown) via socket communication. Point cloud data was acquired using the Intel RealSense SDK 2.0. A software system customized based on Python programming in Visual Studio Code can synchronize the control of the robotic arm 10 with the reading of the point cloud data.

[0026] Figure 3 shows an example of a processing flow for a medical support robot device 1 to autonomously perform auscultation by bringing a stethoscope (medical instrument) 40 into contact with the body surface of a specimen 50. This processing flow mainly includes the following components (A) to (C): (A) Reconstruction of body surface shape: Acquisition of point cloud data covering the entire chest of sample 50 (process (1)), and registration (alignment) of the acquired point cloud data to reconstruct the shape of the entire chest (process (2)). (B) Estimation of placement position: Estimation of the placement position of the stethoscope 40 based on the reconstructed body shape and anatomical structure and landmarks on the body surface of specimen 50 (process (3)) (process (4)). (C) Constant load passive scanning: The stethoscope 40 is placed at the position estimated in (B) above while maintaining a constant contact force.

[0027] The following explains these three components (A), (B), and (C). (A) Reconstruction of body surface shape (registration of point cloud data) To accurately reconstruct the entire three-dimensional shape of sample 50, the position of the LiDAR camera 30 is changed to acquire several point cloud datasets, and registration (alignment) is performed. This process also deals with noisy and partially overlapping data. A common approach to this is to combine sampling-based coarse alignment, such as ICP (iterative closest point) (see, e.g., F. Pomerleau, F. Colas, R. Siegwart, and S. Magnenat, “Comparing ICP variants on real-world data sets: Open-source library and experimental protocol,” Auton. Robots, vol. 34, no. 3, pp. 133-148, 2013.), with iterative local search. As an advanced registration algorithm, pairwise global registration is widely used, which is more than an order of magnitude faster and more robust to noise than general registration pipeline processing. This approach can also be employed for registering multiple surfaces to obtain models of large scenes or objects. This procedure is known as multi-way registration.

[0028] To further improve registration accuracy, it is important to obtain the relative position of each dataset. Assuming that the point cloud data paired with the position data used to capture the point cloud is known, the system coordinates of the captured dataset (local coordinates in the medical support robot device 1; the same applies hereinafter) can be converted to global coordinates, thus minimizing registration errors. An advantage of the medical support robot device developed by the inventors is that the position of the LiDAR camera 30 when each point cloud dataset is captured can be accurately obtained based on encoders implemented in each joint of the robot arm 10.

[0029] Following multiway registration, a collection of captured point cloud data from the body surface of sample 50 {P i Given}, the set of position and orientation information for each point cloud data set in global coordinates.

[0030]

number

[0031]

number

[0032]

Number

[0033]

Number

[0034]

Number

[0035]

Number

[0036]

Number

[0037]

Number

[0038]

number

[0039]

number

[0040] Next, all the poses

[0041]

number

[0042]

number

[0043]

number

[0044]

number

[0045]

number

[0046]

number

[0047]

number

[0048]

number

[0049] Based on multiway registration, the registration pipeline processing in this embodiment is as follows: First, a LiDAR camera 30 attached to the end effector 20 of the robot arm 10 moves to several points away from the sample 50 to capture (photograph) the surface of the sample 50. Next, each coordinate of the captured dataset is transformed into a global coordinate system based on the robot arm 10. Multiway registration is performed using the point cloud dataset represented in global coordinates. Each dataset is then combined into a single dataset using the transformations estimated for each dataset. Finally, filters are applied to the combined data to remove inlier and outlier point clouds.

[0050] (B) Estimation of placement location Auscultation is primarily performed to examine the circulatory and respiratory systems, namely heart sounds and breath sounds. In this embodiment, the examination of the circulatory system will be described as an example. In the examination of the circulatory system, the stethoscope is mainly placed in contact with four locations on the specimen 50 to listen to the sounds of the tricuspid valve, mitral valve, pulmonary valve, and aortic valve. As shown in Figure 4, the sounds of the tricuspid valve, mitral valve, pulmonary valve, and aortic valve can be heard roughly at the left side of the lower sternum near the fifth intercostal space, above the apex of the heart in the fifth intercostal space to the left of the midclavicular line (approximately 10 cm from the midline), above the medial end of the left second intercostal space, and above the medial end of the right second intercostal space, respectively. Therefore, in order for the medical support robot device 1 to autonomously auscultate these four locations, it must recognize each of these positions.

[0051] In this embodiment, the positions of the nipple and umbilicus of specimen 50 are used to estimate each position on the body surface. The nipple and umbilicus are readily recognizable landmarks on the body surface for the medical support robot device 1. The nipple is roughly located in the fourth intercostal space (Figure 4). Its position can serve as a landmark for finding each placement position based on the anatomical relationship between each rib and its intercostal space. For example, in the literature JA Cook, SS Tholpady, A. Momeni, and MW Chu, “Predictors of internal mammary vessel diameter: A computed tomographic angiography-assisted anatomic analysis,” J. Plast. Reconstr. Aesthetic Surg., vol. 69, no. 10, pp. 1340-1348, 2016, the height of the intercostal spaces and ribs is measured using CT scan data. Based on this anatomical statistical data, the craniocaudal length between the fourth intercostal space (nipple position) and the second intercostal space (position of the aortic valve and pulmonary valve) was 39 mm, and the craniocaudal length between the fourth intercostal space (nipple position) and the fifth intercostal space (position of the mitral valve and tricuspid valve) was an average of 20.6 mm. These lengths can be used as a reference for the height of each valve on the y-axis (crown-caudal axis) of the system coordinates (local coordinates in the medical support robot device 1) of the medical support robot device 1. In addition, the position of the abdominal midline is required as a reference for estimating the width of each valve on the x-axis (left-right axis) of the system coordinates of the medical support robot device 1. The midline can be identified by drawing a straight line connecting the midpoint of the line segment connecting the left and right nipples to the navel (Figure 5). The midclavicular line, where the mitral valve is located, is approximately 100 mm away from the midline along the x-axis (see, for example, RA Sayeed and GE Darling, “Surface Anatomy and Surface Landmarks for Thoracic Surgery,” Thorac. Surg. Clin., vol. 17, no. 4, pp. 449-461, 2007).Since the other valves are located near the medial end of the intercostal space, the stethoscope should be placed around the edge of the sternum. The average width of the male sternum is 25.99 mm (see, for example, JS Bruce, “Utility of the sternum to estimate sex and age,” Bost. Univ. Theses Diss., pp. 1-93, 2014, [Online]. Available: https: / / hdl.handle.net / 2144 / 15320). Next, we assume that the positions of the aortic and pulmonary valves along the left-right axis are ±13 mm from the midline. In this way, an anatomical map showing the placement of each valve on the body surface in two-dimensional space can be created based on the reference positions of each valve, as shown in Figure 4. Note that the positions of each examination site may vary slightly depending on the size of the specimen 50, etc. Therefore, the positions of each examination site described above may have some leeway. Furthermore, for example, the measured distance between the left and right nipples, which are landmarks on the body surface of specimen 50, may be compared with a statistically average value, and the lengths used to identify each organ and each examination location described above may be enlarged or reduced according to the difference between the measured value and the average value, and used for estimating the examination location described later.

[0052] As described above, in this embodiment, in order to estimate the placement positions for the stethoscope, it is necessary to identify the positions of the nipples and umbilicus from the image information of the appearance of the specimen 50 captured by the LiDAR camera 30. By combining the identified positions of the nipples and umbilicus with an anatomical map of the body surface, the placement positions of the stethoscope 40 for listening to the sounds of the four valves can be estimated. The overall procedure for estimating the placement positions is as follows (Figure 5): (i) Using a template matching method, the positions of the nipples and umbilicus are extracted as two-dimensional pixel positions from each of the multiple color images captured by the LiDAR camera 30. (ii) Using the extracted positions of the nipples and umbilicus, the position of the midline of the abdomen is identified by drawing a straight line connecting the midpoint of the line segment connecting the left and right nipples to the umbilicus. (iii) Based on the identified midline position, the positions of each diagnostic site in the two-dimensional pixel space of the color image can be estimated based on the anatomical map described above (Figure 4). (iv) The extracted two-dimensional pixel positions in the color image are projected onto the reconstructed three-dimensional coordinate space of the body surface by a transformation based on unique camera parameters.

[0053] While the above described examinations of the circulatory system, respiratory examinations typically involve listening to the upper, middle, and lower lobes of the left and right lungs, as well as the trachea, using a stethoscope from the anterior chest and back. The locations for these examinations can be determined using the same methods as described above. However, the locations for each examination can be more approximate compared to when listening to heart sounds.

[0054] (C) Constant load passive scanning The inventors of this application have developed a novel end effector 20 having a spring-driven passive scanning mechanism for safely installing a stethoscope (medical instrument) 40. Figure 6 shows an example of the configuration of the developed end effector 20. The role of this passive scanning mechanism is to ensure that the stethoscope 40 is in contact with the body surface of the specimen 50 with a safe and constant contact force, regardless of the pushing displacement by the robot arm 10 when placing the stethoscope 40 at the estimated placement position of the specimen 50. The safe and constant contact force may be a predetermined constant value, based on the general pressing force values ​​used by doctors when applying a stethoscope to a patient's body. Here, a certain error may occur between the actual position of the stethoscope 40 and its placement position on the body surface, for example, due to the displacement of the body surface of the specimen 50 caused by the patient's respiratory movements. The end effector 20 according to this embodiment can correct this error while maintaining the contact force within a certain range.

[0055] The end effector 20 shown in Figure 6 comprises a linear servo actuator 202 (L1220PT, MigthyZap, South Korea), a linear spring 204, an optical distance sensor 206 (ZX-LD100L, Omron, Japan), and a linear guide 208 (SSE2B6-70, Misumi, Japan). The linear servo actuator 202 moves vertically up and down relative to the body surface of the specimen 50 while maintaining the compression amount of the linear spring 204 at a predetermined constant amount. In this embodiment, the linear spring constant is 0.45 N / mm (Newtons / millimeter), and two springs are inserted. The optical distance sensor 206 also measured the compression amount of the linear spring 204 in real time. The linear servo actuator 202 was controlled via an Arduino-based PID control controller 62 (IR-STS01, MigthyZap, South Korea). The values ​​measured by the optical distance sensor 206 were transferred to the client computer device 60 via a DAQ (Data Acquisition) tool (Analog Discovery 2, Digilent, USA). The software system of the client computer device 60, which was customized based on Python programming in Visual Studio Code, synchronized the control of the linear servo actuator 202 with the reading process of the measured value data from the optical distance sensor 206. The position of the linear servo actuator 202 was controlled based on feedback from the optical distance sensor 206 using a PID (Proportional Integral Differential) control scheme. More specifically, as shown in Figure 6, the distance from the optical distance sensor 206 to the body surface of the specimen 50 was measured by the optical distance sensor 206, and the measured value was output to the PID control controller 62. By comparing this measured value with the target value (the distance between the optical distance sensor 206 and the body surface of the specimen 50 that should be maintained), the amount of compression of the linear spring 204 was determined, taking into account the displacement of the body surface of the specimen 50 caused by the patient's respiratory movements, etc., and this was output as a command value to the linear servo actuator 202. The linear servo actuator 202 changes the compression state of the linear spring 204 based on the command value.

[0056] (flowchart) Below, an example of the processing performed by the medical support robot device 1 according to this embodiment will be explained using the flowcharts in Figures 7 to 11.

[0057] Figure 7 shows an example flowchart of the process (Figure 3(1)) in which the LiDAR camera 30 takes images of the specimen 50. First, the LiDAR camera 30 moves to a predetermined initial position (step S102). While moving within the imaging area (in this embodiment, the entire chest of the specimen 50) (step S104), the LiDAR camera 30 acquires 3D point cloud information (acquired 3D point cloud information) at each imaging location (step S106). At this time, the LiDAR camera 30 also acquires 2D color image information at each imaging location and position and angle information of the LiDAR camera 30 at each imaging location (camera position coordinate information) (steps S108, S110). The above process continues until the entire imaging area has been moved (step S112). Note that the above process can be executed by controlling the robot arm 10 and the LiDAR camera 30 with control signals from the client computer device 60. (The same applies to each process described below.)

[0058] Figure 8 shows an example flowchart of the process (Figure 3(2)) for reconstructing the surface shape of the specimen 50 using multiple acquired 3D point cloud information and multiple camera position coordinate information obtained by the process in Figure 7. The acquired 3D point cloud information obtained from N locations (N is an integer of 2 or more) is converted from local coordinates to global coordinates based on the position and angle information (position coordinate information) of the LiDAR camera 30 at each imaging location (step S202). The correspondence between overlapping points in each acquired 3D point cloud information is searched (step S204). The N acquired 3D point cloud information are integrated into one 3D point cloud information (integrated 3D point cloud information) based on the correspondence between overlapping points searched in step S204 (step S206).

[0059] Figure 9 shows an example flowchart of the process for extracting landmarks on the body surface of sample 50 (Figure 3 (3)). The nipples and umbilicus of sample 50 are extracted from each of the image information acquired from N locations using a template matching method or the like (step S302). The pixel position information of the extracted nipples and umbilicus is converted into coordinates on the point cloud coordinate system (step S304). The position of the abdominal midline is identified from the position information (coordinates) of the nipples and umbilicus on the point cloud coordinate system and the point cloud distribution of the entire sample (step S306).

[0060] Figure 10 shows an example of a flowchart for the process of estimating the placement position of the stethoscope 40 (Figure 3 (4)). First, the diagnostic site is specified (step S402). In this embodiment, the sites to be auscultated are the tricuspid valve, mitral valve, pulmonary valve, and aortic valve. These specifications may be entered by, for example, a user (operator, etc.) of the medical support robot device 1 using an input / output user interface such as a keyboard on the client computer device 60. This input may be stored in a hard disk drive or RAM, and read from the hard disk drive or RAM when step S402 is executed. Next, the placement position near the diagnostic site is determined based on anatomical statistical information of the three-dimensional position of the organs (step S404). The position information of the nipple, navel, and abdominal midline on the point cloud coordinates is converted to a placement position on the point cloud coordinates for the diagnostic site (step S406). The medical support robot device 1 moves based on the placement position on the point cloud coordinates of the diagnostic site converted in step S406 (step S408). This is followed by step S508 in Figure 11.

[0061] Figure 11 shows an example of a flowchart for constant-load passive scanning (Figure 3(C)). First, the medical support robot device 1 performs calibration of each component (step S502). Also, the load (target load) to be applied when the stethoscope (medical instrument) 40 is brought into contact with the body surface of the specimen 50 is set (step S504). This setting may be entered by, for example, the user (operator, etc.) of the medical support robot device 1 using an input / output user interface such as a keyboard on the client computer device 60. This input may then be saved on a hard disk drive or RAM. Next, the client computer device 60 of the medical support robot device 1 calculates the required spring compression amount necessary for the target load set in step S504 (step S506). Before auscultation is started, the medical support robot device 1 performs the above steps S502 to S506 as preparation.

[0062] Following step S408 in Figure 10, based on the position on the point cloud coordinate system of the diagnostic site converted in step S406 in Figure 10, the robot arm 10 of the medical support robot device 1 operates so that the linear servo actuator 202 of the constant-load passive scanning mechanism 20 holding the stethoscope 40 moves to the position on the point cloud coordinate system of the diagnostic site, bringing the stethoscope 40 into contact with the body surface of the specimen 50 and pressing it with a predetermined contact force (step S508). The optical distance sensor 206 measures the length of the linear spring 204 (step S510). If the length of the linear spring 204 measured in step S510 reaches a limit value (step S512: Yes), further processing cannot be continued, so the medical support robot device 1 moves to its initial position and terminates the process (step S518). In this case, for example, processing such as changing the initial value of the contact force of the stethoscope 40 in step S508 may be performed to re-execute the auscultation process.

[0063] Furthermore, if the length of the linear spring 204 measured in step S510 has not reached the limit value (step S512: No), the client computer device 60 of the medical support robot device 1 determines whether the length of the linear spring 204 measured in step S510 has reached the required spring compression amount calculated in step S506 (step S514). If it has not reached the required spring compression amount (step S514: No), the contact force is insufficient, so the process returns to step S508, and the linear servo actuator 202 operates to increase the compression amount of the linear spring 204. Also, if the client computer device 60 of the medical support robot device 1 determines in step S514 that the length of the linear spring 204 has reached the required spring compression amount (step S514: Yes), the stethoscope 40 is used to auscultate the diagnostic site. Furthermore, if, for example, the medical instrument 40 is an ultrasound probe, the ultrasound probe is pressed against a specific area on the surface of the specimen 50 and moved, and the diagnostic images detected by the ultrasound probe are output to the client computer device 60 as they occur. The physician can then view the diagnostic image on the client computer device 60 or a remote computer device and make a diagnosis. The client computer device 60 also determines the amount of compression of the linear spring 204, taking into account the displacement of the body surface of the specimen 50 caused by the patient's respiratory movements, etc., and determines whether the stethoscope 40 has reached the target position accurately (step S516). If the stethoscope 40 has not reached the target position accurately, the system may return to step S508 to adjust the position of the constant-load passive scanning mechanism 20 that holds the stethoscope 40.

[0064] (Example of experiment) To verify the proof of concept of the medical support robot device according to this embodiment, which enables autonomous positioning of the stethoscope 40 based on the external appearance information of the specimen 50 while ensuring patient safety, the inventors conducted three main types of experiments. In all experiments, a white male torso mannequin was used as the specimen 50. Since the mannequin lacked nipples and a navel, markers were placed at their respective locations.

[0065] First, the accuracy of the reconstructed body surface obtained by the RGB-D camera 30, as described in "(A) Reconstruction of Body Surface Shape (Registration of Point Cloud Data)" above, was evaluated. Figure 12A shows the position of the target on the mannequin in the multiway registration experiment. Figure 12B shows the position of the target on the mannequin and the four diagnostic sites in this experiment. For multiway registration, point cloud data was acquired at five positions, namely, -200 mm, -100 mm, 0 mm, 100 mm, and 200 mm along the x-axis, with the midline of the specimen 50 as the origin, as shown in Figure 12A (captured by the LiDAR camera 30). Furthermore, the performance of multiway registration was compared with and without feedback from the position of the LiDAR camera 30 (Equation (1)). Here, feedback refers to the process of transforming each point cloud data to the system coordinates of the medical support robot device 1 (local coordinates in the medical support robot device 1) based on the position information of the LiDAR camera 30 (Equation (1)). Furthermore, since the accuracy of the point cloud depends on the distance between the LiDAR camera 30 and the subject, point cloud data was acquired by changing the distance of the LiDAR camera 30 from the sample 50 in three stages (250 mm, 300 mm, and 350 mm from the upper edge of the chest of the sample 50). To evaluate the accuracy of multiway registration, the robotic arm 10 was moved to four targets on the sample 50, namely the left and right nipples, the epigastric region, and the navel, based on the reconstructed body surface data (Figure 12A), and the distance between the tip of the end effector 20 and each target was measured in three-dimensional coordinate space. The position of each target was manually indicated on the body surface data reconstructed in this experiment.

[0066] Furthermore, to eliminate other factors (e.g., errors due to the assembly of mechanical parts) without using a registration method, a jig was attached to the robot arm 10 to pinpoint the center of the end effector 20 (Figure 12A). Twelve tests were conducted for each condition.

[0067] Next, the accuracy of the method for estimating the placement position of the stethoscope 40, as described in "(B) Estimation of Placement Position" above, was evaluated. Using the reconstructed body surface data (height of LiDAR camera 30: 300 mm) used in the above experiment, the positions of four placement locations, namely the tricuspid valve, mitral valve, pulmonary valve, and aortic valve, were estimated. Markers were placed on the mannequin for each placement location based on the anatomical map shown in Figure 4. The mannequin's position was randomly varied slightly in 12 patterns.

[0068] Furthermore, the safety of the end effector 20 having the passive scanning mechanism described in "(C) Constant Load Passive Scanning" above was evaluated by measuring both static and dynamic contact forces. "Static contact force" refers to the contact force measured when the medical instrument (stethoscope) 40 is in contact with the body surface and stationary. "Dynamic contact force" refers to the contact force generated during the movement from the air to the body surface where the medical instrument (stethoscope) 40 is brought into contact. In this experiment, the static contact force was measured after pressing the stethoscope 40 5 mm against the ground. Three types of contact forces (5N, 10N, 15N) were set. Twelve trials were performed for each condition. The dynamic contact force was measured when the stethoscope 40 moved from the air to the body surface of the mannequin. The initial position of the stethoscope 40 was 5 mm from the body surface of the specimen 50 along the z-axis (the vertical direction of the specimen 50 (mannequin) lying supine). From this position, the stethoscope 40 was moved 10 mm along the z-axis and pressed 5 mm against the body surface of the specimen 50. The target contact force in this experiment was set to 5 N.

[0069] (Experimental results) (1) Reconstruction of body surface shape Figure 13 shows the results of multi-way registration, separated by the presence or absence of positional feedback from the LiDAR camera 30. The upper panel shows the estimation results for each height without positional feedback, and the lower panel shows the estimation results for each height with positional feedback. As shown in Figure 13, at all heights, the reconstruction with positional feedback more accurately represents the chest shape of the sample 50 than the reconstruction without positional feedback from the LiDAR camera 30. Figure 14 shows the results of the registration error, which changes depending on the height of the LiDAR camera 30. As shown in Figure 14, the error decreases as the distance between the LiDAR camera 30 and the target decreases. A two-tailed t-test with a 90% confidence interval was used to determine whether there was a significant difference in accuracy depending on the height of the LiDAR camera 30. There was a significant difference in the error of the three-dimensional spatial coordinates between LiDAR camera 30 heights of 250 mm and 350 mm (p<0.05). (2) Estimation of placement location Figure 15 shows the results of the estimated positioning error for each placement position. The error for the tricuspid valve was smaller compared to the errors for the other valves. A two-tailed t-test with a 90% confidence interval was also used for positioning accuracy dependent on the position of the target valve. There was a significant difference in the error of the three-dimensional spatial coordinates between the tricuspid valve and the other valves (p<0.01). (3) Contact force of constant load passive scanning mechanism Figure 16A shows the results of the static contact force generated by the passive scanning mechanism in this example. Figure 16B shows the results of the dynamic contact force generated by the passive scanning mechanism in this example. The results for the static contact force showed that the generated contact force (vertical axis) was accurately achieved relative to the target force (horizontal axis) under all conditions. Based on the results for the dynamic contact force, the time-series measurements increased (0.5 seconds) when in contact with the body surface of sample 50, slightly exceeding the target value. However, the measurements were immediately adjusted to match the target value. The maximum measured contact force was 5.36 N, which was 7.2% of the target value.

[0070] According to the medical support robot device of this embodiment described above, when performing medical procedures on a specimen using medical instruments, the medical support robot device can automatically determine the position where the medical instruments should be placed in contact, without requiring human intervention. Furthermore, it is possible to move the medical instruments across the body surface while maintaining a predetermined contact force. This function makes it possible to operate medical support robot devices, which conventionally could only be operated semi-automatically, in a fully automatic manner.

[0071] Furthermore, the full automation of medical support robotic devices makes it possible to maintain the quality of medical care even in countries with declining birth rates and an aging population. Conventional auscultation and ultrasound examinations are important tests for detecting abnormal clinical signs in clinical settings. However, conventional auscultation and ultrasound examinations required physical contact between the physician and the patient. Therefore, physicians risked contracting infections through patients, and required physician skill to apply the stethoscope or ultrasound probe with optimal contact force to the specimen. Because the medical support robotic device according to this embodiment is fully automated, telemedicine becomes possible, which is beneficial in terms of protecting physicians from infection, and allows patients to receive high-quality medical services even in areas with limited medical resources.

[0072] Although one embodiment of the present invention has been described so far, it goes without saying that the present invention is not limited to the above-described embodiments and may be implemented in various different forms within the scope of its technical concept.

[0073] For example, in the embodiment described above, the nipple and navel were used as landmarks on the body surface of the specimen 50, but any visually prominent part of the human body would suffice. For example, joints such as the shoulder, clavicle, or pelvis are easily identifiable from their appearance, and it is assumed that each joint can be used as a landmark. It is possible to use known software (for example, the open-source software "OpenPose") to estimate the positional information of each joint of the specimen 50.

[0074] Furthermore, in the above embodiment, the position of the abdominal midline was determined from the positions of the landmarks, the nipple and the navel, and the position of each diagnostic site on the point cloud coordinate system was estimated using the anatomical positional relationship between this abdominal midline and each diagnostic site. However, depending on the positional relationship between the landmark and the diagnostic site, the position of the diagnostic site may be estimated directly from the position of the landmark.

[0075] Furthermore, while the above embodiment described in detail the case of performing auscultation on the specimen 50 using a stethoscope, this is merely one example. For example, the medical instrument may be another medical instrument such as an ultrasound probe used in ultrasound examinations. For example, the present invention is also applicable when a medical instrument such as an ultrasound probe is pressed against a specific area on the surface of the specimen 50 with a predetermined contact force and moved.

[0076] Furthermore, the scope of the present invention is not limited to the illustrative and described exemplary embodiments, but also includes all embodiments that produce effects equivalent to those aimed at by the present invention. Moreover, the scope of the present invention is not limited to the combination of features defined by each claim, but can be defined by any desired combination of specific features from all disclosed features. [Explanation of Symbols]

[0077] 1…Medical support robotic device 10…Robot arm 15...Base 20… Constant load passive scanning mechanism (end effector) 202… Linear servo actuator 206... Distance sensor 206… Optical distance sensor 208... Linear guide 25…Force / torque sensor 30…RGB-D camera (LiDAR camera) 40…Medical instruments (stethoscope) 50... Samples 60…Client computer device 62…PID control controller

Claims

1. A medical support method performed by a medical support robot device that performs medical procedures on a specimen using medical instruments, wherein the medical support robot device comprises a robot arm and an imaging sensor for imaging the specimen, and the robot arm and the imaging sensor are arranged such that their relative positions are fixed. A step of acquiring acquired 3D point cloud information, which is 3D point cloud information of the sample, 2D image information, and coordinate information of the imaging position, by imaging the sample from multiple imaging positions, wherein the coordinates of the multiple acquired 3D point cloud pieces and the coordinates of the multiple 2D image pieces are associated by the imaging sensor. A step of generating integrated 3D point cloud information, which is a single 3D point cloud information representing the surface shape of a part of the specimen, using multiple acquired 3D point cloud information and multiple imaging position coordinate information, A step of determining the position of a specific part on the specimen predetermined in multiple two-dimensional image information, A step of estimating the position of the diagnostic site on the specimen where the medical procedure is performed in the one integrated three-dimensional point cloud information using anatomical statistical information and the position of a specific site on the specimen, Medical support methods, including those mentioned above.

2. The step of determining the location of a specific part on the sample includes extracting the locations of a plurality of specific parts on the sample in each of the plurality of two-dimensional image information by matching processing, The aforementioned estimation step is, From the positions of multiple specific parts on the specimen in each of the extracted multiple two-dimensional image information, the position of the diagnostic site in each of the multiple two-dimensional image information is estimated using the anatomical statistical information that shows the relationship between the positions of multiple specific parts on the specimen and the position of the diagnostic site, The estimated location of the diagnostic site in each of the plurality of two-dimensional image information is projected onto the integrated three-dimensional point cloud information using the coordinate information of the imaging position. A medical support method according to claim 1, including the method described in claim 1.

3. The coordinate information of the multiple imaging positions is the local coordinate information of the medical support robot device. The step of generating the aforementioned one integrated 3D point cloud information is: A step of converting the local coordinate information of each of the multiple acquired 3D point cloud pieces into global coordinates, based on the local coordinate information of the imaging position at the time each of the multiple acquired 3D point cloud pieces was acquired. The steps include: exploring the correspondence between overlapping points in the aforementioned multiple acquired 3D point cloud information; The steps include: integrating the multiple acquired 3D point cloud information into a single unified 3D point cloud information based on the correspondence between the searched overlapping points; A medical support method according to claim 1, including the method described in claim 1.

4. The medical support method according to claim 1, wherein the specific area on the specimen includes any of the left and right nipples of the specimen, the navel, and any of the joints of the specimen.

5. A step of obtaining a target load to be applied when the medical device is brought into contact with the diagnostic site, The steps include bringing the medical device into contact with the estimated location of the diagnostic site using the target load, The medical support method according to claim 1, further comprising:

6. The medical support robot device includes a spring for bringing the medical instrument into contact with the diagnostic site with the target load. The medical support method according to claim 5, wherein the amount of compression of the spring required for the target load is calculated from the target load obtained.

7. The steps include measuring the amount of compression of the spring while the medical instrument is in contact with the diagnostic site, A step of adjusting the amount of compression of the spring while the medical device is in contact with the diagnostic site, using the amount of compression of the spring required for the calculated target load and the amount of compression of the spring measured while the medical device is in contact with the diagnostic site; The medical support method according to claim 6, further comprising:

8. A medical support robot device that performs medical procedures on a specimen using medical instruments, the medical support robot device comprising a robot arm, an imaging sensor, and a computer device, wherein the robot arm and the imaging sensor are arranged such that their relative positions are fixed. The aforementioned imaging sensor is By imaging the sample from multiple imaging positions, the acquired 3D point cloud information, which is the 3D point cloud information of the sample, 2D image information, and coordinate information of the imaging position are acquired at each imaging position, and the coordinates of the multiple acquired 3D point cloud pieces and the coordinates of the multiple 2D image pieces are associated by the imaging sensor. The aforementioned computer device Using the multiple acquired 3D point cloud pieces and the coordinate information of the multiple imaging positions, an integrated 3D point cloud piece is generated, which is a single 3D point cloud piece showing the surface shape of a part of the specimen. The position of a specific part on the specimen, predetermined in multiple two-dimensional image information, Using anatomical statistical information and the location of a specific site on the specimen, the position of the diagnostic site on the specimen where the medical procedure is performed is estimated in the one integrated three-dimensional point cloud information. Medical support robotic device.

9. The coordinate information of the multiple imaging positions is the local coordinate information of the medical support robot device. The generation of the aforementioned single integrated 3D point cloud information is Based on the local coordinate information of the imaging position at the time each of the multiple acquired 3D point cloud pieces was acquired, the local coordinate information of each of the multiple acquired 3D point cloud pieces is converted to global coordinates. In the aforementioned multiple acquired 3D point cloud data, the correspondence between overlapping points is searched. This includes integrating the multiple acquired 3D point cloud information into a single unified 3D point cloud information based on the correspondence between the searched overlapping points. The medical support robot device according to claim 8.

10. The aforementioned computer device Obtain the target load to be applied when the medical device is brought into contact with the diagnostic site. The medical support robot device according to claim 8 or 9, which outputs a control signal to bring the medical instrument into contact with the estimated position of the diagnostic site with the target load.

11. The medical support robot device includes a spring for bringing the medical instrument into contact with the diagnostic site with the target load. The medical support robot device according to claim 10, wherein the computer device calculates the amount of spring compression required for the target load from the acquired target load.

12. The aforementioned computer device The amount of compression of the spring is obtained while the medical instrument is in contact with the diagnostic site. The medical support robot device according to claim 11, wherein the amount of compression of the spring required for the calculated target load and the amount of compression of the spring obtained while the medical device is in contact with the diagnostic site are used to calculate the amount of compression of the spring while the medical device is in contact with the diagnostic site.

13. A computer program that causes a medical support robot device to execute the medical support method described in any one of claims 1 to 7.

Images