Ultrasound probe operation support system and operation support program

Abstract

This reduces the difference in the relative positional relationship between the ultrasound probe and the object being treated before and after treatment on the subject. [Solution] The ultrasonic probe 16a is marked with a probe detection mark 20, and the object to be treated T is marked with an object detection mark 22. The pre-treatment relative relationship calculation unit 48 calculates the pre-treatment relative relationship between the ultrasonic probe 16a and the object to be treated T during pre-treatment ultrasonic transmission and reception, based on a pre-treatment camera image that includes images of the probe detection mark 20 and the object detection mark 22 formed on the object to be treated T before treatment. The current relative relationship calculation unit 50 calculates the current relative relationship between the ultrasonic probe 16a and the object to be treated T, based on a current camera image that includes images of the probe detection mark 20 and the object detection mark 22 formed at the present time. The probe control support unit 52 performs control to bring the current relative relationship closer to the pre-treatment relative relationship.

Description

Technical Field

[0001] This specification discloses improvements to an operation support system and an operation support program for an ultrasonic probe.

Background Art

[0002] An ultrasonic diagnostic apparatus is known that transmits ultrasonic waves from an ultrasonic probe toward a subject, receives reflected waves from the subject in the ultrasonic probe, and performs various processes such as forming an ultrasonic tomographic image representing a cross section in the subject, forming a Doppler image representing the velocity of tissues (such as blood) in the subject, or performing various measurements based on a reception signal formed from the reflected waves.

[0003] Conventionally, in an ultrasonic diagnostic apparatus, in order for an examiner to appropriately perform a desired process, it is necessary to acquire an appropriate reception signal. In order to acquire an appropriate reception signal, it is necessary to properly position and orient the ultrasonic probe. Therefore, various techniques have been proposed to assist in properly positioning and orienting the ultrasonic probe.

[0004] For example, Patent Document 1 discloses an ultrasonic diagnostic apparatus including an ROI mark setting unit that sets a region of interest designated by an examiner on an ultrasonic image, a camera that captures an object with an AR mark for the object and an ultrasonic probe with an AR mark for the ultrasonic probe to obtain a camera image, a non-object coordinate conversion unit that calculates a body surface curved surface approximating the body surface of the object based on the AR mark for the object in the camera image, an ultrasonic probe coordinate conversion unit that calculates 3D coordinate information of the ultrasonic probe based on the AR mark for the ultrasonic probe in the camera image, a 3D processor that calculates the position of the region of interest on the body surface of the object based on the region of interest, the body surface curved surface of the object, and the 3D coordinate information of the ultrasonic probe, and an ROI / camera image synthesis unit that superimposes and displays an ROI projection mark indicating the position of the region of interest on the camera image.

[0005] Furthermore, Patent Document 2 discloses a robotic surgical system comprising a first robotic arm for holding an ultrasound probe, a second robotic arm for holding surgical instruments, and a master input device for remotely operating a plurality of robotic arms, which can operate in a first mode in which the second robotic arm is fixed in a predetermined position and the first robotic arm is moved in response to the operation of the master input device, or in a second mode in which the second robotic arm is moved in response to the operation of the master input device and the first robotic arm is moved in response to a user command. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2013-255658 [Patent Document 2] Patent No. 4999012 specification [Overview of the project] [Problems that the invention aims to solve]

[0007] Incidentally, when performing a procedure (e.g., surgery) on a subject's target object (e.g., organ), ultrasound waves may be transmitted to and received from the target object before the procedure and after the procedure has started (i.e., during or after the procedure). In such cases, it is desirable to keep the position and orientation of the ultrasound probe, or more specifically, the relative position and orientation of the ultrasound probe with respect to the target object (sometimes referred to as "relative positional relationship" in this specification), as similar as possible between before the procedure and after the procedure has started.

[0008] For example, before a procedure, ultrasound waves are transmitted and received from an ultrasound probe to the object to be treated on the subject, forming an ultrasound tomographic image. The physician performing the procedure can then confirm the object to be treated by reviewing this ultrasound tomographic image. Since the ultrasound probe would get in the way of performing the procedure on the object, it is moved before the procedure begins. After the procedure has started, ultrasound waves are transmitted and received again from the object to be treated using the ultrasound probe, forming another ultrasound tomographic image, for the purpose of checking the progress and results of the procedure. Here, in order to suitably confirm the results of the procedure, it is desirable that the ultrasound tomographic image formed before the procedure and the ultrasound tomographic image formed after the start of the procedure are as close to the same cross-section as possible. In other words, it is desirable that the relative position and orientation relationship between the ultrasound probe and the object to be treated be as similar as possible between before and after the procedure on the subject.

[0009] The purpose of the ultrasound probe operation support system disclosed herein is to reduce the difference in the relative positional relationship between the ultrasound probe and the object being treated before and after treatment on the subject. [Means for solving the problem]

[0010] The ultrasonic probe operation support system disclosed herein is characterized by comprising: a pre-treatment relative relationship calculation unit that calculates the pre-treatment relative relationship, which is the relative position and orientation relationship between the ultrasonic probe and the object to be treated when ultrasonic waves are transmitted and received to the subject in order to form an ultrasonic image, based on a pre-treatment camera image obtained by taking a camera of a probe detection mark attached to the ultrasonic probe and an object detection mark attached to the object to be treated, before treatment on the subject; a current relative relationship calculation unit that calculates the current relative relationship, which is the relative position and orientation relationship between the ultrasonic probe and the object to be treated, based on a current camera image obtained by taking a camera of the probe detection mark and the object detection mark, after treatment on the subject has started; and a probe control support unit that performs control to bring the current relative relationship closer to the pre-treatment relative relationship.

[0011] The probe control support unit may adjust the position or orientation of the robot arm gripping the ultrasonic probe to bring the current relative relationship closer to the pre-treatment relative relationship.

[0012] The probe control support unit may notify the operator operating the ultrasonic probe of the position and orientation of the ultrasonic probe that constitute the current relative relationship.

[0013] The probe control support unit may display on its display unit a camera image taken by a camera of the ultrasonic probe and the object to be treated, which shows the position and orientation of the ultrasonic probe so as to represent the current relative relationship.

[0014] The pre-treatment relative relationship calculation unit converts position and orientation information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is identified based on the image of the probe detection mark included in the pre-treatment camera image, into position and orientation information in the robot coordinate system grasped by the robot control device that controls the robot arm, according to the arm-probe position and orientation relationship, which is the fixed position and orientation relationship between the robot arm and the ultrasonic probe grasped by the robot arm, and converts position and orientation information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the pre-treatment camera image, into position and orientation information in the robot coordinate system The current relative relationship calculation unit converts the pre-treatment relative relationship in the robot coordinate system to position and orientation information in the robot coordinate system, which is determined based on the image of the probe detection mark included in the current camera image according to the position and orientation relationship between the arm probes, and converts the position and orientation information in the robot coordinate system, which is determined based on the image of the object detection mark included in the current camera image according to the position and orientation relationship between the arm probes, and converts the position and orientation information in the robot coordinate system, which is determined based on the image of the object detection mark included in the current camera image according to the position and orientation relationship between the arm probes, and calculates the current relative relationship in the robot coordinate system.

[0015] The pre-treatment relative relationship calculation unit converts position and orientation information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is identified based on the image of the probe detection mark included in the pre-treatment camera image, into position and orientation information in the robot coordinate system grasped by the robot control device that controls the robot arm, according to the arm-camera position and orientation relationship, which is the fixed position and orientation relationship between the robot arm and the camera grasped by the robot arm, and converts position and orientation information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the pre-treatment camera image, into position and orientation information in the robot coordinate system, according to the arm-camera position and orientation relationship, and converts position and orientation information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the pre-treatment camera image, into position and orientation information in the robot coordinate system. Alternatively, the pre-treatment relative relationship in the robot coordinate system is calculated, and the current relative relationship calculation unit converts position and orientation information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is identified based on the image of the probe detection mark included in the current camera image, into position and orientation information in the robot coordinate system, and converts position and orientation information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the current camera image, into position and orientation information in the robot coordinate system, and calculates the current relative relationship in the robot coordinate system.

[0016] The pre-treatment camera image and the current camera image are obtained by capturing the probe detection mark, the object detection mark, and the arm detection mark attached to the robot arm with the camera. The pre-treatment relative relationship calculation unit converts the position and orientation information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is determined based on the image of the arm detection mark included in the pre-treatment camera image, into position and orientation information in the robot coordinate system, according to the coordinate system relationship indicating the relationship between the camera coordinates and the robot coordinates, which is obtained based on the position and orientation of the robot arm in the camera coordinate system, which is determined based on the position and orientation of the robot arm in the robot coordinate system, which is determined based on the position and orientation of the robot arm in the camera coordinate system, which is determined based on the image of the probe detection mark included in the pre-treatment camera image, into position and orientation information in the robot coordinate system, according to the coordinate system relationship. The position and orientation information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the R image, is converted into position and orientation information in the robot coordinate system, and the pre-treatment relative relationship in the robot coordinate system is calculated. The current relative relationship calculation unit converts the position and orientation information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is identified based on the image of the probe detection mark included in the current camera image, into position and orientation information in the robot coordinate system, which is converted into position and orientation information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the current camera image, into position and orientation information in the robot coordinate system, which is calculated.

[0017] Furthermore, the ultrasonic probe operation support program disclosed herein is an ultrasonic probe operation support program characterized in that it causes a computer to function as follows: a pre-treatment relative relationship calculation unit that calculates the pre-treatment relative relationship, which is the relative position and orientation relationship between the ultrasonic probe and the object to be treated when ultrasonic waves are transmitted and received to the subject in order to form an ultrasonic image, based on a pre-treatment camera image obtained by taking a pre-treatment camera image of the probe detection mark attached to the ultrasonic probe and the object to be treated with a camera, before treatment on the subject; a current relative relationship calculation unit that calculates the current relative relationship, which is the relative position and orientation relationship between the ultrasonic probe and the object to be treated, based on a current camera image obtained by taking a camera image of the probe detection mark and the object detection mark, after treatment on the subject has started; and a probe control support unit that performs control to bring the current relative relationship closer to the pre-treatment relative relationship. [Effects of the Invention]

[0018] The ultrasonic probe operation support system disclosed herein can reduce the difference in the relative positional relationship between the ultrasonic probe and the object being treated before and after treatment on the subject. [Brief explanation of the drawing]

[0019] [Figure 1] This is a schematic diagram of the configuration of the ultrasonic probe operation support system according to the first embodiment. [Figure 2] This is a conceptual diagram showing the endoscope camera, ultrasound probe, and object to be treated in the first embodiment. [Figure 3] This is a schematic diagram of the configuration of an ultrasound diagnostic device. [Figure 4] This figure shows an example of an ultrasound probe guidance screen. [Figure 5] This flowchart shows the pre-treatment process of the ultrasound diagnostic device according to the first embodiment. [Figure 6]It is a flowchart showing the flow of processing after the start of treatment of the ultrasonic diagnostic apparatus according to the first embodiment. [Figure 7] It is a schematic configuration diagram of an operation support system for an ultrasonic probe according to the second embodiment. [Figure 8] It is a conceptual diagram showing an external camera, an ultrasonic probe, and a subject in the second embodiment. [Figure 9] It is a schematic configuration diagram of an operation support system for an ultrasonic probe according to the third embodiment. [Figure 10] It is a flowchart showing the flow of processing before treatment of the ultrasonic diagnostic apparatus according to the third embodiment. [Figure 11] It is a flowchart showing the flow of processing after the start of treatment of the ultrasonic diagnostic apparatus according to the third embodiment. [Figure 12] It is a schematic configuration diagram of an operation support system for an ultrasonic probe according to the fourth embodiment. [Figure 13] It is a flowchart showing the flow of processing before treatment of the ultrasonic diagnostic apparatus according to the fourth embodiment. [Figure 14] It is a flowchart showing the flow of processing after the start of treatment of the ultrasonic diagnostic apparatus according to the fourth embodiment.

Embodiments for Carrying Out the Invention

[0020] <First Embodiment> FIG. 1 is a schematic configuration diagram of an operation support system 10 for an ultrasonic probe according to the first embodiment. The operation support system 10 for an ultrasonic probe includes a robot control device 12 including a plurality of robot arms 12a, an endoscope 14, and an ultrasonic diagnostic apparatus 16 including an ultrasonic probe 16a. The robot control device 12, the endoscope 14, and the ultrasonic diagnostic apparatus 16 are communicably connected to each other via a communication line 18 such as a WAN (Wide Area Network) or a LAN (Local Area Network).

[0021] The robot control device 12 includes multiple robot arms 12a, a processor consisting of a CPU (Central Processing Unit), a communication interface consisting of a network adapter, and an input interface consisting of a surgeon console. The processor of the robot control device 12 controls the robot arms 12a according to instructions from an operator such as a doctor. The operator may directly input instructions to the robot control device 12 using the input interface of the robot control device 12, or may input instructions remotely via a communication line 18.

[0022] The robotic arm 12a grasps a surgical instrument (e.g., scissors or forceps) for the procedure. In other words, the robotic control device 12 controls the procedure (e.g., surgery) performed on the subject E using the surgical instrument.

[0023] In the first embodiment, the robot arm 12a grasps the endoscope 14 and the ultrasound probe 16a. In other words, in the first embodiment, the robot control device 12 controls the position and orientation of the endoscope 14 and the position and orientation of the ultrasound probe 16a.

[0024] The endoscope 14 includes a lens, an image sensor, a processor consisting of a CPU, and a communication interface consisting of a network adapter. The endoscope 14 is a camera that images the inside of the body cavity of the subject E. The image sensor of the endoscope 14 forms an endoscopic image, and the communication interface of the endoscope 14 transmits the endoscopic image to the ultrasound diagnostic device 16. Alternatively, the endoscope 14 may be directly connected to the ultrasound diagnostic device 16 by a cable or the like without going through the communication line 18, and the endoscopic image may be directly transmitted to the ultrasound diagnostic device 16.

[0025] Figure 2 is a conceptual diagram showing an endoscope 14, an ultrasound probe 16a, and the object to be treated T (an organ in the example of Figure 2). In the first embodiment, it is assumed that laparoscopic surgery is performed on a subject E. In laparoscopic surgery, a small hole is made in the abdomen A of subject E, and a tube (port) P is attached to the hole. The abdominal cavity AC is inflated with gas, and the endoscope 14, ultrasound probe 16a, and treatment instruments (not shown) are inserted into the abdominal cavity AC through the tube P. In the first embodiment, the ultrasound probe 16a is a drop-in type probe that is inserted into the body cavity (in the abdominal cavity in the example of Figure 2). As described above, in the first embodiment, the endoscope 14 and ultrasound probe 16a are grasped by a robotic arm 12a, and their position and orientation are controlled by a robotic control device 12.

[0026] The ultrasound probe 16a is marked with a probe detection mark 20. The probe detection mark 20 is a mark used to detect the position and orientation of the ultrasound probe 16a. The probe detection mark 20 is captured by a camera to obtain a camera image (endoscopic image in the first embodiment), and the position and orientation of the ultrasound probe 16a can be detected by analyzing the image of the probe detection mark 20 captured in the camera image. An example of the probe detection mark 20 is an AR (Argumented Reality) marker.

[0027] Furthermore, the object to be treated T is marked with an object detection mark 22. The object detection mark 22 is a mark used to detect the position and orientation of the object to be treated T. By capturing the object detection mark 22 with a camera to acquire a camera image and analyzing the image of the object detection mark 22 captured in the camera image, the position and orientation of the object detection mark 22 can be detected. An example of the object detection mark 22 is an AR marker. The object detection mark 22 has a different pattern from the probe detection mark 20.

[0028] The procedure for treating object T will now be described. First, before treatment of object T, ultrasound is transmitted and received to object T by an ultrasound probe 16a inserted into the abdominal cavity AC. This forms an ultrasound tomography image in the ultrasound diagnostic device 16. In this specification, this ultrasound tomography image is referred to as the pre-treatment ultrasound tomography image. The operator can confirm the cross-section of object T using the pre-treatment ultrasound tomography image. At the time when ultrasound is transmitted and received to object T in order to form the pre-treatment ultrasound tomography image (referred to as the pre-treatment ultrasound transmission and reception time in this specification), the endoscope 14 captures the probe detection mark 20 and the object detection mark 22 to form a camera image (endoscopic image in the first embodiment). In this specification, this camera image is referred to as the pre-treatment camera image. The pre-treatment camera image includes images of the probe detection mark 20 and images of the object detection mark 22. In particular, the image of the probe detection mark 20 provides information indicating the position and orientation of the ultrasound probe 16a during pre-treatment ultrasound transmission and reception, and the image of the object detection mark 22 provides information indicating the position and orientation of the object to be treated T (specifically its surface) during pre-treatment ultrasound transmission and reception. The pre-treatment camera image is transmitted from the endoscope 14 to the ultrasound diagnostic device 16.

[0029] Subsequently, the operator initiates treatment on the target object T. Because the ultrasound probe 16a would interfere with the treatment of the target object T, the ultrasound probe 16a is temporarily removed from the abdominal cavity AC. At a minimum, the position and orientation of the ultrasound probe 16a are changed from those during the pre-treatment ultrasound transmission and reception.

[0030] After the procedure begins, the ultrasound probe 16a is inserted into the abdominal cavity AC again, and ultrasound is attempted again to send and receive ultrasound signals to the object T being treated using the ultrasound probe 16a, in order to check the progress and results of the procedure. In this specification, the ultrasound tomography image formed after the start of the procedure (i.e., at the present time) is called the present ultrasound tomography image. Here, in order to suitably compare the pre-procedure ultrasound tomography image and the present ultrasound tomography image, it is preferable to make the position and orientation of the ultrasound probe 16a at the present time as similar as possible to the position and orientation of the ultrasound probe 16a during the pre-procedure ultrasound transmission and reception.

[0031] To this end, at the present moment, the endoscope 14 captures images of the probe detection mark 20 and the object detection mark 22 to form a camera image (endoscopic image in the first embodiment). In this specification, this camera image is referred to as the current camera image. The current camera image also includes images of the probe detection mark 20 and the object detection mark 22. In particular, the image of the probe detection mark 20 provides information indicating the position and orientation of the ultrasound probe 16a at the present moment, and the image of the object detection mark 22 provides information indicating the position and orientation of the object to be treated T (specifically its surface) at the present moment. The current camera image may be a moving image, and a current camera image that always includes images of the probe detection mark 20 and the object detection mark 22 at the present moment is formed. The current camera image is transmitted from the endoscope 14 to the ultrasound diagnostic device 16.

[0032] As will be explained in more detail later, in the ultrasound diagnostic device 16, the pre-treatment camera image and the current camera image are analyzed to help ensure that the position and orientation of the ultrasound probe 16a after the start of processing (i.e., at the present time) are as close as possible to the position and orientation of the ultrasound probe 16a during the pre-treatment ultrasound transmission and reception.

[0033] Figure 3 is a schematic diagram of the ultrasound diagnostic device 16. The ultrasound diagnostic device 16 is a medical device installed in medical institutions such as hospitals.

[0034] The ultrasonic probe 16a is a device that transmits and receives ultrasonic waves to the object T of a subject E. The ultrasonic probe 16a has a vibrating element array consisting of multiple vibrating elements that transmit and receive ultrasonic waves to the object T. In the ultrasonic probe 16a, the vibrating element array is formed from multiple vibrating elements arranged in a single row. A transmission signal is supplied to each vibrating element from the transmitting / receiving unit 30 (described later), causing each vibrating element to generate ultrasonic waves.

[0035] As described above, the ultrasonic probe 16a is marked with a probe detection mark 20.

[0036] The transmitting / receiving unit 30 transmits a transmission signal to the ultrasonic probe 16a (specifically, each vibrating element in the vibrating element array) under control from the control unit 46, which will be described later. The transmitting / receiving unit 30 also receives received signals from each vibrating element that receives reflected waves from the object to be treated T. The transmitting / receiving unit 30 has an adder and a plurality of delay units corresponding to each vibrating element, and performs phase-aligned addition processing by using the adder and the plurality of delay units to align the phases of the received signals from each vibrating element and add them together. As a result, a received beam signal is formed in which information indicating the signal strength of the reflected waves from the object to be treated T is aligned in the depth direction of the object to be treated T.

[0037] The signal processing unit 32 performs various signal processing operations on the received beam signal from the transmitting / receiving unit 30, including filtering and detection, which involves applying a bandpass filter.

[0038] The image forming unit 34 forms an ultrasonic tomographic image (B-mode image) representing a cross-section (particularly the ultrasonic transmitting and receiving wavefront) of the object T to be treated, based on the received beam signal processed by the signal processing unit 32.

[0039] The display control unit 36 ​​controls the display 38 to display various images, including the ultrasound tomography image formed by the image forming unit 34.

[0040] The display unit, the display 38, is a display device composed of, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

[0041] Although not shown in Figure 3, the ultrasound diagnostic apparatus 16 may include a Doppler signal forming unit that forms a Doppler signal indicating the movement of the object T to be treated based on the received signal obtained by the transmitting / receiving unit 30, a color Doppler image forming unit that forms a color Doppler image based on the Doppler signal, or a measurement unit that performs various measurements based on the received signal obtained by the transmitting / receiving unit 30.

[0042] The ultrasound diagnostic apparatus 16 comprises a transmitting / receiving unit 30, a signal processing unit 32, an image forming unit 34, and a display control unit 36, as well as the aforementioned Doppler signal forming unit, color Doppler image forming unit, and measurement unit, all of which are comprised of a processor. The processor comprises at least one of a general-purpose processing unit (e.g., a CPU) and a dedicated processing unit (e.g., a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a programmable logic device). The processor may not consist of a single processing unit, but rather of multiple processing units located in physically separate locations working together. Furthermore, each of the above-mentioned parts may be realized through the cooperation of hardware such as a processor and software.

[0043] The communication interface 40 is comprised of, for example, a network adapter. The communication interface 40 performs the function of communicating with other devices (particularly the robot control device 12 and the endoscope 14) via the communication line 18. In particular, in the first embodiment, the communication interface 40 receives endoscopic images from the endoscope 14, receives position and orientation information indicating the position and orientation of each robot arm 12a from the robot control device 12, and transmits control signals to the robot control device 12 for controlling the robot arm 12a (i.e., the ultrasound probe 16a).

[0044] The input interface 42 consists of, for example, buttons, a trackball, a touch panel, etc. The input interface 42 is used to input commands from an operator using the ultrasound diagnostic device 16 to the ultrasound diagnostic device 16.

[0045] The memory 44 is composed of components such as an HDD (Hard Disk Drive), SSD (Solid State Drive), eMMC (embedded Multi Media Card), ROM (Read Only Memory), or RAM (Random Access Memory). The memory 44 stores an operation support program for the ultrasound probe, which is used to operate each part of the ultrasound diagnostic device 16. The operation support program for the ultrasound probe can also be stored on a computer-readable non-temporary storage medium such as a USB (Universal Serial Bus) memory or CD-ROM. The ultrasound diagnostic device 16 can read and execute the operation support program for the ultrasound probe from such a storage medium. Since the ultrasound diagnostic device 16 performs the functions described below by reading the operation support program for the ultrasound probe, it can be said that the ultrasound diagnostic device 16 is a computer program product.

[0046] The control unit 46 is composed of at least one general-purpose processor (e.g., a CPU) and a dedicated processor (e.g., a GPU, ASIC, FPGA, or programmable logic device). The control unit 46 may not consist of a single processing unit, but rather of multiple processing units located in physically separate locations working together. The control unit 46 controls each part of the ultrasound diagnostic device 16. As shown in Figure 3, the control unit 46 also functions as a pre-treatment relative relationship calculation unit 48, a current relative relationship calculation unit 50, and a probe control support unit 52, according to the ultrasound probe operation support program stored in the memory 44.

[0047] The pre-treatment relative relationship calculation unit 48 calculates the relative positional relationship (referred to as the pre-treatment relative relationship in this specification) between the ultrasound probe 16a and the object to be treated T during pre-treatment ultrasound transmission and reception, based on the pre-treatment camera image formed by the endoscope 14 before treatment of the object to be treated T.

[0048] As described above, the pre-treatment camera image includes an image of the probe detection mark 20 indicating the position and orientation of the ultrasound probe 16a during pre-treatment ultrasound transmission and reception, and an image of the object detection mark 22 indicating the position and orientation of the object T to be treated during pre-treatment ultrasound transmission and reception. The pre-treatment relative relationship calculation unit 48 detects the position and orientation of the ultrasound probe 16a during pre-treatment ultrasound transmission and reception in the camera coordinate system of the endoscope 14 by analyzing the image of the probe detection mark 20 in the pre-treatment camera image. Similarly, the pre-treatment relative relationship calculation unit 48 detects the position and orientation (of the surface, hereafter the same) of the object T to be treated during pre-treatment ultrasound transmission and reception in the camera coordinate system of the endoscope 14 by analyzing the image of the object detection mark 22 in the pre-treatment camera image. Note that a known method can be used to detect the position and orientation of the ultrasound probe 16a or the object T to be treated in the camera coordinate system from the image of the probe detection mark 20 or the object detection mark 22 included in the pre-treatment camera image, so a detailed explanation is omitted here.

[0049] Next, the pre-treatment relative relationship calculation unit 48 converts the position and orientation information (hereinafter simply referred to as the position and orientation information of the ultrasonic probe 16a and the position and orientation information of the object to be treated T) in the camera coordinate system into position and orientation information in the robot coordinate system. The robot coordinate system is a coordinate system grasped by the robot control device 12 and is used to represent the position and orientation of the robot arm 12a.

[0050] In the first embodiment, since the ultrasonic probe 16a is held by the robot arm 12a, the positional relationship between the ultrasonic probe 16a and the robot arm 12a is fixed (does not change). In this specification, the fixed positional relationship between the ultrasonic probe 16a and the robot arm 12a is referred to as the arm-probe positional relationship. The arm-probe positional relationship can be represented, for example, by a vector in the robot coordinate system. In the first embodiment, the pre-treatment relative relationship calculation unit 48 converts the positional information of the ultrasonic probe 16a in the camera coordinate system into positional information in the robot coordinate system according to the arm-probe positional relationship.

[0051] More specifically, the pre-treatment relative relationship calculation unit 48 acquires position and orientation information of the robot arm 12a that grasps the ultrasonic probe 16a in the robot coordinate system, and information indicating the position and orientation relationship between the arm probes from the robot control device 12. The information indicating the position and orientation relationship between the arm probes may be input to the robot control device 12 by, for example, an operator who has the robot arm 12a grasp the ultrasonic probe 16a. The pre-treatment relative relationship calculation unit 48 can obtain position and orientation information of the ultrasonic probe 16a in the robot coordinate system from the position and orientation information of the robot arm 12a in the robot coordinate system and the position and orientation relationship between the arm probes. In this specification, the position and orientation information of the ultrasonic probe 16a during pre-treatment ultrasonic transmission and reception is referred to as pre-treatment probe position and orientation information.

[0052] Furthermore, in the first embodiment, the pre-treatment relative relationship calculation unit 48 converts the position and orientation information of the object to be treated T in the camera coordinate system into position and orientation information in the robot coordinate system, according to the position and orientation relationship between the arm probes. As described above, the pre-treatment relative relationship calculation unit 48 can obtain the position and orientation information of the ultrasonic probe 16a in the camera coordinate system and the position and orientation information of the ultrasonic probe 16a in the robot coordinate system. By aligning the axial directions of the camera coordinate system and the robot coordinate system, a conversion vector from the camera coordinate system to the robot coordinate system can be obtained based on the position and orientation information of the ultrasonic probe 16a in the camera coordinate system and the robot coordinate system. The pre-treatment relative relationship calculation unit 48 can use this conversion vector to convert the position and orientation information of the object to be treated T in the camera coordinate system into position and orientation information in the robot coordinate system. In this specification, the position and orientation information of the object to be treated T during pre-treatment ultrasonic transmission and reception is referred to as pre-treatment object position and orientation information.

[0053] The pre-treatment relative relationship calculation unit 48 calculates the pre-treatment relative relationship based on the pre-treatment probe position and orientation information and pre-treatment object position and orientation information in the robot coordinate system, which were obtained as described above. The pre-treatment relative relationship can be represented by the difference between the pre-treatment probe position and orientation information and the pre-treatment object position and orientation information. The pre-treatment relative relationship calculation unit 48 stores the calculated pre-treatment relative relationship, pre-treatment probe position and orientation information, and pre-treatment object position and orientation information in the memory 44.

[0054] The current relative relationship calculation unit 50 calculates the current positional relationship (referred to as the current relative relationship in this specification) between the ultrasound probe 16a and the object T at the moment of treatment (i.e., at the moment of treatment) based on the current camera image formed by the endoscope 14 after the start of treatment on the object T.

[0055] As described above, the current camera image includes an image of the probe detection mark 20 indicating the current position and orientation of the ultrasound probe 16a, and an image of the object detection mark 22 indicating the current position and orientation of the object to be treated T. The current relative relationship calculation unit 50, similar to the pre-treatment relative relationship calculation unit 48, detects the current position and orientation of the ultrasound probe 16a in the camera coordinate system of the endoscope 14 by analyzing the image of the probe detection mark 20 in the current camera image. Similarly, the current relative relationship calculation unit 50 detects the current position and orientation of the object to be treated T in the camera coordinate system of the endoscope 14 by analyzing the image of the object detection mark 22 in the current camera image.

[0056] Next, the current relative relationship calculation unit 50 converts the position and orientation information of the ultrasonic probe 16a and the object to be treated T in the camera coordinate system into position and orientation information in the robot coordinate system.

[0057] In the first embodiment, the current relative relationship calculation unit 50 converts the position and orientation information of the ultrasonic probe 16a in the camera coordinate system into position and orientation information in the robot coordinate system, according to the position and orientation relationship between the arm probes, in the same manner as the pre-treatment relative relationship calculation unit 48. In this specification, the current position and orientation information of the ultrasonic probe 16a is referred to as the current probe position and orientation information.

[0058] Furthermore, in the first embodiment, the current relative relationship calculation unit 50 converts the position and orientation information of the object to be treated T in the camera coordinate system into position and orientation information in the robot coordinate system, according to the position and orientation relationship between the arm probes, in the same manner as the pre-treatment relative relationship calculation unit 48. In this specification, the position and orientation information of the object to be treated T during pre-treatment ultrasound transmission and reception is referred to as the current object position and orientation information.

[0059] The current relative relationship calculation unit 50 calculates the current relative relationship based on the current probe position and orientation information and the current object position and orientation information in the robot coordinate system, which were obtained as described above. The current relative relationship can be represented by the difference between the current probe position and orientation information and the current object position and orientation information.

[0060] The probe control support unit 52 performs control to bring the current relative relationship calculated by the current relative relationship calculation unit 50 closer to the pre-treatment relative relationship calculated by the pre-treatment relative relationship calculation unit 48 and stored in the memory 44.

[0061] In the first embodiment, since the robot arm 12a is gripping the ultrasonic probe 16a, the probe control support unit 52 transmits a control signal to the robot control device 12 to change the position or orientation of the robot arm 12a gripping the ultrasonic probe 16a. Specifically, the probe control support unit 52 calculates the position and orientation of the ultrasonic probe 16a relative to the position and orientation of the object to be treated, as indicated by the current object position and orientation information, such that the current relative relationship and the pre-treatment relative relationship are the same (the target position and orientation of the ultrasonic probe 16a). The probe control support unit 52 then controls the robot arm 12a gripping the ultrasonic probe 16a so that the position and orientation of the ultrasonic probe 16a, as indicated by the current probe position and orientation information, becomes the target position and orientation of the ultrasonic probe 16a.

[0062] Furthermore, as described above, the robot arm 12a that is gripping the ultrasonic probe 16a can be controlled by the operator. Therefore, the probe control support unit 52 may notify the operator of the current relative position and orientation of the ultrasonic probe 16a. Various methods can be used for notification, but for example, the probe control support unit 52 may display on the display 38 as a guidance screen for the ultrasonic probe 16a an endoscopic image as a camera image, which is an image of the ultrasonic probe 16a and the object to be treated T taken by the endoscope 14, and which shows the current relative position and orientation of the ultrasonic probe 16a.

[0063] Figure 4 shows an example of a guidance screen for the ultrasound probe 16a. As shown in Figure 4, the probe control support unit 52 may, for example, display a guide 54 on the endoscopic image that shows the position and orientation of the ultrasound probe 16a as it is currently relative. In the example in Figure 4, the guide 54 shows the outline of the image of the ultrasound probe 16a as it is currently relative. The operator can change the position and orientation of the ultrasound probe 16a so that the outline of the image of the ultrasound probe 16a in the endoscopic image matches the guide 54, thereby matching the current relative relationship to the pre-treatment relative relationship. The position and orientation of the ultrasound probe 16a shown by the guide 54 can be obtained by converting the position and orientation information of the robot coordinate system that shows the target position and orientation of the ultrasound probe 16a, calculated as described above, into position and orientation information of the camera coordinate system, using the inverse vector of the transformation vector from the camera coordinate system to the robot coordinate system, calculated as described above.

[0064] The processing by the probe control support unit 52 reduces the difference between the relative relationship before treatment and the relative relationship at the current time. As a result, the operator can obtain ultrasound tomographic images of the same cross-section of the object T to be treated, both before and after the start of treatment.

[0065] The configuration overview of the ultrasound diagnostic apparatus 16 according to the first embodiment is as described above. The processing flow of the ultrasound diagnostic apparatus 16 according to the first embodiment will now be explained according to the flowcharts shown in Figures 5 and 6.

[0066] Figure 5 is a flowchart showing the processing flow of the object T to be treated before treatment in the first embodiment.

[0067] In step S10, the ultrasound probe 16a transmits and receives ultrasound waves to and from the object T to be treated. The image forming unit 34 forms a pre-treatment ultrasound tomographic image based on the signal received from the ultrasound probe 16a.

[0068] In step S12, the endoscope 14 captures the probe detection mark 20 and the object detection mark 22 during pre-treatment ultrasound transmission and reception, forming a pre-treatment camera image. The ultrasound diagnostic device 16 receives the pre-treatment camera image from the endoscope 14.

[0069] In step S14, the pre-treatment relative relationship calculation unit 48 acquires pre-treatment probe position and orientation information and pre-treatment target object position and orientation information based on the pre-treatment camera image received in step S12.

[0070] In step S16, the pre-treatment relative relationship calculation unit 48 converts the pre-treatment probe position and orientation information and pre-treatment object position and orientation information in the camera coordinate system, acquired in step S14, into pre-treatment probe position and orientation information and pre-treatment object position and orientation information in the robot coordinate system, based on the position and orientation relationship between the arm probes.

[0071] In step S18, the pre-treatment relative relationship calculation unit 48 calculates the pre-treatment relative relationship based on the pre-treatment probe position and orientation information and pre-treatment object position and orientation information of the robot coordinate system acquired in step S16.

[0072] In step S20, the pre-treatment relative relationship calculation unit 48 stores the pre-treatment relative relationship calculated in step S18, as well as the pre-treatment probe position and orientation information and pre-treatment target object position and orientation information of the robot coordinate system acquired in step S16, in the memory 44.

[0073] Figure 6 is a flowchart showing the processing flow after the start of treatment on the object T in the first embodiment.

[0074] In step S30, the endoscope 14 captures the probe detection mark 20 and the object detection mark 22 at the current moment, forming a current camera image. The ultrasound diagnostic device 16 receives the current camera image from the endoscope 14.

[0075] In step S32, the current relative relationship calculation unit 50 acquires current probe position and orientation information and current object position and orientation information based on the current camera image received in step S30.

[0076] In step S34, the current relative relationship calculation unit 50 converts the current probe position and orientation information and current object position and orientation information in the camera coordinate system, acquired in step S32, into current probe position and orientation information and current object position and orientation information in the robot coordinate system, based on the position and orientation relationship between the arm probes.

[0077] In step S36, the current relative relationship calculation unit 50 calculates the current relative relationship based on the current probe position and orientation information and the current object position and orientation information of the robot coordinate system acquired in step S34.

[0078] In step S38, the probe control support unit 52 performs control to bring the current relative relationship calculated by the current relative relationship calculation unit 50 in step S36 closer to the pre-treatment relative relationship calculated by the pre-treatment relative relationship calculation unit 48 in step S18.

[0079] In step S40, the ultrasound probe 16a, whose position and orientation have been changed so that the current relative relationship matches the pre-treatment relative relationship, transmits and receives ultrasound to the object T to be treated. The image forming unit 34 forms a current ultrasound tomography image based on the signal received from the ultrasound probe 16a. The current ultrasound tomography image will have the same cross-section as the pre-treatment ultrasound tomography image.

[0080] <Second Embodiment> Figure 7 is a schematic diagram of the configuration of the ultrasound probe operation support system 10-2 according to the second embodiment. The ultrasound probe operation support system 10-2 differs from the ultrasound probe operation support system 10 according to the first embodiment in that it has an external camera 60 instead of an endoscope 14. Explanations of aspects that are the same as in the first embodiment will be omitted.

[0081] The external camera 60 consists of a lens, an image sensor, a processor including a CPU, and a communication interface including a network adapter. The external camera 60 is positioned away from the body surface of the subject E and is a camera that images the subject E from the outside.

[0082] Figure 8 is a conceptual diagram showing an external camera 60, an ultrasound probe 16a, and a subject E as the object to be treated T. The external camera 60 is positioned above the subject E by a mechanism such as a camera arm. The external camera 60 may be fixed, or its position and orientation may be changed by an operator. In the second embodiment, the ultrasound probe 16a is also marked with a probe detection mark 20. In the second embodiment, the object to be treated T is also marked with an object detection mark 22. In the second embodiment, the object detection mark 22 may be marked on the body surface of the subject E, which is the object to be treated T.

[0083] In the second embodiment, the pre-treatment camera image, the current camera image, and the guidance screen of the ultrasound probe 16a displayed on the display 38 by the probe control support unit 52 are captured by the external camera 60.

[0084] <Third Embodiment> Figure 9 is a schematic diagram of the configuration of the ultrasonic probe operation support system 10-3 according to the third embodiment. In the first embodiment, the robot arm 12a grasped the ultrasonic probe 16a, but in the third embodiment, the operator O grasps the ultrasonic probe 16a. Also, in the third embodiment, as in the first embodiment, an endoscope 14 is used as the camera. Details that are the same as in the first embodiment will be omitted for brevity.

[0085] In the first embodiment, the pre-treatment relative relationship calculation unit 48 and the current relative relationship calculation unit 50 converted the position and orientation information of the ultrasonic probe 16a and the object to be treated T in the camera coordinate system into position and orientation information in the robot coordinate system, according to the arm-probe position and orientation relationship, which is the fixed position and orientation relationship between the ultrasonic probe 16a and the robot arm 12a that is gripping the ultrasonic probe 16a. However, in the third embodiment, since the operator O is gripping the ultrasonic probe 16a, it is not possible to convert the position and orientation information from the camera coordinate system to the robot coordinate system according to the arm-probe position and orientation relationship.

[0086] In the third embodiment, since the endoscope 14 is held by the robot arm 12a, the positional relationship between the endoscope 14 (specifically its lens) and the robot arm 12a is fixed (does not change). In this specification, the fixed positional relationship between the endoscope 14 and the robot arm 12a is referred to as the arm-camera positional relationship. The arm-camera positional relationship can be represented, for example, by a vector in the robot coordinate system.

[0087] In the third embodiment, the pre-treatment relative relationship calculation unit 48 converts the position and orientation information of the ultrasonic probe 16a in the camera coordinate system, which is identified based on the image of the probe detection mark 20 included in the pre-treatment camera image, into position and orientation information in the robot coordinate system, according to the position and orientation relationship between the arm cameras.

[0088] More specifically, the pre-treatment relative relationship calculation unit 48 acquires position and orientation information of the robot arm 12a that grasps the endoscope 14 in the robot coordinate system, and information indicating the position and orientation relationship between the arm and camera from the robot control device 12. The information indicating the position and orientation relationship between the arm and camera may be input to the robot control device 12 by, for example, an operator who has the robot arm 12a grasp the endoscope 14. The pre-treatment relative relationship calculation unit 48 can obtain the position and orientation of the endoscope 14 in the robot coordinate system from the position and orientation information of the robot arm 12a in the robot coordinate system and the position and orientation relationship between the arm and camera. That is, the pre-treatment relative relationship calculation unit 48 can obtain the origin of the camera coordinates in the robot coordinate system, and the orientation of the endoscope 14 (lens) (i.e., the three axis directions of the camera coordinates). Based on the origin of the camera coordinates and the three axis directions of the camera coordinates in the robot coordinate system, the pre-treatment relative relationship calculation unit 48 can determine a transformation vector from the camera coordinate system to the robot coordinate system.

[0089] The pre-treatment relative relationship calculation unit 48 uses the transformation vector to convert the position and orientation information of the ultrasonic probe 16a in the camera coordinate system into position and orientation information in the robot coordinate system.

[0090] Furthermore, the pre-treatment relative relationship calculation unit 48 converts the position and orientation information of the object to be treated T in the camera coordinate system, which is identified based on the image of the object detection mark 22 included in the pre-treatment camera image, into position and orientation information in the robot coordinate system, according to the position and orientation relationship between the arm probes. Specifically, the pre-treatment relative relationship calculation unit 48 uses the above conversion vector to convert the position and orientation information of the object to be treated T in the camera coordinate system into position and orientation information in the robot coordinate system.

[0091] Similarly, the current relative relationship calculation unit 50 converts the position and orientation information of the ultrasonic probe 16a in the camera coordinate system, which is identified based on the image of the probe detection mark 20 included in the current camera image, into position and orientation information in the robot coordinate system, according to the position and orientation relationship between the arm and camera. More specifically, the current relative relationship calculation unit 50 uses the above conversion vector to convert the position and orientation information of the ultrasonic probe 16a in the camera coordinate system into position and orientation information in the robot coordinate system.

[0092] Furthermore, the current relative relationship calculation unit 50 converts the position and orientation information of the object to be treated T in the camera coordinate system, which is identified based on the image of the object detection mark 22 included in the current camera image, into position and orientation information in the robot coordinate system, according to the position and orientation relationship between the arm cameras. Specifically, the current relative relationship calculation unit 50 uses the above conversion vector to convert the position and orientation information of the object to be treated T in the camera coordinate system into position and orientation information in the robot coordinate system.

[0093] The following describes the processing flow of the ultrasound diagnostic apparatus 16-3 according to the third embodiment, following the flowcharts shown in Figures 10 and 11.

[0094] Figure 10 is a flowchart showing the processing flow before treatment of the object T in the third embodiment.

[0095] In step S50, the ultrasound probe 16a transmits and receives ultrasound waves to and from the object T to be treated. The image forming unit 34 forms a pre-treatment ultrasound tomographic image based on the signal received from the ultrasound probe 16a.

[0096] In step S52, the endoscope 14 captures the probe detection mark 20 and the object detection mark 22 during pre-treatment ultrasound transmission and reception, forming a pre-treatment camera image. The ultrasound diagnostic device 16 receives the pre-treatment camera image from the endoscope 14.

[0097] In step S54, the pre-treatment relative relationship calculation unit 48 acquires pre-treatment probe position and orientation information and pre-treatment target object position and orientation information based on the pre-treatment camera image received in step S52.

[0098] In step S56, the pre-treatment relative relationship calculation unit 48 converts the pre-treatment probe position and orientation information and pre-treatment target object position and orientation information in the camera coordinate system, acquired in step S54, into pre-treatment probe position and orientation information and pre-treatment target object position and orientation information in the robot coordinate system, based on the position and orientation relationship between the arm cameras.

[0099] In step S58, the pre-treatment relative relationship calculation unit 48 calculates the pre-treatment relative relationship based on the pre-treatment probe position and orientation information and pre-treatment object position and orientation information of the robot coordinate system acquired in step S56.

[0100] In step S60, the pre-treatment relative relationship calculation unit 48 stores the pre-treatment relative relationship calculated in step S58, as well as the pre-treatment probe position and orientation information and pre-treatment target object position and orientation information of the robot coordinate system acquired in step S56, in the memory 44.

[0101] Figure 11 is a flowchart showing the processing flow after treatment begins on object T.

[0102] In step S70, the endoscope 14 captures the probe detection mark 20 and the object detection mark 22 at the current moment, forming a current camera image. The ultrasound diagnostic device 16 receives the current camera image from the endoscope 14.

[0103] In step S72, the current relative relationship calculation unit 50 acquires current probe position and orientation information and current object position and orientation information based on the current camera image received in step S70.

[0104] In step S74, the current relative relationship calculation unit 50 converts the current probe position and orientation information and current object position and orientation information in the camera coordinate system, acquired in step S72, into current probe position and orientation information and current object position and orientation information in the robot coordinate system, based on the position and orientation relationship between the arm cameras.

[0105] In step S76, the current relative relationship calculation unit 50 calculates the current relative relationship based on the current probe position and orientation information and the current object position and orientation information of the robot coordinate system acquired in step S74.

[0106] In step S78, the probe control support unit 52 performs control to bring the current relative relationship calculated by the current relative relationship calculation unit 50 in step S76 closer to the pre-treatment relative relationship calculated by the pre-treatment relative relationship calculation unit 48 in step S58.

[0107] In step S80, the ultrasound probe 16a, whose position and orientation have been changed so that the current relative relationship matches the pre-treatment relative relationship, transmits and receives ultrasound to the object T to be treated. The image forming unit 34 forms a current ultrasound tomography image based on the signal received from the ultrasound probe 16a. The current ultrasound tomography image has the same cross-section as the pre-treatment ultrasound tomography image.

[0108] <Fourth Embodiment> Figure 12 is a schematic diagram of the configuration of the ultrasound probe operation support system 10-4 according to the fourth embodiment. The ultrasound probe operation support system 10-4 differs from the ultrasound probe operation support system 10 according to the first embodiment in that it has an external camera 60 instead of an endoscope 14, and that the operator O is holding the ultrasound probe 16a. Explanations of aspects that are the same as in the first embodiment will be omitted.

[0109] In the fourth embodiment, as in the third embodiment, since operator O is holding the ultrasonic probe 16a, it is not possible to convert positional information from the camera coordinate system to the robot coordinate system according to the positional relationship between the arm probes. Also, in the fourth embodiment, since an external camera 60 is used instead of the endoscope 14, it is not possible to convert positional information from the camera coordinate system to the robot coordinate system according to the positional relationship between the arm cameras, as in the third embodiment.

[0110] Therefore, in the fourth embodiment, an arm detection mark 70 is attached to at least one of the robot arms 12a. The arm detection mark 70 is a mark having a pattern that indicates the position and orientation of the robot arm 12a. As a result, the position and orientation of the robot arm 12a can be detected by capturing an image of the arm detection mark 70 with an external camera 60 to acquire an external camera image, and then analyzing the image of the arm detection mark 70 captured in the external camera image. An example of an arm detection mark 70 is an AR marker.

[0111] The external camera 60 captures the probe detection mark 20 attached to the ultrasound probe 16a, the object detection mark 22 attached to the object to be treated T (for example, the body surface of the subject E), and the arm detection mark 70 to form an external camera image.

[0112] In the fourth embodiment, the pre-treatment relative relationship calculation unit 48 determines the position and orientation of the robot arm 12a in the camera coordinate system based on the image of the arm detection mark 70 included in the pre-treatment camera image formed by the external camera 60. The pre-treatment relative relationship calculation unit 48 then obtains an inter-coordinate system relationship showing the relationship between the camera coordinates and the robot coordinates based on the determined position and orientation of the robot arm 12a in the camera coordinate system and the position and orientation of the robot arm 12a in the robot coordinate system. If necessary, the robot arm 12a may be moved to obtain multiple combinations of the position and orientation of the robot arm 12a in the camera coordinate system and the position and orientation of the robot arm 12a in the robot coordinate system, and the inter-coordinate system relationship may be obtained based on these multiple combinations. The inter-coordinate system relationship is, so to speak, a transformation vector from the camera coordinate system to the robot coordinate system.

[0113] The pre-treatment relative relationship calculation unit 48 converts the position and orientation information of the ultrasonic probe 16a in the camera coordinate system, which is identified based on the image of the probe detection mark 20 included in the pre-treatment camera image, into position and orientation information in the robot coordinate system, according to the coordinate system relationships acquired as described above.

[0114] Furthermore, the pre-treatment relative relationship calculation unit 48 converts the position and orientation information of the object to be treated T in the camera coordinate system, which is identified based on the image of the object detection mark 22 included in the pre-treatment camera image, into position and orientation information in the robot coordinate system, according to the relationship between coordinate systems.

[0115] Similarly, the current relative relationship calculation unit 50 converts the position and orientation information of the ultrasonic probe 16a in the camera coordinate system, which is identified based on the image of the probe detection mark 20 included in the current camera image, into position and orientation information in the robot coordinate system, according to the relationship between coordinate systems.

[0116] Furthermore, the current relative relationship calculation unit 50 converts the position and orientation information of the object to be treated T in the camera coordinate system, which is identified based on the image of the object detection mark 22 included in the current camera image, into position and orientation information in the robot coordinate system, according to the relationship between coordinate systems.

[0117] The following describes the processing flow of the ultrasound diagnostic apparatus 16 according to the fourth embodiment, following the flowcharts shown in Figures 13 and 14.

[0118] Figure 13 is a flowchart showing the processing flow before treatment of the object T in the fourth embodiment.

[0119] In step S90, the ultrasound probe 16a transmits and receives ultrasound to and from the object T to be treated. The image forming unit 34 forms a pre-treatment ultrasound tomographic image based on the signal received from the ultrasound probe 16a.

[0120] In step S92, the external camera 60 captures the probe detection mark 20, object detection mark 22, and arm detection mark 70 during pre-treatment ultrasound transmission and reception, forming a pre-treatment camera image. The ultrasound diagnostic device 16 receives the pre-treatment camera image from the external camera 60.

[0121] In step S94, the pre-treatment relative relationship calculation unit 48 acquires pre-treatment probe position and orientation information, pre-treatment target object position and orientation information, and robot arm 12a position and orientation information based on the pre-treatment camera image received in step S92.

[0122] In step S96, the pre-treatment relative relationship calculation unit 48 acquires the relationship between coordinate systems based on the position and orientation of the robot arm 12a in the camera coordinate system identified in step S94 (pre-treatment probe position and orientation information) and the position and orientation of the robot arm 12a in the robot coordinate system.

[0123] In step S98, the pre-treatment relative relationship calculation unit 48 converts the pre-treatment probe position and orientation information and pre-treatment object position and orientation information in the camera coordinate system, acquired in step S94, into pre-treatment probe position and orientation information and pre-treatment object position and orientation information in the robot coordinate system, based on the coordinate system relationships acquired in step S96.

[0124] In step S100, the pre-treatment relative relationship calculation unit 48 calculates the pre-treatment relative relationship based on the pre-treatment probe position and orientation information and pre-treatment object position and orientation information of the robot coordinate system acquired in step S98.

[0125] In step S102, the pre-treatment relative relationship calculation unit 48 stores the pre-treatment relative relationship calculated in step S100, as well as the pre-treatment probe position and orientation information and pre-treatment target object position and orientation information of the robot coordinate system acquired in step S98, in the memory 44.

[0126] Figure 14 is a flowchart showing the processing flow after the start of treatment on the object T in the fourth embodiment.

[0127] In step S110, the external camera 60 captures the probe detection mark 20 and the object detection mark 22 at the current moment, forming a current camera image. The ultrasound diagnostic device 16 receives the current camera image from the external camera 60.

[0128] In step S112, the current relative relationship calculation unit 50 acquires current probe position and orientation information and current object position and orientation information based on the current camera image received in step S30.

[0129] In step S114, the current relative relationship calculation unit 50 converts the current probe position and orientation information and current object position and orientation information in the camera coordinate system, acquired in step S112, into current probe position and orientation information and current object position and orientation information in the robot coordinate system, based on the coordinate system relationships acquired in step S96.

[0130] In step S116, the current relative relationship calculation unit 50 calculates the current relative relationship based on the current probe position and orientation information and the current object position and orientation information of the robot coordinate system acquired in step S114.

[0131] In step S118, the probe control support unit 52 performs control to bring the current relative relationship calculated by the current relative relationship calculation unit 50 in step S116 closer to the pre-treatment relative relationship calculated by the pre-treatment relative relationship calculation unit 48 in step S100.

[0132] In step S120, the ultrasound probe 16a, whose position and orientation have been changed so that the current relative relationship matches the pre-treatment relative relationship, transmits and receives ultrasound to the object T to be treated. The image forming unit 34 forms a current ultrasound tomography image based on the signal received from the ultrasound probe 16a. The current ultrasound tomography image has the same cross-section as the pre-treatment ultrasound tomography image.

[0133] The ultrasonic probe operation support system described above has been explained, but the ultrasonic probe operation support system described above is not limited to the embodiments described above, and various modifications are possible as long as they do not deviate from the spirit of the invention.

[0134] For example, in each of the above embodiments, the ultrasound probe operation support systems 10, 10-2, 10-3, and 10-4 were used to control the pre-treatment ultrasound tomography image and the current ultrasound tomography image to be the same cross-section. However, the ultrasound probe operation support systems 10, 10-2, 10-3, and 10-4 can also be used for other purposes, as long as the relative position and orientation of the ultrasound probe 16a with respect to the treatment target T are to be the same before treatment and after treatment begins. For example, the ultrasound probe operation support systems 10, 10-2, 10-3, and 10-4 can also be used when performing color Doppler image formation and various measurements before and after treatment, by sending and receiving ultrasound to the treatment target T while keeping the position and orientation of the ultrasound probe 16a the same before and after treatment begins.

[0135] Furthermore, in each of the above embodiments, the position and orientation information of the ultrasonic probe 16a and the position and orientation information of the object to be treated T in the camera coordinate system were converted to position and orientation information in the robot coordinate system. However, if the position and orientation of the endoscope 14 or the external camera 60 do not change before the procedure and after the start of the procedure (i.e., at the present time), it is not necessary to convert from the camera coordinate system to the robot coordinate system. In this case, the pre-procedure relative relationship calculation unit 48 calculates the pre-procedure relative relationship based on the position and orientation information of the ultrasonic probe 16a and the position and orientation information of the object to be treated T in the camera coordinate system, and the current relative relationship calculation unit 50 calculates the current relative relationship based on the position and orientation information of the ultrasonic probe 16a and the position and orientation information of the object to be treated T in the camera coordinate system.

[0136] Furthermore, in each of the above embodiments, the functions of the pre-treatment relative relationship calculation unit 48, the current relative relationship calculation unit 50, and the probe control support unit 52 were performed by the control unit 46 of the ultrasound diagnostic device 16. However, these functions do not necessarily have to be performed by the ultrasound diagnostic device 16. For example, these functions may be performed by a robot control device 12, an endoscope 14 (or external camera 60), and a server computer that is communicatively connected to the ultrasound diagnostic device 16. Also, at least one of the above functions may be performed by the robot control device 12. Moreover, all of the above functions may not be performed by a single device, but rather by the cooperation of multiple devices. [Explanation of symbols]

[0137] 10, 10-2, 10-3, 10-4 Ultrasound probe operation support system, 12 Robot control device, 12a Robot arm, 14 Endoscope, 16 Ultrasound diagnostic device, 16a Ultrasound probe, 20 Probe detection mark, 22 Object detection mark, 30 Transceiver unit, 32 Signal processing unit, 34 Image forming unit, 36 Display control unit, 38 Display, 40 Communication interface, 42 Input interface, 44 Memory, 46 Control unit, 48 Pre-treatment relative relationship calculation unit, 50 Current relative relationship calculation unit, 52 Probe control support unit, 54 Guide, 60 External camera.

Claims

1. A pre-treatment relative relationship calculation unit calculates the pre-treatment relative relationship, which is the relative positional relationship between the ultrasound probe and the object to be treated when ultrasound is transmitted and received to the subject in order to form an ultrasound image, based on a pre-treatment camera image obtained by taking a picture of the probe detection mark attached to the ultrasound probe and the object to be treated with a camera before treatment on the subject. At the present time, after the start of treatment on the subject, a current relative relationship calculation unit calculates the current relative relationship, which is the relative positional relationship between the ultrasound probe and the object to be treated, based on the current camera image obtained by capturing the probe detection mark and the object detection mark with a camera. A probe control support unit that performs control to bring the current relative relationship closer to the pre-treatment relative relationship, An ultrasonic probe operation support system characterized by comprising the following features.

2. The probe control support unit changes the position or orientation of the robot arm gripping the ultrasonic probe, thereby bringing the current relative relationship closer to the pre-treatment relative relationship. The ultrasonic probe operation support system according to feature 1.

3. The probe control support unit notifies the operator operating the ultrasonic probe of the position and orientation of the ultrasonic probe that constitute the current relative relationship. The ultrasonic probe operation support system according to feature 1.

4. The probe control support unit displays on the display unit a camera image taken by a camera of the ultrasonic probe and the object to be treated, which shows the position and orientation of the ultrasonic probe so as to represent the current relative relationship. The ultrasonic probe operation support system according to feature 3.

5. The pre-treatment relative relationship calculation unit is: In accordance with the arm-probe position-orientation relationship, which is the fixed position-orientation relationship between the robot arm and the ultrasonic probe grasped by the robot arm, the position-orientation information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is identified based on the image of the probe detection mark included in the pre-treatment camera image, is converted into position-orientation information in the robot coordinate system understood by the robot control device that controls the robot arm. Depending on the positional relationship between the arm probes, the positional information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the pre-treatment camera image, is converted into positional information in the robot coordinate system. The relative relationship before the procedure in the robot coordinate system is calculated, The aforementioned current relative relationship calculation unit, Depending on the positional relationship between the arm probes, the positional information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is identified based on the image of the probe detection mark included in the current camera image, is converted into positional information in the robot coordinate system. Depending on the positional relationship between the arm probes, the positional information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the current camera image, is converted into positional information in the robot coordinate system. The current relative relationship in the robot coordinate system is calculated. The ultrasonic probe operation support system according to feature 1.

6. The pre-treatment relative relationship calculation unit is: In accordance with the arm-camera position-orientation relationship, which is the fixed position-orientation relationship between the robot arm and the camera grasped by the robot arm, the position-orientation information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is identified based on the image of the probe detection mark included in the pre-treatment camera image, is converted into position-orientation information in the robot coordinate system understood by the robot control device that controls the robot arm. Depending on the positional relationship between the arm cameras, the positional information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the pre-treatment camera image, is converted into positional information in the robot coordinate system. The relative relationship before the procedure in the robot coordinate system is calculated, The aforementioned current relative relationship calculation unit, Depending on the positional relationship between the arm cameras, the positional information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is identified based on the image of the probe detection mark included in the current camera image, is converted into positional information in the robot coordinate system. Depending on the positional relationship between the arm cameras, the positional information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the current camera image, is converted into positional information in the robot coordinate system. The current relative relationship in the robot coordinate system is calculated. The ultrasonic probe operation support system according to feature 1.

7. The pre-treatment camera image and the current camera image are obtained by capturing the probe detection mark, the object detection mark, and the arm detection mark attached to the robot arm with the camera. The pre-treatment relative relationship calculation unit is: Based on the position and orientation of the robot arm in the camera coordinate system, which is determined based on the image of the arm detection mark included in the pre-treatment camera image, and the position and orientation of the robot arm in the robot coordinate system as understood by the robot control device that controls the robot arm, the position and orientation information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is determined based on the image of the probe detection mark included in the pre-treatment camera image, is converted into position and orientation information in the robot coordinate system, according to the inter-coordinate system relationship indicating the relationship between the camera coordinates and the robot coordinates. Depending on the relationship between the coordinate systems, the position and orientation information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the pre-treatment camera image, is converted into position and orientation information in the robot coordinate system. The relative relationship before the procedure in the robot coordinate system is calculated, The aforementioned current relative relationship calculation unit, Depending on the relationship between the coordinate systems, position and orientation information indicating the position and orientation of the ultrasonic probe in the camera coordinate system, which is identified based on the image of the probe detection mark included in the current camera image, is converted into position and orientation information in the robot coordinate system. Depending on the relationship between the coordinate systems, the position and orientation information indicating the position and orientation of the object to be treated in the camera coordinate system, which is identified based on the image of the object detection mark included in the current camera image, is converted into position and orientation information in the robot coordinate system. The current relative relationship in the robot coordinate system is calculated. The ultrasonic probe operation support system according to feature 1.

8. Computers, A pre-treatment relative relationship calculation unit calculates the pre-treatment relative relationship, which is the relative positional relationship between the ultrasound probe and the object to be treated when ultrasound is transmitted and received to the subject in order to form an ultrasound image, based on a pre-treatment camera image obtained by taking a picture of the probe detection mark attached to the ultrasound probe and the object to be treated with a camera before treatment on the subject. At the present time, after the start of treatment on the subject, a current relative relationship calculation unit calculates the current relative relationship, which is the relative positional relationship between the ultrasound probe and the object to be treated, based on the current camera image obtained by capturing the probe detection mark and the object detection mark with a camera. A probe control support unit that performs control to bring the current relative relationship closer to the pre-treatment relative relationship, An ultrasonic probe operation support program characterized by its function as such.

Images