Ultrasound diagnostic support equipment and ultrasound diagnostic support program

Abstract

This allows the user to easily identify areas of the target tissue of the subject that have been scanned by the ultrasound beam and areas that have not been scanned by the ultrasound beam. [Solution] The 3D model forming unit 62 acquires a 3D model 56 of the target tissue T, where each position is represented by coordinates in the model coordinate system. The probe position and orientation detection unit 64 acquires the position and orientation of the ultrasonic probe 18a. The position and orientation information conversion unit 66 converts the position and orientation information indicating the position and orientation of the ultrasonic probe 18a into converted position and orientation information in the model coordinate system. The scanned area identification unit 68 identifies the model scanning plane MCS in the model coordinate system based on the converted position and orientation information when the ultrasonic beam is scanned over the target tissue T, and identifies the area of ​​the 3D model 56 through which the model scanning plane MCS passes as the scanned area SA. The display control unit 46 displays the scanned area SA and the unscanned area NSA in the 3D model 56 in different display modes.

Description

Technical Field

[0001] This specification discloses improvements to an ultrasonic diagnostic support device and an ultrasonic diagnostic support program.

Background Art

[0002] Conventionally, an ultrasonic diagnostic apparatus that forms an ultrasonic tomographic image representing the inside of a subject based on reflected waves from the subject when ultrasonic waves are transmitted to the subject is known. Specifically, an ultrasonic beam is scanned on an ultrasonic scanning plane from an ultrasonic probe, and an ultrasonic tomographic image representing a cross-section of the subject on the ultrasonic scanning plane is formed based on the reflected waves of the ultrasonic beam from the subject.

[0003] Conventionally, various techniques related to ultrasonic diagnostic apparatuses have been proposed.

[0004] For example, Patent Document 1 discloses an ultrasonic diagnostic apparatus having a position sensor that detects the position of an ultrasonic probe, which estimates an object region based on a plurality of ultrasonic images obtained by transmitting and receiving ultrasonic waves to an object in a subject and the position information of the ultrasonic probe detected by the position sensor, and further specifies an un-imaged region of the ultrasonic image in the object region based on the plurality of ultrasonic images and the position information.

[0005] Also, Patent Document 2 discloses an ultrasonic diagnostic apparatus having a camera, which estimates the position and orientation of a subject from a first external image obtained by photographing the subject with the camera, determines the position and orientation of the ultrasonic probe from a second external image obtained by photographing an AR marker attached to the ultrasonic probe with the camera, and specifies an examination site based on the position and orientation of the subject and the position and orientation of the ultrasonic probe.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

[0007] Incidentally, when scanning a target tissue of a subject with an ultrasound beam, there are times when the operator, such as a physician, or the user or administrator of the ultrasound image (collectively referred to as "user" in this specification), may want to know which part of the target tissue the ultrasound beam has scanned, that is, which part of the target tissue the received signal corresponding to has been acquired.

[0008] While this is merely one example, before performing procedures such as tumor removal on target tissue, the operator may check the ultrasound image representing the tumor within the target tissue to confirm its location and condition. In this case, it is necessary to scan the ultrasound beam across the ultrasound scanning plane containing the tumor to form an ultrasound image that appropriately represents the tumor. In particular, if multiple tumors are scattered within the target tissue, it is preferable to appropriately form multiple ultrasound images representing each tumor. In such cases, it would be beneficial for the user to be able to understand whether the ultrasound beam has been appropriately scanned across the ultrasound scanning plane containing the tumor so that appropriate ultrasound images are formed.

[0009] The purpose of the ultrasound diagnostic support device disclosed herein is to enable the user to easily identify areas in the target tissue of a subject that have been scanned by an ultrasound beam and areas that have not been scanned by an ultrasound beam. [Means for solving the problem]

[0010] The ultrasound diagnostic support device disclosed herein includes a processor, the processor acquires a three-dimensional model of a target tissue formed based on medical volume data acquired by a medical device other than an ultrasound diagnostic device, wherein each position of the three-dimensional model is represented by coordinates in a model coordinate system, the processor acquires position and orientation information indicating the position and orientation of an ultrasound probe that scans ultrasound on an ultrasound scanning plane including the target tissue, the processor converts the position and orientation information into converted position and orientation information in the model coordinate system, the processor identifies a model scanning plane which is a virtual ultrasound scanning plane in the model coordinate system based on the converted position and orientation information when an ultrasound beam is scanned over the target tissue, the processor identifies the region of the three-dimensional model through which the model scanning plane passes as a scanned region, and the processor displays the three-dimensional model on a display unit, the display control unit configured to display the scanned region and the unscanned region which is a region not scanned by the ultrasound beam in different display modes.

[0011] The three-dimensional model is provided with information indicating the location of a particular area of ​​interest within the target tissue, and the processor is configured to display this information on the display unit.

[0012] The processor is preferably configured to evaluate the received signal obtained by scanning an ultrasonic beam with an ultrasonic scanning surface corresponding to the model scanning surface that defines the scanned area, and to display the scanned area in a manner that shows the evaluation result of the received signal.

[0013] The processor is preferably configured to display an ultrasonic tomographic image formed based on a received signal obtained by scanning an ultrasonic beam with an ultrasonic scanning plane corresponding to the model scanning plane, on the cross-section of the scanned region defined by the model scanning plane.

[0014] The processor is preferably configured to display on a display unit a medical tomographic image reconstructed by cutting out the medical volume data in a cross section based on the conversion position and orientation information, and an ultrasonic tomographic image formed based on the received signal acquired at the current position and orientation of the ultrasonic probe, and to display a guide image indicating the range of the ultrasonic scanning plane superimposed on the medical tomographic image.

[0015] The processor is preferably configured to detect the relative position and orientation of the ultrasound probe with respect to the position and orientation of the target tissue, based on images obtained by taking pictures of the ultrasound probe and the detection marks attached to the target tissue with a camera.

[0016] Furthermore, the ultrasound diagnostic support program disclosed herein is characterized by causing a computer to acquire a three-dimensional model of a target tissue formed based on medical volume data acquired by a medical device other than an ultrasound diagnostic device, wherein each position of the three-dimensional model is represented by coordinates in the model coordinate system; acquiring position and orientation information indicating the position and orientation of an ultrasound probe that scans ultrasound on an ultrasound scanning plane including the target tissue; converting the position and orientation information into converted position and orientation information in the model coordinate system; identifying the ultrasound scanning plane in the model coordinate system based on the converted position and orientation information when an ultrasound beam is scanned over the target tissue; identifying the region of the three-dimensional model through which the ultrasound scanning plane passes as a scanned region; displaying the three-dimensional model on a display unit; and displaying the scanned region and the unscanned region, which is a region where the ultrasound beam has not been scanned, in different display modes within the three-dimensional model. [Effects of the Invention]

[0017] The ultrasound diagnostic support device disclosed herein allows the user to easily identify areas in the target tissue of a subject that have been scanned by the ultrasound beam and areas that have not been scanned by the ultrasound beam. [Brief explanation of the drawing]

[0018] [Figure 1] It is a schematic configuration diagram of an ultrasonic diagnosis support system according to this embodiment. [Figure 2] It is a conceptual diagram showing a camera, an ultrasonic probe, and a target tissue. [Figure 3] It is a schematic configuration diagram of an ultrasonic diagnostic apparatus according to this embodiment. [Figure 4] It is a diagram showing an example of a three-dimensional model of a target tissue. [Figure 5] It is a diagram showing an example of a captured image of a camera. [Figure 6] It is a diagram showing an example of display of a medical tomographic image and an ultrasonic tomographic image. [Figure 7] It is a diagram showing a first example of an ultrasonic scanning plane in a model coordinate system. [Figure 8] It is a diagram showing a first example of a scanned area. [Figure 9] It is a diagram showing a second example of an ultrasonic scanning plane in a model coordinate system. [Figure 10] It is a diagram showing a second example of a scanned area. [Figure 11] It is a diagram showing a second example of a scanned area. [Figure 12] It is a diagram showing a first display example of a three-dimensional model. [Figure 13] It is a diagram showing a second display example of a three-dimensional model. [Figure 14] It is a diagram showing a third display example of a three-dimensional model. [Figure 15] It is a diagram showing a display example of a three-dimensional model, a medical tomographic image, and an ultrasonic tomographic image.

Mode for Carrying Out the Invention

[0019] Figure 1 is a schematic diagram of the configuration of the ultrasound diagnostic support system 10 according to this embodiment. The ultrasound diagnostic support system 10 consists of a medical device 12, a medical image analysis server 14, a camera 16, and an ultrasound diagnostic device 18 as an ultrasound diagnostic support device including an ultrasound probe 18a. The medical device 12, the medical image analysis server 14, and the ultrasound diagnostic device 18 are connected to each other so as to be able to communicate via a communication line 20 such as a WAN (Wide Area Network) or LAN (Local Area Network). In addition, the camera 16 and the ultrasound diagnostic device 18 are connected so as to be able to communicate via wired or wireless connection.

[0020] Medical device 12 consists of, for example, a CT (Computed Tomography) device or an MRI (Magnetic Resonance Imaging) device. In this embodiment, medical device 12 is a device other than an ultrasound diagnostic device. Medical device 12 is a device that forms medical volume data about the target tissue of a subject. The target tissue is the tissue that is scanned by the ultrasound beam of the ultrasound diagnostic device 18. In this embodiment, since the target tissue is treated by an operator after the ultrasound beam has been scanned, the target tissue can also be said to be the tissue that is treated by the operator. Medical volume data is data in which voxels, each containing data, are arranged in three dimensions. Medical volume data has positional information for each voxel. In this embodiment, the position of each voxel constituting the medical volume data is represented by coordinates in the medical data coordinate system. Prior to the ultrasound beam scanning the target tissue by the ultrasound diagnostic device 18, medical device 12 forms medical volume data including the target tissue and transmits the formed medical volume data to the medical image analysis server 14.

[0021] The medical image analysis server 14 consists of a server computer equipped with, for example, a processor, memory, and a communication interface. The medical image analysis server 14 is a device that analyzes various medical image data formed by various modalities, such as medical volume data formed by the medical device 12, or ultrasound tomographic images or ultrasound volume data formed by the ultrasound diagnostic device 18. Prior to the ultrasound diagnostic device 18 scanning the target tissue with an ultrasound beam, the medical image analysis server 14 transmits medical volume data including the target tissue to the ultrasound diagnostic device 18. Alternatively, the medical volume data may be transmitted directly from the medical device 12 to the ultrasound diagnostic device 18.

[0022] Camera 16 consists of a lens, an image sensor, a processor, and a communication interface. Camera 16 captures images of the ultrasound probe 18a, particularly the probe detection mark (details described later) attached to the ultrasound probe 18a. The image sensor of camera 16 forms an image, and the camera 16's communication interface transmits the image to the ultrasound diagnostic device 18 in real time.

[0023] Figure 2 is a conceptual diagram showing a camera 16, an ultrasound probe 18a, and the target tissue T (an organ in the example in Figure 2) of the patient. In this embodiment, it is assumed that laparoscopic surgery is performed on the target tissue T. Figure 2 shows the state in which the ultrasound probe 18a has been inserted into the abdominal cavity AC prior to the laparoscopic surgery. A small hole is made in the abdomen AB of the patient, and a tube (port) P is attached to the hole. The abdominal cavity AC is inflated with gas, and the camera 16 and ultrasound probe 18a are inserted into the abdominal cavity AC through the tube P. In this embodiment, the ultrasound probe 18a is a drop-in type probe that is inserted into the body cavity (in the example in Figure 2, into the abdominal cavity AC). By scanning the ultrasound beam with the ultrasound probe 18a over the ultrasound scanning surface including the target tissue T, an ultrasound tomographic image of the target tissue T is formed. In this embodiment, the position of the camera 16 is fixed. The ultrasound probe 18a is grasped by forceps held by the operator, or by an arm AM such as a robotic arm that operates under the control of the operator. In other words, the position and orientation (i.e., the ultrasonic scanning plane) of the ultrasound probe 18a are controlled by the operator.

[0024] The ultrasound probe 18a is marked with a probe detection mark 30. The probe detection mark 30 is a mark used to detect the position and orientation of the ultrasound probe 18a. The probe detection mark 30 is captured by the camera 16 to obtain an image, and the position and orientation of the ultrasound probe 18a can be detected by analyzing the image of the probe detection mark 30 captured in the image. An example of a probe detection mark 30 is an AR (Argumented Reality) marker.

[0025] Furthermore, the surface of the target tissue T is marked with an object detection mark 32. The object detection mark 32 is a mark used to detect the position and orientation (on the surface) of the target tissue T. The object detection mark 32 is captured by the camera 16 to obtain an image, and the position and orientation of the object detection mark 32 can be detected by analyzing the image of the object detection mark 32 captured in the image. An example of an object detection mark 32 is an AR marker. The object detection mark 32 has a different pattern from the probe detection mark 30.

[0026] In this embodiment, prior to treatment of the target tissue T (for example, removal of a tumor within the target tissue T), ultrasound waves are transmitted to and received from the target tissue T, and an ultrasound tomographic image or ultrasound volume data representing the target tissue T is formed based on the received signal obtained. At this time, it is desirable for the user to be able to understand which areas of the target tissue T have been scanned by the ultrasound beam and which areas have not been scanned by the ultrasound beam. For example, if there is a tumor in the target tissue T and the tumor is to be removed, in order for the operator to confirm the state of the tumor using an ultrasound tomographic image before the operation, it is necessary to scan the tumor with an ultrasound beam (in other words, the tumor must be included in the scanned area). In particular, if there are multiple tumors scattered in the target tissue T, it is desirable to scan all of the tumors with the ultrasound beam to prevent any tumors from being missed during removal. In this embodiment, assistance is provided to enable the user to easily understand which areas of the target tissue T have been scanned by the ultrasound beam and which areas have not been scanned by the ultrasound beam.

[0027] Figure 3 is a schematic diagram of the ultrasound diagnostic device 18. The ultrasound diagnostic device 18 is a medical device installed in medical institutions such as hospitals.

[0028] The ultrasonic probe 18a is a device that transmits and receives ultrasonic waves to a target tissue T of a subject. The ultrasonic probe 18a has a vibrating element array consisting of multiple vibrating elements that scan the target tissue T with an ultrasonic beam. In the ultrasonic probe 18a, the vibrating element array is formed from multiple vibrating elements arranged in a single row. A transmission signal is supplied to each vibrating element from the transmitting / receiving unit 40 (described later), causing each vibrating element to generate an ultrasonic beam that scans the ultrasonic scanning surface.

[0029] The size of the ultrasonic scanning surface (width, which is the length in the scanning direction of the ultrasonic beam, and depth) is determined according to known information such as the structure of the ultrasonic probe 18a and the settings of the ultrasonic diagnostic device 18. For example, the width of the ultrasonic scanning surface (scan width) is determined according to the number of vibrating elements in the ultrasonic probe 18a. The depth of the ultrasonic scanning surface (penetration depth) is determined according to the transmission intensity of the ultrasonic beam (i.e., the intensity of the transmission signal provided from the transmitting / receiving unit 40 to the ultrasonic probe 18a). The position and orientation (direction) of the ultrasonic scanning surface are determined based on the position and orientation of the ultrasonic probe 18a. In other words, since the size of the ultrasonic scanning surface is determined by known information of the ultrasonic diagnostic device 18, the ultrasonic scanning surface can be defined if the position and orientation of the ultrasonic probe 18a are known.

[0030] As described above, the ultrasonic probe 18a is marked with a probe detection mark 30.

[0031] The transmitting / receiving unit 40 transmits a transmission signal to the ultrasonic probe 18a (specifically, each vibrating element in the vibrating element array) under control from the processor 60 described later. The transmitting / receiving unit 40 also receives received signals from each vibrating element that receives reflected waves from the target tissue T. The transmitting / receiving unit 40 has an adder and a plurality of delays corresponding to each vibrating element, and performs phase-aligned addition processing by using the adder and the plurality of delays to align the phases of the received signals from each vibrating element and add them together. As a result, a received beam signal is formed in which information indicating the signal intensity of the reflected waves from the target tissue T is aligned in the depth direction of the target tissue T.

[0032] The signal processing unit 42 performs various signal processing operations on the received beam signal from the transmitting / receiving unit 40, including filtering and detection, which involves applying a bandpass filter.

[0033] The image forming unit 44 forms an ultrasound tomographic image (B-mode image) representing a cross-section (particularly the ultrasound transmitting and receiving wavefront) of the target tissue T based on the received beam signal processed by the signal processing unit 42. The image forming unit 44 also forms a medical tomographic image showing a cross-section of medical volume data based on the medical volume data received from the medical device 12 or the medical image analysis server 14.

[0034] The display control unit 46 controls the display 48 to display various images, including the ultrasound tomography image formed by the image forming unit 44.

[0035] The display unit, the display 48, is a display device composed of, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

[0036] The ultrasonic diagnostic apparatus 18 comprises a transmitting / receiving unit 40, a signal processing unit 42, an image forming unit 44, and a display control unit 46, all of which are composed of a processor (which may be the processor 60 described later, or a different processor from the processor 60).

[0037] The communication interface 50 is comprised of, for example, a network adapter. The communication interface 50 performs the function of communicating with other devices (particularly the medical device 12, the medical image analysis server 14, and the camera 16) via the communication line 20. Specifically, the communication interface 50 receives medical volume data from the medical device 12 or the medical image analysis server 14 and receives captured images from the camera 16.

[0038] The input interface 52 consists of, for example, buttons, a trackball, a touch panel, etc. The input interface 52 is used to input commands from an operator using the ultrasound diagnostic device 18 to the ultrasound diagnostic device 18.

[0039] The memory 54 is composed of components such as an HDD (Hard Disk Drive), SSD (Solid State Drive), eMMC (embedded Multi Media Card), ROM (Read Only Memory), or RAM (Random Access Memory). The memory 54 stores ultrasound diagnostic support programs for operating each part of the ultrasound diagnostic device 18. The ultrasound diagnostic support programs can also be stored on a computer-readable non-temporary storage medium such as a USB (Universal Serial Bus) memory or CD-ROM. The ultrasound diagnostic device 18 can read and execute the ultrasound diagnostic support programs from such storage media.

[0040] Furthermore, as shown in Figure 3, the memory 54 stores the 3D model 56. The 3D model 56 is formed by the 3D model forming unit 62, which will be described later. Details of the 3D model 56 will be described later.

[0041] The processor 60 performs its functions as a 3D model formation unit 62, a probe position and orientation detection unit 64, a position and orientation information conversion unit 66, a scanned area identification unit 68, and a received signal evaluation unit 70, according to the ultrasound diagnostic support program stored in the memory 54. The details of the functions of each part performed by the processor 60 will be described below.

[0042] The 3D model forming unit 62 forms a 3D model 56 of the target tissue T based on medical volume data received from the medical device 12 or the medical image analysis server 14. Thus, the 3D model forming unit 62 obtains the 3D model 56 of the target tissue T.

[0043] Figure 4 shows an example of a 3D model 56 of the target tissue T. Hereafter, the target tissue T will be assumed to be the liver. The 3D model 56 is formed from medical volume data including the liver as the target tissue T. Since known methods can be used to form the 3D model 56 based on the medical volume data, a detailed explanation will be omitted here, but the 3D model formation unit 62 forms the 3D model 56 using techniques such as volume rendering or surface rendering.

[0044] Since the medical volume data contains positional information for each voxel, and the 3D model 56 is formed from the medical volume data, the 3D model 56 also contains positional information (coordinates) indicating each location. Each location in the 3D model 56 is represented by coordinates in the model coordinate system (which is the same coordinate system as the medical data coordinate system). In the figures in this specification, the model coordinate system is represented by the Xm axis, Ym axis, and Zm axis.

[0045] Furthermore, the 3D model 56 is accompanied by information indicating the location of the area of ​​interest, which is a part of the target tissue T that the user should pay attention to. In this embodiment, the 3D model 56 is accompanied by information indicating the location of the tumor TM within the target tissue T as the area of ​​interest. The information indicating the location of the tumor TM may be set by automatic calculation when the 3D model forming unit 62 forms the 3D model 56, or it may be set manually by an operator or the like. Since a tumor TM is not usually represented by the coordinates of a single point, the 3D model 56 has information indicating the coordinate region (which may also be called the location and size of the tumor) occupied by the tumor TM. As shown in Figure 4, if multiple tumor TMs are scattered in the target tissue T, the 3D model 56 has information indicating multiple coordinate regions corresponding to each tumor TM.

[0046] The 3D model 56 may be formed by the medical device 12 or the medical image analysis server 14, and the ultrasound diagnostic device 18 may receive the formed 3D model 56 from the medical device 12 or the medical image analysis server 14. In that case, the processor 60 only needs to receive (acquire) the 3D model 56 and does not need to perform the function of a 3D model forming unit 62.

[0047] The probe position and orientation detection unit 64 detects and acquires the position and orientation of the ultrasonic probe 18a. In this embodiment, the probe position and orientation detection unit 64 detects the position and orientation of the ultrasonic probe 18a based on the captured image formed by the camera 16. The position and orientation of the ultrasonic probe 18a may constantly change due to operator actions, but the probe position and orientation detection unit 64 continuously detects the position and orientation of the ultrasonic probe 18a.

[0048] Figure 5 shows an example of an image IM captured by camera 16. The image IM includes an image of a probe detection mark 30 indicating the current position and orientation of the ultrasound probe 18a. The probe position and orientation detection unit 64 detects the current position and orientation of the ultrasound probe 18a in the camera coordinate system of camera 16 by analyzing the image of the probe detection mark 30 in the image IM. The camera coordinate system is a coordinate system with the position of camera 16 as the origin. For example, it is a coordinate system defined by three axes: the direction of the optical axis of the lens of camera 16 as the z axis, the direction perpendicular to the z axis as the x axis, and the direction perpendicular to the x and z axes as the y axis. Note that a known method can be used to detect the position and orientation of the ultrasound probe 18a in the camera coordinate system from the image of the probe detection mark 30 included in the image IM, so a detailed explanation is omitted here. The position of the ultrasonic probe 18a may be expressed, for example, by the coordinates of a representative point of the ultrasonic probe 18a (for example, the attachment position of the probe detection mark 30), and the orientation of the ultrasonic probe 18a may be expressed, for example, by the rotation angle around each of the three axes of the camera coordinate system.

[0049] The probe position and orientation detection unit 64 may further detect the current position and orientation of the target tissue T.

[0050] The captured image IM also includes an image of an object detection mark 32 indicating the position and orientation of the target tissue T. The probe position and orientation detection unit 64 detects the position and orientation of the target tissue T in the camera coordinate system of the camera 16 by analyzing the image of the object detection mark 32 in the captured image IM. Since known methods can be used to detect the position and orientation of the target tissue T in the camera coordinate system from the image of the object detection mark 32 included in the captured image IM, a detailed explanation is omitted here. The position of the target tissue T may be represented, for example, by the coordinates of a representative point of the target tissue T (for example, the attachment position of the object detection mark 32), and the orientation of the target tissue T may be represented, for example, by the rotation angle around each of the three axis directions of the camera coordinate system.

[0051] Furthermore, the probe position and orientation detection unit 64 may detect the position and orientation of the ultrasonic probe 18a relative to the position and orientation of the target tissue T (surface). This makes it possible to obtain the position and orientation of the ultrasonic probe 18a relative to the target tissue T, while absorbing fluctuations in the position or orientation of the target tissue T.

[0052] The probe position and orientation detection unit 64 sequentially detects the current position and orientation of the ultrasound probe 18a based on the captured images IM (which may be frames constituting a moving image) sent sequentially in real time from the camera 16. In other words, the probe position and orientation detection unit 64 can be said to detect changes in the position and orientation of the ultrasound probe 18a.

[0053] The position and orientation of the ultrasonic probe 18a may be detected by methods other than analyzing the captured image IM. For example, the ultrasonic probe 18a may be equipped with a magnetic sensor or an acceleration sensor, and the processor 60 may acquire position and orientation information indicating the current position and orientation of the ultrasonic probe 18a from these sensors. In this case, the position and orientation of the ultrasonic probe 18a are expressed in coordinates in a real-space coordinate system with respect to an origin determined by sensor calibration.

[0054] The position and orientation information conversion unit 66 converts position and orientation information, which indicates the position and orientation of the ultrasonic probe 18a as defined in the camera coordinate system or real-space coordinate system, into position and orientation information in the model coordinate system. In this specification, the position and orientation information of the ultrasonic probe 18a converted to the model coordinate system is referred to as "converted position and orientation information". An example of a method for converting to converted position and orientation information is as follows.

[0055] Figure 6 shows an example of displaying medical tomographic images (MCI) and ultrasound tomographic images (USI). First, the image forming unit 44 forms a medical tomographic image (MCI) of a predetermined cross-section of the medical volume data received from the medical device 12 or the medical image analysis server 14. The predetermined cross-section may be one that the image forming unit 44 automatically selects to show characteristic structures in the medical volume data, or it may be a cross-section specified by the operator.

[0056] Furthermore, the image forming unit 44 forms an ultrasonic tomographic image (i.e., a real-time ultrasonic tomographic image) USI representing a cross-section of the target tissue T on the ultrasonic scanning plane, based on the received signal obtained by scanning the ultrasonic beam with the ultrasonic probe 18a at its current position and orientation over the target tissue T on the ultrasonic scanning plane.

[0057] The display control unit 46 then displays the formed medical tomography image (MCI) and ultrasound tomography image (USI) on the display 48. To facilitate comparison between the two images, the display control unit 46 may display the medical tomography image (MCI) and ultrasound tomography image (USI) side by side. Since the ultrasound tomography image (USI) is a real-time ultrasound tomography image, the ultrasound scanning plane changes when the position and orientation of the ultrasound probe 18a changes, so the content of the ultrasound tomography image (USI) changes dynamically according to the position or orientation of the ultrasound probe 18a.

[0058] Here, the operator compares the medical tomography image MCI and the ultrasound tomography image USI displayed on the display 48, and adjusts the position and orientation of the ultrasound probe 18a so that the cross-section of the ultrasound tomography image USI is the same as the cross-section of the medical tomography image MCI. When the cross-section of the ultrasound tomography image USI is the same as the cross-section of the medical tomography image MCI, the predetermined cross-section corresponding to the medical tomography image MCI specified in the medical volume data and the ultrasound transmitting and receiving wavefronts are the same cross-section. Here, the predetermined cross-section corresponding to the medical tomography image MCI is specified in the medical volume data, and can therefore be represented in the medical data coordinate system, i.e., the model coordinate system. On the other hand, the position and orientation of the ultrasound probe 18a are represented in the camera coordinate system or the real space coordinate system, and the ultrasound transmitting and receiving wavefronts are determined based on the position and orientation of the ultrasound probe 18a. Therefore, when the cross-section of the ultrasound tomography (USI) image becomes the same as the cross-section of the medical tomography (MCI) image, the operator can input a calibration instruction to the processor 60, thereby obtaining the relationship between the model coordinate system and the camera coordinate system or real-space coordinate system from the position and orientation relationship between a predetermined cross-section in the medical volume data at that time and the ultrasound scanning plane (i.e., the ultrasound probe 18a). In other words, the position and orientation information conversion unit 66 can convert position and orientation information indicating the position and orientation of the ultrasound probe 18a into converted position and orientation information in the model coordinate system.

[0059] The scanned area identification unit 68 identifies scanned areas in the 3D model 56 representing the target tissue T, which are areas where the ultrasound beam has been scanned (i.e., where a received signal has been acquired). The details of the processing of the scanned area identification unit 68 will be explained below with reference to Figures 7 to 10.

[0060] Figure 7 shows a first example of the model scanning plane MCS in the model coordinate system. Figure 7 shows the position and orientation of the ultrasonic probe 18a in the model coordinate system (i.e., the position and orientation indicated by the transformed position and orientation information). As described above, the ultrasonic scanning plane is determined based on information known to the ultrasonic diagnostic device 18, such as the scan width and penetration depth, as well as the position and orientation of the ultrasonic probe 18a. Therefore, the scanned area identification unit 68 identifies the model scanning plane MCS, which is the ultrasonic scanning plane in the model coordinate system, based on the known information and the transformed position and orientation information. In this specification, in order to distinguish between the ultrasonic scanning plane in the actual space where the ultrasonic beam is actually scanned and the virtual ultrasonic scanning plane identified in the model coordinate system, the ultrasonic scanning plane in the actual space is simply referred to as the ultrasonic scanning plane, and the virtual ultrasonic scanning plane in the model coordinate system is referred to as the model scanning plane MCS.

[0061] When the ultrasonic beam is scanned on the ultrasonic scanning surface corresponding to the model scanning surface MCS shown in Figure 7 (in other words, when a transmission signal is supplied from the transmitting / receiving unit 40 to the ultrasonic probe 18a), the scanned area identification unit 68 identifies the region of the 3D model 56 that the identified model scanning surface MCS passes through as the scanned area. In other words, the scanned area identification unit 68 identifies the cross-section of the 3D model 56 defined by the model scanning surface MCS as the scanned area.

[0062] Figure 8 shows a first example of a scanned region SA. When the scanned region identification unit 68 identifies a model scanning plane MCS as shown in Figure 7, it identifies the region of the 3D model 56 that the model scanning plane MCS passes through as the scanned region SA. In Figure 8 (and Figures 10 and 11), the scanned region SA is indicated by diagonal lines. In the example in Figure 8, the scanned region SA is the cross-section of the 3D model 56 by the model scanning plane MCS. Note that the model scanning plane MCS shown in Figure 7 has a sufficient scan width and penetration depth to traverse the 3D model 56, so in the example in Figure 8, the entire cross-section of the 3D model 56 by the model scanning plane MCS is the scanned region SA. However, if the model scanning plane MCS is too small to traverse the 3D model 56 (small scan width or shallow penetration depth), then only a part of the cross-section of the 3D model 56 by the model scanning plane MCS will be identified as the scanned region SA. The scanned area identification unit 68 stores information indicating the scanned area SA (the coordinate region of the scanned area SA) in the memory 54.

[0063] When the operator changes the position or orientation of the ultrasonic probe 18a, the position or orientation of the ultrasonic probe 18a in the model coordinate system changes, and the position and orientation of the model scanning plane MCS also change. Figure 9 shows a second example of the model scanning plane MCS in the model coordinate system. When the ultrasonic beam is scanned on the ultrasonic scanning plane corresponding to the model scanning plane MCS shown in Figure 9, the scanned area identification unit 68 again identifies the region of the 3D model 56 that the model scanning plane MCS passes through as the scanned area SA.

[0064] Figure 10 shows a second example of a scanned region SA. When the scanned region identification unit 68 identifies a model scanning plane MCS as shown in Figure 9, it identifies the region of the 3D model 56 that the model scanning plane MCS passes through as a scanned region SA (SA2 in Figure 10). The scanned region identification unit 68 also stores information indicating the scanned region SA2 in the memory 54.

[0065] Figure 11 shows a third example of scanned region SA. When the ultrasonic probe 18a gradually moves the ultrasonic scanning plane while scanning the ultrasonic beam from the position shown in Figure 7 to the position shown in Figure 9, the entire region of the 3D model 56 between scanned region SA1 and scanned region SA2 is identified as scanned region SA, as shown in Figure 11.

[0066] Thus, the scanned region identification unit 68 can identify one or more independent (non-contiguous) surfaces (cross-sections) in the three-dimensional model 56 as scanned regions SA, or it can identify a three-dimensional scanned region SA as a collection of such surfaces.

[0067] The received signal evaluation unit 70 evaluates the received signal obtained by scanning the ultrasonic probe 18a with an ultrasonic beam over the ultrasonic scanning surface corresponding to the model scanning surface MCS identified by the scanned area identification unit 68 (i.e., defining the scanned area SA). More specifically, the received signal evaluation unit 70 evaluates the received signal from the perspective of whether the ultrasonic tomographic image formed based on the received signal is sufficiently visible.

[0068] Various methods can be considered for evaluating the received signal, but for example, it can be done using a learning model for image analysis such as U-net. U-net is, for example, trained to take an ultrasound tomography image as input and detect artifacts such as side shadows that appear in the ultrasound tomography image. Therefore, the received signal evaluation unit 70 inputs the ultrasound tomography image formed by the image forming unit 44 based on the received signal to the trained U-net, and detects artifacts and other issues occurring in the ultrasound tomography image based on the output of U-net. Then, the received signal evaluation unit 70 evaluates the received signal corresponding to the ultrasound tomography image according to the quantity and quality of the detected artifacts.

[0069] The display control unit 46 displays the 3D model 56 on the display 48. The display control unit 46 also displays the scanned area SA identified by the scanned area identification unit 68 and the unscanned area, which is the area where the ultrasonic beam has not been scanned, in different display modes on the 3D model 56.

[0070] Figure 12 shows a first example of the display of the 3D model 56. In the example in Figure 12, the display control unit 46 displays the 3D model 56 in a manner in which scanned areas SA are filled with a predetermined color (represented by shading in Figure 12) and unscanned areas NSA are not filled. In the example in Figure 12, the scanned area SA is a single surface, but as shown in Figure 11, if a 3D area within the 3D model 56 is identified as the scanned area SA, the display control unit 46 displays the 3D model 56 in which the 3D scanned area SA is filled with a predetermined color.

[0071] The display control unit 46 displays the scanned area SA and the unscanned area NSA in different display modes, allowing the user to easily understand the area in the target tissue T that has been scanned by the ultrasound beam.

[0072] As described above, the 3D model 56 contains information indicating the coordinate region occupied by the area of ​​interest (tumor TM in this embodiment). Therefore, the display control unit 46 should display information indicating the position of the area of ​​interest on the display 48. In this embodiment, as shown in Figure 12, the display control unit 46 indicates the region occupied by the tumor TM by displaying the tumor image TMI in the coordinate region indicated by the tumor TM on the 3D model 56.

[0073] By displaying scanned areas (SA) and unscanned areas (NSA) in different display modes, and also displaying tumor image (TMI), the user can easily determine whether or not the ultrasound beam has been scanned over the tumor (TM), which is a part of interest. In particular, as shown in Figure 12, when multiple tumors (TM) are scattered, the display control unit 46 displays multiple tumor image (TMI), allowing the user to easily identify tumors (TM) that have not yet been scanned by the ultrasound beam.

[0074] Figure 13 shows a second display example of the 3D model 56. The display control unit 46 should display the scanned region SA in such a way that the evaluation result of the received signal evaluation unit 70 for the received signal obtained from the ultrasonic scanning surface CS corresponding to the model scanning surface MCS that defines the scanned region SA can be seen. For example, when filling the scanned region SA with a predetermined color, the display control unit 46 should display the scanned region SA in such a way that the color becomes darker the better the evaluation result of the received signal evaluation unit 70 is, and lighter the color the worse the evaluation result is. For example, in the example in Figure 13, the color of scanned region SA2 is lighter than that of scanned region SA1, indicating that the evaluation result for the received signal corresponding to scanned region SA2 is poor.

[0075] By displaying the scanned area SA so that the evaluation results of the received signal evaluation unit 70 can be seen, the user can easily identify areas that require further scanning of the ultrasound beam, even if those areas have already been scanned. For example, even if a tumor TM has been scanned with an ultrasound beam, if many artifacts occur in the ultrasound tomographic image formed from the received signal obtained, the user may not be able to clearly identify the tumor TM. In such cases, it is preferable to scan the tumor again with the ultrasound beam. In such cases, the user can easily understand that further scanning of the tumor with the ultrasound beam is necessary.

[0076] In this embodiment, the display control unit 46 displays a 3D model 56 in which the scanned area SA is filled, thereby displaying the scanned area SA and the unscanned area NSA in different display modes. However, the method for displaying the scanned area SA and the unscanned area NSA in different display modes is not limited to this.

[0077] Figure 14 shows a third display example of the 3D model 56. For example, as shown in Figure 14, the display control unit 46 may display an ultrasonic tomographic image USI formed by the image forming unit 44 based on a received signal obtained by scanning an ultrasonic beam across an ultrasonic scanning plane corresponding to the model scanning plane MCS, on a cross section as a scanned region SA defined by the model scanning plane MCS. If a 3D scanned region SA, which is a collection of such cross sections, is identified, the display control unit 46 may display the 3D scanned region SA as ultrasonic volume data.

[0078] Figure 15 shows an example of displaying the 3D model 56, the medical tomographic image (MCI), and the ultrasound tomographic image (USI). The display control unit 46 may display the 3D model 56, the medical tomographic image (MCI), and the ultrasound tomographic image (USI), which indicate the scanned region (SA), on the display 48. The medical tomographic image (MCI) is an image reconstructed by cutting out medical volume data received from the medical device 12 or the medical image analysis server 14 using a cross-section based on the transformed position and orientation information of the ultrasound probe 18a. The ultrasound tomographic image (USI) is an ultrasound tomographic image (i.e., a real-time ultrasound tomographic image) formed based on the received signal acquired at the current position and orientation of the ultrasound probe 18a.

[0079] When the position and orientation information of the ultrasound probe 18a is converted into converted position and orientation information, the image forming unit 44 can identify a cross-section of medical volume data that represents the same cross-section as the current position and orientation of the ultrasound probe 18a (i.e., the current ultrasound transmitting and receiving wavefront). The image forming unit 44 can then form a medical tomographic image (MCI) of the identified cross-section. Similarly, the image forming unit 44 can form a (real-time) ultrasound tomographic image (USI) corresponding to the current ultrasound transmitting and receiving wavefront. The display control unit 46 can display the medical tomographic image (MCI) and ultrasound tomographic image (USI) thus formed on the display 48. In other words, once the position and orientation information of the ultrasound probe 18a is converted into converted position and orientation information, it becomes possible to always display the medical tomographic image (MCI) and ultrasound tomographic image (USI) representing the current ultrasound scanning plane in synchronous manner. That is, when the ultrasound scanning plane changes, both the medical tomographic image (MCI) and the ultrasound tomographic image (USI) will represent the changed ultrasound scanning plane. Both images will always represent the same cross-section.

[0080] By displaying the 3D model 56 showing the scanned region SA, the medical tomographic image MCI, and the ultrasound tomographic image USI on the display 48, the operator can operate the ultrasound probe 18a so that the target region becomes the scanned region SA in the 3D model 56 while confirming the medical tomographic image MCI or ultrasound tomographic image USI corresponding to the current ultrasound scanning plane.

[0081] Furthermore, the display control unit 46 may display a guide image GF superimposed on the medical tomographic image MCI to indicate the range of the ultrasound scanning plane. Since the medical tomographic image MCI is constructed by reconstructing medical volume data, it is a rectangular image. On the other hand, when the ultrasound probe 18a performs sector scanning, the ultrasound beam is operated in a fan shape, so the ultrasound scanning plane (i.e., the area in the ultrasound tomographic image USI where the appearance of the target tissue T is shown) becomes wider as the depth increases. The range of the ultrasound scanning plane in the medical tomographic image MCI indicates the range in which the ultrasound beam can be scanned at the position and orientation of the ultrasound probe 18a corresponding to the cross-section of the medical tomographic image MCI.

[0082] By displaying a guide image GF superimposed on the medical tomographic image MCI, users can easily understand which areas of the medical tomographic image MCI are the scanned areas SA.

[0083] The treatment support device relating to this disclosure has been described above, but the treatment support device relating to this disclosure is not limited to the embodiments described above, and various modifications are possible as long as they do not deviate from the spirit of the disclosure.

[0084] For example, in this embodiment, the ultrasound probe 18a was a drop-in type probe, but the ultrasound probe 18a may be of other types. For example, the ultrasound probe 18a may be a probe that is in contact with the body surface of the subject.

[0085] Furthermore, in each of the above embodiments, the functions of the image forming unit 44, the display control unit 46, the 3D model forming unit 62, the probe position and orientation detection unit 64, the position and orientation information conversion unit 66, the scanned area identification unit 68, and the received signal evaluation unit 70 were all provided by the ultrasound diagnostic device 18. However, these functions do not necessarily have to be performed by the ultrasound diagnostic device 18. For example, these functions may be performed by other devices such as the medical image analysis server 14. Also, instead of one device performing all of the above functions, these functions may be performed through the cooperation of multiple devices.

[0086] In this embodiment, each process is executed on any computer. Furthermore, any computer may execute these processes using a processor as hardware, a program as software, or a combination thereof. In that case, the processor is configured to work in cooperation with the program to execute the various processes in this embodiment, and can function as a unit or means in this embodiment. Also, the execution order of the processes by the processor is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other system capable of executing each process. A processor may consist of one or more hardware components, and the type of hardware is not limited. For example, a processor may consist of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a programmable logic device such as an FPGA (Field Programmable Gate Array), a dedicated circuit for executing a specific process such as an ASIC (Application Specific Integrated Circuit), a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). Furthermore, the type of hardware may be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of a given processor, these components may reside in physically separate devices or in the same device. Also, in any embodiment, the order of each process performed by the processor is not limited to the order described above and may be changed as appropriate. Hardware is composed of electrical circuits (circuitry) that combine circuit elements such as semiconductor elements. Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a set of program modules, each function of which may be implemented by a processor configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located on physically separate devices. Program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0087] This invention can also be applied to programs and program products. [Explanation of Symbols]

[0088] 10 Ultrasound diagnostic support system, 12 Medical device, 14 Medical image analysis server, 16 Camera, 18 Ultrasound diagnostic device, 18a Ultrasound probe, 30 Probe detection mark, 32 Object detection mark, 40 Transmitting / receiving unit, 42 Signal processing unit, 44 Image forming unit, 46 Display control unit, 48 Display, 50 Communication interface, 52 Input interface, 54 Memory, 56 3D model, 60 Control unit, 62 3D model forming unit, 64 Probe position and orientation detection unit, 66 Position and orientation information conversion unit, 68 Scanned area identification unit, 70 Received signal evaluation unit.

Claims

1. Equipped with a processor, The aforementioned processor, A three-dimensional model of a target tissue formed based on medical volume data acquired by a medical device other than an ultrasound diagnostic device, wherein each position of the three-dimensional model is represented by coordinates in the model coordinate system, Position and orientation information is obtained that indicates the position and orientation of the ultrasound probe scanning ultrasound in the ultrasound scanning surface including the target tissue. The position and orientation information is converted into transformed position and orientation information in the model coordinate system. Based on the transformed position and orientation information obtained when scanning the target tissue with an ultrasonic beam, a model scanning plane, which is a virtual ultrasonic scanning plane in the model coordinate system, is identified, and the region of the 3D model through which the model scanning plane passes is identified as the scanned region. A display control unit that displays the three-dimensional model on a display unit, wherein the scanned area and the unscanned area, which is an area where the ultrasonic beam has not been scanned, are displayed in different display modes in the three-dimensional model. It is structured in such a way. An ultrasound diagnostic support device characterized by the following features.

2. The aforementioned three-dimensional model is further enhanced with information indicating the location of the area of ​​interest within the target tissue. The processor is configured to display information indicating the location of the part of interest on the display unit. The ultrasound diagnostic support device according to feature 1.

3. The aforementioned processor, The received signal obtained by scanning the ultrasonic beam with the ultrasonic scanning surface corresponding to the model scanning surface that defines the scanned region is evaluated. The scanned area is displayed in a manner that shows the evaluation result of the received signal. It is structured in such a way. The ultrasound diagnostic support device according to feature 1.

4. The processor is configured to display an ultrasonic tomographic image formed based on a received signal obtained by scanning an ultrasonic beam with an ultrasonic scanning plane corresponding to the model scanning plane, on a cross-section of the scanned region defined by the model scanning plane. The ultrasound diagnostic support device according to feature 1.

5. The aforementioned processor, The medical tomographic image reconstructed by cutting out the medical volume data in a cross-section based on the conversion position and orientation information, and the ultrasound tomographic image formed based on the received signal acquired at the current position and orientation of the ultrasound probe, are displayed on the display unit. A guide image indicating the range of the ultrasound scanning surface is superimposed on the aforementioned medical tomographic image. It is structured in such a way. The ultrasound diagnostic support device according to feature 1.

6. The processor is configured to detect the relative position and orientation of the ultrasound probe with respect to the position and orientation of the target tissue, based on images obtained by capturing the ultrasound probe and detection marks attached to the target tissue with a camera. The ultrasound diagnostic support device according to feature 1.

7. On the computer, A three-dimensional model of a target tissue formed based on medical volume data acquired by a medical device other than an ultrasound diagnostic device, wherein each position of the three-dimensional model is represented by coordinates in the model coordinate system, Position and orientation information indicating the position and orientation of the ultrasound probe scanning ultrasound waves in the ultrasound scanning surface including the target tissue is acquired. The position and orientation information is converted into transformed position and orientation information in the model coordinate system. Based on the transformed position and orientation information obtained when scanning the target tissue with an ultrasonic beam, the ultrasonic scanning plane in the model coordinate system is identified, and the region of the 3D model through which the ultrasonic scanning plane passes is identified as the scanned region. The three-dimensional model is displayed on the display unit, and in the three-dimensional model, the scanned area and the unscanned area, which is the area where the ultrasonic beam has not been scanned, are displayed in different display modes. An ultrasound diagnostic support program characterized by the following features.

Images