Treatment support device and treatment support program

Abstract

To allow the operator to easily identify the incision site on the subject's body surface for performing treatment on the target tissue within the subject. [Solution] Camera 12 captures the body surface portion near the tumor TM within the subject E and forms a display image. Volume data acquisition unit 50 acquires ultrasonic volume data 60 corresponding to the ultrasonic irradiation area, which is formed based on the received signal obtained by transmitting and receiving ultrasonic waves in the ultrasonic irradiation area within the subject E, including the tumor TM, from an ultrasonic probe 14 in contact with the body surface portion of the subject E. Reconstruction processing unit 54 cuts out the ultrasonic volume 60 at a cross section CS that is perpendicular to the depth direction of the ultrasonic volume data 60 and includes the treatment target area (the area occupied by the tumor TM), and reconstructs it. Display control unit 36 ​​displays the display image 24B and the reconstructed ultrasonic image 70 on the display 38.

Description

[Technical Field]

[0001] This specification discloses improvements to a treatment support device and a treatment support program. [Background technology]

[0002] An ultrasound diagnostic device is known that transmits ultrasound waves from an ultrasound probe in contact with the surface of a subject's body to the subject, receives reflected waves from the subject with the ultrasound probe, and forms an ultrasound tomographic image of the subject based on the received signal formed from the reflected waves.

[0003] Traditionally, when performing procedures on a subject (such as surgery), an ultrasound diagnostic device has been used by a physician or other operator to examine the inside of the subject.

[0004] For example, Patent Document 1 discloses an image processing device that acquires a three-dimensional image (volume data) representing the tissue to be treated within a subject, formed by an ultrasound diagnostic device, and renders the three-dimensional image from a lateral viewpoint that views the cutting surface from a line of sight direction intersecting the normal to the cutting surface of the tissue to be treated (i.e., from the side of the cutting surface), thereby forming and displaying a lateral viewpoint image, which is an ultrasound image of the cutting surface viewed from the side. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2024-52409 [Overview of the project] [Problems that the invention aims to solve]

[0006] Incidentally, if the tissue to be treated is located within the subject, the operator must make an incision in the subject's body surface before performing the procedure on the tissue. From the perspective of reducing invasiveness to the subject, it is desirable that the incision on the subject's body surface be in the vicinity of the tissue to be treated, such as directly above the tissue (in the depth direction).

[0007] Needless to say, before incising the subject's body surface, the operator cannot see the tissue to be treated. Therefore, conventionally, the operator would examine an ultrasound image showing the tissue to be treated before incision and determine the incision site on the subject's body surface based on the ultrasound image.

[0008] Traditionally, identifying the incision site from ultrasound images has sometimes been time-consuming or difficult. For example, when an ultrasound probe is placed near the incision site and ultrasound waves are transmitted and received against the tissue to be treated, the resulting ultrasound image represents the tissue to be treated as seen from the side of the operator's line of sight facing the incision site. Therefore, identifying the incision site from this ultrasound image sometimes required the operator's experience.

[0009] The purpose of the treatment support device disclosed herein is to enable the operator to easily determine the location of an incision on the body surface of the subject for performing treatment on the target tissue within the subject. [Means for solving the problem]

[0010] The treatment support device disclosed herein includes an image acquisition unit that acquires a display image obtained by photographing a portion of the body surface near the tissue to be treated within the subject, and an ultrasound probe that contacts the body surface portion and provides ultrasound irradiation in the ultrasound irradiation area within the subject, including the tissue to be treated. The device is characterized by comprising: a volume data acquisition unit that acquires ultrasonic volume data corresponding to the ultrasonic irradiation area, formed based on a received signal obtained by transmitting and receiving sound waves; a treatment area identification unit that identifies a treatment area, which is the area occupied by the treatment target tissue in the ultrasonic volume data; a reconstruction processing unit that forms a reconstructed ultrasonic image by cutting out the ultrasonic volume data at a cross section that is perpendicular to the depth direction of the ultrasonic volume data and includes the treatment target area, and reconstructing it; and a display control unit that displays the display image and the reconstructed ultrasonic image on a display unit.

[0011] The system further includes a probe position and orientation information acquisition unit that acquires position and orientation information indicating the position and orientation of the ultrasound probe when the ultrasound volume data is acquired, the image acquisition unit acquires a detection image obtained by photographing a probe detection marker attached to the ultrasound probe in contact with the body surface, the volume data acquisition unit acquires the ultrasound volume data having coordinate information representing each position of the ultrasound volume data with respect to the position and orientation information in the camera coordinate system of the camera that took the display image, and the display control unit displays a partial image on the display unit, which is the part of the display image corresponding to the reconstructed ultrasound image, based on the coordinate information contained in the ultrasound volume data.

[0012] The display control unit may display the partial image and the reconstructed ultrasound image on the display unit with their orientations and sizes aligned.

[0013] The display control unit may display a position indicator on the display unit that shows the position in the partial image corresponding to the position of the treatment target area in the reconstructed ultrasound image.

[0014] The partial image includes an image of a surface feature on the body surface of the subject, and the treatment support device further comprises a distance measuring unit that measures the distance between the surface feature and the position indicated by the position indicator in real space based on the distance between the image of the surface feature and the position indicator in the partial image, and the display control unit may cause the distance measured by the distance measuring unit to be displayed on the display unit.

[0015] The reconstruction processing unit may determine the position of the cut-out cross section in the depth direction to the position where the area of ​​the treatment target region in the cut-out cross section is maximized.

[0016] The reconstruction processing unit may form the reconstructed ultrasound image based on partial volume data, which is a part of the ultrasound volume data and includes the cut-out cross section, having thickness in the depth direction of the ultrasound volume data.

[0017] The volume data acquisition unit acquires the ultrasound volume data including Doppler signals indicating the blood flow velocity of the subject at each position; the reconstruction processing unit forms a blood flow image representing the blood flow of the subject in the excised cross section based on the Doppler signals; the blood flow image is superimposed on the reconstructed ultrasound image to form a reconstructed Doppler image; and the display control unit displays the reconstructed Doppler image on the display unit.

[0018] Furthermore, the treatment support device disclosed herein includes a volume data acquisition unit that acquires ultrasonic volume data corresponding to the ultrasonic irradiation area, which is formed based on a received signal obtained by transmitting and receiving ultrasonic waves in an ultrasonic irradiation area within the subject including the target tissue from an ultrasonic probe in contact with the body surface portion near the target tissue within the subject, and which includes a Doppler signal indicating the blood flow velocity of the subject at each position; a treatment area identification unit that identifies a treatment area corresponding to the target tissue in the ultrasonic volume data; and a unit perpendicular to the depth direction of the ultrasonic volume data. A cut-out cross-section that is a cross-section including the treatment target area, forms a reconstructed ultrasonic image by cutting out and reconstructing the ultrasonic volume data including the Doppler signal, forms a blood flow image representing the blood flow of the subject in the cut-out cross-section based on the Doppler signal, and a reconstruction processing unit that forms a reconstructed Doppler image in which the blood flow image is superimposed on the reconstructed ultrasonic image, and a display control unit that causes the reconstructed Doppler image to be displayed on a display unit.

[0019] The treatment support program disclosed in this specification causes a computer to function as an imaging image acquisition unit that acquires a display imaging image obtained by imaging a surface part of the body near a treatment target tissue in a subject, and the treatment target tissue from an ultrasonic probe that abuts on the surface part. A volume data acquisition unit that acquires ultrasonic volume data corresponding to the ultrasonic irradiation region, which is formed based on a reception signal obtained by transmitting and receiving ultrasonic waves in the ultrasonic irradiation region in the subject including the treatment target tissue, and the treatment target tissue in the ultrasonic volume data. A treatment target region specifying unit that specifies a treatment target region that is the region occupied by the treatment target tissue, and a reconstruction processing unit that forms a reconstructed ultrasonic image by cutting out and reconstructing the ultrasonic volume data in a cut-out cross-section that is perpendicular to the depth direction of the ultrasonic volume data and includes the treatment target region, and a display control unit that causes the display imaging image and the reconstructed ultrasonic image to be displayed on a display unit.

Effect of the Invention

[0020] According to the treatment support device disclosed in this specification, an operator can easily grasp a cut position on the surface of the subject for performing treatment on a treatment target tissue in the subject.

Brief Description of the Drawings

[0021] [Figure 1] It is a schematic configuration diagram of a treatment support system according to this embodiment. [Figure 2] It is a diagram showing an example of a detection imaging image. [Figure 3]This figure shows an example of a photographed image for display. [Figure 4] This is a schematic diagram of the configuration of the ultrasound diagnostic apparatus according to this embodiment. [Figure 5] This is a conceptual diagram illustrating the process of forming ultrasonic volume data. [Figure 6] This figure shows a cross-section of the volume data obtained using ultrasound. [Figure 7] This is a diagram showing multiple cross-sections. [Figure 8] This figure shows the data portion used to form the reconstructed ultrasound image. [Figure 9] This figure shows an example of a reconstructed ultrasound image. [Figure 10] This figure shows examples of displaying captured images and reconstructed ultrasound images. [Figure 11] This is a diagram showing a partial image of a photograph taken for display purposes. [Figure 12] This figure shows an example of a location indicator display. [Figure 13] This figure shows an example of how the distance between the nipple and the recommended incision site is displayed. [Figure 14] This figure shows an example of a reconstructed Doppler image. [Figure 15] This figure shows an example of a reconstructed Doppler image. [Figure 16] This figure shows an example of displaying captured images and reconstructed Doppler images. [Modes for carrying out the invention]

[0022] Figure 1 is a schematic diagram of the configuration of the treatment support system 10 according to this embodiment. The treatment support system 10 is composed of a camera 12 and an ultrasound diagnostic device 16 as a treatment support device including an ultrasound probe 14. The camera 12 and the ultrasound diagnostic device 16 are connected to each other so as to be able to communicate with each other.

[0023] In this embodiment, the ultrasound probe 14 is fitted with a probe detection marker 20. The probe detection marker 20 is a marker for detecting the position and orientation of the ultrasound probe 14. In addition, a body surface detection marker 22 is fitted to the body surface of the subject E. The output marker 22 has a different pattern from the probe detection marker 20 and is a marker for detecting the position and orientation of the subject E on the body surface. An example of the probe detection marker 20 and the body surface detection marker 22 is an AR (Argumented Reality) marker.

[0024] Camera 12 includes a lens, an image sensor, and a CPU (Central Processing Unit). The system includes a processor and a communication interface, such as a network adapter. The camera 12 captures images of the ultrasound probe 14 (specifically, the probe detection marker 20) and the subject E (specifically, the body surface detection marker 22). The position and orientation of the ultrasound probe 14 are detected by the ultrasound diagnostic device 16 using the images obtained from such capture, as described later. In this specification, such captured images are referred to as detection images.

[0025] Figure 2 shows an example of a detection image 24A. As described above, the detection image 24A includes images of the probe detection marker 20 and the body surface detection marker 22. The ultrasound diagnostic device 16 can detect the position and orientation of the ultrasound probe 14 by analyzing the image of the probe detection marker 20 captured in the detection image 24A. Furthermore, the ultrasound diagnostic device 16 can detect the position and orientation of the body surface of the subject E by analyzing the image of the body surface detection marker 22 captured in the detection image 24A. Details of the detection process for the position and orientation of the ultrasound probe 14 and subject E will be described later.

[0026] Furthermore, camera 12 captures images of the body surface area near the tissue to be treated (details described later) within the subject E. In this specification, such captured images are referred to as display images.

[0027] Figure 3 shows an example of a display image 24B. Although not limited to this, this embodiment describes an example where the tissue to be treated is a tumor TM located in the breast of subject E. Therefore, in this embodiment, as shown in Figure 3, the display image 24B includes an image of the body surface area near the tumor TM, i.e., the area around the breast. In particular, the display image 24B is an image of the area around the breast taken from the front of subject E. If the body surface area near the tissue to be treated is shown in the detection image 24A, the detection image 24A may be used as the display image 24B.

[0028] In this specification, the concept encompassing the detection image 24A and the display image 24B is simply referred to as the "captured image." The position and orientation of the camera 12 when capturing the detection image 24A and the position and orientation of the camera 12 when capturing the display image 24B are kept the same.

[0029] The image sensor of camera 12 forms an image, and the image is transmitted to the ultrasound diagnostic device 16 via the communication interface of camera 12.

[0030] Figure 4 is a schematic diagram of the ultrasound diagnostic device 16. The ultrasound diagnostic device 16 is a medical device installed in medical institutions such as hospitals.

[0031] The ultrasonic probe 14 is a device that transmits and receives ultrasonic waves to a subject E. The ultrasonic probe 14 has a vibrating element array consisting of multiple vibrating elements that transmit and receive ultrasonic waves to the subject E. The vibrating element array is formed from multiple vibrating elements arranged in one direction (array direction). A transmission signal is supplied to each vibrating element from the transmitting / receiving unit 30 (described later), causing each vibrating element to generate ultrasonic waves. Specifically, the ultrasonic probe 14 scans an ultrasonic beam on a plane (scanning plane) parallel to the array direction.

[0032] As described above, the ultrasonic probe 14 is equipped with a probe detection marker 20.

[0033] The transmitting / receiving unit 30 transmits a transmission signal to the ultrasonic probe 14 (specifically, each vibrating element in the vibrating element array) under control from the control unit 46, which will be described later. As a result, an ultrasonic beam is scanned from the ultrasonic probe 14 across the scanning plane. The transmitting / receiving unit 30 also receives received signals from each vibrating element that receives reflected waves from the subject E. The transmitting / receiving unit 30 has an adder and a plurality of delays corresponding to each vibrating element, and performs phase-aligned addition processing by using the adder and the plurality of delays to align the phases of the received signals from each vibrating element and add them together. As a result, a received beam signal is formed in which information indicating the signal intensity of the reflected wave from the subject E is aligned in the depth direction of the subject E. Multiple received beam signals corresponding to one scanning plane constitute frame data.

[0034] The signal processing unit 32 performs various signal processing operations on the received beam signal from the transmitting / receiving unit 30, including filtering and detection, which involves applying a bandpass filter.

[0035] The image forming unit 34 forms an ultrasonic tomographic image (B-mode image) representing a cross-section of the subject E (particularly the scanning plane of the ultrasonic beam) based on the received beam signal processed by the signal processing unit 32.

[0036] The display control unit 36 ​​controls the display of the ultrasound tomography image formed by the image forming unit 34 on the display 38. As will be described in more detail later, the display control unit 36 ​​also displays the reconstructed ultrasound image and the display image 24B formed by the control unit 46 on the display 38.

[0037] The display unit, the display 38, is a display device composed of, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

[0038] The ultrasound diagnostic apparatus 16 comprises a transmitting / receiving unit 30, a signal processing unit 32, an image forming unit 34, and a display control unit 36, all of which are comprised of a processor. The processor includes at least one general-purpose processing unit (e.g., a CPU) and a dedicated processing unit (e.g., a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a programmable logic device). The processor may not consist of a single processing unit, but rather a combination of multiple processing units located in physically separate locations. Furthermore, each of the above components may be realized through the cooperation of hardware such as a processor and software.

[0039] The communication interface 40 is composed of, for example, a network adapter. The communication interface 40 performs the function of communicating with other devices (especially the camera 12). In particular, the communication interface 40 receives captured images (detection captured image 24A and display captured image 24B) from the camera 12. Thus, in this embodiment, the communication interface 40 functions as an image acquisition unit.

[0040] The input interface 42 consists of, for example, buttons, a trackball, a touch panel, etc. The input interface 42 is used to input commands from an operator, such as a doctor, who is using the ultrasound diagnostic device 16 to the ultrasound diagnostic device 16.

[0041] The memory 44 consists of HDD (Hard Disk Drive), SSD (Solid State Drive), eMMC (embedded Multi Media Card), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The memory 44 contains each of the ultrasound diagnostic device 16 A procedure support program for operating the unit is stored. Note that the procedure support program is: For example, the program can be stored on a computer-readable non-temporary storage medium such as a USB (Universal Serial Bus) memory or a CD-ROM. The ultrasound diagnostic device 16 can read and execute the treatment support program from such a storage medium. Since the ultrasound diagnostic device 16 performs the functions described below by reading the treatment support program, it can be said that the ultrasound diagnostic device 16 is a computer program product.

[0042] The control unit 46 is composed of at least one general-purpose processor (e.g., a CPU) and a dedicated processor (e.g., a GPU, ASIC, FPGA, or programmable logic device). The control unit 46 may not consist of a single processing unit, but rather of multiple processing units located in physically separate locations working together. The control unit 46 controls each part of the ultrasound diagnostic device 16. As shown in Figure 4, the control unit 46 also functions as a probe position and orientation information acquisition unit 48, a volume data acquisition unit 50, a treatment target area identification unit 52, a reconstruction processing unit 54, and a distance measurement unit 56, according to the treatment support program stored in the memory 44.

[0043] The probe position and orientation information acquisition unit 48 acquires position and orientation information indicating the position and orientation of the ultrasound probe 14. In particular, the probe position and orientation information acquisition unit 48 acquires position and orientation information indicating the position and orientation of the ultrasound probe 14 when each frame data for forming the ultrasound volume data described later is acquired.

[0044] In this embodiment, the probe position and orientation information acquisition unit 48 acquires position and orientation information by analyzing the detection image 24A acquired by the camera 12 to detect the position and orientation of the ultrasonic probe 14. As described above, the detection image 24A includes an image of the probe detection marker 20 for detecting the position and orientation of the ultrasonic probe 14 (see Figure 2). The probe position and orientation information acquisition unit 48 acquires position and orientation information by analyzing the image of the probe detection marker 20 in the detection image 24A. The position of the ultrasonic probe 14 may be expressed, for example, in three-dimensional coordinates in the camera coordinate system. The orientation of the ultrasonic probe 14 may be expressed, for example, in rotation angles with respect to each of three predetermined orthogonal axes in the camera coordinate system. Note that a known method can be used to detect the position and orientation of the ultrasonic probe 14 in the camera coordinate system from the image of the probe detection marker 20 included in the detection image 24A, so a detailed explanation is omitted here.

[0045] As described above, the detection image 24A also includes an image of the body surface detection marker 22 for detecting the position and posture of the subject E's body surface (see Figure 2). The probe position and posture information acquisition unit 48 may detect the position and posture of the subject E's body surface by analyzing the image of the body surface detection marker 22 in the detection image 24A. Then, the probe position and posture information acquisition unit 48 may detect the position and posture of the ultrasound probe 14 relative to the position and posture of the subject E's body surface. This makes it possible to obtain the position and posture of the ultrasound probe 14 relative to the subject E, while absorbing fluctuations in the subject E's position or posture.

[0046] The probe position and orientation information acquisition unit 48 may detect the position and orientation of the ultrasonic probe 14 by means other than analyzing the detection image 24A. For example, the ultrasonic probe 14 may be equipped with position and orientation sensors such as a magnetic sensor or an acceleration sensor, and the position and orientation of the ultrasonic probe 14 may be detected based on the detected values ​​of the position and orientation sensors.

[0047] The position and orientation of the scanning plane over which the ultrasonic beam is scanned are determined by the position and orientation of the ultrasonic probe 14. Therefore, the position and orientation information acquired by the probe position and orientation information acquisition unit 48 can be said to indicate the position and orientation of the scanning plane corresponding to each frame data for forming the ultrasonic volume data. Furthermore, each position (coordinate) on the frame data is based on the position and orientation of the ultrasonic probe 14 at the time the frame data was acquired. Therefore, it can be identified. Thus, it can be said that each frame data is accompanied by coordinate information indicating the position (coordinate) of each frame data, based on the position and orientation of the ultrasound probe 14 when the frame data was acquired.

[0048] The probe position and orientation information acquisition unit 48 associates position and orientation information, which indicates the position and orientation of the ultrasonic probe 14 at the time the frame data was acquired, with the frame data acquired by the transmitting and receiving unit 30, and stores it in the memory 44. As a result, a sequence of frame data consisting of multiple frame data, each associated with position and orientation information, is stored in the memory 44.

[0049] The volume data acquisition unit 50 acquires ultrasonic volume data corresponding to the ultrasonic irradiation area, which is formed based on the received signal obtained by transmitting and receiving ultrasonic waves from the ultrasonic probe 14 in contact with the body surface of the subject E in the ultrasonic irradiation area within the subject E, including the tissue to be treated.

[0050] FIG. 5 is a conceptual diagram showing the concept of the formation process of ultrasonic volume data 60. In the present embodiment, the volume data acquisition unit 50 forms ultrasonic volume data 60 based on the frame data sequence 62 stored in the memory 44. The frame data sequence 62 is composed of a plurality of frame data 64. Each frame data 64 is data corresponding to the scanning surface of the ultrasonic probe 14. Specifically, each frame data 64 is obtained by the ultrasonic probe 14 contacting the subject E while being moved by the operator in the sweep direction, which is the direction perpendicular to the scanning surface, and transmitting and receiving ultrasonic waves to the subject E (particularly the treatment target tissue). In particular, in the present embodiment, since the treatment target tissue is the tumor TM in the breast, the ultrasonic probe 14 is brought into contact with the breast of the subject E so that the depth direction of the scanning surface of the ultrasonic probe 14 is substantially perpendicular to the coronal plane of the subject E (in other words, substantially parallel to the front-rear direction of the subject E). At least a part of the frame data 64 included in each frame data sequence 62 includes data related to the tumor TM (a signal indicating the signal intensity of the reflected wave from the tumor TM).

[0051] In FIG. 5, in the data space where the ultrasonic volume data 60 is defined, the X D axis direction, Y D axis direction, and Z D axis direction, which are orthogonal to each other, are shown. In the example of FIG. 5, the volume data acquisition unit 50 first makes the orientation of each frame data 64 parallel to the X D Y D plane. In particular, the depth direction of the frame data 64 is set to be parallel to the Y D axis. Then, the ultrasonic volume data 60 is formed by arranging and synthesizing the frame data 64 in the arrangement direction (the Z D Y D axis direction in the example of FIG. 5), which is the direction perpendicular to the X D Y

[0052] plane. As a method of forming the ultrasonic volume data 60 from the plurality of frame data 64, a known method can be used, so a detailed description is omitted here.The frame data 64 for forming the ultrasonic volume data 60 may be a plurality of received beam signals corresponding to a single scanning plane formed by the transmitting / receiving unit 30 (or a plurality of received beam signals processed by the signal processing unit 32), or it may be a single ultrasonic tomographic image formed by the image forming unit 34 based on the plurality of received beam signals.

[0053] As described above, each frame data 64 is associated with position and orientation information indicating the position and orientation of the ultrasound probe 14 when the frame data 64 was acquired, and has coordinate information indicating each position of each frame data 64 based on the position and orientation of the ultrasound probe 14. Therefore, the ultrasound volume data 60, which is composed of multiple frame data 64, also has coordinate information indicating each position of the ultrasound volume data 60 based on the position and orientation of the ultrasound probe 14 when each frame data 64 was acquired. In this embodiment, each position of the ultrasound volume data 60 is also a 3D coordinate in the camera coordinate system. This is how it is represented. In addition, since at least some of the frame data 64 included in each frame data column 62 contains data about the tumor TM, which is the tissue being treated, the ultrasound volume data 60 also contains data about the tumor TM.

[0054] In this embodiment, the ultrasonic volume data 60 was composed of multiple frame data 64. However, a 2D array probe in which vibrating elements are arranged in two dimensions may be used as the ultrasonic probe 14, and the volume data acquisition unit 50 may directly form the ultrasonic volume data 60 based on the three-dimensional received signal from the ultrasonic probe 14. Even in this case, the ultrasonic volume data 60 has coordinate information indicating each position of the ultrasonic volume data 60, based on the position and orientation of the ultrasonic probe 14 when the three-dimensional received signal was acquired.

[0055] The treatment area identification unit 52 identifies the treatment area, which is the region occupied by the treatment target tissue (tumor TM in this embodiment) in the ultrasound volume data 60. As for how to identify the treatment area, the treatment area identification unit 52 can use a learning model that has been trained to identify each tissue (e.g., organs or tumors) from the ultrasound volume data 60, which is input to the ultrasound volume data 60. Alternatively, the display control unit 36 ​​may display the ultrasound volume data 60 on the display 38, allowing the operator to input an instruction to identify the treatment area in the ultrasound volume data 60, and the treatment area identification unit 52 may identify the treatment area according to that instruction.

[0056] The reconstruction processing unit 54 determines a cross-section in the ultrasonic volume data 60, cuts out the ultrasonic volume data 60 at the determined cross-section, and reconstructs it to form a reconstructed ultrasonic image.

[0057] Figure 6 shows a cross-section CS in the ultrasonic volume data 60. In this embodiment, the reconstruction processing unit 54 processes the ultrasonic volume data 60 in the depth direction (Y D Ultrasound volume data 60 is extracted and reconstructed using a section CS that is perpendicular to the axial direction and includes the treatment area (in this embodiment, the area occupied by the tumor TM).

[0058] Figure 7 shows multiple excision sections CS. As shown in Figure 7, multiple excision sections CS can be defined that are perpendicular to the depth direction of the ultrasound volume data 60 and contain the tumor TM (for example, CS1 to CS4 in Figure 7). Here, the reconstruction processing unit 54 should determine the position of the excision section CS in the depth direction to the position where the area of ​​the tumor TM in the excision section CS is maximized. As described above, since the area occupied by the tumor TM in the ultrasound volume data 60 is identified by the treatment target area identification unit 52, the reconstruction processing unit 54 can calculate the area of ​​the tumor TM in the excision section CS. By determining the position of the excision section CS in the depth direction to the position where the area of ​​the tumor TM in the excision section CS is maximized, a reconstructed ultrasound image in which the tumor TM is more prominently represented can be formed.

[0059] Figure 8 shows the data portion used to form a reconstructed ultrasound image. The reconstruction processing unit 54 may form a reconstructed ultrasound image based on partial volume data 60a, which is a part of the ultrasound volume data 60 and has a thickness t in the depth direction, and includes an excision section CS (preferably the excision section CS where the area of ​​the tumor TM is maximized). By forming a reconstructed ultrasound image based on partial volume data 60a with a thickness t in the depth direction, a reconstructed ultrasound image with a sense of depth can be formed. However, if the thickness t is too large, the image quality may deteriorate due to blurring of the reconstructed ultrasound image, and if the thickness t is too small, the sense of depth will be reduced, so the reconstruction processing unit 54 may set the thickness t appropriately. For example, the thickness t may be set based on the size of the tumor TM in the depth direction.

[0060] Figure 9 shows an example of a reconstructed ultrasound image 70. The reconstructed ultrasound image 70 is an ultrasound tomographic image including the tumor TM, which is the tissue to be treated. As described above, since the excision section CS is a plane perpendicular to the depth direction of the ultrasound volume data 60, the reconstructed ultrasound image 70 is an image representing a cross-section of the subject E viewed from the depth direction. In this embodiment, as described above, the depth direction of the ultrasound volume data 60 is approximately perpendicular to the coronal plane of the subject E, and the excision section CS is also approximately perpendicular to the coronal plane of the subject E, so the reconstructed ultrasound image 70 is an image representing a cross-section viewed from the depth direction, that is, a cross-section parallel to the coronal plane of the subject E.

[0061] Figure 10 shows an example of displaying the captured image 24B and the reconstructed ultrasound image 70. The display control unit 36 ​​displays the captured image 24B and the reconstructed ultrasound image 70 on the display 38. As described above, the captured image 24B is a photograph of the body surface of the subject E (in other words, the optical axis of the lens of the camera 12 is approximately parallel to the depth direction of the subject E), while the reconstructed ultrasound image 70 is an image representing a cross-section viewed from the depth direction. Therefore, the captured image 24B and the reconstructed ultrasound image 70 appear as if they were viewed from the same line of sight. In this embodiment, the captured image 24B is an image of the subject E taken from the front, and the reconstructed ultrasound image 70 is an image representing a cross-section parallel to the coronal plane of the subject E. Therefore, both the captured image 24B and the reconstructed ultrasound image 70 appear as if the subject E were viewed from the front.

[0062] Since the display image 24B is a photograph of the body surface of subject E, the location of tumor TM is naturally not shown in the display image 24B. On the other hand, the reconstructed ultrasound image 70 shows the location of tumor TM. Therefore, by comparing the display image 24B and the reconstructed ultrasound image 70, which are images viewed from the same line of sight, the operator can identify the location corresponding to tumor TM on the body surface of subject E. In other words, the operator can easily determine the incision site on the body surface of subject E for treatment of tumor TM within subject E.

[0063] Furthermore, if the tumor TM is cancerous, the surrounding tissue may be pulled towards the tumor, causing unnatural deformation. This type of deformation is also called "contracture." As shown in the reconstructed ultrasound image 70, the operator can easily identify this type of "contracture."

[0064] The display control unit 36 ​​may display the entire captured image 24B and the reconstructed ultrasound image 70 on the display 38, but as shown in Figure 10, it is preferable to display the partial captured image 24Ba, which is the portion of the captured image 24B corresponding to the reconstructed ultrasound image 70, and the reconstructed ultrasound image 70 on the display 38.

[0065] Figure 11 shows a partial image 24Ba in the display image 24B. As described above, the coordinate information representing each position in the ultrasound volume data 60 is coordinate information in the camera coordinate system, and the coordinate information representing each position in the ultrasound volume data 60 is based on the position and orientation of the ultrasound probe 14 (i.e., the body surface position of the subject E) when the ultrasound volume data 60 (each frame data 64 in this embodiment) was acquired. Therefore, the display control unit 36 ​​can identify the partial image 24Ba corresponding to the reconstructed ultrasound image 70 in the display image 24B based on the coordinate information of the ultrasound volume data 60.

[0066] The display control unit 36 ​​extracts the identified partial image 24Ba from the display image 24B and displays the partial image 24Ba and the reconstructed ultrasound image 70 as shown in Figure 10. Displayed in I38. Preferably, the partial image 24Ba and the reconstructed ultrasound image 70 are displayed side by side. Since the partial image 24Ba represents the portion corresponding to the reconstructed ultrasound image 70 (the reconstructed ultrasound image 70 will represent the internal structure of the portion represented by the partial image 24Ba), the operator can more easily identify the location of the tumor TM on the body surface of the subject E.

[0067] To make the correspondence between the partial image 24Ba and the reconstructed ultrasound image 70 clearer, the display control unit 36 ​​may display the partial image 24Ba and the reconstructed ultrasound image 70 on the display 38 with their orientations and sizes matching each other.

[0068] By matching the display orientation and display size of the partial image 24Ba and the reconstructed ultrasound image 70, the position of the tumor TM in the partial image 24Ba can be determined based on the position of the tumor TM in the reconstructed ultrasound image 70. For example, the display control unit 36 ​​can identify the coordinates of the pixel group corresponding to the tumor TM in the reconstructed ultrasound image 70, and the pixel group indicated by the same coordinates in the partial image 24Ba can be set as the position of the tumor TM.

[0069] Therefore, as shown in Figure 12, the display control unit 36 ​​may display a position indicator 72 on the display 38 that indicates the position in the partial image 24Ba corresponding to the position of the tumor TM in the reconstructed ultrasound image 70. In the example in Figure 12, an X mark as the position indicator 72 is superimposed on the position of the tumor TM in the partial image 24Ba. This makes it even easier for the operator to identify the position corresponding to the tumor TM on the body surface of the subject E.

[0070] The operator may determine the incision position on the body surface based on surface features on the body surface of subject E. For example, the incision position may be set at a location XX millimeters toward the foot from the surface feature. Therefore, if the partial image 24Ba contains an image of a surface feature, the distance measuring unit 56 may measure the distance between the surface feature in real space and the position indicated by the position indicator 72 (referred to as the "recommended incision position" in this specification) based on the distance between the image of the surface feature and the position indicator 72 in the partial image 24Ba.

[0071] The processing performed by the distance measurement unit 56 will be explained with reference to Figure 12. In this embodiment, the body surface feature is assumed to be a nipple NP. First, the distance measurement unit 56 detects an image of the nipple NP from the partial image 24Ba. For example, the distance measurement unit 56 uses a known object detection technique (e.g., a learning model such as R-CNN (Regional Convolutional Neural Network)) to... The image of the nipple NP can be detected from the partially captured image 24Ba. The distance measuring unit 56 then calculates the distance between the image of the nipple NP and the position indicator 72 in the partially captured image 24Ba. This distance is calculated from the coordinates of the image of the nipple NP (for example, its representative position (such as the center position)) and the coordinates of the position indicator 72 in the partially captured image 24Ba.

[0072] The distance between the nipple NP image and the position indicator 72 in the partial image 24Ba can also be converted to the distance between the nipple NP and the recommended incision site in real space using known techniques. Specifically, the distance between the nipple NP and the recommended incision site in real space can be calculated based on the focal length of the camera 12, the distance from the camera 12 to the body surface of the subject E, and the distance between the nipple NP image and the position indicator 72 in the partial image 24Ba.

[0073] The display control unit 36 ​​displays on the display 38 the distance between the body surface feature in real space and the position indicated by the position indicator, as measured by the distance measurement unit 56. Figure 13 shows an example of displaying the distance between the nipple NP and the recommended incision location. In the example in Figure 13, the display control unit 36 ​​displays a message window 74 indicating the distance from the nipple near the partial image 24Ba.

[0074] The ultrasound diagnostic device 16 is capable of Doppler measurement to measure the velocity of the tissue and blood flow of the subject E. Doppler measurement measures the velocity of the tissue and blood flow of the subject based on the difference between the frequency of the transmitted wave from the ultrasound probe 14 and the frequency of the reflected wave from the subject's tissue and blood flow. In this embodiment, particular attention is paid to the fact that blood flow can be measured by Doppler measurement.

[0075] The transmitting / receiving unit 30 can transmit a transmission wave for Doppler measurement along with a transmission wave for acquiring frame data 64 to form ultrasonic volume data 60. For example, the transmission wave for acquiring frame data 64 and the transmission wave for Doppler measurement can be transmitted alternately. This allows obtaining Doppler signals representing blood flow velocity at each position on the scanning plane along with the frame data 64. Therefore, the volume data acquisition unit 50 can acquire ultrasonic volume data 60 that includes Doppler signals indicating the blood flow velocity of the subject E at each position.

[0076] Figure 14 shows an example of a reconstructed Doppler image 80. The reconstruction processing unit 54 extracts ultrasound volume data 60 containing Doppler signals from the aforementioned excision section CS (see Figure 6) to form a reconstructed ultrasound image 70, and also forms a blood flow image 82 representing the blood flow of the subject E in the excision section CS based on the Doppler signals contained in the ultrasound volume data 60. The blood flow image 82 may be, for example, a color Doppler image in which the direction and velocity of the blood flow of the subject E are represented by hue. The reconstruction processing unit 54 then forms an image (referred to as "reconstructed Doppler image 80" in this specification) in which the blood flow image 82 is superimposed on the reconstructed ultrasound image 70.

[0077] Figure 15 shows an example of the display of the reconstructed Doppler image 80. The display control unit 36 ​​displays the reconstructed Doppler image 80 on the display 38. It is known that blood flow increases around tumors TM, especially cancer. Therefore, by checking the reconstructed Doppler image 80, the operator can suspect that cancer may be present in areas with high blood flow (≒blood flow velocity). In particular, since cancer metastasizes, displaying the reconstructed Doppler image 80, which shows the blood flow near the site where the cancer was located, after a procedure to remove cancer can assist the operator in confirming whether any cancer remains after removal or whether it has metastasized to the surrounding area.

[0078] As shown in Figure 16, the display control unit 36 ​​may display the reconstructed Doppler image 80 and the display-ready captured image 24B (especially the partial captured image 24Ba) side by side. In this case as well, it is preferable to display the partial captured image 24Ba and the reconstructed Doppler image 80 on the display 38 with their display orientation and display size matching each other.

[0079] The treatment support device relating to this disclosure has been described above, but the treatment support device relating to this disclosure is not limited to the embodiments described above, and various modifications are possible as long as they do not deviate from the spirit of the disclosure.

[0080] For example, in each of the above embodiments, the treatment support device is an ultrasound diagnostic device 16, and the functions of the display control unit 36, volume data acquisition unit 50, treatment target area identification unit 52, reconstruction processing unit 54, and distance measurement unit 56 were all performed by the ultrasound diagnostic device 16. However, these functions do not necessarily have to be performed by the ultrasound diagnostic device 16. For example, these functions may be performed by a server computer or the like that is communicatively connected to the ultrasound diagnostic device 16. In that case, the processor of the server computer or the like that acting as the treatment support device acquires ultrasound volume data 60 from the ultrasound diagnostic device (performing the function of the volume data acquisition unit 50), identifies the treatment target area in the ultrasound volume data 60 ( The device may function as a treatment area identification unit 52, form a reconstructed ultrasound image 70 from the ultrasound volume data 60 (functioning as a reconstruction processing unit 54), and display the reconstructed ultrasound image 70 on a display accessible to the operator (functioning as a display control unit 36). Furthermore, instead of one device performing all of the above functions, multiple devices may work together to perform each of these functions. [Explanation of Symbols]

[0081] 10 Treatment support system, 12 Camera, 14 Ultrasound probe, 16 Ultrasound diagnostic device, 20 Probe detection marker, 22 Body surface detection marker, 24A Detection image, 24B Display image, 24Ba Partial image, 30 Transmitter / receiver unit, 32 Signal processing unit, 34 Image forming unit, 36 Display control unit, 38 Display, 40 Communication interface, 42 Input interface, 44 Memory, 46 Control unit, 48 Probe position and orientation information acquisition unit, 50 Volume data acquisition unit, 52 Treatment target area identification unit, 54 Reconstruction processing unit, 56 Distance measurement unit, 60 Ultrasound volume data, 60a Partial volume data, 62 Frame data sequence, 64 Frame data, 70 Reconstructed ultrasound image, 72 Position indicator, 74 Message window, 80 Reconstructed Doppler image, 82 Blood flow image, CS Sectional section.

Claims

1. An image acquisition unit that acquires a display image obtained by photographing the body surface area near the tissue to be treated within the subject, A volume data acquisition unit acquires ultrasonic volume data corresponding to the ultrasonic irradiation area, which is formed based on a received signal obtained by transmitting and receiving ultrasonic waves from an ultrasonic probe in contact with the body surface portion in the ultrasonic irradiation area within the subject, including the tissue to be treated. A treatment target area identification unit identifies the treatment target area, which is the region occupied by the treatment target tissue in the aforementioned ultrasound volume data, A reconstruction processing unit that forms a reconstructed ultrasound image by cutting out the ultrasound volume data at a cross section that is perpendicular to the depth direction of the ultrasound volume data and includes the treatment target area, and then reconstructing the ultrasound volume data. A display control unit that displays the aforementioned captured image for display and the reconstructed ultrasound image on a display unit, A treatment support device characterized by being equipped with the following features.

2. A probe position and orientation information acquisition unit acquires position and orientation information indicating the position and orientation of the ultrasonic probe when the ultrasonic volume data is acquired, Furthermore, The image acquisition unit acquires a detection image obtained by photographing a probe detection marker attached to an ultrasound probe that is in contact with the body surface, The volume data acquisition unit acquires the ultrasonic volume data having coordinate information representing each position of the ultrasonic volume data with respect to the position and orientation information in the camera coordinate system of the camera that captured the display image, The display control unit causes the display unit to display a partial image, which is a portion of the display image corresponding to the reconstructed ultrasound image, based on the coordinate information contained in the ultrasound volume data. The treatment support device according to feature 1.

3. The display control unit causes the display unit to display the partial image and the reconstructed ultrasound image in a manner that matches their display orientation and display size. The treatment support device according to feature 2.

4. The display control unit causes the display unit to display a position indicator on the display unit that indicates the position in the partial image corresponding to the position of the treatment target area in the reconstructed ultrasound image. The treatment support device according to feature 3.

5. The aforementioned partial image includes an image of a surface feature on the body surface of the subject, The aforementioned treatment support device is A distance measuring unit measures the distance between the body surface feature and the position indicated by the position indicator in real space, based on the distance between the image of the body surface feature and the position indicator in the partial image. Furthermore, The display control unit causes the distance measured by the distance measuring unit to be displayed on the display unit. The treatment support device according to feature 4.

6. The reconstruction processing unit determines the position of the cut-out cross section in the depth direction to the position where the area of ​​the treatment target region in the cut-out cross section is maximized. The treatment support device according to feature 1.

7. The reconstruction processing unit forms the reconstructed ultrasound image based on partial volume data, which is a part of the ultrasound volume data and includes the cut-out cross-section and has thickness in the depth direction of the ultrasound volume data. The treatment support device according to feature 1 or 6.

8. The volume data acquisition unit acquires the ultrasound volume data, which includes Doppler signals indicating the blood flow velocity of the subject at each position. The reconstruction processing unit forms a blood flow image representing the blood flow of the subject in the excised cross section based on the Doppler signal, and forms a reconstructed Doppler image in which the blood flow image is superimposed on the reconstructed ultrasound image. The display control unit causes the reconstructed Doppler image to be displayed on the display unit. The treatment support device according to feature 1.

9. A volume data acquisition unit acquires ultrasonic volume data corresponding to the ultrasonic irradiation area, which is formed based on a received signal obtained by transmitting and receiving ultrasonic waves in an ultrasonic irradiation area within the subject including the target tissue from an ultrasonic probe in contact with the body surface portion near the target tissue within the subject, and which includes a Doppler signal indicating the blood flow velocity of the subject at each position. A treatment target area identification unit identifies a treatment target area corresponding to the treatment target tissue in the aforementioned ultrasound volume data, A reconstruction processing unit forms a reconstructed ultrasound image by cutting out and reconstructing the ultrasound volume data including the Doppler signal at a section perpendicular to the depth direction of the ultrasound volume data and including the treatment area, a blood flow image representing the blood flow of the subject at the section based on the Doppler signal, and a reconstructed Doppler image in which the blood flow image is superimposed on the reconstructed ultrasound image. A display control unit that displays the reconstructed Doppler image on a display unit, A treatment support device characterized by being equipped with the following features.

10. Computers, An image acquisition unit that acquires a display image obtained by photographing the body surface area near the tissue to be treated within the subject, A volume data acquisition unit acquires ultrasonic volume data corresponding to the ultrasonic irradiation area, which is formed based on a received signal obtained by transmitting and receiving ultrasonic waves from an ultrasonic probe in contact with the body surface portion in the ultrasonic irradiation area within the subject, including the tissue to be treated. A treatment target area identification unit identifies the treatment target area, which is the region occupied by the treatment target tissue in the aforementioned ultrasound volume data, A reconstruction processing unit that forms a reconstructed ultrasound image by cutting out the ultrasound volume data at a cross section that is perpendicular to the depth direction of the ultrasound volume data and includes the treatment target area, and then reconstructing the ultrasound volume data. A display control unit that displays the aforementioned captured image for display and the reconstructed ultrasound image on a display unit, A treatment support program characterized by functioning as such.

Images