PROCESS SUPPORT DEVICE AND PROCESS SUPPORT PROGRAM

Abstract

One task is to enable an operator to easily identify an incision position on a subject's body surface for performing a procedure on a target tissue in the subject. A camera images a section of the body surface surrounding a tumor in a patient to create a captured image for display. A volumetric data acquisition unit captures ultrasound volumetric data corresponding to an ultrasound irradiation area containing the tumor in the patient. This data is generated based on received signals obtained by transmitting and receiving ultrasound within the irradiation area using an ultrasound probe in contact with the patient's body surface. A reconstruction processing unit extracts the ultrasound volumetric data into a cross-sectional section perpendicular to a depth direction of the ultrasound volumetric data and including a target area (the area occupied by the tumor) to reconstruct the ultrasound volumetric data.A display control shows the captured image and a reconstructed ultrasound image on a display.

Description

BACKGROUND OF THE INVENTION 1. Field of the invention

[0001] The present description discloses improvements to a process support device and a process support program. 2. Description of the state of the art

[0002] An ultrasound diagnostic device is known which transmits ultrasound from an ultrasound probe, which is in contact with a body surface of a subject, to the subject, receives reflected waves from the subject through the ultrasound probe and generates an ultrasound tomography image of the subject based on received signals formed from the reflected waves.

[0003] In the prior art, in a case where a procedure (for example, an operation) is performed on a subject, an ultrasound diagnostic device is used by an operator, such as a doctor, to examine the inside of the subject.

[0004] For example, JP2024-52409A discloses an image processing device that captures a three-dimensional image (volume data) formed by an ultrasound diagnostic device representing a procedure target tissue in a subject and renders the three-dimensional image from a side viewpoint, in which a section for cutting the procedure target tissue is seen from a side intersecting a normal line of the section, in a line-of-sight direction (i.e., a side of the section), to form and display a side viewpoint image, which is an ultrasound image in which the section is seen from the side. SUMMARY OF THE INVENTION

[0005] In a case where the target tissue is present in the patient, the operator must incise the patient's body surface at the target tissue before performing the procedure. From the perspective of minimizing invasiveness to the patient, it is desirable that the incision position on the patient's body surface be located in the vicinity of the target tissue, such as directly above it (in one depth direction).

[0006] It goes without saying that the operator cannot see the target tissue before incising the patient's body surface. Therefore, in the prior art, the operator performs the task of reviewing the ultrasound tomography image, which shows the target tissue before incision, and specifying the incision position on the patient's body surface based on the ultrasound tomography image.

[0007] With current technology, specifying the incision position from the ultrasound tomography image can be time-consuming and difficult. For example, the ultrasound tomography image, generated by bringing the ultrasound probe into contact with the area surrounding the incision site to transmit and receive ultrasound to and from the target tissue, depicts the target tissue from the side in the visual line of sight of the operator facing the incision site. Therefore, operator experience may be required to specify the incision position from the ultrasound tomography image.

[0008] One function of a procedure support device disclosed in the present description is to enable an operator to easily identify an incision position on a body surface of a subject for performing a procedure on a target tissue of the subject.

[0009] The present specification discloses a procedure support device comprising: an image acquisition unit that acquires an image for display obtained by imaging a body surface section within the environment of a procedure target tissue in a subject; a volume data acquisition unit that acquires ultrasound volume data corresponding to an ultrasound irradiation area containing the procedure target tissue in the subject, formed on the basis of received signals obtained by transmitting and receiving ultrasound in the ultrasound irradiation area using an ultrasound probe in contact with the body surface section; a procedure target area specification unit that specifies a procedure target area occupied by the procedure target tissue in the ultrasound volume data;a reconstruction processing unit that forms a reconstructed ultrasound image by cutting out the ultrasound volume data in a cut-out cross-section perpendicular to a depth direction of the ultrasound volume data and including the procedure target area to reconstruct the ultrasound volume data; and a display control that displays the acquired image for display and the reconstructed ultrasound image on a display unit.

[0010] The procedure support device may further comprise: a probe position and orientation information acquisition unit, which acquires position and orientation information specifying the position and orientation of the ultrasound probe in a case where ultrasound volume data are acquired; the image acquisition unit acquiring an image for detection obtained by imaging a probe detection marker attached to the ultrasound probe in contact with the body surface section; the volume data acquisition unit acquiring the ultrasound volume data, the coordinate information specifying each position of the ultrasound volume data, with reference to the position and orientation information in a camera coordinate system of a camera that acquired the acquired image for display; and the display control acquiring a partial acquired image.that is a section of the recorded image for display, which corresponds to the reconstructed ultrasound image on the display unit based on the coordinate information contained in the ultrasound volume data.

[0011] The display control can show the partial scan image and the reconstructed ultrasound image on the display unit with adjusted display directions and display sizes.

[0012] The display control can show a position indicator on the display unit that indicates a position in the partial image that corresponds to a position of the target area in the reconstructed ultrasound image.

[0013] The partial image can contain an image of a body surface feature object on a body surface of the subject; the procedure support device can further include a distance measuring unit that measures a distance between the body surface feature object and the position indicated by the position indicator in real space, based on a distance between the image of the body surface feature object and the position indicator in the partial image; and the display control can display the distance measured by the distance measuring unit on the display unit.

[0014] The reconstruction processing unit can determine a position of the cut-out cross-section in the depth direction as a position where an area of ​​the process target region in the cut-out cross-section is largest.

[0015] The reconstruction processing unit can form the reconstructed ultrasound image based on partial volume data, which are a part of the ultrasound volume data, including the cut-out cross-section and a thickness in the depth direction of the ultrasound volume data.

[0016] The volume data acquisition unit can acquire the ultrasound volume data, which includes a Doppler signal indicating the velocity of blood flow in the subject at each position; the reconstruction processing unit can create a blood flow image representing the subject's blood flow in the cut-out cross-section based on the Doppler signal and create a reconstructed Doppler image in which the blood flow image is superimposed on the reconstructed ultrasound image; and the display control can display the reconstructed Doppler image on the display unit.

[0017] Furthermore, the present description discloses a procedure support device comprising: a volume data acquisition unit that acquires ultrasound volume data corresponding to an ultrasound irradiation area containing a procedure target tissue in a patient, which is formed on the basis of received signals obtained by transmitting and receiving ultrasound in the ultrasound irradiation area using an ultrasound probe in contact with a body surface section in the vicinity of the procedure target tissue in the patient, wherein the ultrasound volume data includes a Doppler signal indicating a velocity of blood flow in the patient at each position; a procedure target area specification unit that specifies a procedure target area for a procedure corresponding to the procedure target tissue in the ultrasound volume data;a reconstruction processing unit that forms a reconstructed ultrasound image by extracting the ultrasound volume data containing the Doppler signal in a cut-out cross-section perpendicular to a depth direction of the ultrasound volume data and containing the procedure target area, in order to reconstruct the ultrasound volume data forming a blood flow image representing the blood flow of the subject in the cut-out cross-section based on the Doppler signal, and forming a reconstructed Doppler image in which the blood flow image is superimposed on the reconstructed ultrasound image; and a display control that displays the reconstructed Doppler image on a display unit.

[0018] Furthermore, the present description discloses a process support program that causes a computer to function as: an image acquisition unit that acquires an image for display obtained by imaging a body surface section in the environment of a procedure target tissue in a subject; a volume data acquisition unit that acquires ultrasound volume data corresponding to an ultrasound irradiation area containing the procedure target tissue in the subject, which is formed on the basis of received signals obtained by transmitting and receiving ultrasound in the ultrasound irradiation area using an ultrasound probe in contact with the body surface section;a target area specification unit that specifies a target area occupied by the target tissue in the ultrasound volume data; a reconstruction processing unit that forms a reconstructed ultrasound image by cutting out the ultrasound volume data in a cut-out cross-section perpendicular to a depth direction of the ultrasound volume data and including the target area to reconstruct the ultrasound volume data; and a display control that displays the acquired image for display and the reconstructed ultrasound image on a display unit.

[0019] With the procedure support device disclosed in the present description, the operator can easily identify the incision position on the body surface of the subject for performing the procedure on the target tissue of the subject. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a diagram that schematically shows a configuration of a process support system according to the present embodiment. Fig. Figure 2 is a diagram showing an example of a captured image for detection. Fig. Figure 3 is a diagram showing an example of a captured image for display. Fig. Figure 4 is a diagram that schematically shows a configuration of an ultrasound diagnostic device according to the present embodiment. Fig. Figure 5 is a concept diagram that illustrates a concept of processing the generation of ultrasound volume data. Fig. Figure 6 is a diagram showing a cut-out cross-section of the ultrasound volume data. Fig. Figure 7 is a diagram showing several cut-out cross-sections. Fig. Figure 8 is a diagram showing a data section used to create a reconstructed ultrasound image. Fig. Figure 9 is a diagram showing an example of the reconstructed ultrasound image. Fig. Figure 10 is a diagram showing a display example of the captured image for display and the reconstructed ultrasound image. Fig. 11 is a diagram showing a partial image within the captured image for display. Fig. Figure 12 is a diagram showing an example display of a position indicator. Fig. Figure 13 is a diagram showing an example of the distance between a nipple and a recommended incision position. Fig. Figure 14 is a diagram showing an example of a reconstructed Doppler image. Fig. Figure 15 is a diagram showing a display example of the reconstructed Doppler image. Fig. Figure 16 is a diagram showing a display example of the captured image and the reconstructed Doppler image. DESCRIPTION OF PREFERRED EXECUTION FORMS

[0020] Fig. Figure 1 is a diagram schematically showing a configuration of a process support system 10 according to the present embodiment. The process support system 10 includes a camera 12 and an ultrasound diagnostic device 16, which is a process support device containing an ultrasound probe 14. The camera 12 and the ultrasound diagnostic device 16 are connected to each other in such a way that they can communicate with each other.

[0021] In the present embodiment, a probe detection marker 20 is attached to the ultrasound probe 14. The probe detection marker 20 is a marker for detecting the position and orientation of the ultrasound probe 14. Furthermore, a body surface detection marker 22 is attached to the body surface of a subject E. The body surface detection marker 22 has a different pattern than the probe detection marker 20 and is a marker for detecting the position and orientation of the body surface of the subject E. Examples of the probe detection marker 20 and the body surface detection marker 22 include a marker for augmented reality (AR).

[0022] The camera 12 includes, in addition to a lens and an image sensor, a processor comprising a central processing unit (CPU) and the like, a communication interface comprising a network adapter and the like. The camera 12 images the ultrasound probe 14 (in particular the probe detection marker 20) and the subject E (in particular the body surface detection marker 22). The position and orientation of the ultrasound probe 14 are detected by the ultrasound diagnostic device 16, as described below, by using a captured image obtained through such imaging. In this description, such a captured image is referred to as a captured image for detection.

[0023] Fig. Figure 2 is a diagram showing an example of a captured image 24A for detection. As described above, the image of the probe detection marker 20 and the image of the body surface detection marker 22 are included in the captured image 24A for detection. The ultrasound diagnostic device 16 can detect the position and orientation of the ultrasound probe 14 by analyzing the image of the probe detection marker 20 shown in the captured image 24A for detection. Furthermore, the ultrasound diagnostic device 16 can detect the position and orientation of the body surface of the subject E by analyzing the image of the body surface detection marker 22 shown in the captured image 24A for detection. Details of the processing for detecting the positions and orientations of the ultrasound probe 14 and the subject E are described later.

[0024] Furthermore, camera 12 images a section of the body surface in the vicinity of a target tissue (details will be described later) of the subject E. In this description, such a captured image is referred to as a captured image for display.

[0025] Fig. Figure 3 is a diagram showing an example of a recorded image 24B for display. Although the present disclosure is not limited to this, the present embodiment describes an example in which the target tissue of the procedure is a tumor TM in the breast of the subject E. Therefore, in the present embodiment, as in Fig. Figure 3 shows that the acquired image 24B for display is an image of a body surface section near the tumor TM, that is, a circumference of the breast. Specifically, the acquired image 24B for display is an image in which the edge of the breast is depicted from a frontal view of the subject E. In a case where the body surface section near the procedure target tissue is shown in the acquired image 24A for detection, the acquired image 24A for detection can be used as the acquired image 24B for display.

[0026] In this description, the concepts comprising the captured image 24A for detection and the captured image 24B for display are simply referred to as a single captured image. The position and orientation of the camera 12 in a case where the captured image 24A for detection is taken, and the position and orientation of the camera 12 in a case where the captured image 24B for display is taken, are set to be the same.

[0027] A captured image is formed by the image sensor of the camera 12, and the captured image is transmitted via the communication interface of the camera 16 to the ultrasound diagnostic device 12.

[0028] Fig. Figure 4 is a diagram that schematically shows a configuration of the ultrasound diagnostic device 16. The ultrasound diagnostic device 16 is a medical device installed in a medical facility, such as a hospital.

[0029] The ultrasound probe 14 is a device that transmits ultrasound to and receives it from the subject E. The ultrasound probe 14 includes a transducer element arrangement consisting of several transducer elements that transmit ultrasound waves to and receive them from the subject. The transducer element arrangement consists of the several transducer elements arranged in one direction (arrangement direction). In a case where a transmission signal from a transmission and reception unit 30, which will be described later, is supplied to each transducer element, each transducer element generates the ultrasound. In particular, the ultrasound probe 14 scans an ultrasound beam on a plane (scanning plane) that is parallel to the arrangement direction.

[0030] As described above, the probe detection marker 20 is attached to the ultrasound probe 14.

[0031] The transmission and reception unit 30 transmits the transmission signal to the ultrasound probe 14 (in particular, each transducer element of the transducer element arrangement) under the control of a controller 46, which will be described later. As a result, the ultrasound beam is sampled by the ultrasound probe 14 at the scanning plane. In addition, the transmission and reception unit 30 receives a received signal from each transducer element that has received reflected waves from the subject E. The transmission and reception unit 30 comprises an adder and several delay devices corresponding to the respective transducer elements and performs phase alignment and addition processing of the received signals from the transducer elements using the adder and the multiple delay devices.As a result, a received beam signal is generated, in which information indicating the signal intensity of the wave reflected by subject E is arranged in the depth direction of subject E. Several received beam signals corresponding to a scanning plane constitute single-image data.

[0032] A signal processing unit 32 performs various types of signal processing, including filter processing by applying a bandpass filter, detection processing and the like, on the received beam signal from the transmit and receive unit 30.

[0033] An image generation unit 34 generates an ultrasound tomography image (B-mode image) representing a cross-section (in particular a scanning plane of the ultrasound beam) of the subject E, based on the received beam signal which was subjected to signal processing in the signal processing unit 32.

[0034] A display controller 36 controls the display of the ultrasound tomography image, generated by the image generation unit 34, on a display 38. Furthermore, as will be described in detail later, the display controller 36 displays a reconstructed ultrasound image and the acquired image 24B, generated by the controller 46, on the display 38.

[0035] The display 38 as a display unit is a display device configured, for example, by a liquid crystal display or organic electroluminescence (EL).

[0036] The transmission and reception unit 30, the signal processing unit 32, the image generation unit 34, and the display controller 36, which are contained in the ultrasound diagnostic device 16, are configured by a processor. The processor includes at least one general-purpose processing device (for example, a CPU or the like) and one dedicated processing device (for example, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a programmable logic device). The processor need not be configured by a single processing device but can instead be configured by cooperation of several processing devices located at physically separate positions.Furthermore, each of the units described above can be implemented through the collaboration of hardware, such as a processor, and software.

[0037] A communication interface 40 is configured, for example, by a network adapter. The communication interface 40 has the function of communicating with another device (in particular, the camera 12). Specifically, the communication interface 40 receives the captured images (captured image 24A for detection and captured image 24B for display) from the camera 12. As described above, in the present embodiment, the communication interface 40 functions as a capture unit for captured images.

[0038] An input interface 42 is configured, for example, by a button, a trackball, or a touch panel. The input interface 42 is used to input an instruction from an operator, such as a physician using the ultrasound diagnostic device 16, to the ultrasound diagnostic device 16.

[0039] A memory 44 contains a hard disk drive (HDD), a solid-state drive (SSD), an embedded MultiMediaCard (eMMC), read-only memory (ROM), random-access memory (RAM), or the like. The memory 44 stores a procedure support program for operating each unit of the ultrasound diagnostic device 16. Alternatively, the procedure support program can also be stored on a non-volatile, computer-readable storage medium, such as a Universal Serial Bus (USB) drive or a CD-ROM. The ultrasound diagnostic device 16 can read the procedure support program from such a storage medium and execute the procedure support program.Since the ultrasound diagnostic device 16 reads the procedure support program in order to have the functions described later, it can be said that the ultrasound diagnostic device 16 is a computer program product.

[0040] A controller 46 comprises at least one general-purpose processor (such as a CPU) and one dedicated processor (such as a GPU, ASIC, FPGA, or programmable logic circuit). The controller 46 need not be configured by a single processing device but can instead be configured through cooperation of multiple processing devices located at physically separate positions. The controller 46 controls each unit of the ultrasonic diagnostic device 16. Furthermore, as shown in Fig. 4 shown, the control 46 functions as a recording unit 48 for probe position and orientation information, a volume data recording unit 50, a specification unit 52 for process target area, a reconstruction processing unit 54 and a distance measurement unit 56 in accordance with the process support program stored in the memory 44.

[0041] The probe position and orientation information acquisition unit 48 acquires position and orientation information that specifies the position and orientation of the ultrasound probe 14. In particular, the probe position and orientation information acquisition unit 48 acquires the position and orientation information that specifies the position and orientation of the ultrasound probe 14 in a case where single-image data are acquired to generate ultrasound volume data, which will be described later.

[0042] In the present embodiment, the probe position and orientation information acquisition unit 48 acquires the position and orientation information by analyzing the image 24A for detection captured by the camera 12 in order to detect the position and orientation of the ultrasound probe 14. As described above, the captured image 24A for detection contains the image of the probe detection marker 20 for detecting the position and orientation of the ultrasound probe 14 (see Fig. 2) The probe position and orientation information acquisition unit 48 acquires the position and orientation information by analyzing the image of the probe detection marker 20 in the captured image 24A for detection. The position of the ultrasound probe 14 can, for example, be expressed by three-dimensional coordinates in a camera coordinate system. The orientation of the ultrasound probe 14 can, for example, be expressed by an angle of rotation with respect to each of three predetermined orthogonal axes in the camera coordinate system. Since a known method can be used to detect the position and orientation of the ultrasound probe 14 in the camera coordinate system from the image of the probe detection marker 20 contained in the captured image 24A for detection, detailed descriptions of this method are omitted here.

[0043] As described above, the recorded image 24A also contains the image of the body surface detection marker 22 for detecting the position and orientation of the body surface of the subject E (see Fig. 2) The probe position and orientation information acquisition unit 48 can detect the position and orientation of the body surface of the subject E by analyzing the image of the body surface detection marker 22 in the captured image 24A for detection. Furthermore, the probe position and orientation information acquisition unit 48 can detect the position and orientation of the ultrasound probe 14 relative to the position and orientation of the body surface of the subject E. Consequently, the position and orientation of the ultrasound probe 14 relative to the subject E can be maintained, compensating for any variation in the position or orientation of the subject E.

[0044] The probe position and orientation information acquisition unit 48 can detect the position and orientation of the ultrasound probe 14 by using a method other than analyzing the recorded image 24A for detection. For example, a position and orientation sensor, such as a magnetic sensor or an accelerometer, can be provided in the ultrasound probe 14, and the position and orientation of the ultrasound probe 14 can be detected based on detection values ​​from the position and orientation sensor.

[0045] The position and orientation of the scanned plane are determined by the position and orientation of the ultrasound probe 14. Therefore, the position and orientation information acquired by the probe position and orientation acquisition unit 48 specifies the position and orientation of the scan plane corresponding to each image data point for generating the ultrasound volume data. Furthermore, each position (coordinate) on the image data point can be specified based on the position and orientation of the ultrasound probe 14 when the image data point is acquired. Therefore, each image data point is accompanied by coordinate information specifying each position (coordinate) of the image data point with reference to the position and orientation of the ultrasound probe 14 when the image data point is acquired.

[0046] The acquisition unit 48 for probe position and orientation information stores the single-image data acquired by the transmission and reception unit 30 in memory 44 in conjunction with the position and orientation information, which specifies the position and orientation of the ultrasound probe 14 in a case where the single-image data is acquired. Consequently, a single-image data sequence containing multiple single-image data, each of which is associated with the position and orientation information, is stored in memory 44.

[0047] The volume data acquisition unit 50 acquires the ultrasound volume data corresponding to the ultrasound irradiation area in the subject E including the procedure target tissue, which are formed on the basis of the received signals obtained by transmitting and receiving the ultrasound in the ultrasound irradiation area using the ultrasound probe 14, which is in contact with the body surface section of the subject E.

[0048] Fig. Figure 5 is a concept diagram illustrating a concept for processing and generating ultrasound volume data 60. In the present embodiment, the volume data acquisition unit 50 generates the ultrasound volume data 60 based on a single-image data sequence 62 stored in the memory 44. The single-image data sequence 62 is composed of several single-image data points 64. Each single-image data point 64 corresponds to the scanning plane of the ultrasound probe 14. Specifically, the single-image data points 64 are obtained by transmitting and receiving ultrasound to and from the subject E (especially the target tissue) while the ultrasound probe 14, which is in contact with the subject E, is moved by the operator in a pivoting direction perpendicular to the scanning plane.In particular, in the present embodiment, since the target tissue of the procedure is the tumor TM in the breast, the ultrasound probe 14 is brought into contact with the breast of the subject E, such that the depth direction of the scanning plane of the ultrasound probe 14 is substantially perpendicular to a coronal plane of the subject E (in other words, substantially parallel to a front-back direction of the subject E). At least a portion of the image data 64 contained in each image sequence 62 includes data (a signal indicating the signal intensity of the wave reflected from the tumor TM) relating to the tumor TM.

[0049] Fig. 5 shows an X D -axis direction, a Y D -axis direction and a Z D -Axis directions that are perpendicular to each other in a data space where the ultrasound volume data 60 are defined. In the example of Fig. 5 makes the volume data acquisition unit 50 initially the direction of each individual image data 64 parallel to an X D Y D -plane. In particular, the depth direction of the single image data 64 is parallel to the Y D -axis. Then the individual image data 64 are arranged in one direction (Z). D -Axis direction in the example of Fig. 5), perpendicular to the X D Y D The image plane is arranged and combined to form the ultrasound volume data 60. Since a known method can be used as a method for forming the ultrasound volume data 60 from the multiple individual image data 64, its detailed description is omitted here.

[0050] The single image data 64 for forming the ultrasound volume data 60 can be the multiple received beam signals (or multiple received beam signals processed by the signal processing unit 32) corresponding to a sampling plane formed by the transmission and receiving unit 30, or can be an ultrasound tomography image formed by the image generation unit 34 based on the multiple received beam signals.

[0051] As described above, each image data 64 is linked to position and orientation information specifying the position and orientation of the ultrasound probe 14 when the image data 64 is acquired, and includes coordinate information specifying the position of each image data 64 based on the position and orientation of the ultrasound probe 14. Therefore, the ultrasound volume data 60, which contains the multiple image data 64, also includes coordinate information specifying each position of the ultrasound volume data 60 relative to the position and orientation of the ultrasound probe 14 when each image data 64 is acquired. In the present embodiment, each position of the ultrasound volume data 60 is also expressed by its three-dimensional coordinates in the camera coordinate system.Since the data relating to the tumor TM, which is the target tissue of the procedure, are contained in at least some of the image data 64 contained in each image sequence 62, the ultrasound volume data 60 also contain the data relating to the tumor TM.

[0052] In the present embodiment, the ultrasonic volume data 60 are configured from the multiple single-image data 64, but a 2D array probe, in which the transducer elements are arranged in two dimensions, can be used as the ultrasonic probe 14, and the volume data acquisition unit 50 can directly generate the ultrasonic volume data 60 based on a three-dimensional received signal from the ultrasonic probe 14. Even in this case, the ultrasonic volume data 60 contain the coordinate information that specifies each position of the ultrasonic volume data 60 based on the position and orientation of the ultrasonic probe 14 in a case where the three-dimensional received signal is acquired.

[0053] The procedure target area specification unit 52 specifies a procedure target area occupied by the procedure target tissue (tumor TM in the present embodiment) in the ultrasound volume data 60. As a method of specifying the procedure target area, the procedure target area specification unit 52 can use a learning model trained to identify any tissue (for example, an organ or a tumor) from the ultrasound volume data 60, with the ultrasound volume data 60 as an input. Alternatively, the operator can input an instruction to specify the procedure target area in the ultrasound volume data 60 by displaying the ultrasound volume data 60 on the display 38 with the display control 36 or the like, and the procedure target area specification unit 52 can specify the procedure target area in accordance with the instruction.

[0054] The reconstruction processing unit 54 forms the reconstructed ultrasound image by determining the cross-section in the ultrasound volume data 60 and cutting out the ultrasound volume data 60 in the determined cross-section to reconstruct the ultrasound volume data 60.

[0055] Fig. Figure 6 is a diagram showing a sectioned cross-section CS in the ultrasonic volume data 60. In the present embodiment, the reconstruction processing unit 54 sections the ultrasonic volume data 60 in the sectioned cross-section CS perpendicular to the depth direction (Y). D -axis) of the ultrasound volume data 60 and including the target area of ​​the procedure (in the present embodiment the area occupied by the tumor TM) to reconstruct the ultrasound volume data 60.

[0056] Fig. Figure 7 is a diagram showing several cut-out cross-sections CS. As in Fig. As shown in Figure 7, the multiple excised cross-sections CS can be defined perpendicular to the depth direction of the ultrasound volume data 60 and including the tumor TM (for example, CS1 to CS4 in Figure 7). Fig. 7) Here, the reconstruction processing unit 54 can determine the position of the excised cross-section CS in the depth direction as the position where the area of ​​the tumor TM in the excised cross-section CS is largest. As described above, since the specification unit 52 for the procedure target area specifies the area occupied by the tumor TM in the ultrasound volume data 60, the reconstruction processing unit 54 can calculate the area of ​​the tumor TM in the excised cross-section CS. By determining the position of the excised cross-section CS in the depth direction as the position where the area of ​​the tumor TM in the excised cross-section CS is largest, it is possible to generate the reconstructed ultrasound image in which the tumor TM is depicted at a larger size.

[0057] Fig. Figure 8 is a diagram showing a data section used to generate the reconstructed ultrasound image. The reconstruction processing unit 54 can generate the reconstructed ultrasound image based on partial volume data 60a, which is a part of the ultrasound volume data 60, containing the cut-out cross-section CS (preferably the cut-out cross-section CS in which the tumor area TM is largest) and having a thickness t in the depth direction of the ultrasound volume data 60. The reconstructed ultrasound image with a stereoscopic effect can be generated by generating the reconstructed ultrasound image based on the partial volume data 60a, which has a thickness t in the depth direction.Image quality can be degraded due to blurring of the reconstructed ultrasound image if the thickness t is too large, and the stereoscopic effect is reduced if the thickness t is too small. Therefore, the reconstruction processing unit 54 can adjust the thickness t appropriately. For example, the thickness t can be adjusted based on the size of the tumor TM in the depth direction.

[0058] Fig. Figure 9 is a diagram showing an example of a reconstructed ultrasound image 70. The reconstructed ultrasound image 70 is the ultrasound tomography image containing the tumor TM, which is the target tissue of the procedure. As described above, the cut-out cross-section CS is a plane perpendicular to the depth direction of the ultrasound volume data 60, and thus the reconstructed ultrasound image 70 is an image showing a cross-section of the subject E as seen from the depth direction.In the present embodiment, as described above, since the depth direction of the ultrasound volume data 60 is essentially perpendicular to the coronal plane of the subject E and the cut-out cross-section CS is also essentially perpendicular to the coronal plane of the subject E, the reconstructed ultrasound image 70 is an image showing the cross-section seen from the depth direction, that is, the cross-section parallel to the coronal plane of the subject E.

[0059] Fig. Figure 10 is a diagram showing a display example of the captured image 24B for display and the reconstructed ultrasound image 70. The display control 36 displays the captured image 24B for display and the reconstructed ultrasound image 70 on the display 38. As described above, the captured image 24B for display is obtained by imaging the body surface of the subject E (in other words, an optical axis of the lens of camera 12 is essentially parallel to the depth direction of the subject E), and the reconstructed ultrasound image 70 is an image representing the cross-section seen from the depth direction; thus, the captured image 24B for display and the reconstructed ultrasound image 70 are images seen from the same line of sight.In the present embodiment, since the recorded image 24B for display is an image obtained by imaging the subject E from the front, and the reconstructed ultrasound image 70 is an image showing the cross-section parallel to the coronal plane of the subject E, both the recorded image 24B for display and the reconstructed ultrasound image 70 are images obtained by viewing the subject E from the front.

[0060] Since the acquired image 24B for display is obtained by imaging the body surface of subject E, the position of the tumor TM is not naturally shown in the acquired image 24B for display. On the other hand, the position of the tumor TM is shown in the reconstructed ultrasound image 70. Therefore, the operator can specify the position corresponding to the tumor TM on the body surface of subject E by comparing the reconstructed ultrasound image 70 and the acquired image 24B for display, which are images viewed in the same line of sight. That is, the operator can easily identify the incision position on the body surface of subject E for performing the procedure on the tumor TM in subject E.

[0061] Furthermore, in a case where the tumor TM is malignant, peripheral tissue of the tumor TM can be pulled towards the tumor side and abnormally deformed. Such deformation is also referred to as "anchoring." The operator can easily verify such an "anchored" condition using the reconstructed ultrasound image 70.

[0062] The display control 36 can display the entire acquired image 24B and the reconstructed ultrasound image 70 on the display 38, but the display control 36 can be preferred, as in Fig. Figure 10 shows a partial image 24Ba, which is a section of the recorded image 24B for display, corresponding to the reconstructed ultrasound image 70, and displays the reconstructed ultrasound image 70 on the display 38.

[0063] Fig. Figure 11 is a diagram showing the partial image 24Ba in the captured image 24B for display. As described above, since the coordinate information representing each position of the ultrasound volume data 60 is the coordinate information in the camera coordinate system, and the coordinate information representing each position of the ultrasound volume data 60 is based on the position and orientation (i.e., the body surface position of the subject E) of the ultrasound probe 14 in a case where the ultrasound volume data 60 (in the present embodiment, each individual image data 64) are acquired, the display control 36 can specify the partial image 24Ba corresponding to the reconstructed ultrasound image 70 in the captured image 24B for display based on the coordinate information contained in the ultrasound volume data 60.

[0064] The display control 36 extracts the specified partial image 24Ba from the acquired image 24B for display and shows the partial image 24Ba and the reconstructed ultrasound image 70 on the display 38, as shown in Fig. Figure 10 shows that it is preferred that the partial image 24Ba and the reconstructed ultrasound image 70 be displayed side by side. Since the partial image 24Ba is an image representing a section corresponding to the reconstructed ultrasound image 70 (the reconstructed ultrasound image 70 represents an internal structure of the section shown by the partial image 24Ba), the operator can more easily specify a position corresponding to the tumor TM on the body surface of the subject E.

[0065] To make the correspondence between the partial image 24Ba and the reconstructed ultrasound image 70 more understandable, the display control 36 can display the partial image 24Ba and the reconstructed ultrasound image 70 on the display 38 with adapted display directions and display sizes.

[0066] In a case where the display directions and display sizes of partial image 24Ba and the reconstructed ultrasound image 70 are aligned, it is possible to specify the position of the tumor TM in partial image 24Ba based on the position of the tumor TM in the reconstructed ultrasound image 70. For example, the display control 36 can specify coordinates of a pixel group corresponding to the tumor TM in the reconstructed ultrasound image 70 and set the pixel group specified by the same coordinates in partial image 24Ba as the position of the tumor TM.

[0067] Therefore, as in Fig. Figure 12 shows that the display control 36 on the display 38 displays a position indicator 72, which indicates a position in the partial image 24Ba that corresponds to the position of the tumor TM in the reconstructed ultrasound image 70. In the example of Fig. In section 12, an X-mark is superimposed as the position indicator 72 at the position of the tumor TM in the partial image 24Ba. As a result, the operator can more easily specify the position corresponding to the tumor TM on the body surface of the subject E.

[0068] The operator can determine the incision position on the body surface with reference to a body surface feature object on the body surface of the subject E. For example, the incision position is a position ◯◯ millimeters from the body surface feature object to one side of the foot. Therefore, in a case where the image of the body surface feature object is included in the partial image 24Ba, the distance measuring unit 56 can measure a distance between the body surface feature object in the partial image 24Ba and the position indicated by the position indicator 72 in real space (referred to in this description as a “recommended incision position”) based on the distance between the image of the body surface feature object and the position indicator 72 in the partial image 24Ba.

[0069] The processing carried out by the distance measuring unit 56 is described with reference to Fig. 12. In the present embodiment, it is assumed that the body surface feature object is a nipple NP. First, the distance measuring unit 56 detects an image of the nipple NP from the partial image 24Ba. The distance measuring unit 56 can detect the image of the nipple NP from the partial image 24Ba, for example, by using a known object detection technique (for example, a learning model such as a regional convolutional neural network (R-CNN)). Then, the distance measuring unit 56 calculates a distance between the image of the nipple NP and the position indicator 72 in the partial image 24Ba. The distance is calculated from the coordinates of the image of the nipple NP (for example, a representative position (center position or the like) thereof) in the partial image 24Ba and the coordinates of the position indicator 72.

[0070] The conversion of the distance between the image of the nipple NP and the position indicator 72 in the partial image 24Ba into the distance between the nipple NP and the recommended incision position in real space can also be performed using a known technique. In particular, the distance between the nipple NP and the recommended incision position in real space can be calculated based on the focal length of the camera 12, the distance from the camera 12 to the body surface of the subject E, and the distance between the image of the nipple NP and the position indicator 72 in the partial image 24Ba.

[0071] The display control 36 shows the distance between the body surface feature object and the position indicated by the position indicator in real space, which is measured by the distance measuring unit 56, on the display 38. Fig. Figure 13 is a diagram showing an example of the distance between the nipple (NP) and the recommended incision position. In the example of Fig. 13 The display control 36 shows a message window 74 indicating the distance from the nipple in the vicinity of the partial image 24Ba.

[0072] The ultrasound diagnostic device 16 can perform Doppler measurements to measure tissue or blood flow velocity in the subject E. Doppler measurement measures the tissue or blood flow velocity of the subject based on the difference between the frequency of transmitted waves from the ultrasound probe 14 and the frequency of reflected waves from the subject's tissue or blood flow. In the present embodiment, particular attention is paid to the fact that blood flow can be measured by Doppler measurement.

[0073] The transmission and reception unit 30 can transmit the transmission waves for Doppler measurement together with the transmission waves for acquiring the single-image data 64 to generate the ultrasound volume data 60. For example, the transmission waves for acquiring the single-image data 64 and the transmission waves for Doppler measurement can be transmitted alternately. As a result, a Doppler signal representing the velocity of blood flow at each position in the scanning plane can be obtained along with the single-image data 64. Therefore, the volume data acquisition unit 50 can acquire the ultrasound volume data 60, which contains the Doppler signal indicating the velocity of blood flow in the subject E at each position.

[0074] Fig. Figure 14 is a diagram showing an example of a reconstructed Doppler image 80. The reconstruction processing unit 54 slices the ultrasound volume data 60, which contains the Doppler signal, into the sliced ​​cross-section CS (see Figure 14). Fig. 6) to form the reconstructed ultrasound image 70, and forms a blood flow image 82, which represents the blood flow of the subject E in the cut-out cross-section CS, based on the Doppler signal contained in the ultrasound volume data 60. The blood flow image 82 may, for example, be a color Doppler image in which the direction and velocity of the blood flow of the subject E are represented by a color tone. Then the reconstruction processing unit 54 forms an image (referred to in this description as a “reconstructed Doppler image 80”) in which the blood flow image 82 is superimposed on the reconstructed ultrasound image 70.

[0075] Fig. Figure 15 is a diagram showing a display example of the reconstructed Doppler image 80. The display control 36 displays the reconstructed Doppler image 80 on the display 38. It is known that a blood flow rate is increased in the tumor TM, particularly at the periphery of a cancerous tumor. Therefore, the operator can review the reconstructed Doppler image 80 and infer that cancer is present in a case where there is a section with a high blood flow rate (≈ blood flow velocity). In particular, since cancer metastasizes, in a case where the reconstructed Doppler image 80, which represents the blood flow rate in the vicinity of a site where cancer is present, is displayed after a cancer resection procedure, it is possible to assist the operator in verifying whether the cancer, which was presumed to have been resected, remains or whether the cancer has metastasized to the periphery.

[0076] As in Fig. As shown in Figure 16, the display control 36 can display the reconstructed Doppler image 80 and the acquired image 24B (especially the partial acquisition image 24Ba) side by side. In this case, too, the partial acquisition image 24Ba and the reconstructed Doppler image 80 can be displayed on the display 38 with adjusted display directions and sizes.

[0077] Although the process support device according to the present disclosure has been described above, the process support device according to the present disclosure is not limited to the embodiment described above, and various modifications can be made without deviating from the core of the present disclosure.

[0078] For example, in the embodiment described above, the process support device is the ultrasonic diagnostic device 16, and each of the functions of the display control 36, the volume data acquisition unit 50, the target area specification unit 52, the reconstruction processing unit 54, and the distance measurement unit 56 is included in the ultrasonic diagnostic device 16, but each of these functions need not necessarily be performed by the ultrasonic diagnostic device 16. For example, these functions can be provided by a server computer or the like that is communicatively connected to the ultrasonic diagnostic device 16.In this case, the processor of the server computer or the like can act as the process support device, acquiring the ultrasound volume data 60 from the ultrasound diagnostic device or the like (functioning as the volume data acquisition unit 50), specifying the procedure target area in the ultrasound volume data 60 (functioning as the procedure target area specification unit 52), generating the reconstructed ultrasound image 70 from the ultrasound volume data 60 (functioning as the reconstruction processing unit 54), and displaying the reconstructed ultrasound image 70 on the display that can be viewed by the operator (functioning as the display controller 36). Furthermore, it is possible that not all of the functions described above are provided by a single device, and the functions described above can be provided through the cooperation of several devices. Reference symbol list 10 Process Support System 12 Camera 14 ultrasound probe 16 Ultrasound diagnostic device 20 probe detection markers 22 Body surface detection marker 24A image captured for detection 24B captured image for display 24Ba partial image 30 transmission and reception unit 32 Signal processing unit 34 Image generation unit 36 Display control 38 ads 40 Communication interface 42 Input interface 44 storage 46 Control 48 Acquisition unit for probe position and orientation information 50 volume data acquisition unit 52 Specification unit for process target area 54 Reconstruction Processing Unit 56 Distance measuring unit 60 ultrasound volume data 60a Partial volume data 62 single-image data sequence 64 individual image data 70 reconstructed ultrasound image 72 Position indicator 74 message windows 80 reconstructed Doppler image 82 Blood flow diagram CS cut cross section QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 2024-52409A

[0004]

Claims

[1] Process support device comprising: a captured image capture unit that captures a captured image for display, which is obtained by imaging a body surface section in an environment of a procedure target tissue in a subject under investigation; a volume data acquisition unit that acquires ultrasound volume data corresponding to an ultrasound irradiation area containing the procedure target tissue in the subject, which is formed on the basis of received signals obtained by transmitting and receiving ultrasound in the ultrasound irradiation area using an ultrasound probe in contact with the body surface section; a specification unit for procedure target area that specifies a procedure target area occupied by the procedure target tissue in the ultrasound volume data; a reconstruction processing unit that forms a reconstructed ultrasound image by cutting out the ultrasound volume data in a cut-out cross-section perpendicular to a depth direction of the ultrasound volume data and including the process target area to reconstruct the ultrasound volume data; and a display control that displays the captured image and the reconstructed ultrasound image on a display unit. [2] Process support device according to claim 1, further comprising: a probe position and orientation information acquisition unit that captures position and orientation information specifying the position and orientation of the ultrasound probe in a case where ultrasound volume data is acquired. wherein the image acquisition unit acquires an image for detection obtained by imaging a probe detection marker attached to the ultrasound probe in contact with the body surface section, The volume data acquisition unit acquires the ultrasound volume data, the coordinate information specifying each position of the ultrasound volume data, with reference to the position and orientation information in a camera coordinate system of a camera that captured the image for display, and The display control displays a partial image, which is a section of the captured image for display, corresponding to the reconstructed ultrasound image on the display unit based on the coordinate information contained in the ultrasound volume data. [3] Process support device according to claim 2, wherein the display control displays the partial image and the reconstructed ultrasound image with adapted display directions and display sizes thereof on the display unit. [4] Process support device according to claim 3, wherein the display control displays a position indicator on the display unit which indicates a position in the partial image that corresponds to a position of the process target area in the reconstructed ultrasound image. [5] Process support device according to claim 4, wherein the partial image contains an image of a body surface feature object on a body surface of the subject, The process support device further comprises a distance measuring unit that measures a distance between the body surface feature object and the position indicated by the position indicator in a real space based on a distance between the image of the body surface feature object and the position indicator in the partial image, and The display control shows the distance measured by the distance measuring unit on the display unit. [6] Process support device according to claim 1, wherein the reconstruction processing unit determines a position of the cut-out cross-section in the depth direction as a position at which an area of ​​the process target area in the cut-out cross-section is largest. [7] Process support device according to claim 1 or 6, wherein the reconstruction processing unit forms the reconstructed ultrasound image based on partial volume data which are a part of the ultrasound volume data, including the cut-out cross-section and having a thickness in the depth direction of the ultrasound volume data. [8] Process support device according to claim 1, wherein the volume data acquisition unit acquires the ultrasound volume data, which includes a Doppler signal indicating the velocity of blood flow in the subject at each position, The reconstruction processing unit creates a blood flow image representing the blood flow of the subject in the cut-out cross-section based on the Doppler signal and forms a reconstructed Doppler image in which the blood flow image is superimposed on the reconstructed ultrasound image, and The display control shows the reconstructed Doppler image on the display unit. [9] Process support device comprising: a volume data acquisition unit that acquires ultrasound volume data corresponding to an ultrasound irradiation area containing a procedure target tissue in a subject, which is formed on the basis of received signals obtained by transmitting and receiving ultrasound in the ultrasound irradiation area using an ultrasound probe in contact with a body surface section in the vicinity of the procedure target tissue in the subject, wherein the ultrasound volume data includes a Doppler signal indicating a velocity of blood flow in the subject at each position; a specification unit for procedure target area, which specifies a procedure target area corresponding to the procedure target tissue in the ultrasound volume data; a reconstruction processing unit that forms a reconstructed ultrasound image by extracting the ultrasound volume data containing the Doppler signal in a cut-out cross-section perpendicular to a depth direction of ultrasound volume data and the procedure target area, in order to reconstruct the ultrasound volume data that forms a blood flow image representing the blood flow of the subject in the cut-out cross-section based on the Doppler signal, and that forms a reconstructed Doppler image in which the blood flow image is superimposed on the reconstructed ultrasound image; and a display control that displays the reconstructed Doppler image on a display unit. [10] Process support program that causes a computer to act as: a captured image capture unit that captures a captured image for display, which is obtained by imaging a body surface section in an environment of a procedure target tissue in a subject under investigation; a volume data acquisition unit that acquires ultrasound volume data corresponding to an ultrasound irradiation area containing the procedure target tissue in the subject, which is formed on the basis of received signals obtained by transmitting and receiving ultrasound in the ultrasound irradiation area using an ultrasound probe in contact with the body surface section; a specification unit for procedure target area that specifies a procedure target area occupied by the procedure target tissue in the ultrasound volume data; a reconstruction processing unit that forms a reconstructed ultrasound image by cutting out the ultrasound volume data in a cut-out cross-section perpendicular to a depth direction of the ultrasound volume data and including the process target area to reconstruct the ultrasound volume data; and a display control that displays the captured image and the reconstructed ultrasound image on a display unit.

Images