Ultrasound diagnostic system and method for controlling the ultrasound diagnostic system

Abstract

This invention provides an ultrasound diagnostic system and a control method for the ultrasound diagnostic system that can estimate the position and orientation of an ultrasound probe even when the ultrasound probe is in a blind spot in the field of view of an optical camera. [Solution] The ultrasound diagnostic system comprises an ultrasound probe (1) having a reference figure projection light source (11) that projects a reference figure, a projection member (4) onto which an image of the reference figure is projected from the reference figure projection light source (11), a diagnostic device (2) connected to the ultrasound probe (1), and an optical camera (3) connected to the diagnostic device (2) that acquires an optical image of the projection member (4). The diagnostic device (2) has a probe position estimation unit (24) that estimates the position and orientation of the ultrasound probe (1) based on the image of the reference figure projected onto the projection member (4) that is captured in the optical image.

Description

Technical Field

[0001] The present invention relates to an ultrasonic diagnostic system for estimating the position and orientation of an ultrasonic probe and a control method for the ultrasonic diagnostic system.

Background Art

[0002] Conventionally, an ultrasonic examination of a subject has been performed by taking an ultrasonic image representing a tomographic plane in the subject using a so-called ultrasonic probe. At this time, for example, in order for a user such as a doctor to confirm the position and orientation of the ultrasonic probe corresponding to the taken ultrasonic image after the examination of the subject, the position and orientation of the ultrasonic probe may be measured. As one method of measuring the position and orientation of the ultrasonic probe, for example, as disclosed in Patent Document 1, a technique of measuring the position and orientation of the ultrasonic probe by arranging a marker on the ultrasonic probe and photographing the marker with an optical camera is known. The method of measuring the position and orientation of the ultrasonic probe using a marker has an advantage that the necessary equipment is inexpensive and can be easily installed as compared with, for example, a method of measuring the position and orientation of the ultrasonic probe using a so-called magnetic sensor or the like.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, during the ultrasonic examination of the subject, the marker arranged on the ultrasonic probe may enter a blind spot in the visual field of the optical camera, such as being covered by the body of the subject. In this case, since the marker cannot be photographed by the optical camera, there is a problem that it is difficult to measure the position and orientation of the ultrasonic probe using the technique disclosed in Patent Document 1.

[0005] This invention was made to solve the problems of the conventional methods, and aims to provide an ultrasound diagnostic system and a control method for the ultrasound diagnostic system that can estimate the position and orientation of an ultrasound probe even when the ultrasound probe is in a blind spot in the field of view of an optical camera. [Means for solving the problem]

[0006] The above objective can be achieved with the following configuration. [1] An ultrasonic probe having a reference figure projection light source that projects a reference figure, A projection member onto which an image of a reference figure is projected from a reference figure projection light source, Diagnostic equipment connected to an ultrasound probe, An optical camera connected to a diagnostic device and which acquires an optical image of the projection target, Equipped with, The diagnostic device is Probe position estimation unit estimates the position and orientation of the ultrasonic probe based on the image of a reference figure projected onto the projection target member captured in the optical image. An ultrasound diagnostic system. [2] A marker is attached to the ultrasound probe. The ultrasound diagnostic system according to [1], wherein the probe position estimation unit further estimates the position and orientation of the ultrasound probe by taking into account a marker placed on the ultrasound probe that is visible in the optical image. [3] The system further includes a blind spot determination unit that determines whether or not the marker of the ultrasound probe has entered a blind spot in the field of view of the optical camera by analyzing the optical image acquired by the optical camera. The ultrasound diagnostic system according to [2], wherein the probe position estimation unit estimates the position and orientation of the ultrasound probe based solely on the image of the reference figure projected onto the projection member while the blind spot status determination unit determines that the marker has entered the blind spot of the optical camera. [4] The ultrasound diagnostic system according to [3], wherein the blind spot determination unit determines that the ultrasound probe has entered the blind spot of the optical camera if a condition in which a marker installed on the ultrasound probe cannot be read from the optical image continues for a predetermined period of time or longer. [5] The ultrasonic diagnostic system according to [3] or [4], wherein the reference figure projection light source stops projecting the reference figure onto the projection target when the blind spot status determination unit determines that the marker is located outside the blind spot of the optical camera, and starts projecting the reference figure onto the projection target when the blind spot status determination unit determines that the marker has entered the blind spot of the optical camera. [6] Equipped with additional memory, The diagnostic device has an image acquisition unit that acquires an ultrasound image of the subject by transmitting and receiving an ultrasound beam using an ultrasound probe. The ultrasound diagnostic system according to any one of [1] to [5], wherein the probe position estimation unit stores the estimated position and orientation information of the ultrasound probe and the ultrasound image in memory, linking them together. [7] Project a reference figure from the ultrasonic probe onto the object to be projected onto, An optical camera is used to acquire an optical image of the object to be projected onto. The position and orientation of the ultrasonic probe are estimated based on the image of a reference figure projected onto the projected object, which is captured in the optical image. A method for controlling an ultrasound diagnostic system. [Effects of the Invention]

[0007] The present invention relates to an ultrasonic diagnostic system comprising: an ultrasonic probe having a reference figure projection light source that projects a reference figure; a projected member onto which an image of the reference figure is projected from the reference figure projection light source; a diagnostic device connected to the ultrasonic probe; and an optical camera connected to the diagnostic device that acquires an optical image of the projected member. The diagnostic device has a probe position estimation unit that estimates the position and orientation of the ultrasonic probe based on the image of the reference figure projected onto the projected member captured in the optical image. Therefore, the position and orientation of the ultrasonic probe can be estimated even when the ultrasonic probe is in a blind spot in the field of view of the optical camera. [Brief explanation of the drawing]

[0008] [Figure 1] It is a block diagram showing the configuration of the ultrasonic diagnostic system according to Embodiment 1 of the present invention. [Figure 2] It is a block diagram showing the internal configuration of the ultrasonic probe and the diagnostic device in Embodiment 1 of the present invention. [Figure 3] It is a block diagram showing the internal configuration of the transmission / reception circuit in Embodiment 1 of the present invention. [Figure 4] It is a block diagram showing the internal configuration of the image generation unit in Embodiment 1 of the present invention. [Figure 5] It is a diagram schematically showing an example of the appearance of the ultrasonic probe in Embodiment 1 of the present invention. [Figure 6] It is a diagram schematically showing an example of the projection member in Embodiment 1 of the present invention. [Figure 7] It is a diagram schematically showing an example of an optical image in which the projection member and the ultrasonic probe are shown. [Figure 8] It is a diagram schematically showing an example of an AR display image in which a virtual display is added to the optical image. [Figure 9] It is a flowchart showing the operation of the ultrasonic diagnostic system according to Embodiment 1 of the present invention. [Figure 10] It is a diagram schematically showing a modified example of the appearance of the ultrasonic probe in Embodiment 1 of the present invention. [Figure 11] It is a block diagram showing the internal configuration of the ultrasonic probe and the diagnostic device in Embodiment 2 of the present invention.

Embodiments for Carrying Out the Invention

[0009] Hereinafter, embodiments of this invention will be described based on the accompanying drawings. The description of the constituent elements described below is based on typical embodiments of the present invention, but the present invention is not limited to such embodiments. In this specification, a numerical range represented by "~" means a range including the numerical values described before and after "~" as the lower limit value and the upper limit value. In this specification, "identical" and "the same" shall include the error ranges generally acceptable in the technical field.

[0010] Embodiment 1 Fig. 1 shows the configuration of an ultrasonic diagnostic system according to Embodiment 1 of the present invention. The ultrasonic system includes an ultrasonic probe 1, a diagnostic device 2 connected to the ultrasonic probe 1, one optical camera 3 connected to the diagnostic device 2, and a projection member 4 composed of a so-called screen or the like having a planar shape. The ultrasonic probe 1 has a reference figure projection light source 11 for projecting a reference figure described later onto the projection member 4.

[0011] Fig. 2 shows the internal configurations of the ultrasonic probe 1 and the diagnostic device 2. The ultrasonic probe 1 and the diagnostic device 2 are connected to each other by so-called wired communication or so-called wireless communication. The ultrasonic probe 1 includes a vibrator array 12 and a transmission / reception circuit 13 connected thereto.

[0012] The diagnostic device 2 includes an image generation unit 21 connected to the transmission / reception circuit 13. In the diagnostic device 2, a display control unit 22 and a monitor 23 are sequentially connected to the image generation unit 21. Further, the diagnostic device 2 includes a probe position estimation unit 24 connected to the optical camera 3. An AR (Augmented Reality) display image generation unit 25 is connected to the optical camera 3, the image generation unit 21, and the probe position estimation unit 24. Also, a memory 26 is connected to the image generation unit 21 and the probe position estimation unit 24. The optical camera 3, the AR display image generation unit 25, and the memory 26 are connected to the display control unit 22. Also, a device control unit 27 is connected to the optical camera 3, the reference figure projection light source 11 and the transmission / reception circuit 13 of the ultrasonic probe 1, the image generation unit 21, the display control unit 22, the probe position estimation unit 24, the AR display image generation unit 25, and the memory 26. An input device 28 is connected to the device control unit 27.

[0013] The image acquisition unit 29 is composed of a transmitting / receiving circuit 13 and an image generation unit 21. Furthermore, the processor 30 for the diagnostic device 2 is composed of the image generation unit 21, a display control unit 22, a probe position estimation unit 24, an AR display image generation unit 25, and a device control unit 27.

[0014] The ultrasound probe 1 is used to capture so-called ultrasound images representing cross-sectional planes within a subject by transmitting an ultrasound beam into the subject while in contact with the subject's body surface and receiving ultrasound echoes reflected from within the subject.

[0015] The transducer array 12 of the ultrasonic probe 1 has a plurality of ultrasonic transducers arranged in one or two dimensions. Each of these ultrasonic transducers transmits ultrasound according to a drive signal supplied from the transmitting / receiving circuit 13, and also receives ultrasonic echoes from the subject and outputs a signal based on the ultrasonic echoes. Each ultrasonic transducer is constructed by forming electrodes at both ends of a piezoelectric body made of, for example, a piezoelectric ceramic represented by PZT (Lead Zirconate Titanate), a polymer piezoelectric element represented by PVDF (Poly Vinylidene Di Fluoride), or a piezoelectric single crystal represented by PMN-PT (Lead Magnesium Niobate-Lead Titanate).

[0016] The image acquisition unit 29, which consists of a transmitting / receiving circuit 13 and an image generation unit 21, acquires an ultrasonic image by transmitting and receiving an ultrasonic beam using an ultrasonic probe 1.

[0017] The transmitting / receiving circuit 13 transmits ultrasonic waves from the transducer array 12 and generates a sound line signal based on the received signal acquired by the transducer array 12, under the control of the device control unit 27. As shown in Figure 2, the transmitting / receiving circuit 13 includes a pulser 41 connected to the transducer array 12, and an amplifier 42, an AD (Analog to Digital) converter 43, and a beamformer 44 connected sequentially in series from the transducer array 12.

[0018] The pulser 41 includes, for example, multiple pulse generators, and based on a transmission delay pattern selected in response to a control signal from the device control unit 27, it supplies each drive signal to the multiple ultrasonic transducers of the transducer array 12, adjusting the delay amount, so that the ultrasonic waves transmitted from the transducers form an ultrasonic beam. In this way, when a pulsed or continuous wave voltage is applied to the electrodes of the ultrasonic transducers of the transducer array 12, the piezoelectric material expands and contracts, generating pulsed or continuous wave ultrasonic waves from each ultrasonic transducer, and an ultrasonic beam is formed from the combined wave of these ultrasonic waves.

[0019] The transmitted ultrasonic beam is reflected by an object, such as a part of the subject, and propagates toward the transducer array 12 of the ultrasonic probe 1. The ultrasonic echo propagating toward the transducer array 12 in this way is received by each ultrasonic transducer that makes up the transducer array 12. At this time, each ultrasonic transducer that makes up the transducer array 12 expands and contracts upon receiving the propagating ultrasonic echo, generating a received signal which is an electrical signal, and outputs these received signals to the amplification unit 42.

[0020] The amplification unit 42 amplifies the signals input from each ultrasonic transducer constituting the transducer array 12 and transmits the amplified signals to the AD conversion unit 43. The AD conversion unit 43 converts the signals transmitted from the amplification unit 42 into digital received data. The beamformer 44 performs so-called receive focus processing by adding each received data received from the AD conversion unit 43 with a corresponding delay. Through this receive focus processing, each received data converted by the AD conversion unit 43 is added in phase and a sound ray signal with a focused ultrasonic echo is obtained.

[0021] As shown in Figure 3, the image generation unit 21 has a configuration in which a signal processing unit 45, a DSC (Digital Scan Converter) 46, and an image processing unit 47 are connected in series in sequence.

[0022] The signal processing unit 45 receives the sound line signal from the transmitting / receiving circuit 13, applies a sound velocity value set by the device control unit 27 to correct for attenuation due to distance according to the depth of the ultrasonic reflection position, and then performs envelope detection processing to generate a B-mode image signal, which is tomographic image information of the tissue within the subject.

[0023] The DSC46 converts the B-mode image signal generated by the signal processing unit 45 into an image signal that follows the scanning method of a normal television signal (raster conversion). The image processing unit 47 performs various necessary image processing, such as gradation processing, on the B-mode image signal input from the DSC 46, and then sends the B-mode image signal to the display control unit 22, the AR display image generation unit 25, and the memory 26. Hereafter, the B-mode image signal processed by the image processing unit 47 will be referred to as the ultrasound image.

[0024] Here, the ultrasound probe 1 has a housing H as shown in Figure 5, for example. Inside the housing H are a transducer array 12 and a transmitting / receiving circuit 13. The housing H has an array housing H1 which is the part that contacts the body surface of the subject and houses the transducer array 12, and a gripping part H2 for a user such as a doctor to grasp the ultrasound probe 1. The reference graphic projection light source 11 can be attached, for example, to the tip H2A of the gripping part H2 which is located on the opposite side from the array housing H1. In Figure 5, as an example, the ultrasound probe 1 is depicted as a so-called wired probe that is connected to the diagnostic device 2 via a connecting cable CW.

[0025] The reference figure projection light source 11 is a light source for projecting a reference figure, which can be used as an AR marker such as ArUco (Augmented Reality University of Cordoba), toward the projection target 4. The reference figure projection light source 11 can be configured, for example, by a small projector or projection device.

[0026] The projection target member 4 is a member having a projection area A1 that has a planar shape and no pattern, as shown in Figure 6, for example. The projection area A1 displays an image R1 of a reference figure projected by the reference figure projection light source 11. The projection target member 4 may have a marker M1 that can be used as an AR marker such as ArUco. This marker M1 is different from the reference figure projected onto the projection target member 4 from the reference figure projection light source 11. The type of projection target member 4 is not particularly limited as long as it has a projection area A1, and may be a screen as shown in Figure 6, or a wall or ceiling of an inspection room.

[0027] The optical camera 3 acquires an optical image of the projection target 4. The optical camera 3 includes an image sensor such as a so-called CCD (Charge Coupled Device) image sensor or a so-called CMOS (Complementary Metal-Oxide-Semiconductor) image sensor, and under the control of the device control unit 27, acquires an optical image Q of the ultrasonic probe 1 and the projection target 4, as shown in Figure 7, for example. In the example in Figure 7, an optical image Q is shown of the ultrasonic probe 1 held in the user's hand J and the projection target 4 onto which an image R1 of a reference figure is projected by a reference figure projection light source 11 attached to the ultrasonic probe 1. The optical camera 3 can be fixed and positioned in a location where the ultrasonic probe 1 and the projection target 4 can be clearly photographed.

[0028] The probe position estimation unit 24 estimates the position and orientation of the ultrasonic probe 1 based on the image R1 of a reference figure projected onto the projection member 4, which is captured in the optical image Q acquired by the optical camera 3. In this process, the probe position estimation unit 24 first reads the marker M1 placed on the projection member 4 to estimate the position and orientation of the projection member 4 relative to the optical camera 3. Next, the probe position estimation unit 24 reads the image R1 of the reference figure projected onto the projection member 4 to estimate the position and orientation of the ultrasonic probe 1 relative to the projection member 4. Finally, the probe position estimation unit 24 can estimate the position and orientation of the ultrasonic probe 1 relative to the optical camera 3 based on the position and orientation information of the projection member 4 relative to the optical camera 3 and the position and orientation information of the ultrasonic probe 1 relative to the projection member 4.

[0029] Here, since the position and orientation of the projected member 4 relative to the optical camera 3 are fixed, the probe position estimation unit 24 can also store the position and orientation of the projected member 4 relative to the optical camera 3 in advance. In this case, the probe position estimation unit 24 can estimate the position and orientation of the ultrasonic probe 1 relative to the optical camera 3 based on the stored information on the position and orientation of the projected member 4 relative to the optical camera 3 and the information on the position and orientation of the ultrasonic probe 1 relative to the projected member 4, which is estimated by reading the image R1 of the reference figure projected onto the projected member 4, without reading the marker M1 placed on the projected member 4.

[0030] The probe position estimation unit 24 can read the marker M1 on the projected member 4 and the image R1 of the reference figure by using a known algorithm for reading the figure used as an AR marker. For example, if the marker M1 on the projected member 4 and the image R1 of the reference figure represent ArUco, the marker M1 on the projected member 4 and the image R1 of the reference figure can be read using the ArUco algorithm included in the OpenCV® library.

[0031] Generally, when reading a figure used as an AR marker and estimating the distance between the optical camera 3 and the figure, so-called intrinsic parameters including the focal length, optical center, and shear coefficient of the optical camera 3's lens, so-called extrinsic parameters representing the rotational and translational degrees of freedom of the optical camera 3, and the lens distortion coefficient are required. Before estimating the position and orientation of the ultrasonic probe 1, the probe position estimation unit 24 obtains the intrinsic parameters, extrinsic parameters, and distortion coefficient from the optical camera 3, and estimates the position and orientation of the ultrasonic probe 1 relative to the optical camera 3 from the obtained intrinsic parameters, extrinsic parameters, and distortion coefficient, as well as the reading results of the marker M1 on the projected member 4 and the image R1 of the reference figure.

[0032] The AR display image generation unit 25 generates an AR display image E, such as the one shown in Figure 8, based on the optical image Q acquired by the optical camera 3, the ultrasonic image U acquired by the image acquisition unit 29, and the position and orientation information of the ultrasonic probe 1 estimated by the probe position estimation unit 24, and displays the AR display image E on the monitor 23 via the display control unit 22. In the example in Figure 8, the AR display image E shows the ultrasonic probe 1 held in the user's hand J and the projected member 4 onto which the image R1 of a reference figure is projected by the reference figure projection light source 11 attached to the ultrasonic probe 1. The AR display image E also virtually shows three direction vectors D representing the orientation of the ultrasonic probe 1 and the ultrasonic image U displayed extending from the tip of the ultrasonic probe 1. By checking the image thus virtually shown, the user can easily visually grasp the orientation of the ultrasonic probe 1 relative to the subject and the positional relationship of the acquired ultrasonic image U relative to the subject.

[0033] Memory 26 stores the ultrasound image U acquired by the image acquisition unit 29 and the position and orientation information of the ultrasound probe 1 estimated by the probe position estimation unit 24, linking them together. Users who are not proficient in interpreting ultrasound images U may have difficulty understanding which part of the subject the ultrasound image U represents. However, by checking the position and orientation information of the ultrasound probe 1 stored in correspondence with the ultrasound image U, they can easily understand which part of the subject the ultrasound image U represents.

[0034] For memory 26, for example, recording media such as flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), FD (Flexible Disk), MO disk (Magneto-Optical disk), MT (Magnetic Tape), RAM (Random Access Memory), CD (Compact Disc), DVD (Digital Versatile Disc), SD card (Secure Digital card), or USB memory (Universal Serial Bus memory) can be used.

[0035] The display control unit 22, under the control of the device control unit 27, performs predetermined processing on the ultrasound image U acquired by the image acquisition unit 29 and the AR display image E generated by the AR display image generation unit 25, and displays them on the monitor 23.

[0036] The monitor 23 displays ultrasound images U, etc., under the control of the display control unit 22, and has a display device such as an LCD (Liquid Crystal Display) or an organic EL display (Organic Electroluminescence Display).

[0037] The device control unit 27 controls each part of the diagnostic device 2, the transmitting and receiving circuit 13 of the ultrasonic probe 1, the reference figure projection light source 11, and the optical camera 3 based on a control program stored in advance.

[0038] The input device 28 is for the user to perform input operations and consists of devices such as a keyboard, mouse, trackball, touchpad, and touch sensor placed on top of the monitor 23.

[0039] In this embodiment, each process is executed on any computer. Furthermore, any computer may execute these processes using a processor 30 as hardware, a program as software, or a combination thereof. In this case, the processor 30 is configured to cooperate with the program to execute the various processes in this embodiment, and can function as a unit or means in this embodiment. Also, the execution order of the processes by the processor 30 is not limited to the order described and may be changed as appropriate. Any computer may be a general-purpose computer, a computer designed for a specific purpose, a workstation, or any other system capable of executing each process.

[0040] The processor 30 may be composed of one or more hardware components, and the type of hardware is not limited. For example, the processor 30 may be composed of hardware such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), FPGA (Field Programmable Gate Array) or other programmable logic devices, an ASIC (Application Specific Integrated Circuit) or other dedicated circuit for executing specific processing, a GPU (Graphic Processing Unit), or an NPU (Neural Processing Unit). Furthermore, the type of hardware may be a combination of different types of hardware. When multiple hardware components are configured to execute one or more processes of the processor 30, these multiple hardware components may reside in physically separate devices or in the same device. Also, in any embodiment, the order of each process performed by the processor 30 is not limited to the order described above and may be changed as appropriate. The hardware is composed of electrical circuits (circuitry) that combine circuit elements such as semiconductor elements.

[0041] Furthermore, the program may be firmware or software such as microcode. Alternatively, the program may be, for example, a group of program modules, each function of which may be implemented by a processor 30 configured to perform its respective function. The program may be program code or multiple code segments stored on one or more non-temporary computer-readable media (e.g., storage media or other storage). The program may be divided and stored on multiple non-temporary computer-readable media located in physically separate devices. Program code or code segments may represent any combination of procedures, functions, subprograms, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Program code or code segments may be connected to other code segments or hardware circuits by sending and receiving information, data, arguments, parameters, or memory contents.

[0042] Next, the operation of the ultrasound diagnostic apparatus according to the embodiment will be described with reference to the flowchart shown in Figure 9. The optical camera 3 is positioned so that the ultrasound probe 1 and the projection target member 4 are within its field of view.

[0043] In step S1, the device control unit 27 performs calibration of the optical camera 3. This calibration allows the acquisition of the intrinsic parameters, extrinsic parameters, and distortion coefficients of the optical camera 3. The device control unit 27 can perform calibration of the optical camera 3 by using, for example, the method described in Z. Zhang, "A flexible new technique for camera calibration", IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.

[0044] In step S2, position information including the position and orientation of the projected member 4 relative to the optical camera 3 is acquired. For example, as shown in Figure 6, if a marker M1 is placed on the projected member 4, an optical image Q showing the marker M1 on the projected member 4 is acquired by the optical camera 3 under the control of the device control unit 27, and the marker M1 shown in the optical image Q is read by the probe position estimation unit 24, thereby acquiring the position information of the projected member 4 relative to the optical camera 3. Furthermore, if the positional relationship between the optical camera 3 and the projected member 4 is stored in advance, the probe position estimation unit 24 can acquire the previously stored position information of the projected member 4 relative to the optical camera 3. In addition, the probe position estimation unit 24 can also acquire a value input by the user via the input device 28 as the position information of the projected member 4 relative to the optical camera 3.

[0045] Once step S2 is complete, the user, such as a physician, grasps the ultrasound probe 1 and begins an ultrasound examination of the subject to capture an ultrasound image U representing a cross-sectional view of the subject. In step S3, the user positions the ultrasound probe 1 on the subject's body surface to capture the ultrasound image U. The image acquisition unit 29 acquires the ultrasound image U. At this time, under the control of the device control unit 27, the transmission and reception of ultrasound is started from multiple transducers of the transducer array 12 according to the drive signal from the pulser 41 of the transmitting and receiving circuit 13 of the ultrasound probe 1. The ultrasound echo from within the subject is received by multiple transducers of the transducer array 12, the received signal, which is an analog signal, is output to the amplification unit 42 for amplification, and the AD conversion unit 43 performs AD conversion to acquire the received data.

[0046] The beamformer 44 performs reception focus processing on this received data, and the resulting sound line signal is sent to the image generation unit 21 of the diagnostic device 2, where the image generation unit 21 generates an ultrasonic image U. At this time, the signal processing unit 45 of the image generation unit 21 performs attenuation correction according to the depth of the ultrasonic reflection position and envelope detection processing on the sound line signal, and the DSC 46 converts it into an image signal following the scanning method of a normal television signal, and the image processing unit 47 performs various necessary image processing such as gradation processing. The ultrasonic image U generated in step S2 in this way is sent to the display control unit 22, the AR display image generation unit 25, and the memory 26.

[0047] In step S4, the device control unit 27 projects a reference figure from the reference figure projection light source 11 attached to the ultrasonic probe 1 toward the projection member 4. As a result, an image R1 of the reference figure is projected onto the projection member 4, for example, as shown in Figure 6.

[0048] In step S5, the optical camera 3, under the control of the device control unit 27, acquires an optical image Q as shown in Figure 7, which captures the image R1 of the reference figure projected onto the projection member 4 in step S4.

[0049] In step S6, the probe position estimation unit 24 estimates the position and orientation of the ultrasonic probe 1 based on the position and orientation of the projected member 4 relative to the optical camera 3 acquired in step S2 and the image R1 of the reference figure captured in the optical image Q acquired in step S5. At this time, the probe position estimation unit 24 estimates the position and orientation of the ultrasonic probe 1 relative to the projected member 4 by reading the image R1 of the reference figure captured in the optical image Q acquired in step S5, and estimates the position and orientation of the ultrasonic probe 1 relative to the optical camera 3 from the information on the position and orientation of the projected member 4 relative to the optical camera 3 acquired in step S2 and the estimated information on the position and orientation of the ultrasonic probe 1 relative to the projected member 4.

[0050] Generally, a technique is known in which the position and orientation of the ultrasound probe 1 relative to the optical camera 3 is estimated by reading a graphic used as an AR marker placed on the ultrasound probe 1 with the optical camera 3. However, depending on the position on the subject to which the ultrasound probe 1 is in contact, the ultrasound probe 1 may enter a blind spot in the field of view of the optical camera 3, i.e., it may go out of the field of view of the optical camera 3. In this case, there is a problem in that the position and orientation of the ultrasound probe 1 cannot be estimated.

[0051] According to the process in step S6, the position and orientation of the ultrasonic probe 1 relative to the optical camera 3 are estimated from the position and orientation information of the projected member 4 relative to the optical camera 3 acquired in step S2 and the estimated position and orientation information of the ultrasonic probe 1 relative to the projected member 4. Therefore, even if the ultrasonic probe 1 is in a blind spot in the field of view of the optical camera 3, the position and orientation of the ultrasonic probe 1 relative to the optical camera 3 can be estimated.

[0052] In step S7, the AR display image generation unit 25 generates an AR display image E as shown in Figure 8 based on the ultrasound image U acquired in step S3, the optical image Q acquired in step S5, and the position and orientation information of the ultrasound probe 1 estimated in step S6, and displays this on the monitor 23. By checking the AR display image E displayed on the monitor 23, the user can easily visually understand the orientation of the ultrasound probe 1 relative to the subject and the positional relationship of the acquired ultrasound image U relative to the subject.

[0053] In step S8, the device control unit 27 determines whether or not to terminate the ultrasound examination of the subject. The device control unit 27 can determine to terminate the ultrasound examination of the subject if, for example, the user inputs an instruction to terminate the ultrasound examination via the input device 28, and to continue the ultrasound examination of the subject if no specific instruction is input by the user via the input device 28.

[0054] As long as it is determined in step S8 to continue the ultrasound examination of the subject, the processes from step S3 to step S8 are repeated. If it is determined in step S8 to end the ultrasound examination of the subject, the operation of the ultrasound diagnostic system according to the flowchart shown in Figure 9 is completed.

[0055] As described above, according to the ultrasound diagnostic system of Embodiment 1 of the present invention, the probe position estimation unit 24 estimates the position and orientation of the ultrasound probe 1 based on the image R1 of a reference figure projected onto the projection member 4, which is captured in the optical image Q acquired by the optical camera 3. Therefore, even when the ultrasound probe 1 is in a blind spot of the optical camera 3's field of view, the position and orientation of the ultrasound probe 1 relative to the optical camera 3 can be estimated.

[0056] Although it is explained that the transmitting / receiving circuit 13 is provided in the ultrasound probe 1, the transmitting / receiving circuit 13 may also be provided in the diagnostic device 2. Furthermore, although it is explained that the image generation unit 21 is provided in the diagnostic device 2, the image generation unit 21 may also be provided in the ultrasound probe 1.

[0057] The diagnostic device 2 may be a stationary type, a portable type that is easy to carry, or a handheld type consisting of, for example, a smartphone or tablet computer. Thus, the types of equipment that make up the diagnostic device 2 are not particularly limited.

[0058] Figure 5 shows that the reference figure projection light source 11 is attached to the housing H of the ultrasonic probe 1 from the outside, but the method of attachment is not particularly limited as long as it is attached to the ultrasonic probe 1. For example, a hole can be formed in the housing H, and the reference figure projection light source 11 can be built into the housing H so as to project the reference figure through the hole.

[0059] In relation to Figure 8, it is explained that the directional vector D and the ultrasound image U are virtually represented in the AR display image E, but the virtual representation is not limited to this. For example, the AR display image E may include either the directional vector D or the ultrasound image U.

[0060] In step S7 of the flowchart in Figure 9, it is explained that an AR display image E is generated and displayed on the monitor 23. However, instead, the ultrasound image U acquired in step S3 and the position and orientation information of the ultrasound probe 1 estimated in step S6 can be linked together and stored in the memory 26. Alternatively, the AR display image E can be generated and displayed on the monitor 23, and the position and orientation information of the ultrasound image U and the ultrasound probe 1 can also be linked together and stored in the memory 26.

[0061] Furthermore, although the flowchart in Figure 8 explains that steps S4 to S6 are performed after step S3, it is also possible to perform step S3 after steps S4 to S6, and to perform steps S3 and S4 to S6 simultaneously.

[0062] Furthermore, as shown in Figure 10, for example, a marker M2 that can be used as an AR marker can be placed on the housing H of the ultrasonic probe 1A. This marker M2 is different from the marker M1 placed on the projected member 4 and the reference figure projected toward the projected member 4 by the reference figure projection light source 11. When the ultrasonic probe 1A is within the field of view of the optical camera 3, the probe position estimation unit 24 can estimate the position and orientation of the ultrasonic probe 1A by taking into account the marker M2 placed on the housing H of the ultrasonic probe 1A, in addition to the image R1 of the reference figure projected onto the projected member 4. This improves the accuracy of estimating the position and orientation of the ultrasonic probe 1A.

[0063] Furthermore, while an example is shown in which the marker M1 placed on the projection target 4, the reference figure projected onto the projection target 4 from the reference figure projection light source 11, and the marker M2 placed on the ultrasonic probe 1A are composed of white and black patterns, the colors are not particularly limited, and they can also be composed of patterns with colors other than white and black.

[0064] Embodiment 2 In Embodiment 1, the position and orientation of the ultrasonic probe 1 are always estimated using the image R1 of the reference figure projected onto the projection member 4. However, the position and orientation of the ultrasonic probe 1 can also be estimated using the image R1 of the reference figure only when the ultrasonic probe 1 is in a blind spot of the optical camera 3's field of view.

[0065] Figure 11 shows the configuration of the ultrasonic probe 1A and diagnostic device 2A in Embodiment 2. The ultrasonic probe 1A is the same as the ultrasonic probe 1 in Embodiment 1, except that a marker M2 is placed in its housing H, as shown in Figure 10.

[0066] Diagnostic device 2A is the same as diagnostic device 2 in Embodiment 1 shown in Figure 2, but further includes a blind spot state determination unit 51 and has a device control unit 27A instead of a device control unit 27. In diagnostic device 2A, the blind spot state determination unit 51 is connected to the optical camera 3. The blind spot state determination unit 51 is connected to the device control unit 27A. The reference figure projection light source 11 is connected to the device control unit 27A. Furthermore, the image generation unit 21, display control unit 22, probe position estimation unit 24, AR display image generation unit 25, device control unit 27A, and blind spot state determination unit 51 constitute a processor 30A for diagnostic device 2A.

[0067] The blind spot determination unit 51 analyzes the optical image Q acquired by the optical camera 3 to determine whether the marker M2 of the ultrasound probe 1A has entered the blind spot of the optical camera 3's field of view. The blind spot determination unit 51 can determine that the ultrasound probe 1A has entered the blind spot of the optical camera 3's field of view if, for example, the marker M2 of the ultrasound probe 1A is not read from the optical image Q in the same manner as the probe position estimation unit 24, and the state in which the marker M2 installed on the ultrasound probe 1A is not read from the optical image Q continues for a predetermined period of time or longer.

[0068] Under the control of the device control unit 27A, the reference figure projection light source 11 stops projecting the reference figure onto the projection target member 4 when the blind spot state determination unit 51 determines that the marker M2 is located outside the blind spot of the optical camera 3's field of view, i.e., within the optical camera 3's field of view. When the blind spot state determination unit 51 determines that the marker M2 has entered the blind spot of the optical camera 3's field of view, it starts projecting the reference figure onto the projection target member 4.

[0069] The probe position estimation unit 24 estimates the position and orientation of the ultrasonic probe 1A relative to the optical camera 3 based on the marker M2 of the ultrasonic probe 1A while it is determined that the marker M2 of the ultrasonic probe 1A is located within the field of view of the optical camera 3. Furthermore, while it is determined that the marker M2 of the ultrasonic probe 1A is in a blind spot of the field of view of the optical camera 3, the probe position estimation unit 24 estimates the position and orientation of the ultrasonic probe 1A based on the image R1 of the reference figure projected from the reference figure projection light source 11 onto the projection target member 4.

[0070] As described above, according to the ultrasound diagnostic system of Embodiment 2, even when the position and orientation of the ultrasound probe 1A are estimated based on the image R1 of the reference figure projected onto the projection member 4 only when the ultrasound probe 1A is in a blind spot in the field of view of the optical camera 3, the position and orientation of the ultrasound probe 1A relative to the optical camera 3 can be estimated regardless of whether the ultrasound probe 1A is in a blind spot in the field of view of the optical camera 3 or not. [Explanation of symbols]

[0071] 1,1A Ultrasonic probe, 2,2A Diagnostic device, 3 Optical camera, 4 Projected member, 11 Reference figure projection light source, 12 Transducer array, 13 Transmit / receive circuit, 21 Image generation unit, 22 Display control unit, 23 Monitor, 24 Probe position estimation unit, 25 AR display image generation unit, 26 Memory, 27,27A Device control unit, 28 Input device, 29 Image acquisition unit, 30,30A Processor, 41 Pulsar, 42 Amplifier unit, 43 AD conversion unit, 44 Beamformer, 45 Signal processing unit, 46 DSC, 47 Image processing unit, 51 Blind spot status determination unit, A1 Projected area, CW Connection cable, D Direction vector, E AR display image, H Housing, H1 Array housing unit, H2 Gripping unit, H2A Tip unit, J Hand, M1,M2 Marker, Q Optical image, R1 Image, U Ultrasonic image.

Claims

1. An ultrasonic probe having a reference figure projection light source that projects a reference figure, A projection member onto which an image of the reference figure is projected from the reference figure projection light source, A diagnostic device connected to the aforementioned ultrasound probe, An optical camera connected to the diagnostic device and which acquires an optical image of the projection member, Equipped with, The diagnostic device is A probe position estimation unit estimates the position and orientation of the ultrasonic probe based on the image of the reference figure projected onto the projection member as captured in the optical image. An ultrasound diagnostic system.

2. A marker is attached to the ultrasonic probe. The ultrasound diagnostic system according to claim 1, wherein the probe position estimation unit further estimates the position and orientation of the ultrasound probe by taking into account the marker installed on the ultrasound probe that is captured in the optical image.

3. The system further includes a blind spot determination unit that determines whether or not the marker of the ultrasonic probe has entered a blind spot in the field of view of the optical camera by analyzing the optical image acquired by the optical camera. The ultrasonic diagnostic system according to claim 2, wherein the probe position estimation unit estimates the position and orientation of the ultrasonic probe based solely on the image of the reference figure projected onto the projection member while the blind spot state determination unit has determined that the marker has entered the blind spot of the optical camera.

4. The ultrasound diagnostic system according to claim 3, wherein the blind spot condition determination unit determines that the ultrasound probe has entered the blind spot of the optical camera if the state in which the marker installed on the ultrasound probe cannot be read from the optical image continues for a predetermined period of time or longer.

5. The ultrasonic diagnostic system according to claim 3, wherein the reference figure projection light source stops projecting the reference figure onto the projection member when the blind spot state determination unit determines that the marker is located outside the blind spot of the optical camera, and starts projecting the reference figure onto the projection member when the blind spot state determination unit determines that the marker has entered the blind spot of the optical camera.

6. With even more memory, The diagnostic device has an image acquisition unit that acquires an ultrasound image of a subject by transmitting and receiving an ultrasound beam using the ultrasound probe. The ultrasound diagnostic system according to claim 1, wherein the probe position estimation unit stores the estimated position and orientation information of the ultrasound probe and the ultrasound image in the memory, linking them together.

7. A reference figure is projected from the ultrasonic probe onto the object to be projected onto. An optical image of the projection member is acquired using an optical camera. The position and orientation of the ultrasonic probe are estimated based on the image of the reference figure projected onto the projection member as captured in the optical image. A method for controlling an ultrasound diagnostic system.

Images