Ultrasonic diagnostic apparatus, image display method, and program

Abstract

To easily and appropriately set the direction of a reference image such as an optical image displayed together with an ultrasonic image of puncture.SOLUTION: An ultrasonic diagnostic apparatus includes an ultrasonic image generation unit, an optical imaging unit, and a control unit. The ultrasound image generation unit generates ultrasound image data from a reception signal of an ultrasound probe that transmits and receives ultrasound waves to and from a subject. The optical imaging unit generates optical image data by optically imaging puncture using a puncture needle into a subject. The control part simultaneously displays an ultrasonic image 610 of the ultrasonic image data and an optical image 620 as a reference image on the basis of the optical image data. At the same time, the control unit detects the position of the user in the optical image 620 of the optical image data, and sets the orientation of the optical image 620 to be displayed so that the detected position of the user is on the lower side.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an ultrasound diagnostic apparatus, an image display method, and a program. [Background technology]

[0002] Conventionally, there has been known an ultrasound diagnostic apparatus that uses an ultrasound probe to irradiate ultrasound into the interior of a subject, receives and analyzes the reflected waves, and displays an ultrasound image of the interior of the subject. The subject is, for example, a living patient.

[0003] Ultrasound diagnostic devices are used not only to display ultrasound images of the internal living body of a subject, but also to insert a puncture needle into a target location while visually confirming the location of the puncture needle and a specific site (target) in the subject. A puncture needle is a hollow needle used to collect samples from a target in the subject, drain fluids, or inject or place drugs or markers into the target. This allows for quick, reliable, and easy treatment of the target in the subject.

[0004] In recent years, ultrasound-guided puncture procedures such as nerve blocks and central venous catheterization have been attracting attention. To avoid complications due to incorrect puncture during central venous catheterization, there is a need to reduce the difficulty of the procedure. Ultrasound diagnostic devices that assist in the puncture procedure are known to reduce the difficulty.

[0005] For example, an ultrasound diagnostic device is known that displays an ultrasound image and an optical image captured by an optical camera attached to an ultrasound probe side by side (see Patent Document 1). When a puncture needle is used, this ultrasound diagnostic device can flip the orientation of the optical image up, down, left, and right based on operational input from a user such as a doctor. The user manually changes the orientation of the optical image to match the insertion direction of the puncture needle. This allows the user to intuitively recognize the insertion mode of the puncture needle. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2023-121441 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-282792 Summary of the Invention [Problem to be solved by the invention]

[0007] However, in actual medical practice, it is cumbersome for a user to adjust the orientation of an optical image by operating the display or operation unit of the ultrasound diagnostic device. This is particularly true when the user has both hands occupied while holding the ultrasound probe and the puncture needle. Therefore, it is difficult to perform an operation to change the orientation of the optical image. Furthermore, if the ultrasound diagnostic device is far from the user, it is impossible to perform an operation to change the orientation of the optical image. Therefore, there is a demand for an easy and appropriate way to set the orientation of the optical image.

[0008] An object of the present invention is to easily and appropriately set the orientation of a reference image such as an optical image to be displayed together with an ultrasound image of a puncture. [Means for solving the problem]

[0009] In order to solve the above problem, the ultrasonic diagnostic apparatus of the invention described in claim 1 comprises: an ultrasound image generating unit that generates ultrasound image data from a reception signal of an ultrasound probe that transmits and receives ultrasound to and from a subject; an optical imaging unit that optically images the puncture of the subject using the puncture needle and generates optical image data; The system is equipped with a control unit that simultaneously displays an ultrasound image of the ultrasound image data and a reference image based on the optical image data, detects the position of the user in the optical image of the optical image data, and sets the orientation of the reference image to be displayed so that the detected user position is at the bottom.

[0010] The invention described in claim 2 is the ultrasound diagnostic device described in claim 1, The control unit performs image analysis on the optical image data to detect the position of the user in the optical image.

[0011] The invention described in claim 3 is the ultrasonic diagnostic apparatus described in claim 2, the optical imaging unit includes a first optical imaging unit that optically images the puncture of the puncture needle into the subject and generates first optical image data, and a second optical imaging unit that optically images the puncture of the puncture needle into the subject in an imaging direction different from that of the first optical imaging unit and generates second optical image data; The control unit simultaneously displays an ultrasound image of the ultrasound image data and a reference image based on the first optical image data, and performs image analysis of at least one of the first optical image data and the second optical image data to detect the position of the user in the first optical image of the first optical image data.

[0012] The invention described in claim 4 is the ultrasonic diagnostic apparatus described in claim 2, The control unit detects the position of the puncture needle in the optical image based on pattern information obtained by decoding symbols contained in the optical image of the optical image data, and sets the orientation of the reference image to be displayed so that the position of the user corresponding to the detected position of the puncture needle is at the bottom.

[0013] The invention described in claim 5 is the ultrasonic diagnostic apparatus described in claim 1, The reference image is the optical image.

[0014] The invention described in claim 6 is the ultrasonic diagnostic apparatus described in claim 1, The reference image is an illustration image that indicates the positions and directions of the ultrasound probe and the puncture needle in the optical image.

[0015] The invention described in claim 7 is the ultrasonic diagnostic apparatus described in claim 6, The control unit performs image analysis on the optical image data to obtain the positions and orientations of the ultrasound probe and the puncture needle, and generates illustration image data of an illustration image that indicates the positions and orientations of the ultrasound probe and the puncture needle.

[0016] The invention described in claim 8 is the ultrasonic diagnostic apparatus described in claim 1, The control unit performs image analysis on the optical image data to obtain a puncture mode for the puncture based on the positions and directions of the ultrasound probe and the puncture needle, and sets display elements corresponding to the obtained puncture mode to be displayed together with the ultrasound image and the reference image.

[0017] The image display method of the invention described in claim 9 comprises: an ultrasound image generating step of generating ultrasound image data from a reception signal of an ultrasound probe that transmits and receives ultrasound to and from the subject; an optical imaging step of optically imaging the puncture of the subject using the puncture needle to generate optical image data; The method includes a control process of simultaneously displaying an ultrasound image of the ultrasound image data and a reference image based on the optical image data, detecting the position of the user in the optical image of the optical image data, and setting the orientation of the reference image to be displayed so that the detected user position is at the bottom.

[0018] The program of the invention described in claim 10 is an ultrasound image generating unit that generates ultrasound image data from a reception signal of an ultrasound probe that transmits and receives ultrasound to and from a subject; an optical imaging unit that optically images the puncture of the subject using the puncture needle and generates optical image data; a control unit that simultaneously displays an ultrasound image of the ultrasound image data and a reference image based on the optical image data, detects a position of a user in the optical image of the optical image data, and sets an orientation of the reference image to be displayed so that the detected position of the user is at the bottom; Function as. [Effects of the Invention]

[0019] According to the present invention, the orientation of a reference image displayed together with an ultrasound image of a puncture can be easily and appropriately changed. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a diagram showing the external configuration of an ultrasound diagnostic apparatus according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing the functional configuration of the ultrasound diagnostic apparatus. [Figure 3] 1 is a perspective view showing a cross-sectional ultrasound probe, a puncture needle, an optical camera, and a laser pointer according to a first embodiment. FIG. [Figure 4] 1 is a perspective view showing a parallel method ultrasound probe, a puncture needle, an optical camera, and a laser pointer according to a first embodiment. FIG. [Figure 5] FIG. [Figure 6] FIG. 10 is a diagram showing a table of schematic diagrams showing the positional relationship of objects for each pattern information in the parallel method, cross method, camera viewpoint, and optical image direction. [Figure 7] FIG. 10 is a diagram illustrating an example of an optical image. [Figure 8] FIG. 10 is a diagram illustrating an example of an optical image. [Figure 9] 10 is a flowchart showing a first optical image setting process. [Figure 10] FIG. 10 is a perspective view showing a cross-sectional ultrasound probe, a puncture needle, an optical camera, and a laser pointer according to a second embodiment. [Figure 11] FIG. 10 is a perspective view showing another example of the cross-sectional ultrasound probe, the puncture needle, the optical camera, and the laser pointer of the second embodiment. [Figure 12] 10 is a flowchart showing a second optical image setting process. [Figure 13] FIG. [Figure 14] FIG. [Figure 15] FIG. [Figure 16]FIG. [Figure 17] FIG. 1 is a diagram showing a two-dimensional code. [Figure 18] 10 is a flowchart showing a third optical image setting process. [Figure 19] FIG. 2 is a diagram illustrating a user and a voice input unit. DETAILED DESCRIPTION OF THE INVENTION

[0021] Advantages and features provided by one or more embodiments of the present invention will be more fully understood from the following detailed description and the accompanying drawings. However, these drawings are for illustrative purposes only and are not intended to define the limits of the present invention. Hereinafter, first to fourth embodiments and first and second modifications of the present invention will be described with reference to the drawings. However, the scope of the present invention is not limited to the disclosed embodiments.

[0022] (First embodiment) A first embodiment of the present invention will be described with reference to Figs. 1 to 9. First, the device configuration of this embodiment will be described with reference to Figs. 1 and 2. Fig. 1 is a diagram showing the external configuration of an ultrasound diagnostic device 100 of this embodiment. Fig. 2 is a block diagram showing the functional configuration of the ultrasound diagnostic device 100.

[0023] An ultrasound diagnostic device 100 of this embodiment is installed in a medical facility such as a hospital. In this embodiment, the ultrasound diagnostic device 100 is used by a user such as a doctor or technician to perform a puncture operation using a puncture needle 5. As shown in FIG. 1 , the ultrasound diagnostic device 100 includes an ultrasound diagnostic device main body 1, an ultrasound probe 2, an optical camera 31, and a laser pointer 41. The optical camera 31 and the laser pointer 41 are connected to the ultrasound diagnostic device main body 1 via cables 32 and 42, respectively.

[0024] The ultrasound probe 2 is connected to the ultrasound diagnostic device main body 1. The ultrasound probe 2 transmits ultrasound waves (transmitted ultrasound waves) into a subject, such as a patient's living body, and receives reflected waves of the ultrasound waves (reflected ultrasound waves: echoes) reflected within the subject. The ultrasound probe 2 has an ultrasound probe main body 21, a cable 22, and a connector 23. The ultrasound probe main body 21 is the head of the ultrasound probe 2 and transmits and receives ultrasound waves. The cable 22 is connected to the ultrasound probe main body 21 and the connector 23. The cable 22 is a cable through which a drive signal for the ultrasound probe main body 21 and a received ultrasound signal flow. The connector 23 is a plug connector for connecting to a receptacle connector (not shown) of the ultrasound diagnostic device main body 1. Here, the connector 23 is a common connector for the cable 32 of the optical camera 31 and the cable 42 of the laser pointer 41.

[0025] The ultrasound diagnostic device main body 1 is connected to the ultrasound probe main body 21 via a connector 23 and a cable 22. The ultrasound diagnostic device main body 1 transmits an electrical drive signal to the ultrasound probe main body 21, causing the ultrasound probe main body 21 to transmit ultrasound waves to the subject. The ultrasound probe 2 generates a reception signal, which is an electrical signal, in response to the ultrasound reflected from inside the subject and received by the ultrasound probe main body 21. The ultrasound diagnostic device main body 1 creates an image of the internal state of the subject as ultrasound image data based on the reception signal generated by the ultrasound probe 2.

[0026] The ultrasound probe main body 21 has transducers 211 (FIG. 2) at the tip side. The number of transducers 211 can be set arbitrarily, and in practice, it is, for example, 192. The transducers are arranged, for example, in a one-dimensional array in the scanning direction (azimuth direction, long axis direction). The transducers may also be arranged in a two-dimensional array. In this embodiment, a linear scanning electronic scanning probe is adopted as the ultrasound probe 2. However, the ultrasound probe 2 may be either an electronic scanning type or a mechanical scanning type. The ultrasound probe 2 may also be any of a linear scanning type, a sector scanning type, or a convex scanning type. The ultrasound diagnostic device main body 1 and the ultrasound probe 2 may be configured to communicate wirelessly instead of by wire via a cable 22. This wireless communication may be UWB (Ultra Wide Band) or the like.

[0027] The ultrasound diagnostic apparatus main body 1 has an operation unit 11 and a display unit 17. The operation unit 11 accepts various operation inputs from the user. The operation unit 11 has operation elements such as push buttons, encoders, lever switches, joysticks, trackballs, keyboards, touchpads, and multifunction switches.

[0028] The display unit 17 has a display panel such as an LCD (Liquid Crystal Display) or an EL (Electro-Luminescence) display. The display unit 17 displays display information such as an ultrasound image based on ultrasound image data. In particular, the display unit 17 displays a composite image of the ultrasound image captured by the ultrasound probe 2 and the optical image captured by the optical camera 31.

[0029] In the ultrasound diagnostic device 100, the optical camera 31 and the laser pointer 41 are in a predetermined positional relationship with respect to the ultrasound probe 2 (ultrasound probe body 21). To achieve this, the optical camera 31 and the laser pointer 41 are attached to the ultrasound probe body 21 via a detachable attachment 202. The positional relationship of the optical camera 31 and the laser pointer 41 with respect to the ultrasound probe 2 is determined by the attachment 202. However, the attachment 202 may be configured to be able to adjust the attitude of the optical camera 31 and the laser pointer 41.

[0030] The attachment 202 is, for example, a screw-type clamping member, and is attached to the ultrasonic probe body 21 so as to clamp the ultrasonic probe body 21 from both the left and right sides. The attachment 202 is made of, for example, a material that can withstand disinfectants, such as POM (polyacetal). When the attachment 202 is attached to the ultrasonic probe body 21, the ultrasonic probe 2, optical camera 31, and laser pointer 41 are aligned. For this reason, a probe notch (not shown) is provided on the outer surface of the ultrasonic probe body 21. Furthermore, a protrusion (not shown) that fits into the probe notch is provided on the inner peripheral surface of the attachment 202.

[0031] The optical camera 31 is, for example, a general fiberscope camera that acquires an optical image signal using a built-in imaging element. The optical camera 31 has, for example, a zoom lens, and can magnify and image the object to be imaged (here, the body surface region of the subject O). The optical camera 31 is attached to the proximal end side of the ultrasound probe body 21.

[0032] The laser pointer 41 is, for example, a general semiconductor laser that outputs visible color laser light (for example, red laser light with a wavelength of 635 nm to 690 nm). The laser pointer 41 is attached to the proximal end side of the ultrasound probe body 21. The laser pointer 41 emits laser light onto the body surface of the subject 0 to form a predetermined projection image 401. The laser pointer 41 outputs the laser light so that the projection shape of the laser light, which is the projection image 401, onto the body surface of the subject 0 is a line shape, using a built-in diffraction grating or slit.

[0033] In the puncture operation, the puncture needle 5 is assumed to be inserted into the subject 0, for example, by the user freehand. The user operates the ultrasound diagnostic device 100 by contacting the ultrasonic beam transmitting / receiving surface of the ultrasound probe 2 with the body surface of the subject 0, thereby obtaining ultrasound image data of the inside of the subject 0. The user looks at the display unit 17 and confirms the position of the puncture target, such as a blood vessel, tissue, or lesion, inside the subject 0, which is displayed in the ultrasound image within the composite image. The user performs the puncture operation by determining the target insertion position and target posture of the puncture needle 5 when inserting it into the subject 0 from the optical image of the composite image. At this time, a projection image 401 is formed on the body surface of the subject by the laser light of the laser pointer 41 within the optical image. The target insertion position and target posture of the puncture needle 5 are displayed in the projection image 401. This allows the user to perform the puncture operation accurately.

[0034] 2, the ultrasound diagnostic device main body 1 includes an operation unit 11, a transmission unit 12, a reception unit 13, an ultrasound image generation unit 14, an optical image generation unit 141, an oscillation control unit 142, an image synthesis unit 15, a display control unit 16, a display unit 17, a control unit 18, and a storage unit 19. The optical camera 31 and the optical image generation unit 141 function as an optical imaging unit 30.

[0035] The operation unit 11 accepts various operation inputs from the user and outputs the operation signals to the control unit 18. The operation unit 11 may be formed integrally with the display screen of the display unit 17 and may include a touch panel that accepts touch inputs from the user.

[0036] The transmitting unit 12, under the control of the control unit 18, supplies a driving signal, which is an electrical signal, to the ultrasonic probe 2, causing the ultrasonic probe 2 to generate transmitted ultrasonic waves. The transmitting unit 12 generates transmitted ultrasonic waves by, for example, driving a continuous portion (e.g., 64) of a plurality of transducers (e.g., 192) arranged in the ultrasonic probe 2. Then, the transmitting unit 12 performs scanning by shifting the driven transducers in the scanning direction each time a transmitted ultrasonic wave is generated.

[0037] The receiving unit 13 receives the received signal, which is an analog electrical signal received from the ultrasound probe 2, and amplifies and AD (Analog to Digital) converts it in accordance with the control of the control unit 18. The receiving unit 13 adjusts the time phase of the digital received signal after AD conversion by providing a delay time for each individual path corresponding to each transducer, and then adds the signals (phased addition) to generate sound ray data.

[0038] The ultrasound image generating unit 14 performs envelope detection processing, logarithmic compression, and the like on the sound ray data from the receiving unit 13 under the control of the control unit 18. The ultrasound image generating unit 14 adjusts the dynamic range and gain of the sound ray data after processing such as logarithmic compression to convert the brightness, and generates B (Brightness) mode image data. In other words, B mode image data is tomographic image data in which the strength of the received signal when the image mode is B mode is represented by brightness. However, the ultrasound image generating unit 14 may also be configured to generate color Doppler image data or the like and superimpose it on the B mode image data.

[0039] The optical image generating unit 141 acquires optical image signals from the optical camera 31 via the cable 32 and generates optical image data in accordance with the control of the control unit 18. For example, the optical image generating unit 141 generates optical image data of a moving image by continuously generating optical image data in frame units based on the optical image signals sequentially obtained from the optical camera 31. Note that the optical image generating unit 141 may be configured to be built into the optical camera 31.

[0040] The oscillation control unit 142, under the control of the control unit 18, controls the drive current flowing through the laser diode of the laser pointer 41, and controls the on / off operation of the output operation of the laser light.

[0041] The image synthesis unit 15, under the control of the control unit 18, acquires ultrasound image data from the ultrasound image generation unit 14 and acquires optical image data from the optical image generation unit 141. The image synthesis unit 15 generates synthetic image data that displays an ultrasound image of the ultrasound image data and an optical image of the optical image data on the same display screen. The image synthesis unit 15 outputs the generated synthetic image data to the display control unit 16. The image synthesis unit 15 generates synthetic image data in real time every time new ultrasound image data is acquired and / or every time new optical image data is acquired. The image synthesis unit 15 outputs the generated synthetic image data to the display control unit 16.

[0042] The image composition unit 15 may be capable of changing the display mode of the ultrasound image and / or the optical image in the composite image in accordance with user settings input to the control unit 18 or the operation unit 11. The image composition unit 15 may also generate composite image data after performing predetermined image processing on the input ultrasound image data or input optical image data.

[0043] The display control unit 16 is, for example, a DSC (Digital Scan Converter). Under the control of the control unit 18, the display control unit 16 performs processing such as coordinate conversion on the composite image data input from the image synthesis unit 15 to convert the composite image data into an image signal for display.

[0044] The display unit 17 displays the composite image on the display panel in accordance with the image signal output from the display control unit 16 under the control of the control unit 18. The display unit 17 also displays various display information input from the control unit 18 on the display panel.

[0045] The control unit 18 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The control unit 18 reads various processing programs stored in the ROM, loads them into the RAM, and controls each unit of the ultrasound diagnostic apparatus 100 in cooperation with the CPU and the loaded programs. The ROM is composed of non-volatile memory such as a semiconductor. The ROM stores a system program corresponding to the ultrasound diagnostic apparatus 100, various processing programs executable on the system program, and various data such as a gamma table. In particular, the ROM stores a first optical image setting program for executing a first optical image setting process described below. These programs are stored in the RAM in the form of computer-readable program codes. The CPU sequentially executes operations in accordance with the program codes in the RAM. The RAM forms a work area for temporarily storing various programs executed by the CPU and data related to these programs.

[0046] The storage unit 19 is a storage unit such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores information such as ultrasound image data in a writable and readable manner.

[0047] For each component of the ultrasound diagnostic apparatus 100, some or all of the functions of each functional block can be implemented as a hardware circuit such as an integrated circuit. An example of an integrated circuit is an LSI (Large Scale Integration). Depending on the level of integration, an LSI may also be referred to as an IC (Integrated Circuit), a system LSI, a super LSI, or an ultra LSI. Furthermore, the method of implementing the integrated circuit is not limited to an LSI. The integrated circuit may also be implemented using a dedicated circuit or a general-purpose processor. The integrated circuit may also utilize a field programmable gate array (FPGA) or a reconfigurable processor, which allows the reconfiguration of the connections and settings of circuit cells within the LSI. Furthermore, some or all of the functions of each functional block may be implemented by software. In this case, the software is stored on one or more storage media, such as a ROM, an optical disk, or a hard disk, and is executed by a processor.

[0048] Next, crossover and parallel puncture modes will be described with reference to Figures 3 and 4. Figure 3 is a perspective view showing the crossover ultrasound probe 2, puncture needle 5, optical camera 31, and laser pointer 41 of this embodiment. Figure 4 is a perspective view showing the parallel puncture mode ultrasound probe 2, puncture needle 5, optical camera 31, and laser pointer 41 of this embodiment.

[0049] The cross method is a method of inserting the puncture needle 5 so that the long axis direction of the ultrasound probe 2 is perpendicular to the puncture needle 5. The long axis direction of the ultrasound probe 2 is the one-dimensional array direction of the transducers 211, and is the scanning direction (azimuth direction). The short axis direction of the ultrasound probe 2 is the direction perpendicular to the long axis direction, and is the elevation direction. The parallel method is a method of inserting the puncture needle 5 so that the long axis direction of the ultrasound probe 2 is parallel to the puncture needle 5.

[0050] The cross-sectional puncture method will be described with reference to Figure 3. Figure 3 is a perspective view of the ultrasound probe 2, viewed obliquely from above. The optical camera 31 and laser pointer 41 are attached by an attachment 202, side by side in the minor axis direction, corresponding to the central position 201 in the major axis direction of the ultrasound probe body 21. The optical camera 31 is attached so that the tip 210 of the ultrasound probe body 21, the projected image 401, and the region to be observed are reflected in the optical image. The region to be observed is the region to be observed in the ultrasound image of the body surface of the subject 0. However, the minor axis direction in Figure 3 has been shifted in the drawing for ease of viewing.

[0051] The projection image 401 guides the target position and target posture of the puncture needle 5 within the optical image, with the tip 210 as the reference position. For example, the projection image 401 presents a line shape that starts at the central position 201 on the body surface of the subject 0 and extends in the minor axis direction. The target posture of the puncture needle 5 is, for example, an appropriate orientation of the puncture needle 5 in a planar view (meaning a view from above the body surface of the subject 0).

[0052] The puncture needle 5 is inserted into a target site 501 in the subject 0 via a target insertion position 502, for example. Here, the extension direction of the projection image 401 of the linear laser light becomes the target posture of the puncture needle 5 when inserting the puncture needle 5 into the subject 0. For example, the target insertion position 502 is a position 2 cm vertically and 2 cm in the minor axis direction from the target site 501. At this time, the elevation angle (puncture angle) of the puncture needle 5 with respect to the minor axis direction is 45°. However, the puncture angle, target site 501, and target insertion position 502 are not limited to these examples.

[0053] The parallel puncture method will be described with reference to Figure 4. Figure 4 is a perspective view of the ultrasound probe 2 as seen obliquely from above. The optical camera 31 and laser pointer 41 are attached by an attachment 202 side by side along the long axis direction, corresponding to the center position 203 in the short axis direction of the ultrasound probe body 21. The optical camera 31 is attached so that the tip of the ultrasound probe body 21, the projected image 401, and the region to be observed are reflected in the optical image. The ultrasound image obtained by the ultrasound probe 2 is an ultrasound image corresponding to the scan cross section 240 of the ultrasound probe 2. The scan cross section 240 is a cross section passing through the acoustic axis direction of ultrasound, which is perpendicular to the long axis direction and the short axis direction.

[0054] The projection image 401 guides the target position and target posture of the puncture needle 5 within the optical image, with the tip 210 as the reference position. For example, the projection image 401 has a line shape on the body surface of the subject 0 that starts at the central position 201 and extends in the longitudinal direction.

[0055] The puncture needle 5 is inserted into a target site 501 in the subject 0 via a target insertion position 502, for example. Here, the extension direction of the projection image 401 of the linear laser light becomes the target posture of the puncture needle 5 when inserting the puncture needle 5 into the subject 0. For example, the target insertion position 502 is a position 2 cm vertically and 2 cm in the long axis direction from the target site 501. At this time, the elevation angle (puncture angle) of the puncture needle 5 with respect to the short axis is 45°. However, the puncture angle, target site 501, and target insertion position 502 are not limited to these examples.

[0056] Next, the operation of the ultrasound diagnostic device 100 will be described with reference to Figs. 5 to 9. Fig. 5 is a diagram showing a composite image 600. Fig. 6 is a diagram showing a table of schematic diagrams showing the positional relationship of objects in the parallel method, cross method, camera viewpoint, and optical image direction for each piece of pattern information. Fig. 7 is a diagram showing an example of an optical image. Fig. 8 is a diagram showing an example of an optical image. Fig. 9 is a flowchart showing the first optical image setting process.

[0057] Referring to FIG. 5, a composite image 600 will be described as an example of a composite image of composite image data generated by the image composition unit 15.

[0058] The composite image 600 includes an ultrasound image 610 and an optical image 620. The ultrasound image 610 is a B-mode image of the subject 0 based on ultrasound image data generated by the ultrasound image generation unit 14. The optical image 620 is an optical image based on optical image data generated by the optical image generation unit 141. The optical image 620 is, for example, an image captured when a user inserts a puncture needle from below to above using the cross-sectional method. The optical image 620 includes the center position 201 of the ultrasound probe body 21 and the body surface of the subject 0. The projection image 401 is omitted from the optical image 620. The optical image 620 may also be configured to include a first center line indicating the horizontal center of the optical image and a second center line indicating the vertical center. The optical image 620 does not show the user, but depicts the user facing downward, which is an appropriate orientation for puncture.

[0059] Hereinafter, the up, down, left, and right directions refer to the up, down, left, and right directions of the image plane or the directions of an object such as a user that correspond to the up, down, left, and right directions.

[0060] In the composite image 600, the ultrasound image 610 and the optical image 620 are configured to be divided into left and right sections on the display screen, but this is not limiting. For example, in the composite image, the ultrasound image and the optical image may be configured to be divided into top and bottom sections on the display screen, or the positions of the ultrasound image and the optical image may be swapped.

[0061] Next, the appropriate orientation of the optical image to be displayed will be explained with reference to Figure 6. In Figure 6, the vertical axis represents the parallel method, cross method, and camera viewpoint, and the positional relationship of the object regarding optical imaging for each item is shown in a table of schematic diagrams. The objects are the user, ultrasound probe body 21, optical camera 31, and puncture needle 5. The schematic diagram is a view of the plane of the body surface of subject 0 viewed from the vertical direction. Here, the laser pointer 41 and its projected image 401 are omitted.

[0062] Each schematic diagram in FIG. 6 is a plan view. In FIG. 6, the ultrasound probe body 21 is shown as a rectangle. The longitudinal direction of the rectangle corresponds to the long axis direction of the ultrasound probe body 21. Similarly, the short side direction of the rectangle corresponds to the short axis direction of the ultrasound probe body 21. The optical camera 31 is shown as an isosceles triangle. The apex angle of the isosceles triangle is the imaging direction (optical axis direction) of the optical camera 31. Furthermore, the imaging range 310 of the optical camera 31 is shown as a semicircle. The approximate center of the semicircle is the imaging plane of the optical camera 31. The puncture needle 5 is shown by an arrow. The direction of the arrow is the insertion direction of the puncture needle 5.

[0063] In Figure 6, the schematic diagram of the parallel method shows the user positioned on the lower side. The parallel method can be divided into four states depending on the position of the user relative to the ultrasound probe body 21, optical camera 31, and puncture needle 5. Similarly, the schematic diagram of the cross method shows the user positioned on the lower side. The cross method can also be divided into four states depending on the position of the user relative to the ultrasound probe body 21, optical camera 31, and puncture needle 5.

[0064] In FIG. 6, the schematic view of the camera viewpoint is a schematic view in which the optical camera 31 is positioned on the lower side and the imaging direction is upward. In the schematic view of the camera viewpoint, the ultrasound probe body 21 is omitted. Depending on the schematic view of the camera viewpoint, the schematic views of the parallel method and the cross method are classified into four pieces of pattern information. The four pieces of pattern information are called patterns 01, 02, 03, and 04.

[0065] In FIG. 6, the schematic diagram of the optical image orientation shows the optical image and object arrangement corresponding to the schematic diagram of the camera viewpoint. In the schematic diagram of the optical image orientation, the optical camera 31 and the puncture needle 5 are shown. The optical image 620 is shown as a large rectangle. In addition, in the schematic diagram of the optical image orientation, the user is positioned at the bottom. In this way, the optical image 620 is oriented in an appropriate display direction from the user's viewpoint. The appropriate orientation of the optical image 620 is determined by the direction (position) of the user and the puncture needle 5 relative to the optical image 620.

[0066] 7 and 8 show examples of an optical image 620. The optical image 620 in FIG. 7 is an optical image corresponding to the parallel method pattern 01 in FIG. 6. The optical image 620 in FIG. 7 shows the ultrasound probe body 21, the user's hand, the puncture needle 5 (or a syringe with the puncture needle), and the body surface of the subject 0. The optical image 620 in FIG. 7 is an optical image in a direction appropriate for the user's viewpoint from below. However, if the displayed optical image 620 in FIG. 7 is rotated in a planar view, it will not be an appropriate image for the user's viewpoint. For this reason, it is necessary to change the orientation of the displayed optical image 620, which has a different orientation, to the orientation of the optical image 620 in FIG. 7.

[0067] The optical image 620 in FIG. 8 is an optical image corresponding to the parallel method pattern 03 in FIG. 6. The optical image 620 in FIG. 8 also shows the ultrasound probe body 21, the user's hand, the puncture needle 5 (or the syringe with the puncture needle), and the body surface of the subject 0. The displayed optical image 620 in FIG. 8 is an optical image in a direction appropriate for the user's viewpoint from below. However, if the optical image 620 in FIG. 8 is also rotated in a planar view, it will not be an appropriate image for the user's viewpoint. For this reason, it is necessary to change the orientation of the displayed optical image 620, which has a different orientation, to the orientation of the optical image 620 in FIG. 8.

[0068] Next, with reference to FIG. 9 , the first optical image setting process executed by the ultrasound diagnostic device 100 will be described. The optical camera 31 and laser pointer 41 are attached to the ultrasound probe 2 in advance. In the ultrasound diagnostic device 100, the control unit 18 executes a composite image display process. As the composite image display process, the control unit 18 scans the subject with the ultrasound probe 2, generates a projection image with the laser pointer 41, and captures the imaging range with the optical camera 31. The control unit 18 generates ultrasound image data with the ultrasound image generation unit 14 and generates optical image data with the optical image generation unit 141. The control unit 18 generates composite image data with the image synthesis unit 15 and displays a live composite image based on the composite image data on the display unit 17. The user performs ultrasound-guided puncture. Specifically, the user inserts the puncture needle 5 into the target area of ​​the subject using the parallel or cross method while visually checking the ultrasound image and optical image of the composite image.

[0069] The first optical image setting process is a process for setting the orientation of the composite image of the optical image data generated in the composite image process to an appropriate direction based on the optical image. For example, when the composite image display process is executed, the control unit 18 executes the first optical image setting process in accordance with the first optical image setting program in the ROM.

[0070] First, the control unit 18 acquires the generated optical image data from the optical image generation unit 141 (step S11). The control unit 18 performs image analysis on the optical image data acquired in step S11 and determines whether or not the user's finger, hand, wrist, or arm at the end of the puncture needle 5 is detected (step S12). If the user's finger or the like (finger, hand, wrist, or arm) is not detected (step S12; NO), the process proceeds to step S11. If the user's finger or the like is detected (step S12; YES), the control unit 18 detects the direction (position) of the user (or their body) relative to the optical image from the detected user's finger or the like (step S13).

[0071] If the optical image has a sufficient angle of view in step S12, the control unit 18 may be configured to perform image analysis of the optical image data to detect the user's body and face. In this configuration, in step S13, the control unit 18 detects the user's direction relative to the optical image from the recognized user's body, etc.

[0072] The control unit 18 determines an appropriate orientation of the optical image from the user's viewpoint relative to the current optical image based on the user's direction detected in step S13 (step S14). That is, it is determined that the optical image to be displayed is directed downward with the detected user's direction facing downward. The control unit 18 sets the orientation of the optical image of the optical image data to the orientation determined in step S14 (step S15). Thereafter, the processing proceeds to step S11. The setting in step S15, for example, is performed by setting the image synthesis unit 15 to synthesize the optical image of the optical image data in the determined orientation. However, the present invention is not limited to this, and for example, the optical image data itself may be changed so that the optical image is set in the determined orientation.

[0073] Furthermore, since the process proceeds to step S11 after step S15, when the user's position changes, the orientation of the optical image also changes (follows) in real time.

[0074] As shown in the first optical image setting process, in step S11, the optical image data does not need to be acquired all at once. The user moves the ultrasound probe 2 (by moving its position or tilting its angle, for example) to expand the imaging range of the optical camera 31. This allows the direction of the user and the direction of the puncture needle 5 to be detected.

[0075] As described above, according to this embodiment, the ultrasound diagnostic device 100 includes an ultrasound image generation unit 14, an optical imaging unit 30, and a control unit 18. The ultrasound image generation unit 14 generates ultrasound image data from a reception signal of the ultrasound probe 2, which transmits and receives ultrasound to and from the subject. The optical imaging unit 30 optically images the insertion of the puncture needle 5 into the subject and generates optical image data. The control unit 18 simultaneously displays an ultrasound image of the ultrasound image data and an optical image as a reference image based on the optical image data on the display unit 17. The control unit 18 also detects the position of the user in the optical image of the optical image data and sets the orientation of the displayed optical image so that the detected position of the user is at the bottom.

[0076] Therefore, the orientation of the optical image displayed together with the ultrasound image of the puncture can be easily and appropriately set so that the user's position is at the bottom, without the user having to perform any operation input on the ultrasound diagnostic device 100. This allows the moving direction (orientation) of the needle image to be automatically adjusted to match the insertion direction of the puncture needle 5. This allows the user to intuitively and easily perform the puncture procedure, reducing the burden on the user.

[0077] Furthermore, the control unit 18 performs image analysis on the optical image data to determine the user's position in the optical image. Therefore, the orientation of the optical image displayed together with the ultrasound image of the puncture can be easily and automatically set appropriately so that the user's position is at the bottom, without any action by the user.

[0078] (Second embodiment) A second embodiment of the present invention will be described with reference to Figures 10 to 12. Figure 10 is a perspective view showing the cross-sectional ultrasound probe 2, the puncture needle 5, the optical cameras 31 and 33, and the laser pointer 41 of this embodiment. Figure 11 is a perspective view showing another example of the cross-sectional ultrasound probe 2, the puncture needle 5, the optical cameras 31 and 33, and the laser pointer 41 of this embodiment. Figure 12 is a flowchart showing the second optical image setting process.

[0079] In the first embodiment, the orientation of the optical image to be displayed is set to an appropriate direction using only the optical image data from the optical camera 31. In this embodiment, the orientation of the optical image to be displayed is set to an appropriate direction using the optical image data from both the optical cameras 31 and 33.

[0080] In this embodiment, as in the first embodiment, an ultrasonic diagnostic device 100 is used as the device configuration. However, in the ultrasonic diagnostic device 100 of this embodiment, a description of the configuration of the same parts as those of the ultrasonic diagnostic device 100 of the first embodiment will be omitted, and different parts will be mainly described.

[0081] As shown in FIG. 10 , the ultrasound diagnostic apparatus 100 of this embodiment further includes an optical camera 33. The optical imaging unit 300 includes the optical camera 33 and an optical image generation unit 141. The optical camera 33 is similar to the optical camera 31, but its imaging direction is different from that of the optical camera 31. The optical camera 33 is attached to, for example, an attachment 202. The optical camera 33 preferably has, for example, a fisheye lens, etc., to widen the imaging range. With a wide imaging range, it is possible to capture images of the user's face, body, etc. in addition to the puncture needle 5 and the user's hand. The optical camera 33 is connected to the optical image generation unit 141 via a cable (not shown) and a connector 23.

[0082] The optical image generation unit 141 acquires an optical image signal from the optical camera 31 via the cable 32 under the control of the control unit 18, and generates first optical image data. Similarly, the optical image generation unit 141 acquires an optical image signal from the optical camera 33, and generates second optical image data.

[0083] The image synthesis unit 15, under the control of the control unit 18, acquires ultrasonic image data from the ultrasonic image generation unit 14 and acquires first and second optical image data from the optical image generation unit 141. The image synthesis unit 15 generates synthetic image data of a synthetic image of an ultrasonic image of the ultrasonic image data and a first optical image of the first optical image data.

[0084] As shown in FIG. 11, the optical camera 33 may be attached to a position other than the attachment 202, such as the ultrasound diagnostic device main body 1.

[0085] The ROM of the control unit 18 stores a second optical image setting program for executing a second optical image setting process, which will be described later, in place of the first optical image setting program.

[0086] Next, the second optical image setting process executed by the ultrasound diagnostic apparatus 100 of this embodiment will be described with reference to Figure 12. As in the first embodiment, the control unit 18 first executes a composite image display process. However, in the composite image display process of this embodiment, composite image data of an ultrasound image and a first optical image based on the first optical image data from the optical image generation unit 141 is displayed on the display unit 17. The user inserts the puncture needle 5 into the target site of the subject using the parallel method or cross method while visually checking the ultrasound image and the optical image of the composite image.

[0087] The second optical image setting process is a process for setting the orientation of the composite image of the optical image data generated in the composite image process to an appropriate direction based on the optical image. For example, when the composite image display process is executed, the control unit 18 executes the second optical image setting process in accordance with the second optical image setting program in the ROM.

[0088] First, the control unit 18 acquires the generated first optical image data and second optical image data from the optical image generation unit 141 (step S21). The control unit 18 performs image analysis on at least one of the first and second optical image data acquired in step S21 (step S22). In step S22, the control unit 18 determines whether or not a user's finger, hand, wrist, or arm at the end of the puncture needle 5 is detected from the image analysis results. If a user's finger or the like is not detected (step S22; NO), the process proceeds to step S21. If a user's finger or the like is detected (step S22; YES), the control unit 18 detects the direction (position) of the user (or their body) relative to the first optical image from the detected user's finger or the like (step S23).

[0089] Based on the user's direction detected in step S23, the control unit 18 determines an appropriate orientation of the first optical image from the user's viewpoint relative to the current first optical image (step S24). The control unit 18 sets the orientation of the optical image of the optical image data to the orientation determined in step S24 (step S25). Thereafter, the process proceeds to step S21.

[0090] As described above, according to this embodiment, the device includes the optical imaging unit 30 as a first optical imaging unit and the optical imaging unit 300 as a second optical imaging unit. The optical imaging unit 30 optically images the insertion of the puncture needle into the subject and generates first optical image data. The optical imaging unit 300 optically images the insertion of the puncture needle into the subject in an imaging direction different from that of the optical imaging unit 30 and generates second optical image data. The control unit 18 simultaneously displays an ultrasound image of the ultrasound image data and a first optical image as a reference image based on the first optical image data. The control unit 18 also performs image analysis of at least one of the first optical image data and the second optical image data to detect the user's position in the first optical image of the first optical image data. Therefore, using multiple optical imaging units can expand the recognizable imaging range and more accurately detect the user's position (direction) in the first optical image.

[0091] (Third embodiment) A third embodiment of the present invention will be described with reference to Fig. 13 to Fig. 16. Fig. 13 is a diagram showing a composite image 630. Fig. 14 is a diagram showing a composite image 660. Fig. 15 is a diagram showing a composite image 700. Fig. 16 is a diagram showing a composite image 740.

[0092] In the first embodiment, the composite image 600 of the composite image data includes an ultrasound image 610 and an optical image 620. In this embodiment, the composite image of the composite image data includes an ultrasound image and an illustration image.

[0093] In this embodiment, similarly to the first embodiment, the ultrasound diagnostic device 100 is used as the device configuration. However, instead of the first optical image setting process described above, an illustration image data generation process is executed.

[0094] An example of a composite image displayed in this embodiment will be described with reference to FIGS. 13 and 14. As shown in FIG. 13, a composite image 630 is a composite image corresponding to the parallel method, and includes an ultrasound image 640 and an illustration image 650. The illustration image 650 is an image that displays only the parts of the ultrasound probe main body 21 and the puncture needle 5 among the schematic diagrams of the parallel method in FIG. 6. Here, the illustration image 650 is assumed to be an illustration image corresponding to the schematic diagram of the parallel method pattern 03 in FIG. 6. The illustration image 650 includes, for example, a needle image 651 of the puncture needle 5 and a probe image 652 of the ultrasound probe main body 21.

[0095] As shown in Fig. 14, a composite image 660 is a composite image corresponding to the cross-sectional method, and includes an ultrasound image 670 and an illustration image 680. The illustration image 680 is an image that displays only the ultrasound probe body 21 and the puncture needle 5 parts from among the schematic diagrams of the cross-sectional method in Fig. 6. Here, the illustration image 680 is assumed to be an illustration image that corresponds to the schematic diagram of pattern 03 of the cross-sectional method in Fig. 6. The illustration image 680 includes, for example, a needle image 681 of the puncture needle 5 and a probe image 682 of the ultrasound probe body 21.

[0096] Because illustration image 650 is easier to see than an optical image, it is made smaller in size than optical image 620 in composite image 600 in Fig. 5. Therefore, ultrasound image 640 is larger and has a larger visible range than ultrasound image 610 in composite image 600. The same applies to illustration image 680 and ultrasound image 670.

[0097] Additionally, the composite image 660 has a vertical line 690 in the horizontal center as a UI (User Interface) for puncture assistance. In the cross-sectional method, the presence of line 690 makes it easier for the user to confirm the puncture needle portion near line 690 in the ultrasound image 670. In the parallel-angle method, the needle image in the ultrasound image appears diagonally to the left and right. For this reason, line display is unnecessary in the ultrasound image 640.

[0098] Next, the illustration image data generation process executed by the ultrasound diagnostic device 100 will be described with reference to Fig. 9. In the ultrasound diagnostic device 100, the control unit 18 first executes a composite image display process. The composite image display process of this embodiment is similar to the composite image display process of the first embodiment, but the contents of the composite image data are different. The control unit 18 causes the display unit 17 to display composite image data of a composite image of the generated ultrasound image data and an illustration image of the illustration image data generated in the illustration image data generation process described below.

[0099] The illustration image data generation process will be described mainly focusing on the differences from the first optical image setting process of Fig. 9. In the illustration image data generation process, steps S11 to S14 of the first optical image setting process are common. In parallel with this, the control unit 18 performs image analysis of the optical image data acquired in step S11 to detect the puncture needle 5. From the detected puncture needle 5, the control unit 18 detects the position and (insertion) direction of the puncture needle 5 in the optical image. The control unit 18 also performs image analysis of the optical image of the optical image data to detect the position and direction of the ultrasound probe main body 21 in the optical image. The control unit 18 determines whether the schematic diagram is a parallel method or a cross-sectional method from the table of Fig. 6 based on the user's direction in step S13, the detected position and direction of the puncture needle 5, and the detected position and direction of the ultrasound probe main body 21.

[0100] The control unit 18 generates illustration image data of the determined schematic diagram corresponding to the orientation of the optical image of step S14. The illustration image of the illustration image data includes a needle image corresponding to the detected position and orientation of the puncture needle 5 and a probe image corresponding to the detected position and orientation of the ultrasound probe main body 21. The control unit 18 causes the image synthesis unit 15 to generate composite image data of a composite image of the illustration image of the illustration image data and the ultrasound image of the ultrasound image data. At this time, if the determined schematic diagram corresponds to the cross-sectional view, the control unit 18 generates part data of a line 690 as a UI. The control unit 18 causes the image synthesis unit 15 to synthesize the part data of the line 690 into the composite image data.

[0101] In the illustration image data generation process, similar to the first optical image setting process, after the illustration image data is generated, the process proceeds to the first step (acquiring the first optical image data). Therefore, when the user's position, the position and direction of the puncture needle 5, and the ultrasound probe 2 change, the positions of the illustration image, the needle image, and the probe image change (follow) in real time.

[0102] As described above, according to this embodiment, the reference image displayed together with the ultrasound image is an illustration image showing the position and orientation of the ultrasound probe 2 (ultrasound probe main body 21) and the puncture needle 5 in the optical image. The control unit 18 performs image analysis on the optical image data to acquire the position and orientation of the ultrasound probe 2 and the puncture needle 5. The control unit 18 generates illustration image data of an illustration image showing the position and orientation of the ultrasound probe 2 and the puncture needle 5. Therefore, even without an optical image, the user can easily confirm the positional relationship and orientation of the ultrasound probe 2 and the puncture needle 5, and can also confirm the puncture mode (parallel method or cross method). Furthermore, because the illustration image is easy to see, the display size of the illustration image can be smaller than that of the optical image. In this case, the display size of the ultrasound image can be increased, allowing the user to perform the puncture procedure more intuitively and easily.

[0103] Furthermore, the control unit 18 performs image analysis on the optical image data to acquire the puncture mode (parallel method or cross method) based on the positions and orientations of the ultrasound probe 2 and the puncture needle 5. The control unit 18 sets UI parts (lines 690 in the parallel method) as display elements corresponding to the acquired puncture mode to be displayed together with the ultrasound image and the illustration image. Therefore, the UI parts to be displayed superimposed on the optical image and ultrasound image can be automatically changed according to the puncture mode, improving user operability.

[0104] Note that the UI to be combined into the combined image depending on whether the parallel or cross-sectional method is used is not limited to the line 690. For example, a configuration may be adopted in which a combined image 700 is displayed as shown in FIG. 15 . The combined image 700 is a combined image when the parallel method is determined. The combined image 700 includes an ultrasound image 710, an optical image 720 replaced with an illustration image, and a needle trajectory prediction line 730 as a UI. The needle trajectory prediction line 730 is a linear part that indicates a predicted insertion path of the puncture needle 5 in the parallel method. Of the three lines of the needle trajectory prediction line 730, the middle line is a center line (the trajectory of the puncture needle 5). The two lines above and below the middle line indicate the width of the range in which the puncture needle 5 is expected to move. When the parallel method is determined, the control unit 18 analyzes the optical image data to predict a predicted insertion path of the puncture needle 5 and generates part data of the needle trajectory prediction line 730 corresponding to the prediction result. The control unit 18 causes the image synthesis unit 15 to synthesize the part data of the needle trajectory prediction line 730 into the synthetic image data.

[0105] Note that calculation of the predicted insertion path of the puncture needle 5 is not limited to prediction based only on analysis of optical image data from the optical camera 31. In predicting the predicted path, a stereo camera system, a TOF (Time Of Flight) sensor, a millimeter wave sensor, LiDAR (Light Detection And Ranging), etc. may also be used or used in combination. The stereo camera system is a measurement system that uses two optical cameras that can measure the depth direction of the puncture needle 5.

[0106] Alternatively, as shown in FIG. 16 , a configuration may be adopted in which a composite image 740 is displayed. The composite image 740 is a composite image when the crossover method is determined. The composite image 740 has an ultrasound image 750, an optical image 760 in place of an illustration image, and a mark 770 as a UI. The mark 770 is a rectangular part that indicates a predicted insertion path of the puncture needle 5 in the crossover method. When the crossover method is determined, the control unit 18 analyzes the optical image data to predict a predicted insertion path of the puncture needle 5 and generates part data of the mark 770 corresponding to the prediction result. The control unit 18 causes the image synthesis unit 15 to synthesize the part data of the mark 770 into the composite image data.

[0107] (Fourth embodiment) A fourth embodiment of the present invention will be described with reference to Fig. 17 and Fig. 18. Fig. 17 is a diagram showing two-dimensional codes 801, 802, 803, and 804. Fig. 18 is a flowchart showing a third optical image setting process.

[0108] In the first embodiment, the optical image data of the optical camera 31 is analyzed and the orientation of the optical image to be displayed is set to an appropriate direction. In this embodiment, the orientation of the optical image to be displayed is set by the user by reading a two-dimensional code.

[0109] In this embodiment, as in the first embodiment, an ultrasonic diagnostic device 100 is used. However, in the ultrasonic diagnostic device 100 of this embodiment, a description of the configuration of the same parts as those of the ultrasonic diagnostic device 100 of the first embodiment will be omitted, and different parts will be mainly described.

[0110] The ROM of the control unit 18 stores a third optical image setting program for executing a third optical image setting process, which will be described later, in place of the first optical image setting program.

[0111] Next, with reference to Fig. 17, the two-dimensional codes to be read by the optical camera 31 of the ultrasound diagnostic device 100 will be described. As shown in Fig. 17, the read targets are four types of two-dimensional codes 801, 802, 803, and 804. The two-dimensional codes 801, 802, 803, and 804 are QR (Quick Response) Codes (registered trademark), and contain different identification information. However, the read targets are not limited to two-dimensional codes, and may be other types of two-dimensional codes or symbols such as barcodes.

[0112] The "upper", "lower", "left", and "right" written above the two-dimensional codes 801 to 804 indicate, for example, the position of the puncture needle 5 within the optical image. That is, according to the table in the schematic diagram of Fig. 6, the two-dimensional code 801 includes the identification information of pattern 04, in which the puncture needle 5 is positioned on the upper side. However, the insertion direction of the puncture needle 5 is downward.

[0113] Similarly, two-dimensional code 802 includes identification information for pattern 03, in which the puncture needle 5 is located on the bottom. Two-dimensional code 803 includes identification information for pattern 02, in which the puncture needle 5 is located on the left. Two-dimensional code 804 includes identification information for pattern 01, in which the puncture needle 5 is located on the right.

[0114] The two-dimensional codes 801 to 804 are printed on, for example, paper. The user holds the paper on which the two-dimensional codes 801 to 804 are printed. Alternatively, the two-dimensional codes 801 to 804 may be displayed on a part of the display screen of the display unit 17.

[0115] Next, the operation of the ultrasound diagnostic device 100 will be described with reference to Fig. 18. As in the first embodiment, the control unit 18 first executes composite image display processing. While visually checking the ultrasound image and optical image of the composite image, the user inserts the puncture needle 5 into the target site of the subject using the parallel or cross method.

[0116] The third optical image setting process is a process for setting the orientation of the composite image of the optical image data generated in the composite image process to an appropriate orientation based on the result of reading the two-dimensional code. For example, when the composite image display process is executed, the control unit 18 executes the third optical image setting process in accordance with the second optical image setting program in the ROM.

[0117] The user visually checks the optical image of the composite image being displayed on the display unit 17. If the user wishes to change the orientation of the optical image so that the user is positioned at the bottom in the optical image, the user moves the optical camera 31 on the ultrasound probe main body 21. The user then causes the moved optical camera 31 to read one of the two-dimensional codes 801 to 804 on paper or being displayed.

[0118] First, the control unit 18 acquires the generated optical image data from the optical image generation unit 141 (step S31). The control unit 18 decodes the two-dimensional code in the optical image of the optical image data acquired in step S31 and determines whether or not the two-dimensional code (pattern information) is detected (step S32). If the two-dimensional code is not detected (step S32; NO), the process proceeds to step S31. If the two-dimensional code is detected (step S32; YES), the control unit 18 determines an appropriate orientation of the optical image from the user's viewpoint relative to the current optical image based on the pattern information of the detected two-dimensional code (step S33). For example, the control unit 18 performs image analysis on the optical image to detect the position (direction) of the puncture needle 5. The control unit 18 determines the orientation of the optical image so that the position of the puncture needle 5 corresponds to the pattern information and the user's position corresponding to the position of the puncture needle 5 is downward. The control unit 18 sets the orientation of the optical image of the optical image data to the direction determined in step S33 (step S34). Thereafter, the process proceeds to step S31.

[0119] As described above, according to this embodiment, the control unit 18 detects the position of the puncture needle 5 in the optical image based on pattern information obtained by decoding the two-dimensional code included in the optical image of the optical image data. The control unit 18 sets the orientation of the displayed optical image so that the user's position corresponding to the detected position of the puncture needle 5 is at the bottom. Therefore, by optically capturing a two-dimensional code with the desired pattern information, the user can easily and appropriately set the orientation of the optical image displayed together with the ultrasound image of the puncture so that the user's position is at the bottom. This allows the user to perform the puncture procedure intuitively and easily.

[0120] (First Modification) A first modification of the fourth embodiment will be described. The third embodiment was configured to read a two-dimensional code on an optical image to obtain pattern information, and then determine and change the orientation of the optical image. This modification is configured to recognize the voice uttered by the user to obtain pattern information, and then determine and change the orientation of the optical image.

[0121] In this modified example, the ultrasound diagnostic device 100 is also used as the device configuration. However, the ultrasound diagnostic device main body 1 includes an audio input unit (not shown) connected to the control unit 18. The audio input unit is a microphone that receives audio input from the user and outputs the audio signal to the control unit 18. The audio to be recognized is assumed to be "Pattern 01," "Pattern 02," "Pattern 03," or "Pattern 04." However, the audio to be recognized is not limited to the above examples as long as it corresponds to each piece of pattern information.

[0122] Next, the operation of the ultrasound diagnostic device 100 will be described. As in the first embodiment, the control unit 18 first executes composite image display processing. While visually checking the ultrasound image and optical image of the composite image, the user inserts the puncture needle 5 into the target area of ​​the subject using the parallel or cross method.

[0123] The control unit 18 executes an optical image setting process similar to the third optical image setting process. In step S32, the control unit 18 performs voice recognition of the voice data via the voice input unit and determines whether pattern information or corresponding voice information is detected. If voice information is detected (step S32; YES), in step S33, the control unit 18 determines an appropriate optical image orientation for the user's viewpoint relative to the current optical image based on the pattern information of the detected voice information.

[0124] As described above, according to this modified example, the user can easily and appropriately set the orientation of the optical image displayed together with the ultrasound image of the puncture so that the user's position is at the bottom by vocalizing the desired pattern information.

[0125] (Second Modification) A second modification of the third embodiment will be described with reference to Fig. 19. Fig. 19 is a diagram showing a user 9 and voice input units 901 and 902.

[0126] In the fourth embodiment, a two-dimensional code on an optical image is read to acquire pattern information, and the orientation of the optical image is then determined and changed. In this modified example, a voice input unit is used to detect the orientation of an object such as the ultrasound probe body 21, and the orientation of the optical image is then determined and changed.

[0127] In this modification, the ultrasound diagnostic device 100 is also used as the device configuration. However, the ultrasound diagnostic device 100 includes audio input units 901 and 902 connected to the control unit 18 via a cable. The audio input units 901 and 902 are microphones. The audio input units 901 and 902 receive input of sound waves from the user 9 and output the audio signal to the control unit 18. The audio input units 901 and 902 are disposed, for example, on the ultrasound probe 2 or the housing of the ultrasound diagnostic device main unit 1.

[0128] Consider the case where a user 9 is positioned and audio input units 901 and 902 are arranged as shown in FIG. 19. The audio input units 901 and 902 are arranged at a distance d apart. An axis perpendicular to the axis of distance d intersects with the direction of the sound waves of the audio emitted by the user 9 at an angle θ. In this case, the audio signal of audio input unit 901 is represented by X(t). The audio signal of audio input unit 902 is represented by X(t-τ). t is time. τ is the time difference. θ is expressed by the following equation (1). θ=sin -1 (cτ / d) …(1) where c is the speed of sound.

[0129] Therefore, by analyzing the time difference τ of the audio signals, it is possible to calculate the direction from the user 9 to the audio input units 901, 902. For example, if the audio input units 901, 902 are arranged side by side in the longitudinal direction of the ultrasound probe body 21, the orientation of the ultrasound probe body 21 (optical camera 31) on a plane from the user 9 can be determined.

[0130] In the first optical image setting process, for example, in detecting the direction of the user in step S13, the control unit 18 may use information on the analysis result of the audio signal input from the audio input units 901 and 902. The information on the analysis result of the audio signal is, for example, the direction of the user 9 from the ultrasound probe body 21 (optical camera 31). Furthermore, in the third embodiment, the direction of the user 9 from the ultrasound probe body 21 may be used to generate illustration image data.

[0131] In the above description, an example has been disclosed in which a ROM is used as a computer-readable medium for the program according to the present invention, but this is not limiting. Other computer-readable media include non-volatile memories such as flash memories and portable recording media such as CD-ROMs. Furthermore, a carrier wave is also applicable to the present invention as a medium for providing data for the program according to the present invention via a communication line.

[0132] The above-described embodiments and modifications are merely examples of the ultrasound diagnostic apparatus, image display method, and program according to the present invention, and are not intended to be limiting. For example, at least two of the above-described embodiments and modifications may be appropriately combined.

[0133] While embodiments and variations of the present invention have been described and illustrated in detail, the disclosed embodiments and variations are made for purposes of illustration and example only, and are not intended to be limiting. The scope of the present invention should be interpreted by the terms of the appended claims. [Explanation of symbols]

[0134] 100 Ultrasound diagnostic equipment 1. Ultrasound diagnostic device 11 Control section 12 Transmitter 13 Receiving unit 14 Ultrasound image generation unit 141 Optical image generation unit 142 Oscillation control section 15 Image synthesis unit 16 Display control unit 17 Display section 18 Control Unit 19 Memory section 2 Ultrasonic probe 21 Ultrasonic probe body 22, 32, 42 cables 23 Connector 30,300 Optical imaging unit 31,33 Optical camera 41 Laser Pointer 901,902 Audio input section

Claims

1. an ultrasound image generating unit that generates ultrasound image data from a reception signal of an ultrasound probe that transmits and receives ultrasound to and from a subject; an optical imaging unit that optically images the puncture of the subject using the puncture needle and generates optical image data; a control unit that simultaneously displays an ultrasound image of the ultrasound image data and a reference image based on the optical image data, detects a user's position in the optical image of the optical image data, and sets the orientation of the reference image to be displayed so that the detected user position is at the bottom.

2. The ultrasound diagnostic apparatus according to claim 1 , wherein the control unit performs image analysis on the optical image data to detect the position of the user in the optical image.

3. the optical imaging unit includes a first optical imaging unit that optically images the puncture of the puncture needle into the subject and generates first optical image data, and a second optical imaging unit that optically images the puncture of the puncture needle into the subject in an imaging direction different from that of the first optical imaging unit and generates second optical image data; 3. The ultrasound diagnostic device of claim 2, wherein the control unit simultaneously displays an ultrasound image of the ultrasound image data and a reference image based on the first optical image data, and performs image analysis of at least one of the first optical image data and the second optical image data to detect the position of the user in the first optical image of the first optical image data.

4. 3. The ultrasound diagnostic device according to claim 2, wherein the control unit detects the position of the puncture needle in the optical image based on pattern information obtained by decoding a symbol included in the optical image of the optical image data, and sets the orientation of the reference image to be displayed so that the position of the user corresponding to the detected position of the puncture needle is at the bottom.

5. The ultrasonic diagnostic apparatus according to claim 1 , wherein the reference image is the optical image.

6. The ultrasonic diagnostic apparatus according to claim 1 , wherein the reference image is an illustration image showing the positions and directions of the ultrasonic probe and the puncture needle in the optical image.

7. The ultrasound diagnostic device of claim 6, wherein the control unit performs image analysis on the optical image data to obtain the positions and directions of the ultrasound probe and the puncture needle, and generates illustration image data of an illustration image showing the positions and directions of the ultrasound probe and the puncture needle.

8. 2. The ultrasound diagnostic device according to claim 1, wherein the control unit performs image analysis on the optical image data to obtain a puncture mode based on positions and directions of the ultrasound probe and the puncture needle, and sets a display element corresponding to the obtained puncture mode to be displayed together with the ultrasound image and the reference image.

9. an ultrasound image generating step of generating ultrasound image data from a reception signal of an ultrasound probe that transmits and receives ultrasound to and from the subject; an optical imaging step of optically imaging the puncture of the subject using the puncture needle to generate optical image data; an ultrasound image based on the ultrasound image data and a reference image based on the optical image data simultaneously, and a control step of detecting a user's position in the optical image based on the optical image data and setting the orientation of the reference image to be displayed so that the detected user position is at the bottom.

10. an ultrasound image generating unit that generates ultrasound image data from a reception signal of an ultrasound probe that transmits and receives ultrasound to and from a subject; an optical imaging unit that optically images the puncture of the subject using the puncture needle and generates optical image data; a control unit that simultaneously displays an ultrasound image of the ultrasound image data and a reference image based on the optical image data, detects a position of a user in the optical image of the optical image data, and sets an orientation of the reference image to be displayed so that the detected position of the user is at the bottom; A program to function as a

Images