Ultrasound diagnostic device and method for controlling the ultrasound diagnostic device

Abstract

The present invention provides an ultrasonic diagnostic device and a method for controlling the ultrasonic diagnostic device in which an ultrasonic image generation unit (31) generates an ultrasonic image including a site being examined, and an optical camera (33) generates an optical image including a subject in contact with an ultrasonic probe (1). A display control unit (43) causes a monitor (41) to display the ultrasonic image and the optical image, and a first image memory (35) stores therein ultrasonic images and optical images generated in a certain past period of time later than a time specified by a user. An image selection unit (37) selects one frame of ultrasonic images and one frame of optical images from among the ultrasonic images and the optical images stored in the first image memory (35), and a second image memory (39) stores therein the one frame of ultrasonic images and the one frame of optical images in association with each other. This makes it possible to save the trouble of having to re-select a desired image when using an optical image instead of a body mark.

Description

[Technical Field]

[0001] The present invention relates to an ultrasonic diagnostic apparatus that generates an ultrasonic image including an examination site of a subject, and a method for controlling the ultrasonic diagnostic apparatus. [Background technology]

[0002] When examining a subject using an ultrasound diagnostic device, it is important to record an ultrasound image including the area to be examined in association with a body mark for identifying the position and orientation of the ultrasound probe at the time the ultrasound image was generated, for use in subsequent diagnosis. Conventionally, when adding a body mark to an ultrasound image, it was necessary to manually add a probe mark representing the position and orientation of the ultrasound probe at the time the ultrasound image was generated to the body mark to be added to the ultrasound image for each ultrasound image, which was extremely time-consuming. In response to this, Patent Documents 1 and 2 propose storing an ultrasound image in association with an optical image (moving image) of the subject at the time the ultrasound image was generated. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-131099 [Patent Document 2] Japanese Patent Application Publication No. 2019-193778 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Documents 1 and 2, optical images can be referenced instead of body marks. However, simply storing an ultrasound image and an optical image in association with each other, as in Patent Documents 1 and 2, poses a problem in that, for example, during a later diagnosis, each time the images from the examination are viewed again, it is time-consuming to reselect the desired frame of ultrasound image and frame of optical image from the associated and stored ultrasound images.

[0005] An object of the present invention is to provide an ultrasonic diagnostic apparatus and a method for controlling an ultrasonic diagnostic apparatus that can eliminate the need to reselect a desired image when using an optical image instead of a body mark. [Means for solving the problem]

[0006] In order to achieve the above object, the present invention provides an ultrasonic wave probe and an apparatus main body connected to the ultrasonic wave probe, The device body is an ultrasound image generating unit that generates an ultrasound image including the examination point from a received signal obtained by transmitting and receiving an ultrasound beam to and from the examination point of the subject using an ultrasound probe; an optical camera for generating an optical image including the object in contact with the ultrasound probe; The monitor and a display control unit that displays the ultrasound image and the optical image on a monitor; a first image memory that stores ultrasound images and optical images generated during a certain period of time from a timing designated by a user; an image selection unit that selects one frame of an ultrasound image and one frame of an optical image from among the ultrasound images and optical images stored in the first image memory; and a second image memory that stores one frame of an ultrasonic image and one frame of an optical image in association with each other.

[0007] Here, it is preferable that the display control unit, in response to an instruction from the user, scrolls back a past ultrasound image specified by the user from among the ultrasound images stored in the first image memory and displays it on the monitor, and scrolls back a past optical image specified by the user from among the optical images stored in the first image memory and displays it on the monitor.

[0008] Preferably, the image selection unit selects one frame of an ultrasonic image and one frame of an optical image based on a user's designation.

[0009] Furthermore, it is preferable that the image selection unit automatically selects at least one of one frame of an ultrasound image and one frame of an optical image.

[0010] Furthermore, it is preferable that the image selection unit automatically selects both one frame of an ultrasound image and one frame of an optical image.

[0011] Furthermore, it is preferable that the image selection unit automatically selects one of the one frame of ultrasound image and the one frame of optical image, and automatically selects the other of the one frame of ultrasound image and the one frame of optical image from among the ultrasound images and optical images stored in the first image memory that were generated within a certain period before and after the generation time of one of the images.

[0012] Furthermore, it is preferable that the image selection unit selects one of the one frame of ultrasound image and the one frame of optical image based on a user specification, and automatically selects the other of the one frame of ultrasound image and the one frame of optical image.

[0013] In addition, it is preferable that the image selection unit automatically selects one of the ultrasound images and optical images stored in the first image memory from among the ultrasound images and optical images generated within a certain period before or after the generation time of the other image.

[0014] In addition, when a user processes at least one of a frame of ultrasound image and a frame of optical image, it is preferable that the second image memory stores the processed frame of ultrasound image and the processed frame of optical image in association with each other.

[0015] The present invention also provides a control method for an ultrasonic diagnostic apparatus including an ultrasonic probe and an apparatus main body connected to the ultrasonic probe, the method comprising: an ultrasound image generating unit included in the device body generating an ultrasound image including the examination point from a received signal obtained by transmitting and receiving an ultrasound beam to and from the examination point of the subject using an ultrasound probe; A step in which an optical camera included in the device body generates an optical image including the object in contact with the ultrasound probe; a step in which a display control unit included in the device main body displays the ultrasound image and the optical image on a monitor included in the device main body; a step of storing, in a first image memory of the device body, ultrasound images and optical images generated during a certain period of time from a timing designated by a user; an image selection unit included in the device body selecting one frame of an ultrasound image and one frame of an optical image from the ultrasound images and optical images stored in the first image memory; and a step of storing one frame of an ultrasonic image and one frame of an optical image in association with each other in a second image memory provided in the main body of the apparatus. [Effects of the Invention]

[0016] In the present invention, one frame of ultrasound image and one frame of optical image are stored in association with each other. As a result, according to the present invention, when a user wants to re-view the images from the examination, for example, during a later diagnosis, the user can avoid the need to re-select a desired frame of ultrasound image and one frame of optical image from the ultrasound images and optical images stored in the first image memory. Furthermore, by referring to one frame of optical image, the user can easily understand the position and orientation of the ultrasound probe 1 at the time of generating the one frame of ultrasound image associated with this one frame of optical image. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a block diagram illustrating a configuration of an ultrasonic diagnostic apparatus according to an embodiment. [Figure 2] FIG. 2 is a block diagram illustrating a configuration of a transmission / reception circuit according to an embodiment. [Figure 3] FIG. 2 is a block diagram illustrating a configuration of an ultrasound image generating unit according to an embodiment. [Figure 4] 1 is a flowchart illustrating an operation of an ultrasound diagnostic apparatus according to an embodiment. [Figure 5] FIG. 1 is a conceptual diagram of an embodiment showing a display screen of a monitor during an examination of a subject. [Figure 6] FIG. 10 is a conceptual diagram of another embodiment showing the display screen of the monitor after the freeze button is pressed. [Figure 7A] 1 is a conceptual diagram of one embodiment of an optical image captured of a subject undergoing an abdominal examination. [Figure 7B] 1 is a conceptual diagram of another embodiment of an optical image taken of a subject undergoing abdominal examination. DETAILED DESCRIPTION OF THE INVENTION

[0018] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The ultrasonic diagnostic apparatus and the method for controlling the ultrasonic diagnostic apparatus of the present invention will be described in detail below based on preferred embodiments shown in the accompanying drawings.

[0019] Fig. 1 is a block diagram showing the configuration of an embodiment of an ultrasonic diagnostic apparatus according to the present invention. The ultrasonic diagnostic apparatus shown in Fig. 1 is a handheld ultrasonic diagnostic apparatus, and includes an ultrasonic probe 1 and a device main body 3 connected to the ultrasonic probe 1. The ultrasonic diagnostic apparatus of this embodiment is realized by the ultrasonic probe 1, the device main body 3, and an application program for ultrasonic diagnosis that runs on the device main body 3.

[0020] The ultrasonic probe 1 scans an examination location on a subject with an ultrasonic beam and outputs sound ray signals corresponding to an ultrasound image of the examination location. As shown in Fig. 1, the ultrasonic probe 1 includes a transducer array 11, a transmission / reception circuit 13, and a battery 15. The transducer array 11 and the transmission / reception circuit 13 are bidirectionally connected, and the transmission / reception circuit 13 is connected to a device control unit 47 of the device main body 3, which will be described later. The ultrasonic probe 1 also includes a built-in battery 15.

[0021] The transducer array 11 has a plurality of ultrasound transducers arranged one-dimensionally or two-dimensionally. Each of these transducers transmits ultrasound waves in accordance with a drive signal supplied from the transmission / reception circuit 13, and receives reflected waves from the subject and outputs an analog reception signal. Each vibrator is made of, for example, PZT (Lead Zirconate Titanate). Representative piezoelectric ceramics include PVDF (Polyvinylidene Difluoride). It is composed of an element with electrodes formed on both ends of a piezoelectric body made of a polymer piezoelectric element such as lead magnesium niobate-lead titanate (PMN-PT) or a piezoelectric single crystal such as lead magnesium niobate-lead titanate (PMN-PT).

[0022] The transmission / reception circuit 13, under the control of the device control unit 47, causes the transducer array 11 to transmit ultrasonic beams and generates acoustic ray signals by performing reception focusing processing on reception signals output from the transducer array 11 that have received ultrasonic echoes. As shown in Fig. 2, the transmission / reception circuit 13 has a pulser 51 connected to the transducer array 11, and an amplifier 53, an AD (Analog-to-Digital) converter 55, and a beamformer 57 that are connected in series from the transducer array 11 in this order.

[0023] The pulser 51 includes, for example, a plurality of pulse generators, and performs a transmission focusing process in which the delay amount of each drive signal is adjusted and supplied to the plurality of transducers in the transducer array 11 so that the ultrasonic waves transmitted from the plurality of transducers form an ultrasonic beam based on the transmission delay pattern selected by the device control unit 47. When this transmission focusing process applies a pulsed or continuous wave voltage to the electrodes of the transducers in the transducer array 11, the piezoelectric material expands and contracts, and pulsed or continuous wave ultrasonic waves are generated from each transducer, and an ultrasonic beam is formed from the composite wave of these ultrasonic waves.

[0024] The transmitted ultrasonic beam is reflected by an object such as a part of the subject, and propagates toward the transducer array 11 of the ultrasonic probe 1. Each transducer constituting the transducer array 11 expands and contracts upon receiving the ultrasonic echo propagating toward the transducer array 11 in this manner, generating received signals which are electrical signals, and outputs these received signals to the amplifier unit 53.

[0025] The amplifier 53 amplifies the signals input from each transducer constituting the transducer array 11 and transmits the amplified signals to the AD converter 55. The AD converter 55 converts the analog signals transmitted from the amplifier 53 into digital received data and outputs the received data to the beamformer 57.

[0026] The beam former 57 performs a reception focusing process in which each piece of reception data converted by the AD conversion unit 55 is delayed and added in accordance with the sound speed or sound speed distribution set based on the reception delay pattern selected by the device control unit 47. By this reception focusing process, each piece of reception data converted by the AD conversion unit 55 is phased and added, and a sound ray signal in which the focus of the ultrasonic echo is narrowed is generated.

[0027] The battery 15 is built into the ultrasonic probe 1 and supplies power to each circuit of the ultrasonic probe 1.

[0028] Next, the device main body 3 generates an ultrasound image including the examination point of the subject based on the sound ray signal generated by the ultrasound probe 1, and displays the ultrasound image including the examination point of the subject. The device main body 3 is, for example, a handheld terminal device such as a smartphone or a tablet PC (Personal Computer), and as shown in FIG. The imaging system includes an ultrasound image generating unit 31, an optical camera 33, a first image memory 35, an image selecting unit 37, a second image memory 39, a monitor 41, a display control unit 43, an input device 45, and a device control unit 47.

[0029] The ultrasonic image generating unit 31 is connected to the transmitting / receiving circuit 13 of the ultrasonic probe 1, and the ultrasonic image generating unit 31 is sequentially connected to a display control unit 43 and a monitor 41. The first image memory 35 is connected to each of the ultrasonic image generating unit 31 and the display control unit 43. The optical camera 33 is sequentially connected to the first image memory 35 and an image selection unit 37, and the image selection unit 37 is connected to a second image memory 39 and a display control unit 43, respectively. The transmitting / receiving circuit 13, the ultrasonic image generating unit 31, the optical camera 33, the display control unit 43, the first image memory 35, the image selection unit 37, and the second image memory 39 are connected to a device control unit 47, which is connected to an input device 45.

[0030] The ultrasonic probe 1 and the device main body 3 are connected wirelessly using wireless communication such as Wi-Fi (Wireless Fidelity), or are connected wired using a cable such as a USB (Universal Serial Bus) cable.

[0031] The ultrasonic image generating unit 31 generates an ultrasonic image (ultrasonic image signal) including the examination point of the subject from reception signals obtained by transmitting and receiving ultrasonic beams to and from the examination point of the subject using the ultrasonic probe 1 (more strictly, the transducer array 11), and more specifically, from sound ray signals generated from the reception signals by the transmission and reception circuit 13, under the control of the device control unit 47. As shown in Fig. 3, the ultrasonic image generating unit 31 has a configuration in which the signal processing unit 21, the DSC 23, and the image processing unit 25 are connected in series.

[0032] The signal processing unit 21 generates image information data corresponding to an ultrasound image based on the sound ray signals generated by the transmitting / receiving circuit 13. More specifically, the signal processing unit 21 performs signal processing on the sound ray signals generated by the beam former 57 of the transmitting / receiving circuit 13, for example, correcting for attenuation caused by the propagation distance in accordance with the depth of the position where the ultrasound is reflected, and then performs envelope detection processing to generate image information data representing tomographic image information regarding tissues within the subject.

[0033] A DSC (Digital Scan Converter) 23 raster-converts the image information data generated by the signal processing unit 21 into an image signal that conforms to the scanning method of a normal television signal.

[0034] The image processing unit 25 performs various image processing on the image signal input from the DSC 23, such as brightness correction, tone correction, sharpness correction, image size correction, refresh rate correction, scanning frequency correction, and color correction in accordance with the display format of the monitor 41, to generate an ultrasound image, and outputs the processed ultrasound image to the first image memory 35 and the display control unit 43.

[0035] Under the control of the device control unit 47, the optical camera 33 uses the ultrasonic probe 1 to capture an image of the subject during an examination of the examination area, thereby generating an optical image (camera image) including the subject in contact with the ultrasonic probe 1.

[0036] The first image memory 35 stores, under the control of the device control unit 47, a series of multiple frames of ultrasound images (moving images) generated by the ultrasound image generation unit 31 and a series of multiple frames of optical images (moving images) generated by the optical camera 33 for each examination. The first image memory 35 sequentially stores ultrasound images (still images) and optical images (still images) independently until the storage capacity of the first image memory 35 is full. It is not necessary to match the generation times (timestamps) of the ultrasound images and optical images when they are saved. In other words, it is not necessary to save ultrasound images and optical images of frames with the same generation time as a set. Thereafter, the ultrasound images and optical images of the newest frames are sequentially overwritten on the ultrasound images and optical images of the oldest frames, respectively, in the first image memory 35. This causes the first image memory 35 to store ultrasound images and optical images generated within a certain period of time from the ultrasound image and optical image of the newest frame.

[0037] Under the control of the device control unit 47, the image selection unit 37 selects one frame of an ultrasound image and one frame of an optical image from the ultrasound images and optical images stored in the first image memory . The method for selecting one frame of ultrasound image and one frame of optical image will be described in detail later.

[0038] Under the control of the device control unit 47, the second image memory 39 stores one frame of ultrasound image and one frame of optical image selected by the image selection unit 37 in association with each other.

[0039] The display control unit 43, under the control of the device control unit 47, causes the monitor (display unit) 41 to display various types of information. The monitor 41 displays, for example, ultrasound images, optical images, and also an operation screen for selecting one frame of ultrasound image and one frame of optical image. The monitor 41 is not particularly limited, but examples thereof include an LCD (Liquid Crystal Display) and an organic EL (Electro-Luminescence) display. The monitor 41 is disposed on one side of the device body 3. In contrast, the optical camera 33 is disposed on the other side of the device body 3, i.e., on the side opposite to the side on which the monitor 41 is disposed.

[0040] The input device 45 receives various instructions input by a user (examiner) of the ultrasound diagnostic device. The input device 45 is not particularly limited, but may include, for example, various buttons and a touch panel on which the user performs touch operations to input various instructions.

[0041] The device control unit 47 controls each part of the ultrasonic probe 1 and the device main body 3 based on a pre-stored program and user instructions input from the input device 45 .

[0042] The ultrasonic image generating unit 31 , the image selecting unit 37 , the display control unit 43 and the device control unit 47 are configured by a processor 49 .

[0043] Next, the operation of the ultrasonic diagnostic apparatus will be described with reference to the flowchart of FIG.

[0044] When examining a subject, the user first brings the ultrasonic probe 1 into contact with the area to be examined on the subject to begin transmitting and receiving ultrasonic waves, and then uses the optical camera 33 to capture an image of the subject, including the ultrasonic probe 1, that is examining the area to be examined.

[0045] In this case, under the control of the device control unit 47, the transmission / reception circuit 13 transmits and receives ultrasonic waves while the ultrasonic probe 1 is in contact with the examination location of the subject, and generates a sound ray signal (step S1).

[0046] That is, in accordance with the drive signal from the pulser 51, ultrasonic beams are transmitted from the plurality of transducers of the transducer array 11 to the examination location of the subject. The ultrasonic echo from the inspection point based on the ultrasonic beam transmitted from the pulser 51 is received by each transducer of the transducer array 11, and a received signal, which is an analog signal, is output from each transducer of the transducer array 11 that receives the ultrasonic echo. The received signal output from each transducer of the transducer array 11 is amplified by the amplifier 53 and AD converted by the AD converter 55 to obtain received data. The beamformer 57 performs reception focus processing on this reception data, thereby generating sound ray signals.

[0047] Next, under the control of the device control unit 47, the ultrasound image generation unit 31 generates an ultrasound image including the examination area of ​​the subject based on the sound ray signals generated by the beamformer 57 of the transmission / reception circuit 13 (step S2).

[0048] That is, the sound ray signals generated by the beam former 57 are subjected to various signal processing by the signal processing unit 21, and image information data representing tomographic image information relating to tissues within the subject is generated. The image information data generated by the signal processing unit 21 is raster converted by the DSC 23, and then subjected to various image processing by the image processing unit 25, whereby ultrasound images (moving images) are generated sequentially.

[0049] Meanwhile, under the control of the device control unit 47, the optical camera 33 images the subject using the ultrasonic probe 1 while the examination area is being examined, and optical images (moving images) including the subject in contact with the ultrasonic probe 1 are sequentially generated (step S3). That is, an ultrasonic image of the examination location of the subject and an optical image of the subject including the ultrasonic probe 1 examining the examination location are simultaneously generated.

[0050] The ultrasonic images generated by the ultrasonic image generating unit 31 and the optical images generated by the optical camera 33 are sequentially stored in the first image memory 35 under the control of the device control unit 47 .

[0051] Furthermore, under the control of the device control unit 47, the display control unit 43 sequentially displays the ultrasound images generated by the ultrasound image generation unit 31 and the optical images generated by the optical camera 33 on the monitor 41, as shown in FIG. 5 (step S4). Therefore, while viewing the ultrasound image and optical image displayed on the display screen of the monitor 41 arranged on one side of the device main body 3, the user can use the ultrasound probe 1 to generate an ultrasound image including the examination point of the subject, and at the same time, use the optical camera 33 arranged on the other side of the device main body 3 to generate an optical image including the subject with the ultrasound probe 1 in contact with the examination point.

[0052] Fig. 5 is a conceptual diagram of one embodiment showing the display screen of the monitor during an examination of a subject. An ultrasound image 61 is displayed in the upper region of the display screen of the monitor 41 shown in Fig. 5, and an optical image 63 is displayed in the region below the ultrasound image 61 from the center to the right edge. A first operation screen 65 is displayed in the region below the optical image 63. A freeze button 67 is located in the center of this first operation screen.

[0053] Here, while viewing the ultrasound image and optical image displayed sequentially on the monitor 41 of the device main body 3, the user presses the freeze button 67 at the timing when the user thinks that the desired ultrasound image including the examination area of ​​the subject has been generated (step S5).

[0054] When the freeze button 67 is pressed, under the control of the device control unit 47, the ultrasound images and optical images generated during a certain period of time from the time when the freeze was specified by the user are stored in the first image memory 35 (step S6). In other words, the ultrasound images and optical images generated during a certain period of time from the time when the freeze was specified by the user and already stored in the first image memory 35 are held in the first image memory 35 as they are.

[0055] Furthermore, when the freeze button 67 is pressed, the display control unit 43 displays a second operation screen 71 on the monitor 41 as shown in FIG. 6, instead of the first operation screen 65 shown in FIG.

[0056] Fig. 6 is a conceptual diagram of another embodiment showing the display screen of the monitor after the freeze button is pressed. Fig. 6 shows a state in which a second operation screen 71 is displayed instead of the first operation screen 65 on the display screen of the monitor 41 shown in Fig. 5. From the left end to the center of this second operation screen 71, a slide bar 73 for ultrasound images is arranged on the upper side, and a slide bar 75 for optical images (camera images) is arranged on the lower side. In addition, a save button 77 is arranged on the right side of the second operation screen 71.

[0057] Next, the user uses the slide bar 73 for ultrasound images and the slide bar 75 for optical images to select the desired frame of ultrasound image and the desired frame of optical image, respectively, from among the ultrasound images and optical images stored in the first image memory 35.

[0058] The left and right direction of the slide bar 73 for ultrasound images represents the time axis, with the right end position corresponding to the generation time of the ultrasound image of the latest frame among the ultrasound images stored in the first image memory 35, and the left end position corresponding to the generation time of the ultrasound image of the oldest frame.

[0059] By sequentially moving the triangle knob from the right end position of the ultrasound image slide bar 73 to the left, the user can scroll back from the latest frame of ultrasound image to the previous frame of ultrasound image and display them on the monitor 41. Furthermore, by stopping the movement of the knob, the user can specify, from the ultrasound images stored in the first image memory 35, one frame of ultrasound image that corresponds to the generation time of the position where the movement of the knob is stopped. The operation of the slide bar 75 for the optical image is similar.

[0060] That is, when the user moves the knob of the slide bar 73 for the ultrasound image left or right, in response to this instruction, the display control unit 43 scrolls back a past ultrasound image specified by the user by moving the knob from among the ultrasound images stored in the first image memory 35 and displays it on the monitor 41. Also, when the user stops moving the knob at a desired position, one frame of ultrasound image corresponding to the generation time of the position where the movement of the knob is stopped is specified (step S7).

[0061] Furthermore, when the user moves the knob of the slide bar 75 for optical images left and right, in response to this instruction, the display control unit 43 scrolls back from among the optical images stored in the first image memory 35 a past optical image designated by the user by moving the knob, and displays it on the monitor 41. Furthermore, when the user stops moving the knob at a desired position, one frame of optical image corresponding to the generation time of the position where the knob movement is stopped is designated (step S8).

[0062] After the user has designated one desired frame of an ultrasound image and one desired frame of an optical image, the user then presses the save button 77 .

[0063] In response to this, the image selection unit 37, under the control of the device control unit 47, selects one frame of an ultrasound image and one frame of an optical image from the ultrasound images and optical images stored in the first image memory 35 based on the user's specifications (step S9).

[0064] Subsequently, under the control of the device control unit 47, one frame of ultrasound image and one frame of optical image selected by the image selection unit 37 are associated with each other and stored in the second image memory 39 (step S10).

[0065] In the ultrasound diagnostic device of this embodiment, one frame of ultrasound image and one frame of optical image are stored in association with each other. This eliminates the need for the user to reselect a desired frame of ultrasound image and one frame of optical image from the ultrasound images and optical images stored in the first image memory 35 when reviewing the images from the examination, for example, during a later diagnosis. Furthermore, by referring to one frame of optical image, the user can easily understand the position and orientation of the ultrasound probe 1 at the time of generating the one frame of ultrasound image associated with this one frame of optical image.

[0066] The user can perform various processes on at least one of the one frame of ultrasound image and one frame of optical image specified by the user.

[0067] In this case, when the user presses, for example, the ultrasound image 61 on the display screen shown in FIG. 6, the mode enters an ultrasound image processing mode, and various processes can be performed on the ultrasound image 61. In the processing mode, the user can, for example, add annotations to the ultrasound image 61 and perform various image processing including image sharpening. The same is true for the optical image; by pressing the optical image 63, the user can enter the optical image processing mode and perform various processes on the optical image.

[0068] In this way, when the user processes at least one of a frame of ultrasound image and a frame of optical image, the processed frame of ultrasound image and the processed frame of optical image are associated and stored in the second image memory 39.

[0069] In addition, the image selection unit 37 can not only select one frame of ultrasound image and one frame of optical image specified by the user, but also automatically select at least one of one frame of ultrasound image and one frame of optical image.

[0070] For example, when the freeze button 67 is pressed, the image selector 37 may automatically select both one frame of an ultrasound image and one frame of an optical image.

[0071] The method for automatically selecting an ultrasound image is not particularly limited as long as it can automatically select an ultrasound image showing the examination location of the subject, or more specifically, an ultrasound image showing the examination location most clearly. For example, when the examination location is specified in advance by the user, the image selection unit 37 can automatically select one frame of ultrasound image including the examination location by identifying the examination location included in the ultrasound image for each ultrasound image stored in the first image memory 35 through image recognition processing.

[0072] The image selection unit 37 can use, for example, a machine learning determination model as the image recognition process for identifying the examination location included in the ultrasound image. The judgment model is a trained model that uses training ultrasound images containing the same examination location of an arbitrary subject as training data and learns the relationship between the training ultrasound images and the examination locations contained in these training ultrasound images using multiple training data. Based on the learning results, the determination model receives an ultrasound image to be determined as an input, and outputs a determination result (prediction result) of the examination location contained in this ultrasound image. The image selection unit 37 selects one frame of ultrasound image including the examination location from the ultrasound images based on the result of the determination by the determination model.

[0073] Furthermore, the method for automatically selecting an optical image is not particularly limited as long as it is possible to select an optical image that shows the entire subject so as to identify the position and orientation of the ultrasound probe 1 at the time of generating the ultrasound image. For example, the image selection unit 37 can automatically select one frame of an optical image that includes the examination point and the ultrasound probe 1 in contact with the subject by identifying the ultrasound probe 1 included in each optical image stored in the first image memory 35 through image recognition processing.

[0074] For example, FIGS. 7A and 7B are both conceptual diagrams of optical images of a subject undergoing abdominal examination. In the optical image shown in Fig. 7A, the subject's abdomen, including the ultrasound probe, appears larger than in the optical image shown in Fig. 7B, but because the subject is not entirely visible, it is difficult to tell that the position where the ultrasound probe is in contact with the subject's abdomen. In contrast, in the optical image shown in Fig. 7B, the subject is entirely visible, so it is clear that the position where the ultrasound probe is in contact with the subject's abdomen. In this case, the image selection unit 37 automatically selects the optical image shown in FIG. 7B, in which the entire subject is captured. The image selection unit 37 can similarly use a machine learning-based determination model as the image recognition process for identifying the ultrasound probe 1 included in the optical image.

[0075] By automatically selecting both one frame of an ultrasound image and one frame of an optical image, the user can save time and effort in manually selecting images. Note that if the user is not satisfied with the automatically selected image, they may manually select another image.

[0076] Furthermore, the image selector 37 may limit the range of one frame of ultrasound image and one frame of optical image to be selected from all ultrasound images and optical images stored in the first image memory 35. When ultrasound images and optical images are stored in the first image memory 35, their generation time (timestamp) is stored. The range of image selection can be limited using this generation time information.

[0077] In this case, the image selection unit 37 automatically selects one of the one frame of ultrasound image and one frame of optical image, and automatically selects the other of the one frame of ultrasound image and one frame of optical image from the ultrasound images and optical images stored in the first image memory 35 that were generated within a certain period before and after the generation time of one of the images. It should be noted that it is not necessary for the generation time of one image to be exactly the same as the generation time of the other image; in other words, it is not necessary to select an image with the same timestamp as one image; it is sufficient to select the other image from within a certain period before or after the generation time of the one image.

[0078] By limiting the range of image selection, images can be selected more efficiently and in a shorter time than selecting one frame of ultrasound image and one frame of optical image from all ultrasound images and optical images stored in the first image memory 35.

[0079] It is desirable that the certain period before and after the generation time be, for example, at least one second and at most two seconds. If this certain period is shorter than one second, there is a high possibility that the image contains the effects of camera shake when the user presses the freeze button 67. On the other hand, if the certain period is longer than two seconds, there is a high possibility that the position of the ultrasound probe 1 has been moved and an image of a position other than the examination area is included.

[0080] Alternatively, when the user specifies one of a frame of ultrasound image and a frame of optical image from the ultrasound images and optical images stored in the first image memory 35, the image selection unit 37 may select this one image based on the user's specification, and automatically select the other image of the frame of ultrasound image and the frame of optical image.

[0081] In this case, the user designates a desired frame of ultrasound image by, for example, moving the knob of the ultrasound image slide bar 73 left and right and then stopping the movement of the knob at a desired position. In response to this, the image selection unit 37 selects the frame of ultrasound image manually designated by the user from the ultrasound images stored in the first image memory 35. When a user specifies one frame of an ultrasound image, the image selection unit 37 automatically selects one frame of an optical image from the optical images stored in the first image memory 35. When automatically selecting one frame of an optical image, the knob of the optical image slide bar 75 shown in Fig. 6 may be automatically moved to a position corresponding to the generation time of the one frame of the optical image to be automatically selected.

[0082] By automatically selecting one of the images, the user can save time and effort in manually selecting the other image. If the user is not satisfied with the automatically selected other image, the user may manually select another image.

[0083] The one image manually designated by the user is not limited to an ultrasound image, and the user may designate an optical image as the one image. However, since the ultrasound image is an image used for diagnosis and the optical image is an image used as a body mark indicating the position and orientation of the ultrasound probe 1 when the ultrasound image was generated, it is desirable that the one image manually designated by the user be an ultrasound image.

[0084] Similarly, the image selection unit 37 may automatically select one of the ultrasound images and optical images stored in the first image memory from among the ultrasound images and optical images generated within a certain period before or after the generation time of the other image.

[0085] In either case, when one frame of an ultrasound image or one frame of an optical image is selected, the one image may be stored in the second image memory 39 first, and then when the other image is selected, the other image may be stored in the second image memory 39. Alternatively, after both one frame of an ultrasound image and one frame of an optical image are selected, both images may be stored in the second image memory 39.

[0086] As shown in FIG. 1, the device main body 3 may be provided with the ultrasound image generating unit 31, but this is not limited thereto. The entire ultrasound image generating unit 31 or only the signal processing unit 21 may be provided on the ultrasound probe 1 side.

[0087] In the device of the present invention, the processing units (Processing Units) that perform various processes, such as the transmitting and receiving circuit 13, the ultrasonic image generating unit 31, the image selecting unit 37, the display control unit 43, and the device control unit 47, are provided. The hardware configuration of the first image memory 35 and the second image memory 39 may be dedicated hardware or may be various processors or computers that execute programs. Also, the first image memory 35 and the second image memory 39 may be flash memory, an SD card (Secure Digital card Secure Digital Card), USB memory (Universal Serial Bus memory) Alternatively, a recording medium such as a serial bus memory may be used, or an HDD (Hard Disk Drive), an SSD (Solid State Drive), or an external server may also be used.

[0088] The various processors include CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), and so on, which are general-purpose processors that execute software (programs) and function as various processing units. This includes programmable logic devices (PLDs), which are processors whose circuit configuration can be changed after manufacturing, such as programmable gate arrays (Programmable Gate Arrays), and dedicated electrical circuits, such as application specific integrated circuits (ASICs), which are processors with circuit configurations designed specifically for performing specific processes.

[0089] A single processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types, for example, a combination of multiple FPGAs, or a combination of an FPGA and a CPU, etc. Also, multiple processing units may be configured with one of the various processors, or two or more of the multiple processing units may be combined into one processor.

[0090] For example, as typified by server and client computers, one processor is configured by combining one or more CPUs and software, and this processor functions as multiple processing units. Another form is the use of a processor that realizes the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by system-on-chip (SoC).

[0091] Furthermore, the hardware configuration of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0092] The method of the present invention can be implemented by, for example, a program that causes a computer to execute each step. Also, a computer-readable recording medium on which this program is recorded can be provided.

[0093] Although the present invention has been described in detail above, the present invention is not limited to the above-described embodiments, and various improvements and modifications may be made without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0094] 1 Ultrasound probe, 3 Device body, 11 Transducer array, 13 Transmitting / receiving circuit, 15 Battery, 21 Signal processing unit, 23 DSC, 25 Image processing unit, 31 Ultrasound image generation unit, 33 Optical camera, 35 First image memory, 37 Image selection unit, 39 Second image memory, 41 Monitor, 43 Display control unit, 45 Input device, 47 Device control unit, 49 Processor, 51 Pulser, 53 Amplification unit, 55 AD conversion unit, 57 Beamformer, 61 Ultrasound image, 63 Optical image, 65 First operation screen, 67 Freeze button, 71 Second operation screen, 73 Slide bar for ultrasound image, 75 Slide bar for optical image, 77 Save button

Claims

1. An ultrasonic probe and a device main body connected to the ultrasonic probe, The device body includes: an ultrasonic image generating unit that generates an ultrasonic image including an examination point from a received signal obtained by transmitting and receiving an ultrasonic beam to and from the examination point of a subject using the ultrasonic probe; an optical camera for generating an optical image including the subject in contact with the ultrasound probe; The monitor and a display control unit that displays the ultrasonic image and the optical image on the monitor; a first image memory that stores a plurality of frames of the ultrasonic images and a plurality of frames of the optical images generated during a certain period of time from a timing designated by a user; an image selection unit that selects one frame of an ultrasound image and one frame of an optical image from the ultrasound images and the optical images stored in the first image memory; a second image memory that stores the one frame of ultrasound image and the one frame of optical image in association with each other.

2. 2. The ultrasound diagnostic apparatus according to claim 1, wherein the display control unit, in response to an instruction from a user, scrolls back a past ultrasound image specified by the user from among the ultrasound images stored in the first image memory and displays it on the monitor, and scrolls back a past optical image specified by the user from among the optical images stored in the first image memory and displays it on the monitor.

3. The ultrasound diagnostic apparatus according to claim 1 , wherein the image selection unit selects each of the one frame of ultrasound image and the one frame of optical image based on a designation by the user.

4. The ultrasound diagnostic apparatus according to claim 1 , wherein the image selection unit automatically selects at least one of the one frame of ultrasound image and the one frame of optical image.

5. The ultrasound diagnostic apparatus according to claim 4 , wherein the image selection unit automatically selects both the one frame of ultrasound image and the one frame of optical image.

6. 6. The ultrasound diagnostic device according to claim 5, wherein the image selection unit automatically selects one of the one frame of ultrasound image and the one frame of optical image, and automatically selects the other of the one frame of ultrasound image and the one frame of optical image from among the ultrasound images and the optical images stored in the first image memory that were generated within a certain period before and after the generation of the one of the images.

7. 5. The ultrasound diagnostic device according to claim 4, wherein the image selection unit selects one of the one frame of ultrasound image and the one frame of optical image based on a designation by the user, and automatically selects the other of the one frame of ultrasound image and the one frame of optical image.

8. 8. The ultrasound diagnostic device according to claim 7, wherein the image selection unit automatically selects the other image from among the ultrasound images and the optical images stored in the first image memory that were generated within a certain period before or after the generation of the other image.

9. 9. The ultrasound diagnostic device according to claim 1, wherein, when the user processes at least one of the one frame of ultrasound image and the one frame of optical image, the second image memory stores the one frame of ultrasound image and the one frame of optical image in association with each other after the processing.

10. A method for controlling an ultrasonic diagnostic apparatus including an ultrasonic probe and an apparatus main body connected to the ultrasonic probe, comprising: an ultrasound image generating unit included in the device body generating an ultrasound image including an examination point from a received signal obtained by transmitting and receiving an ultrasound beam to and from the examination point of a subject using the ultrasound probe; a step in which an optical camera included in the device body generates an optical image including the subject in a state in which the ultrasound probe is in contact with the subject; a step in which a display control unit included in the device main body displays the ultrasound image and the optical image on a monitor included in the device main body; a step of storing, in a first image memory of the device body, a plurality of frames of the ultrasound images and a plurality of frames of the optical images generated during a certain period of time from a timing designated by a user; an image selection unit included in the device body selecting one frame of an ultrasound image and one frame of an optical image from the ultrasound images and the optical images stored in the first image memory; a step of storing the one frame of ultrasound image and the one frame of optical image in association with each other in a second image memory included in the device body.

Images