PANORAMIC ULTRASOUND IMAGE GENERATION DEVICE AND PANORAMIC ULTRASOUND IMAGE GENERATION PROGRAM

Abstract

The image quality of a panoramic ultrasound image is improved. Alternatively, the computational effort required to generate the panoramic ultrasound image is reduced. A probe information acquisition unit acquires multiple positional information points that specify the position of an ultrasound probe in a case where a received signal corresponding to each frame used to generate a panoramic ultrasound image is acquired. A combination processing unit arranges a second frame by shifting the second frame relative to a first frame in a scanning direction, based on a difference between a first position (the position of the ultrasound probe in a case where the received signal corresponding to the first frame forming the panoramic ultrasound image is acquired) and a second position (the position of the ultrasound probe in a case where the received signal corresponding to the second frame forming the panoramic ultrasound image is acquired), and then combines the first and second frames.

Description

BACKGROUND OF THE INVENTION 1. Field of the invention

[0001] The present description discloses improvements to a panoramic ultrasound image generation device and a panoramic ultrasound image generation program. 2. Description of the state of the art

[0002] An ultrasound diagnostic device is known that transmits ultrasound waves from an ultrasound probe to a subject, receives a reflected wave from the subject in the ultrasound probe, and generates an ultrasound tomography image representing a cross-section of the subject based on a received signal formed from the reflected wave. Since an ultrasound beam is scanned by the ultrasound probe onto a scanning surface, the ultrasound tomography image is an image representing a cross-section of the subject on the scanning surface.

[0003] In the prior art, a panoramic ultrasound image is generated by arranging and combining several ultrasound tomography images acquired while an ultrasound probe is moved in a scanning direction (in particular, a scanning direction of an ultrasound beam). In this description, each ultrasound tomography image that forms the panoramic ultrasound image is referred to as a "single image". The panoramic ultrasound image is an ultrasound tomography image that, compared to a single image, extends in the scanning direction of the ultrasound beam.

[0004] For example, JP4299015B discloses an ultrasound image processing device that generates a panoramic ultrasound image by combining a single image n-1 and a single image n, which is a single image taken at a time point after the single image n-1. In particular, in the ultrasound image processing device disclosed in JP4299015B, a rectangular reference area is set in the single image n-1, a rectangular area corresponding to the reference area of ​​the single image n-1 is specified from the single image n, the single image n-1 is rotated so that the reference area and the rectangular area coincide, and then the single image n and the single image n-1 are combined to generate a panoramic ultrasound image. SUMMARY OF THE INVENTION

[0005] As a method of generating a panoramic ultrasound image, in a method of performing registration of two individual images (movement or rotation of at least one individual image) based on a similarity between individual images (for example, a similarity between a reference area of ​​individual image n-1 and a rectangular area of ​​individual image n, as disclosed in JP4299015B) and then performing the combination, the accuracy of the registration of the individual images may be low, and consequently the image quality of the panoramic ultrasound image may be low.

[0006] Furthermore, the method of registering two individual images based on the similarity between the images and then combining them has a problem in that the computational effort for registering the two individual images is increased, i.e., the computational effort for generating the panoramic ultrasound image is increased.

[0007] One purpose of the panoramic ultrasound image generation device disclosed in the present description is to improve the image quality of the panoramic ultrasound image. Alternatively, the purpose of the panoramic ultrasound image generation device disclosed in the present description is to reduce the computational effort required to generate the panoramic ultrasound image.

[0008] A panoramic ultrasound image generation device disclosed in the present description is a panoramic ultrasound image generation device that generates a panoramic ultrasound image by combining several individual images, which include a first individual image obtained by scanning with an ultrasound beam from an ultrasound probe located at a first position towards a subject, and a second individual image obtained by scanning with the ultrasound beam from the ultrasound probe located at a second position, which differs from the first position, towards a subject, comprising: a probe information acquisition unit that acquires position information indicating a position of the ultrasound probe in a case where a received signal corresponding to each individual image for generating the panoramic ultrasound image is received;and a combination processing unit that generates the panoramic ultrasound image by combining the multiple individual images, wherein the combination processing unit is configured to combine the first individual image and the second individual image by shifting the first individual image and the second individual image based on a difference between the first position and the second position located in a scanning direction of the ultrasound beam from the first position.

[0009] The probe information acquisition unit can further acquire orientation information specifying the orientation of the ultrasound probe in a case where the received signal corresponding to each frame for generating the panoramic ultrasound image is acquired, and the combination processing unit can rotate the second frame relative to the first frame in accordance with a rotation amount corresponding to a difference between a first orientation, which is an orientation of the ultrasound probe in a case where the ultrasound probe is in the first position, and a second orientation, which is an orientation of the ultrasound probe in a case where the ultrasound probe is in the second position, and can combine the first frame and the second frame.

[0010] The combination processing unit can, based on the first position and the second position, set a reference area in a superimposed section, which is a section of the first frame that corresponds to a section in which a scanning area corresponding to the first frame and a scanning area corresponding to the second frame overlap, and can specify a correspondence area, which is an area in the second frame that corresponds to the reference area of ​​the first frame, and can combine the first frame and the second frame by shifting the first frame and the second frame based on a difference between a position of the reference area and a position of the correspondence area.

[0011] The panoramic ultrasound image generation device may further include an intermediate image generation unit that generates an intermediate image for interpolation between the first image and the second image in the panoramic ultrasound image based on the first image and the second image in a case where the second position is separated from the first position by a distance equal to or greater than a first threshold distance, wherein the combination processing unit can generate the panoramic ultrasound image by combining the first image, the second image and the intermediate image arranged between the first image and the second image.

[0012] The panoramic ultrasound image generation device may further include a notification processing unit that notifies an operator in a case where an angle formed between a scanning direction of the ultrasound beam from the ultrasound probe in a case where the ultrasound probe is at the first position and a direction from the first position to the second position is equal to or greater than a first threshold angle.

[0013] The panoramic ultrasound image generation device may further include a notification processing unit that notifies an operator in a case where an angle formed between a scanning direction of the ultrasound beam in the first orientation and a scanning direction of the ultrasound beam in the second orientation is equal to or greater than a second threshold angle.

[0014] The panoramic ultrasound image generation device may further include a notification processing unit that notifies an operator in a case where the second position is separated from the first position by a distance equal to or greater than a second threshold distance.

[0015] The panoramic ultrasound image generation device may further include a camera that images the subject and the ultrasound probe, and a display control that displays the panoramic ultrasound image together with several captured images obtained by imaging the ultrasound probe and the subject, wherein the captured images are arranged in an imaging sequence, in a case in which received signals corresponding to each of the several individual images are captured, on a display unit.

[0016] Furthermore, a panoramic ultrasound image generation program disclosed in the present description causes a computer, which generates a panoramic ultrasound image by combining several individual images, namely a first image obtained by scanning with an ultrasound beam from an ultrasound probe located at a first position towards a subject, and a second image obtained by scanning with the ultrasound beam from the ultrasound probe located at a second position, which differs from the first position, towards a subject, to function as: a probe information acquisition unit that acquires position information indicating the position of the ultrasound probe in a case where a received signal corresponding to each individual image for forming the panoramic ultrasound image is received;and a combination processing unit that generates the panoramic ultrasound image by combining the multiple individual images, wherein the combination processing unit is configured to combine the first individual image and the second individual image by shifting the first individual image and the second individual image based on a difference between the first position and the second position located in a scanning direction of the ultrasound beam from the first position.

[0017] According to the panoramic ultrasound image generation device disclosed in this description, it is possible to improve the image quality of the panoramic ultrasound image. Alternatively, it is possible to reduce the computational effort required to generate the panoramic ultrasound image with the panoramic ultrasound image disclosed in this description. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a schematic diagram of a configuration of a panoramic ultrasound imaging system according to the present embodiment. Fig. Figure 2 is a diagram showing an example of an image captured by a camera. Fig. Figure 3 is a diagram showing multiple scanning areas corresponding to multiple individual images. Fig. Figure 4 is a schematic diagram of a configuration of an ultrasound diagnostic device according to the present embodiment. Fig. Figure 5 is a representation that shows an angle as alignment information. Fig. Figure 6 is a first diagram showing ultrasound probes located at a first position and a second position. Fig. Figure 7 is a first diagram showing a first single image and a second single image that are to be combined. Fig. Figure 8 is a second diagram showing the ultrasound probes located at the first and second positions. Fig. Figure 9 is a second diagram showing the first and second images to be combined. Fig. Figure 10 is a diagram showing an example of a panoramic ultrasound image. Fig. Figure 11 is a diagram showing a reference range. Fig. Figure 12 is a diagram showing a superimposed section. Fig. Figure 13 is a third diagram showing the ultrasound probes located at the first and second positions. Fig. Figure 14 is a diagram showing an intermediate frame. Fig. Figure 15 is a fourth diagram showing the ultrasound probes located at the first and second positions. Fig. Figure 16 is a fifth diagram showing the ultrasound probes located at the first and second positions. Fig. Figure 17 is a diagram showing a display example of the panoramic ultrasound image and the recorded image. Fig. Figure 18 is a flowchart showing a flow of processing of the ultrasonic diagnostic device according to the present embodiment. DESCRIPTION OF PREFERRED EXECUTION FORMS

[0018] Fig. Figure 1 is a schematic diagram of a configuration of a panoramic ultrasound imaging system 10 according to the present embodiment. The panoramic ultrasound imaging system 10 is configured to include a camera 12 and an ultrasound diagnostic device 16, which is a panoramic ultrasound imaging device containing an ultrasound probe 14. The camera 12 and the ultrasound diagnostic device 16 are interconnected in a communicative manner.

[0019] A probe detection marker 20 is attached to the ultrasound probe 14. The probe detection marker 20 is a marker for detecting the position and orientation of the ultrasound probe 14. An example of the probe detection marker 20 is a marker for augmented reality (AR).

[0020] The camera 12 is configured to include, in addition to a lens and an image sensor, a processor (configured as a CPU and the like), a communication interface (configured as a network adapter and the like), and the like. The camera 12 images a patient E and the ultrasound probe 14 (in particular, the probe detection marker 20). The camera 12 can be an extracorporeal camera that images the body surface of the patient E, or it can be an intracorporeal camera, such as an endoscope that is inserted into the patient. A captured image is formed by the image sensor of the camera 12, and the captured image is transmitted from a communication interface of the camera 12 to the ultrasound diagnostic device 16.

[0021] Fig. Figure 2 is a diagram showing an example of a captured image 22 from camera 12. As described above, the captured image 22 contains the image of the probe detection marker 20. The ultrasound diagnostic device 16 can detect at least one of the position and orientation of the ultrasound probe 14 by analyzing the image of the probe detection marker 20 shown in the captured image 22. Details of the detection processing of the position and orientation of the ultrasound probe 14 are described below.

[0022] The ultrasound diagnostic device 16, as the panoramic ultrasound image generation device, generates a panoramic ultrasound image. Fig. Figure 3 is a diagram showing multiple scanning surfaces SP, corresponding to several individual images used to generate the panoramic ultrasound image. In the drawings of this description, including... Fig. 3, is a horizontal direction in which a scanning direction of the ultrasound beam emitted by the ultrasound probe 14 runs, (in particular a scanning direction in a case in which a first single image is captured among the several single images to generate the panoramic ultrasound image) an X-axis, a depth direction is a Y-axis and a horizontal direction perpendicular to the X-axis and the Y-axis is a Z-axis.

[0023] An operator of the ultrasound diagnostic device 16, such as a physician, generates several individual images corresponding to each scanning area SP located in a scanning direction of the ultrasound beam while moving the ultrasound probe 14 in the scanning direction (X-axis direction) of the ultrasound beam and bringing the ultrasound probe 14 into contact with the patient E. Specifically, the operator positions the ultrasound probe 14 at a first position 14a and scans the scanning area SPa with the ultrasound beam to generate the first individual image corresponding to the scanning area SPa. Next, the ultrasound probe 14 is positioned at a second position 14b, spaced apart from the first position 14a in the scanning direction, to scan the scanning area SPb with the ultrasound beam, thereby generating the second individual image corresponding to the scanning area SPb.Similarly, several individual images are generated, corresponding to each scanning area SP.

[0024] The ultrasound diagnostic device 16 generates a panoramic ultrasound image by combining the multiple individual images, including the first and second images. Details of the processing involved in generating the panoramic ultrasound image are described below.

[0025] Fig. Figure 4 is a schematic diagram of a configuration of the ultrasound diagnostic device 16. The ultrasound diagnostic device 16 is a medical device that is installed in medical facilities, such as a hospital.

[0026] The ultrasound probe 14 is a device that transmits ultrasound waves to and receives them from the subject E. The ultrasound probe 14 comprises a vibrating element assembly consisting of several vibrating elements that transmit ultrasound waves to and receive them from the subject E. The vibrating element assembly is formed from several vibrating elements arranged in one direction (arrangement direction). A transmission signal is supplied to each vibrating element by a transmission and reception unit 30 described below, so that each vibrating element generates ultrasound waves. In particular, the ultrasound probe 14 scans the ultrasound beam on a plane (scanning surface SP) that is parallel to the arrangement direction.

[0027] As described above, the probe detection marker 20 is attached to the ultrasound probe 14.

[0028] The transmission and reception unit 30 transmits the transmission signal to the ultrasound probe 14 (specifically, each vibrating element of the vibrating element assembly) under the control of a controller 46 (described below). As a result, the ultrasound beam is scanned by the ultrasound probe 14 onto the scanning surface SP. Furthermore, the transmission and reception unit 30 receives a received signal from each vibrating element that has received a reflected wave from the subject E. The transmission and reception unit 30 includes an adder and several delays corresponding to each vibrating element and performs phase alignment and addition processing of the received signals from each vibrating element using the adder and the multiple delays.As a result, a received beam signal is generated in which information indicating a signal intensity of the wave reflected by the subject E is arranged in a depth direction of the subject.

[0029] A signal processing unit 32 performs various types of signal processing, including filter processing by applying a bandpass filter, detection processing and the like, on the received beam signal from the transmit and receive unit 30.

[0030] An image generation unit 34 generates an ultrasound tomography image (B-mode image) representing a cross-section (specifically, a scanning area of ​​the ultrasound beam) of the subject E, based on the received beam signal, which has undergone signal processing in the signal processing unit 32. The ultrasound tomography image generated by the image generation unit 34 is also used as a single image to generate the panoramic ultrasound image.

[0031] A display control 36 performs control to display various images, including the ultrasound tomography image produced by the image generation unit 34 or a panoramic ultrasound image described below, on a display 38.

[0032] The display 38 as a display unit is, for example, a display device configured using a liquid crystal display, organic electroluminescence (EL) or the like.

[0033] The transmission and reception unit 30, the signal processing unit 32, the image generation unit 34, and the display control 36, which are contained in the ultrasound diagnostic device 16, are configured by a processor. The processor is configured to include at least one general-purpose processing device (for example, a CPU or the like) and one dedicated processing device (for example, a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a programmable logic device).The processor need not be configured using a single processing device, but can instead be configured through the cooperation of multiple processing devices located in physically separate positions. Furthermore, each of the units described above can be implemented through the cooperation of hardware, such as a processor, and software.

[0034] A communication interface 40 is configured, for example, using a network adapter or the like. The communication interface 40 has the function of communicating with another device (in particular, the camera 12). In particular, the communication interface 40 receives the captured image from the camera 12.

[0035] An input interface 42 is configured using, for example, a button, a trackball, a touch panel, and the like. The input interface 42 is used to input an instruction from an operator using the ultrasound diagnostic device 16 to the ultrasound diagnostic device 16.

[0036] Memory 44 is configured to contain a hard disk drive (HDD), a solid-state drive (SSD), an embedded MultiMediaCard (eMMC), read-only memory (ROM), random-access memory (RAM), or the like. A panoramic ultrasound image generation program for operating each unit of the ultrasound diagnostic device 16 is stored in memory 44. The panoramic ultrasound image generation program can also be stored, for example, on a computer-readable non-volatile storage medium, such as a Universal Serial Bus (USB) drive or a CD-ROM. The ultrasound diagnostic device 16 can read and execute the panoramic ultrasound image generation program from such a storage medium.The ultrasound diagnostic device 16 reads the panoramic ultrasound image generation program to have the functions described below, and thus it can be said that the ultrasound diagnostic device 16 is a computer program product.

[0037] The controller 46 is configured to include at least one general-purpose processor (such as a CPU) and one dedicated processor (such as a GPU, ASIC, FPGA, or programmable logic circuit). The controller 46 can be configured to have multiple processing devices located at physically separate positions and working together, instead of being configured using a single processing device. The controller 46 controls each unit of the ultrasonic diagnostic device 16. Furthermore, as described in Fig. 4 shown, the control 46 functions as a probe information acquisition unit 48, a combination processing unit 50, an intermediate image generation unit 52 and a notification processing unit 54 in accordance with the panoramic ultrasound image generation program stored in the memory 44.

[0038] The probe information acquisition unit 48 acquires position information indicating the position of the ultrasound probe 14. Specifically, the probe information acquisition unit 48 acquires multiple position information pieces indicating the position of the ultrasound probe 14 when the received signal corresponding to each individual frame used to generate the panoramic ultrasound image is acquired. Furthermore, the probe information acquisition unit 48 acquires orientation information indicating the orientation of the ultrasound probe 14. Specifically, the probe information acquisition unit 48 acquires multiple orientation information pieces indicating the orientation of the ultrasound probe 14 when the received signal corresponding to each individual frame used to generate the panoramic ultrasound image is acquired.

[0039] In the present embodiment, the probe information acquisition unit 48 acquires the position and orientation information by analyzing the captured image 22 taken by the camera 12 in order to detect the position and orientation of the ultrasound probe 14. As described above, the captured image 22 contains the image of the probe detection marker 20, which indicates the position and orientation of the ultrasound probe 14 (see Fig. 2) The probe information acquisition unit 48 acquires the position and orientation information by analyzing the image of the probe detection marker 20 in the captured image 22. The position information can be, for example, three-dimensional coordinates in a camera coordinate system. The orientation information can be a rotation angle with respect to a predetermined axis (for example, the X-axis, the Y-axis, or the Z-axis) in the camera coordinate system. Since a known method can be used to detect the position and orientation of the ultrasound probe 14 in the camera coordinate system from the image of the probe detection marker 20 contained in the captured image 22, a detailed description of this method is omitted here.

[0040] The probe information acquisition unit 48 performs calibration before detecting the position and orientation of the ultrasound probe 14 based on the captured image 22. Specifically, the operator sets the position and orientation of the ultrasound probe 14 to a predetermined position and orientation and, in this state, enters the calibration instruction to the ultrasound diagnostic device 16. When the calibration instruction is entered, the probe information acquisition unit 48 detects the position and orientation of the ultrasound probe 14 based on the probe detection marker 20 contained in the captured image 22 and stores the position and orientation in memory 44 as a reference position and orientation.Subsequently, the probe information acquisition unit 48 detects the position of the ultrasound probe 14 as a relative position with respect to a known reference position and detects the orientation of the ultrasound probe 14 as a relative orientation with respect to a known reference orientation.

[0041] In the present embodiment, it is assumed that the operator enters the calibration instruction at the position and orientation of the ultrasound probe 14 when the first single image among the multiple images used to generate the panoramic ultrasound image is being acquired, and that the position and orientation of the ultrasound probe 14 at this time are the reference position and orientation. Of course, the position and orientation of the ultrasound probe 14 are not limited to this when performing the calibration.

[0042] Fig. Figure 5 is a diagram showing an angle θ as the orientation information. For example, in a case where calibration is performed in an orientation where the ultrasound probe 14 is brought into perpendicular contact with the surface of the subject E (in other words, in a state where the scanning direction of the ultrasound beam is parallel to the surface of the subject E), the inclination of the ultrasound probe 14 in an XY plane can be represented by the angle θ. As described above, the orientation information can be represented by a rotation angle in each of the XY plane (about the Z-axis), a YZ plane (about the X-axis), and an XZ plane (about the Y-axis) with respect to the orientation of the ultrasound probe 14 in a case of calibration.

[0043] The probe information acquisition unit 48 can detect at least one of the position and orientation information of the ultrasound probe 14 by a method other than analyzing the recorded image 22. For example, a position and orientation sensor, such as a magnetic sensor or an accelerometer, can be provided in the ultrasound probe 14, and at least one of the position and orientation information can be detected based on a detection value of the position and orientation sensor.

[0044] The scanning area SP (see Fig. 3) is determined by the position and orientation of the ultrasound probe 14. Therefore, it can be said that the position and orientation information acquired by the probe information acquisition unit 48 specifies the position and orientation of the scanning area SP, which corresponds to each individual image for generating the panoramic ultrasound image.

[0045] The probe information acquisition unit 48 stores in the memory 44 the position information, which specifies the position of the ultrasound probe 14 in a case in which the transmission and receiving unit 30 acquires the received signal corresponding to each individual image, and the orientation information, which specifies the orientation of the ultrasound probe 14 in a case in which the transmission and receiving unit 30 acquires the received signal corresponding to each individual image, in conjunction with the received signal or the individual image formed from the received signal.

[0046] The Combination Processing Unit 50 combines multiple individual images to create a panoramic ultrasound image. The Combination Processing Unit 50 can create a panoramic ultrasound image by combining three or more individual images, but the following describes the processing of combining two individual images (a first individual image and a second individual image). After a third individual image (that is, the combination of the second individual image and the third individual image, the combination of the third individual image and a fourth individual image, etc.), the combination processing of the first individual image and the second individual image can be performed in the same way.

[0047] In the present description, a position of the ultrasound probe 14 in a case where the received signal corresponding to the first frame is acquired is referred to as a first position, an orientation of the ultrasound probe 14 in a case where the received signal corresponding to the first frame is acquired (in other words, in a case where the ultrasound probe 14 is in the first position) is referred to as a first orientation, a position of the ultrasound probe 14 in a case where the received signal corresponding to the second frame is acquired and which differs from the first position is referred to as a second position, and an orientation of the ultrasound probe 14 in a case where the received signal corresponding to the second frame is acquired (in other words, in a case where the ultrasound probe 14 is in the second position)is referred to as a second orientation.

[0048] Fig. Figure 6 is a first diagram showing the ultrasound probe 14, located at position 14a in the first position and position 14b in the second. In the example of Fig. 6. The surface of the subject E, on which the ultrasound probe 14 is moved, is flat, and there is almost no change in the orientation of the ultrasound probe 14 when it is moved to acquire multiple images. In this case, the combination processing unit 50 can combine the multiple images based on the position information of the ultrasound probe 14.

[0049] In particular, the combination processing unit 50 combines the first and second images by shifting (translating) the first and second images based on a difference between the first position 14a, specified by the position information acquired by the probe information acquisition unit 48, and the second position 14b, also specified by the position information acquired by the probe information acquisition unit 48 (the second position 14b lies in the scanning direction of the ultrasound beam from the first position 14a). The difference (a distance) between the first position 14a and the second position 14b, specified by the position information, is a distance in the camera coordinate system or in real space.On the other hand, as described above, the position of the scanning area SP is determined by the position of the ultrasound probe 14, and its size (in particular, its width in the scanning direction) is also known. Furthermore, the number of pixels of the single image corresponding to the scanning area SP (in particular, the number of pixels in the scanning direction) is predetermined. Therefore, the combined processing unit 50 can calculate the distance in the camera coordinate system or the real space per unit pixel of the single image in the scanning direction.

[0050] Fig. Figure 7 is a diagram showing the positions of a first frame 60a and a second frame 60b to be combined. The combination processing unit 50 calculates a positional relationship in the scanning direction (generally a transverse direction of the image) between the first frame 60a and the second frame 60b, which corresponds to the positional relationship in the scanning direction between the scanning area SPa and the scanning area SPb, based on the distance in the camera coordinate system or real space per unit pixel of the frame in the scanning direction and the difference between the first position 14a and the second position 14b (in other words, the positional difference between the scanning area SPa and the scanning area SPb).In particular, the combination processing unit 50 calculates the difference (ΔPx) in position in the scanning direction between the first frame 60a and the second frame 60b with respect to the number of pixels. The combination processing unit 50 then aligns the second frame 60b by shifting it in the scanning direction by the calculated difference ΔPx relative to the first frame 60a and then combines the first frame 60a and the second frame 60b.

[0051] Fig. Figure 8 is a second diagram showing the ultrasound probes 14, located at the first position 14a and the second position 14b. In the example of Fig. 8 The surface of the subject E, on which the ultrasound probe 14 is moved, exhibits irregularities, and the orientation of the ultrasound probe 14 changes when the probe is moved to acquire multiple images. In this case, the combination processing unit 50 combines the multiple images based on the position and orientation information of the ultrasound probe 14.

[0052] In particular, the combination processing unit 50 rotates the second frame 60b relative to the first frame 60a in accordance with a rotation amount corresponding to a difference between the first orientation specified by the orientation information acquired by the probe information acquisition unit 48 and the second orientation specified by the orientation information acquired by the probe information acquisition unit 48, and combines the first frame 60a and the second frame 60b by shifting the first frame 60a and the second frame 60b on the basis of the difference between the first position 14a and the second position 14b.As described above, since the orientation of the scanning surface SP is determined by the orientation of the ultrasound probe 14, the combination processing unit 50 can obtain the difference in orientation between the scanning surface SPa and the scanning surface SPb based on the difference between the first orientation and the second orientation (in other words, the difference in position between the scanning surface SPa and the scanning surface SPb).

[0053] Fig. Figure 9 is a diagram showing the positions and orientations of the first image 60a and the second image 60b, which are to be combined. As in Fig. 8 and Fig. As shown in Figure 9, the combination processing unit 50 rotates the second image 60b relative to the first image 60a such that the orientation of the scanning area SPb is the same as the orientation of the scanning area SPa. Furthermore, the combination processing unit 50 aligns the second image 60b by shifting it in the scanning direction by the calculated difference ΔPx relative to the first image 60a and then combines the first image 60a and the second image 60b.

[0054] Fig. Figure 10 shows an example of a panoramic ultrasound image 62 generated by the combination processing unit 50. As described above, in the present embodiment, the combination processing unit 50 determines the position of the second image 60b relative to the first image 60a, or the position and orientation based on the first position 14a and the second position 14b of the ultrasound probe 14 acquired by the probe information acquisition unit 48, and combines the first image 60a and the second image 60b. Therefore, in the present embodiment, since the first image 60a and the second image 60b are not registered based on the similarity between the images, the accuracy of the registration between the first image 60a and the second image 60b can be improved, and thus the image quality of the panoramic ultrasound image 62 can be improved.Furthermore, in the present embodiment, since the first single image 60a and the second single image 60b are registered on the basis of the first position 14a and the second position 14b without it being necessary to calculate the similarity between the single images for the registration of the first single image 60a and the second single image 60b, the computational effort for the registration (i.e. the generation of the panoramic ultrasound image 62) of the first single image 60a and the second single image 60b can be reduced.

[0055] Fig. Figure 11 is a diagram showing a reference area 64 set in the first image 60a. In the present embodiment, in addition to the first position 14a and the second position 14b of the ultrasound probe 14, the position of the second image 60b relative to the first image 60a, or the position and orientation based on the similarity between the images of the first image 60a and the second image 60b, can be determined. In particular, the combined processing unit 50 sets a portion of the first image 60a as the reference area 64.Next, the combination processing unit 50 performs image matching processing or the like between the reference area 64 and the second frame 60b to specify a correspondence area 66, which is an area in the second frame 60b that corresponds to the reference area 64 (exhibiting a similarity equal to or greater than a predetermined similarity threshold). Then, the combination processing unit 50 combines the first frame 60a and the second frame 60b by shifting the first frame 60a and the second frame 60b based on the difference between the reference area 64 and the correspondence area 66. Specifically, the combination processing unit 50 determines the position of the second frame 60b relative to the first frame 60a, or its position and orientation such that the reference area 64 and the correspondence area 66 overlap.

[0056] In the present embodiment, since the first position 14a and the second position 14b of the ultrasound probe 14 can also be used, for example, after the position of the second single image 60b in relation to the first single image 60a or the position and orientation have been determined on the basis of the first position 14a and the second position 14b, in a case where the reference area 64 and the correspondence area 66 differ from each other, processing of fine-tuning the position of the second single image 60b in relation to the first single image 60a or the position and orientation can be carried out so that the reference area 64 and the correspondence area 66 overlap each other.

[0057] Fig. Figure 12 is a diagram showing a superimposed section 68. Here, the reference area 64 must be in the superimposed section 68 (hatched section in Fig. 12) is set, which is a section of the first frame 60a corresponding to a section in which the scanning area SPa corresponding to the first frame 60a and the scanning area SPb corresponding to the second frame 60b overlap. This is because, in a case where the reference area 64 is set in a section other than the superimposed section 68, there should be no area in the second frame 60b where the similarity to the reference area 64 is equal to or greater than the similarity threshold, and the correspondence area 66 is not detected from the second frame 60b. In the present embodiment, the combination processing unit 50 can specify the section in which the scanning area SPa and the scanning area SPb overlap based on the first position 14a and second position 14b detected by the probe information acquisition unit 48.Therefore, the combination processing unit 50 sets the reference area 64 in the superimposed section 68 based on the first position 14a and the second position 14b.

[0058] The processing of the intermediate image generation unit 52 is described with reference to Fig. 13 and Fig. 14 described. Fig. Figure 13 is a third diagram showing the ultrasound probe 14, which is located at the first position 14a and the second position 14b. Fig. Figure 14 is a diagram showing an intermediate image 60m. In a case where the operator rapidly moves the ultrasound probe 14 in the scanning direction when acquiring multiple images to generate the panoramic ultrasound image 62, the first position 14a and the second position 14b may be separated from each other, as shown in Fig. Figure 13 shows that in such a case, there may be no superimposed section between the scanning area SPa and the scanning area SPb, and there may be a gap between the scanning area SPa and the scanning area SPb. In a case where the first single image 60a, corresponding to the scanning area SPa, and the second single image 60b, corresponding to the scanning area SPb, are combined as they are, the first single image 60a and the second single image 60b are discontinuous in the panoramic ultrasound image 62.

[0059] Therefore, in a case where the second position 14b is separated from the first position 14a by a distance equal to or greater than the first threshold distance, the intermediate image generation unit 52 generates the intermediate image 60m, which interpolates between the first image 60a and the second image 60b in the panoramic ultrasound image 62. The first threshold distance can be predetermined. For example, the first threshold distance is set to a distance between the first position 14a and the second position 14b such that the scanning area SPa and the scanning area SPb do not overlap.

[0060] In particular, the intermediate frame generation unit 52 generates the intermediate frame 60m based on the first frame 60a and the second frame 60b. A known technology can be used as a method for generating the intermediate frame 60m based on the first frame 60a and the second frame 60b. For example, the intermediate frame 60m can be generated by applying a spatiotemporal filter to the first frame 60a and the second frame 60b. The filter type can be a smoothing filter, an anisotropic filter, an application filter, or a combination thereof. Furthermore, the intermediate frame 60m can be nonlinearly corrected by detecting the local change of the second frame 60b relative to the first frame 60a.

[0061] In a case where the intermediate image 60m is generated by the intermediate image generation unit 52, the combination processing unit 50 combines the first image 60a, the second image 60b, and the intermediate image 60m, which is positioned between the first image 60a and the second image 60b, to generate the panoramic ultrasound image 62. In this case, the intermediate image 60m can be positioned midway between the first image 60a and the second image 60b.

[0062] The processing of the notification processing unit 54 is carried out with reference to Fig. 15 and Fig. 16 described. Fig. Figure 15 is a fourth diagram showing the ultrasound probe 14, which is located at the first position 14a and the second position 14b. Fig. Figure 15 is a top view of the ultrasound probe 14. In order to adequately generate the panoramic ultrasound image 62, it is necessary for the operator to move the ultrasound probe 14 in the scanning direction when acquiring multiple individual images. That is to say, in Fig. 15 the ultrasound probe 14 can be moved from the first position 14a to the second position 14b1 in the scanning direction, but in a case where the second position 14b2 is a position to which the first position 14a has transitioned in the Z-axis direction, it is not possible to generate a suitable panoramic ultrasound image 62.

[0063] Therefore, in a case where an angle φ1 exists between the scanning direction (X-axis direction in the example of Fig. 15) If the angle of the ultrasonic beam from the ultrasonic probe 14, in a case where the ultrasonic probe 14 is positioned at the first position 14a, and the direction from the first position 14a to the second position 14b is equal to or greater than a predetermined first threshold angle, the notification processing unit 54 shall notify the operator. The notification procedure may be carried out, for example, by displaying a warning on the display 38, emitting a warning tone, causing a light-emitting unit to emit light, or a combination thereof.

[0064] Fig. Figure 16 is a fifth diagram showing the ultrasound probe 14, which is located at the first position 14a and the second position 14b. Fig. Figure 16 is also a top view of the ultrasound probe 14. To adequately generate the panoramic ultrasound image 62, it is also necessary to keep the scanning direction of the ultrasound beam in the ultrasound probe 14 constant when multiple individual images are acquired. If the scanning direction of the ultrasound beam fluctuates while multiple individual images are acquired, it is not possible to generate a suitable panoramic ultrasound image 62.

[0065] Therefore, in a case where an angle φ2 exists between the scanning direction (X-axis direction in the example of Fig. 16) the ultrasound beam in the first orientation, which is the orientation of the ultrasound probe 14 in a case where the ultrasound probe 14 is located at the first position 14a, and the scanning direction of the ultrasound beam in the second orientation, which is the orientation of the ultrasound probe 14 in a case where the ultrasound probe 14 is located at the second position 14b, is equal to or greater than a predetermined second threshold angle, the notification processing unit 54 shall notify the operator.

[0066] Furthermore, as in Fig. 13 shown, in a case where the first position 14a and the second position 14b are separated by a distance equal to or greater than the first threshold distance, the intermediate image generation unit 52 the intermediate image 60m, but from the point of view of generating a suitable panoramic ultrasound image 62 it is undesirable that the first position 14a and the second position 14b are separated.

[0067] Therefore, the notification processing unit 54 can notify the operator in a case where the second position 14b is separated from the first position 14a by a distance equal to or greater than the second threshold distance. Here, the second threshold distance can be the same as the first threshold distance or different from it.

[0068] Fig. Figure 17 is a diagram showing a display example of the panoramic ultrasound image 62 and the acquired image 22. The display controller 36 displays the panoramic ultrasound image 62 generated by the combination processing unit 50 on the display 38. Furthermore, the display controller 36 can display the panoramic ultrasound image 62 together with the multiple acquired images 22 obtained by imaging the ultrasound probe 14 and the subject E, the acquired images 22 arranged in an imaging sequence, in a case where the received signals corresponding to each of the multiple individual images used to generate the panoramic ultrasound image 62 are captured. In particular, as shown in Fig. As shown in Figure 17, the display control 36 displays the multiple acquired images 22 side by side in the left and right directions in the imaging sequence such that each individual image forming the panoramic ultrasound image 62 and the acquired image 22, which is acquired in a case where the received signal corresponding to the individual image is captured, correspond to each other above or below the panoramic ultrasound image 62. As a result, the operator can easily determine which position of the examinee E each individual image forming the panoramic ultrasound image 62 corresponds to, or the position and orientation of the ultrasound probe 14 in a case where each individual image is acquired.

[0069] The outline of the configuration of the panoramic ultrasound imaging system 10 according to the present embodiment is as described above. Below is a flow of processing of the ultrasound diagnostic device 16 with reference to a [reference to a document / ... Fig. The flowchart shown in section 18 is described.

[0070] In step S10, the operator moves the ultrasound probe 14 in the scanning direction of the ultrasound beam. The image generation unit 34 generates several individual images, corresponding to each scanning area SP arranged in the scanning direction of the ultrasound beam, based on the received signal detected by the ultrasound probe 14, which transmits the ultrasound waves to and receives them from the examinee E. During this time, the camera 12 images the ultrasound probe 14 (in particular the probe detection marker 20) and transmits the captured image 22 to the ultrasound diagnostic device 16.The probe information acquisition unit 48 acquires several position information pieces that indicate the position of the ultrasound probe 14 in a case in which the received signal corresponding to each individual image is acquired, and several orientation information pieces that indicate the orientation of the ultrasound probe 14 in a case in which the received signal corresponding to each individual image is acquired, by analyzing the acquired image 22.

[0071] In step S12, the notification processing unit 54 determines whether, in a case where the received signals corresponding to two adjacent frames in the acquisition sequence are acquired, the position or orientation of the ultrasound probe 14 meets the notification condition, based on the position and orientation information acquired in step S10. For example, the notification processing unit 54 determines whether the angle φ1 (see Fig. 15), which is formed between the scanning direction of the ultrasound beam from the ultrasound probe 14 in a case where the ultrasound probe 14 is located at the first position 14a, and the direction from the first position 14a to the second position 14b, is equal to or greater than the predetermined first threshold angle. Alternatively, the notification processing unit 54 determines whether the angle φ2 (see Fig. 16), which is formed between the scanning direction of the ultrasound beam in the first orientation and the scanning direction of the ultrasound beam in the second orientation, is equal to or greater than the predetermined second threshold angle. Alternatively, it is determined whether the second position 14b is separated from the first position 14a by a distance equal to or greater than the second threshold distance. If the notification condition is found to be met, the process proceeds to step S14.

[0072] In step S14, the notification processing unit 54 notifies the operator. If step S12 determines that the notification condition is not met, the process proceeds to step S16.

[0073] In step S16, the combination processing unit 50 sets the reference area 64 in the first single image 60a based on the position and orientation information acquired in step S10. The combination processing unit 50 sets the reference area 64 in the superimposed section 68 (see Fig. 12) on the basis of the first position 14a and the second position 14b.

[0074] In step S18, the combination processing unit 50 performs an affine transformation of the second frame 60b based on the position and orientation information acquired in step S10. Specifically, the combination processing unit 50 calculates a difference (ΔPx) in the position in the scanning direction between the first frame 60a and the second frame 60b with respect to the number of pixels corresponding to the position relationship in the scanning direction between the scanning area SPa and the scanning area SPb, based on the difference between the first position 14a and the second position 14b, and moves the second frame 60b in the scanning direction by the calculated difference ΔPx while remaining parallel with respect to the first frame 60a.Furthermore, the combination processing unit 50 rotates the second single image 60b relative to the first single image 60a such that the orientation of the scanning surface SPb is the same as the orientation of the scanning surface SPa based on the difference between the first orientation and the second orientation.

[0075] In step S20, the combination processing unit 50 specifies the correspondence area 66 in the second frame 60b, which corresponds to the reference area 64 set in step S16, and determines whether the similarity between the reference area 64 and the correspondence area 66 is equal to or greater than a predetermined similarity threshold. If the similarity between the reference area 64 and the correspondence area 66 is less than a predetermined similarity threshold, the process returns to step S18, and the combination processing unit 50 adjusts the position or orientation of the second frame 60b so that the similarity between the reference area 64 and the correspondence area 66 is equal to or greater than the predetermined similarity threshold.In a case where the similarity between the reference area 64 and the correspondence area 66 is equal to or higher than a predetermined similarity threshold, the process proceeds to step S22.

[0076] In step S22, the combination processing unit 50 combines the first single image 60a and the second single image 60b, which were subjected to affine transformation in step S18.

[0077] In step S24, the combination processing unit 50 determines whether the combination processing for all individual images to generate the panoramic ultrasound image 62 is complete. If there is still one remaining individual image, the process returns to step S16, and the processing of steps S16 to S22 is repeated for the next individual image (for example, the second individual image 60b and the third individual image). If step S24 determines that the combination processing for all individual images to generate the panoramic ultrasound image 62 is complete, the process terminates.

[0078] Although the panoramic ultrasound image generation device according to the present disclosure has been described above, the panoramic ultrasound image generation device according to the present disclosure is not limited to the embodiment described above, and various modifications can be made without deviating from the scope of protection of the present disclosure.

[0079] Furthermore, in each of the embodiments described above, for example, the panoramic ultrasound image generation device is the ultrasound diagnostic device 16, and each of the functions of the image generation unit 34, the display control 36, the probe information acquisition unit 48, the combination processing unit 50, the intermediate single image generation unit 52, and the notification processing unit 54 is provided in the ultrasound diagnostic device 16. However, it is not absolutely necessary that each of these functions be performed by the ultrasound diagnostic device 16. For example, these functions can be performed by a server computer or the like that is communicatively connected to the ultrasound diagnostic device 16.Furthermore, it is possible that not all of the functions described above are performed by one device, and the functions described above can be performed by the cooperation of several devices. Reference symbol list 10 Panoramic ultrasound imaging system 12 Camera 14 ultrasound probe 14a first position 14b second position 16 Ultrasound diagnostic device 20 Probe detection marking 22 recorded images 30 transmission and reception unit 32 Signal processing unit 34 Image generation unit 36 Display control 38 ads 40 Communication interface 42 Input interface 44 storage 46 Control 48 Probe Information Acquisition Unit 50 combination processing units 52 Intermediate image generation unit 54 Notification processing unit 60a first single image 60b second single image 60m intermediate image 62 Panoramic ultrasound image 64 Reference range 66 Correspondence area 68 superimposed section QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] JP 4299015B [0004, 0005]

Claims

[1] Panoramic ultrasound image generating device, producing a panoramic ultrasound image by combining several individual images, including a first individual image obtained by scanning with an ultrasound beam from an ultrasound probe located at a first position towards a subject, and a second individual image obtained by scanning with the ultrasound beam from the ultrasound probe located at a second position, different from the first position, towards the subject, comprising: a probe information acquisition unit that acquires position information indicating the position of the ultrasound probe in a case where a received signal corresponding to each frame for generating the panoramic ultrasound image is acquired; and a combination processing unit that generates the panoramic ultrasound image by combining the multiple individual images, wherein the combination processing unit is configured to combine the first individual image and the second individual image by shifting the first individual image and the second individual image based on a difference between the first position and the second position located in a scanning direction of the ultrasound beam from the first position. [2] Panoramic ultrasound image generation device according to claim 1, wherein the probe information acquisition unit further acquires orientation information that specifies an orientation of the ultrasound probe in a case in which the received signal corresponding to each individual frame for generating the panoramic ultrasound image is acquired, and The combination processing unit rotates the second single image relative to the first single image in accordance with a rotation amount that corresponds to a difference between a first orientation, which is an orientation of the ultrasound probe in a case where the ultrasound probe is in the first position, and a second orientation, which is an orientation of the ultrasound probe in a case where the ultrasound probe is in the second position, and combines the first single image and the second single image. [3] Panoramic ultrasound image generation device according to claim 1 or 2, wherein the combination processing unit based on the first position and the second position, a reference area is established in a superimposed section, which is a section of the first frame, corresponding to a section in which a scanning area corresponding to the first frame and a scanning area corresponding to the second frame overlap, and a correspondence area, which is an area in the second frame that corresponds to the reference area of ​​the first frame, is specified, and the first frame and the second frame are combined by shifting the first frame and the second frame based on a difference between a position of the reference area and a position of the correspondence area. [4] Panoramic ultrasound image generation device according to one of claims 1 to 3, further comprising: an intermediate image generation unit that generates an intermediate image for interpolation between the first image and the second image in the panoramic ultrasound image based on the first image and the second image in a case where the second position is separated from the first position by a distance equal to or greater than a first threshold distance, wherein the combination processing unit generates the panoramic ultrasound image by combining the first single image, the second single image and the intermediate single image that is arranged between the first single image and the second single image. [5] Panoramic ultrasound image generation device according to any one of claims 1 to 4, further comprising: a notification processing unit that notifies an operator in a case where an angle formed between a scanning direction of the ultrasonic beam from the ultrasonic probe in a case where the ultrasonic probe is at the first position and a direction from the first position to the second position is equal to or greater than a first threshold angle. [6] Panoramic ultrasound image generation device according to claim 2, further comprising: a notification processing unit that notifies an operator in a case where an angle formed between a scanning direction of the ultrasonic beam in the first orientation and a scanning direction of the ultrasonic beam in the second orientation is equal to or greater than a second threshold angle. [7] Panoramic ultrasound image generation device according to claim 1, further comprising: a notification processing unit that notifies an operator in a case where the second position is separated from the first position by a distance equal to or greater than a second threshold distance. [8] Panoramic ultrasound image generation device according to any one of claims 1 to 7, further comprising: a camera that images the patient and the ultrasound probe; and a display control that displays the panoramic ultrasound image together with several recorded images obtained by imaging the ultrasound probe and the subject, wherein the recorded images are arranged in an imaging sequence, in a case in which received signals corresponding to each of the several individual images are captured, on a display unit. [9] Panoramic ultrasound image generation program which causes a computer to generate a panoramic ultrasound image by combining several individual images, including a first individual image obtained by scanning with an ultrasound beam from an ultrasound probe located at a first position towards a subject, and a second individual image obtained by scanning with the ultrasound beam from the ultrasound probe located at a second position, which differs from the first position, towards the subject, in order to function as: a probe information acquisition unit that acquires position information indicating the position of the ultrasound probe in a case where a received signal corresponding to each frame for generating the panoramic ultrasound image is acquired; and a combination processing unit that generates the panoramic ultrasound image by combining the multiple individual images, wherein the combination processing unit is configured to combine the first individual image and the second individual image by shifting the first individual image and the second individual image based on a difference between the first position and the second position located in a scanning direction of the ultrasound beam from the first position.

Images