Ultrasound diagnostic apparatus and storage medium
By using deep learning technology and a learned model, the imaging conditions of the ultrasound diagnostic device are automatically adjusted, solving the problem that users have difficulty in selecting the optimal parameters and improving the accuracy of the imaging site and the examination results.
Patent Information
- Application Number
- CN202410610652.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-29
- Filing Date
- 2024-05-16
- Publication Date
- 2024-11-29
AI Technical Summary
In ultrasound diagnostic devices, it is difficult for users to select the optimal imaging parameters for each imaging area, resulting in an inability to properly adjust imaging conditions and affecting the examination results.
Using deep learning technology, the imaging sites are identified based on the ultrasound images of the subject, and the imaging conditions are automatically changed after the model has been learned, thereby improving the accuracy of imaging site identification.
By generating input images of predetermined sizes, the accuracy of inference of photographic parts is improved, ensuring that inspections are conducted under appropriate photographic conditions.
Smart Images

Figure CN119033403A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an ultrasonic diagnostic apparatus capable of changing imaging conditions, and a storage medium including instructions executed by the ultrasonic diagnostic apparatus. Background Art
[0002] When scanning a subject using an ultrasonic diagnostic apparatus, the user sets imaging conditions for each imaging site before starting scanning of the subject.
[0003] Prior art literature
[0004] Patent Literature
[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2021-171559 Summary of the invention
[0006] Problem that the invention aims to solve
[0007] The imaging conditions include various parameters. Therefore, it is difficult for the user to select the optimal parameters for each imaging part. Therefore, in the ultrasonic diagnostic apparatus, a preset in which the imaging conditions are predetermined is prepared for each imaging part. When imaging a subject, the user can set the imaging conditions corresponding to the imaging part by selecting the preset corresponding to the imaging conditions of the subject.
[0008] However, depending on the user, there are cases where an appropriate preset cannot be selected or where parameter adjustment corresponding to the imaging part cannot be fully performed, and there are many cases where it is difficult to perform an examination of the subject under appropriate imaging conditions.
[0009] As a method to deal with this problem, the following technology is being studied: using deep learning technology to determine the imaging part of the subject based on the ultrasonic image of the subject, and automatically changing the imaging conditions when the current imaging conditions set by the user are not suitable for the imaging part of the subject.
[0010] When estimating the imaging part of the subject, an input image is created based on the ultrasonic image of the subject, and the input image is input to the learned neural network to estimate the imaging part.
[0011] However, depending on the ultrasonic viewing angle and ultrasonic depth, the estimated imaging part may not match the actual imaging part. In this case, if the imaging conditions are automatically changed, the imaging conditions may be changed to imaging conditions that are not suitable for the actual imaging part.
[0012] Therefore, a technology capable of improving the accuracy of distinguishing an imaging part is desired.
[0013] Technical Solution
[0014] A first aspect of the present invention is an ultrasonic diagnostic apparatus comprising:
[0015] Ultrasonic probe;
[0016] Display; and
[0017] Processor: communicates with the ultrasound probe and the display,
[0018] Wherein, the processor executes:
[0019] setting conditions for obtaining an ultrasonic image of a subject;
[0020] According to the condition, the ultrasonic probe transmits an ultrasonic beam, the ultrasonic probe receives an echo from the subject, and generates an ultrasonic image of the subject based on the echo received by the ultrasonic probe; and
[0021] Based on the ultrasonic image displayed on the display, an input image to be input to the learned model is produced;
[0022] Each input image produced by the processor is produced as:
[0023] The length of the subject in the depth direction has a first length, and
[0024] The length in a direction perpendicular to the depth direction of the subject has a second length.
[0025] A second aspect of the present invention is a storage medium storing instructions executable by a processor that communicates with an ultrasonic probe, wherein the instructions cause the processor to execute:
[0026] setting conditions for obtaining an ultrasonic image of a subject;
[0027] According to the condition, the ultrasonic probe transmits an ultrasonic beam, the ultrasonic probe receives an echo from the subject, and generates an ultrasonic image of the subject based on the echo received by the ultrasonic probe; and
[0028] Based on the ultrasonic image displayed on the display, an input image to be input to the learned model is produced;
[0029] Each input image produced by the processor is produced as:
[0030] The length of the subject in the depth direction has a first length, and
[0031] The length in a direction perpendicular to the depth direction of the subject has a second length.
[0032] Beneficial Effects
[0033] In the present invention, each input image is produced such that the length in the depth direction of the subject has a first length, and the length in the direction orthogonal to the depth direction of the subject has a second length. Therefore, the input image is produced to have the predetermined first length and second length, regardless of the shape or size of the ultrasonic image used to produce the input image. Therefore, since the size of the input image can be made universal, the accuracy of inference of the imaging part can be improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 This is a diagram showing how a subject is scanned by the ultrasonic diagnostic apparatus 1 according to the first embodiment of the present invention.
[0035] Figure 2 is a block diagram of the ultrasonic diagnostic apparatus 1 .
[0036] Figure 3 This is a schematic diagram of the original image.
[0037] Figure 4 It is an explanatory diagram of the original image P1.
[0038] Figure 5 It is an explanatory diagram of the original image P2.
[0039] Figure 6 This is a diagram for explaining a method of creating a training image PA1 from an original image P1.
[0040] Figure 7 This is a diagram illustrating preprocessing.
[0041] Figure 8 This is a diagram for explaining a method of creating a training image PA2 based on an original image P2.
[0042] Fig. 9 This is a diagram illustrating preprocessing.
[0043] Fig.10 It is a schematic diagram of original images P1 to Pn and training images PA1 to PAn created by preprocessing the original images P1 to Pn.
[0044] Fig.11 This is a diagram to illustrate the correct answer data.
[0045] Fig.12 It is an explanatory diagram of a method for creating the learned model 31.
[0046] Fig.13 FIG. 1 is a diagram showing an example of a flowchart executed in the examination of a subject.
[0047] Fig.14 It is an explanatory diagram of step ST11.
[0048] Fig.15 This is a diagram showing how the next new subject 53 is examined.
[0049] Fig.16 It is an explanatory diagram of step ST11.
[0050] Fig.17 2 is a diagram showing a modified example of the blank area.
[0051] Fig.18 2 is a diagram showing a modified example of the blank area.
[0052] Fig.19 2 is a diagram showing a modified example of the blank area.
[0053] Fig. 20 A diagram showing a modified example of a cropped image. DETAILED DESCRIPTION
[0054] Hereinafter, specific embodiments will be described, but the present invention is not limited to the following embodiments.
[0055] (1) First Embodiment
[0056] Figure 1 FIG. 1 is a diagram showing a state where a subject is scanned by an ultrasonic diagnostic apparatus 1 according to an embodiment of the present invention. Figure 2 is a block diagram of the ultrasonic diagnostic apparatus 1 .
[0057] The ultrasonic diagnostic apparatus 1 includes an ultrasonic probe 2, a transmission beamformer 3, a transmitter 4, a receiver 5, a reception beamformer 6, a processor 7, a display unit 8, a memory 9, and a user interface 10. The ultrasonic diagnostic apparatus 1 is an example of an ultrasonic image display system of the present invention.
[0058] The ultrasonic probe 2 has a plurality of vibration elements 2a arranged in an array. The transmission beam former 3 and the transmitter 4 drive the plurality of vibration elements 2a arranged in the ultrasonic probe 2, and transmit ultrasonic waves from the vibration elements 2a. The ultrasonic waves transmitted from the vibration elements 2a are transmitted to the subject 52 (refer to Figure 1 ) is reflected inside the waveguide, and the reflected echo is received by the vibration element 2a. The vibration element 2a converts the received echo into an electrical signal, and outputs the electrical signal as an echo signal to the receiver 5. The receiver 5 performs prescribed processing on the echo signal, and outputs it to the reception beamformer 6. The reception beamformer 6 performs reception beamforming on the signal received from the receiver 5, and outputs the echo data.
[0059] The receive beamformer 6 may be a hardware beamformer or a software beamformer. When the receive beamformer 6 is a software beamformer, the receive beamformer 6 may include one or more processors including i) a graphics processing unit (GPU), ii) a microprocessor, iii) a central processing unit (CPU), iv) a digital signal processor (DSP), and v) other types of processors capable of performing logical operations. The processor constituting the receive beamformer 6 may be a processor different from the processor 7, or may be constituted by the processor 7.
[0060] The ultrasound probe 2 may include all or part of the circuits for performing transmit beamforming and / or receive beamforming. For example, all or part of the transmit beamformer 3 , the transmitter 4 , the receiver 5 , and the receive beamformer 6 may be provided in the ultrasound probe 2 .
[0061] The processor 7 controls the transmit beamformer 3, the transmitter 4, the receiver 5, and the receive beamformer 6. In addition, the processor 7 communicates electronically with the ultrasonic probe 2. The processor 7 controls which vibration element 2a is in an active state and the shape of the ultrasonic beam transmitted from the ultrasonic probe 2. The processor 7 also communicates electronically with the display unit 8. The processor 7 can process echo data to generate an ultrasonic image. The term "electronic communication" can be defined as including both wired communication and wireless communication. According to one embodiment, the processor 7 can include a central processing unit (CPU). According to other embodiments, the processor 7 can include other electronic components such as a digital signal processor, a field programmable gate array (FPGA), a graphics processing unit (GPU), or other types of processors that can perform processing functions, and one or more processors. According to other embodiments, the processor 7 can include multiple electronic components that can perform processing functions. For example, the processor 7 can include two or more electronic components selected from a list of electronic components including a central processing unit, a digital signal processor, a field programmable gate array, and a graphics processing unit.
[0062] The processor 7 can also include a complex demodulator (not shown) for demodulating the RF data. In other embodiments, demodulation can be performed at an early stage in the processing chain.
[0063] In addition, the processor 7 can generate various ultrasonic images (e.g., B-mode images, color Doppler images, M-mode images, color M-mode images, spectral Doppler images, elastic imaging images, TVI images, distortion images, distortion velocity images, etc.) based on the data obtained through the processing performed by the receive beamformer 6. In addition, one or more modules can generate these ultrasonic images.
[0064] The image beams and / or image frames can be stored, and timing information indicating the time when the data was acquired from the memory can be recorded. The module can, for example, include a scan conversion module that performs a scan conversion operation to convert the image frame from the coordinate beam space to the display space coordinates. An image processor module can also be provided, which reads the image frame from the memory during the processing of the subject and displays the image frame in real time. The image processor module can store the image frame in the image memory, and the ultrasound image is read from the image memory and displayed on the display unit 8.
[0065] In this specification, the term "image" can broadly refer to both a visible image and data representing the visible image. In addition, the term "data" can include raw data, which is ultrasound data before scan conversion operation, and image data, which is data after scan conversion operation.
[0066] It should be noted that the above processing tasks that the processor 7 is responsible for may also be executed by multiple processors.
[0067] In addition, when the reception beamformer 6 is a software beamformer, the processing performed by the beamformer may be performed by a single processor or may be performed by a plurality of processors.
[0068] The display unit 8 is, for example, an LED (Light Emitting Diode) display unit, an LCD (Liquid Crystal Display) display unit, or an organic EL (Electro-Luminescence) display unit. The display unit 8 displays an ultrasonic image. Figure 1 As shown, the display unit 8 includes the display monitor 18 and the touch panel 181, but the display unit 8 may be composed of a single display unit instead of the display monitor 18 and the touch panel 181. Alternatively, two or more display devices may be provided instead of the display monitor 18 and the touch panel 181.
[0069] The memory 9 is an optional known data storage medium. In one example, the ultrasonic image display system includes a non-temporary storage medium and a temporary storage medium as a memory. In addition, the ultrasonic image display system can also include multiple memories. The non-temporary storage medium is a non-volatile storage medium such as an HDD (Hard Disk Drive), a ROM (Read Only Memory), etc. The non-temporary storage medium can include a mobile storage medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), etc. The program executed by the processor 7 is stored in the non-temporary storage medium. The temporary storage medium is a volatile storage medium such as a RAM (Random Access Memory).
[0070] The memory 9 stores one or more instructions that can be executed by the processor 7. The one or more instructions enable the processor 7 to perform various actions.
[0071] It should be noted that the processor 7 may also be configured to be wired or wirelessly connectable to the external storage device 15. In this case, instructions executed by the processor 7 may be stored in both the memory 9 and the external storage device 15 in a distributed manner.
[0072] The user interface 10 can receive input from a user 51 (e.g., an operator). For example, the user interface 10 receives input of instructions and information from the user 51. The user interface 10 includes a keyboard, hard keys, a trackball, a rotary control, and soft keys. The user interface 10 may also include a touch screen that displays soft keys.
[0073] The ultrasonic diagnostic apparatus 1 is configured as described above.
[0074] When scanning a subject using an ultrasonic diagnostic apparatus, the user sets imaging conditions for each imaging site before starting scanning of the subject.
[0075] The imaging conditions include various parameters. Therefore, it is difficult for the user to select the optimal parameters for each imaging part. Therefore, in the ultrasonic diagnostic apparatus, a preset in which the imaging conditions are predetermined is prepared for each imaging part. When imaging a subject, the user can set the imaging conditions corresponding to the imaging part by selecting the preset corresponding to the imaging conditions of the subject.
[0076] However, depending on the user, there are cases where an appropriate preset cannot be selected or where parameter adjustment corresponding to the imaging part cannot be fully performed, and there are many cases where it is difficult to perform an examination of the subject under appropriate imaging conditions.
[0077] As a method to deal with this problem, the following technology is being studied: using deep learning technology to determine the imaging part of the subject based on the ultrasonic image of the subject, and automatically changing the imaging conditions when the current imaging conditions set by the user are not suitable for the imaging part of the subject.
[0078] When estimating the imaging part of the subject, an input image is created based on the ultrasonic image of the subject, and the input image is input to the learned neural network to estimate the imaging part.
[0079] However, depending on the ultrasonic viewing angle and ultrasonic depth, the estimated imaging part may not match the actual imaging part. In this case, if the imaging conditions are automatically changed, the imaging conditions may be changed to imaging conditions that are not suitable for the actual imaging part.
[0080] Therefore, in order to cope with the above-mentioned problem, the ultrasonic diagnostic apparatus 1 according to the first embodiment is configured to improve the estimation accuracy of the imaging part. The first embodiment will be described in detail below.
[0081] It should be noted that, in the first embodiment, the learned model is used to infer the imaging part of the subject, and whether to change the imaging conditions is determined based on the inference result. Therefore, in the first embodiment, before examining the subject, a learning phase for generating a learned model is performed to generate a learned model suitable for inferring the imaging part of the subject. Therefore, the learning phase for generating the learned model is first described below. Then, after explaining the learning phase, a method for automatically changing the imaging conditions during the examination of the subject is explained.
[0082] (About the learning stage)
[0083] Figure 3 to Figure 12 This is an illustration of the learning phase.
[0084] In the learning phase, first, an original image serving as a basis for generating a training image is prepared.
[0085] Figure 3 This is a schematic diagram of original images P1 to Pn.
[0086] In this embodiment, ultrasonic images Pi (i=1 to n) are prepared as original images. Ultrasonic images Pi include ultrasonic images obtained from medical facilities such as hospitals, ultrasonic images obtained from medical equipment manufacturers, etc. For example, 5,000 to 10,000 original images are prepared as original images.
[0087] The original images P1 to Pn include images of various parts to be examined. Examples of the examination target parts include, but are not limited to, the abdomen, breasts, and kidneys. Various parts to be examined by ultrasonic examination can be used as the examination target parts.
[0088] Each original image is described below.
[0089] Figure 4 It is an explanatory diagram of the original image P1.
[0090] Figure 4 The subject 100 and an original image P1 obtained by imaging the subject 100 are shown.
[0091] exist Figure 4 The subject 100 is shown in the upper part. An enlarged view of a cross section 101 of the subject 100 is shown on the right side of the subject 100. An area 102 is shown in the cross section 101. The area 102 is an area representing the cross section of the breast of the subject. The area 102 is a square area. The longitudinal length RD1 (cm) of the area 102 represents the length in the depth direction (y direction) of the subject 100. In addition, the transverse length RW1 (cm) of the area 102 represents the length in the direction orthogonal to the depth direction of the subject 100 (the width direction of the subject 100).
[0092] exist Figure 4 The lower part of is a schematic diagram of the original image P1 of the area 102.
[0093] The original image P1 is a square having four sides 21, 22, 23, and 24. The longitudinal length D1 (cm) of the original image P1 represents the length in the depth direction (y direction) of the subject 100 (i.e., the length RD1 of the region R1). In addition, the lateral length W1 (cm) of the original image P1 represents the length in the direction (width direction of the subject 100) orthogonal to the depth direction of the subject 100 (i.e., the length RW1 of the region R1). Therefore, the length D1 of the original image P1 represents the length RD1 of the region 102, and the length W1 of the original image P1 represents the length RW1 of the region 102. The length D1 of the original image P1 (the length RD1 of the region 102) is, for example, 4 (cm), and the length W1 of the original image P1 (the length RW1 of the region 102) is, for example, 4 (cm).
[0094] Figure 5It is an explanatory diagram of the original image P2.
[0095] Figure 5 The subject 110 and an original image P2 obtained by imaging the subject 110 are shown.
[0096] exist Figure 5 The subject 110 is shown in the upper part. An enlarged view of a cross section 111 of the subject 110 is shown on the right side of the subject 110. An area 112 is shown in the cross section 111. The area 112 is an area representing the cross section of the kidney of the subject. The area 112 is a roughly trapezoidal area. The longitudinal length RD2 (cm) of the area 112 represents the length in the depth direction (y direction) of the subject 110. In addition, the length RWS2 (cm) between the corners RC1 and RC2 of the area 112 represents the length of the upper side in the direction orthogonal to the depth direction of the subject (the width direction of the subject). In addition, the length RWL2 (cm) between the corners RC3 and RC4 of the area 112 represents the length of the lower side in the width direction of the subject.
[0097] exist Figure 5 The lower part of is a schematic diagram of the original image P2 of the area 112.
[0098] The original image P2 has a roughly trapezoidal shape. The original image P2 has four sides 26, 27, 28 and 29. The sides 26, 28 and 29 are straight lines, but the side 27 has an arc shape. The longitudinal length D2 (cm) of the original image P2 represents the length in the depth direction (y direction) of the subject (i.e., the length RD2 of the region 112). The length WS2 (cm) between the corners C1 and C2 of the original image P2 represents the length RWS2 of the region 112 of the subject 110. In addition, the length WL2 (cm) between the corners C3 and C4 of the original image P2 represents the length RWL2 of the region 112 of the subject 110. Therefore, the length D2 of the original image P2 represents the length RD2 of the region 112, the length WS2 of the original image P2 represents the length RWS2 of the region 112, and the length WL2 of the original image P2 represents the length RWL2 of the region 112. The length D2 of the original image P2 (the length RD2 of the region 112) is, for example, 10 (cm), the length WS2 of the original image P2 (the length RWS2 of the region 112) is, for example, 5 (cm), and the length WL2 of the original image P2 (the length RWL1 of the region 112) is, for example, 10 (cm).
[0099] Similarly, ultrasonic images obtained by imaging various parts of various subjects are prepared as original images. Figure 4 Show the original square image, Figure 5 Although a substantially trapezoidal original image is shown, an ultrasonic image of other various shapes (eg, sectoral shape) obtained by ultrasonic examination can also be used as the original image.
[0100] A training image is created using each of these original images P1 to Pn. Hereinafter, a method of creating a training image from each original image will be described.
[0101] Figure 6 This is a diagram for explaining a method of creating a training image PA1 from an original image P1.
[0102] In this embodiment, the training image PA1 is created by preprocessing the original image P1.
[0103] As described above, the original image P1 is a square, and the original image P1 has a size of D1 = W1 = 4 cm. On the other hand, the training image PA1 is also a square like the original image P1, but the training image PA1 has a larger size than the original image P1. In this embodiment, the training image PA1 has a size of DA1 = WA1 = 6 cm, but the size of the training image is not limited to 6 cm, and can be shorter or longer than 6 cm.
[0104] In this embodiment, preprocessing is performed to produce a training image (DA1=WA1=6cm) from the original image P1 (D1=W1=4cm). The preprocessing is described below. It should be noted that the preprocessing is a process that can be performed by a device having an image processing function, for example, a general computer can be used as such a device.
[0105] Figure 7 This is a diagram illustrating preprocessing.
[0106] Figure 7 The middle figures show schematic diagrams (a) and (b) for explaining the pretreatment step.
[0107] First, the schematic diagram (a) will be described.
[0108] The schematic diagram (a) shows the contour F of the original image P1 and the training image PA1. The contour F is indicated by a dotted line. The contour F is shown in such a way that the upper side of the contour F coincides with the upper side of the original image P1.
[0109] The length D1 of the original image P1 in the depth direction of the subject is 4 cm. In addition, the length W1 of the original image P1 in the width direction of the subject is also 4 cm. Therefore, the length D1 of the original image P1 in the depth direction is shorter than the length DA1 of the training image in the depth direction by ΔD (= 3 cm), and further, the length W1 of the original image P1 in the width direction is shorter than the length WA1 of the training image in the width direction by ΔW (= ΔW1 + ΔW2) (ΔW = 3 cm). Therefore, in order to eliminate the insufficient portion ΔD1 in the depth direction of the original image P1 and the insufficient portion ΔW in the width direction of the original image P1, a zero-filling process is performed on the original image P1 to fill the blank area BL around the original image P1 with zero data to meet the size of the training image. The image after the zero-filling process is performed on the original image P1 is shown in the schematic diagram (b). In the schematic diagram (b), the blank area BL subjected to the zero-filling process is represented as an area filled with black. The size of the original image P1 is smaller than the size of the training image, but by performing the above-mentioned zero padding process as a pre-processing of the original image P1, the size of the original image P1 can be made consistent with the size of the training image PA1. It should be noted that in this embodiment, the training image PA1 of the desired size is produced by performing the zero padding process. However, as long as the training image can have the desired size, a process different from the zero padding process can also be performed.
[0110] It should be noted that, before or after the zero-filling process is performed on the original image P1, other pre-processing is performed as needed, but the description of other pre-processing is omitted here.
[0111] In this way, the training image PA1 can be created based on the original image P1.
[0112] Next, an example of creating a training image based on the original image P2 will be described.
[0113] Figure 8 This is a diagram for explaining a method of creating a training image PA2 based on an original image P2.
[0114] As described above, the original image P2 is a substantially trapezoidal shape. On the other hand, the training image PA2 is a square shape like the training image PA1 described above, and has the same size as the training image PA1 (DA2=WA2=6 cm).
[0115] In the present embodiment, preprocessing is performed to create a training image PA2 (DA2=WA2=6 cm) from the original image P2 (substantially trapezoidal). The preprocessing will be described below.
[0116] Fig. 9 This is a diagram illustrating preprocessing.
[0117] exist Fig. 9The middle figures show schematic diagrams (a) to (e) for explaining the pretreatment process.
[0118] First, the schematic diagram (a) will be described.
[0119] The outline diagram (a) shows the contour F of the original image P2 and the training image PA2. The contour F is indicated by a dotted line. The contour F is shown in the figure such that the upper side of the contour F coincides with the upper side of the original image P2.
[0120] The length D2 of the original image P2 in the depth direction of the subject is 10 cm, and the length WS2 of the upper side of the original image P2 is 5 cm. Therefore, in the case of the original image P2, the length WS2 of the upper side is 2 cm shorter than the width WA2 of the training image, but the length D2 in the depth direction is 3 cm longer than the length DA2 in the depth direction of the training image. Therefore, the part suitable for the training image is cut from the original image P2.
[0121] The schematic diagram (b) shows how a portion suitable for the training image is cut out from the original image P2.
[0122] The depth length D2 of the original image P2 is longer than the length DA2 of the training image. Therefore, the range from the body surface position Q1 of the original image P2 to the position Q2 6 cm lower in the depth direction is used as the image portion for the training image.
[0123] On the other hand, since the length WS2 of the upper side of the original image P2 is shorter than the length WA2 of the training image, the range from the position Q3 of the upper left corner C1 to the position Q4 of the upper right corner C2 of the original image P2 is used as the image portion used for the training image.
[0124] Therefore, the portion surrounded by the positions Q1, Q2, Q3 and Q4 of the original image P2 is cut out as the image portion PE2 used for the training image. The schematic diagram (c) shows an image (hereinafter referred to as a "cut-out image") PE2 cut out from the original image P2.
[0125] Next, a zero-filling process is performed on the cropped image PE2 to fill the blank areas BL1 and BL2 along the sides of the cropped image PE2 with zero data to meet the size of the training image. Schematic diagram (d) shows the cropped image PE2 before the zero-filling process is performed, and schematic diagram (e) shows the cropped image PE2 after the zero-filling process is performed. The area that has been zero-filled is displayed as an area filled with black. Therefore, the size of the cropped image PE2 obtained based on the original image P2 is smaller than the size of the training image, but by performing the above-mentioned zero-filling process as a preprocessing, the training image PA2 can be produced based on the original image P2. It should be noted that in the present embodiment, the training image PA2 of the desired size is produced by performing the zero-filling process, but as long as the training image can have the desired size, the training image can also be produced by performing preprocessing other than the zero-filling process.
[0126] It should be noted that, before or after the zero-filling process is performed on the original image P2, other pre-processing is performed as needed, but the description of other pre-processing is omitted here.
[0127] In this way, the training image PA2 can be created based on the original image P2.
[0128] Similarly, preprocessing is performed on other original images to generate square training images of 6 cm in length and 6 cm in width. Fig.10 As shown, training images PA1 to PAn created to have the same size can be prepared from original images P1 to Pn.
[0129] Next, the correct answer data is marked on the training images PA1 to PAn (refer to Fig.11 ).
[0130] Fig.11 This is a diagram to illustrate the correct answer data.
[0131] The training image PA1 is an image of a breast, so the training image PA1 is labeled "breast" as the correct answer data.
[0132] In addition, the training image PA2 is an image of kidneys. Therefore, the training image PA2 is labeled with "kidneys" as the correct answer data.
[0133] Similarly, the correct answer data are also labeled for the other training images PA3 to PAn. Therefore, the correct answer data are labeled for all the training images PA1 to PAn.
[0134] Then, if Fig.12As shown, by making the neural network 30 learn the above-mentioned training images PA1~PAn, a learned model 31 is produced. The learned model 31 is stored in a memory or an external storage device. It should be noted that the learned model 31 can be produced using any learning algorithm used in AI learning, machine learning, deep learning, etc. For example, the learned model 31 can be produced by learning with a teacher or by learning without a teacher.
[0135] In the first embodiment, the learned model 31 is used to automatically change the imaging conditions. Fig.13 An example of a method of automatically changing imaging conditions will be described.
[0136] Fig.13 FIG. 1 is a diagram showing an example of a flowchart executed in the examination of a subject.
[0137] In step ST1, the user 51 places the subject 52 (reference Figure 1 ) is guided to the examination room and the subject 52 is made to lie on the examination bed.
[0138] In addition, the user 51 operates the user interface 10 (refer to Figure 2 ), input patient information, set imaging conditions for obtaining an ultrasonic image of the subject, and other necessary settings. It should be noted that the imaging conditions include any conditions related to obtaining an ultrasonic image, such as conditions for transmitting an ultrasonic beam, conditions for receiving echoes from the subject, and conditions for processing data used to produce an ultrasonic image based on the received echoes.
[0139] Here, it is assumed that the imaging part of the subject is the "breast." Therefore, the user sets the imaging conditions for the breast.
[0140] After the user completes the preparation for the examination, he starts the examination of the subject 52. Fig.13 In the figure, t0 is used to represent the time when the inspection starts.
[0141] In addition, Fig.13 In the figure, "subject", "imaging part" and "imaging condition" are shown on the time axis. "Subject" means the subject to be examined, "imaging part" means the imaging part of the subject, and "imaging condition" means the imaging condition set in the ultrasonic diagnostic apparatus. For example, at the examination start time t0, the figure shows that "subject" is the subject 52, "imaging part" is the breast, and "imaging condition" is the imaging condition for the breast.
[0142] The user 51 operates the probe while pressing the ultrasonic probe 2 against the imaging part of the subject 52 to scan the subject 52. Here, since the imaging part of the subject is the breast, Figure 1As shown, the user 51 presses the ultrasonic probe 2 against the breast of the subject 52. The ultrasonic probe 2 transmits ultrasonic waves and receives echoes reflected from the subject 52. The received echoes are converted into electrical signals and output as echo signals to the receiver 5 (see FIG. 5 ). Figure 2 ). The receiver 5 performs prescribed processing on the echo signal and outputs it to the reception beamformer 6. The reception beamformer 6 performs reception beamforming on the signal received from the receiver 5 and outputs the echo data.
[0143] The processor 7 generates an ultrasonic image based on the echo data. The ultrasonic image is displayed on the display unit 8.
[0144] The user 51 checks the ultrasonic image displayed on the display unit 8 and saves the ultrasonic image as necessary. Then, the user 51 continues to perform the examination of the subject.
[0145] On the other hand, after starting the examination of the subject at time t0, the processor 7 periodically determines whether to change the imaging conditions and executes a process 41 for automatically changing the imaging conditions as needed. In this embodiment, the first process 41 is executed at time t1 after the examination start time t0. The process 41 is described below.
[0146] When the process 41 starts, first, in step ST10, the processor 7 determines the imaging part included in the ultrasonic image obtained between time t0 and time t1. The determination step ST10 will be described below.
[0147] First, in step ST11 , the processor generates an input image 71 to be input to the learned model 31 based on the ultrasonic image 61 obtained between time t0 and time t1 and displayed on the display unit 8 .
[0148] Fig.14 It is an explanatory diagram of step ST11.
[0149] When one ultrasonic image 61 is obtained between time t0 and time t1, the processor can generate an input image 71 for input to the learned model 31 based on the ultrasonic image 61. On the other hand, when multiple ultrasonic images are obtained between time t0 and time t1, the processor can select one ultrasonic image 61 from the multiple ultrasonic images and generate an input image 71 for input to the learned model 31 based on the selected ultrasonic image 61. When multiple ultrasonic images are obtained between time t0 and time t1, the processor can typically select the ultrasonic image obtained last between time t0 and time t1 (the ultrasonic image obtained just before time t1) as the ultrasonic image 61.
[0150] The ultrasonic image 61 is a rectangular image (D1 = W1 = 4 cm). Therefore, the ultrasonic image 61 is compared with the reference image 61. Figure 7 The same preprocessing as the method for producing the training image PA1 described above is performed to generate the input image 71. It should be noted that the size of the input image 71 is the same as the size of the training image PA1 described above (DA1 = WA1 = 6 cm). Therefore, the processor generates an input image 71 (DA1 = WA1 = 6 cm) larger than the ultrasonic image 61 based on the ultrasonic image 61 (D1 = W1 = 4 cm). Specifically, the preprocessing is performed as follows.
[0151] The processor performs zero padding processing on the ultrasonic image 61 to fill the blank area 161 around the ultrasonic image 61 with zero data to meet the size of the input image 71. Here, the blank area 161 is set along three sides 612, 613 and 614 of the four sides 611 to 614 of the ultrasonic image 61. Fig.14 (a) is a schematic diagram showing an ultrasonic image 61 before zero-filling processing. Fig.14 (b) is a schematic diagram showing an ultrasonic image 61 after zero padding. Therefore, the ultrasonic image 61 itself is smaller than the size of the input image 71, but by performing the above-mentioned zero padding as pre-processing of the ultrasonic image 61, the input image 71 of a desired size can be produced from the ultrasonic image 61. It should be noted that in this embodiment, the input image 71 of a desired size is produced by performing zero padding. However, as long as the input image 71 can have a desired size, pre-processing other than zero padding can also be performed.
[0152] It should be noted that, before or after the zero-filling process is performed on the ultrasonic image 61, other pre-processing is performed as necessary, but the description of other pre-processing is omitted here. After the input image 71 is generated, the process proceeds to step ST12.
[0153] In step ST12 , the processor 7 estimates the part represented by the input image 71 using the learned model 31 .
[0154] The processor 7 inputs the input image 71 to the learned model 31, and uses the learned model 31 to infer the part included in the input image 71. In the inference step, the processor calculates the probability that each imaging part is included in the input image 71. Then, the processor infers the imaging part included in the input image 71 based on the calculated probability for each imaging part.
[0155] Here, it is assumed that the probability of the breast exceeds the threshold value. Therefore, the processor infers that the imaging part included in the input image 71 is the breast. After the imaging part is inferred, the process proceeds to step ST20.
[0156] In step ST20, the processor determines whether to change the condition based on the estimated imaging part.
[0157] First, in step ST21, the processor determines whether the currently set imaging condition is the imaging condition corresponding to the imaging part estimated in step ST12. If the currently set imaging condition is the imaging condition corresponding to the imaging part estimated in step ST12, the processor proceeds to step ST22. On the other hand, if the currently set imaging condition is not the imaging condition corresponding to the imaging part estimated in step ST12, the processor proceeds to step ST23.
[0158] At time t1, the imaging condition set is the imaging condition for breast. On the other hand, the imaging part estimated in step ST12 is the breast. Therefore, since the currently set imaging condition (imaging condition for breast) is the imaging condition corresponding to the imaging part estimated in step ST12 (breast), the process proceeds to step ST22, the processor 7 determines that the imaging condition is not to be changed, and ends the process 41.
[0159] On the other hand, the user 51 continues to examine the subject 52 while operating the ultrasonic probe 2 after time t1. In addition, the processor also regularly executes the above process 41 after time t1. Here, in the process 41 executed after time t1, it is assumed that it is determined that the imaging conditions have not been changed (step ST22). Therefore, the automatic change of the imaging conditions is not performed, and the breast examination of the subject is ended. The end time of the mammography of the subject is represented by "t2". The user prepares for the examination of the next new subject.
[0160] Fig.15 This is a diagram showing how the next new subject is examined.
[0161] Hereinafter, a case will be described in which the imaging part of the new subject 53 is different from the imaging part of the previous subject 52. Here, a case will be described in which the imaging part of the previous subject 52 is the breast and the imaging part of the new subject 53 is the kidney.
[0162] After the user has finished the breast examination of the subject 52 that was just performed, he / she prepares for the kidney examination of the new subject 53. In this case, since the imaging site is changed from the breast to the kidney, the user needs to change the imaging conditions from the breast imaging conditions to the kidney imaging conditions. However, the following considers the case where the user starts the kidney examination of the new subject 53 without changing the imaging conditions.
[0163] The user starts the kidney examination of the subject 53 at time t3.
[0164] User 51 starts the kidney examination of subject 53 from time t3, but since the imaging conditions are not changed, the imaging conditions set are still the breast imaging conditions. Therefore, the user starts the kidney examination of subject 53 under the breast imaging conditions. Fig.15 As shown, a user 51 presses a probe 52 against the abdomen of a subject 53 to perform a kidney examination.
[0165] On the other hand, the processor 7 starts the kidney examination of the subject 53 at time t3 and then periodically executes the process 41. In the present embodiment, a case where the process 41 is executed at time t4 after time t3 will be described.
[0166] When the process 41 starts, first, in step ST10, the processor 7 determines the imaging part included in the ultrasonic image obtained between time t3 and time t4. The determination step ST10 will be described below.
[0167] First, in step ST11 , the processor pre-processes the ultrasonic image 62 obtained between time t3 and time t4 and displayed on the display unit 2 , and generates an input image 72 to be input to the learned model 31 .
[0168] Fig.16 It is an explanatory diagram of step ST11.
[0169] When one ultrasonic image 62 is obtained between time t3 and time t4, the processor can generate an input image 72 for input to the learning model 31 based on the ultrasonic image 62. On the other hand, when multiple ultrasonic images are obtained between time t3 and time t4, the processor can select one ultrasonic image 62 from the multiple ultrasonic images and generate an input image 72 for input to the learned model 31 based on the selected ultrasonic image 62. When multiple ultrasonic images are obtained between time t3 and time t4, the processor can typically select the ultrasonic image obtained last between time t3 and time t4 (for example, the ultrasonic image obtained just before time t4) as the ultrasonic image 62.
[0170] The ultrasonic image 62 is a substantially trapezoidal image. Therefore, the processor performs the same processing as the reference processing on the ultrasonic image 62. Fig. 9 The same preprocessing as the training image preparation method described above is performed to generate the input image 72. It should be noted that the size of the input image 72 is the same as the size of the training image PA2 described above, which is DA2 = WA2 = 6 cm. Therefore, the processor generates a rectangular input image 72 based on the roughly trapezoidal ultrasonic image 62. Specifically, the preprocessing is performed as follows.
[0171] like Fig.16As shown in (a), regarding the depth direction of the subject, the processor sets the range from position Q1 on the body surface of the ultrasonic image 62 to position Q2 separated by only 6 cm in the depth direction as the image portion for creating the input image. Also, regarding the width direction, the range from position Q3 at the upper left corner C1 of the ultrasonic image 62 to position Q4 at the upper right corner C2 is set as the image portion used for the training image.
[0172] Therefore, if Fig.16 As shown in (b), the processor determines the region 621 defined by Q1 to Q4 of the ultrasonic image 62 as the image portion for creating the input image, and cuts the image portion 621 from the ultrasonic image 62 (refer to Fig.16 (c)).
[0173] Next, the processor performs a zero-filling process of filling blank areas 622 and 623 along the sides of the cropped image 621 with zero data on the cropped image 621 cropped from the ultrasonic image 62 so as to meet the size of the input image 72 . Fig.16 (d) shows a schematic diagram of the cropped image 621 before zero-filling the blank areas 622 and 623. Fig.16 (e) shows a schematic diagram of a cropped image 621 after zero padding is performed on the blank areas 622 and 623. Therefore, an input image 72 of a desired size can be produced from the ultrasonic image 62. It should be noted that in this embodiment, the input image 72 of a desired size is produced by performing zero padding. However, preprocessing other than zero padding may be performed as long as the input image 72 can have a desired size.
[0174] It should be noted that, before or after the zero-filling process is performed on the ultrasonic image 62, other pre-processing is performed as necessary, but the description of other pre-processing is omitted here. After the input image 72 is generated, the process proceeds to step ST12.
[0175] In step ST12 , the processor 7 estimates the part represented by the input image 72 using the learned model 31 .
[0176] The processor 7 inputs the input image 72 to the learned model 31, and uses the learned model 31 to infer the part included in the input image 72. In the inference step, the processor calculates the probability that each imaging part is included in the input image 72. Then, the processor infers the imaging part included in the input image 72 based on the probability calculated for each imaging part.
[0177] Here, it is assumed that the probability of the kidney is the highest. Therefore, the processor infers that the imaging part included in the input image is the kidney. After the imaging part is inferred, the process proceeds to step ST20.
[0178] In step ST20, the processor determines whether to change the condition based on the estimated imaging part.
[0179] First, in step ST21, the processor determines whether the currently set imaging condition is the imaging condition corresponding to the imaging part estimated in step ST12. If the currently set imaging condition is the imaging condition corresponding to the imaging part estimated in step ST12, the processor proceeds to step ST22. On the other hand, if the currently set imaging condition is not the imaging condition corresponding to the imaging part estimated in step ST12, the processor proceeds to step ST23.
[0180] At time t4, the imaging condition set is the imaging condition for breast. On the other hand, the imaging part estimated in step ST12 is the kidney. Therefore, since the currently set imaging condition (the imaging condition for breast) is not the imaging condition corresponding to the imaging part estimated in step ST12 (the kidney), the process proceeds to step ST23.
[0181] In step ST23, the processor determines to change the imaging condition. Then, the process proceeds to step ST24, and the imaging condition is changed from the imaging condition for the breast to the imaging condition for the kidney. Fig.13 2 shows a state where the imaging condition is changed to the kidney immediately after time t4.
[0182] Therefore, the user starts imaging of the kidney under the imaging condition of the breast, but the processor automatically changes the imaging condition to the imaging condition of the kidney while the user is imaging the kidney. Therefore, even if the user forgets to change the imaging condition, after the processor changes the imaging condition, the user can still image the kidney under the imaging condition of the kidney, thereby obtaining a high-quality kidney image.
[0183] On the other hand, the user 51 continues to examine the kidney of the subject 53 while operating the ultrasonic probe 2 after time t4, and the processor periodically executes the above-mentioned process 41. Here, at time t5 after time t4, the flow of process 41 is executed.
[0184] When the flow of process 41 starts at time t5, in step ST11, by using Fig. 9 The ultrasonic image 63 displayed on the display unit 8 is preprocessed by a method to generate an input image 73. In step ST12, the input image 73 is input to the learned model 31 to estimate the imaging part. Then, in step ST20, it is determined whether to change the imaging conditions, and the process ends.
[0185] On the other hand, the user 51 continues to examine the subject 53 while operating the ultrasonic probe 2 after time t5. The processor also periodically executes the above process 41 after time t5. It should be noted that, in the process 41 after time t5, it is determined that the imaging conditions will not be changed (step ST22), and the examination of the subject 53 is completed at time t6.
[0186] When the examination of the subject 53 is completed, the flow of the process 41 is also regularly executed in the examination of the next new subject. Similarly, the flow of the process 41 is regularly executed every time a new subject is examined.
[0187] exist Fig.13 , the process 41 is schematically shown to be executed periodically even after the examination of the subject 53. Specifically, after time t6, in step ST11, the ultrasonic images 64, 65 ... 6m displayed on the display unit 8 are preprocessed to generate input images 74, 75 ... 7m, and in step ST12, the imaging part included in the input images 74, 75 ... 7m is estimated. Then, in step ST20, it is determined whether to change the imaging conditions, and the imaging conditions are changed as needed (step ST24), and the flow of process 41 ends.
[0188] As described above, in the present embodiment, the input images 71 to 7m are generated in a predetermined size regardless of the imaging conditions of the ultrasonic images 61 to 6m. Therefore, even if the user (or processor) changes the ultrasonic angle of view or the ultrasonic depth according to the imaging part of the subject, or the user (or processor) changes the ultrasonic angle of view or the ultrasonic depth during the examination of the subject, in step ST11, the input images 71 to 7m of the predetermined size can be obtained. Therefore, for example, even if the processor changes the imaging conditions at time t4, or the user manually changes the ultrasonic depth during the examination of the subject 52 (or 53), in step ST11, the input images 71 to 7m of the predetermined size can be obtained. Therefore, in step ST12, the processor performs inference based on the input images 71 to 7m of the same size, so that the accuracy of the imaging part can be improved, and a stable inference result can be obtained.
[0189] In addition, in the present embodiment, the length of the input image in the depth direction is the length measured based on the body surface of the subject. However, as long as an input image of a predetermined size is generated, the length of the input image in the depth direction may be set to the length measured based on a reference plane different from the body surface of the subject (e.g., a surface contained in the body of the subject, the surface of an organ).
[0190] It should be noted that, in this embodiment, Fig.14As shown, the blank area 161 is set along three sides 612, 613, and 614 among the four sides 611 to 614 of the ultrasonic image 61. However, the blank area does not necessarily have to be set along three sides, and various blank areas can be set.
[0191] Figure 17 to Figure 19 2 is a diagram showing a modified example of the blank area.
[0192] Fig.17 An example is shown in which the blank area 162 is set along the three sides 611, 612, and 614 of the ultrasonic image 61. Therefore, a zero filling process of filling the blank area 162 along the three sides 611, 612, and 614 with zeros is performed.
[0193] Fig.18 An example is shown in which the blank area 163 is set along both sides 613 and 614 of the ultrasonic image 61. Therefore, a zero filling process of filling the blank area 163 along both sides 613 and 614 with zeros is performed.
[0194] Fig.19 An example is shown in which the blank area 164 is set along the four sides 611 to 614 of the ultrasonic image 61. Therefore, a zero filling process is performed to fill the blank area 164 along the four sides 611 to 614 with zeros.
[0195] In this way, as long as an input image of a desired size can be created, the blank area can be set in an arbitrary shape for the ultrasonic image.
[0196] In addition, in this embodiment, if Fig.16 As shown in FIG. 6 , the cropped image 621 is cropped based on the position Q1 of the body surface of the ultrasonic image 62. However, the cropped image can be cropped based on any position of the ultrasonic image 62.
[0197] Fig. 20 A diagram showing a modified example of a cropped image.
[0198] exist Fig. 20 In the figure, as shown in the schematic diagram (a1), with respect to the depth direction, an example is shown in which the range from position Q11 below position Q1 to position Q21 only 6 cm lower in the depth direction is set as the image portion used for the training image. Therefore, the area surrounded by positions Q11, Q21, Q3, and Q4 is set as a cropped image 631 (refer to the schematic diagram (a2)). Then, blank areas 632 and 633 are set on the sides of the cropped image 631 (schematic diagram (a3)), and zero filling processing is performed (schematic diagram (a4)). In this way, the reference position for cropping the cropped image does not necessarily have to be the body surface, and the image can be cropped with the desired position as the reference according to the imaging conditions of the ultrasonic image, etc.
[0199] Description of Reference Numerals
[0200] 1: Ultrasonic diagnostic equipment
[0201] 2: Ultrasonic probe
[0202] 3: Transmit beamformer
[0203] 4: Transmitter
[0204] 5: Receiver
[0205] 6: Receive beamformer
[0206] 7: Processor
[0207] 8: Display unit
[0208] 9: Memory
[0209] 10: User Interface
[0210] 18: Display Monitor
[0211] 21, 22, 23, 24, 26, 27, 28, 29: Edge
[0212] 30: Neural Networks
[0213] 31: Learning the model
[0214] 41: Process
[0215] 51: User
[0216] 52, 53: Subject
[0217] 61~6m: Ultrasound image
[0218] 71~7m: Input image
[0219] 100, 110: Subject
[0220] 101, 111: Cross section
[0221] 102, 112: Area
[0222] 161, 162, 164, 622, 623, 632, 633: blank area
[0223] 163, 621, 631: Crop image
[0224] 181: Touch panel
[0225] 611, 612, 613, 614: Edge
Claims
1. An ultrasonic diagnostic device, comprising: Ultrasonic probe; a display unit; and Processor: communicates with the ultrasonic probe and the display unit, in, The processor performs: setting conditions for obtaining an ultrasonic image of a subject; According to the condition, the ultrasonic probe transmits an ultrasonic beam, the ultrasonic probe receives an echo from the subject, and generates an ultrasonic image of the subject based on the echo received by the ultrasonic probe; and creating an input image to be input to the learned model based on the ultrasonic image displayed on the display unit; Each input image produced by the processor is produced as: The length of the subject in the depth direction has a first length, and The length in a direction orthogonal to the depth direction of the subject has a second length.
2. The ultrasonic diagnostic apparatus according to claim 1, in, Producing an input image based on the ultrasonic image comprises: The ultrasonic image is preprocessed.
3. The ultrasonic diagnostic apparatus according to claim 2, in, When the length of the subject in the ultrasonic image in the depth direction is greater than a predetermined length, The pre-processing comprises: cropping a portion of the ultrasound image; and The input image is produced based on the cropped image.
4. The ultrasonic diagnostic apparatus according to claim 3, in, The pre-processing comprises: A predetermined process is performed on the cropped image to produce the input image of a desired size.
5. The ultrasonic diagnostic apparatus according to claim 4, in, The predetermined processing includes zero padding processing.
6. The ultrasonic diagnostic apparatus according to claim 2, in, When the length of the subject in the ultrasonic image in the depth direction is smaller than the predetermined length, The pre-processing comprises: Prescribed processing is performed on the ultrasonic image to create the input image of a desired size.
7. The ultrasonic diagnostic apparatus according to claim 6, in, The prescribed processing includes zero padding processing.
8. The ultrasonic diagnostic apparatus according to claim 1, in, The first length is a length between a position on the body surface of the subject and a position inside the body of the subject.
9. The ultrasonic diagnostic apparatus according to claim 1, in, The second length is equal to the first length.
10. The ultrasonic diagnostic apparatus according to claim 1, in, The learned model is produced by learning a plurality of training images through a neural network. The training images are: The length of the subject in the depth direction has a first length, and The length in a direction perpendicular to the depth direction of the subject has a second length.
11. The ultrasonic diagnostic apparatus according to claim 1, in, The processor The input image is input to the learned model, and a part included in the input image is estimated using the learned model.
12. A storage medium, storing instructions executable by a processor that communicates with an ultrasound probe, in, The instructions cause the processor to: setting conditions for obtaining an ultrasonic image of a subject; According to the condition, the ultrasonic probe transmits an ultrasonic beam, the ultrasonic probe receives an echo from the subject, and generates an ultrasonic image of the subject based on the echo received by the ultrasonic probe; and creating an input image to be input to the learned model based on the ultrasonic image displayed on the display unit; Each input image produced by the processor is produced as: The length of the subject in the depth direction has a first length, and The length in a direction perpendicular to the depth direction of the subject has a second length.
Citation Information
Patent Citations
Ultrasound diagnostic device
JP2021171559A
