Programs, ultrasound diagnostic equipment, ultrasound diagnostic systems, imaging diagnostic equipment, and training equipment.

Abstract

The objective is to provide image diagnostic technology that utilizes machine learning models. [Solution] One aspect of the present disclosure relates to a machine learning model trained using training data that includes first ultrasonic image data based on a received signal received by an ultrasonic transducer, first ground truth data which is first region information associated with a detection target of the first ultrasonic image data, and second ground truth data which is either first location information associated with a detection target of the first ultrasonic image data, or second region information based on the first location information.

Description

Technical Field

[0001] The present disclosure relates to a machine learning model, a program, an ultrasonic diagnostic apparatus, an ultrasonic diagnostic system, an image diagnostic apparatus, and a training apparatus.

Background Art

[0002] With the recent progress of deep learning technology, machine learning models have come to be used in various applications. For example, in the medical field, it has been proposed to use machine learning models for image diagnosis of ultrasonic image data.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] As indicators for evaluating cardiac function, there are measurement items such as ejection fraction (EF) of the left ventricle and diameter of the inferior vena cava (IVC), and accurate and reproducible measurements are desired. Currently, in the manual measurement of EF, EF is calculated by a user tracing the endocardium in an ultrasonic image. Also, in the manual measurement of the IVC diameter, the IVC diameter is measured by a user designating the vessel wall of the inferior vena cava based on the hepatic vein. These manual measurements are complicated and errors can occur due to user operation. Also, it is known that there are variations for each captured image and errors can occur.

[0005] A semi-automatic EF measurement method is generally known. In this semi-automatic method, the endocardium is automatically traced, but two points on the mitral annulus and one point on the apex must be specified by the user. Furthermore, in order for the user to specify the two points on the mitral annulus and one point on the apex, the ultrasound image must be frozen and these points selected on the still image. As a result, the measurement is time-consuming and cumbersome, making real-time measurement difficult.

[0006] Furthermore, unlike other organs, the heart is a part of the body that moves significantly with each beating. For example, when calculating the ejection fraction (EF) to evaluate cardiac function in real time, tracing the endocardium alone is insufficient to adequately identify the left ventricular region, requiring techniques to evaluate cardiac function with good accuracy.

[0007] In light of the above issues, one of the objectives of this disclosure is to provide image diagnostic technology that utilizes machine learning models. [Means for solving the problem]

[0008] One aspect of the present disclosure relates to a machine learning model trained using training data that includes first ultrasonic image data based on a received signal received by an ultrasonic transducer, first ground truth data which is first region information associated with a detection target of the first ultrasonic image data, and second ground truth data which is either first location information associated with a detection target of the first ultrasonic image data, or second region information based on the first location information. [Effects of the Invention]

[0009] According to this disclosure, it is possible to provide image diagnostic technology that utilizes machine learning models. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is a schematic diagram showing the training and inference processes of a machine learning model according to one embodiment of the present disclosure. [Figure 2]Figure 2A shows an example ultrasound image of the left ventricular region, and Figure 2B shows an example ultrasound image of the inferior vena cava diameter. [Figure 3] Figure 3 is a schematic diagram showing a machine learning model for the left ventricular region according to one embodiment of the present disclosure. [Figure 4] Figure 4 is a schematic diagram illustrating the training and inference processes of a machine learning model according to another embodiment of the present disclosure. [Figure 5] Figure 5 is a schematic diagram illustrating the training and inference processes of a machine learning model according to another embodiment of the present disclosure. [Figure 6] Figure 6 is a schematic diagram illustrating the training and inference processes of a machine learning model according to another embodiment of the present disclosure. [Figure 7] Figure 7 is a schematic diagram showing an ultrasound diagnostic apparatus according to one embodiment of the present disclosure. [Figure 8] Figure 8 is a block diagram showing the hardware configuration of an ultrasound diagnostic apparatus according to one embodiment of the present disclosure. [Figure 9] Figure 9 is a block diagram showing the hardware configuration of a training device and an image diagnostic device according to one embodiment of the present disclosure. [Figure 10] Figure 10 is a block diagram showing the functional configuration of a training device according to one embodiment of the present disclosure. [Figure 11] Figure 11A shows training data for the left ventricular region according to one embodiment of the present disclosure, and Figure 11B shows the ground truth data for region detection results according to one embodiment of the present disclosure. [Figure 12] Figure 12 is a schematic diagram showing the training process of a machine learning model for EF measurement according to one embodiment of the present disclosure. [Figure 13] Figure 13 is a block diagram showing the functional configuration of an ultrasound diagnostic device according to one embodiment of the present disclosure. [Figure 14] Figure 14 shows the network architecture of a machine learning model for EF measurement according to one embodiment of the present disclosure. [Figure 15] Figure 15 is a schematic diagram showing the detection of the target area for EF measurement according to one embodiment of the present disclosure. [Figure 16] Figures 16A to C are schematic diagrams showing contour extraction processing according to an embodiment of the present disclosure. [Figure 17] Figure 17A is a diagram showing training data of the inferior vena cava region according to an embodiment of the present disclosure, and Figure 17B is a diagram showing correct answer data of region detection results according to an embodiment of the present disclosure. [Figure 18] Figure 18 is a schematic diagram showing a training process of a machine learning model for IVC diameter measurement according to an embodiment of the present disclosure. [Figure 19] Figure 19 is a diagram showing a network architecture of a machine learning model for IVC diameter measurement according to an embodiment of the present disclosure. [Figure 20] Figure 20A is a diagram showing training data of the left ventricular region according to an embodiment of the present disclosure, and Figure 20B is a diagram showing correct answer data of region detection results according to an embodiment of the present disclosure. [Figure 21] Figure 21 is a schematic diagram showing a training process of a machine learning model for EF measurement according to an embodiment of the present disclosure. [Figure 22] Figure 22 is a diagram showing a network architecture of a machine learning model for EF measurement according to an embodiment of the present disclosure. [Figure 23] Figure 23 is a schematic diagram showing target region detection for EF measurement according to an embodiment of the present disclosure. [Figure 24] Figure 24A is a diagram showing training data of the inferior vena cava region according to an embodiment of the present disclosure, and Figure 24B is a diagram showing correct answer data of region detection results according to an embodiment of the present disclosure. [Figure 25] Figure 25 is a schematic diagram showing target region detection for IVC diameter measurement according to an embodiment of the present disclosure. [Figure 26] Figure 26 is a diagram showing a network architecture of a machine learning model for IVC diameter measurement according to an embodiment of the present disclosure. MODE FOR CARRYING OUT THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0012] [Summary of this disclosure] The following embodiments disclose a training device for training a machine learning model for estimating a target area in an ultrasound image, and an ultrasound diagnostic device, an imaging diagnostic device, and an ultrasound diagnostic system that use the trained machine learning model to estimate the target area and calculate indicators related to cardiac function (e.g., EF, IVC diameter, etc.).

[0013] More specifically, the machine learning model according to the embodiment described later extracts region information from the input ultrasound image that indicates a portion of the target area (e.g., the contour of the left ventricular cavity region, the inferior vena cava region, etc.) and positional information that indicates another portion of the target area (e.g., the left and right valve annulus ends, hepatic vein points, etc.). The ultrasound diagnostic device, imaging diagnostic device, and ultrasound diagnostic system estimate the target area based on the extracted region information and positional information, and calculate an index for evaluating cardiac function based on the estimated target area. The machine learning model according to this embodiment can extract the target area from a beating heart more effectively than directly extracting the target area.

[0014] [System Configuration] First, a system for realizing training and inference processing using a machine learning model according to one embodiment of this disclosure will be described. Figure 1 is a schematic diagram showing the training and inference processing of a machine learning model according to one embodiment of this disclosure.

[0015] As shown in Figure 1, the training device 50 stores the machine learning model 10 to be trained and trains the machine learning model 10 using the training data stored in the training data database (DB) 20. The machine learning model 10 to be trained may be implemented as any appropriate type of machine learning model, such as a neural network. For example, if the machine learning model 10 to be trained is implemented as a neural network, the training device 50 may perform supervised learning using the training data obtained from the training data DB 20 and update the parameters of the machine learning model 10 according to any known training algorithm, such as backpropagation.

[0016] Once the machine learning model 10 is trained, the trained machine learning model 10 is stored in the ultrasound diagnostic device 100, and the ultrasound diagnostic device 100 may use the trained machine learning model 10 to estimate the detection result of the target region from ultrasound image data acquired by sending and receiving ultrasound signals to and from the subject 30 via the ultrasound probe. For example, if the machine learning model 10 is trained to extract the left ventricular pericardium boundary and the left and right valve annular ends from cardiac ultrasound image data, the ultrasound diagnostic device 100 may extract the left ventricular pericardium boundary and the left and right valve annular ends from the ultrasound image data of the subject 30, as shown in Figure 2A. The ultrasound diagnostic device 100 can estimate the left ventricular region in real time based on the region defined by the extracted left ventricular pericardium boundary and the left and right valve annular ends, and measure the left ventricular ejection fraction (EF) from the fluctuating left ventricular region.

[0017] Alternatively, if the machine learning model 10 is trained to extract the inferior vena cava and hepatic veins from cardiac ultrasound image data, the ultrasound diagnostic device 100 may extract the inferior vena cava region and hepatic veins from the ultrasound image data of the subject 30, as shown in Figure 2B. The ultrasound diagnostic device 100 can measure the inferior vena cava (IVC) diameter defined by the extracted inferior vena cava region and hepatic veins in real time.

[0018] In one embodiment, a machine learning model 10 for detecting the left ventricular region may extract region detection results from multiple channels from ultrasound image data. In the embodiment shown in Figure 3, channel 0 indicates the left ventricular endocardium boundary, and channels 1 and 2 indicate the weighted locations of the region detection results at the left and right annular ends, respectively. The ultrasound diagnostic device 100 may extract the contour or boundary of the left ventricular region based on the detection results of these three channels. Although not shown, a machine learning model 10 for detecting the inferior vena cava and hepatic vein may similarly extract region detection results from multiple channels from ultrasound image data. For example, channel 0 may indicate the inferior vena cava region, channel 1 may indicate the location of the hepatic vein, and the ultrasound diagnostic device 100 may estimate the inferior vena cava (IVC) diameter based on the detection results of these two channels.

[0019] A system configuration according to one embodiment of the present disclosure has been described with reference to Figure 1, but the system configuration according to the present disclosure is not limited thereto. Figure 4 is a schematic diagram showing the training and inference processes of a machine learning model according to another embodiment of the present disclosure. In the illustrated embodiment, the training device 50 is the same as in the embodiment shown in Figure 1 in that it trains the machine learning model 10 to be trained, but in this embodiment, the ultrasound diagnostic system 1 has an ultrasound diagnostic device 100 and an image diagnostic device 200, and the image diagnostic device 200 stores the trained machine learning model 10 and estimates the region detection result from ultrasound image data acquired from the ultrasound diagnostic device 100. According to this embodiment, even if the ultrasound diagnostic device 100 does not have computing resources to execute the machine learning model 10, the machine learning model 10 can be executed by utilizing the more abundant computing resources of the image diagnostic device 200, which is realized by a server located on a network or the like.

[0020] Figure 5 is a schematic diagram showing the training and inference processes of a machine learning model according to another embodiment of the present disclosure. In the illustrated embodiment, the training device 50 may train the machine learning model 10 to be trained, which is stored in the model database (DB) 40, and store the trained machine learning model 10 in the model DB 40. The ultrasound diagnostic device 100 may access the model DB 40 and use the trained machine learning model 10. For example, when ultrasound image data is acquired from a patient 30, the ultrasound diagnostic device 100 may pass the acquired ultrasound image data to the trained machine learning model 10 stored in the model DB 40 and obtain region detection results from the model DB 40. According to this embodiment, even if the ultrasound diagnostic device 100 does not have storage resources to store the machine learning model 10, it can use the machine learning model 10 stored in the model DB 40.

[0021] Figure 6 is a schematic diagram showing the training and inference processes of a machine learning model according to another embodiment of the present disclosure. In the illustrated embodiment, the training device 50 trains the machine learning model 10 to be trained, which is stored in the model DB 40, and stores the trained machine learning model 10 in the model DB 40, similar to the embodiment shown in Figure 5. However, in this embodiment, the ultrasound diagnostic system 1A includes an ultrasound diagnostic device 100 and an image diagnostic device 200, and the image diagnostic device 200 may pass ultrasound image data acquired from the ultrasound diagnostic device 100 to the trained machine learning model 10 in the model DB 40 and obtain region detection results from the model DB 40. According to this embodiment, even if the ultrasound diagnostic device 100 does not have computing resources to execute the machine learning model 10, and / or does not have storage resources to store the machine learning model 10, the machine learning model 10 can be used by utilizing the more abundant computing and storage resources of the model DB 40 and the image diagnostic device 200, which are implemented by a server located on a network or the like.

[0022] [Hardware configuration of ultrasound diagnostic equipment] Figure 7 shows an example of the external appearance of the ultrasound diagnostic device 100. Figure 8 is a block diagram showing an example of the configuration of the main parts of the control system of the ultrasound diagnostic device 100.

[0023] The ultrasound diagnostic device 100 visualizes the shape or movement within the subject 30 as an ultrasound image. The ultrasound diagnostic device 100 according to this embodiment is used, for example, to capture an ultrasound image (i.e., a tomographic image) of the area to be detected and to perform an examination of the area to be detected.

[0024] As shown in Figure 7, the ultrasound diagnostic device 100 comprises an ultrasound diagnostic device body 1010 and an ultrasound probe 1020. The ultrasound diagnostic device body 1010 and the ultrasound probe 1020 are connected via a cable 1030.

[0025] The ultrasonic probe (ultrasonic transducer) 1020 transmits an ultrasonic beam (for example, about 1 to 30 MHz) into the subject 30 (for example, the human body), and also functions as an acoustic sensor that receives ultrasonic echoes reflected from the transmitted ultrasonic beam within the subject 30 and converts them into electrical signals.

[0026] The user operates the ultrasound diagnostic device 100 by bringing the ultrasound beam transmitting and receiving surface of the ultrasound probe 1020 into contact with the body surface of the area to be detected on the subject 30, and performs the examination. Any type of ultrasound probe, such as a convex probe, linear probe, sector probe, or three-dimensional probe, can be used for the ultrasound probe 1020.

[0027] The ultrasonic probe 1020 is configured to include, for example, a plurality of transducers (e.g., piezoelectric elements) arranged in a matrix, and a channel switching device (e.g., a multiplexer) for switching the on / off state of the plurality of transducers individually or in block units (hereinafter referred to as "channels").

[0028] Each transducer of the ultrasound probe 1020 converts a voltage pulse generated by the ultrasound diagnostic device main unit 1010 (transmitter 1012) into an ultrasound beam and transmits it into the patient 30. It then receives the ultrasound echo reflected within the patient 30, converts it into an electrical signal (hereinafter referred to as the "received signal"), and outputs it to the ultrasound diagnostic device main unit 1010 (receiver 1013).

[0029] As shown in Figure 8, the ultrasound diagnostic device main unit 1010 includes an operation input unit 1011, a transmission unit 1012, a reception unit 1013, an ultrasound image generation unit 1014, a display image generation unit 1015, an output unit 1016, and a control unit 1017.

[0030] The transmitting unit 1012, receiving unit 1013, ultrasonic image generation unit 1014, and display image generation unit 1015 are composed of dedicated or general-purpose hardware (electronic circuits) such as DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), and PLD (Programmable Logic Device), which are appropriate for each process, and work in cooperation with the control unit 1017 to realize each function.

[0031] The operation input unit 1011 receives input such as commands to instruct the start of diagnosis or information about the subject 30. The operation input unit 1011 may have, for example, an operation panel with multiple input switches, a keyboard, and a mouse. The operation input unit 1011 may also be a touch panel provided integrally with the output unit 1016.

[0032] The transmitting unit 1012 is a transmitter that sends voltage pulses as drive signals to the ultrasonic probe 1020 in accordance with the instructions of the control unit 1017. The transmitting unit 1012 may include, for example, a high-frequency pulse oscillator and a pulse setting unit. The transmitting unit 1012 may adjust the voltage pulses generated by the high-frequency pulse oscillator to the voltage amplitude, pulse width, and transmission timing set by the pulse setting unit and send them to each channel of the ultrasonic probe 1020.

[0033] The transmitting unit 1012 has a pulse setting unit for each of the multiple channels of the ultrasonic probe 1020, and the voltage amplitude, pulse width, and transmission timing of the voltage pulse can be set for each of the multiple channels. For example, the transmitting unit 1012 may change the target depth or generate different pulse waveforms by setting an appropriate delay time for the multiple channels.

[0034] The receiving unit 1013 is a receiver that receives and processes the received signal relating to the ultrasonic echo generated by the ultrasonic probe 1020, in accordance with the instructions of the control unit 1017. The receiving unit 1013 may include a preamplifier, an AD converter, and a receiving beamformer.

[0035] The receiver 1013 uses a preamplifier to amplify the received signals related to the weak ultrasonic echo for each channel, and then uses an AD converter to convert the received signals into digital signals. The receiver 1013 then uses a receiving beamformer to combine the received signals of multiple channels into a single signal, which can then be converted into acoustic line data.

[0036] The ultrasound image generation unit 1014 acquires a received signal (acoustic line data) from the receiving unit 1013 and generates an ultrasound image (i.e., a tomographic image) of the inside of the subject 30.

[0037] The ultrasound image generation unit 1014, for example, continuously stores the signal intensity of the ultrasound echo detected after the ultrasound probe 1020 transmits a pulsed ultrasound beam in the depth direction in line memory over time. Then, as the ultrasound beam from the ultrasound probe 1020 scans inside the patient 30, the ultrasound image generation unit 1014 sequentially stores the signal intensity of the ultrasound echo at each scanning position in line memory, generating two-dimensional data in frame units. The ultrasound image generation unit 1014 can then generate an ultrasound image representing the two-dimensional structure in a cross-section including the ultrasound transmission direction and the ultrasound scanning direction by converting the signal intensity of the two-dimensional data into brightness values.

[0038] The ultrasonic image generation unit 1014 may also include, for example, an envelope detection circuit that performs envelope detection on the received signal acquired from the receiving unit 1013, a logarithmic compression circuit that performs logarithmic compression on the signal intensity of the received signal detected by the envelope detection circuit, and a dynamic filter that removes noise components contained in the received signal using a bandpass filter whose frequency characteristics are changed according to the depth.

[0039] The display image generation unit 1015 acquires ultrasound image data from the ultrasound image generation unit 1014 and generates a display image that includes the display area of ​​the ultrasound image. The display image generation unit 1015 then sends the generated display image data to the output unit 1016. The display image generation unit 1015 may sequentially update the display image each time it acquires a new ultrasound image from the ultrasound image generation unit 1014 and display the display image in video format on the output unit 1016.

[0040] Furthermore, the display image generation unit 1015 may, in accordance with the instructions of the control unit 1017, generate a display image in which a graphic representation of the time-series data of the detected object is embedded within the display area along with the ultrasonic image.

[0041] The display image generation unit 1015 may perform predetermined image processing, such as coordinate transformation processing or data interpolation processing, on the ultrasound image output from the ultrasound image generation unit 1014 before generating the display image.

[0042] The output unit 1016 acquires display image data from the display image generation unit 1015 in accordance with the instructions of the control unit 1017 and outputs the display image. For example, the output unit 1016 may be composed of a liquid crystal display, an organic EL display, or a CRT display, and may display the display image.

[0043] The control unit 1017 controls the entire ultrasound diagnostic device 100 by controlling the operation input unit 1011, the transmission unit 1012, the reception unit 1013, the ultrasound image generation unit 1014, the display image generation unit 1015, and the output unit 1016 according to their respective functions.

[0044] The control unit 1017 may include a CPU (Central Processing Unit) 1171 as an arithmetic / control device, a ROM (Read Only Memory) 1172 and a RAM (Random Access Memory) 1173 as main memory. The ROM 1172 stores basic programs and basic setting data. The CPU 1171 reads a program from the ROM 172 according to the processing content, stores it in the RAM 1173, and executes the stored program to centrally control the operation of each functional block of the ultrasound diagnostic device main unit 1010 (transmitter 1012, receiver 1013, ultrasound image generation unit 1014, display image generation unit 1015, and output unit 1016).

[0045] [Hardware configuration of training equipment and image processing equipment] Next, with reference to Figure 9, the hardware configuration of the training device 50 and image processing device 200 according to one embodiment of the present disclosure will be described. Figure 9 is a block diagram showing the hardware configuration of the training device 50 and image processing device 200 according to one embodiment of the present disclosure.

[0046] The training device 50 and the image processing device 200 may each be implemented by computing devices such as a server, personal computer (PC), smartphone, or tablet, and may have a hardware configuration such as that shown in Figure 9. Specifically, the training device 50 and the image processing device 200 each have a drive device 101, a storage device 102, a memory device 103, a processor 104, a user interface (UI) device 105, and a communication device 106, which are interconnected via bus B.

[0047] The programs or instructions that implement the various functions and processes described later in the training device 50 and the image processing device 200 may be stored on a removable storage medium such as a CD-ROM (Compact Disk-Read Only Memory) or flash memory. When the storage medium is set in the drive device 101, the programs or instructions are installed from the storage medium to the storage device 102 or memory device 103 via the drive device 101. However, the programs or instructions do not necessarily have to be installed from the storage medium; they may also be downloaded from any external device via a network or the like.

[0048] The storage device 102 is implemented by a hard disk drive or the like, and stores files, data, etc., used to execute the installed program or instructions, along with the installed program or instructions.

[0049] The memory device 103 is implemented using random access memory, static memory, etc., and when a program or instruction is activated, it reads the program or instruction, data, etc. from the storage device 102 and stores it. The storage device 102, the memory device 103, and the removable storage medium may be collectively referred to as a non-transitory storage medium.

[0050] The processor 104 may be implemented by one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), processing circuits, etc., which may consist of one or more processor cores, and executes various functions and processes of the training device 50 and image processing device 200, which will be described later, according to data such as programs, instructions, and parameters necessary to execute the programs or instructions stored in the memory device 103.

[0051] The user interface (UI) device 105 may consist of input devices such as a keyboard, mouse, camera, and microphone, output devices such as a display, speaker, headset, and printer, and input / output devices such as a touch panel, and realizes an interface between the user and the training device 50 and the image processing device 200. For example, the user operates the training device 50 and the image processing device 200 by operating a GUI (Graphical User Interface) displayed on the display or touch panel with a keyboard, mouse, etc.

[0052] The communication device 106 is implemented by various communication circuits that perform wired and / or wireless communication processing with external devices, the Internet, LAN (Local Area Network), cellular networks, and other communication networks.

[0053] However, the hardware configuration described above is merely an example, and the training device 50 and image processing device 200 according to this disclosure may be implemented by any other suitable hardware configuration.

[0054] [First Embodiment] Next, a training device 50 according to one embodiment of the present disclosure will be described. This embodiment will be described with a focus on a machine learning model 10 that is trained to detect the left ventricular region for EF measurement. Figure 10 is a block diagram showing the functional configuration of the training device 50 according to one embodiment of the present disclosure. As shown in Figure 10, the training device 50 has a data acquisition unit 51 and a training unit 52.

[0055] The data acquisition unit 51 acquires training data for the machine learning model 10 to be trained. Specifically, the data acquisition unit 51 acquires training data that includes ultrasound image data and ground truth data consisting of region information associated with the detection target of the ultrasound image data, and / or location information associated with the detection target of the ultrasound image data or region information based on said location information.

[0056] For example, the data acquisition unit 51 may acquire ultrasound image data showing the heart, as shown in Figure 11A, from the training data DB 20 as training data to be input to the machine learning model 10. The data acquisition unit 51 may also acquire data showing the left ventricular region in the ultrasound image data, as shown in Figure 11B, and coordinates indicating the positions of the left and right valve annular ends, from the training data DB 20 as output training data to be output from the machine learning model 10. The data showing the left ventricular region is a dataset associated with confidence information (for example, a numerical value indicating the confidence that the central pixel block of the ultrasound image data corresponds to the left ventricular region). A pixel block is one of the divided regions when an image is divided into multiple regions, and may consist of a group of multiple pixels or a single pixel. If necessary, the data acquisition unit 51 may preprocess the acquired training ultrasound image data. This preprocessing may include, for example, contrast adjustment and noise reduction.

[0057] Furthermore, the data acquisition unit 51 may perform data augmentation on the training data acquired from the training data DB 20 to increase the amount of training data. For example, the data acquisition unit 51 may perform enlargement, reduction, repositioning, deformation, etc. on the training ultrasound image data acquired from the training data DB 20.

[0058] The training unit 52 compares the output results of the machine learning model 10 being trained, which include data showing the left ventricular region and coordinates indicating the positions of the left and right annular valvular ends, with the ground truth data, and updates the parameters of the machine learning model 10 according to the error between the output results and the ground truth data. As shown in Figure 12, the training unit 52 may also input training ultrasound image data to the machine learning model 10 and obtain data showing the left ventricular region in the ultrasound image data and coordinates indicating the positions of the left and right annular valvular ends from the machine learning model 10. For example, in the example shown in Figure 12, the machine learning model 10 being trained outputs data showing the left ventricular region as shown and coordinates (100,115) and (155,95) indicating the positions of the left and right annular valvular ends. The training unit 52 compares the detection result with the ground truth data of the left ventricular region in the input training ultrasound image data and the coordinates (110,100) and (160,100) indicating the positions of the left and right annular ends, and updates the parameters of the machine learning model 10 according to the error between the detection result and the ground truth data. For example, if the machine learning model 10 is implemented by a convolutional neural network, the training unit 52 may continue to adjust the parameters of the machine learning model 10 according to the error between the output result and the ground truth data according to the backpropagation method until a predetermined termination condition is met.

[0059] Once the training of the machine learning model 10 is completed in this manner, the trained machine learning model 10 may be provided to the ultrasound diagnostic device 100. Alternatively, the trained machine learning model 10 may be provided to the model DB 40 and / or the image diagnostic device 200.

[0060] Next, an ultrasound diagnostic apparatus 100 according to one embodiment of the present disclosure will be described. The ultrasound diagnostic apparatus 100 uses a machine learning model 10 trained by a training device 50 to perform ultrasound diagnosis based on ultrasound signals transmitted to and received from a subject 30. Figure 13 is a block diagram showing the functional configuration of the ultrasound diagnostic apparatus 100 according to one embodiment of the present disclosure. As shown in Figure 13, the ultrasound diagnostic apparatus 100 has a data acquisition unit 110 and an inference unit 120.

[0061] The data acquisition unit 110 acquires ultrasound image data to be inferred. Specifically, the data acquisition unit 110 acquires ultrasound image data generated based on the received signal received from the subject 30 by the ultrasound transducer 1120. If necessary, the data acquisition unit 110 may perform preprocessing on the acquired ultrasound image data, such as noise suppression, contrast normalization, and image resizing, for input into the trained machine learning model 10.

[0062] The inference unit 120 inputs the ultrasound image data to be inferred into the trained machine learning model 10 and obtains the inference result. Specifically, the inference unit 120 inputs the ultrasound image data to be inferred into the trained machine learning model 10 and obtains data indicating the contour of the left ventricular region and coordinates indicating the positions of the left and right annular valvular ends from the machine learning model 10. For example, if the trained machine learning model 10 is implemented as a U-net type convolutional neural network as shown in Figure 14, the inference unit 120 inputs the ultrasound image data to be inferred into the input layer of the convolutional neural network and obtains data indicating the left ventricular region and coordinates indicating the positions of the left and right annular valvular ends from the output layer. In the illustrated example, data indicating the left ventricular region and the coordinates of the left annular valvular end (110,100) and the right annular valvular end (160,100) are output.

[0063] The inference unit 120 determines the measurement target area based on the data representing the left ventricular region estimated by the machine learning model 10, and the coordinates of the left and right annular ends. Specifically, when data representing the left ventricular region and the coordinates of the left and right annular ends (110, 100) are obtained from the machine learning model 10, the inference unit 120 derives a straight line connecting the left and right annular ends, as shown in Figure 15, and superimposes this straight line onto the depicted left ventricular region. The inference unit 120 then determines the area enclosed by the depicted left ventricular region and the straight line as the measurement target area, and can estimate the left ventricular ejection fraction based on the volume of this measurement target area.

[0064] Here, the data showing the contour of the left ventricular region may be obtained by following a procedure such as those shown in Figures 16A-C. That is, as shown in Figure 16A, the centroid of the left ventricular region is first determined from the data showing the left ventricular region, which is the output result from the machine learning model 10, and contour points are searched outward from the determined centroid. Next, as shown in Figure 16B, the point where the confidence level of the output result first falls below a threshold may be determined as a contour point. This process is carried out by changing the angle of the search line extending from the centroid in various ways, as shown in Figure 16C, thereby determining contour points on each search line. Then, these contour point data are spline interpolated to obtain the contour line. Furthermore, the volume of the region to be measured may be estimated based on the contour line determined from the data showing the left ventricular region in this way. For example, the volume of the region to be measured may be derived according to the Modified Simpson method (disk method). Specifically, the long axis (L) of two cross-sections of the apical 2-chamber or 4-chamber image is divided equally into 20 discs, and the inner diameter of the short axis (a) perpendicular to the long axis of each disc is divided. i and b i The volume is calculated from the sum of the cross-sectional areas of the disks. Assuming each disk is elliptical, the left intracellular cavity area (V) is determined. That is, the left intracellular cavity area (V) of the object being measured is

number

[0065] The machine learning model 10, which estimates the detection target area and coordinates of a specific location as described above, is not limited to EF measurement and may also be used for IVC diameter measurement. First, regarding the training process, for example, the data acquisition unit 51 may acquire ultrasound image data showing the inferior vena cava, as shown in Figure 17A, from the training data DB 20 as training data to be input to the machine learning model 10. In addition, the data acquisition unit 51 may acquire data showing the area of ​​the inferior vena cava region in the ultrasound image data and coordinates showing the location of the hepatic veins, as training data to be output from the machine learning model 10, as shown in Figure 17B, from the training data DB 20. The data acquisition unit 51 may, if necessary, perform preprocessing and / or data augmentation on the acquired training ultrasound image data.

[0066] The training unit 52 may input training ultrasound image data to the machine learning model 10 and obtain data indicating the inferior vena cava region and coordinates indicating the location of the hepatic vein from the machine learning model 10. For example, as shown in Figure 18, training ultrasound image data stored in the training data DB 20 may be input to the machine learning model 10 to be trained, and data indicating the inferior vena cava region and coordinates (120, 115) indicating the location of the hepatic vein may be output as detection results. The training unit 52 compares the detection result with the ground truth data of the inferior vena cava region and coordinates (130, 110) indicating the location of the hepatic vein in the input training ultrasound image data, and updates the parameters of the machine learning model 10 according to the error between the detection result and the ground truth data.

[0067] For example, if the machine learning model 10 is implemented by a convolutional neural network, the training unit 52 may continue to adjust the parameters of the machine learning model 10 according to the error between the output result and the ground truth data in accordance with the backpropagation method until a predetermined termination condition is met. Once the training of the machine learning model 10 is completed in this way, the trained machine learning model 10 may be provided to the ultrasound diagnostic device 100. Alternatively, the trained machine learning model 10 may be provided to the model DB 40 and / or the image diagnostic device 200.

[0068] Next, regarding the inference process, the data acquisition unit 110 acquires ultrasound image data generated based on the received signal received from the subject 30 by the ultrasound probe 1120. If necessary, the data acquisition unit 110 may perform preprocessing on the acquired ultrasound image data for input into the trained machine learning model 10. The inference unit 120 inputs the ultrasound image data to be inferred into the trained machine learning model 10 and obtains data indicating the inferior vena cava region and coordinates indicating the location of the hepatic vein from the machine learning model 10. For example, if the trained machine learning model 10 is implemented as a U-net type convolutional neural network as shown in Figure 19, the inference unit 120 inputs the ultrasound image data to be inferred into the input layer of the convolutional neural network and obtains data indicating the inferior vena cava region and coordinates indicating the location of the hepatic vein from the output layer. In the illustrated example, data indicating the inferior vena cava region and the coordinates of the hepatic vein (110,100) are output. The inference unit 120 can then estimate the IVC diameter from a region determined based on the inferior vena cava region data estimated by the machine learning model 10 and coordinates indicating the location of the hepatic veins. Specifically, the inference unit 120 sets a measurement point search line based on a preset distance from the hepatic vein location, finds two intersections between the measurement point search line and the estimated inferior vena cava region, and estimates the distance between these intersections as the IVC diameter. The inference unit 120 may also find the distances between intersections corresponding to multiple measurement point search lines at different angles, and use the distance between the intersections with the minimum distance as the IVC diameter. Alternatively, the inference unit 120 may determine the angle of the measurement point search line based on the inferior vena cava region data and estimate the IVC diameter from the determined measurement point search line.

[0069] [Second Example] Next, a target region detection process using the machine learning model 10 according to the second embodiment of this disclosure will be described. The machine learning model 10 according to the first embodiment, upon receiving ultrasound image data, detected data indicating the left ventricular region or inferior vena cava in the ultrasound image data, and coordinates indicating the location of the left and right valve annulus ends or hepatic veins. On the other hand, the machine learning model 10 according to the second embodiment, upon receiving ultrasound image data, detects data indicating the left ventricular region or inferior vena cava in the ultrasound image data, and data indicating the region where the left and right valve annulus ends or hepatic veins exist. For example, the data indicating such a region may be in the form of a heatmap representing the confidence level of the location of the target to be detected. The heatmap data may be in the form of data indicating the confidence level or probability that the target to be detected exists at each location on the map.

[0070] Regarding the training process, the data acquisition unit 51 may acquire ultrasound image data showing the heart, as shown in Figure 20A, from the training data DB 20 as training data to be input to the machine learning model 10. The data acquisition unit 51 may also acquire data showing the left ventricular region in the ultrasound image data, as shown in Figure 20B, and heatmap data showing the confidence level of the positions of the left and right valve annular ends, from the training data DB 20 as output training data to be output from the machine learning model 10. If necessary, the data acquisition unit 51 may preprocess the acquired training ultrasound image data. Such preprocessing may include, for example, contrast adjustment and noise reduction.

[0071] Furthermore, the data acquisition unit 51 may perform data augmentation on the training data acquired from the training data DB 20 to increase the amount of training data. For example, the data acquisition unit 51 may perform scaling, repositioning, deformation, etc. on the training ultrasound image data acquired from the training data DB 20. In addition, the data acquisition unit 51 may modify the heatmap data representing the confidence level of the detection target's position after data augmentation, within a range where the peak position of the confidence level does not change. Specifically, in the case of a heatmap corresponding to the distance from the peak position of the confidence level, the data acquisition unit 51 modifies the heatmap that has changed due to data augmentation to a heatmap corresponding to the distance before data augmentation and the reference distance. This makes it possible to make the confidence level value of the heatmap reflect the distance from the detection target's position.

[0072] The training unit 52 trains the machine learning model 10 to be trained using training data. Specifically, the training unit 52 may input training ultrasound image data to the machine learning model 10 and obtain data showing the left ventricular region in the ultrasound image data and heatmap data showing the confidence level of the positions of the left and right annular valvular ends from the machine learning model 10. As shown in Figure 21, the training unit 52 compares the left ventricular region data and the data showing the confidence level of the positions of the left and right annular valvular ends output from the machine learning model 10 to be trained with the ground truth data and updates the parameters of the machine learning model 10 according to the error between the output result and the ground truth data. For example, if the machine learning model 10 is implemented by a convolutional neural network, the training unit 52 may continue to adjust the parameters of the machine learning model 10 according to the error between the output result and the ground truth data according to backpropagation until a predetermined termination condition is met.

[0073] Once the training of the machine learning model 10 is completed in this manner, the trained machine learning model 10 may be provided to the ultrasound diagnostic device 100. Alternatively, the trained machine learning model 10 may be provided to the model DB 40 and / or the image diagnostic device 200.

[0074] For the inference process, the data acquisition unit 110 acquires ultrasound image data generated based on the received signal received from the subject 30 by the ultrasound probe 1120. The data acquisition unit 110 may perform necessary preprocessing on the acquired ultrasound image data, such as noise suppression, contrast normalization, and image resizing, for input to the trained machine learning model 10.

[0075] The inference unit 120 inputs the ultrasound image data to be inferred into the trained machine learning model 10 and obtains left ventricular region data and heatmap data indicating the confidence level of the positions of the left and right valve annulus from the machine learning model 10. For example, if the trained machine learning model 10 is implemented as a U-net type convolutional neural network as shown in Figure 22, the inference unit 120 inputs the ultrasound image data to be inferred into the input layer of the convolutional neural network and obtains left ventricular region data and heatmap data indicating the confidence level of the positions of the left and right valve annulus from the output layer.

[0076] The inference unit 120 determines the measurement target area based on data representing the left ventricular region estimated by the machine learning model 10 and heatmap data indicating the confidence level of the positions of the left and right annular valvular ends. Specifically, once data representing the left ventricular region and heatmap data indicating the confidence level of the positions of the left and right annular valvular ends are obtained from the machine learning model 10, as shown in Figure 23, the inference unit 120 derives a straight line connecting the positions with the highest confidence level between the left and right annular valvular ends in the heatmap data and superimposes this straight line onto the depicted left ventricular region. The inference unit 120 then determines the area surrounded by the depicted left ventricular region and the straight line as the measurement target area and can estimate the left ventricular ejection fraction based on the volume of the measurement target area. Alternatively, the volume of the measurement target area may be estimated based on the contour line determined from the estimated data representing the left ventricular region.

[0077] Here, data showing the contour of the left ventricular region may be obtained by following the procedure below. That is, the centroid of the left ventricular region is first determined from the data showing the left ventricular region, which is the output result from the machine learning model 10, and contour points are searched outward from the determined centroid. Next, the point where the confidence level of the output result first falls below a threshold may be determined as a contour point. This process determines contour points on each search line by changing the angle of the search line extending from the centroid. Then, these contour point data are spline interpolated to obtain the contour line. In addition, the volume of the region to be measured may be estimated based on the contour line determined from the data showing the left ventricular region in this way. For example, the volume of the region to be measured may be derived according to the Modified Simpson method (disk method). Specifically, the long axis (L) of the two cross-sections of the apical 2-chamber or 4-chamber image is divided equally into 20 disks, and the inner diameter of the short axis (a) perpendicular to the long axis of each disk is divided. i and b i The volume is calculated from the sum of the cross-sectional areas of the disks. Assuming each disk is elliptical, the left intracellular cavity area (V) is determined. That is, the left intracellular cavity area (V) of the object being measured is

number

[0078] The machine learning model 10, which estimates the detection target area and the confidence level regarding a specific location as described above, is not limited to EF measurement and may also be used for IVC diameter measurement. First, regarding the training process, for example, the data acquisition unit 51 may acquire ultrasound image data showing the inferior vena cava, as shown in Figure 24A, from the training data DB 20 as training data to be input to the machine learning model 10. In addition, the data acquisition unit 51 may acquire data showing the inferior vena cava region in the ultrasound image data and heatmap data showing the confidence level of the hepatic vein location, as output training data to be output from the machine learning model 10, as shown in Figure 24B, from the training data DB 20. The data acquisition unit 51 may, if necessary, perform preprocessing and / or data augmentation on the acquired training ultrasound image data.

[0079] The training unit 52 may input training ultrasound image data into the machine learning model 10 and obtain data indicating the inferior vena cava region in the ultrasound image data and heatmap data indicating the confidence level of the hepatic vein location from the machine learning model 10. The training unit 52 compares the detection result with the ground truth data of the inferior vena cava region and the heatmap data indicating the confidence level of the hepatic vein location in the input training ultrasound image data, and updates the parameters of the machine learning model 10 according to the error between the detection result and the ground truth data.

[0080] For example, if the machine learning model 10 is implemented by a convolutional neural network, the training unit 52 may continue to adjust the parameters of the machine learning model 10 according to the error between the output result and the ground truth data in accordance with the backpropagation method until a predetermined termination condition is met. Once the training of the machine learning model 10 is completed in this way, the trained machine learning model 10 may be provided to the ultrasound diagnostic device 100. Alternatively, the trained machine learning model 10 may be provided to the model DB 40 and / or the image diagnostic device 200.

[0081] Next, regarding the inference process, the data acquisition unit 110 acquires ultrasound image data generated based on the received signal received from the subject 30 by the ultrasound probe 1120. If necessary, the data acquisition unit 110 may perform preprocessing on the acquired ultrasound image data for input into the trained machine learning model 10. As shown in Figure 25, the inference unit 120 inputs the ultrasound image data to be inferred into the trained machine learning model 10 and acquires an image showing the inferior vena cava region and heatmap data showing the confidence level of the hepatic vein location from the machine learning model 10. For example, if the trained machine learning model 10 is implemented as a U-net type convolutional neural network as shown in Figure 26, the inference unit 120 inputs the ultrasound image data to be inferred into the input layer of the convolutional neural network and acquires an image showing the inferior vena cava region and heatmap data showing the confidence level of the hepatic vein location from the output layer. The inference unit 120 can then estimate the IVC diameter from a region defined based on the image of the inferior vena cava region estimated by the machine learning model 10 and the coordinates showing the highest confidence level of the hepatic vein location.

[0082] According to the embodiment described above, the ultrasound diagnostic apparatus 100 may be configured to include an ultrasound transducer 1120 that transmits and receives ultrasound to and from a subject 30, and an output means that uses a machine learning model 10 to output inference results associated with the detection target from ultrasound image data based on the received signal received by the ultrasound transducer 1120.

[0083] Furthermore, the ultrasound diagnostic device 100 may be configured to have output means that, using a machine learning model 10, outputs a detection region associated with the target to be detected as a first inference result from ultrasound image data based on the received signal received by the ultrasound transducer 1120, outputs a detection position associated with the target to be detected as a second inference result, and outputs a detection result associated with the target to be detected as a third inference result based on the detection region and the detection position.

[0084] Furthermore, the ultrasound diagnostic device 100 may be configured to include a confidence level generation means that generates a confidence level associated with the detection target from ultrasound image data based on the received signal received by the ultrasound transducer 1120 using a machine learning model 10, and a position information acquisition means that obtains the coordinates of the maximum confidence level based on the confidence level. In addition, the ultrasound diagnostic device 100 may be configured to include a shape recognition means that recognizes the shape of the detection target based on the coordinates of the maximum confidence level, and an output means that outputs information about the shape of the detection target.

[0085] According to the embodiment described above, the ultrasound diagnostic system 1 may be configured to include a measurement position determination means for determining the measurement position of a detection target based on the coordinates of the maximum confidence value, a measurement means for measuring the detection target based on the measurement position, and an output means for outputting measurement information of the measured detection target.

[0086] According to the embodiment described above, the machine learning model 10 may be trained using training data that includes ultrasound image data based on the received signal received by the ultrasound probe, region information associated with the detection target of the ultrasound image data (e.g., left ventricular region, inferior vena cava, etc.), and position information associated with the detection target of the ultrasound image data (e.g., coordinates of the left and right valve annular ends, coordinates of the hepatic vein, etc.), or region information based on position information (e.g., heatmap data of the left and right valve annular ends, heatmap data of the hepatic vein, etc.). Here, the region information based on position information is not limited to heatmap data, but may be any type of data including distance from the position coordinates of the detection target and confidence level. The region information may also be image data.

[0087] Although embodiments of this disclosure have been described in detail above, this disclosure is not limited to the specific embodiments described above, and various modifications and changes are possible within the scope of the gist of this disclosure as described in the claims. [Explanation of Symbols]

[0088] 1. Ultrasound diagnostic system 10 Machine Learning Models 20. Training data database (DB) 30 subjects 40 Model Database (DB) 50 training equipment 100 Ultrasound diagnostic equipment 200 imaging diagnostic devices

Claims

1. A first ultrasonic image data based on the received signal received by the ultrasonic probe, The first ground truth data is first region information associated with the detection target of the first ultrasonic image data, The first ultrasound image data is a second ground truth data which is either first location information associated with the detection target, or second region information based on the first location information, A machine learning model trained using training data that includes [specific data].

2. The machine learning model according to claim 1, wherein the second correct data is the second region information.

3. The machine learning model according to claim 2, wherein the second region information includes information such as the distance from the first position coordinates associated with the detection target and a degree of confidence.

4. The machine learning model according to claim 1, wherein the first region information and the second region information are image data.

5. The machine learning model according to claim 3, wherein the second region information is first heatmap information obtained by converting information including the distance from a first position coordinate associated with the detection target and the confidence level into a heatmap.

6. The machine learning model according to claim 1, wherein the training data further includes a third ground truth data which is a second location information associated with the detection target of the first ultrasonic image data, or a third region information based on the second location information.

7. The machine learning model according to claim 1, wherein the machine learning model is composed of a convolutional neural network.

8. A program that enables a computer to implement an output function that outputs an inference result associated with the detection target from a second ultrasonic image data based on a received signal received by an ultrasonic probe, using a machine learning model according to any one of claims 1 to 7.

9. An ultrasonic probe that transmits and receives ultrasound waves to and from a subject, An inference unit that uses a machine learning model according to any one of claims 1 to 7 to output an inference result associated with the detection target from a second ultrasonic image data based on a received signal received by the ultrasonic probe, An ultrasound diagnostic device.

10. An ultrasonic diagnostic apparatus having an inference unit that, using a predetermined machine learning model, outputs a detection region associated with a detection target as a first inference result from a second ultrasonic image data based on a received signal received by the ultrasonic transducer, outputs a detection position associated with the detection target as a second inference result, and outputs a detection result associated with the detection target as a third inference result based on the detection region and the detection position.

11. An inference unit that uses a predetermined machine learning model to generate a confidence score associated with the detection target from a second ultrasonic image data based on the received signal from the ultrasonic probe, and obtains the coordinates of the maximum confidence score based on the confidence score. An ultrasound diagnostic device.

12. The ultrasonic diagnostic apparatus according to claim 11, wherein the inference unit recognizes the shape of the object to be detected based on the coordinates of the maximum confidence value and outputs information about the shape of the object to be detected.

13. The ultrasonic diagnostic apparatus according to claim 11, wherein the inference unit determines the measurement position of the detection target based on the coordinates of the maximum confidence value, measures the detection target based on the measurement position, and outputs the measurement information of the measured detection target.

14. An ultrasonic probe that transmits and receives ultrasound waves to and from a subject, An output unit that outputs an inference result associated with the detection target from a second ultrasonic image data based on a received signal received by the ultrasonic probe, using a machine learning model according to any one of claims 1 to 7, An ultrasound diagnostic system.

15. An image diagnostic apparatus having an inference unit that outputs an inference result associated with the detection target from a second ultrasonic image data based on a received signal received by an ultrasonic transducer, using a machine learning model according to any one of claims 1 to 7.

16. A first ultrasonic image data based on the received signal received by the ultrasonic probe, The first ground truth data is first region information associated with the detection target of the first ultrasonic image data, The first ultrasound image data is a second ground truth data which is either first location information associated with the detection target, or second region information based on the first location information, A training device that performs machine learning using training data that includes the following.

Images