Ultrasonic time series data processing device and ultrasonic time series data processing program

Abstract

To improve prediction accuracy of a feature shown by an ultrasonic image.SOLUTION: By repeating transmission / reception of an ultrasonic wave with respect to the same scan surface set in a target subject, target time-series data in a time-series is acquired in a reception unit 16, an image generation unit 18, a tracking processing unit 20, an elasto-processing unit 22 or a Doppler processing unit 24. A feature prediction unit 48 inputs the target time-series data to a learning machine that is learned so as to predict and output the feature shown by the time-series data on the basis of the input time-series data, and predicts the feature shown by the target time-series data on the basis of the output data of the learning machine with respect to the target time-series data. A display control unit 26 notifies the user of the prediction result by the feature prediction unit 48.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This specification discloses an ultrasound time-series data processing device and an ultrasound time-series data processing program. [Background technology]

[0002] Conventionally, an ultrasound diagnostic device transmits and receives ultrasound waves toward a subject, and generates an ultrasound image based on the resulting frame data (multiple ultrasound beam data corresponding to one ultrasound image). Conventionally, technologies have been proposed for automatically diagnosing diseases displayed in ultrasound images.

[0003] Patent Document 1 discloses an ultrasound image diagnosis support method that uses ultrasound images (still images) showing a tumor as a disease and other ultrasound images (still images) as training data to train a model to identify a tumor in an input ultrasound image, inputs an ultrasound image to be determined into the trained model, and identifies a tumor in the ultrasound image to be determined. In particular, the ultrasound image diagnosis support method disclosed in Patent Document 1, based on the knowledge that speckle noise and the like occurs singly, treats a tumor candidate region occurring at the same position in consecutive frames, which are ultrasound images that are consecutive in time series to the ultrasound image to be determined, as the final tumor region. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2018 / 180386 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, as in Patent Document 1, by inputting an ultrasound image into a trained model (learning machine), it becomes possible to predict the features shown by the ultrasound image. However, there is room for improvement in predicting the features of ultrasound images using a learning machine. For example, if a learning machine could predict the features of ultrasound images while taking into account the movement of the subject represented by a time-series sequence of ultrasound images (i.e., video images), it is expected that the prediction accuracy would be improved.

[0006] An object of the ultrasound time-series data processing device disclosed in this specification is to improve the prediction accuracy of features indicated by ultrasound images. [Means for solving the problem]

[0007] The ultrasound time series data processing device disclosed in the present specification is characterized in that it comprises: a feature prediction unit that inputs target time series data, which is time series data generated by repeatedly transmitting and receiving ultrasound waves multiple times for the same scanning plane set within a target subject, to a feature prediction learner that is trained to predict and output features related to the time series data based on the input time series data, and predicts features related to the target time series data based on the output of the feature prediction learner in response to the input; and a notification unit that notifies a user of the prediction result of the feature prediction unit. In addition, the feature may be at least one of the type of examination performed using the time series data, the type of tissue cross section indicated by the time series data, the measurement position to be set, the disease indicated by the time series data, or the type of artifact occurring in the time series data. In addition, the time series data may be at least one of a time series of ultrasound images, a time series of frame data that is data before imaging, trajectory information that represents the trajectory of speckles, time series elastography information that represents the time change in the amount of distortion of the test area, or a time series Doppler signal that represents the time change in the velocity of the test area or blood.

[0008] According to this configuration, the feature prediction learning device is trained using time-series learning time-series data, and is thereby able to output features of the time-series data taking into account changes in signals over time in the time-series data. By inputting time-series target time-series data to a feature prediction learning device trained in this way, it becomes possible to predict features of an ultrasound image with higher accuracy than, at least, when an ultrasound image is input to a learning device trained using ultrasound images, which are still images, and the features of the ultrasound image are predicted.

[0009] The feature prediction unit may input the target time series data to a feature prediction learner selected based on user instructions from among a plurality of feature prediction learners trained to predict and output features that differ from one another.

[0010] According to this configuration, the user can obtain a prediction result of a desired feature regarding the target time series data.

[0011] The feature prediction unit may input the target time series data to each of a plurality of feature prediction learners trained to predict and output different features, and predict features related to the target time series data based on a combination of outputs from the plurality of feature prediction learners.

[0012] This configuration is expected to further improve the accuracy of feature prediction for target time series data. For example, the feature prediction unit predicts the type of test to be performed using the target time series data based on the output of a feature prediction learner that predicts the type of test to be performed using the time series data, and identifies multiple diseases that can be identified from the predicted type of test. Then, the feature prediction unit inputs the target time series data to a feature prediction learner that predicts the disease indicated by the time series data, thereby predicting the disease indicated by the target time series data by selecting from the multiple diseases previously identified. This is expected to further improve the accuracy of disease prediction.

[0013] The target time series data and the learning time series data may be time series data corresponding to the same period in the pulsation cycle of the subject.

[0014] According to this configuration, the period of the learning time series data and the target time series data in the pulsation cycle is the same, thereby improving the output accuracy of the feature prediction learning device, and thereby improving the prediction accuracy of the features indicated by the target time series data.

[0015] The apparatus may further include a time-series data generating unit that generates the target time-series data.

[0016] According to this configuration, in an ultrasound diagnostic apparatus that transmits and receives ultrasound to and from a subject to generate target time-series data, it is possible to predict features related to the target time-series data.

[0017] The feature prediction unit may repeatedly predict features of the target time series data based on the current time point.

[0018] According to this configuration, the user can obtain prediction results of features related to the latest target time series data.

[0019] The feature prediction unit may predict features of the target time-series data based on an electrocardiogram waveform acquired from the target subject, with a time point at which the heartbeat interval of the target subject becomes stable as a reference.

[0020] According to this configuration, it is possible to predict the characteristics of target time series data acquired in a state where the heartbeat interval is stable.

[0021] The feature prediction unit may predict features of the target time-series data based on a time point when a user operates an ultrasonic probe that transmits and receives ultrasonic waves.

[0022] According to this configuration, the user can obtain a prediction of the characteristics of the target time-series data every time he or she operates the ultrasound probe.

[0023] The ultrasound time series data processing program disclosed in the present specification causes a computer to function as a feature prediction unit that inputs target time series data, which is time series data generated by repeatedly transmitting and receiving ultrasound waves multiple times for the same scanning plane set within a target subject, into a feature prediction learner that is trained to predict and output features related to the time series data based on the input time series data, using a combination of training time series data, which is time series data showing changes in signals over time, generated by repeatedly transmitting and receiving ultrasound waves multiple times for the same scanning plane set within the subject, and information representing features related to the training time series data, and predicts features related to the target time series data based on the output of the feature prediction learner in response to the input, and a notification unit that notifies a user of the prediction result of the feature prediction unit. [Effects of the Invention]

[0024] According to the ultrasound time-series data processing device disclosed in this specification, it is possible to improve the prediction accuracy of features indicated by ultrasound images. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a block diagram of an ultrasonic diagnostic apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 is a conceptual diagram showing a time-series frame data sequence. [Figure 3] FIG. 10 is a diagram showing an example of a notification screen for notifying a prediction result of a feature prediction unit. [Figure 4] FIG. 10 is a first conceptual diagram showing the timing at which a prediction process is performed by a feature prediction unit and target time series data. [Figure 5] FIG. 10 is a second conceptual diagram showing the timing at which the feature prediction unit performs prediction processing and target time series data. [Figure 6] FIG. 10 is a third conceptual diagram showing the timing at which the feature prediction unit performs prediction processing and target time series data. [Figure 7] 4 is a flowchart showing the flow of processing performed by the ultrasound diagnostic apparatus according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0026] 1 is a block diagram of an ultrasound diagnostic apparatus 10 serving as an ultrasound time-series data processing apparatus according to this embodiment. The ultrasound diagnostic apparatus 10 is a medical device installed in a medical institution such as a hospital and used during ultrasound examinations.

[0027] As will be described in detail later, the ultrasound diagnostic device 10 is a device that scans an object with an ultrasound beam and generates an ultrasound image based on the frame data obtained thereby. More specifically, the ultrasound diagnostic device 10 generates a tomographic image (B-mode image) in which the amplitude intensity of the reflected wave from the scanning plane is converted into brightness based on the frame data.

[0028] The ultrasound diagnostic device 10 also has a speckle tracking function that tracks small specks (speckles) in the time direction in multiple B-mode images arranged in time series. The speckle tracking function is used, for example, when evaluating the movement (contraction / expansion) of a specific part of the myocardium. The ultrasound diagnostic device 10 also has an elastography function that applies pressure to the area to be examined by, for example, pressing the probe 12 against the surface of the subject's body, estimates the amount of strain of the area to be examined corresponding to the pressure, and represents the amount of strain of the area to be examined in color and superimposes it on the B-mode image. The ultrasound diagnostic device 10 also has a color flow mapping function (color Doppler function) that calculates the movement velocity of tissue within the subject based on the difference in frequency between the transmitted wave and the reflected wave, and represents the movement velocity in color and superimposes it on the B-mode image.

[0029] The probe 12, which is an ultrasonic probe, is a device that transmits ultrasonic waves and receives reflected waves. Specifically, the probe 12 is placed in contact with the body surface of a subject, transmits ultrasonic waves toward the subject, and receives the reflected waves reflected by tissue within the subject. A transducer element array consisting of a plurality of transducer elements is provided within the probe 12. A transmission signal, which is an electrical signal, is supplied to each transducer element included in the transducer element array from a transmitter 14, which will be described later, thereby generating an ultrasonic beam (transmission beam). Furthermore, each transducer element included in the transducer element array receives reflected waves from the subject, converts the reflected waves into a reception signal, which is an electrical signal, and transmits the received signal to a receiver 16, which will be described later.

[0030] During transmission of ultrasonic waves, the transmitter 14 supplies multiple transmission signals in parallel to the probe 12 (specifically, the transducer element array) under the control of a processor 46 (described later). This causes ultrasonic waves to be transmitted from the transducer element array. The transmitter 14 supplies the transmission signals to the probe 12 so that the transmission beam transmitted from the probe 12 is electronically scanned within the scanning plane.

[0031] When receiving reflected waves, the receiving unit 16 receives multiple received signals in parallel from the probe 12 (specifically, the transducer array). The receiving unit 16 performs phasing addition (delay addition) on the multiple received signals, thereby generating received beam data.

[0032] 2 is a conceptual diagram showing a time-series frame data sequence. An ultrasonic beam is scanned from the probe 12 in the θ direction on the scanning plane, and one frame of data F corresponding to one ultrasonic image made up of a plurality of received beam data is generated. The same scanning plane set inside the subject is scanned with the ultrasonic beam multiple times. This generates a plurality of time-series frame data F (i.e., a frame data sequence).

[0033] The generated frame data F is sent to the image generating unit 18, the elastography processing unit 22, or the Doppler processing unit 24.

[0034] Returning to FIG. 1 , the image generator 18 is configured with a digital scan converter and has functions such as coordinate conversion, pixel interpolation, and frame rate conversion. The image generator 18 performs various signal processing such as gain correction, logarithmic amplification, and filtering on each receive beam data sequence from the receiver 16. Then, the image generator 18 generates a B-mode image in which the amplitude (intensity) of the reflected wave is represented by brightness, based on frame data consisting of multiple receive beam data sequences after signal processing. The image generator 18 generates a time-series ultrasound image sequence (B-mode image sequence) based on the time-series frame data sequence. The generated B-mode images are sent to the display controller 26 and the processor 46.

[0035] The tracking processing unit 20 executes speckle tracking processing to track speckles in the time direction based on the time-series B-mode image sequence generated by the image generation unit 18. The speckle tracking processing may be the same as conventional processing, and therefore a detailed description thereof will be omitted here. Trajectory information representing the trajectories of the speckles obtained by the speckle tracking processing is sent to the display control unit 26 and the processor 46.

[0036] The elastography processing unit 22 estimates the amount of strain of the test site corresponding to pressure applied by, for example, pressing the probe 12 against the body surface of the subject, based on the received beam data from the receiving unit 16. The process for estimating the amount of strain may be the same as conventional processes, and therefore a detailed description thereof will be omitted here. In particular, the elastography processing unit 22 estimates the time change in the amount of strain of the test site based on the time-series received beam data string from the receiving unit 16. Time-series elastography information representing the time change in the estimated amount of strain is sent to the display control unit 26 and the processor 46.

[0037] The Doppler processing unit 24 calculates the velocity of the test site within the scanning plane or the velocity of the blood flowing within the test site based on the received beam data from the receiving unit 16. The process for calculating the velocity of the test site or the blood may be the same as conventional processes, so a detailed description will be omitted here. In particular, the Doppler processing unit 24 calculates the time change in the velocity of the test site within the scanning plane or the blood flowing within the test site based on the time series received beam data string from the receiving unit 16. A time series Doppler signal (time series velocity signal) indicating the time change in the velocity of the test site or the blood is sent to the display control unit 26 and the processor 46.

[0038] The display control unit 26 displays the B-mode images generated by the image generation unit 18 on a display 28, which serves as a display unit configured, for example, with a liquid crystal panel. In particular, the display control unit 26 displays a time-series B-mode image sequence (i.e., a moving image) on the display 28. The display control unit 26 also displays the results of speckle tracking processing on the display 28 based on trajectory information from the tracking processing unit 20. The display control unit 26 also displays the time-varying distortion amount of the test site on the display 28 by representing it with color, based on the time-series elastography information from the elastography processing unit 22, by superimposing it on the B-mode image sequence (moving image). The display control unit 26 also displays the time-varying distortion amount of the test site or blood velocity on the display 28 by representing it with color, based on the time-series Doppler signal from the Doppler processing unit 24. The display control unit 26 also displays the prediction results of a feature prediction unit 48, which will be described later, on the display 28.

[0039] Each of the transmitter 14, receiver 16, image generator 18, tracking processor 20, elastography processor 22, Doppler processor 24, and display controller 26 is configured with one or more processors, chips, electric circuits, etc. Each of the units may be realized by a combination of hardware and software.

[0040] The input interface 30 is configured by, for example, a button, a trackball, a touch panel, etc. The input interface 30 is used to input user instructions to the ultrasound diagnostic apparatus 10.

[0041] The electrocardiograph 32 includes a plurality of electrodes attached to the subject, and outputs electrocardiogram waveform data, which is the heartbeat waveform of the subject, to the processor 46.

[0042] The memory 34 includes a hard disk drive (HDD), a solid state drive (SSD), an embedded multimedia card (eMMC), a read-only memory (ROM), or a random access memory (RAM). The memory 34 stores an ultrasound time-series data processing program for operating each component of the ultrasound diagnostic device 10. The ultrasound time-series data processing program may be stored in a computer-readable non-transitory storage medium such as a universal serial bus (USB) memory or a CD-ROM. The ultrasound diagnostic device 10 or another computer can read and execute the ultrasound time-series data processing program from such a storage medium. As shown in FIG. 1 , the memory 34 also stores an examination type prediction learner 36, a tissue cross-section prediction learner 38, a measurement position prediction learner 40, a disease prediction learner 42, and an artifact prediction learner 44. Each learner corresponds to a feature prediction learner.

[0043] Each of the learners, the examination type prediction learner 36, the tissue cross section prediction learner 38, the measurement position prediction learner 40, the disease prediction learner 42, and the artifact prediction learner 44, is configured with a learning model such as an RNN (Recurrent Neural Network), an LSTM (Long Short Term Memory) which is a type of RNN, a CNN (Convolutional Neural Network), or a DQN (Deep Q-Network) that uses a deep reinforcement learning algorithm. The above-mentioned learners may all be the same type of learning model, or may be different learning models.

[0044] Each learning device uses a combination of training time-series data, which is time-series data indicating time changes in signals generated by repeatedly transmitting and receiving ultrasound waves to the same scanning plane set within the subject, and information (labels) indicating features of the training time-series data, as training data. The learning device is trained to predict and output features related to the time-series data based on the input time-series data. Specifically, the training time-series data is input to the learning device, and the learning device predicts and outputs features indicated by the training time-series data. The computer performing the training process calculates the error between the output data and the label attached to the training time-series data using a predetermined loss function, and adjusts each parameter of the learning device (e.g., the weight and bias of each neuron) to minimize the error. By repeating this learning process, the learning device becomes able to output features related to the input time-series data with high accuracy.

[0045] In this embodiment, the learning time-series data is a time-series sequence of ultrasound images generated by the image generator 18, but the learning time-series data may also be a time-series sequence of frame data that is data before imaging. Furthermore, in the disease prediction learner 42, the learning time-series data may also be trajectory information generated by the tracking processor 20, time-series elastography information generated by the elastography processor 22, or time-series Doppler signals generated by the Doppler processor 24.

[0046] The subject's target region for the training time series data may be a pulsating region. In this case, the training time series data may be data corresponding to a predetermined period in the pulsation cycle of the subject's region. For example, the subject's electrocardiogram waveform may be acquired from the electrocardiograph 32, and the training time series data may be time series data based on a received beam data sequence acquired during the period between R waves in the electrocardiogram waveform.

[0047] The "feature" mentioned above is at least one of the following: the type of examination performed using the time-series data, the type of tissue cross section indicated by the time-series data, the measurement position to be set in the ultrasound image corresponding to the time-series data, the disease indicated by the time-series data, or the type of artifact occurring in the time-series data. In this embodiment, as will be described below, a learning device is provided for each feature.

[0048] The test type prediction learning device 36 is trained to predict and output the type of test to be performed using the time series data based on the input time series data, using a combination of learning time series data and information indicating the type of test to be performed using the learning time series data as learning data.

[0049] The tissue cross section prediction learning device 38 is trained to predict and output the tissue cross section shown by the ultrasound image corresponding to the time series data based on the input time series data, using as learning data a combination of learning time series data and information indicating the tissue cross section (e.g., four-chamber cross section, two-chamber cross section of the heart) shown by the ultrasound image corresponding to the learning time series data.

[0050] The measurement position prediction learning device 40 is trained to use a combination of learning time series data and information indicating the measurement position to be set in a measurement using the learning time series data (for example, the position of tissue or the cursor position in a Doppler examination) as learning data, and to predict and output the measurement position to be set in a measurement using the time series data based on the input time series data.

[0051] The disease prediction learning device 42 is trained to use a combination of learning time series data and information indicating the disease indicated by the learning time series data as learning data, and to predict and output the disease indicated by the time series data based on the input time series data.

[0052] The artifact prediction learning device 44 is trained to use a combination of learning time series data and information indicating the type of artifact that will occur in the ultrasound image corresponding to the learning time series data as learning data, and to predict and output the type of artifact that will occur in the ultrasound image corresponding to the time series data based on the input time series data.

[0053] In this embodiment, each of the above learning units is trained by a computer other than the ultrasound diagnostic device 10, and the trained learning units are stored in the memory 34. However, the time-series data acquired by the ultrasound diagnostic device 10 may be used as training time-series data, and the learning process of each learning unit may be performed by the ultrasound diagnostic device 10. In this case, the processor 46 functions as a learning processing unit that performs the learning process of each learning unit.

[0054] The processor 46 includes at least one of a general-purpose processing device (e.g., a CPU (Central Processing Unit)) and a dedicated processing device (e.g., a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a programmable logic device). The processor 46 may not be a single processing device, but may be configured by the cooperation of multiple processing devices located in physically separate locations. As shown in FIG. 1, the processor 46 functions as a feature prediction unit 48 in accordance with an ultrasound time-series data processing program stored in the memory 34.

[0055] The feature prediction unit 48 inputs time series data (referred to herein as "target time series data") generated by repeatedly transmitting and receiving ultrasound waves to and from the same scanning plane set within the subject (referred to herein as "target subject") for feature prediction into the trained learning device. In this embodiment, the target time series data is a time series of ultrasound images generated by the image generation unit 18, but the target time series data may also be a time series of frame data that is data before imaging. Furthermore, in the disease prediction learning device 42, the target time series data may be trajectory information generated by the tracking processing unit 20, time series elastography information generated by the elastography processing unit 22, or time series Doppler signals generated by the Doppler processing unit 24.

[0056] When the target time series data is a sequence of ultrasound images, the receiving unit 16 and the image generating unit 18 correspond to the time series data generating unit, and when the target time series data is a sequence of time series frame data that is data before imaging, the receiving unit 16 corresponds to the time series data generating unit. When the target time series data is trajectory information, the receiving unit 16, the image generating unit 18, and the tracking processing unit 20 correspond to the time series data generating unit, when the target time series data is time series elastography information, the receiving unit 16 and the elastography processing unit 22 correspond to the time series data generating unit, and when the target time series data is time series Doppler signals, the receiving unit 16 and the Doppler processing unit 24 correspond to the time series data generating unit.

[0057] It should be noted that previously acquired time-series data (frame data sequence, ultrasound image sequence, trajectory information, time-series elastography information, or time-series Doppler signals) is temporarily stored in a cine memory (not shown) or the like for a predetermined period of time. For example, time-series data for multiple heartbeats is stored based on the electrocardiogram waveform of the target subject acquired by the electrocardiograph 32. As will be described in detail later, the feature prediction unit 48 selects target time-series data from the stored time-series data.

[0058] Each learning device predicts features of the target time series data based on the input target time series data and outputs output data indicating the prediction results. The feature prediction unit 48 predicts features of the target time series data based on the output data of each learning device.

[0059] Specifically, the feature prediction unit 48 inputs the target time series data into the trained examination type prediction learner 36 to predict the type of examination to be performed using the target time series data. The feature prediction unit 48 also inputs the target time series data into the trained tissue cross-section prediction learner 38 to predict the tissue cross-section indicated by the ultrasound image corresponding to the target time series data. The feature prediction unit 48 also inputs the target time series data into the trained measurement position prediction learner 40 to predict the measurement position to be set for measurement using the target time series data. Predicting the type of examination to be performed using the target time series data, predicting the tissue cross-section indicated by the ultrasound image corresponding to the target time series data, or predicting the measurement position to be set can be useful for those unfamiliar with diagnostics using ultrasound images. Furthermore, the prediction of the tissue cross-section can also be useful for performing automatic processing. For example, when automatically tracing the outline of the heart shown in a B-mode image, the ultrasound diagnostic device 10 needs to know in advance that the heart is shown in the B-mode image. By predicting the tissue cross section, the ultrasound diagnostic apparatus 10 can determine that the heart is displayed in the B-mode image, and then proceed to automatic tracing of the heart contour.

[0060] Furthermore, the feature prediction unit 48 inputs the target time series data into the trained disease prediction learner 42 to predict the disease indicated by the target time series data. As described above, the feature prediction unit 48 can input the trajectory information generated by the tracking processing unit 20, the time-series elastography information generated by the elastography processing unit 22, or the time-series Doppler signal generated by the Doppler processing unit 24 as the target time series data to the disease prediction learner 42. The disease prediction learner 42 trained using the trajectory information as training data predicts the disease indicated by the target time series data based on the speckle trajectory. The disease prediction learner 42 trained using the time-series elastography information as training data predicts the disease indicated by the target time series data based on the time change in the amount of strain at the test site. Furthermore, the disease prediction learner 42 trained using the time-series Doppler signal as training data predicts the disease indicated by the target time series data based on the time change in the motion velocity of tissue within the subject (e.g., the time change in blood flow).

[0061] Furthermore, the feature prediction unit 48 predicts the type of artifact that will occur in the ultrasound image corresponding to the target time series data by inputting the target time series data to the trained artifact prediction learner 44. Predicting the type of artifact that will occur in the ultrasound image corresponding to the target time series data can be useful for those who are unfamiliar with identifying artifacts that occur in ultrasound images.

[0062] In this embodiment, multiple learners trained to predict and output different features are stored in the memory 34. Therefore, the feature prediction unit 48 inputs the target time series data to a learner selected from the multiple learners based on a user instruction. The user can directly specify the learner to which the target time series data is input using the input interface 30. The feature prediction unit 48 may also select a learner to which the target time series data is input based on the operation settings of the ultrasound diagnostic apparatus 10 determined by the user using the input interface 30. For example, when the operation mode of the ultrasound diagnostic apparatus 10 is a color Doppler mode that performs a color Doppler function, the feature prediction unit 48 inputs the target time series data to the measurement position prediction learner 40 that predicts the position of a Doppler cursor. When the operation mode of the ultrasound diagnostic apparatus 10 is a speckle tracking mode that performs speckle tracking, the feature prediction unit 48 inputs the target time series data (trajectory information in this case) to the disease prediction learner 42.

[0063] The feature prediction unit 48 may also input target time series data to each of multiple learners and predict features related to the target time series data based on a combination of the outputs of the multiple learners. For example, the feature prediction unit 48 predicts the type of examination to be performed using the target time series data based on the output of the test type prediction learner 36, and identifies multiple diseases that can be identified from the predicted test type. Identification of multiple diseases based on the test type can be performed, for example, by referencing a table that associates the test type with multiple diseases that can be identified from the test type. The feature prediction unit 48 then inputs the target time series data into the trained disease prediction learner 42, and predicts the disease indicated by the target time series data by selecting it from the multiple identified diseases. This is expected to further improve the accuracy of disease prediction. Similarly, the feature prediction unit 48 predicts the tissue cross-section indicated by the ultrasound image corresponding to the target time series data based on the output of the tissue cross-section prediction learner 38, and identifies multiple diseases that can be identified from the predicted tissue cross-section. Identification of multiple diseases based on tissue cross sections can also be performed by, for example, referring to a table that associates tissue cross sections with multiple diseases that can be identified from the tissue cross sections. Then, the feature prediction unit 48 inputs the target time series data into the trained disease prediction learner 42, and predicts the disease indicated by the target time series data by selecting it from the multiple identified diseases. This is expected to further improve the accuracy of disease prediction.

[0064] The feature prediction unit 48 may input the target time series data to all the learning devices and obtain the prediction results of each feature in all the learning devices in parallel.

[0065] As described above, in this embodiment, each learning device is trained using training time-series data. This allows each learning device to output features of the time-series data in consideration of time changes in the time-series data (frame data sequence, ultrasound image sequence, trajectory information, time-series elastography information, or time-series Doppler signals). By inputting target time-series data to a learning device trained in this way, it becomes possible to predict features of an ultrasound image with higher accuracy than, at least, when an ultrasound image is input to a learning device trained using ultrasound images, which are still images, and the features of the ultrasound image are predicted.

[0066] The target time series data and the training time series data may be time series data corresponding to the same period in the subject's pulse cycle in the electrocardiogram waveform acquired by the electrocardiograph 32. For example, if the training time series data is time series data based on a receive beam data sequence acquired during the period between R waves in the electrocardiogram waveform, the feature prediction unit 48 may also set the target time series data as time series data based on a receive beam data sequence acquired during the period between R waves in the electrocardiogram waveform. By making the training time series data and the target time series data have the same period in the pulse cycle, the output accuracy of each learner can be improved. In other words, the prediction accuracy of the feature indicated by the target time series data by the feature prediction unit 48 is improved.

[0067] The display control unit 26 notifies the user of the prediction result of the feature prediction unit 48. That is, the display control unit 26 functions as a notification unit. In the present embodiment, the prediction result of the feature prediction unit 48 is displayed on the display 28 by the display control unit 26 as described below, but in addition to or instead of this, the prediction result of the feature prediction unit 48 may be notified to the user by audio output or the like.

[0068] 3 is a diagram showing an example of a notification screen 50 displayed on the display 28 to notify the user of the prediction result of the feature prediction unit 48. The notification screen 50 displays an ultrasound image 52 (moving image) generated based on the target time-series data and a prediction result 54 of the feature prediction unit 48. The notification screen 50 displays the prediction result of the disease indicated by the target time-series data by the feature prediction unit 48 using the disease prediction learner 42.

[0069] The timing at which the feature prediction unit 48 performs the above-mentioned prediction process and the method for selecting target time series data to be subjected to the prediction process will be described below.

[0070] First, the feature prediction unit 48 can execute a prediction process at the timing when an instruction is received from the user. For example, in response to a user inputting a prediction instruction to the ultrasound diagnostic apparatus 10 using the input interface 30, the feature prediction unit 48 predicts the features of the target time series data by inputting the target time series data obtained up to that point into a learning device.

[0071] Fig. 4 is a diagram showing target time-series data when a prediction process is executed at the timing instructed by a user. In Fig. 4 (as well as Figs. 5 and 6 described later), the horizontal axis represents the time axis, and the electrocardiographic waveform of the target subject acquired by the electrocardiograph 32 is shown.

[0072] 4, the user inputs a prediction instruction to the ultrasound diagnostic apparatus 10 at time t1, and the feature prediction unit 48 executes the prediction process at time t1. In this case, the feature prediction unit 48 selects target time series data from the stored time series data with time t1 as the reference.

[0073] For example, as indicated by reference numeral 60 in FIG. 4 , the feature prediction unit 48 may use time series data acquired during a period of one heartbeat immediately prior to time t1, which is the execution time of the prediction process, as the target time series data. Here, time series data from R wave to R wave of the electrocardiogram waveform is used as the target time series data. Also, as indicated by reference numeral 62 in FIG. 4 , the feature prediction unit 48 may use time series data acquired during a predetermined period immediately prior to time t1 as the target time series data. Furthermore, as indicated by reference numeral 64 in FIG. 4 , the feature prediction unit 48 may use a predetermined number of data acquired at the timing of the R wave of the electrocardiogram waveform as the target time series data. As indicated by reference numeral 64, the multiple data included in the target time series data do not have to be consecutive in time.

[0074] Furthermore, the feature prediction unit 48 may repeatedly execute the prediction process at predetermined time intervals without relying on a prediction instruction from the user. In this case, the feature prediction unit 48 repeatedly executes prediction of features for the target time series data (e.g., the target time series data indicated by reference numerals 60, 62, and 64 in FIG. 4) based on the execution time of the prediction process (i.e., the current time). This allows the user to obtain prediction results for features for the latest target time series data.

[0075] The feature prediction unit 48 may also perform the prediction process based on the electrocardiogram waveform of the target subject acquired by the electrocardiograph 32. Specifically, as shown in FIG. 5, the feature prediction unit 48 may perform the prediction process at a timing when the heartbeat interval of the target subject becomes stable. The heartbeat interval of the target subject can be determined, for example, by detecting R waves from the electrocardiogram waveform and measuring the time between R waves. In the example of FIG. 5, the prediction process is performed at time t2, when the heartbeat interval for three heartbeats becomes stable. In this case, the feature prediction unit 48 performs feature prediction on the target time series data (e.g., time series data acquired during a period of one heartbeat immediately before time t2 (reference numeral 66 in FIG. 5), time series data acquired during a predetermined time immediately before time t2, or a predetermined number of data acquired at the timing of the R wave of the electrocardiogram waveform with time t2 as the reference) based on the execution time of the prediction process (time t2 in the example of FIG. 5). This allows feature prediction of the target time series data acquired when the heartbeat interval is stable.

[0076] Furthermore, the feature prediction unit 48 may be configured to execute the prediction process at a timing based on the time point when the user operates the probe 12. Specifically, the feature prediction unit 48 may be configured to execute the prediction process at a timing when the attitude of the probe 12 has been stable for a predetermined period of time after the attitude of the probe 12 has been changed. The feature prediction unit 48 can identify the attitude of the probe 12 based on, for example, a detection signal from an acceleration sensor provided in the probe 12. Furthermore, the feature prediction unit 48 can identify a change in the attitude of the probe 12 based on an image change in the B-mode image generated by the image generation unit 18. This allows the user to obtain a prediction of the feature of the target time series data every time the user operates the probe 12. In particular, the user can obtain a prediction of the feature of the target time series data every time the user changes the attitude of the probe 12.

[0077] For example, in FIG. 6, assume that at time t3, the user changes the posture of the probe 12 from posture 1 to posture 2 and then maintains the posture of the probe 12 at posture 2. If the feature prediction unit 48 determines that the posture of the probe 12 has stabilized (the amount of change has been maintained at or below a predetermined amount) for a predetermined time T from time t3 when the posture change of the probe 12 is detected, the feature prediction unit 48 executes a prediction process at time t4, which is the time after the predetermined time T from time t3. In this case, the feature prediction unit 48 executes prediction of features regarding target time series data (e.g., time series data acquired during a period of one heartbeat immediately before time t4 (reference numeral 68 in FIG. 6), time series data acquired during a predetermined time immediately before t4, or a predetermined number of data acquired at the timing of an R wave of an electrocardiogram waveform with time t4 as the reference) based on the execution time of the prediction process (time t4 in the example of FIG. 6).

[0078] Similarly, suppose that at time t5, the user changes the posture of the probe 12 from posture 2 to posture 3 and then maintains the posture of the probe 12 at posture 3. If the feature prediction unit 48 determines that the posture of the probe 12 has stabilized for a predetermined time T from time t5, when the posture change of the probe 12 is detected, the feature prediction unit 48 executes a prediction process at time t6, which is the time after the predetermined time T from time t5. In this case, the feature prediction unit 48 executes prediction of features regarding target time series data (e.g., time series data (reference numeral 70 in FIG. 6) acquired during a period of one heartbeat immediately before time t6, time series data acquired during a predetermined time immediately before t6, or a predetermined number of data acquired at the timing of an R wave of an electrocardiogram waveform with time t6 as the reference) based on the execution time of the prediction process (time t6 in the example of FIG. 6).

[0079] The flow of processing performed by the ultrasound diagnostic apparatus 10 according to this embodiment will be described below with reference to the flowchart shown in FIG.

[0080] In step S10, the ultrasound diagnostic apparatus 10 sets an operation mode in response to a user instruction from the input interface 30. The operation mode may be, for example, a B-mode for generating a B-mode image, a speckle tracking mode for performing speckle tracking, an elastography mode for performing an elastography function, or a color Doppler mode for performing a color flow mapping function.

[0081] In step S12, ultrasonic waves are transmitted and received from the probe 12 to the subject, thereby acquiring time-series data (frame data sequence, ultrasonic image sequence, trajectory information, time-series elastography information, or time-series Doppler signals). The time-series data is stored in a cine memory or the like.

[0082] In step S14, the feature prediction unit 48 sets the type of feature to be predicted for the target time series data. In other words, the feature prediction unit 48 selects a learning unit to which the target time series data is input from among the multiple learning units stored in the memory 34. The feature prediction unit 48 may select a learning unit based on an instruction input by the user to the input interface 30, or may select a learning unit based on the operation mode of the ultrasound diagnostic device 10 set in step S10.

[0083] In step S16, the feature prediction unit 48 determines whether it is time to execute the prediction process. As described above, the timing to execute the prediction process may be when an instruction is received from the user, when a predetermined time has elapsed since the previous prediction execution if the prediction process is performed at predetermined time intervals, when the heartbeat interval of the subject has stabilized, or when the posture of the probe 12 has stabilized for a predetermined time after a change in posture of the probe 12. The feature prediction unit 48 waits until it is time to execute the prediction process, and if it is time to execute the prediction process, proceeds to step S18.

[0084] In step S18, the feature prediction unit 48 selects target time series data from the time series data held in step S12, based on the execution time of the prediction process. For example, the feature prediction unit 48 selects, as the target time series data, time series data acquired during a period of one heartbeat immediately before the execution time of the prediction process, time series data acquired during a predetermined time immediately before the execution time of the prediction process, or a predetermined number of data acquired at the timing of the R wave of an electrocardiogram waveform based on the execution time of the prediction process.

[0085] In step S20, the feature prediction unit 48 inputs the target time series data selected in step S18 to the learning device selected in step S14. Then, the feature prediction unit 48 predicts features related to the target time series data based on the output data of the learning device.

[0086] In step S22, the display control unit 26 notifies the user of the prediction result of the feature prediction unit 48 (see FIG. 3).

[0087] In step S24, processor 46 determines whether an instruction to stop the prediction process has been input by a user. If an instruction to stop the prediction process has not been input, the process returns to step S16 and repeats the processes from step S16 to S24. If an instruction to stop the prediction process has been input, the process ends.

[0088] Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications are possible without departing from the spirit of the present invention.

[0089] For example, in the present embodiment, the ultrasound time-series data processing device is the ultrasound diagnostic device 10, but the ultrasound time-series data processing device is not limited to the ultrasound diagnostic device 10 and may be another computer. In this case, each trained learner is stored in a memory accessible from the computer serving as the ultrasound time-series data processing device, and the processor of the computer fulfills the functions of the feature prediction unit 48 and the display control unit 26. [Explanation of symbols]

[0090] 10 Ultrasound diagnostic device, 12 Probe, 14 Transmitter, 16 Receiver, 18 Image generator, 20 Tracking processor, 22 Elastomeric processor, 24 Doppler processor, 26 Display controller, 28 Display, 30 Input interface, 34 Memory, 36 Examination type prediction learner, 38 Tissue cross section prediction learner, 40 Measurement position prediction learner, 42 Disease prediction learner, 44 Artifact prediction learner, 46 Processor, 48 Feature prediction unit.

Claims

1. a feature prediction unit that inputs target time series data, which is time series data generated by repeatedly transmitting and receiving ultrasonic waves multiple times on the same scanning plane set inside the target object, to a feature prediction learner that is trained to predict and output features related to the time series data based on the input time series data, using a combination of training time series data, which is time series data showing changes over time in signals, generated by repeatedly transmitting and receiving ultrasonic waves multiple times on the same scanning plane set inside the object, and information indicating features related to the training time series data, as training data, and predicts features related to the target time series data based on the output of the feature prediction learner in response to the input; a notification unit that notifies a user of a prediction result of the feature prediction unit; a time series data generation unit that generates the target time series data; Equipped with the feature prediction unit predicts, at the time point, features of the target time series data acquired during a period of one heartbeat of the subject immediately before a time point at which the posture of the ultrasound probe that transmits and receives ultrasound waves has been stabilized for a predetermined period of time since the posture of the ultrasound probe has been changed; 1. An ultrasonic time-series data processing device comprising:

2. The feature is at least one of a type of examination performed using the time series data, a type of tissue cross section indicated by the time series data, a measurement position to be set, a disease indicated by the time series data, or a type of artifact occurring in the time series data.

2. The ultrasonic time-series data processing device according to claim 1,

3. The time series data is at least one of a time series ultrasound image sequence, a time series frame data sequence which is data before imaging, trajectory information which represents a speckle trajectory, time series elastography information which represents a time change in the amount of strain of the test site, or a time series Doppler signal which represents a time change in the velocity of the test site or blood.

2. The ultrasonic time-series data processing device according to claim 1,

4. the feature prediction unit inputs the target time series data to a feature prediction learning device selected based on a user instruction from among a plurality of feature prediction learning devices trained to predict and output the features different from each other; 2. The ultrasonic time-series data processing device according to claim 1,

5. the feature prediction unit inputs the target time series data to each of a plurality of the feature prediction learners, each of which has been trained to predict and output the features different from one another, and predicts features related to the target time series data based on a combination of outputs from the plurality of the feature prediction learners.

2. The ultrasonic time-series data processing device according to claim 1,

6. the target time series data and the learning time series data are time series data corresponding to the same period in a pulsation cycle of the subject; 2. The ultrasonic time-series data processing device according to claim 1,

7. Computer, a feature prediction unit that inputs target time series data, which is time series data generated by repeatedly transmitting and receiving ultrasonic waves multiple times on the same scanning plane set inside the target object, to a feature prediction learner that is trained to predict and output features related to the time series data based on the input time series data, using a combination of training time series data, which is time series data showing changes over time in signals, generated by repeatedly transmitting and receiving ultrasonic waves multiple times on the same scanning plane set inside the object, and information indicating features related to the training time series data, as training data, and predicts features related to the target time series data based on the output of the feature prediction learner in response to the input; a notification unit that notifies a user of a prediction result of the feature prediction unit; a time series data generation unit that generates the target time series data; It functions as the feature prediction unit predicts, at the time point, features of the target time series data acquired during a period of one heartbeat of the subject immediately before a time point at which the posture of the ultrasound probe that transmits and receives ultrasound waves has been stabilized for a predetermined period of time since the posture of the ultrasound probe has been changed; 1. An ultrasonic time series data processing program comprising:

Images