Method for providng information on strain quantification and device for providng information on strain quantification using the same
An AI-powered cardiac ultrasound analysis system segments heart structures and calculates strain parameters using motion vector fields and spline curves, addressing manual measurement flaws and enhancing cardiovascular diagnosis accuracy.
Patent Information
- Application Number
- KR1020240115280
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2024-06-07
- Filing Date
- 2024-08-27
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2044-08-27
AI Technical Summary
Conventional strain analysis methods in cardiovascular diagnosis, particularly for cardiac function evaluation, are time-consuming and subjective, leading to unreliable results due to manual measurement and reliance on expert proficiency, with limited scope in measuring parameters beyond left ventricle size and function.
An artificial neural network-based prediction model for cardiac ultrasound images that segments heart structures, determines motion vector fields, and calculates strain parameters, using spline curves for accurate motion estimation and quantification, including left atrial volume and diastolic function.
Provides a highly reliable and automated system for strain quantification, overcoming manual measurement errors and limitations of conventional methods by offering precise cardiac muscle movement evaluation and early diagnosis of cardiopulmonary diseases.
Smart Images

Figure 112024093814005-PAT00013_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a method for providing information on strain quantification and a device for providing information on strain quantification using the same. Background Technology
[0002] Strain is a technique used to determine whether there is an abnormality in an organ by measuring the strain of micro-muscles.
[0003] In particular, in relation to cardiovascular diagnosis, strain is used to evaluate cardiac function by measuring the movement of the heart muscle. In this case, strain is measured via echocardiography and can be expressed as a percentage of how much the length changes when the muscle contracts or relaxes. Through this, the contraction and relaxation states of target areas of the heart (e.g., the left ventricle) can be quantitatively analyzed.
[0004] Meanwhile, while such quantitative analysis of strain is essential for the diagnosis and treatment of heart disease, manual measurement is time-consuming and results can vary depending on the subjectivity of the expert. Furthermore, manual measurement of strain requires multiple steps, and errors can occur at each step.
[0005] In other words, manual measurement of strain can rely on the experience of experts, and there is a problem in that the reliability of the analysis results is insufficient depending on the proficiency of the medical staff.
[0006] Accordingly, there is a continuous demand for the development of an information provision system for strain quantification that can provide results with high accuracy and reproducibility.
[0007] The background description of the invention is provided to facilitate a better understanding of the present invention. The matters described in the background description should not be construed as an acknowledgment that they exist as prior art. The problem to be solved
[0008] With the recent advancement of artificial intelligence technology, various techniques for strain analysis have been proposed.
[0009] Meanwhile, the proposed artificial intelligence-based strain analysis techniques are mainly limited to measuring the size and volume of the left ventricle (LV) and calculating parameters such as the LV ejection fraction (LVEF) to evaluate left ventricular systolic function, so there may be limitations in their scope of application.
[0010] The inventors of the present invention intended to construct an artificial neural network-based prediction model capable of analyzing various parameters, including not only the systolic function of the left ventricle but also the measurement of left atrial (LA) volume and further evaluation of left ventricular diastolic function (LVDF).
[0011] In particular, the inventors of the present invention intended to provide an information-providing system capable of motion estimation and quantitative strain analysis for target cardiac regions such as the LV or LA by constructing it to segment the anatomical structures of the heart for a single model and simultaneously determine the motion vector field.
[0012] Accordingly, the inventors of the present invention recognized that they could solve the problem of errors occurring as multiple steps are taken in the manual measurement of strain.
[0013] In this regard, the inventors of the present invention recognized that when predicting motion vector fields between adjacent frames, the similarity between global structural features and local features can be considered, thereby enabling motion estimation that is robust to image noise.
[0014] As a result, the inventors of the present invention were able to apply a prediction model trained to predict motion field vectors between adjacent frames for a segmented target heart region to a new information providing system.
[0015] Furthermore, the inventors of the present invention were able to perform geometric modeling that enables motion estimation of a target area by defining a spline curve using spline mathematical techniques.
[0016] At this time, the inventors of the present invention recognized that motion estimation is possible without folding by setting an ROI (Region of interest) based on the inner wall of the heart structure, which is the target area, and generating a spline curve.
[0017] As a result, the inventors of the present invention can expect to be able to accurately evaluate the movement of the heart muscle by providing a new information provision system based on spatiotemporal motion modeling, thereby providing not only visual information of the dysfunctional area but also highly reliable quantitative strain information.
[0018] Accordingly, the problem that the present invention aims to solve is to provide a method, device, and system for providing information on strain quantification, configured to determine a motion vector field for a target heart region using a prediction model for a cardiac ultrasound image obtained from an individual, and to provide information related to strain quantification based thereon.
[0019] The problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem
[0020] To solve the problem described above, a method for providing information on strain quantification according to an embodiment of the present invention is provided. The method for providing information on strain quantification, implemented by a processor, comprises the steps of: receiving a cardiac ultrasound image including a target heart region of an object; dividing the target heart region using the cardiac ultrasound image as input and determining a motion vector field for the target heart region within the received cardiac ultrasound image using a prediction model trained to determine a motion vector field based on the divided target heart region; and determining a strain quantification parameter based on the motion vector field.
[0021] At this time, the target cardiac region is at least one of LV (Left Ventricle), RV (Right Ventricle), LA (Left Atrium) and RA (Right Atrium).
[0022] According to a feature of the present invention, a cardiac ultrasound image is a video composed of a plurality of frames, and a prediction model may be configured to determine a motion vector field based on at least one frame and an adjacent frame for at least one frame selected among the plurality of frames.
[0023] According to another feature of the present invention, a plurality of frames includes a plurality of frames having a first resolution for a target heart region and a plurality of frames having a second resolution, and the step of determining a motion vector field may include, using a prediction model, a step of determining a correlation for a plurality of frames having a first resolution, a step of determining a correlation for a plurality of frames having a second resolution, a step of integrating motion features based on the correlation for each of the first resolution and the second resolution, and a step of determining a motion vector field based on the integrated motion features.
[0024] According to another feature of the present invention, the first resolution or the second resolution has a resolution greater than the other one, and the step of determining a motion vector field based on integrated motion features may further include the step of determining a feature map for a plurality of frames corresponding to the larger resolution, and the step of determining a motion vector field based on the feature map and the integrated motion features.
[0025] According to another feature of the present invention, a plurality of frames includes a first frame for a target heart region and a second frame that is a frame before or after the first frame, and the step of determining a motion vector field may include the step of determining a first motion vector field for the first frame using a prediction model, and the step of estimating a second motion vector field for the second frame based on the first motion vector field.
[0026] According to another feature of the present invention, the step of determining a motion vector field may include the step of determining a spline curve using a spline mathematical technique to estimate motion for a target heart region using a prediction model.
[0027] According to another feature of the present invention, the step of determining a spline curve may further include the step of determining a heart wall within a target heart region, the step of determining an intermediate layer for the heart wall, the step of extending the intermediate layer to determine a Region of Interest (ROI), and the step of obtaining a spline curve for the ROI.
[0028] According to another feature of the present invention, the step of determining a spline curve may include the step of determining a plurality of spline curve layers to obtain a spline surface composed of a plurality of spline curve layers.
[0029] According to another feature of the present invention, the step of correcting the determined spline curve may be further included.
[0030] According to another feature of the present invention, the step of correcting a spline curve may include the step of determining the curvature of the spline curve, and the step of correcting the spline curve by excluding data points among the data points forming the spline curve whose curvature is greater than or equal to a predetermined level.
[0031] According to another feature of the present invention, the step of correcting the spline curve may further include a step of smoothing by assigning weights to data points corresponding to specific regions of the target heart region during the process of generating the spline curve.
[0032] According to another feature of the present invention, the prediction model is a model further trained to classify a cross-sectional view of an ultrasound image using a cardiac ultrasound image as input, and the step of determining a motion vector field further includes, using the prediction model, a step of classifying a cross-sectional view for a received ultrasound image, a step of segmenting a target heart region for an ultrasound image corresponding to a classified cross-sectional view, and a step of determining a motion vector field for a target heart region, and the step of determining a strain quantitative parameter may further include a step of determining a strain quantitative parameter corresponding to a classified view.
[0033] According to another feature of the present invention, the method may further include the step of outputting and providing a mask for a target heart region segmented by a prediction model.
[0034] According to another feature of the present invention, in claim 1, the target heart region is LA, and the step of determining a strain quantitative parameter may include the step of determining a strain curve for LA based on a motion vector field, and the step of determining a quantitative parameter for LA based on the strain curve.
[0035] To solve the problem described above, a device for providing information on strain quantification according to an embodiment of the present invention is provided.
[0036] The above device includes a communication unit configured to receive a cardiac ultrasound image including a target cardiac region of an object, and a processor functionally connected to the communication unit. In this case, the processor is configured to divide the target cardiac region using a cardiac ultrasound image as input, determine a motion vector field for the target cardiac region within the received cardiac ultrasound image using a prediction model learned to determine a motion vector field based on the divided target cardiac region, and determine a strain quantitative parameter based on the motion vector field, wherein the target cardiac region is at least one of LV, RV, LA, and RA.
[0037] According to a feature of the present invention, a cardiac ultrasound image is a video composed of a plurality of frames, and a prediction model may be configured to determine a motion vector field based on at least one frame and an adjacent frame for at least one frame selected among the plurality of frames.
[0038] According to another feature of the present invention, a plurality of frames includes a plurality of frames having a first resolution for a target heart region and a plurality of frames having a second resolution, and a processor may be further configured to use a prediction model to determine a correlation for a plurality of frames having a first resolution and a correlation for a plurality of frames having a second resolution, integrate motion features based on the correlation for each of the first resolution and the second resolution, and determine a motion vector field based on the integrated motion features.
[0039] According to another feature of the present invention, the first resolution or the second resolution has a resolution greater than the other one, and the processor may be further configured to determine a feature map for a plurality of frames corresponding to the larger resolution and to determine a motion vector field based on the feature map and the integrated motion features.
[0040] According to another feature of the present invention, a plurality of frames includes a first frame for a target heart region and a second frame that is a frame before or after the first frame, and the processor may be further configured to determine a first motion vector field for the first frame using a prediction model and to estimate a second motion vector field for the second frame based on the first motion vector field.
[0041] According to another feature of the present invention, the processor may be further configured to determine a spline curve using a spline mathematical technique to estimate motion for a target heart region using a prediction model.
[0042] According to another feature of the present invention, the processor may be further configured to determine a heart wall within a target heart region, determine an intermediate layer with respect to the heart wall, extend the intermediate layer to determine an ROI, and obtain a spline curve for the ROI.
[0043] According to another feature of the present invention, the processor may be further configured to determine a plurality of spline curve layers to obtain a spline surface composed of a plurality of spline curve layers.
[0044] According to another feature of the present invention, the prediction model is a model further trained to classify cross-sectional views of an ultrasound image using an ultrasound image as input, and the processor may be further configured to classify cross-sectional views of a received ultrasound image using the prediction model, segment a target heart region for an ultrasound image corresponding to a classified cross-sectional view, determine a motion vector field for a target heart region, and determine a strain quantification parameter corresponding to a classified view.
[0045] According to another feature of the present invention, the target cardiac region is LA, and the processor may be further configured to determine a strain curve for LA based on a motion vector field, and to determine a quantitative parameter for LA based on the strain curve.
[0046] In order to solve the problem described above, a system for providing information on strain quantification according to an embodiment of the present invention is provided.
[0047] The above system includes an internal memory configured to store a prediction model learned to divide a target heart region based on a divided target heart region and to determine a motion vector field based on a cardiac ultrasound image including a target heart region of an object as input, and to access the internal memory, and to use the prediction model to determine a motion vector field for a target heart region within a received cardiac ultrasound image and to determine a strain quantitative parameter based on the motion vector field, wherein the target heart region is at least one of LV (Left Ventricle), RV (Right Ventricle), LA (Left Atrium), and RA (Right Atrium).
[0048] Specific details of other embodiments are included in the detailed description and drawings. Effects of the invention
[0049] The present invention can provide an information provision system using an artificial neural network-based prediction model capable of analyzing various strain quantitative parameters, including not only the systolic function of the left ventricle but also the measurement of left atrial volume and further evaluation of left ventricular diastolic function.
[0050] Accordingly, the present invention can overcome the limitations of conventional strain quantitative analysis methods, which are limited to measuring the size and volume of the left ventricle and calculating parameters such as LV ejection fraction to evaluate left ventricular contractile function.
[0051] In particular, the present invention provides an information providing system capable of predicting motion vector fields between adjacent frames for cardiac ultrasound images, thereby providing highly reliable information for strain quantification.
[0052] Accordingly, the present invention can overcome the limitations of conventional information provision systems based on strain (or strain rate) analysis that only consider regional characteristics.
[0053] The present invention can overcome the limitations of conventional information provision systems that require the application of motion noise post-processing algorithms because they rely on image quality and expert experience in strain analysis of cardiac ultrasound images and consider only local features.
[0054] Furthermore, the present invention enables motion estimation of a heart structure without folding by establishing an ROI based on an intermediate layer of the inner wall for the target area of the heart structure and generating a spline curve.
[0055] Accordingly, the present invention provides a new information provision system based on spatiotemporal motion modeling, which allows for accurate evaluation of cardiac muscle movement, thereby enabling the provision of not only visual information of the dysfunctional area but also highly accurate quantitative strain information.
[0056] Furthermore, the present invention can solve the problem of errors occurring in manual measurement of strain by automating a procedure performed through multiple steps using a single model to determine quantitative strain parameters.
[0057] In other words, the present invention can contribute to the early diagnosis and good treatment prognosis of cardiopulmonary diseases by providing information on strain quantification.
[0058] The effects according to the present invention are not limited to those exemplified above, and a wider variety of effects are included within the present invention. Brief explanation of the drawing
[0059] FIG. 1 is a schematic diagram illustrating an information providing system using cardiac ultrasound images according to an embodiment of the present invention. FIG. 2a is a schematic diagram illustrating a device for providing information on strain quantification according to an embodiment of the present invention. FIG. 2b is a schematic diagram illustrating a user device that receives information from a device for providing information on strain quantification according to an embodiment of the present invention. FIGS. 3a to 3k illustrate a schematic flowchart for explaining a method for providing information on strain quantification based on a motion vector field in a device for providing information on strain quantification according to an embodiment of the present invention, and an exemplary procedure for determining a motion vector field. FIG. 4 illustrates the structure of a prediction model applied to various embodiments of the present invention. FIG. 5 illustrates, exemplarily, the modeling procedure of a prediction model applied to various embodiments of the present invention. FIGS. 6a to 6d illustrate the evaluation results of a prediction model used in a device for providing information on strain quantification according to various embodiments of the present invention in LV. FIG. 7 illustrates the evaluation results of a prediction model used in a device for providing information on strain quantification according to various embodiments of the present invention in LA and LV. FIG. 8 illustrates the evaluation results of a prediction model used in a device for providing information on strain quantification according to various embodiments of the present invention in RV to LV. Specific details for implementing the invention
[0060] The advantages and features of the present invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. In connection with the description of the drawings, similar reference numerals may be used for similar components.
[0061] In this document, expressions such as "have," "can have," "include," or "can include" refer to the existence of the relevant feature (e.g., numerical values, functions, actions, or components, etc.) and do not exclude the existence of additional features.
[0062] In this document, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.
[0063] Expressions such as "first," "second," "first," or "second" used in this document may modify various components regardless of order and / or importance, and are used merely to distinguish one component from another without limiting such components. For example, the first user device and the second user device may represent different user devices regardless of order or importance. For example, without departing from the scope of rights set forth in this document, the first component may be named the second component, and similarly, the second component may be renamed the first component.
[0064] Where it is stated that a certain component (e.g., a first component) is "(operatively or communicatively) coupled with" or "connected to" another component (e.g., a second component), it should be understood that the said certain component may be directly connected to the said other component or connected through another component (e.g., a third component). On the other hand, where it is stated that a certain component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between the said certain component and the said other component.
[0065] As used in this document, the expression “configured to” may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” does not necessarily mean only “specifically designed to” in hardware. Instead, in some situations, the expression “device configured to” may mean that the device is “capable of” together with other devices or components. For example, the phrase “a processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing said operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or an application processor) capable of performing said operations by executing one or more software programs stored in a memory device.
[0066] The terms used in this document are used merely to describe specific embodiments and are not intended to limit the scope of other embodiments. Singular expressions may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by those skilled in the art described in this document. Terms used in this document that are defined in general dictionaries may be interpreted as having the same or similar meaning as they have in the context of the relevant technology, and are not to be interpreted in an ideal or overly formal sense unless explicitly defined in this document. In some cases, even terms defined in this document may not be interpreted to exclude the embodiments of this document.
[0067] The features of each of the various embodiments of the present invention may be combined or combined with one another, either partially or wholly, and as will be fully understood by those skilled in the art, various technical interlocking and operation are possible, and each embodiment may be implemented independently of one another or together in an interlocking relationship.
[0068] For clarity in the interpretation of this specification, the terms used in this specification are defined below.
[0069] As used in this specification, the term "cardiac ultrasound image" may refer to an image that non-invasively visualizes the structure and function of the heart using ultrasound technology.
[0070] At this time, the cardiac ultrasound image may be a two-dimensional cardiac ultrasound image, but is not limited to this and may be a three-dimensional cardiac ultrasound image.
[0071] Furthermore, the cardiac ultrasound image may be a periodic cardiac ultrasound image (or video) composed of multiple frames.
[0072] In various embodiments, the cardiac ultrasound image may be a B-mode ultrasound image, but is not limited thereto.
[0073] Meanwhile, in cardiac ultrasound images, a motion vector field between adjacent frames may be determined for each frame of the image according to a method for providing information on strain quantification according to an embodiment of the present invention.
[0074] According to another feature of the present invention, the cardiac ultrasound image may be a time-series echocardiogram image having a sequence of 2 beats.
[0075] For example, a 2-bit echocardiogram may include frames of end systole (ES) and end diastole (ED), and furthermore, may be a time-series echocardiogram including mid (ES-ED) frames between end systole and end diastole, and mid (ED-ES) frames between end diastole and end systole.
[0076] As used in this specification, the term "target cardiac region" may refer to a region of the anatomical structures of the heart in a cardiac ultrasound image.
[0077] In one embodiment of the present invention, the target cardiac region may be one region selected from the right ventricle anterior wall, right ventricle (RV), anterior wall of aorta, aorta, posterior wall of aorta, left atrium (LA), and posterior wall of left atrium.
[0078] The term "motion vector field" as used in this specification refers to a vector field which is a space expressed by a vector function, and may be a plurality of vectors that present the direction and magnitude according to the movement of the fine muscles of an organ within an image (or frame).
[0079] More specifically, within this specification, the motion vector field may refer to vectors for each of a plurality of pixels forming a target heart region in a frame of a cardiac ultrasound image. Accordingly, by determining the motion vector field between adjacent frames, strain analysis based on the global structural features of the organ may be possible.
[0080] As used herein, the term "prediction model" may be a model trained to take a cardiac ultrasound image as input, segment a target heart region, and determine a motion vector field based on the segmented target heart region.
[0081] According to a feature of the present invention, the prediction model may be a model trained to take a cardiac ultrasound image as input, classify a cross-sectional view of the ultrasound image, segment the classified target heart region, and determine a motion vector field based on the segmented target heart region.
[0082] Here, the prediction model can classify at least one of the following for a cardiac ultrasound image: a B-mode PLAX (Parasternal Long-Axis) section, a B-mode PSAX (Parasternal Short-Axis) AV (Aortic Valve) level section, a B-mode PSAX MV (Mitral Valve) level section, a B-mode A4C (Apical 4-Chamber) section, a B-mode A2C (Apical 2-Chamber) section, a B-mode A3C (Apical 3-Chamber) section, a B-mode Subcostal 4-Chamber section, a B-mode Suprasternal section, a B-mode LVOT (Left Ventricular Outflow Tract) section, and a B-mode RV (Right Ventricular) section.
[0083] However, it is not limited to this, and the prediction model may be a model trained to classify cross-sections corresponding to M-modes and Doppler modes.
[0084] Furthermore, in various embodiments of the present invention, the prediction model may be trained to distinguish the direction of the cardiac ultrasound image as right or left.
[0085] In other words, the prediction model may be a hybrid model combining various modules that support strain quantification and the automation of image processing.
[0086] In various embodiments of the present invention, the prediction model may comprise a cross-sectional view classification unit that classifies cross-sectional views for an input cardiac ultrasound image, a target heart region division unit that divides a target heart region, and a motion estimation unit that estimates a motion vector field.
[0087] However, it is not limited to this, and the prediction model may consist of a wider variety of combinations of modules that provide information on strain quantification.
[0088] According to another embodiment of the present invention, the prediction model may be a model trained to estimate a motion vector field based on adjacent frames.
[0089] In other words, the prediction model learns the spatiotemporal characteristics of cardiac ultrasound images and can estimate the motion vector field relative to adjacent frames for each of multiple frames.
[0090] According to another feature of the present invention, a prediction model can be trained to classify two classes of motion vector fields corresponding to the x-axis and y-axis, respectively, in the forward direction of the frame of the image, using a 2D cardiac ultrasound image as input.
[0091] At this time, based on the output value in forward motion, motion vector fields corresponding to the x-axis and y-axis of the backward direction of the video frame can be obtained.
[0092] For example, parameters can be shared in a layer prior to the output layer of a prediction model, so that when a reverse sequence is applied to forward motion, it may be possible to obtain motion vector fields corresponding to the x-axis and y-axis respectively for reverse motion.
[0093] In other words, this makes it possible to estimate motion in both directions of the frame.
[0094] However, it is not limited to this, and the prediction model may be trained to classify four classes of motion vector fields corresponding to the forward and reverse x-axis and y-axis of the frame, respectively.
[0095] According to another embodiment of the present invention, the prediction model may be a model trained to estimate motion using information on multi-scale resolutions by deriving the correlation between frames corresponding to the resolution for each of a plurality of frames having a plurality of resolutions and integrating them.
[0096] In other words, this allows for the effective integration of information collected at various resolutions, making more precise motion estimation possible.
[0097] According to another embodiment of the present invention, the prediction model may be a model trained to determine feature maps for a plurality of frames corresponding to a large resolution to obtain information about the context, and to estimate motion together with the context information and the motion features of the aforementioned multi-scale resolution.
[0098] Accordingly, through motion estimation with fused context features, reliable motion estimation can be achieved by clearly distinguishing between the target heart region and the background region (noise).
[0099] In various embodiments of the present invention, the motion estimation unit of the prediction model may be subject to unsupervised learning-based training. Furthermore, the target heart region segmentation unit and the cross-section classification unit may each be subject to supervised learning-based training in which a specific region within a specific frame or image and the corresponding cross-section are labeled.
[0100] In this case, the prediction model may be a semi-supervised learning model that can be selectively unsupervised or supervised depending on the learning objective, such as the prediction of motion vector fields and / or the segmentation of regions and / or the classification of cross-sections.
[0101] In another embodiment of the present invention, the prediction model may be a model in which geometric modeling is performed to determine a spline curve using spline mathematical techniques to estimate motion for a target heart region.
[0102] As used in this specification, the term "spline curve" refers to a smooth curve composed of data points that can visually represent a target heart region in a cardiac ultrasound image.
[0103] In various embodiments of the present invention, the spline curve may be at least one of a B-spline curve, a Catmull-Rom spline curve, a Bezier curve, a NURBS (Non-Uniform Rational B-Splines) curve, a Hypotrochoid curve, a Hermite spline, and a Parametric curve.
[0104] Preferably, the spline curve may be a B-spline curve, but is not limited thereto.
[0105] In various embodiments of the present invention, a spline surface composed of layers including a time concept for a spline curve can be created.
[0106] In more diverse embodiments of the present invention, a wall is determined within a target heart region, an intermediate layer is determined with respect to the wall, the intermediate layer is extended to determine a Region of Interest (ROI), and a spline curve for the determined ROI may be generated.
[0107] Here, when the ROI is expanded by determining the endocardium and epicardium layers of the heart based on the middle layer, it is possible to analyze changes in the ROI with high accuracy, thereby providing reliable motion estimation results.
[0108] In other words, since layered speckle tracking, which divides the heart muscle into multiple layers and allows for the individual analysis of deformation in each layer, is possible, the function of the heart muscle can be evaluated more precisely.
[0109] In more diverse embodiments of the present invention, the generated spline curve may be post-processed based on curvature or weight.
[0110] In a specific example, in the case of the LV, since distortion of the spline curve may occur at the annulus point, which is the location of the annulus, a ring-shaped structure of the heart valve, a correction may be performed by excluding data points with curvature above a predetermined level through curvature-based cutting.
[0111] In another embodiment of the LV, the apex, which is the lowest part of the heart, has a pointed shape, and weighted B-Spline Curve Smoothing can be performed to reflect its structural features well. More specifically, by applying high weights to data points to reduce the influence of the smooth penalty term, it is possible to achieve a smooth and detailed representation of the severely curved apex.
[0112] This allows for clearer contouring and motion estimation of target cardiac regions, such as the LV or LV wall.
[0113] According to one embodiment of the present invention, the prediction model may be a model trained to estimate motion based on a 2D cardiac ultrasound image having a 2-bit sequence labeled for end systole (ES), end diastole (ED), mid between end systole and end diastole (ES-ED), and mid between end diastole and end systole (ED-ES).
[0114] In various embodiments, the prediction model, meanwhile, may be based on at least one algorithm selected from DenseNet-121, U-net, VGG net, DenseNet, FCN (Fully Convolutional Network) having an encoder-decoder structure, DNN (deep neural network) such as SegNet, DeconvNet, DeepLAB V3+, Transformer such as Lawin+, SegFormer, Swin, SqueezeNet, AlexNet, ResNet18, MobileNet-v2, GoogLeNet, ResNet-v2, ResNet50, RetinaNet, ResNet101, Inception-v3, HRNet, ResNeXt, and EfficientNet.
[0115] Furthermore, the prediction model may be an ensemble model based on at least two of the aforementioned algorithm models. However, it is not limited to this.
[0116] The term “strain quantitative parameter” as used in this specification is a strain parameter used to evaluate cardiac function and may include information on longitudinal deformation of the heart muscle.
[0117] At this time, the strain quantification parameter may be at least one of longitudinal strain, which indicates the degree of deformation in the longitudinal direction of the heart muscle; radial strain, which indicates the degree of deformation in the radial direction of the heart muscle; and circumferential strain, which indicates the degree of deformation in the circumferential direction of the heart muscle.
[0118] In various embodiments, the strain quantification parameter may be a length-type strain, but is not limited thereto.
[0119] For example, the strain quantification parameter may be a global longitudinal strain that includes a local minimum between the ED (End-Diastole) and AVC (Aortic Valve Closure) in the cardiac cycle, a global minimum between the ED and AVC, and a minimum throughout the cardiac cycle.
[0120] According to another embodiment of the present invention, a strain curve may be used to determine a strain quantitative parameter for LA.
[0121] Here, the strain curve for LA can represent storage (Rervoir), conduction (Conduit), and contraction (Contraction).
[0122] This mechanism can be determined by extracting pre-A time based on ECG (Electrocardiogram) data, but the accuracy may be very low.
[0123] Accordingly, in various embodiments of the present invention, a strain quantitative parameter based on a strain curve can be determined, thereby providing quantitative information on LA strain with higher accuracy.
[0125] Hereinafter, with reference to FIGS. 1 and FIGS. 2a to 2b, a device for providing information on strain quantification according to various embodiments of the present invention will be described in detail.
[0126] FIG. 1 illustrates a system for providing information on cardiac strain quantification using a device for providing information on cardiac strain quantification according to an embodiment of the present invention. FIG. 2a illustrates an exemplary configuration of a medical device receiving information on cardiac strain quantification according to an embodiment of the present invention. FIG. 2b illustrates an exemplary configuration of a server for providing information on cardiac strain quantification according to an embodiment of the present invention.
[0127] First, referring to FIG. 1, the information providing system (1000) may be a system configured to provide information related to cardiac strain quantification based on cardiac ultrasound images of an object. In this case, the information providing system (1000) may be composed of a medical device (100) that receives information related to cardiac strain quantification, an ultrasound imaging diagnostic device (200) that provides cardiac ultrasound images, and an information providing server (300) that generates information related to cardiac strain quantification based on the received cardiac ultrasound images.
[0128] In various embodiments of the present invention, an information providing server (300) may be mounted on an ultrasound imaging diagnostic device (200), and in this case, various information related to measurements may be displayed on a display unit (not shown) of the ultrasound imaging diagnostic device (200).
[0129] That is, the user can use an ultrasound imaging diagnostic device (200) to check information related to cardiac strain quantification at the same time as the diagnosis.
[0130] In various embodiments, the medical device (100) is an electronic device that provides a user interface for displaying information related to cardiac strain quantification, and may include at least one of a smartphone, a tablet PC (Personal Computer), a laptop and / or a PC.
[0131] The medical device (100) receives the results of the quantitative analysis of cardiac strain for an object from the information provision server (300), and may display the received results through a display unit (not shown).
[0132] The information providing server (300) may include a general-purpose computer, laptop, and / or data server, etc., that performs various operations to determine information related to cardiac strain quantification based on cardiac ultrasound images provided from an ultrasound imaging diagnostic device (200), such as an ultrasound diagnostic device. In this case, the information providing server (300) may be a device for accessing a web server that provides web pages or a mobile web server that provides mobile websites, but is not limited thereto.
[0133] More specifically, the information providing server (300) receives a cardiac ultrasound image from an ultrasound imaging diagnostic device (200), performs segmentation of a target heart region and motion estimation within the received cardiac ultrasound image, and performs calculations to determine a strain quantitative parameter based on this. At this time, the information providing server (300) can perform various functions for cardiac strain quantitative analysis from the cardiac ultrasound image using a prediction model.
[0134] The information providing server (300) can provide cardiac strain quantitative parameters, furthermore, masks for segmented target cardiac regions, classified cross-sections, etc. to the medical device (100).
[0135] In this way, the information provided from the information provision server (300) may be provided as a web page through a web browser installed on the medical device (100), or provided in the form of an application or a program. In various embodiments, such data may be provided in a form included in a platform in a client-server environment.
[0136] Next, with reference to FIGS. 2a and 2b, the components of the information providing server (300) of the present invention will be described in detail.
[0137] First, referring to FIG. 2a, the medical device (100) may include a memory interface (110), one or more processors (120) and a peripheral interface (130). Various components within the medical device (100) may be connected by one or more communication buses or signal lines.
[0138] The memory interface (110) is connected to the memory (150) and can transmit various data to the processor (120). Here, the memory (150) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain data.
[0139] In various embodiments, memory (150) may store at least one of an operating system (151), a communication module (152), a graphical user interface module (GUI) (153), a sensor processing module (154), a telephone module (155), and an application module (156). Specifically, the operating system (151) may include instructions for processing basic system services and instructions for performing hardware operations. The communication module (152) may communicate with at least one of one or more other devices, computers, and servers. The graphical user interface module (GUI) (153) may process a graphical user interface. The sensor processing module (154) may process sensor-related functions (e.g., processing voice input received using one or more microphones (192)). The telephone module (155) may process telephone-related functions. The application module (156) may perform various functions of a user application, such as electronic messaging, web browsing, media processing, exploration, imaging, and other processing functions. In addition, the medical device (100) may store one or more software applications (156-1, 156-2) (e.g., information provision applications) associated with any one type of service in memory (150).
[0140] In various embodiments, the memory (150) can store a digital assistant client module (157) (hereinafter, DA client module), and accordingly, can store commands and various user data (158) for performing client-side functions of the digital assistant.
[0141] Meanwhile, the DA client module (157) can obtain voice input, text input, touch input and / or gesture input from a user through various user interfaces (e.g., I / O subsystem (140)) provided in the medical device (100).
[0142] Additionally, the DA client module (157) can output data in the form of audiovisual and tactile signals. For example, the DA client module (157) can output data consisting of a combination of at least two of voice, sound, notifications, text messages, menus, graphics, videos, animations, and vibrations. Furthermore, the DA client module (157) can communicate with a digital assistant server (not shown) using a communication subsystem (180).
[0143] In various embodiments, the DA client module (157) may collect additional information about the surrounding environment of the medical device (100) from various sensors, subsystems, and peripheral devices to construct a context associated with user input. For example, the DA client module (157) may provide context information along with user input to a digital assistant server to infer the user's intent. Here, context information that may accompany user input may include sensor information, e.g., lighting, ambient noise, ambient temperature, images of the surrounding environment, video, etc. As another example, context information may include the physical state of the medical device (100) (e.g., device orientation, device location, device temperature, power level, speed, acceleration, motion pattern, cellular signal strength, etc.). As another example, the situation information may include information related to the software status of the medical device (100) (e.g., processes running on the medical device (100), installed programs, past and present network activity, background services, error logs, resource usage, etc.).
[0144] In various embodiments, the memory (150) may include additional or deleted instructions, and furthermore, the medical device (100) may include additional configurations in addition to the configuration shown in 2a, or exclude some configurations.
[0145] The processor (120) can control the overall operation of the medical device (100) and can execute various commands to implement an interface that provides information related to cardiac strain quantification by running an application or program stored in memory (150).
[0146] The processor (120) may correspond to a computing device such as a CPU (Central Processing Unit) or an AP (Application Processor). Additionally, the processor (120) may be implemented in the form of an integrated chip (IC), such as a System on Chip (SoC) that integrates various computing devices such as a Neural Processing Unit (NPU).
[0147] The peripheral interface (130) is connected to various sensors, subsystems, and peripheral devices and can provide data to enable the medical device (100) to perform various functions. Here, it can be understood that the medical device (100) performs a function by the processor (120).
[0148] The peripheral interface (130) may receive data from a motion sensor (160), a light sensor (light sensor) (161), and a proximity sensor (162), thereby enabling the medical device (100) to perform orientation, light, and proximity detection functions, etc. As another example, the peripheral interface (130) may receive data from other sensors (163) (positioning system—GPS receiver, temperature sensor, biometric sensor), thereby enabling the medical device (100) to perform functions related to other sensors (163).
[0149] In various embodiments, the medical device (100) may include a camera subsystem (170) connected to a peripheral interface (130) and an optical sensor (171) connected thereto, thereby enabling the medical device (100) to perform various shooting functions such as taking photos and recording video clips.
[0150] In various embodiments, the medical device (100) may include a communication subsystem (180) connected to a peripheral interface (130). The communication subsystem (180) is composed of one or more wired / wireless networks and may include various communication ports, radio frequency transceivers, and optical transceivers.
[0151] In various embodiments, the medical device (100) includes an audio subsystem (190) connected to a peripheral interface (130), and the audio subsystem (190) includes one or more speakers (191) and one or more microphones (192), so that the medical device (100) can perform voice-operated functions, such as voice recognition, voice cloning, digital recording, and telephone functions.
[0152] In various embodiments, the medical device (100) may include an I / O subsystem (140) connected to a peripheral interface (130). For example, the I / O subsystem (140) may control a touch screen (143) included in the medical device (100) through a touch screen controller (141). As an example, the touch screen controller (141) may detect user contact and movement or interruption of contact and movement using any one of a plurality of touch sensing technologies, such as capacitive, resistive, infrared, surface acoustic wave technology, proximity sensor array, etc. As another example, the I / O subsystem (140) may control other input / control devices (144) included in the medical device (100) through other input controller(s) (142). As an example, other input controller(s) (142) can control one or more pointer devices such as buttons, rocker switches, thumb wheels, infrared ports, USB ports and styluses.
[0153] Next, referring to FIG. 2b, the information providing server (300) may include a communication interface (310), memory (320), I / O interface (330), and a processor (340), and each component may communicate with one or more communication buses or signal lines.
[0154] The communication interface (310) can be connected to the medical device (100) and the ultrasound imaging diagnostic device (200) via a wired / wireless communication network to exchange data. For example, the communication interface (310) can receive cardiac ultrasound images from the ultrasound imaging diagnostic device (200) and transmit information regarding cardiac strain quantification, such as strain quantification parameters or data obtained during the process of determining them (e.g., segmented masks), to the medical device (100).
[0155] Meanwhile, a communication interface (310) that enables the transmission and reception of such data includes a communication pod (311) and a wireless circuit (312), wherein the wired communication port (311) may include one or more wired interfaces, for example, Ethernet, Universal Serial Bus (USB), FireWire, etc. Additionally, the wireless circuit (312) may transmit and receive data with an external device via an RF signal or an optical signal. Furthermore, wireless communication may use at least one of a plurality of communication standards, protocols, and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.
[0156] The memory (320) can store various data used in the information provision server (300). For example, the memory (320) can store cardiac ultrasound images, or predictive models modeled for quantitative analysis of cardiac strain from cardiac ultrasound images, or various modules constituting the predictive models, and furthermore, parameters for learning these predictive models.
[0157] In various embodiments, the memory (320) may include a volatile or non-volatile recording medium capable of storing various data, commands and information. For example, the memory (320) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain data.
[0158] In various embodiments, the memory (320) may store at least one configuration of an operating system (321), a communication module (322), a user interface module (323), and one or more applications (324).
[0159] An operating system (321) (e.g., an embedded operating system such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) may include various software components and drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and may support communication between various hardware, firmware, and software components.
[0160] The communication module (323) can support communication with other devices through the communication interface (310). The communication module (320) may include various software components for processing data received by the wired communication port (311) or wireless circuit (312) of the communication interface (310).
[0161] The user interface module (323) can receive user requests or inputs from a keyboard, touch screen, microphone, etc. through the I / O interface (330) and provide a user interface on the display.
[0162] The application (324) may include a program or module configured to be executed by one or more processors (330). Here, the application for providing information related to cardiac strain quantification may be implemented on a server farm.
[0163] The I / O interface (330) can connect at least one of an input / output device (not shown) of the information providing server (300), such as a display, keyboard, touch screen, and microphone, to the user interface module (323). The I / O interface (330) can receive user input (e.g., voice input, keyboard input, touch input, etc.) together with the user interface module (323) and process commands based on the received input.
[0164] The processor (340) is connected to the communication interface (310), memory (320), and I / O interface (330) to control the overall operation of the information provision server (300) and can perform various commands for providing information through applications or programs stored in the memory (320).
[0165] The processor (340) may correspond to a computing device such as a CPU (Central Processing Unit) or an AP (Application Processor). Additionally, the processor (340) may be implemented in the form of an integrated chip (IC), such as a System on Chip (SoC) that integrates various computing devices. Alternatively, the processor (340) may include a module for computing artificial neural network models, such as a Neural Processing Unit (NPU).
[0167] Hereinafter, a method for providing information on strain quantification according to an embodiment of the present invention will be described with reference to FIGS. 3a to 3k.
[0168] FIGS. 3a to 3k illustrate a schematic flowchart for explaining a method for providing information on strain quantification based on a motion vector field in a device for providing information on strain quantification according to an embodiment of the present invention, and an exemplary procedure for determining a motion vector field.
[0169] First, referring to FIG. 3a, a cardiac ultrasound image of an object is received according to a method for providing information on strain quantification according to an embodiment of the present invention (S310). Then, a target heart region is segmented using a prediction model (S320), a motion vector field is determined (S330), and finally, strain quantification parameters are determined based on the motion vector field (S340).
[0170] More specifically, in the step (S310) where the cardiac ultrasound image is received, a 2D cardiac ultrasound image may be received.
[0171] For example, in the step (S310) where a cardiac ultrasound image is received, a 2D ultrasound image having a 2-bit sequence may be received.
[0172] According to another feature of the present invention, in the step (S310) where a cardiac ultrasound image is received, a cardiac ultrasound video consisting of a plurality of frames may be received.
[0173] At this time, the received cardiac ultrasound image may have at least one of a B-mode PLAX (Parasternal Long-Axis) section, a B-mode PSAX (Parasternal Short-Axis) AV (Aortic Valve) level section, a B-mode PSAX MV (Mitral Valve) level section, a B-mode A4C (Apical 4-Chamber) section, a B-mode A2C (Apical 2-Chamber) section, a B-mode A3C (Apical 3-Chamber) section, a B-mode Subcostal 4-Chamber section, a B-mode Suprasternal section, a B-mode LVOT (Left Ventricular Outflow Tract) section, and a B-mode RV (Right Ventricular) section. However, it is not limited thereto, and the cardiac ultrasound image may have sections corresponding to M-mode and Doppler mode.
[0174] Meanwhile, referring to FIG. 3b together, according to various embodiments of the present invention, after the step (S310) of receiving an ultrasound image, the step (S350) of classifying a cross-sectional view of the received cardiac ultrasound image using a prediction model may be optionally performed.
[0175] Next, the target heart region for the image with the classified cross-section is segmented using a prediction model (S320).
[0176] According to a feature of the present invention, the target cardiac region may be one region selected from the right ventricle anterior wall, right ventricle (RV), anterior wall of aorta, aorta, posterior wall of aorta, left atrium (LA), and posterior wall of left atrium (LA).
[0177] According to another feature of the present invention, a mask for a target heart region segmented by a prediction model may be output and provided to a medical device (100).
[0178] Next, in the step (S330) where the motion vector field is determined, the motion vector field for the target heart region of the cardiac ultrasound image is determined by the prediction model.
[0179] That is, the prediction model applied to various embodiments of the present invention may be a hybrid model combining various modules that support strain quantification and the automation of image processing processes.
[0180] Referring to FIG. 3c, in one embodiment of the present invention, the prediction model may be composed of a module that performs a plurality of functions, including a motion estimation unit (410) that estimates a motion vector field based on an input cardiac ultrasound image, a target heart region segmentation unit (420) that segments a target heart region, and a cross-sectional view classification unit (430) that classifies cross-sectional views.
[0181] However, it is not limited to this, and prediction models can be composed of more diverse combinations of modules depending on the purpose.
[0182] For example, the prediction model may consist of a motion estimation unit (410) and a target heart region segmentation unit (420), and the cardiac ultrasound image may be received by a cross-sectional classification model separated independently of the prediction model, the cross-sectionals may be classified, and the result thereof may be transmitted to the prediction model.
[0183] The structural features of the prediction model used in various embodiments of the present invention will be described in detail later.
[0184] Referring again to FIG. 3a, in the step (S330) in which a motion vector field is determined in one embodiment of the present invention, for at least one frame selected among a plurality of frames, a motion vector field can be estimated based on an adjacent frame.
[0185] For example, in the step (S330) where the motion vector field is determined, if the motion field is estimated in the forward direction, the motion vector field of the second frame (I2) following the first frame can be determined based on the motion field for the first frame (I1). Furthermore, if the motion field is estimated in the backward direction, the motion vector field of the third frame (I0) prior to the first frame can be determined based on the motion field for the first frame (I1).
[0186] At this time, optionally, if a reverse sequence is applied to the forward motion, it may be possible to obtain motion vector fields corresponding to the x-axis and y-axis respectively for the reverse motion. That is, through this, it may be possible to estimate motion for both directions of the frame.
[0187] In another embodiment of the present invention, the step (S330) of determining the motion vector field can be performed based on multi-scale resolution.
[0188] More specifically, referring to FIG. 3d, a correlation between multiple frames having a first resolution is determined using a prediction model (S330-1), and a correlation between multiple frames having a second resolution higher than the first resolution is determined (S330-2). Then, based on the correlations for the first resolution and the second resolution, respectively, motion features are integrated (S330-3), and a motion vector field is determined based on the integrated motion features (S330-4).
[0189] Referring together with FIG. 3e, according to the step (S330-1) in which a correlation is determined for a plurality of frames having a first resolution, a feature of a plurality of frames having a first resolution is determined in the motion estimation unit (410) of the prediction model (412), and according to the step (S330-2) in which a correlation is determined for a plurality of frames having a second resolution, a feature of a plurality of frames having a second resolution is determined (414).
[0190] At this time, bidirectional feature extraction between pixels of adjacent frames is performed in a plurality of frames having a first resolution and correlations are determined (412), and bidirectional feature extraction between pixels of adjacent frames is performed in a plurality of frames having a second resolution of a higher resolution and correlations are determined (414).
[0191] Next, in the motion feature integration step (S330-3), motion features for the two resolutions are integrated.
[0192] Next, in the step where the motion vector field is determined (S330-4), after the context data of the feature map is extracted at a second resolution of high resolution, the context data is used to estimate the motion together with the motion features of the integrated multi-scale resolution.
[0193] At this time, through motion estimation with fused context features, reliable and precise motion estimation can be achieved by clearly distinguishing between the target heart region and the background region (noise).
[0194] Referring again to FIG. 3a, in the step (S330) where a motion vector field is determined in one embodiment of the present invention, motion estimation may be possible by determining the contouring and spline curve of the ROI-based target heart region using a prediction model in which geometric modeling is performed.
[0195] More specifically, referring to FIG. 3f, a heart wall is determined within a segmented target heart region (S330-5), an intermediate layer is determined for the heart wall (S330-6), the intermediate layer is extended to determine an ROI for motion estimation (S330-7), and a spline curve for the ROI is determined (S330-8).
[0196] Referring to FIG. 3g, in the step (S330-7) where the ROI is determined, the intermediate layer is extended to determine the ROI that includes the endocardium and epicardium layers of the heart. As a result, a highly accurate analysis is possible in the motion analysis of the ROI of the target heart region, thereby providing a highly reliable motion estimation result.
[0197] Returning to Fig. 3f, in the step where the spline curve is determined (S330-8), a spline curve is generated, which is a curve composed of data points.
[0198] In other words, since layered speckle tracking, which divides the heart muscle into multiple layers and allows for the individual analysis of deformation in each layer, is possible, the function of the heart muscle can be evaluated more precisely.
[0199] Meanwhile, in the step where the spline curve is determined (S330-8), at least one spline curve among a B-Spline curve, a Catmull-Rom Spline curve, a Bezier curve, a NURBS (Non-Uniform Rational B-Splines) curve, a Hypotrochoid curve, and a Parametric curve may be generated.
[0200] Preferably, the spline curve may be a B-spline curve, but is not limited thereto.
[0201] Referring to FIG. 3f and FIG. 3h together, according to various embodiments of the present invention, in the step (S330-8) where a spline curve is determined, a spline surface consisting of a plurality of layers including a time concept for the spline curve can be generated. That is, a spline surface can be generated according to the generation of a spline curve multiple times.
[0202] At this time, the generated spline surface may be post-processed through smoothing.
[0203] According to another feature of the present invention, after the step (S330-8) in which a spline curve is determined, post-processing can be performed on the generated spline curve based on curvature or weight.
[0204] More specifically, referring to (a) and (b) of FIG. 3i, in the case of LV, since distortion of the spline curve may occur at the annulus point, which is the location of the annulus, a ring-shaped structure of the heart valve, a correction may be performed by excluding data points with curvature above a predetermined level through curvature-based cutting.
[0205] Referring to Fig. 3j, the apex, the lowest part of the heart in the LV, has a pointed shape, and weighted B-spline curve smoothing can be performed to accurately reflect its structural features. More specifically, different weights can be assigned to each data point, but with lower weights specifically applied to the apex, so that these data points have less influence than other regions during the smoothing process. Consequently, the curvature of the region around the annulus point can be smoothed out while the curvature of the apex is maintained (see smoothing with weights).
[0206] Therefore, clearer contouring and motion estimation of target cardiac regions, such as the LV or LV wall, may be possible.
[0207] Referring again to FIG. 3a, in the step (S340) where the strain quantitative parameter is determined, at least one parameter among the longitudinal strain, which indicates the degree of deformation in the longitudinal direction of the heart muscle, the radial strain, which indicates the degree of deformation in the radial direction of the heart muscle, and the circumferential strain, which indicates the degree of deformation in the circumferential direction of the heart muscle, may be determined.
[0208] According to various embodiments of the present invention, a strain curve may be used to determine a strain quantitative parameter for LA.
[0209] More specifically, with reference to Fig. 3j, the strain curve for LA can represent storage (Rervoir), conduction (Conduit), and contraction. These mechanisms can be determined by extracting pre-A time based on ECG (Electrocardiogram) data, but the accuracy may be very low.
[0210] Accordingly, in various embodiments of the present invention, a strain quantitative parameter based on a strain curve can be determined, thereby providing quantitative information on LA strain with higher accuracy.
[0212] According to the information provision method according to the various embodiments of the present invention above, various processes for strain analysis for LV, as well as LA, RV, and RA, can be performed using a geometric modeling-based hybrid prediction model.
[0213] That is, the present invention can provide an information provision system using an artificial neural network-based prediction model capable of analyzing various strain quantitative parameters, including not only the systolic function of the left ventricle but also the measurement of left atrial volume and further evaluation of left ventricular diastolic function.
[0214] Accordingly, the present invention can overcome the limitations of conventional strain quantitative analysis methods, which are limited to measuring the size and volume of the left ventricle and calculating parameters such as LV ejection fraction to evaluate left ventricular contractile function.
[0216] Hereinafter, the structure of a prediction model and a learning method used in various embodiments of the present invention will be described with reference to FIGS. 4 and 5.
[0217] FIG. 4 illustrates the structure of a prediction model applied to various embodiments of the present invention. FIG. 5 illustrates the modeling procedure of a prediction model applied to various embodiments of the present invention.
[0218] First, referring to FIG. 4, the hybrid prediction model (400) may be composed of a plurality of modules, including a motion estimation unit (410) for estimating motion, a target heart region segmentation unit (420) for segmenting the target heart region, and a cross-sectional view classification unit (430).
[0219] More specifically, when a cardiac ultrasound video is input into a target cardiac region segmentation unit (420) of a U-Net structure, features of the left region can be copied and pasted to the same level position of the right region through a skip connection connecting the left block to the right block in the U-shaped segmentation unit. Through this, features lost as the input data becomes smaller during the prediction of the motion vector field can be corrected. Then, a finally segmented mask can be output through upsampling.
[0220] That is, the target heart region segmentation unit (420) of the aforementioned structural features can be modeled to determine and efficiently learn motion vector fields between adjacent frames by considering global structural features and local feature similarities. Accordingly, it is resistant to noise and can provide reliable results for strain analysis regardless of image quality.
[0221] Next, through the target heart region segmentation unit (420), some data from the downsampling process of the input ultrasound image is transmitted to the motion estimation unit (410), and motion feature extraction for multi-scale resolution is performed therefrom. Subsequently, context data corresponding to the large resolution feature map and resolution-specific integrated motion features pass through a B-spline layer to determine the motion vector field for the target heart region.
[0222] In various embodiments of the present invention, data downsampled through the target heart region segmentation unit (420) is transmitted to the cross-sectional view classification unit (430), and based on this, the cross-sectional view classification of the cardiac ultrasound image is classified and the direction of the right or left is classified.
[0223] In other words, by using a single model of this hybrid structure, various processing for motion estimation can be performed on cardiac ultrasound images.
[0224] Accordingly, the present invention can overcome the problem of errors occurring as multiple steps are taken in manual strain measurement by providing information related to strain quantification based on a single model.
[0225] Meanwhile, the structural characteristics of the prediction model are not limited to those mentioned above and can have a wider variety of structures.
[0227] The prediction model used in various embodiments of the present invention can be trained to estimate a motion vector field through cyclic motion modeling.
[0228] More specifically, referring to FIG. 5, the prediction model can use 2-bit cardiac ultrasound images for training, each labeled with End systole (ES), End diastole (ED), mid between End systole and End diastole (ES-ED), and mid between End diastole and End systole (ED-ES) frames.
[0229] More specifically, when there exists a first set of input data in 2-bit units and a second set of input data different from the first set, and when it is to determine the motion vector field of a frame in the forward direction, the prediction model can be trained to determine the motion vector field of the first frame of the newly input second set based on the last frame of the first set. That is, the motion vector field of the first frame of the second set can be determined based on the motion field of the last frame of the first set, which is not '0'.
[0230] Next, if the motion vector field of the frame is to be determined in reverse, the prediction model can be trained to determine the motion vector field of the last frame of the newly input second set based on the first frame of the first set.
[0231] As such, the prediction model applied to various embodiments of the present invention can estimate a motion vector field by considering the spatiotemporal features of a time series of cardiac ultrasound images through bi-directional cyclic motion modeling.
[0232] Meanwhile, bidirectional circular motion modeling can be performed through Equation 1 based on bidirectional circular motion, consistency constraints, and sparse labels for specific frames.
[0233] [Mathematical Formula 1]
[0234]
[0235] Here is the motion vector estimated in the forward direction of frame t, and can mean the motion vector estimated in the inverse direction of frame t. Furthermore, y is the mask label, is a model prediction, and p can mean omega defined as an arbitrary point in image space.
[0236] More specifically, the first formula (Bi-directional cyclic motions) is a loss function that uses forward motion vectors and backward motion vectors to ensure that the (t+1)%T and (T-1)%T images match the current image, and a formula that imposes constraints on the derivatives of each motion vector to be close to zero, and the second formula (Sparse labels) is a segmentation loss that is applied sparsely to the frame 't' where the ground truth (GT) is located.
[0237] Through this modeling, the prediction model can enable bidirectional motion estimation based on recognized frames.
[0238] Meanwhile, in another embodiment of the present invention, the prediction model can be modeled by the following Equation 2 for differential motion.
[0239] [Mathematical Formula 2]
[0240]
[0241] Here, (t) is parameterized over the interval [0,1], and the stop vector field (SVF) can be generated as a path of the diffusional transformation field. Furthermore, the identity transformation from the initial state It has, and SVF(v) can be integrated over the interval [0,1] (t).
[0242] Through this, it is possible to obtain a differentiable and inversely function-containing diffeomorphic deformation field in motion estimation, establish a one-to-one correspondence, and preserve the spatial phase. Furthermore, the folding problem is minimized, and simultaneous forward and inverse modeling becomes possible.
[0243] Meanwhile, in another embodiment of the present invention, the prediction model can be modeled for motion estimation based on a B-spline curve, and the B-spline curve can be defined by the following mathematical formula 3.
[0244] [Mathematical Formula 3]
[0245]
[0246] Here is a basis function, and can be a data point that is a control point.
[0248] Meanwhile, in another embodiment of the present invention, the prediction model may be modeled to perform smoothing on a B-spline curve by the following mathematical formulas 4 and 5.
[0249] [Mathematical Formula 4]
[0250]
[0251] [Mathematical Formula 5]
[0252]
[0253] Here, Equation 4 is a formula that incorporates a penalty term to minimize the curvature of the curve during the B-spline curve fitting process, and Equation 5 may be a formula obtained by expanding Equation 4 into matrix form.
[0254] In particular, the prediction model can be modeled to perform weight-based smoothing on B-spline curves by the following mathematical formula 6.
[0255] [Mathematical Formula 6]
[0256]
[0257] Here, Equation 6 is a formula set to mitigate distortion resulting from the calculation of Equation 4 by increasing the weights of data points in cases where there are places with inherently large curvature, such as the apex of the LV.
[0258] Through such mathematical modeling, the prediction model can accurately estimate the motion of the generated spline curve. Meanwhile, the modeling of the prediction model is not limited to the aforementioned method and can be performed through a wider variety of methods.
[0260] Evaluation: Evaluation of the diagnostic performance of a device for providing information on strain quantification
[0261] Hereinafter, with reference to FIGS. 6a to 6d, FIGS. 7, and FIGS. 8, the evaluation results of a prediction model applied to a device for providing information on strain quantification according to various embodiments of the present invention will be described.
[0262] More specifically, referring to Fig. 6a, the results of deriving GLS strain quantitative parameters for cardiac ultrasound images corresponding to A2C, A3C, and A4C cross-sections of the LV using a prediction model are shown.
[0263] Furthermore, referring to FIGS. 6b to 6d, the evaluation results for motion-based strain quantification are illustrated.
[0264] In all results, the predictive model shows results with a slope close to 1, as the predicted values from various conditions and seeds are very similar to the actual values in both individuals with heart disease (ST-Elevation Myocardial Infarction), individuals with LBBB (Left Bundle Branch Block), and individuals with DCM (Dilated Cardiomyopathy), as well as normal individuals.
[0265] This suggests that the prediction performance of strain quantitative parameters, including GLS, is excellent and stable.
[0266] Next, referring to Figure 7(a), the evaluation results for the LA strain quantitative parameters of STEMI individuals using the prediction model also show excellent prediction performance.
[0267] In contrast, referring to Figures 7 (b) and (c), it appears that the diagnostic performance is somewhat lower when using EchoPAC, a conventional cardiac ultrasound image analysis software, as the predicted value of the LA strain quantitative parameter differs from the actual value.
[0268] Next, referring to Figure 8, the evaluation results for the quantitative parameters of RV strain of PHT (Pulmonary Hypertension) subjects using the prediction model show that the prediction performance has a Pearson correlation coefficient of r of 0.72.
[0270] According to the information provision method according to the various embodiments of the present invention above, various processes for strain analysis for LV, as well as LA, RV, and RA, can be performed using a geometric modeling-based hybrid prediction model.
[0271] That is, the present invention can provide an information provision system using an artificial neural network-based prediction model capable of analyzing various strain quantitative parameters, including not only the systolic function of the left ventricle but also the measurement of left atrial volume and further evaluation of left ventricular diastolic function.
[0272] Accordingly, the present invention can overcome the limitations of conventional strain quantitative analysis methods, which are limited to measuring the size and volume of the left ventricle and calculating parameters such as LV ejection fraction to evaluate left ventricular contractile function.
[0273] In other words, the present invention can contribute to the early diagnosis of diseases and a good treatment prognosis by providing information on strain quantification.
[0275] Although embodiments of the present invention have been described in more detail with reference to the attached drawings, the present invention is not necessarily limited to these embodiments and may be modified in various ways within the scope of the technical spirit of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical spirit of the present invention, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present invention shall be interpreted by the claims below, and all technical spirits within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols
[0276] 100: Medical staff device 110: Memory interface 120: Processor 130: Peripheral Interfaces 140: I / O Subsystem 141: Touchscreen Controller 142: Other Input Controllers 143: Touch screen 144: Other Input Control Devices 150: Memory 151: Operating System 152: Communication Module 153: GUI Module 154: Sensor processing module 155: Phone module 156: Applications 156-1, 156-2: Application 157: Digital Assistant Client Module 158: User data 160: Motion sensor 161: Light sensor 162: Proximity sensor 163: Other sensors 170: Camera subsystem 171: Optical sensor 180: Communications subsystem 190: Audio Subsystem 191: Speaker 192: Microphone 300: Information server 310: Communication Interface 311: Wired communication port 312: Wireless circuit 320: Memory 321: Operating System 322: Communication Module 323: User Interface Module 324: Application 330: I / O interface 340: Processor 1000: Information provision system
Claims
Claim 1 A method for providing information on strain quantification implemented by a processor, comprising: receiving a cardiac ultrasound image including a target heart region of an object; determining a motion vector field for the target heart region within the received cardiac ultrasound image using a prediction model trained to divide the target heart region using the cardiac ultrasound image as input and determine the motion vector field based on the divided target heart region; and determining a strain quantification parameter based on the motion vector field, wherein the step of determining the motion vector field includes determining a spline curve using a spline mathematical technique to estimate motion for the target heart region using the prediction model, and wherein the target heart region is at least one of LV (Left Ventricle), RV (Right Ventricle), LA (Left Atrium), and RA (Right Atrium). Claim 2 A method for providing information on strain quantification according to claim 1, wherein the cardiac ultrasound image is a video consisting of a plurality of frames, and the prediction model is configured to determine a motion vector field based on a frame adjacent to the at least one frame selected among the plurality of frames. Claim 3 A method for providing information on strain quantification according to claim 2, wherein the plurality of frames includes a plurality of frames having a first resolution for the target heart region and a plurality of frames having a second resolution, and the step of determining the motion vector field comprises: a step of determining a correlation for the plurality of frames having the first resolution using the prediction model; a step of determining a correlation for the plurality of frames having the second resolution; a step of integrating motion features based on the correlation for each of the first resolution and the second resolution; and a step of determining the motion vector field based on the integrated motion features. Claim 4 A method for providing information on strain quantification according to claim 3, wherein the first resolution or the second resolution has a resolution greater than the other one, and the step of determining a motion vector field based on the integrated motion features further comprises the step of determining a feature map for a plurality of frames corresponding to the larger resolution, and the step of determining a motion vector field based on the feature map and the integrated motion features. Claim 5 A method for providing information on strain quantification according to claim 2, wherein the plurality of frames includes a first frame for the target heart region and a second frame that is a frame before or after the first frame, and the step of determining the motion vector field includes the step of determining a first motion vector field for the first frame using the prediction model, and the step of estimating a second motion vector field for the second frame based on the first motion vector field. Claim 6 delete Claim 7 A method for providing information on strain quantification, wherein the step of determining the spline curve further comprises: determining a heart wall within the target heart region; determining an intermediate layer for the heart wall; extending the intermediate layer to determine a Region of Interest (ROI); and obtaining a spline curve for the ROI. Claim 8 A method for providing information on strain quantification, wherein the step of determining the spline curve in claim 1 includes the step of determining the plurality of spline curve layers to obtain a spline surface composed of a plurality of spline curve layers. Claim 9 A method for providing information on strain quantification, further comprising the step of correcting the determined spline curve in claim 1. Claim 10 A method for providing information on strain quantification, wherein the step of correcting the spline curve comprises: a step of determining the curvature of the spline curve; and a step of correcting the spline curve to cut the spline curve by excluding data points among the data points forming the spline curve where the curvature is greater than or equal to a predetermined level. Claim 11 A method for providing information on strain quantification, wherein the step of correcting the spline curve further includes the step of smoothing by assigning weights to data points corresponding to specific regions of the target heart region during the process of generating the spline curve. Claim 12 A method for providing information on strain quantification according to claim 1, wherein the prediction model is a model further trained to classify a cross-sectional view of an ultrasound image using a cardiac ultrasound image as input, and the step of determining the motion vector field further comprises: a step of classifying a cross-sectional view for the received ultrasound image using the prediction model; a step of segmenting a target heart region for the ultrasound image corresponding to the classified cross-sectional view; and a step of determining a motion vector field for the target heart region, and the step of determining the strain quantification parameter further comprises a step of determining a strain quantification parameter corresponding to the classified view. Claim 13 A method for providing information on strain quantification, further comprising the step of outputting and providing a mask for the target heart region segmented by the prediction model in claim 1. Claim 14 A method for providing information on strain quantification according to claim 1, wherein the target cardiac region is LA, and the step of determining the strain quantification parameter comprises the step of determining a strain curve for the LA based on the motion vector field, and the step of determining a quantification parameter for the LA based on the strain curve. Claim 15 A device for providing information on strain quantification, comprising a communication unit configured to receive a cardiac ultrasound image including a target cardiac region of an object, and a processor functionally connected to the communication unit, wherein the processor is configured to divide a target cardiac region using a cardiac ultrasound image as input, determine a motion vector field for the target cardiac region within the received cardiac ultrasound image using a prediction model learned to determine a motion vector field based on the divided target cardiac region, and determine a strain quantification parameter based on the motion vector field, and further configured to determine a spline curve using a spline mathematical technique to estimate motion for the target cardiac region using the prediction model, wherein the target cardiac region is at least one of LV, RV, LA, and RA. Claim 16 A device for providing information on strain quantification, wherein, in claim 15, the cardiac ultrasound image is a video consisting of a plurality of frames, and the prediction model is configured to determine a motion vector field based on a frame adjacent to the at least one frame selected among the plurality of frames. Claim 17 A device for providing information on strain quantification, wherein, in claim 16, the plurality of frames comprises a plurality of frames having a first resolution and a plurality of frames having a second resolution for the target heart region, and the processor is further configured to determine a correlation for the plurality of frames having the first resolution and a correlation for the plurality of frames having the second resolution using the prediction model, integrate motion features based on the correlation for each of the first resolution and the second resolution, and determine a motion vector field based on the integrated motion features. Claim 18 A device for providing information on strain quantification, wherein, in claim 17, the first resolution or the second resolution has a resolution greater than the other one, and the processor is further configured to determine a feature map for a plurality of frames corresponding to the greater resolution and to determine a motion vector field based on the feature map and the integrated motion features. Claim 19 A device for providing information on strain quantification, wherein, in claim 16, the plurality of frames includes a first frame for the target heart region and a second frame that is a frame before or after the first frame, and the processor is further configured to determine a first motion vector field for the first frame using the prediction model and to estimate a second motion vector field for the second frame based on the first motion vector field. Claim 20 delete Claim 21 A device for providing information on strain quantification, wherein the processor is further configured to determine a heart wall within the target heart region, determine an intermediate layer with respect to the heart wall, extend the intermediate layer to determine an ROI, and obtain a spline curve for the ROI. Claim 22 In claim 15, the device for providing information on strain quantification is further configured such that the processor determines the plurality of spline curve layers to obtain a spline surface composed of the plurality of spline curve layers. Claim 23 A device for providing information on strain quantification, wherein, in claim 15, the prediction model is a model further trained to classify cross-sectional views of an ultrasound image using a cardiac ultrasound image as input, and the processor is further configured to classify cross-sectional views of the received ultrasound image using the prediction model, segment a target heart region for the ultrasound image corresponding to the classified cross-sectional views, determine a motion vector field for the target heart region, and determine a strain quantification parameter corresponding to the classified view. Claim 24 A device for providing information on strain quantification, wherein, in claim 15, the target cardiac region is LA, and the processor is further configured to determine a strain curve for the LA based on the motion vector field, and to determine a quantitative parameter for the LA based on the strain curve. Claim 25 A system for providing information on cardiac strain quantification, comprising: an internal memory configured to store a prediction model learned to divide a target cardiac region based on a cardiac ultrasound image including a target cardiac region of an object and to determine a motion vector field based on the divided target cardiac region, and a processing unit configured to access the internal memory, and to determine a motion vector field for the target cardiac region within the received cardiac ultrasound image using the prediction model, determine a strain quantification parameter based on the motion vector field, and determine a spline curve using a spline mathematical technique to estimate motion for the target cardiac region using the prediction model, wherein the target cardiac region is at least one of LV (Left Ventricle), RV (Right Ventricle), LA (Left Atrium), and RA (Right Atrium).
Citation Information
Patent Citations
Ultrasound system and method for estimating motion of particle based on vector doppler
KR101348771B1
Method and apparatus for augmenting echocardiography data
KR1020230055004A
Method for evaluating scoliosis and device evaluating scoliosis using the same
KR102389067B1
Systems and methods for machine learning based physiological motion measurement
US20210133984A1
Combined B-mode / tissue doppler approach for improved cardiac motion estimation in echocardiographic images
US9629615B1
