Method for providing guidelines of echocardiography images and device usinng the same

Abstract

The present invention provides a method for providing guidance for a cardiac ultrasound image implemented by a processor, comprising the steps of receiving a cardiac ultrasound image of an object captured, and determining probe guidance based on the received cardiac ultrasound image using a prediction model learned to determine probe guidance according to ultrasound probe movement with the cardiac ultrasound image as input, and a device using the same.

Description

Technology Field

[0001] The present invention relates to a method for providing guidelines for cardiac ultrasound images and a device for providing guidelines for cardiac ultrasound images using the same. Background Technology

[0002] Cardiac ultrasound examination is performed by projecting ultrasound waves onto the three-dimensional structure of the heart in multiple planes to acquire images of the heart and measure hemodynamic variables.

[0003] At this time, the medical team positions the ultrasound probe in a location where it is easy to obtain ultrasound images so as to acquire multi-dimensional images through anatomical structures around the heart, such as between the ribs, and records images by finding appropriate cross-sections through rotation and tilting.

[0004] Furthermore, measurements associated with morphological changes in the heart can be determined from appropriate tomographic images, and specific diseases can be diagnosed based on these measurements. In other words, these measurements can serve as clinical values ​​for the diagnosis of heart disease and, furthermore, for monitoring prognosis.

[0005] Meanwhile, the determination of measurements obtained through cardiac ultrasound examinations depends on the interpretation of medical staff, and the reliability of the results may also depend on the expertise of the medical staff.

[0006] In other words, since measurements can vary significantly depending on the proficiency of medical staff, there is a continuous demand for the development of a new guideline provision system capable of deriving highly accurate measurements from cardiac ultrasound images.

[0007] The background description of the invention is provided to facilitate a better understanding of the present invention. The matters described in the background description should not be construed as an acknowledgment that they exist as prior art. The problem to be solved

[0008] Meanwhile, to solve the aforementioned problems, the inventors of the present invention intended to develop a guideline providing system based on an artificial neural network trained to segment multiple regions of a cardiac ultrasound image.

[0009] In this regard, the inventors of the present invention noted that for various measurement values, a B-mode image is acquired and an M-mode or Doppler (color Doppler) image is determined therefrom, and the user performs a process for measurement within the image to determine the measurement value.

[0010] The inventors of the present invention recognized that by applying an artificial neural network, they could complement the clinical process to provide information by selecting frames capable of determining measurements without additional work for mode switching, and provide highly reliable information.

[0011] Furthermore, the inventors of the present invention recognized that by applying an artificial neural network, they could solve the problem of conventional processes that have difficulty determining measurements due to time constraints in emergency situations.

[0012] As a result, the inventors of the present invention developed an artificial neural network-based guideline provision system.

[0013] Accordingly, the inventors of the present invention could expect that by providing a new guideline provision system, it would be possible to provide highly reliable analysis results for cardiac ultrasound images regardless of the proficiency of medical staff.

[0014] In particular, the inventors of the present invention could expect that when a new guideline providing system is applied to a small device, including a handheld type, which has limitations in the task of acquiring measurements, fast and highly accurate decision-making would be possible.

[0015] Accordingly, the problem to be solved by the present invention is to provide a method for providing guidelines for cardiac ultrasound images and a device using the same, configured to provide information on determinable measurement parameters by segmenting the anatomical structure of the heart from a received cardiac ultrasound image using an artificial neural network-based prediction model.

[0016] The problems of the present invention are not limited to those mentioned above, and other unmentioned problems will be clearly understood by those skilled in the art from the description below. means of solving the problem

[0017] In order to solve the problem described above, a method for providing guidelines for cardiac ultrasound images according to one embodiment of the present invention is provided.

[0018] The above method is a method for providing guidelines for a cardiac ultrasound image implemented by a processor, comprising the steps of receiving a cardiac ultrasound image of an object captured, and determining a probe guidance based on the received cardiac ultrasound image using a prediction model trained to determine a probe guidance according to the movement of an ultrasound probe with the cardiac ultrasound image as input.

[0019] According to a feature of the present invention, the method may further include the step of receiving a cross-sectional view of a cardiac ultrasound image, and the step of determining probe guidance may include the step of determining probe guidance based on the received cardiac ultrasound image and the received cross-sectional view using a prediction model.

[0020] According to another feature of the present invention, the step of determining probe guidance may include the step of determining a first probe guidance and a second probe guidance according to movement, respectively, based on a received cardiac ultrasound image using a prediction model.

[0021] According to another feature of the present invention, the second probe guidance may be defined as a guidance with a greater movement than the first probe guidance, and the first probe guidance may include at least one probe operation guidance among hold, probe head upward movement (Tilt Down), probe head downward movement (Tilt Up), probe head left movement (Rock Right), probe head right movement (Rock Left), probe head right tilt (Tilt Right), probe head left tilt (Tilt Left), probe head upward shaking (Rock Down), probe head downward shaking (Rock Up), probe clockwise rotation (Rotate Clockwise), and probe counter-clockwise rotation (Rotate Counter-Clockwise).

[0022] The second probe guidance may include at least one probe operation guidance among upper sliding (Slide UP), lower sliding (Slide DOWN), left sliding (Slide LEFT), and right sliding (Slide RIGHT) to provide for a precise scan of anatomical structures.

[0023] According to another feature of the present invention, the prediction model may be configured to probabilistically predict a first probe guidance and a second probe guidance, respectively, and may further include the step of providing the first probe guidance and the step of selectively providing the second probe guidance when the probability for the first probe guidance is greater than or equal to a predetermined level.

[0024] According to another feature of the present invention, the prediction model may be further configured to segment anatomical structures of the heart using a cardiac ultrasound image as input, and the step of determining probe guidance may include the step of segmenting anatomical structures within the received cardiac ultrasound image using the prediction model and the step of determining probe guidance based on the segmentation result using the prediction model.

[0025] According to another feature of the present invention, the prediction model may be further configured to classify cross-sectional views of an input cardiac ultrasound image using a cardiac ultrasound image as input, and the step of determining probe guidance may include the step of classifying cross-sectional views of a received cardiac ultrasound image using the prediction model and the step of determining probe guidance corresponding to the classified cross-sectional views using the prediction model.

[0026] According to another feature of the present invention, the step of classifying cross-sections may include classifying a received cardiac ultrasound image into at least one cross-section among PLAX, PSAX-AV, PSAX MV, PSAX PM, PSAX APEX, A4C, A3C, and A2C using a prediction model, and the step of determining a corresponding probe guidance may include determining a probe guidance for the at least one cross-section.

[0027] To solve the problem described above, a device for providing guidelines for cardiac ultrasound images according to another embodiment of the present invention is provided.

[0028] The above device includes a communication unit configured to receive a cardiac ultrasound image of an object captured, and a processor functionally connected to the communication unit.

[0029] At this time, the processor may be configured to determine probe guidance based on the received cardiac ultrasound image by using a prediction model trained to determine probe guidance according to the movement of the ultrasound probe with the cardiac ultrasound image as input.

[0030] According to a feature of the present invention, the prediction model may be a model trained to determine probe guidance using a cardiac ultrasound image and a cross-sectional view of the cardiac ultrasound image as input.

[0031] According to another feature of the present invention, the processor may be further configured to determine a first probe guidance and a second probe guidance according to movement, respectively, based on a received cardiac ultrasound image using a prediction model.

[0032] According to another feature of the present invention, the second probe guidance may be defined as guidance with greater movement than the first probe guidance, and the first probe guidance includes at least one probe operation guidance among standard, upward movement of the probe head, downward movement of the probe head, leftward movement of the probe head, rightward movement of the probe, clockwise rotation of the probe, and counterclockwise rotation of the probe, and the second probe guidance may include at least one probe operation guidance among standard, movement toward the ribs, movement toward the ribs, leftward sliding, and rightward sliding.

[0033] According to another feature of the present invention, the prediction model may be configured to probabilistically predict a first probe guidance and a second probe guidance, respectively, and the processor may further include an output unit configured to provide the first probe guidance and, if the probability for the first probe guidance is greater than or equal to a predetermined level, to selectively provide the second probe guidance.

[0034] According to another feature of the present invention, the prediction model may be further configured to segment anatomical structures of the heart using a cardiac ultrasound image as input, and the processor may be further configured to segment anatomical structures within the received cardiac ultrasound image using the prediction model and to determine probe guidance based on the segmentation result using the prediction model.

[0035] According to another feature of the present invention, the prediction model may be further configured to classify cross-sectional views of an input cardiac ultrasound image using a cardiac ultrasound image as input, and the processor may be further configured to classify cross-sectional views of a received cardiac ultrasound image using the prediction model and to determine probe guidance corresponding to the classified cross-sectional views using the prediction model.

[0036] According to another feature of the present invention, the processor may be further configured to classify a received cardiac ultrasound image into at least one cross-section of PLAX, PSAX-AV, PSAX MV, PSAX PM, PSAX APEX, A4C, A3C, and A2C using a prediction model, and to determine probe guidance for at least one cross-section.

[0037] Specific details of other embodiments are included in the detailed description and drawings. Effects of the invention

[0038] The present invention can provide a guideline provision system for cardiac ultrasound images based on an artificial neural network, which enables the acquisition of cardiac measurements based on the anatomical structure of the heart using cardiac ultrasound images.

[0039] In particular, the present invention can provide faster and more accurate quantification results of heart structure by providing a guideline providing system that provides information on frames capable of determining measurements conforming to ASE guidelines within a plurality of divided regions.

[0040] Furthermore, the present invention is applicable to small devices including handheld types that have limitations in performing tasks for acquiring measurements, particularly processes for acquiring specific images, and can provide fast and reliable information using small devices.

[0041] In addition, the present invention provides a prediction model trained to determine probe guidance according to the movement of an ultrasound probe using a cardiac ultrasound image as input, thereby improving the quality of the cardiac ultrasound image and implementing a method for providing guidelines that can guide optimal probe operation tailored to the user's proficiency.

[0042] According to the present invention, probe manipulation can be analyzed in real time based on captured cardiac ultrasound images, and optimal guidance can be determined through a prediction model. Through this, the present invention supports the examiner in acquiring more accurate ultrasound images and, in particular, can provide customized probe manipulation guidance suitable for both beginners and experts.

[0043] In addition, the present invention can provide a quantitative and objective imaging assistance system compared to existing experience-based ultrasound imaging methods by automatically analyzing ultrasound probe movement through a prediction model and determining probe guidance that considers image quality and the alignment of anatomical structures.

[0044] Furthermore, the guideline provision method of the present invention provides real-time feedback, allowing the examiner to immediately adjust probe operation, thereby reducing unnecessary imaging errors during the inspection process and providing the effect of shortening the ultrasound inspection time.

[0045] Consequently, the present invention improves the accuracy and reproducibility of cardiac ultrasound examinations and supports medical professionals in acquiring more stable and reliable cardiac ultrasound images, thereby contributing to the improvement of diagnostic accuracy. In other words, the present invention enables the provision of highly reliable analysis results for cardiac ultrasound images regardless of the examiner's proficiency, and can contribute to establishing more accurate decision-making and treatment plans during the image analysis stage.

[0046] The effects according to the present invention are not limited to those exemplified above, and various other effects are included in this specification. Brief explanation of the drawing

[0047] FIGS. 1a and 1b illustrate a guideline provision system for cardiac ultrasound images using a device for providing guidelines for cardiac ultrasound images according to an embodiment of the present invention. FIG. 2a is a block diagram showing the configuration of a medical device according to one embodiment of the present invention. FIG. 2b is a block diagram showing the configuration of a guideline providing server according to one embodiment of the present invention. FIG. 3 illustrates the procedure of a method for providing guidelines for cardiac ultrasound images according to one embodiment of the present invention. FIGS. 4a and 4c and FIGS. 5 to 7 illustrate, by way of example, the procedure of a method for providing guidelines for cardiac ultrasound images according to one embodiment of the present invention. FIG. 8 illustrates an exemplary user interface based on a guideline provision method according to various embodiments of the present invention. FIGS. 9a and 9b and FIGS. 10a to 10c illustrate the evaluation results of a prediction model used in a guideline provision method according to various embodiments of the present invention. FIGS. 11a to 11f illustrate, by way of example, a procedure for a method of providing guidelines for cardiac ultrasound images according to another embodiment of the present invention. FIGS. 12a to 12c illustrate exemplary user interfaces based on a guideline provision method according to various embodiments of the present invention. FIGS. 13a to 13d illustrate the evaluation results of a prediction model capable of anatomical structure division and cross-sectional view classification used in a guideline provision method according to various embodiments of the present invention. Specific details for implementing the invention

[0048] The advantages of the invention and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below but may be implemented in various different forms. These embodiments are provided merely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims.

[0049] The shapes, sizes, ratios, angles, numbers, etc. disclosed in the drawings for explaining embodiments of the present invention are exemplary, and therefore the present invention is not limited to the depicted details. Furthermore, in describing the present invention, if it is determined that a detailed description of related known technology may unnecessarily obscure the essence of the present invention, such detailed description is omitted. Where terms such as "comprising," "having," or "consisting of" are used in this specification, other parts may be added unless "only" is used. Where a component is expressed in the singular, it includes cases where it includes the plural unless specifically stated otherwise.

[0050] In interpreting the components, they are interpreted to include a margin of error even in the absence of a separate explicit description.

[0051] The features of each of the various embodiments of the present invention may be combined or combined with one another, either partially or wholly, and as will be fully understood by those skilled in the art, various technical interlocking and operation are possible, and each embodiment may be implemented independently of one another or together in an interlocking relationship.

[0052] For clarity in the interpretation of this specification, the terms used in this specification are defined below.

[0053] As used herein, the term "entity" may refer to any object intended to receive guideline information for determining measurements from cardiac ultrasound images. Meanwhile, the entities disclosed herein may be any mammals other than humans, but are not limited thereto.

[0054] As used in this specification, the term "cardiac ultrasound image" refers to a cardiac ultrasound image of an object, which may be a single-frame still image or a video composed of multiple frames.

[0055] At this time, the cardiac ultrasound image may be a 2D or 3D image.

[0056] According to a feature of the present invention, the cardiac ultrasound image may be a B-mode cardiac ultrasound image that serves as the basis for determining the measurement value. However, it is not limited thereto, and the cardiac ultrasound image may include a Doppler cardiac ultrasound image, a color Doppler cardiac ultrasound image, or an M-mode cardiac ultrasound image capable of determining the measurement value.

[0057] As used in this specification, the term "anatomical structure" may refer to an anatomical structure of the heart that is identifiable in cardiac ultrasound images.

[0058] According to a feature of the present invention, the anatomical structure may include one region selected from the right ventricle anterior wall, the right ventricle (RV), the anterior wall of the aorta, the aorta, the posterior wall of the aorta, the left atrium (LA), and the posterior wall of the left atrium (LA). However, it is not limited thereto.

[0059] Here, "parameter" or "measurement" may refer to numerical values ​​such as thickness or diameter that are measurable for a heart structure, and may be used interchangeably in this specification.

[0060] According to a feature of the present invention, the parameter may be at least one of the interventricular septum thickness (IVS), left ventricular diameter (LVID), left ventricular posterior wall diastole (LVVWD), right ventricular outflow tract (RVOT), aortic diameter (Ao), and left atrium diameter (LA). However, it is not limited thereto.

[0061] In this case, "measurable parameter" can be defined as a measurement parameter for which a value corresponding to the parameter can be determined.

[0062] As used in this specification, the term "mode parameter" may be defined as a parameter measurable from a mode selected by the user.

[0063] According to various embodiments of the present invention, a prediction model trained to segment anatomical structures within a frame of a cardiac ultrasound image is presented.

[0064] As used in this specification, the term "prediction model" may be a model configured to take cardiac ultrasound images as input and output segmented anatomical structures.

[0065] Meanwhile, the output value of the prediction model is not limited to what was mentioned above.

[0066] For example, the prediction model may be a model trained to take a cardiac ultrasound image as input and output the location (coordinates) of each anatomical structure.

[0067] As another example, the prediction model may be a model trained to take a cardiac ultrasound image as input, segment regions, and output a probability for each anatomical structure corresponding to each region.

[0068] According to the features of the present invention, the prediction model may be a model further trained to determine and output a location-based connectivity meter, an uncertainty score corresponding to uncertainty, and a capacity meter corresponding to volume for each of the segmented anatomical structures.

[0069] Here, the uncertainty score may include entropy, mutual information between predictions and model posterior (e.g., Bayesian Active Learning by Disagreement; BALD) indicating how related the random variables of 'predictions' and 'model posterior probabilities' are to each other, variation ratios, and mean standard deviation.

[0070] In addition, the prediction model may include a model trained to take cardiac ultrasound images as input, classify the types of cross-sections, and output results and probabilities.

[0071] According to another feature of the present invention, scores for connectivity, uncertainty, and capacity, respectively, are calculated, and a quality evaluation of an image can be performed based on a combination of these scores.

[0072] In various embodiments of the present invention, measurable parameters can be obtained from the image based on the positional relationship of each anatomical structure within the cardiac ultrasound image.

[0073] According to another feature of the present invention, a measurable image mode can be obtained from the corresponding image based on the positional relationship of each anatomical structure.

[0074] According to another feature of the present invention, guidelines for acquiring measurable image modes from a corresponding image can be determined based on the positional relationship of each anatomical structure. For example, lines for acquiring Doppler mode or color Doppler mode images, or boxes for acquiring M-mode images, may be determined and displayed within the base image. Accordingly, the user can more easily acquire images of different modes for determining measurements. That is, images of various modes and measurements can be acquired and measurements determined using a handheld type device with a screen smaller than that of a general ultrasound diagnostic device.

[0075] At this time, the determination of measurable parameters or measurable modes may be performed by a prediction model.

[0076] In other words, the prediction model may be a model further trained to segment anatomical structures in the input cardiac ultrasound image and, simultaneously, predict measurable parameters or measurable modes from the image based on the segmentation results.

[0077] However, it is not limited to this, and it may be possible to predict measurable parameters or measurable modes from an image by a measurement prediction model that exists independently of the prediction model and is trained to predict measurable parameters or modes using the segmentation results of the prediction model as input.

[0078] For example, a measurement prediction model may probabilistically predict measurable parameters or measurable modes.

[0079] According to a feature of the present invention, when a selection of a measurable ultrasound mode is received from a user, an image corresponding to the selected mode and a value for a measurable parameter within the image may be output and provided to the user.

[0080] Furthermore, the prediction model includes an output layer composed of multiple nodes, where the number of nodes may correspond to a predetermined number of regions (number of classes) of anatomical structures.

[0081] According to another feature of the present invention, the prediction model is based on deep learning-based learning and, based on the results, can determine measurable parameters, measurable other modes (M-mode, Doppler, color Doppler), etc., based on a rule base, but the rule base step is not limited thereto and may also be based on machine learning-based learning.

[0082] In more diverse embodiments of the present invention, the prediction model may be a model trained to determine probe guidance according to the movement of the ultrasound probe using a cardiac ultrasound image as input.

[0083] In more diverse embodiments of the present invention, the prediction model may be a model trained to determine probe guidance according to the movement of an ultrasound probe by taking a cardiac ultrasound image and a cross-sectional view of the image as input.

[0084] In this case, the prediction model may be a model trained to determine probe guidance using cardiac ultrasound images and cross-sectional views of the cardiac ultrasound images as input.

[0085] In various embodiments of the present invention, the prediction model may be a model trained to probabilistically predict probe guidance.

[0086] Here, in ultrasound examination, the probe manipulation guide is a system that guides how to adjust the probe for correct image acquisition and may include a first probe guidance for beginners and a second probe guidance for advanceds depending on the magnitude of the probe movement.

[0087] At this time, the first probe guidance may vary by mode.

[0088] In various embodiments of the present invention, the first probe guidance may include at least one probe operation guidance among hold, tilt down, tilt up, leftward, rightward, clockwise, and counterclockwise rotation of the probe in PLAX and A4C shooting modes.

[0089] In various embodiments of the present invention, the first probe guidance may include at least one probe operation guidance among hold, tilt right, tilt left, rock down, rock up, rotate clockwise, and rotate counter-clockwise in PSAX, A2C, A3C, and IVC shooting modes.

[0090] In various embodiments of the present invention, the second probe guidance provides a more sophisticated probe operation method and may include at least one probe operation guidance among upward sliding (Slide UP), downward sliding (Slide DOWN), left sliding (Slide LEFT), and right sliding (Slide RIGHT) to provide for a precise scan of anatomical structures.

[0091] For example, while fixing the starting point position of the probe, you can receive a first guidance-based probe operation guide to find the standard view by learning basic operations such as tilting, rocking, and rotating, and if the standard view has been formed but fine correction is still needed, you can receive a second guidance-based probe operation guide to obtain the optimal ultrasound image by performing a sliding operation through a larger movement.

[0092] However, it is not limited to this, and probe guidance may include various probe operation guidances based on the cross-sectional view.

[0093] Meanwhile, the prediction model may be a model configured to determine probe guidance based on the segmentation of anatomical structures and / or classification of cross-sections, but is not limited thereto.

[0094] In more diverse embodiments of the present invention, the prediction model may be a model trained to classify Odd cardiac ultrasound images using cardiac ultrasound images as input.

[0095] In other words, the predictive model can be used to maintain the correct cross-section in captured cardiac ultrasound images and to detect abnormalities.

[0096] Meanwhile, the prediction model may be based on at least one algorithm selected from DenseNet-121, U-net, VGG net, DenseNet, FCN (Fully Convolutional Network) with encoder-decoder structure, DNN (deep neural network) such as SegNet, DeconvNet, DeepLAB V3+, Transformer such as Lawin+, SegFormer, Swin, SqueezeNet, AlexNet, ResNet18, MobileNet-v2, GoogLeNet, ResNet-v2, ResNet50, RetinaNet, ResNet101, Inception-v3, HRNet, ResNeXt, and EfficientNet. Furthermore, the prediction model may be an ensemble model based on at least two of the aforementioned algorithms. However, it is not limited thereto.

[0097] In more diverse embodiments of the present invention, the cardiac ultrasound image may be a video consisting of a plurality of frames, and the prediction model may exist as a fused model comprising a temporal model that learns changes between time or consecutive frames and a 2D model that analyzes a single image or independent frames.

[0098] For example, if a 2D CNN analyzes individual frames of a cardiac ultrasound image to extract immediate structural information, a temporal model can connect multiple frames to analyze changes over time, thereby performing a comprehensive analysis of the cardiac ultrasound image.

[0099] This allows for the provision of optimal ultrasound probe guidance that considers not only the quality evaluation of a single image but also the continuous flow of the image.

[0101] Hereinafter, with reference to FIGS. 1a, 1b, 2a, and 2b, a guideline provision system for cardiac ultrasound images and a guideline provision device for cardiac ultrasound images using a guideline provision device for cardiac ultrasound images according to an embodiment of the present invention will be described.

[0102] FIGS. 1a and 1b illustrate a guideline provision system for cardiac ultrasound images using a device for providing guidelines for cardiac ultrasound images according to an embodiment of the present invention. FIG. 2a illustrates an exemplary configuration of a medical device receiving guidelines for cardiac ultrasound images according to an embodiment of the present invention. FIG. 2b illustrates an exemplary configuration of a server for providing guidelines for cardiac ultrasound images according to an embodiment of the present invention.

[0103] First, referring to FIG. 1a, the guideline providing system (1000) may be a system configured to provide information related to measurement parameters based on cardiac ultrasound images of an object. In this case, the guideline providing system (1000) may be composed of a medical device (100) that receives information related to measurement parameters, an ultrasound imaging diagnostic device (200) that provides cardiac ultrasound images, and a guideline providing server (300) that generates information related to measurement parameters based on the received cardiac ultrasound images.

[0104] In various embodiments of the present invention, a guideline providing server (300) may be mounted on an ultrasound imaging diagnostic device (200), and in this case, various information related to measurements may be displayed on a display (not shown) of the ultrasound imaging diagnostic device (200).

[0105] That is, the user can use the ultrasound imaging diagnostic device (200) to check information related to heart measurements at the same time as the diagnosis.

[0106] In various embodiments, the medical device (100) is an electronic device that provides a user interface for displaying information associated with measurement parameters, and may include at least one of a smartphone, a tablet PC (Personal Computer), a laptop and / or a PC.

[0107] The medical device (100) receives guidelines for determining measurement parameters for an object from a guideline providing server (300) and can display the received results through a display unit (not shown).

[0108] The guideline provision server (300) may include a general-purpose computer, laptop, and / or data server, etc., that performs various operations to determine information related to cardiac quantification based on cardiac ultrasound images provided from an ultrasound imaging diagnostic device (200), such as an ultrasound diagnostic device. In this case, the guideline provision server (300) may be a device for accessing a web server that provides web pages or a mobile web server that provides mobile websites, but is not limited thereto.

[0109] More specifically, the guideline providing server (300) receives a cardiac ultrasound image from an ultrasound imaging diagnostic device (200) and performs a prediction of anatomical structures for determining measurements within the received cardiac ultrasound image. At this time, the guideline providing server (300) can perform a prediction of the location of anatomical structures within the cardiac ultrasound image using a prediction model.

[0110] Furthermore, the guideline providing server (300) can determine measurable parameters and / or measurable modes based on anatomical structures.

[0111] The guideline providing server (300) can provide measurable parameters and / or measurable modes, furthermore, segmentation results for anatomical structures, etc. to the medical device (100).

[0112] Referring together with FIG. 1b, in a more diverse embodiment, the medical device (100) receives guidelines for probe operation from a guideline providing server (300) and can display the received results through a display unit (not shown).

[0113] More specifically, the guideline providing server (300) receives cardiac ultrasound images and cross-sectional views from the ultrasound imaging diagnostic device (200) and performs a prediction of probe guidance based on the received cardiac ultrasound images. At this time, the guideline providing server (300) can perform a prediction of probe guidance using a prediction model.

[0114] Furthermore, the guideline providing server (300) can determine probe guidance specific to a specific cross-section.

[0115] In this way, information provided from the guideline provision server (300) may be provided as a web page through a web browser installed on the medical device (100), or provided in the form of an application or program. In various embodiments, such data may be provided in a form included in a platform in a client-server environment.

[0116] Next, with reference to FIGS. 2a and 2b, the components of the guideline providing server (300) of the present invention will be described in detail.

[0117] First, referring to FIG. 2a, the medical device (100) may include a memory interface (110), one or more processors (120) and a peripheral interface (130). Various components within the medical device (100) may be connected by one or more communication buses or signal lines.

[0118] The memory interface (110) is connected to the memory (150) and can transmit various data to the processor (120). Here, the memory (150) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain data.

[0119] In various embodiments, memory (150) may store at least one of an operating system (151), a communication module (152), a graphical user interface module (GUI) (153), a sensor processing module (154), a telephone module (155), and an application module (156). Specifically, the operating system (151) may include instructions for processing basic system services and instructions for performing hardware operations. The communication module (152) may communicate with at least one of one or more other devices, computers, and servers. The graphical user interface module (GUI) (153) may process a graphical user interface. The sensor processing module (154) may process sensor-related functions (e.g., processing voice input received using one or more microphones (192)). The telephone module (155) may process telephone-related functions. The application module (156) may perform various functions of a user application, such as electronic messaging, web browsing, media processing, exploration, imaging, and other processing functions. In addition, the medical device (100) may store one or more software applications (156-1, 156-2) associated with any one type of service (e.g., a guideline provision application) in memory (150).

[0120] In various embodiments, the memory (150) can store a digital assistant client module (157) (hereinafter, DA client module), and accordingly, can store commands and various user data (158) for performing client-side functions of the digital assistant.

[0121] Meanwhile, the DA client module (157) can obtain voice input, text input, touch input and / or gesture input from a user through various user interfaces (e.g., I / O subsystem (140)) provided in the medical device (100).

[0122] Additionally, the DA client module (157) can output data in the form of audiovisual and tactile signals. For example, the DA client module (157) can output data consisting of a combination of at least two of voice, sound, notifications, text messages, menus, graphics, videos, animations, and vibrations. Furthermore, the DA client module (157) can communicate with a digital assistant server (not shown) using a communication subsystem (180).

[0123] In various embodiments, the DA client module (157) may collect additional information about the surrounding environment of the medical device (100) from various sensors, subsystems, and peripheral devices to construct a context associated with user input. For example, the DA client module (157) may provide context information along with user input to a digital assistant server to infer the user's intent. Here, context information that may accompany user input may include sensor information, e.g., lighting, ambient noise, ambient temperature, images of the surrounding environment, video, etc. As another example, context information may include the physical state of the medical device (100) (e.g., device orientation, device location, device temperature, power level, speed, acceleration, motion pattern, cellular signal strength, etc.). As another example, the situation information may include information related to the software status of the medical device (100) (e.g., processes running on the medical device (100), installed programs, past and present network activity, background services, error logs, resource usage, etc.).

[0124] In various embodiments, the memory (150) may include additional or deleted instructions, and furthermore, the medical device (100) may include additional configurations in addition to the configuration shown in 2a, or exclude some configurations.

[0125] The processor (120) can control the overall operation of the medical device (100) and can execute various commands to implement an interface that provides information related to measurement parameters by running an application or program stored in memory (150).

[0126] The processor (120) may correspond to a computing device such as a CPU (Central Processing Unit) or an AP (Application Processor). Additionally, the processor (120) may be implemented in the form of an integrated chip (IC), such as a System on Chip (SoC) that integrates various computing devices such as a Neural Processing Unit (NPU).

[0127] The peripheral interface (130) is connected to various sensors, subsystems, and peripheral devices and can provide data to enable the medical device (100) to perform various functions. Here, it can be understood that the medical device (100) performs a function by the processor (120).

[0128] The peripheral interface (130) may receive data from a motion sensor (160), a light sensor (light sensor) (161), and a proximity sensor (162), thereby enabling the medical device (100) to perform orientation, light, and proximity detection functions, etc. As another example, the peripheral interface (130) may receive data from other sensors (163) (positioning system—GPS receiver, temperature sensor, biometric sensor), thereby enabling the medical device (100) to perform functions related to other sensors (163).

[0129] In various embodiments, the medical device (100) may include a camera subsystem (170) connected to a peripheral interface (130) and an optical sensor (171) connected thereto, thereby enabling the medical device (100) to perform various shooting functions such as taking photos and recording video clips.

[0130] In various embodiments, the medical device (100) may include a communication subsystem (180) connected to a peripheral interface (130). The communication subsystem (180) is composed of one or more wired / wireless networks and may include various communication ports, radio frequency transceivers, and optical transceivers.

[0131] In various embodiments, the medical device (100) includes an audio subsystem (190) connected to a peripheral interface (130), and the audio subsystem (190) includes one or more speakers (191) and one or more microphones (192), so that the medical device (100) can perform voice-operated functions, such as voice recognition, voice cloning, digital recording, and telephone functions.

[0132] In various embodiments, the medical device (100) may include an I / O subsystem (140) connected to a peripheral interface (130). For example, the I / O subsystem (140) may control a touch screen (143) included in the medical device (100) through a touch screen controller (141). As an example, the touch screen controller (141) may detect user contact and movement or interruption of contact and movement using any one of a plurality of touch sensing technologies, such as capacitive, resistive, infrared, surface acoustic wave technology, proximity sensor array, etc. As another example, the I / O subsystem (140) may control other input / control devices (144) included in the medical device (100) through other input controller(s) (142). As an example, other input controller(s) (142) can control one or more pointer devices such as buttons, rocker switches, thumb wheels, infrared ports, USB ports and styluses.

[0133] Next, referring to FIG. 2b, the guideline providing server (300) may include a communication interface (310), memory (320), I / O interface (330), and a processor (340), and each component may communicate with one or more communication buses or signal lines.

[0134] The communication interface (310) can be connected to the medical staff device (100) and the ultrasound imaging diagnostic device (200) via a wired / wireless communication network to exchange data. For example, the communication interface (310) can receive a cardiac ultrasound image from the ultrasound imaging diagnostic device (200), determine measurement parameters or guidelines for obtaining them from the image, and transmit them to the medical staff device (100).

[0135] Meanwhile, a communication interface (310) that enables the transmission and reception of such data includes a communication pod (311) and a wireless circuit (312), wherein the wired communication port (311) may include one or more wired interfaces, for example, Ethernet, Universal Serial Bus (USB), FireWire, etc. Additionally, the wireless circuit (312) may transmit and receive data with an external device via an RF signal or an optical signal. Furthermore, wireless communication may use at least one of a plurality of communication standards, protocols, and technologies, such as GSM, EDGE, CDMA, TDMA, Bluetooth, Wi-Fi, VoIP, Wi-MAX, or any other suitable communication protocol.

[0136] The memory (320) can store various data used in the guideline provision server (300). For example, the memory (320) can store cardiac ultrasound images or a prediction model trained to segment anatomical structures within cardiac ultrasound images.

[0137] In various embodiments, the memory (320) may include a volatile or non-volatile recording medium capable of storing various data, commands and information. For example, the memory (320) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, network storage, cloud, and blockchain data.

[0138] In various embodiments, the memory (320) may store at least one configuration of an operating system (321), a communication module (322), a user interface module (323), and one or more applications (324).

[0139] An operating system (321) (e.g., an embedded operating system such as LINUX, UNIX, MAC OS, WINDOWS, VxWorks, etc.) may include various software components and drivers for controlling and managing general system tasks (e.g., memory management, storage device control, power management, etc.) and may support communication between various hardware, firmware, and software components.

[0140] The communication module (323) can support communication with other devices through the communication interface (310). The communication module (320) may include various software components for processing data received by the wired communication port (311) or wireless circuit (312) of the communication interface (310).

[0141] The user interface module (323) can receive user requests or inputs from a keyboard, touch screen, microphone, etc. through the I / O interface (330) and provide a user interface on the display.

[0142] The application (324) may include a program or module configured to be executed by one or more processors (330). Here, the application for providing information associated with measurement parameters may be implemented on a server farm.

[0143] The I / O interface (330) can connect at least one of the input / output devices (not shown) of the guideline providing server (300), such as a display, keyboard, touch screen, and microphone, to the user interface module (323). The I / O interface (330) can receive user input (e.g., voice input, keyboard input, touch input, etc.) together with the user interface module (323) and process commands based on the received input.

[0144] The processor (340) is connected to the communication interface (310), memory (320), and I / O interface (330) to control the overall operation of the guideline provision server (300) and can execute various commands for providing guidelines through an application or program stored in the memory (320).

[0145] The processor (340) may correspond to a computing device such as a CPU (Central Processing Unit) or an AP (Application Processor). Additionally, the processor (340) may be implemented in the form of an integrated chip (IC), such as a System on Chip (SoC) that integrates various computing devices. Alternatively, the processor (340) may include a module for computing artificial neural network models, such as a Neural Processing Unit (NPU).

[0146] In various embodiments, the processor (340) may be configured to predict anatomical structures within a cardiac ultrasound image using predictive models and to automatically determine and provide measurements. Optionally, the processor (340) may be configured to provide information about measurements obtainable from the cardiac ultrasound image and information about measurable modes.

[0148] Hereinafter, with reference to FIGS. 3, FIGS. 4a, FIGS. 4c, and FIGS. 5 to 7, a method for providing guidelines for cardiac ultrasound images according to an embodiment of the present invention will be described in detail.

[0149] FIG. 3 illustrates the procedure of a method for providing guidelines for cardiac ultrasound images according to an embodiment of the present invention. FIGS. 4a to 4c and FIGS. 5 to 7 illustrate, by example, the procedure of a method for providing guidelines for cardiac ultrasound images according to an embodiment of the present invention.

[0150] First, referring to FIG. 3, a guideline provision procedure according to one embodiment of the present invention is as follows. First, a cardiac ultrasound image of an object is received (S310). Next, the location of anatomical structures is determined by a prediction model (S320). Next, measurable parameters are determined based on the anatomical structures (S330).

[0151] More specifically, in the step (S310) where the cardiac ultrasound image is received, a cardiac ultrasound image of the target area, i.e., the heart area, can be received.

[0152] According to a feature of the present invention, in the step (S310) where a cardiac ultrasound image is received, a cardiac ultrasound image of a still cut in B-mode or a cardiac ultrasound video including a plurality of frames may be received.

[0153] However, this is not limited to this, and a wider variety of cardiac ultrasound images that can determine measurements may be received at the step (S310) where the cardiac ultrasound image is received.

[0154] Next, a step (S320) in which anatomical structures are predicted is performed.

[0155] According to a feature of the present invention, the step (S320) of predicting anatomical structures can be performed by a prediction model trained to divide a plurality of regions based on anatomical structures within a cardiac ultrasound image using a received cardiac ultrasound image as input.

[0156] That is, in the step (S320) where anatomical structures are predicted, the location (coordinates) of the anatomical structures within the image can be determined.

[0157] At this time, the prediction model can segment at least one selected region among the right ventricle anterior wall, right ventricle (RV), anterior wall of aorta, aorta, posterior wall of aorta, left atrium (LA), and posterior wall of left atrium within the cardiac ultrasound image.

[0158] Next, measurable parameters are determined based on region-based predicted anatomical structures (S330).

[0159] According to a feature of the present invention, in the step (S330) where a measurable parameter is determined, the measurable parameter may be determined based on the positional relationship of each anatomical structure.

[0160] According to another feature of the present invention, in the step (S330) where a measurable parameter is determined, a measurable parameter may be determined from at least one mode among an M-mode cardiac ultrasound image, a Doppler cardiac ultrasound image, and a color Doppler cardiac ultrasound image based on a B-mode cardiac ultrasound image.

[0161] According to another feature of the present invention, a step of determining a measurable ultrasonic mode based on a division result may be further performed.

[0162] At this time, in the step of determining a measurable ultrasound mode, an image quality evaluation is performed based on the segmentation results for each anatomical structure, and a measurable ultrasound mode can be determined based on the image quality evaluation.

[0163] At this time, an evaluation is performed for each segmented anatomical structure, including a location-based connectivity map, uncertainty score (e.g., entropy meter), and an assessment of the volume-related capacity, and a score is calculated for each. Subsequently, an image quality evaluation can be performed based on the sum of each score. For example, if the sum of the scores is below a predetermined level, it is determined as "low image quality," and if it is above the predetermined level, it is determined as "good image quality" for the ultrasound mode, thereby providing the image quality evaluation result.

[0164] Meanwhile, in various embodiments, the connectivity and capacity levels may be scored based on a rule base, and in the case of entropy, a residual-based score may be calculated. However, it is not limited thereto.

[0165] Meanwhile, the determination of measurable parameters, and furthermore, measurable modes, can be performed by a prediction model configured to take cardiac ultrasound images as input and output segmented anatomical structures.

[0166] In other words, the prediction model may be a model further trained to segment anatomical structures in the input cardiac ultrasound image and, simultaneously, predict measurable parameters or measurable modes from the image based on the segmentation results.

[0167] However, it is not limited to this, and it may be possible to predict measurable parameters or measurable modes from an image by a measurement prediction model that exists independently of the prediction model and is trained to predict measurable parameters or modes using the segmentation results of the prediction model as input.

[0168] For example, referring together with FIG. 4a, when a cardiac ultrasound image (412) is received, it is input into a prediction model (420) in the step (S320) where anatomical structures are predicted. More specifically, segmentation of multiple regions or features related to anatomical structures are extracted for the input cardiac ultrasound image (412). After operations are performed on these extracted features, the location of the anatomical structures within the cardiac ultrasound image (412) can finally be determined. Then, in the step (S330) where measurement parameters are determined, information (442) related to measurements, including measurable parameters and / or measurable modes based on the location of the anatomical structures, is determined from the output segmentation result (432). Here, information (442) related to the measurement may include anatomical structures within the image according to the segmentation, modes measurable from the image (e.g., Doppler AV (LVOT) PW, M mode Ao-LA, M mode and LV, Color Doppler AV and Color Doppler MV), and measurement parameters measurable from the image (e.g., LVPWD and Ao). However, it is not limited thereto, and information (442) related to the measurement may include values ​​for measurable parameters and the position of a probe (not shown).

[0169] In other words, by using a prediction model, it is possible to obtain measurements obtainable in a wider variety of modes without the process of acquiring images with switched ultrasound imaging modes and determining measurements obtainable in a specific switched imaging mode.

[0170] Accordingly, fast and highly accurate decision-making can be achieved by using a small, handheld device that has limitations in the task of acquiring measurements.

[0171] In various embodiments, the prediction model may be a model trained to predict information related to measurements simultaneously with region segmentation of anatomical structures.

[0172] For example, referring to FIG. 4b, in the step (S330) where measurement parameters are determined, a segmentation result (432) is output from the prediction model (420), and at the same time, information (442) related to measurements including measurable parameters and / or measurable modes based on the location of anatomical structures may be determined and output.

[0173] That is, information regarding measurable parameters and / or ultrasound imaging modes and / or segmented anatomical structures and / or measurements (not shown) and / or the position of the probe (not shown) can be automatically determined by the prediction model, allowing the user to determine measurements more quickly and make decisions.

[0174] Meanwhile, the determination of measurable parameters and / or ultrasound imaging modes by the aforementioned prediction model may be performed by a measurement prediction model independent of the prediction model trained to segment anatomical structures. Here, the measurement prediction model may be a model trained to determine measurable parameters and / or ultrasound imaging modes based on positional information of anatomical structures, but is not limited thereto.

[0175] In various embodiments, the prediction model may be a model trained to predict information related to measurements during the process of region segmentation for anatomical structures.

[0176] For example, referring to FIG. 4c, in the step (S330) where measurement parameters are determined, information (442) related to measurements, including measurable parameters and / or measurable modes based on the location of anatomical structures, may be determined from the prediction model (420). That is, the prediction model (420) may provide to the user only the information (442) related to measurements without outputting the segmentation result.

[0177] Here, the prediction model (420) may be a model trained to predict measurable parameters and / or measurable modes, etc. based on the positional relationships of anatomical structures.

[0178] Accordingly, the user can quickly obtain only the information related to the determination of the measurement value.

[0179] Referring again to FIG. 3, in the step (S330) where measurement parameters are determined, an evaluation is performed for each of the divided anatomical structures, and a measurable parameter can be determined based on the evaluation.

[0180] Here, for each anatomical structure, an evaluation of the capacity corresponding to the volume and the aforementioned location-based connectivity, uncertainty score (e.g., entropy meter, mutual information between model posterior probabilities (e.g., Bayesian Active Learning by Disagreement; BALD), deviation rate and mean standard deviation) can be performed.

[0181] In other words, evaluation results for each anatomical structure can be provided, making it possible to determine measurable parameters from low-quality images.

[0182] For example, by referring to FIGS. 5 and 6 together, an entropy map (542) and a connectivity map (642) for each anatomical structure can be determined and provided based on the segmentation result (432) for the anatomical structures of the prediction model (420).

[0183] In various embodiments, the step of determining a guideline within an ultrasound image based on segmented anatomical structures can be further performed to enable the acquisition of an ultrasound image of a different ultrasound mode from the received ultrasound image.

[0184] For example, referring together with FIG. 7, based on the segmentation result (432), guidelines (742), such as lines for acquiring Doppler mode or color Doppler mode images or boxes for acquiring M-mode images, can be determined and displayed within the B-mode image. Accordingly, the user can more easily acquire images of different modes for determining measurements. That is, images of various modes and measurements can be acquired and measurements determined using a handheld device with a screen smaller than that of a general ultrasound diagnostic device.

[0185] In various embodiments, a selection of an ultrasonic mode or measurement parameter determined to be measurable from the user can be received.

[0186] For example, if input is received for a specific ultrasound mode (view), an image corresponding to that ultrasound mode may be output and provided to the user. Furthermore, if input is received for a specific parameter, a value corresponding to the measurement may be output and provided to the user.

[0187] That is, in various embodiments of the present invention, various information associated with the measurement value may be called and provided according to user input.

[0188] Accordingly, medical staff can easily identify measurements from acquired cardiac ultrasound images in accordance with the guideline provision method according to various embodiments of the present invention, thereby enabling them to quickly proceed with the ultrasound analysis step and establish more accurate decision-making and treatment plans.

[0189] Meanwhile, the prediction and determination of measurements regarding anatomical structures are not limited to the aforementioned embodiments.

[0190] Hereinafter, a user interface according to a guideline provision method according to various embodiments of the present invention will be described exemplarily with reference to FIG. 8.

[0191] Referring to FIG. 8, in various embodiments, information associated with a measurement determined by a guideline providing server (300) is provided as a user interface through a touchscreen screen (143) of a user device (100).

[0192] More specifically, the user interface includes a still image (810) that displays the names of anatomical structures determined to be capable of obtaining measurable parameters and / or modes from images acquired during diagnosis using a cardiac ultrasound probe. At this time, the still image (810) includes an image quality indicator (812) that indicates the quality level of the image. The user can check the ultrasound modes obtainable from the still image (810) through the measurable mode indicator (820). Furthermore, the user can check the measurements obtainable from the still image (810) through the measurable parameter indicator (830).

[0193] In various embodiments, the user interface includes a connectivity display (840) that provides information on the connectivity evaluated for each anatomical structure of the heart within the still image (810).

[0194] The user can check the original video of the still image (810) through the original video display section (850).

[0195] In various embodiments, the user interface includes an uncertainty score display (860) that provides information on the uncertainty score evaluated for each anatomical structure of the heart in the still image (810) (e.g., entropy meter, mutual information between model posterior probabilities (e.g., Bayesian Active Learning by Disagreement; BALD), deviation rate and mean standard deviation).

[0196] In various embodiments, the user interface includes a probe location indicator (870) that tracks and provides the location of the probe.

[0197] According to the method of providing guidelines in various embodiments of the present invention, the user can more quickly visually perceive information related to the quantification of heart structure.

[0199] Evaluation 1: Evaluation of the Predictive Model

[0200] Hereinafter, with reference to FIGS. 9a and 9b and 10a to 10c, the evaluation results of a prediction model applied to various embodiments of the present invention will be described. FIGS. 9a and 9b and FIGS. 10a to 10c illustrate the evaluation results of a prediction model applied to a guideline provision method according to various embodiments.

[0201] This evaluation was performed based on the classification of standard and non-standard cross-sections of the quality meter for PLAX cross-sections and / or A4C ​​for the prediction model.

[0202] More specifically, referring to Figures 9a and 9b, the prediction model is shown to have excellent diagnostic performance, with a sensitivity of 0.912 and a specificity of 0.954 in the PLAX cross-section, and a particularly high AUC of 0.963.

[0203] Referring further to Figures 10a and 10b, the prediction model showed a high AUC close to 0.96 in both the PLAX cross-section and the A4C cross-section, and referring to the matrix in Figure 10c, it appears that the prediction model showed a high AUC of 0.9 or higher.

[0204] In other words, the present invention enables the provision of rapid and reliable analysis results for cardiac ultrasound images having various cross-sectional views, regardless of the proficiency of medical personnel, and can contribute to faster and more accurate decision-making and the establishment of treatment plans during the image analysis stage. Hereinafter, with reference to FIGS. 11a to 11f, a method for providing guidelines for cardiac ultrasound images according to another embodiment of the present invention will be described in detail.

[0205] FIGS. 11a to 11f illustrate an exemplary method for providing guidelines for cardiac ultrasound images according to one embodiment of the present invention.

[0206] First, referring to FIG. 11a, a cardiac ultrasound image of an object is received (S1110), and a probe guidance is determined based on the cardiac ultrasound image using a prediction model (S1120).

[0207] Referring to FIG. 11b, in various embodiments of the present invention, a cardiac ultrasound image of an object is received (S1110), a cross-sectional view of the cardiac ultrasound image is received (S1130), and a probe guidance is determined based on the cardiac ultrasound image and the cross-sectional view using a prediction model (S1122).

[0208] Referring to FIG. 11c, in various embodiments of the present invention, a cardiac ultrasound image of an object is received (S1110), a cross-sectional view of the cardiac ultrasound image is received (S1130), and a first probe guidance and a second probe guidance according to movement are determined based on the cardiac ultrasound image and the cross-sectional view using a prediction model (S1124).

[0209] At this time, the second probe guidance is defined as guidance for a user with higher proficiency than the first probe guidance or for larger probe movements, and the first probe guidance includes at least one probe operation guidance among standard, probe head up, probe head down, probe head left, probe head right, probe clockwise, and probe counter-clockwise, and the second probe guidance includes at least one probe operation guidance among rib up, rib down, slide left, and slide right.

[0210] In more diverse embodiments of the present invention, the prediction model is configured to probabilistically predict each of the first probe guidance and the second probe guidance.

[0211] Next, a first probe guidance is provided (S1142), and if the probability or quality score for the first probe guidance is greater than or equal to a predetermined level, a second probe guidance is optionally provided (S1144).

[0212] That is, the information provision method according to various embodiments of the present invention can be configured to provide probe guidance stepwise according to the user's proficiency or degree of movement.

[0213] More specifically, the prediction model first provides a first probe guidance based on ultrasound images, and if the result of the user operating according to the guidance meets a predetermined level of reliability, a second probe guidance that guides more precise operation may be optionally provided.

[0214] Referring together with FIG. 11d, in various embodiments of the present invention, a prediction model (1120) can determine guidance (1122) for probe manipulation required by the examiner by taking a cardiac ultrasound image (1112) and a cross-sectional view (1114) as inputs.

[0215] More specifically, the prediction model (1120) can extract features from a cardiac ultrasound image, determine the probe operation required for the input cross-section, and output probe operation guidance (1122).

[0216] At this time, two levels of guidance can be provided by considering the magnitude of the probe movement, and a first probe guidance (Beginner Guidance) with a relatively small magnitude of probe movement and a second probe guidance (Advanced Guidance) with a relatively large magnitude of probe movement (1144) can be determined respectively.

[0217] For example, the first probe guidance can provide guidelines related to small / precise movement and rotation operations of the probe (Head Up / Down, Left / Right, Rotate Clockwise / Counterclockwise) to help beginners easily follow along. Meanwhile, the second probe guidance includes guidance related to larger operations (Rib Up / Down, Slide Left / Right) and can help experienced users acquire more accurate ultrasound images.

[0218] Through this, you can learn basic operations such as tilting, rocking, and rotating while keeping the probe's starting point position fixed, and receive guidance to find the standard view. If the standard view has been formed but fine correction is still needed, you can receive guidance to obtain the optimal ultrasound image by performing a sliding operation with a larger movement.

[0219] Consequently, the information providing method according to various embodiments of the present invention can contribute to maintaining consistency in ultrasound imaging and acquiring more accurate images by providing stepwise guidance based on the magnitude of probe movement.

[0220] In more diverse embodiments of the present invention, the prediction model may take a cardiac ultrasound image as input, segment the anatomical structures of the heart, and determine probe guidance according to the movement of the ultrasound probe based on the segmentation results.

[0221] In more diverse embodiments of the present invention, the prediction model may take a cardiac ultrasound image as input, classify cross-sections, and determine probe guidance according to ultrasound probe movement corresponding to the cross-sections.

[0222] More specifically, referring further to FIG. 11e, in a more diverse embodiment of the present invention, a cardiac ultrasound image of an object is received (S1110), and anatomical structures within the received cardiac ultrasound image are segmented using a prediction model trained to segment the anatomical structures of the heart and classify cross-sectional views of the input cardiac ultrasound image using the cardiac ultrasound image as input (S1150). Then, cross-sectional views of the received cardiac ultrasound image are classified using the prediction model (S1160), and probe guidance according to the movement of the ultrasound probe is determined based on the segmentation result and the classified cross-sectional views (S1170).

[0223] Referring together with Fig. 11f, a cardiac ultrasound image (1112) of an object and a cross-sectional view (1114) of the image are input into a prediction model (1120), and the prediction model can divide anatomical structures based on this and classify the cross-sectional views to output a cross-sectional view classification result (VC) (1162) and divided anatomical structures (1152).

[0224] At this time, the prediction model (1120) can segment the anatomical structures of the heart using a Fully Convolutional Network (FCN) based network to perform the function of segmenting the anatomical structures of the heart and classifying the cross-sections of the image using the received cardiac ultrasound image (1112) as input, and optionally calculate an uncertainty score for the segmentation result of the anatomical structures. In addition, classification is performed using a CNN based network for cross-section classification, and through this, it can be classified whether the image corresponds to a specific cross-section such as PLAX, PSAX, A4C, etc.

[0225] The prediction model (1120) can extract features from the results of the classification of the segmented anatomical structures and cross-sections and perform a merging process to determine probe operation guidance based on this. In this process, the prediction model (1120) can comprehensively analyze the features of the segmented anatomical structures to determine the probe operation guidance (1122) corresponding to the cross-sections.

[0226] More specifically, the prediction model (1120) can provide small probe manipulation of the first probe guidance (Beginner Guidance) for beginner testers or larger manipulation of the second probe guidance (Advanced Guidance) for experienced users.

[0227] The output probe guidance (1122) is provided to the examiner, and the examiner can operate the probe by referring to it.

[0228] Consequently, the prediction model (1120) used in various embodiments of the present invention can divide anatomical structures and classify cross-sections simultaneously, extract key features based on the results, and then provide optimal probe operation guidance. Through this, the present invention can contribute to providing customized guidelines based on the user's movements and improving the quality of cardiac ultrasound images in real time.

[0229] Meanwhile, in more diverse embodiments of the present invention, the prediction model may be a model trained to distinguish between a normal cross-section and an Odd (abnormal) cardiac ultrasound image using a cardiac ultrasound image as input.

[0230] More specifically, the predictive model can determine whether the captured cardiac ultrasound image contains a medically valid cross-section and detect abnormal ultrasound images (e.g., when the image is taken from the wrong angle, when the ultrasound probe is not placed in the correct position, when anatomical structures are not clearly visible, etc.).

[0231] To this end, the prediction model establishes criteria for maintaining the correct cross-section and can classify images that do not match these criteria as Odd cardiac ultrasound images.

[0232] Consequently, the guideline providing method according to an embodiment of the present invention can analyze the quality of cardiac ultrasound images in real time by utilizing a prediction model trained to classify Odd cardiac ultrasound images, detect incorrect cross-sectional views, and provide an optimal probe manipulation method to correct them. This supports the examiner in acquiring more accurate and stable cardiac ultrasound images and enables the maintenance of consistent image quality regardless of the examiner's proficiency.

[0234] Next, referring to FIGS. 12a and 12b, various embodiments of the present invention can output the segmentation results of cardiac ultrasound images in real time and probabilistically predict and provide probe guidance based thereon.

[0235] Referring to the display on the left, the actual captured cardiac ultrasound image is shown, while the image on the right displays the results of the automatic segmentation of anatomical structures by color, allowing for intuitive verification of their location and shape. This segmentation process is performed automatically via a predictive model and can be updated in real time.

[0236] Additionally, by referring to the guidance information section at the top right, a first probe guidance (e.g., probe_left or standard) and a second probe guidance (e.g., slide_left or slide_right) for the current probe operation can be probabilistically predicted and provided. In this case, the guidance prediction can be performed based on the classification results of the segmented structure and cross-section, and the optimal probe operation direction can be suggested to the inspector in real time.

[0237] By referring to the probability graph at the bottom right, the user interface visually displays the predicted probabilities for various probe manipulation options, allowing the examiner to intuitively understand with high probability how appropriate a specific manipulation is.

[0238] More specifically, in FIG. 12a, the probability of the first probe guidance operation for Probe_left and the second probe guidance operation for Slide_left is highest, and in FIG. 12b, the probability of the first probe guidance operation for Standard and the second probe guidance operation for Slide_right is highest. This may mean that the prediction model according to various embodiments of the present invention predicted that the corresponding operation is required.

[0239] In this case, the second probe guidance may be provided optionally if the first probe guidance corresponds to a standard, but is not limited thereto.

[0240] Consequently, in various embodiments of the present invention, the results of segmenting anatomical structures of a cardiac ultrasound image in real time through a user interface and probabilistically predicted probe manipulation guidance based thereon can be provided.

[0241] In other words, the present invention can support more precise ultrasound image acquisition by providing intuitive visual feedback through a user interface.

[0242] Referring together with FIG. 12c, according to various embodiments of the present invention, the quality evaluation of ultrasound images and probe guidance can be predicted in real time through a user interface, and intuitive visual feedback can be provided to the user.

[0243] More specifically, the predicted cross-section classification result is displayed at the top, allowing the examiner to verify which cardiac cross-section the currently captured ultrasound image corresponds to. In addition, probe guidance prediction based on the examiner's movement is performed, and beginner guidance and advance guidance based on probe movement can be provided, respectively.

[0244] Furthermore, guidance on the direction of probe operation to be adjusted in real time along with the captured ultrasound image is provided through the user interface.

[0245] In addition, quality and structural evaluation scores of the captured images can be provided to the examiner through the user interface. At this time, the Structural Score and Quality Score are output and provided separately, allowing the examiner to quantitatively evaluate the accuracy of anatomical structure recognition and the overall image quality of the currently captured images.

[0246] Consequently, the present invention can support the user in acquiring more accurate and consistent ultrasound images by analyzing ultrasound images in real time and providing probe guidance determined based on predicted cross-sectional views and image quality evaluation results.

[0248] Evaluation 2: Evaluation of the Predictive Model

[0249] Hereinafter, with reference to FIGS. 13a to 13d, the evaluation results of a prediction model applied to various embodiments of the present invention will be explained.

[0250] First, referring to FIG. 13a, the results of an analysis of quality evaluation indicators using entropy values ​​to evaluate the performance of a prediction model according to various embodiments of the present invention are shown. At this time, the prediction performance was evaluated based on AUC (Area Under the Curve), ACC (Accuracy), SEN (Sensitivity), and SPE (Specificity). Furthermore, the performance of the prediction model was compared by cross-section (PLAX, PSAX-AV, PSAX-MV, PSAX-PM, PSAX-APEX, A4C, A2C, and A3C).

[0251] More specifically, when using a prediction model consisting solely of a module that performs cross-section classification, the overall AUC and ACC appear to be low (E vc (Reference). In this case, it was found that prediction performance improves when a module performing the classification of anatomical structures is added (E vc +E seg (Reference), in particular, it was found that the highest prediction performance was observed across all cross-sections when a module for calculating the image quality score was added (E vc +E seg +E structure +E center ).

[0252] The above results may imply that the prediction model of the present invention can precisely evaluate the quality of ultrasound images by utilizing various features, and that quality evaluation is possible with high accuracy.

[0253] Next, referring to FIG. 13b, the results of measuring the probe movement prediction performance of the prediction model for each cross-section (PLAX, PSAX-AV, PSAX-MV, PSAX-PM, PSAX-APEX, A4C, A3C, and A2C) are shown.

[0254] More specifically, regarding accuracy, the prediction model was found to have demonstrated relatively high prediction performance in PSAX-MV and A4C cross-sections.

[0255] In the case of the F1 Score indicator, the highest value of 0.835 was observed in the A4C cross-section, and an F1 Score value of 0.828 was observed in the PSAX-MV cross-section. This may mean that the prediction model demonstrated excellent probe movement prediction performance in the corresponding cross-section.

[0256] Furthermore, referring to the results of the mean accuracy (mAP) analysis, the highest performance was observed with 0.911 in the PSAX-MV cross-section and 0.908 in the A4C cross-section, which may mean that the prediction model is operating with high confidence to predict probe movement in the corresponding cross-section.

[0257] Next, referring to FIG. 13c, the results of measuring the prediction performance for a specific probe movement for each cross-sectional view and analyzing the prediction accuracy of each movement are shown.

[0258] More specifically, the Rock Right movement in the PLAX cross-section showed a high prediction accuracy of 0.998, and the Rock Right movement in the PSAX-AV and PSAX-MV cross-sections showed high prediction performance of 0.967 and 0.989, respectively.

[0259] Referring further to FIG. 13d, a matrix for analyzing the probe movement prediction performance of the prediction model in more detail is shown.

[0260] In this case, when analyzing the relationship between the predicted probe operation direction and the actual operation direction for each cross-section (PLAX, PSAX-MV, A4C, PSAX-AV, PSAX-PM, PSAX-APEX, A3C, and A2C), it may indicate that the more values ​​located on the diagonal, the more accurately the model predicted.

[0261] More specifically, the prediction model was shown to have predicted probe movements with high accuracy in the cross-sections of PLAX, PSAX-MV, and A4C.

[0262] The above results suggest that the prediction model according to various embodiments of the present invention can accurately predict probe movement for each cross-sectional view, and that guidance optimization considering the characteristics of each cross-sectional view is possible to provide accurate probe operation.

[0263] In other words, the present invention provides a predictive model that automatically segments the anatomical structures of the heart and classifies cross-sections, thereby enabling the rapid and accurate recognition of specific anatomical structures within ultrasound images. This allows for the acquisition of more standardized cardiac ultrasound images without relying on the examiner's experience.

[0264] Furthermore, the present invention can automatically calculate an evaluation score based on the segmentation results and cross-sectional classification of the image, thereby quantitatively evaluating the quality of the captured ultrasound image in real time.

[0265] Accordingly, the inspector receives an evaluation score to determine in real time whether the image quality meets certain standards, and if it is insufficient, can receive probe guidance.

[0266] In other words, the present invention makes it possible to provide highly reliable analysis results for cardiac ultrasound images regardless of the examiner's proficiency, and can contribute to establishing more accurate decision-making and treatment plans during the image analysis stage.

[0268] Although embodiments of the present invention have been described in more detail with reference to the attached drawings, the present invention is not necessarily limited to these embodiments and may be modified in various ways within the scope of the technical spirit of the present invention. Accordingly, the embodiments disclosed in the present invention are intended to explain, not limit, the technical spirit of the present invention, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present invention shall be interpreted by the claims below, and all technical spirits within an equivalent scope shall be interpreted as being included within the scope of rights of the present invention. Explanation of the symbols

[0269] 100: Medical staff device 110: Memory interface 120: Processor 130: Peripheral Interfaces 140: I / O Subsystem 141: Touchscreen Controller 142: Other Input Controllers 143: Touch screen 144: Other Input Control Devices 150: Memory 151: Operating System 152: Communication Module 153: GUI Module 154: Sensor processing module 155: Phone module 156: Applications 156-1, 156-2: Application 157: Digital Assistant Client Module 158: User data 160: Motion sensor 161: Light sensor 162: Proximity sensor 163: Other sensors 170: Camera subsystem 171: Optical sensor 180: Communications subsystem 190: Audio Subsystem 191: Speaker 192: Microphone 300: Server for providing guidelines 310: Communication Interface 311: Wired communication port 312: Wireless circuit 320: Memory 321: Operating System 322: Communication Module 323: User Interface Module 324: Application 330: I / O interface 340: Processor

Claims

Claim 1 A method for providing guidelines for a cardiac ultrasound image implemented by a processor, comprising: receiving a cardiac ultrasound image of an object captured by the processor; and determining a first probe guidance and a second probe guidance based on the movement of the received cardiac ultrasound image using a prediction model trained to determine a probe guidance based on the movement of an ultrasound probe using the cardiac ultrasound image as input, wherein the prediction model is configured to probabilistically predict each of the first probe guidance and the second probe guidance, and further comprising: providing the first probe guidance; and if the probability for the first probe guidance is greater than or equal to a predetermined level, selectively providing the second probe guidance. Claim 2 A method for providing guidelines for a cardiac ultrasound image, wherein, in claim 1, the step of receiving a cross-sectional view of the cardiac ultrasound image is further included, and the step of determining the probe guidance is further included, the step of determining the probe guidance based on the received cardiac ultrasound image and the received cross-sectional view using the prediction model. Claim 3 delete Claim 4 A method for providing guidelines for cardiac ultrasound images, wherein, in claim 1, the second probe guidance is defined as guidance with greater movement than the first probe guidance, and the first probe guidance includes at least one probe operation guidance among hold, probe head upward movement (Tilt Down), probe head downward movement (Tilt Up), probe head left movement (Rock Right), probe head right movement (Rock Left), probe head right tilt (Tilt Right), probe head left tilt (Tilt Left), probe head upward shaking (Rock Down), probe head downward shaking (Rock Up), probe clockwise rotation (Rotate Clockwise), and probe counter-clockwise rotation (Rotate Counter-Clockwise), and the second probe guidance includes at least one probe operation guidance among slide up, slide down, slide left, and slide right. Claim 5 delete Claim 6 A method for providing guidance for a cardiac ultrasound image, wherein, in claim 1, the prediction model is further configured to segment anatomical structures of the heart using a cardiac ultrasound image as input, and the step of determining the probe guidance further comprises the step of segmenting the anatomical structures within the received cardiac ultrasound image using the prediction model, and the step of determining the probe guidance based on the segmentation result using the prediction model. Claim 7 A method for providing guidance for a cardiac ultrasound image, wherein, in claim 1, the prediction model is further configured to classify cross-sectional views of the input cardiac ultrasound image by taking a cardiac ultrasound image as input, and the step of determining the probe guidance further comprises the step of classifying cross-sectional views of the received cardiac ultrasound image using the prediction model, and the step of determining a probe guidance corresponding to the classified cross-sectional views using the prediction model. Claim 8 A method for providing guidelines for cardiac ultrasound images, wherein, in claim 7, the step of classifying the cross-section includes classifying the received cardiac ultrasound image into at least one cross-section among PLAX, PSAX-AV, PSAX MV, PSAX PM, PSAX APEX, A4C, A3C, and A2C using the prediction model, and the step of determining the corresponding probe guidance includes determining the probe guidance for the at least one cross-section. Claim 9 A device for providing guidelines for a cardiac ultrasound image, comprising: a communication unit configured to receive a cardiac ultrasound image of an object captured; and a processor functionally connected to the communication unit; wherein the processor is configured to determine a first probe guidance and a second probe guidance based on the movement of the received cardiac ultrasound image using a prediction model learned to determine a probe guidance based on the movement of the ultrasound probe with the cardiac ultrasound image as input, and wherein the prediction model is configured to probabilistically predict each of the first probe guidance and the second probe guidance, and further comprises an output unit configured to provide the first probe guidance and, if the probability for the first probe guidance is greater than or equal to a predetermined level, selectively provide the second probe guidance. Claim 10 In claim 9, the device for providing guidelines for cardiac ultrasound images is a model trained to determine the probe guidance using a cardiac ultrasound image and a cross-sectional view of the cardiac ultrasound image as input. Claim 11 delete Claim 12 A device for providing guidelines for cardiac ultrasound images, wherein the second probe guidance is defined as guidance with greater movement than the first probe guidance, and the first probe guidance includes at least one probe operation guidance among hold, probe head tilt down, probe head tilt up, probe head rock right, probe head rock left, probe head tilt right, probe head tilt left, probe head rock down, probe head rock up, probe clockwise, and probe counter-clockwise; and the second probe guidance includes at least one probe operation guidance among slide up, slide down, slide left, and slide right. Claim 13 delete Claim 14 A device for providing guidance for a cardiac ultrasound image, wherein, in claim 9, the prediction model is further configured to segment anatomical structures of the heart using a cardiac ultrasound image as input, and the processor is further configured to segment the anatomical structures within the received cardiac ultrasound image using the prediction model and to determine probe guidance based on the segmentation result using the prediction model. Claim 15 A device for providing guidelines for cardiac ultrasound images, wherein, in claim 9, the prediction model is further configured to classify cross-sectional views of the input cardiac ultrasound image by taking the cardiac ultrasound image as input, and the processor is further configured to classify cross-sectional views of the received cardiac ultrasound image using the prediction model and to determine probe guidance corresponding to the classified cross-sectional views using the prediction model. Claim 16 A device for providing guidelines for cardiac ultrasound images, wherein the processor is further configured to classify the received cardiac ultrasound image into at least one cross-section of PLAX, PSAX-AV, PSAX MV, PSAX PM, PSAX APEX, A4C, A3C, and A2C using the prediction model, and to determine probe guidance for the at least one cross-section.

Images