Method and apparatus for analyzing biometric data on basis of artificial intelligence model to generate diagnostic information about user, and program therefor

Abstract

The present disclosure provides a method and computer apparatus for analyzing biometric data to generate diagnostic information about a user, and a program therefor. The method according to an embodiment of the present disclosure comprises the steps of: analyzing biometric data on the basis of a pre-trained neural network model to determine a user's condition; on the basis of the user's condition and feature information corresponding to the biometric data, acquiring, from a pre-established database, context data that assists in explaining the user's condition; and generating a prompt on the basis of the context data, and providing the prompt to a language model to generate diagnostic information regarding the user's condition.

Description

Method, device and program for generating user diagnostic information by analyzing biometric data based on an artificial intelligence model

[0001] The present disclosure relates to deep learning technology in the medical field, and more particularly, to a method, device, and program for generating user diagnostic information by analyzing biometric data based on an artificial intelligence model.

[0002] With the advancement of electronic communication, the medical field has continuously developed technologies that assess a user's health status based on biosignals and automatically analyze the presence of disease. Electrocardiogram (ECG) data, in particular, is a key indicator for the early detection of heart disease and is widely utilized in various clinical settings. However, existing analysis technologies have focused on quantifying biosignals or simply classifying them into specific states based on established criteria. This has made it difficult to fully reflect the complex context required in clinical practice, and has limited the ability to provide explanations of analysis results in natural language.

[0003] To overcome these limitations, active efforts have recently been made to utilize large language models (LLMs) in diagnostic applications. LLMs possess the ability to understand and generate diverse text data formats, making them an effective tool for enhancing the explainability of biosignal analysis results by generating diagnostic explanations in natural language.

[0004] One approach to utilizing LLMs is to fine-tune them to generate explanations tailored to specific disease groups or diagnostic tasks. However, fine-tuning LLMs requires large, high-quality medical datasets, and building and managing such datasets presents significant challenges. Furthermore, fine-tuned models are often over-optimized for specific purposes, making them difficult to apply to other diseases or clinical situations.

[0005] Accordingly, a new method is needed to efficiently generate diagnostic explanations in natural language form by utilizing already trained large-scale language models without separate fine-tuning.

[0006] The present disclosure is conceived in response to the aforementioned background technology, and aims to provide a method, device, and program for generating user diagnostic information by analyzing biometric data based on an artificial intelligence model.

[0007] However, the problems to be solved in this disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood based on the description below.

[0008] According to an embodiment of the present disclosure for realizing the task described above, a method for generating user diagnostic information by analyzing biometric data based on an artificial intelligence model, which is performed by a computing device including at least one processor, includes the steps of: analyzing biometric data based on a pre-learned neural network model to determine a user's status; acquiring context data that assists in explaining the user's status from a pre-built database based on the user's status and feature information corresponding to the biometric data; and generating a prompt based on the context data and providing the prompt to a language model to generate diagnostic information regarding the user's status.

[0009] Alternatively, the method further includes a step of inputting the biometric data into a machine learning model to extract and obtain feature information corresponding to the biometric data.

[0010] Alternatively, the user's condition includes the presence or absence of heart disease or a risk group for heart disease, and the step of determining the user's condition includes the step of inputting the biometric data into the pre-trained neural network model to obtain a score corresponding to the user's possibility of heart disease, and determining the user's condition based on the obtained score.

[0011] Alternatively, the above-mentioned pre-built database includes a plurality of first text information regarding each of a plurality of characteristic information extractable from the biometric data and a plurality of second text information generated by an expert regarding a plurality of states identifiable through the biometric data, wherein the first text information and the second text information may each be generated based on text written by a domain expert.

[0012] Alternatively, the step of obtaining context data for assisting in the description of the user's status from the above-mentioned pre-built database includes the step of selecting third text information corresponding to the feature information from among the plurality of first text information based on a similarity between the text information corresponding to the feature information and each of the first text information, the step of selecting fourth text information corresponding to the user's status from among the plurality of second text information based on a similarity between the text information corresponding to the user's status and each of the second text information, and the step of obtaining the context data based on the third text information and the fourth text information.

[0013] Alternatively, the step of selecting the third text information includes a step of identifying a similarity between the text information corresponding to the feature information and each of the first text information based on a distance between a first vector obtained by embedding the text information corresponding to the feature information and a plurality of second vectors obtained by embedding each of the first text information.

[0014] Alternatively, the step of selecting the fourth text information includes a step of identifying a similarity between the text information corresponding to the user's state and each of the second text information based on a distance between a third vector obtained by embedding the text information corresponding to the user's state and a plurality of fourth vectors obtained by embedding each of the second text information.

[0015] Alternatively, the plurality of first text information may be composed of texts each describing a plurality of characteristic information related to the biometric data, and the plurality of second text information may be composed of texts each describing a plurality of states of the user related to the biometric data.

[0016] Alternatively, the step of obtaining context data for assisting in describing the user's status may include the step of identifying first characteristic information related to the user's status among a plurality of characteristic information corresponding to the biometric data, and the step of obtaining context data for assisting in describing the user's status from a pre-built database based on the user's status and the first characteristic information.

[0017] Alternatively, the step of generating diagnostic information about the user's status may include the step of generating a prompt based on the first characteristic information, the user's status, and the context data.

[0018] Alternatively, the biometric data may include electrocardiogram data, and the step of determining the user's condition may include a step of obtaining lead-specific feature information of the electrocardiogram data and global feature information of the electrocardiogram data using the pre-learned neural network model, and the step of obtaining context data may include a step of obtaining context data that assists in explaining the user's condition from a pre-built database using at least one of the lead-specific feature information and the global feature information as a query.

[0019] According to an embodiment of the present disclosure for realizing the task described above, a computing device for generating user diagnostic information by analyzing biometric data includes a processor including at least one core and a memory including program codes executable by the processor, wherein the processor analyzes the biometric data based on a pre-learned neural network model to determine the user's status, obtains context data that assists in explaining the user's status from a pre-built database based on the user's status and feature information of the biometric data corresponding to the biometric data, generates a prompt based on the context data, and provides the prompt to a language model, thereby generating diagnostic information regarding the user's status.

[0020] According to one embodiment of the present disclosure for realizing the task as described above, a computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations for analyzing biometric data to generate user diagnostic information, the operations including an operation for analyzing biometric data to determine a user's status based on a pre-learned neural network model, an operation for obtaining context data that assists in explaining the user's status from a pre-built database based on the user's status and characteristic information of the biometric data corresponding to the biometric data, and an operation for generating a prompt based on the context data and providing the prompt to a language model to generate diagnostic information regarding the user's status.

[0021] According to the method of generating user diagnostic information by analyzing biometric data of the present disclosure, prompts are generated based on feature information extracted from the biometric data and user status information, and these prompts are provided to a Large Language Model (LLM) to generate diagnostic information in natural language. This allows for the provision of descriptive diagnostic information that is easy for medical professionals to understand and utilize.

[0022] In addition, since the present invention can generate diagnostic explanations by directly utilizing pre-learned LLM, it can flexibly respond to various clinical situations without separate fine-tuning, thereby reducing the burden of securing medical data and the cost of maintaining and operating the model.

[0023] FIG. 1 is a block diagram of a computing device that analyzes biometric data to generate user diagnostic information according to one embodiment of the present disclosure.

[0024] FIG. 2 is a flowchart schematically illustrating a method for generating user diagnostic information by analyzing biometric data according to one embodiment of the present disclosure.

[0025] FIG. 3 is an exemplary diagram illustrating a method for obtaining context data based on feature information and state information according to one embodiment of the present disclosure.

[0026] FIG. 4 is an exemplary diagram illustrating a method for obtaining context data based on feature information and state information according to one embodiment of the present disclosure.

[0027] FIG. 5 is a detailed configuration diagram of a computing device according to another embodiment of the present disclosure.

[0028] Below, embodiments of the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present disclosure. The embodiments presented in this disclosure are provided to enable those skilled in the art to utilize or implement the contents of the present disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure may be implemented in various different forms and is not limited to the embodiments described below.

[0029] Throughout the specification of this disclosure, identical or similar drawing numbers refer to identical or similar components. Furthermore, for clarity in the description of this disclosure, drawing numbers for parts unrelated to the description of this disclosure may be omitted in the drawings.

[0030] The term "or" as used herein is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified herein or clear from context, "X employs A or B" should be understood to mean either of the natural inclusive permutations. For example, unless otherwise specified herein or clear from context, "X employs A or B" can be interpreted to mean either X employs A, X employs B, or X employs both A and B.

[0031] The term "and / or" as used herein should be understood to refer to and include all possible combinations of one or more of the related concepts listed.

[0032] The terms "comprises" and / or "comprising" as used herein should be understood to mean the presence of certain features and / or components. However, it should be understood that the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other features, other components, and / or combinations thereof.

[0033] Unless otherwise specified in this disclosure or unless the context makes it clear that the singular form is intended to be referred to, the singular should generally be construed to include “one or more.”

[0034] The term "Nth (N is a natural number)" used in the present disclosure can be understood as an expression used to mutually distinguish components of the present disclosure based on a predetermined standard such as a functional perspective, a structural perspective, or convenience of explanation. For example, components performing different functional roles in the present disclosure can be distinguished as a first component or a second component. However, components that are substantially the same within the technical spirit of the present disclosure but must be distinguished for convenience of explanation may also be distinguished as a first component or a second component.

[0035] The term "acquisition" as used in this disclosure may be understood to mean not only receiving data through a wired or wireless communication network with an external device or system, but also generating data in an on-device form.

[0036] Meanwhile, the term "module" or "unit" used in the present disclosure can be understood as a term referring to an independent functional unit that processes computing resources, such as a computer-related entity, firmware, software or a part thereof, hardware or a part thereof, or a combination of software and hardware. At this time, the "module" or "unit" may be a unit composed of a single element, or a unit expressed as a combination or set of multiple elements. For example, as a narrow concept, a "module" or "unit" may refer to a hardware element of a computing device or a set thereof, an application program that performs a specific function of software, a processing process implemented through software execution, or a set of instructions for program execution, etc. In addition, as a broad concept, a "module" or "unit" may refer to the computing device itself that constitutes the system, or an application running on the computing device, etc. However, since the above-described concept is only an example, the concept of “module” or “part” may be defined in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0037] The term "model" as used herein may be understood as a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or an abstract model of a processing process to solve a specific problem. For example, a neural network "model" may refer to the entire system implemented as a neural network that has problem-solving capabilities through learning. In this case, the neural network can have problem-solving capabilities by optimizing the parameters connecting nodes or neurons through learning. A neural network "model" may include a single neural network or a set of neural networks that are a combination of multiple neural networks.

[0038] The term "data" used in this disclosure may include "image," "signal," and the like. The term "image" used in this disclosure may refer to multidimensional data composed of discrete image elements. In other words, "image" may be understood as a term referring to a digital representation of an object visible to the human eye. For example, "image" may refer to multidimensional data composed of elements corresponding to pixels in a two-dimensional image. "Image" may refer to multidimensional data composed of elements corresponding to voxels in a three-dimensional image.

[0039] The explanation of the above terms is intended to aid understanding of the present disclosure. Therefore, unless explicitly stated as limiting the contents of the present disclosure, it should be noted that the above terms are not intended to limit the technical ideas of the present disclosure.

[0040] FIG. 1 is a block diagram of a computing device that analyzes biometric data to generate user diagnostic information according to one embodiment of the present disclosure.

[0041] A computing device (100) according to an embodiment of the present disclosure may be a hardware device or a part of a hardware device that performs comprehensive processing and calculation of data, or may be a software-based computing environment connected to a communication network. For example, the computing device (100) may be a server that performs intensive data processing functions and shares resources, or may be a client that shares resources through interaction with a server. In addition, the computing device (100) may be a cloud system in which multiple servers and clients interact to comprehensively process data. Since the above description is only one example related to the type of computing device (100), the type of computing device (100) may be configured in various ways within a category understandable to those skilled in the art based on the contents of the present disclosure. For example, the computing device (100) may be implemented as various electronic devices such as a server, a desktop, a laptop, a smartphone, a smart watch, a smart ring, etc.

[0042] Referring to FIG. 1, a computing device (100) according to an embodiment of the present disclosure may include a processor (110), a memory (120), and a communication interface (communication unit) (130). However, FIG. 1 is merely an example, and the computing device (100) may include other components for implementing a computing environment. In addition, only some of the disclosed components may be included in the computing device (100).

[0043] A processor (110) according to an embodiment of the present disclosure may be understood as a configuration unit including hardware and / or software for performing computing operations. For example, the processor (110) may read a computer program to perform data processing for machine learning. The processor (110) may process computational processes such as processing input data for machine learning, feature extraction for machine learning, and error calculation based on backpropagation. The processor (110) for performing such data processing may include a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), or a field programmable gate array (FPGA). The above-described type of processor (110) is only one example, and thus, the type of processor (110) may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0044] The processor (110) is connected to other components of the computing device (100) (i.e., memory (120) and communication interface (130)) and controls the overall operation of the computing device (100).

[0045] The memory (120) according to one embodiment of the present disclosure may be understood as a configuration unit including hardware and / or software for storing and managing data processed in the computing device (100). That is, the memory (120) may store any type of data generated or determined by the processor (110) and any type of data received by the communication interface (130). For example, the memory (120) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory (120) may also include a database system that controls and manages data in a predetermined system. The type of memory (120) described above is only one example, and thus the type of memory (120) can be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

[0046] The memory (120) can structure and organize and manage data, combinations of data, and program codes executable in the processor (110) required for the processor (110) to perform operations. For example, the memory (120) can store bio-data (aortic blood pressure data, brain wave data, and electrocardiogram data) received through a communication interface (130) to be described later. In addition, the memory (120) can store a machine learning model trained to extract feature information, a neural network model trained to identify a patient's condition, and a rule-based model, and can store program codes that operate to perform learning of each model (i.e., a machine learning model, a neural network model, a rule-based model, etc.), program codes that operate the neural network model to receive bio-data (aortic blood pressure data, brain wave data, and electrocardiogram data, etc.) and perform inference according to the purpose of use of the computing device (100), and processed data generated as the program codes are executed.

[0047] A communication interface (130) according to an embodiment of the present disclosure may be understood as a component that transmits and receives data through any known wired or wireless communication system. For example, the communication interface (130) may perform data transmission and reception using a wired or wireless communication system such as a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), wireless broadband internet (WiBro), fifth generation mobile communication (5G), ultrawide-band, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity, near field communication (NFC), or Bluetooth. Since the above-described communication systems are only examples, the wired and wireless communication system for data transmission and reception of the communication interface (130) may be applied in various ways other than the above-described examples.

[0048] The communication interface (130) can receive data required for the processor (110) to perform calculations through wired or wireless communication with any system or any client, etc. In addition, the communication interface (130) can transmit data generated through calculations of the processor (110) through wired or wireless communication with any system or any client, etc. For example, the communication interface (130) can receive a patient's biometric data through communication with a cloud server that performs tasks such as databases in a hospital environment, standardization of medical data, or a computing device, etc. The communication interface (130) can transmit output data of each model, intermediate data derived from the calculation process of the processor (110), processed data, etc. through communication with the aforementioned database, server, or computing device, etc.

[0049] FIG. 2 is a flowchart schematically illustrating a method for generating user diagnostic information by analyzing biometric data according to one embodiment of the present disclosure.

[0050] Referring to FIG. 2, according to one embodiment of the present disclosure, the processor (110) analyzes biometric data using a pre-learned neural network model to determine the user's status (S210).

[0051] Specifically, the processor (110) can obtain the user's biometric data from a biometric signal measuring device (e.g., a smartwatch worn by the user, an electrocardiogram measuring device, etc.) that is connected to the computing device (100) through a communication interface of the computing device. The biometric data can obtain various biometric data that can be measured from the user, such as blood pressure, body temperature, photoreceptor blood flow, heart rate, electroencephalogram data, electrocardiogram data, etc. However, for the convenience of explanation of the present disclosure, the biometric data will be described below assuming electrocardiogram data.

[0052] In addition, the processor (110) may directly obtain biometric data from the user through a sensing unit included in the computing device. For example, the biometric data may include electrocardiogram data. In this case, the processor may detect an electrocardiogram signal from the user as multiple electrodes included in the sensing unit are attached to the user's body, and digitally process the detected electrocardiogram signal (e.g., A / D conversion, etc.) to obtain the user's electrocardiogram data. Depending on the measurement method using the multiple electrodes, the processor (110) may obtain various forms of electrocardiogram data, such as 1 lead, 3 leads, 6 leads, and 12 leads.

[0053] According to one embodiment of the present disclosure, the processor (110) may acquire latent feature information of biometric data. Here, the feature information may include quantitative or standard information that can be identified from a biometric signal corresponding to the biometric data. In addition, the feature information may include latent unique information (patterns, waveform trends, etc.) of the biometric data. For example, when the biometric data is electrocardiogram data, the feature information may include at least one feature point that can be identified or inferred from the electrocardiogram signal and a parameter calculated through the feature point. Specifically, the feature information may include the amplitude of the P wave, the time length of the QRS complex, the potential change of the ST segment, the shape of the T wave, the RR interval, etc.

[0054] In particular, the processor (110) can acquire feature information corresponding to biometric data using an artificial intelligence-based model. Here, the artificial intelligence-based model may include a machine learning model, a neural network model (hereinafter, referred to as a first neural network model), and a rule-based model trained to extract latent feature information from biometric data. For example, the processor may input electrocardiogram (ECG) data into the machine learning-based model to extract and acquire feature information from the ECG data. Specifically, the processor (110) may input ECG data into the machine learning model and extract various feature information from the ECG data. To this end, the processor (110) may use a learning data set including a plurality of ECG data and label data corresponding to the plurality of feature information for each ECG data. The processor (110) may calculate a loss function based on the difference between the output result of the machine learning model and the label data, and optimize the parameters of the machine learning model to minimize the loss function. For example, loss functions such as Mean Squared Error (MSE), Mean Absolute Error (MAE), and Cross Entropy Loss can be used. The processor (110) repeatedly performs training of a machine learning model based on these loss functions, thereby enabling more accurate extraction of feature information from biometric data.

[0055] Meanwhile, the biometric data included in the training data used to train an artificial intelligence model (e.g., a machine learning model, a neural network model, etc.) may have the same size. To this end, the biometric signal corresponding to the biometric data may be commonly acquired with a preset length. For example, the processor (110) may acquire an electrocardiogram signal measured from a patient in units of a preset length or divide the signal into multiple pieces by applying a window of a preset length on the time axis. In this case, the electrocardiogram signal may be measured with a length of 10 seconds each through 12 leads and may be measured at 500 points per second. In addition, the processor (110) may digitally process each acquired electrocardiogram signal to acquire multiple electrocardiogram data of the same size. Meanwhile, the electrocardiogram signal may be a limb lead electrocardiogram signal or a single lead electrocardiogram signal.

[0056] Meanwhile, the first neural network model described above may be trained in advance to identify and extract feature information included in each biometric data based on a learning data set composed of multiple biometric data of the same type. The learning data set may include multiple label data in which feature information included in each biometric data is labeled. The processor (110) may input learning data (multiple biometric data included in the learning data set) into the first neural network model, and may calculate a loss function based on the difference between the output value of the first neural network model during the learning process and the label data corresponding to each biometric data. The loss function may be defined as a cross entropy loss for a classification task, and as a mean squared error (MSE) or mean absolute error (MAE) for a regression task, and may also be defined as an objective function for optimizing the balance between precision and recall. Based on the calculated loss function, the processor (110) can adjust the weights of the first neural network model through backpropagation. By repeating this process, the processor (110) can improve the regression and classification performance of the first neural network model with respect to feature information in the biometric data, and ultimately obtain a first neural network model trained to extract feature information from electrocardiogram data. The first neural network model can be implemented as a multi-layer perceptron (MLP), a convolutional neural network (CNN), a recurrent neural network (RNN), a generative adversarial network, etc.

[0057] According to an embodiment of the present disclosure, when at least one feature information is plural, the first neural network model may include one of a regressor and a classifier corresponding to each feature information according to the type of the plural feature information. Specifically, when the feature information is expressed as a continuous numerical value, the output layer corresponding to the feature information may be configured as a regressor (Regression), and for example, feature information such as QRS duration, PR interval, RR interval, heart rate (HR), and heart rate variability (HRV) may be each calculated through a plurality of regressors included in the first neural network model. Meanwhile, when the feature information is calculated as a discrete category value, the feature information may be calculated through a classifier. For example, feature information such as the presence or absence of a P wave, whether the rhythm is irregular, and whether the QRS waveform is normal or abnormal may be calculated through a classifier included in the first neural network model.

[0058] In addition, the first neural network model may be configured with a multi-task learning structure for predicting multiple feature information in parallel. Specifically, the first neural network model may be configured to process electrocardiogram data input through a common encoder, and simultaneously predict each feature information through multiple output layers (specifically, multiple output layers including either a regressor or a classifier) ​​based on the output of the encoder. Based on this structure, the first neural network model can effectively reflect the inherent correlation between multiple feature information, while individually improving the prediction accuracy of the output corresponding to each feature information.

[0059] Meanwhile, according to one embodiment of the present disclosure, when there is a plurality of at least one piece of feature information, the first neural network model may include a plurality of sub-neural network models corresponding to each piece of feature information. Specifically, each sub-neural network model may be composed of an individual neural network structure designed to predict at least one specific piece of feature information, and each sub-neural network model may independently perform learning and inference based on the same electrocardiogram data. For example, feature information expressed as continuous numerical values, such as QRS interval, PR interval, RR interval, heart rate (HR), and heart rate variability (HRV), may be individually calculated through a plurality of regression-based sub-neural network models. Alternatively, feature information expressed as discrete categorical values, such as the presence or absence of a P wave, whether the rhythm is irregular, and whether the QRS waveform is normal or abnormal, may be individually calculated through a plurality of classification-based sub-neural network models. Based on this structure, each sub-neural network model can be individually learned with a structure optimized for the characteristics of specific feature information, thereby minimizing interference between feature information while improving the prediction accuracy for each feature information. The processor (110) can effectively predict multiple feature information for electrocardiogram data by independently training and executing each of the multiple sub-neural network models.

[0060] Meanwhile, the AI ​​model may include multiple sub-AI models, each classified according to the type of biometric data. For example, the machine learning model may include a first machine learning model that extracts feature information from electrocardiogram (ECG) data, a second machine learning model that extracts feature information from electroencephalogram (EEG) data, and so on.

[0061] In addition, the processor (110) can analyze the extracted feature information to determine the user's condition. The processor (110) can determine the user's condition by obtaining a comparison result (e.g., amount of change, etc.) with the extracted feature information or the biometric data acquired at a previous time (specifically, the user's past biometric data acquired at a previous time and stored in memory or the feature information extracted from the past biometric data). Here, the user's condition may include the presence or absence of a specific disease (e.g., left ventricular systolic dysfunction, myocardial infarction, etc.), the degree of possibility of developing a specific disease, etc., and in addition to the disease, may include general health conditions such as stress, fatigue, sleep state, medication responsiveness, etc.

[0062] In particular, the processor (110) can analyze biometric data using a pre-trained neural network model, obtain output information corresponding to the user's condition, and determine the user's condition based on the obtained output information. The pre-trained neural network model can include a neural network model trained to extract latent feature information from the biometric data and determine the user's condition based on the extracted feature information. The processor (110) can input biometric data into a second neural network model to obtain output information corresponding to the user's condition. For example, the output information corresponding to the user's condition can include a user's stress index, a user's fatigue level, a score corresponding to a user's disease possibility, etc.

[0063] To this end, the second neural network model may be trained in advance to output output information corresponding to the user's condition based on a learning data set consisting of multiple biometric data of the same type. The learning data set may include multiple label data in which the user's condition is labeled. For the example described above, the learning data set may be assigned a label indicating whether the user has a disease. The processor (110) may input learning data (multiple biometric data included in the learning data set) into the second neural network model, and may calculate a loss function based on the difference between the output value of the second neural network model during the learning process and the label data corresponding to each biometric data. The loss function may be defined as a cross entropy loss in the case of a classification task, and may also be defined as an objective function for optimizing the balance between precision and recall. Based on the calculated loss function, the processor (110) can adjust the weights of the second neural network model through backpropagation. By repeating this process, the processor (110) can improve the classification performance of the second neural network model regarding the user's condition, and ultimately obtain a second neural network model trained to predict the user's condition from electrocardiogram data.

[0064] The second neural network model can be implemented as a multi-layer perceptron (MLP), convolutional neural network (CNN), recurrent neural network (RNN), or generative adversarial network.

[0065] Meanwhile, the processor (110) can determine the user's condition based on the output information obtained from the second neural network model. For example, if the processor (110) obtains a probability value corresponding to the user's likelihood of having heart disease from the second neural network model or a score calculated based on the probability value, the processor (110) can determine the severity of the user's heart disease by comparing the obtained score (or probability value) with a reference value.

[0066] Meanwhile, the second neural network model may include multiple sub-neural network models trained to identify different user states. For example, the second neural network model may include multiple sub-neural network models trained to generate scores corresponding to the user's likelihood of left ventricular systolic dysfunction, scores corresponding to the likelihood of left ventricular diastolic dysfunction, and scores corresponding to the likelihood of myocardial infarction.

[0067] Meanwhile, the processor (110) can extract an encoder included in a first neural network model for which learning has been completed, and connect a classifier corresponding to a task type defined according to the user's purpose or environment, particularly a user's state to be identified, to the output layer of the extracted encoder to build a second neural network model. In addition, the processor (110) can fine-tune the second neural network model based on a training data set to which labels corresponding to the user's state described above have been assigned.

[0068] Meanwhile, according to another embodiment of the present disclosure, the processor (110) may extract at least one feature information from biometric data using a pre-trained neural network model, and then the processor (110) may identify the user's status based on the at least one feature information extracted using a rule-based model. Here, the rule-based model may be a model that classifies the user's status by comparing the extracted feature information with one or more pre-defined judgment criteria or thresholds corresponding to the status type, according to one or more pre-defined types of the user's status. For example, the rule-based model may include a conditional judgment structure (If-Then rule), a threshold-based comparison, a rule-based classification structure such as a decision tree, or a rule-based ensemble structure that performs a judgment by combining a plurality of rules.

[0069] At this time, for each state type defined in the rule-based model, the type and number of feature information compared with the judgment criteria may be different. In addition, the processor (110) can simultaneously identify multiple state types of the user by inputting multiple extracted feature information into the acquired rule-based model. For example, if the feature information is extracted from an electrocardiogram signal, the rule-based model can simultaneously identify states such as atrial premature contraction, complete atrioventricular block, and atrial fibrillation by determining whether values ​​such as PR interval, QRS width, and QT interval exceed or fall below a specific standard.

[0070] The processor (110) inputs quantitative feature information derived from a neural network model, compares it with a set of conditions defined in a rule-based model, and if the conditions are satisfied, determines that the user has a specific disease or health condition. In this way, by combining the feature information extraction step through a neural network model with the condition determination step through a rule-based model, the processor (110) can simultaneously secure quantitative accuracy and medical interpretability, and can provide the user with the basis and logic of the diagnostic process beyond simple classification results.

[0071] According to one embodiment of the present disclosure, the processor (110) can simultaneously extract lead-specific feature information of electrocardiogram data and global feature information of electrocardiogram data. Specifically, the lead-specific feature information may include information such as PR interval, width of QRS complex, potential of ST segment, amplitude of R wave and T wave, and presence or absence of T wave inversion as feature values ​​independently calculated for each of 12 leads of the electrocardiogram. The global feature information is an index calculated by synthesizing the entire electrocardiogram signal or the entire lead, and may include numerical information such as heart rate, average RR interval, QT interval or corrected QT interval (QTc), potential average value of the entire signal, and the like. The processor (110) can precisely analyze the user status by extracting lead-specific feature information and global feature information together.

[0072] According to one embodiment of the present disclosure, the processor (110) may extract lead-specific feature information and global feature information through different artificial intelligence models. For example, the machine learning model may include multiple machine learning models trained to predict and output lead-specific feature information and global feature information, respectively. To this end, the machine learning model that outputs lead-specific feature information may be trained based on a training data set composed of electrocardiogram data corresponding to each lead, and the machine learning model that outputs global feature information may be trained based on a training data set composed of multiple electrocardiogram data as a single input data. In this regard, the description of the training method of the above-described machine learning model may be equally applicable. Meanwhile, the same may be applied to the first neural network model.

[0073] Additionally, the artificial intelligence model that extracts lead-specific feature information may include multiple artificial intelligence models corresponding to each lead. Each artificial intelligence model may be individually trained based on a training data set comprised of electrocardiogram data acquired through the corresponding lead.

[0074] In addition, the processor (110) can configure separate processing paths for each type of feature information within a single integrated neural network model. For example, lead-specific feature information can be extracted through a neural network (e.g., a convolutional neural network (CNN)) that is independently applied to the electrocardiogram data of each lead. In this case, temporal features of the waveform of each lead can be detected to individually calculate values ​​such as QRS width and R wave amplitude. Global feature information can be extracted through a neural network (e.g., a recurrent neural network (RNN), a long short-term memory (LSTM) neural network, or a transformer-based structure) that reflects the time-series relationship and overall pattern between all leads. These models support the precise derivation of overall physiological indicators such as heart rate or QTc by learning the temporal and spatial correlation of the entire signal.

[0075] FIG. 3 and FIG. 4 are exemplary diagrams showing a method for obtaining context data based on feature information and state information according to one embodiment of the present disclosure.

[0076] Referring back to FIG. 2, according to one embodiment of the present disclosure, the processor (110) determines the user's status (S210) and then acquires context data that assists in explaining the user's status from a pre-built database (300) based on feature information corresponding to the user's status and biometric data (S220). Here, the context data is information related to feature information or status, and may be reference information for supplementing or assisting in the explanation or judgment of the user's feature information and status. For example, the context data may include descriptive text of biometric data (10) written by a medical domain expert (doctor, medical field expert, etc.), descriptive text of each type of feature information, descriptive text of each type of condition, a document containing diagnostic criteria or interpretation guidelines for a specific disease, etc. In addition to text, the context data may also include images or diagrams containing medical descriptions of feature information and condition types, or visual examples of electrocardiogram waveforms.

[0077] Meanwhile, according to an embodiment of the present disclosure, a database (300) constructed may include a plurality of text information (hereinafter, first text information) regarding each of a plurality of characteristic information extractable from biometric data (10) and a plurality of text information (hereinafter, second text information) generated by an expert regarding a plurality of states that can be identified through the biometric data (10). In this case, the first text information and the second text information may each be generated by an expert. The first and second text information may include tags, metadata, or text information defining an image, such as an image, a diagram, or a visual example.

[0078] The database (300) may be implemented by one or more storage devices or servers, and may include a relational database management system (RDBMS), a non-relational database (NoSQL), a cloud-based storage, etc.

[0079] Referring to FIG. 3, the processor (110) may obtain context data as reference information (or auxiliary information) for generating diagnostic information about the user based on the characteristic information and the user's status. In particular, the processor (110) may search for context data that assists in explaining characteristic information extracted from the user's biometric data (10) within the database (300) or may search for context data that assists in explaining the identified user's status by using at least one of the characteristic information and the user's status.

[0080] In particular, the processor (110) can obtain context data that assists in describing the user's status from the pre-built database (300) by using feature information (e.g., lead-specific feature information and global feature information, etc.) and status as queries. For example, referring to FIG. 4, the processor (110) can determine that the user's feature information is 1) the width of the QRS complex is extended to 130 ms, 2) the QTc interval exceeds a reference value, and 3) a left bundle branch block (LBBB) pattern is detected in leads V1 to V6 using the pre-trained second neural network model (30), and that the user's status is a high-risk group for left ventricular systolic dysfunction (LVSD). At this time, the processor (110) can obtain keywords (e.g., QRS width extension, LBBB pattern, QTc interval extension) corresponding to the feature information based on the feature information. Keywords may be converted according to predefined judgment criteria or templates, or corresponding descriptive phrases may be selected based on a mapping table. Alternatively, the processor (110) may obtain keywords corresponding to the feature information using a natural language generation model. The processor (110) may use keywords of the generated feature information as queries to search for text information (hereinafter, third text information) defining or explaining each feature information among a plurality of first text information within the pre-built database (300). In addition, the processor (110) may use the name of the user's condition identified simultaneously (i.e., left ventricular systolic dysfunction or LVSD) as queries to search for text information (hereinafter, fourth text information) defining the disease or including explanations such as diagnostic criteria or medical opinions among a plurality of second text information within the pre-built database (300).

[0081] Meanwhile, according to one embodiment of the present disclosure, the processor (110) may select third text information corresponding to the feature information from among a plurality of first text information based on the similarity between the text information corresponding to the feature information and each first text information, and may select fourth text information corresponding to the user's state from among a plurality of second text information based on the similarity between the text information corresponding to the user's state and each second text information. In addition, the processor (110) may obtain the third text information and the fourth text information as context data.

[0082] Specifically, the similarity may be an indicator indicating the degree of semantic or contextual similarity between keywords of feature information or status information used as a query (e.g., “QRS interval prolongation” and “left ventricular systolic dysfunction”) and a plurality of text information (specifically, a plurality of first and second text information) stored in the database (300). For example, when the keyword “QTc interval prolongation” is used as a query, the processor (110) may search for sentences or documents containing content semantically similar to the keyword among the first text information. At this time, the similarity may be determined not only by whether words are directly identical, but also by including expressions that have clinically similar meanings.

[0083] Meanwhile, according to one embodiment of the present disclosure, the processor (110) may identify the similarity between the text information corresponding to the feature information and each of the first text information based on the distance between the first vector obtained by embedding the text information corresponding to the feature information and the plurality of second vectors obtained by embedding each of the first text information in order to select the third text information. Similarly, the processor (110) may identify the similarity between the text information corresponding to the user's state and each of the second text information based on the distance between the third vector obtained by embedding the text information corresponding to the user's state and the plurality of fourth vectors obtained by embedding each of the second text information in order to select the fourth text information. The embedding process may be performed through a vectorization technique based on natural language processing, and for example, a pre-trained model such as BERT, Sentence-BERT, or Word2Vec may be used.

[0084] The processor (110) may vectorize the first text information and the second text information using a natural language processing-based embedding technique and store them in a preset vector space. The processor (110) may embed queries (e.g., keywords) corresponding to feature information and state information in the preset vector space to generate a first vector and a third vector, and may compare the first vector and the second vector with a plurality of second vectors and a fourth vector corresponding to a plurality of first text information and a plurality of second text information, respectively, to calculate a similarity. The processor (110) may calculate a cosine similarity or a Euclidean distance based on the distance or angle between these vectors, and may select an item with the highest similarity among the first text information as the third text information and an item with the highest similarity among the second text information as the fourth text information based on the similarity. The selected third text information and the fourth text information may be utilized as context data to supplement the interpretation or explanation of the feature information and the state information.

[0085] Meanwhile, according to one embodiment of the present disclosure, the processor (110) may identify first feature information that determines the user's status from among a plurality of feature information acquired through a pre-learned machine learning model (20), and may acquire first context data corresponding to the identified first feature information and second context data corresponding to the user's status from the constructed database (300). Specifically, when a plurality of feature information are extracted from the biometric data (10) through the pre-learned machine learning model (20), the processor (110) may selectively acquire first context data corresponding to one or more feature information (i.e., the first feature information) that is related to the user's status or substantially contributed to determining the user's status. In this case, the user's status may be a status type determined as abnormal compared to a normal status from among a plurality of pre-defined status types, and the processor (110) may acquire second context data that supports an explanation or interpretation of the corresponding abnormal status.

[0086] Referring again to FIG. 2, according to one embodiment of the present disclosure, the processor (110) may generate a prompt based on feature information, the user's status, and context data, and provide the prompt to a language model (40) to generate diagnostic information about the user's status (S230).

[0087] Specifically, the language model (40) may be a large language model (LLM), which may be a pre-trained text generation model capable of understanding and generating various types of natural language data. For example, the language model (40) may include OpenAI's GPT, Google's PaLM, Meta's LLaMA series, etc. The processor (110) may input a prompt to the language model (40) to output a diagnostic explanation or report for the user containing medical meaning regarding the user's condition in natural language.

[0088] In particular, the processor (110) can input a prompt to the language model (40) to obtain an output in the form of a diagnostic report (400) that interprets or explains the user's condition. The prompt includes extracted feature information, determined condition information, and context data (e.g., third and fourth text information) related thereto, and can be configured so that the language model (40) can generate appropriate diagnostic information. The processor (110) can include instructions to guide the language model (40) to clinically interpret the prompt. For example, an instruction such as "Based on the following ECG features, diagnostic results, and context information, write a medical description of the patient's condition" can be included at the top of the prompt, and structured input information can be arranged below it.

[0089] According to one embodiment of the present disclosure, the processor (110) may additionally include instructions regarding the output format within the prompt to clearly specify the response format of the language model (40). For example, to enable the response generated by the language model (40) to be utilized in a structured form, the processor may request that the diagnosis result be returned in JSON format, or a template including specific fields (e.g., diagnosis name, characteristic information and disease definition, clinical judgment, etc.) may be pre-specified. Through this, the generated diagnosis report (400) may be provided not only as a human-understandable natural language description, but also in a structured data format suitable for machine reading and subsequent processing.

[0090] FIG. 5 is a detailed configuration diagram of a computing device (600) according to another embodiment of the present disclosure. Referring to FIG. 5, a computing device (600) according to an embodiment of the present disclosure includes a processor (610), a memory (620), a communication interface (630), a display (640), a user interface (650), a sensing unit (660), a camera (670), a speaker (680), and a microphone (690). Among the configurations illustrated in FIG. 6, a detailed description of configurations that overlap with those illustrated in FIG. 1 will be omitted.

[0091] The display (640) can display various images and texts. Here, the images include both still images and moving images. The display (640) can also output electrocardiogram data and electrocardiogram signals corresponding to the electrocardiogram data (e.g., electrocardiogram graphs), and can also output diagnostic information acquired through a language model. The display (640) can be implemented as various types of displays, such as an LCD (Liquid Crystal Display Panel), an OLED (Organic Light Emitting Diodes), an LCoS (Liquid Crystal on Silicon), a DLP (Digital Light Processing), etc. In addition, the display (640) can also include a driving circuit, a backlight unit, etc., which can be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (640) can be implemented as a touch screen by being combined with a touch panel.

[0092] The display (640) can function not only as an output interface that outputs images via a touch screen, but also as an input interface that receives a user's touch input. The display (640) can display extracted feature information, identified user status, generated prompts, etc.

[0093] The user interface (650) is a configuration used by the computing device (600) to perform interaction with the user, and may include, but is not limited to, at least one of a touch sensor, a motion sensor, a button, a jog dial, and a switch.

[0094] The sensing unit (660) senses the patient's bio-signals to acquire bio-data. For example, the sensing unit (660) may detect the electrical signals of the patient's heartbeat through multiple electrodes, thereby acquiring the patient's electrocardiogram (ECG) data. Alternatively, the sensing unit (660) may include an optical sensor, and may acquire the patient's optical blood flow data through the optical sensor.

[0095] The camera (670) can capture images of the user by photographing objects around the user. Based on the captured images, the processor (610) can determine the user's biological information (height, age, gender, etc.) and include it in the prompt. The camera (670) can be implemented with an imaging device such as an imaging device having a CMOS structure (CIS, CMOS Image Sensor) or an imaging device having a CCD structure (Charge Coupled Device). However, the present invention is not limited thereto, and the camera (670) can be implemented with a camera module having various resolutions capable of capturing an object. Meanwhile, the camera (670) can be implemented with a depth camera (e.g., an IR depth camera), a stereo camera, an RGB camera, etc.

[0096] The speaker (680) is a component that outputs various audio data that have undergone various processing operations, such as decoding, amplification, and noise filtering, by an audio processing unit (not shown). The speaker (680) can output various notification sounds or voice messages. According to one embodiment of the present disclosure, the processor (610) can convert an electrical signal into a user's voice and output it through the speaker (680). As an example, the speaker (680) can output diagnostic information in audio form.

[0097] The microphone (690) can receive the user's voice. The processor (610) can acquire the user's voice to modify or generate a prompt through the microphone (690).

[0098] Meanwhile, according to one embodiment of the present disclosure, a non-transitory computer-readable medium storing a program that performs a method of generating user diagnostic information by analyzing the aforementioned biometric data may be provided. Here, the non-transitory computer-readable medium refers to a medium that semi-permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, the various applications or programs described above may be stored and provided on a non-transitory computer-readable medium, such as a CD, DVD, hard disk, Blu-ray disc, USB, memory card, or ROM.

[0099] The various embodiments of the present disclosure described above can be combined with additional embodiments and modified within a scope that can be understood by those skilled in the art in light of the detailed description set forth above. It should be understood that the embodiments of the present disclosure are illustrative in all respects and not restrictive. For example, each component described as being single may be implemented in a distributed manner, and similarly, components described as being distributed may be implemented in a combined form. Accordingly, all changes or modifications derived from the meaning, scope, and equivalent concepts of the claims of the present disclosure should be interpreted as being included within the scope of the present disclosure.

Claims

1. A method for generating user diagnostic information by analyzing biometric data, performed by a computing device including at least one processor, A step of analyzing biometric data based on a learned neural network model to determine the user's status; A step of obtaining context data that assists in explaining the user's status from a pre-built database based on the user's status and feature information corresponding to the biometric data; and A step of generating a prompt based on the context data and providing the prompt to a language model to generate diagnostic information about the user's status; method.

2. In paragraph 1, A step of inputting the biometric data into a machine learning model and extracting and obtaining feature information corresponding to the biometric data; method.

3. In paragraph 1, The status of the above user is: Including the possibility of heart disease, The steps to determine the status of the above user are: A step of inputting the biometric data into the learned neural network model to obtain a score corresponding to the possibility of the user having a heart disease, and determining the user's condition based on the obtained score; method.

4. In paragraph 1, The above-mentioned database is, Includes a plurality of first text information regarding each of a plurality of feature information extractable from the biometric data and a plurality of second text information generated by an expert regarding a plurality of states that can be identified through the biometric data, The first text information and the second text information are each generated based on text written by a domain expert. method.

5. In paragraph 4, The step of obtaining context data to assist in explaining the user's status from the above-mentioned pre-built database is as follows: A step of selecting third text information corresponding to the feature information from among the plurality of first text information based on the similarity between the text information corresponding to the feature information and each first text information; A step of selecting fourth text information corresponding to the user's status from among the plurality of second text information based on the similarity between the text information corresponding to the user's status and each second text information; and A step of obtaining the context data based on the third text information and the fourth text information; comprising; method.

6. In paragraph 5, The third step of selecting text information is: A step of identifying the similarity between the text information corresponding to the feature information and each of the first text information based on the distance between the first vector obtained by embedding the text information corresponding to the feature information and the plurality of second vectors obtained by embedding each of the first text information; including; method.

7. In paragraph 5, The fourth step of selecting text information is: A step of identifying the similarity between the text information corresponding to the user's state and each of the second text information based on the distance between the third vector obtained by embedding the text information corresponding to the user's state and the plurality of fourth vectors obtained by embedding the respective second text information; including; method.

8. In paragraph 5, The above plurality of first text information, It consists of texts each describing a plurality of characteristic information related to the above biometric data, The above plurality of second text information, Consisting of texts each describing a plurality of states of the user related to the biometric data, method.

9. In paragraph 1, The step of obtaining context data to assist in describing the status of the above user is as follows: A step of identifying first characteristic information related to the user's status among a plurality of characteristic information corresponding to the biometric data, and a step of obtaining context data that assists in explaining the user's status from a pre-built database based on the user's status and the first characteristic information; method.

10. In paragraph 10, The step of generating diagnostic information about the status of the above user is: A step of generating a prompt based on the first characteristic information, the user's status, and the context data; method.

11. In paragraph 1, The above biometric data is, Contains electrocardiogram data, The steps to determine the status of the above user are: A step of obtaining lead-specific feature information of the electrocardiogram data and global feature information of the electrocardiogram data using the above-mentioned pre-learned neural network model; The step of obtaining the above context data is: A step of obtaining context data that assists in explaining the user's status from a pre-built database using at least one of the lead-specific feature information and global feature information as a query; comprising; method.

12. In a computing device that analyzes biometric data to generate user diagnostic information, a processor comprising at least one core; and a memory including program codes executable by the processor; The above processor, A method for identifying a user's status by analyzing biometric data based on a pre-learned neural network model, obtaining context data that assists in explaining the user's status from a pre-built database based on the user's status and the characteristic information of the biometric data corresponding to the biometric data, generating a prompt based on the context data, and providing the prompt to a language model to generate diagnostic information regarding the user's status. Computing device.

13. A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations for analyzing biometric data to generate user diagnostic information. The above actions are, An action to identify the user's condition by analyzing biometric data based on a pre-learned neural network model; An operation of obtaining context data that assists in explaining the user's status from a pre-built database based on the user's status and the characteristic information of the biometric data corresponding to the biometric data; and An operation of generating a prompt based on the context data and providing the prompt to a language model to generate diagnostic information about the user's status; Computer program.

Images