Server for providing model for predicting mortality and long-term occupancy of post-surgery patient and operation method thereof

Abstract

According to one embodiment of the present disclosure, disclosed is a server for providing a model for predicting mortality and long-term occupancy of a post-surgery patient and an operation method thereof. The server comprises at least one processor for executing at least one instruction stored in a memory, wherein the at least one processor can obtain patient-related information including at least one of clinical information, surgery information, or blood test results of a patient, apply the obtained patient-related information to an artificial intelligence learning model for predicting mortality and long-term occupancy of a patient, and generate, using an ensemble learning model, result predicting at least one of mortality or long-term occupancy of the patient.

Description

A server and its operating method that provide a model for predicting whether a patient will die after surgery and whether the patient will remain in the hospital for a long time

[0001] The present invention relates to a server and a method of operating the server, and more particularly, to a server and a method of operating the same that provides a model for predicting whether a patient will die after surgery and whether the patient will remain in the hospital for a long time.

[0002] Technological advancements have led to the development of assessment models to assess patient status. The APACHE II score is typically used to assess the severity of critically ill patients. The APACHE II score has an area under the curve (AUROC) for mortality of approximately 0.7-0.8.

[0003] However, this score does not include much surgical information, and there is no standard cutoff value for mortality. Therefore, it is difficult to obtain clear results when using the APACHE II score to predict mortality. Furthermore, it is difficult to use it to predict not only mortality but also expected length of hospital stay.

[0004] The purpose of the embodiments disclosed in this disclosure is to provide a server and a method of operating the server that develop and provide an artificial intelligence model that predicts treatment outcomes (death or survival) for critically ill patients after surgery using clinical information, surgical information, blood tests, etc. In addition, since the shortage of intensive care unit beds due to long-term hospitalization has always been a major issue in intensive care units, the purpose of the present disclosure is to provide a server and a method of operating the server that develop and provide an artificial intelligence model that predicts the length of stay of a patient for a specific date (e.g., 30 days) or longer or shorter than that.

[0005] The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0006] According to the present disclosure for achieving the above-described technical task, a server providing a model for predicting whether a patient will die after surgery and whether the patient will remain in the hospital for a long time may be provided. The server may include a memory storing at least one instruction. The server may include at least one processor executing at least one instruction stored in the memory. The at least one processor may obtain patient-related information including at least one of the patient's clinical information, surgical information, or blood test results. The at least one processor may apply the obtained patient-related information to an artificial intelligence learning model that predicts whether the patient will die and whether the patient will remain in the hospital for a long time. The at least one processor may generate a result predicting at least one of the patient's death or whether the patient will remain in the hospital for a long time through the artificial intelligence learning model.

[0007] In addition, according to the present disclosure, a method of operating a server that provides a model for predicting whether a patient will die after surgery and whether the patient will be hospitalized long-term may be provided. The method of operating the server may include a step of acquiring patient-related information including at least one of the patient's clinical information, surgical information, or blood test results. The method of operating the server may include a step of applying the acquired patient-related information to an artificial intelligence learning model that predicts whether the patient will die and whether the patient will be hospitalized long-term. The method of operating the server may include a step of generating a result predicting at least one of the patient's death or whether the patient will be hospitalized long-term through the artificial intelligence learning model.

[0008] In addition, a computer program stored in a computer-readable recording medium for implementing the present disclosure may be further provided.

[0009] In addition, a computer-readable recording medium recording a computer program for implementing the present disclosure may be further provided.

[0010] The aforementioned problem-solving method of the present disclosure can improve the accuracy of predicting whether a patient will die after surgery. Furthermore, the aforementioned problem-solving method of the present disclosure can predict whether a patient will require long-term hospitalization after surgery. This allows for an early assessment of the patient's severity after surgery and provides information on whether a patient will require long-term hospitalization, thereby enhancing the efficiency of intensive care unit clinical management.

[0011] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0012] FIG. 1 is a diagram illustrating a server that provides a model for predicting whether a patient will die after surgery and whether the patient will remain in the hospital for a long time according to one embodiment of the present disclosure.

[0013] FIG. 2 is a diagram for explaining the operation of a server according to one embodiment of the present disclosure.

[0014] FIG. 3 is a flowchart illustrating the operation of a server according to one embodiment of the present disclosure.

[0015] FIG. 4 is a flowchart for explaining the learning operation of an artificial intelligence learning model included in a server according to one embodiment of the present disclosure.

[0016] FIG. 5 is a diagram for explaining training data used for learning an artificial intelligence learning model according to one embodiment of the present disclosure.

[0017] FIG. 6a is a diagram for explaining the learning performance of the first ensemble learning model according to one embodiment of the present disclosure.

[0018] FIG. 6b is a diagram for explaining the inference performance of the first ensemble learning model according to one embodiment of the present disclosure.

[0019] FIG. 6c is a diagram illustrating the learning performance of a second ensemble learning model according to one embodiment of the present disclosure.

[0020] FIG. 6d is a diagram for explaining the inference performance of the second ensemble learning model according to one embodiment of the present disclosure.

[0021] FIGS. 7A to 7E are diagrams for explaining features that affect the performance of the first ensemble learning model according to one embodiment of the present disclosure.

[0022] FIGS. 8A to 8E are diagrams for explaining features that affect the performance of a second ensemble learning model according to one embodiment of the present disclosure.

[0023] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and any content that is common in the technical field to which this disclosure pertains or that overlaps between embodiments is omitted. The terms "part, module, element, block" used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple "parts, modules, elements, blocks" may be implemented as a single component, or a single "part, module, element, block" may include multiple components.

[0024] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0025] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0026] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0027] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0028] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0029] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0030] The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

[0031] FIG. 1 is a diagram illustrating a server that provides a model for predicting whether a patient will die after surgery and whether the patient will remain in the hospital for a long time according to one embodiment of the present disclosure.

[0032] In one embodiment, the server (100) illustrated in FIG. 1 may obtain patient-related information about a patient and, using the obtained patient-related information, generate a result predicting at least one of whether the patient will die or whether the patient will have a long-term stay in the hospital in relation to the time of the patient's hospitalization.

[0033] In one embodiment, the patient-related information acquired by the server (100) may be information about a patient who underwent surgery at a hospital. The predicted result based on the patient-related information acquired by the server (100) may be at least one of the following: whether the patient will die after the surgery or whether the patient will remain in the hospital for a long period after the surgery.

[0034] In one embodiment, the server (100) can, of course, generate results predicting both whether the patient will die after surgery and whether the patient will remain in the hospital for a long time based on the patient-related information acquired. Furthermore, the server (100) can also generate and provide information on the condition of the patient who underwent surgery (e.g., if the patient is predicted to die after surgery and will not remain in the hospital for a long time) based on the results of the post-operative mortality and long-term hospitalization.

[0035] At this time, the server (100) may use an ensemble learning model of artificial intelligence learned based on patient-related information to predict whether the patient will die after surgery and whether the patient will stay in the hospital for a long time. At this time, the ensemble learning model may be a model created by combining a plurality of deep learning models (e.g., XGBoost (eXtreme Gradient Boosting), CatBoost, LGBM (Light Gradient Boosting Model), Random Forest, and MLP (Multi-Layer Perceptron), etc.) learned to predict whether the patient will die after surgery and whether the patient will stay in the hospital for a long time based on patient-related information. Hereinafter, the ensemble learning model used in the server (100) of the present disclosure will be described later with reference to FIGS. 4 to 8e.

[0036] In one embodiment, the server (100) may include a memory (110), at least one processor (130), and a communication unit (120). The memory (110), at least one processor (130), and the communication unit (120) may each be electrically and / or physically connected to each other.

[0037] However, in some embodiments, the server (100) may include fewer or more components than the components illustrated in FIG. 1.

[0038] In one embodiment, the memory (110) can store data supporting various functions of the server (100), a program for the operation of at least one processor (130), can store input / output data (e.g., sentences, music files, still images, videos, etc.), can store a plurality of application programs (or applications) running on the device, at least one data for the operation of the device, and at least one instruction. At least some of these application programs can be downloaded from an external server via wireless communication.

[0039] The memory (110) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory may be a database that is separate from the server (100) but is connected by wire or wirelessly.

[0040] In one embodiment, the memory (110) may store an ensemble learning model trained to predict whether a patient will die after surgery and whether the patient will remain in the hospital for a long period of time. In one embodiment, the memory (110) may store patient-related information regarding a patient for whom the patient's death and long-term hospital stay are to be predicted after surgery.

[0041] In one embodiment, at least one processor (130) may control the overall operations of the server (100). In one embodiment, at least one processor (130) may control the operations of the server (100) by executing at least one instruction stored in the memory (110).

[0042] In one embodiment, at least one processor (130) may execute at least one instruction stored in the memory (110) to perform an operation of predicting at least one of whether the patient will die after surgery or whether the patient will remain in the hospital for a long time based on the acquired patient-related information.

[0043] In one embodiment, at least one processor (130) may be a general-purpose processor, such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-only processor, such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor, such as an NPU. The at least one processor controls processing of patient-related information based on instructions stored in memory or an artificial intelligence model. Alternatively, if the at least one processor is an artificial intelligence-only processor, the artificial intelligence-only processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0044] In one embodiment, at least one processor (130) executes at least one command stored in the memory (110) to predict whether the patient will die after surgery and whether the patient will remain in the hospital for a long time based on the acquired patient-related information, and generates information or notifications to be provided to an external electronic device that provided the patient-related information to the server (100) or to other users of the server (100) based on the predicted result.

[0045] In one embodiment, the communication unit (120) may include one or more components that enable communication with another external server or external electronic device. For example, the communication unit (120) may include at least one of a wired communication module, a wireless communication module, or a short-range communication module.

[0046] In one embodiment, the wired communication module may include various wired communication modules such as a Local Area Network (LAN) module, a Wide Area Network (WAN) module, or a Value Added Network (VAN) module, as well as various cable communication modules such as a Universal Serial Bus (USB), a High Definition Multimedia Interface (HDMI), a Digital Visual Interface (DVI), RS-232 (recommended standard232), power line communication, or plain old telephone service (POTS).

[0047] In one embodiment, the wireless communication module may include a wireless communication module that supports various wireless communication methods such as GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), UMTS (universal mobile telecommunications system), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, and 6G, in addition to a WiFi module and a Wireless Broadband module.

[0048] In one embodiment, the short-range communication module is for short-range communication, and can support short-range communication using at least one of Bluetooth™ RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

[0049] In one embodiment, at least one processor (130) may obtain patient-related information regarding a patient whose death or long-term stay after surgery is to be predicted from an external server or external electronic device via the communication unit (120). Furthermore, at least one processor (130) may provide at least one result of the patient's death or long-term stay predicted through an ensemble learning model to the external server or external electronic device via the communication unit (120).

[0050] In one embodiment, at least one processor (130) may obtain a pre-trained ensemble learning model to predict whether the patient will die or remain in the hospital for a long time after surgery by using patient-related information from an external server or an external electronic device through the communication unit (120).

[0051] FIG. 2 is a diagram illustrating the operation of a server according to one embodiment of the present disclosure. Hereinafter, the same components as those described in FIG. 1 are assigned the same drawing reference numerals, and redundant descriptions are omitted.

[0052] Referring to FIGS. 1 and 2, in one embodiment, the server (100) may obtain patient-related information from an external electronic device (200) via the communication unit (120). At this time, the server (100) may provide the external electronic device (200) with a result predicting at least one of whether the patient is dead or whether the patient will remain in the hospital for a long time via the communication unit (120).

[0053] In one embodiment, a user (300) may provide patient-related information about a patient for which at least one of death or long-term hospitalization is desired to be determined to the server (100) via an external electronic device (200). The user (300) may obtain a result regarding at least one of the predicted death or long-term hospitalization of the patient from the server (100) via the external electronic device (200).

[0054] In one embodiment, the server (100) is a server that processes information by communicating with external electronic devices, and may include an application server, a computing server, a database server, a file server, a proxy server, and a web server. In one embodiment, the external electronic device (200) may include a notebook, desktop, laptop, tablet PC, slate PC, portable terminal, etc. equipped with a web browser. However, the present disclosure is not limited to any one, and the server (100) and the external electronic device (200) may include various servers and electronic devices capable of performing the operations described in FIGS. 1 and 2.

[0055] Additionally, in one embodiment, the user (300) may provide patient-related information to an electronic device including the server (100), rather than an external electronic device (200). The server (100) may obtain patient-related information from the user (300) through a user interface or an input / output interface. The server (100) may also provide the predicted results to the user (300) through the electronic device including the server (100).

[0056] FIG. 3 is a flowchart illustrating the operation of a server according to one embodiment of the present disclosure.

[0057] Referring to FIGS. 1 and 3, in one embodiment, the operating method of the server (100) may include a step (S100) of acquiring patient-related information including at least one of patient clinical information, surgical information, or blood test results.

[0058] In one embodiment, in step S100 of acquiring patient-related information, at least one processor (130) may acquire patient-related information. The patient's clinical information may include information acquired when the patient receives treatment or diagnosis at a hospital prior to surgery, information obtained through a preoperative questionnaire, whether the patient was hospitalized prior to surgery, and the duration of hospitalization.

[0059] In one embodiment, the patient's surgery information may include information about at least one of whether the surgery performed on the patient is a regular surgery (e.g., a surgery scheduled in advance through a medical practitioner) or an emergency surgery (e.g., a surgery not scheduled in advance or a surgery scheduled and performed within a specific date), details about the patient's surgery (e.g., the subject of the surgery, the time of the surgery, etc.), or the department in which the patient's surgery was performed.

[0060] In one embodiment, if the surgery was an emergency, the patient's health may deteriorate significantly compared to a regular surgery, as it was not scheduled. In another embodiment, if the department where the surgery was performed has a history of patients with a high mortality rate, the history may be used as a basis for predicting the patient's death compared to departments with a history of patients with a high mortality rate.

[0061] In one embodiment, when it is desired to predict whether a patient will die using the server (100), the patient-related information may include information such as the date of hospitalization of the patient after surgery.

[0062] In one embodiment, the operating method of the server (100) may include a step (S200) of applying the acquired patient-related information to an ensemble learning model that predicts whether the patient will die and whether the patient will remain in the hospital for a long time.

[0063] In one embodiment, the ensemble learning model may be a pre-trained model that predicts whether a patient will die and whether the patient will remain in the hospital for a long period of time based on training patient-related information including at least one of the patient's clinical information, surgical information, or blood test results. In one embodiment, the ensemble learning model may be a pre-trained model that predicts whether a patient will die and whether the patient will remain in the hospital for a long period of time based on training patient-related information including at least one of the patient's clinical information, surgical information, blood test results, whether the patient will die, the patient's post-operative hospitalization period, or the patient's hospitalization period.

[0064] In one embodiment, in step S200 of predicting whether a patient is dead and whether the patient will remain in the hospital for a long time, at least one processor (130) may apply the acquired patient-related information to an ensemble learning model. At least one processor (130) may input the acquired patient-related information into the ensemble learning model.

[0065] In one embodiment, the ensemble learning model, which is an artificial intelligence learning model, may include a first ensemble learning model and a second ensemble learning model. The first ensemble learning model may be a model trained to predict whether a patient will die after surgery based on acquired patient-related information. The second ensemble learning model may be a model trained to predict whether a patient will remain hospitalized for a long period of time using acquired patient-related information.

[0066] In one embodiment, at least one processor (130) can use at least one of the first ensemble learning model or the second ensemble learning model to determine at least one of whether the patient is dead or whether the patient is expected to remain in the hospital for a long time.

[0067] In one embodiment, the operating method of the server (100) may include a step (S300) of predicting at least one of whether the patient is dead or whether the patient is in a long-term hospital stay through an ensemble learning model.

[0068] In one embodiment, in step S300 of predicting at least one of whether a patient will die or whether the patient will be hospitalized for a long time, at least one processor (130) may use an ensemble learning model to predict at least one of whether the patient will die or whether the patient will be hospitalized for a long time. In one embodiment, the ensemble learning model may be trained to provide a probability of whether the patient will die. In one embodiment, the ensemble learning model may provide a higher probability as the predicted likelihood of the patient's death increases.

[0069] However, the present disclosure is not limited thereto, and the ensemble learning model may be a model trained to predict two outcomes: whether the patient will die or not. At least one processor (130) can use the ensemble learning model to predict whether the patient will die or not. Through this, a clear prediction result can be provided to the user of the server (100).

[0070] In one embodiment, the ensemble learning model may be trained to determine whether a patient will be admitted to the hospital long-term based on the date the patient was admitted to the hospital ward after surgery. In one embodiment, the ensemble learning model may be trained to predict whether a patient will be admitted to the hospital long-term based on 30 days from the date the patient was admitted to the hospital ward after surgery. In one embodiment, if the patient is predicted to be discharged from the hospital ward within 30 days from the date the patient was admitted to the hospital ward after surgery, the ensemble learning model may be trained to predict that the patient will not be admitted to the hospital long-term. In one embodiment, if the patient is predicted to be discharged from the hospital ward within 30 days or more from the date the patient was admitted to the hospital ward after surgery, the ensemble learning model may be trained to predict that the patient will be admitted to the hospital long-term.

[0071] At this time, the date the patient begins hospitalization after surgery may be determined as either the date of surgery or the day after surgery, depending on the type of surgery included in the patient-related information. However, the disclosure is not limited to either one.

[0072] Furthermore, the 30-day period used to determine long-term stay is only an example and is not limited to any specific date. The date used to determine long-term stay may change based on factors such as the type of surgery, department, age, and circumstances of the hospital performing the surgery, as well as pre-entered settings.

[0073] In one embodiment, the operating method of the server (100) may include a step of transmitting a result of predicting at least one of whether the patient is dead or whether the patient is in a long-term condition to an external electronic device (200, see FIG. 2) via a communication unit (120, see FIG. 1). At least one processor (130) may transmit the predicted result in the step (S300) of predicting at least one of whether the patient is dead or whether the patient is in a long-term condition to the external electronic device (200).

[0074] At this time, information transmitted by at least one processor (130) to an external electronic device (200) may vary depending on the predicted result in the step (S300) of predicting at least one of whether the patient is dead or whether the patient is in a long-term condition.

[0075] In one embodiment, if the server (100) predicts both whether the patient will die and whether the patient will remain in the hospital for a long time, the predicted result may be that the patient will not remain in the hospital for a long time and will die. In this case, the patient is predicted to die without being hospitalized for a long time after surgery, which may indicate that the patient's health condition or post-operative prognosis is not good. In this case, at least one processor (130) may provide a notification to an external electronic device (200) that includes information that the patient scheduled for surgery is a critically ill patient. In addition, at least one processor (130) may provide a notification to an external electronic device (200) that includes information that intensive care or intensive observation is required after surgery on the patient.

[0076] In one embodiment, for convenience of explanation, at least one processor (130) is disclosed to only transmit information to an external electronic device (200), but it is of course possible to transmit information to an external server or directly through an electronic device including a server (100).

[0077] FIG. 4 is a flowchart illustrating the learning operation of an artificial intelligence learning model included in a server according to one embodiment of the present disclosure. FIG. 5 is a diagram illustrating training data used for learning an artificial intelligence learning model according to one embodiment of the present disclosure.

[0078] In one embodiment, referring to FIGS. 1 and 4, the artificial intelligence learning model included in the memory (110) may be a pre-learned model to predict whether a patient will die after surgery and whether the patient will remain in the hospital for a long time based on training patient-related information.

[0079] In one embodiment, the server (100) may be provided with a pre-trained artificial intelligence learning model. However, the present disclosure is not limited thereto, and at least one processor (130) may train the artificial intelligence learning model to predict whether a patient will die after surgery and whether the patient will remain hospitalized long-term based on training patient-related information. This training may be performed within the server (100) itself, where the prediction of patient death and long-term hospitalization according to the present disclosure is performed, or may be performed through a separate server and / or system. Hereinafter, for convenience of explanation, it will be described that at least one processor (130) trains an ensemble learning model.

[0080] In one embodiment, an artificial intelligence learning model may include an artificial intelligence model. The artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the computational results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated during the learning process so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized. The artificial neural network may include a deep neural network (DNN), for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a transformer, deep Q-networks, or a boosting algorithm, but is not limited to the examples described above.

[0081] Additionally, the AI ​​learning model may include an ensemble learning model. Ensemble learning models may include XGBoost (eXtreme Gradient Boosting), CatBoost, LGBM (Light Gradient Boosting Model), Random Forest, and MLP (Multi-Layer Perceptron). For convenience of explanation, the learning method of the AI ​​learning model will be described as the learning method of the ensemble learning model.

[0082] Referring to FIGS. 4 and 5, in one embodiment, a learning method of an ensemble learning model may include a step of acquiring patient-related information for learning as training data.

[0083] In one embodiment, the training patient-related information may include at least one of clinical information of the training patient, surgical information of the training patient, death of the training patient, length of stay of the training patient, or blood test of the training patient.

[0084] In one embodiment, a learning method of an ensemble learning model may include a step of dividing learning patient-related information (500) into a learning set (400) and a test set (410). At least one processor (130) may divide learning patient-related information (500) into a learning set (400) and a test set (410).

[0085] In one embodiment, the test set (410) may be data used in the process of determining whether the ensemble learning model is overfitted by the training set (400).

[0086] In one embodiment, the training set (400) may be data used in the process of training an ensemble learning model. In one embodiment, the training method of the ensemble learning model may include a step (S10) of dividing the training set (400) into a training set (401) and a validation set (402). In one embodiment, at least one processor (130) may divide 80% of the training set (400) into a training set (401) and the remaining 20% ​​into a validation set (402). In one embodiment, the operation of dividing the training patient-related information into the training set (401) and the validation set (402) may be performed using a stratified k-fold validation technique. Referring to FIG. 5, the training set (401) may refer to a plurality of data indicated by "V" among the training set (400). The validation set (402) may refer to a plurality of data marked as “T” among the training set (400).

[0087] In one embodiment, a method for learning an ensemble learning model may include a step (S20) of learning an ensemble learning model to predict whether a patient has died and whether the patient will be hospitalized long-term using a training set (401).

[0088] In one embodiment, the training set (401) may be a set used to update multiple weights included in an ensemble learning model among patient-related information for learning. At least one processor (130) may use the training set (401) to update multiple weights included in the ensemble learning model, thereby training the ensemble learning model to predict whether a patient will die after surgery and whether the patient will remain in the hospital for a long time based on the patient-related information.

[0089] In one embodiment, the method for training an ensemble learning model may include a step (S30) of externally verifying the performance of the trained model using a verification set (402). In one embodiment, the verification set (402) may be a set used to confirm whether training of the ensemble learning model has been completed using patient-related information for training. In one embodiment, at least one processor (130) may verify whether training of the ensemble learning model has an accuracy higher than a certain target value based on the result values ​​obtained using the ensemble learning model based on the verification set (402).

[0090] In one embodiment, the ensemble learning model may be a model learned through steps S10, S20, and S30.

[0091] In one embodiment, the ensemble learning model is a model in which learning is performed through an operation of dividing patient-related information for learning into a training set (401) and a verification set (402) and then learning, and the performance of the ensemble learning model, which has been primarily learned through the training set (401), is externally verified through the verification set (402) and then learning is completed, thereby increasing the reliability of the performance of the ensemble learning model.

[0092] In one embodiment, a method for learning an ensemble learning model may include a step of testing the performance of the ensemble learning model using a test set (410), for which learning has been completed through a training set (401) and a verification set (402). At least one processor (130) may perform an inference operation using the test set (410), which is data that has not been used in the learning process, to test the performance of the ensemble learning model, for which learning has been completed through the learning set (400).

[0093] In one embodiment, when 80% of the learning patient-related information (500) is divided into a learning set (400) and the remaining 20% ​​is divided into a test set (410), the training set (401) may be divided into 80% of the learning set (400), i.e., 64% of the learning patient-related information (500), and the verification set (402) may be divided into 20% of the learning set (400), i.e., 16% of the learning patient-related information.

[0094] Through this learning process, XGBoost (eXtreme Gradient Boosting), CatBoost, LGBM (Light Gradient Boosting Model), Random Forest, and MLP (Multi-Layer Perceptron), which are included in the ensemble learning model, can each be learned.

[0095] In one embodiment, the first ensemble learning model may be trained to predict whether a patient is dead using the training set (400). In one embodiment, the first ensemble learning model may be trained to predict whether a patient is dead using first patient-related information including clinical information included in the training set (400), surgical information of the training patient, whether the training patient is dead, and blood tests of the training patient.

[0096] In one embodiment, the second ensemble learning model may be trained to predict whether a patient will be hospitalized long-term by using the training set (400). In one embodiment, the second ensemble learning model may be trained to predict whether a patient will be hospitalized long-term by using second patient-related information including clinical information included in the training set (400), surgical information of the training patient, hospitalization period of the training patient, and blood tests of the training patient.

[0097] In one embodiment, for an ensemble learning model that has completed learning and performance evaluation using patient-related information (500) for learning, inference can be performed using patient-related information (510). However, the present disclosure is not limited thereto, and the patient-related information (510) used to perform inference may be obtained from a hospital other than the specific hospital from which the patient-related information (500) for learning was obtained, and may be information for determining whether the performance of an ensemble learning model trained using information from a specific hospital is maintained even when information from a different hospital is used.

[0098] FIG. 6a is a diagram illustrating the learning performance of a first ensemble learning model according to one embodiment of the present disclosure. FIG. 6b is a diagram illustrating the inference performance of a first ensemble learning model according to one embodiment of the present disclosure.

[0099] Referring to FIG. 6A, in one embodiment, FIG. 6A illustrates an evaluation table (600) that evaluates the learned performance of a first ensemble learning model trained to predict whether a patient will die using patient-related information. In one embodiment, the evaluation table (600) illustrated in FIG. 6A may be a table obtained by evaluating predicted results using patient-related information for learning after completing training of the first ensemble learning model. In one embodiment, the evaluation table (600) illustrated in FIG. 6A may be a table obtained by evaluating predicted results using a test set (410) after completing training of the first ensemble learning model.

[0100] In one embodiment, the evaluation table (600) shows the learned performance for each of the learned XGBoost, CatBoost, LGBM, Random Forest, and MLP models. In one embodiment, the evaluation table (620) shows the AUROC (Area Under the Receiver Operating Characteristic Curve), Accuracy, Precision, Recall, and F1-Score items for each model. In one embodiment, the cut-off value in XGBoost may be set to 0.2. The cut-off value in CatBoost may be set to 0.22. The cut-off value in LGBM may be set to 0.2. The cut-off value in Random Forest may be set to 0.25. The cut-off value in the MLP model may be set to 0.3. In this case, the cut-off value may be a value that serves as a criterion for distinguishing whether a patient dies or does not die when determining whether the patient dies.

[0101] In one embodiment, referring to FIGS. 4, 5, and 6a, each of the XGBoost, CatBoost, and LGBM models may be learned models that replace missing values ​​included in the learning patient-related information present during the learning process with blanks. In contrast, the Random Forest and MLP models may be learned models that replace missing values ​​included in the learning patient-related information present during the learning process with medians. In this case, the median may mean the median of values ​​adjacent to the missing value or the median of the entire data.

[0102] In one embodiment, the first ensemble learning model is illustrated in the evaluation table (600) as an Ensemble Model. The first ensemble learning model may be an ensemble model that is weighted with 0.15 for XGBoost, 0.15 for CatBoost, 0.6 for LGBM, 0.05 for Random Forest, and 0.05 for MLP. However, the present disclosure is not limited thereto, and the first ensemble learning model may be generated by differently weighting each of the XGBoost, CatBoost, LGBM, Random Forest, and MLP models to improve inference performance.

[0103] In one embodiment, the evaluation table (610) illustrated in FIG. 6B may be a table obtained by evaluating predicted results using patient-related information input for actual inference, rather than training data, to describe the inference performance of the first ensemble learning model. In one embodiment, if the patient-related information for training is patient-related information acquired from a specific hospital, the patient-related information input for actual inference may be patient-related information acquired from a different hospital.

[0104] Referring to the evaluation table (610) illustrated in FIG. 6b, the evaluation table (610) illustrated in FIG. 6b is evaluated using the results of whether or not a patient died, predicted using data input for actual inference, not the learning data used in the first ensemble learning model. However, it is illustrated that a certain score or higher was obtained in each evaluation item.

[0105] Through this, not only the learning performance of the first ensemble learning model learned using patient-related information for learning, but also the accuracy of the first ensemble learning model in predicting whether a patient will die can be high when making inferences using the first ensemble learning model.

[0106] FIG. 6c is a diagram illustrating the learning performance of a second ensemble learning model according to one embodiment of the present disclosure. FIG. 6d is a diagram illustrating the inference performance of a second ensemble learning model according to one embodiment of the present disclosure. Hereinafter, any duplicate description of the same configuration as that described in FIGS. 6a and 6b will be omitted.

[0107] Referring to FIG. 6c, in one embodiment, FIG. 6c illustrates an evaluation table (620) that evaluates the learned performance of a second ensemble learning model learned to predict whether a patient will be hospitalized long-term using patient-related information.

[0108] In one embodiment, the second ensemble learning model is illustrated in the evaluation table (620) as an Ensemble Model. The second ensemble learning model may be an ensemble model that is weighted with 0.1 for XGBoost, 0.2 for CatBoost, 0.05 for LGBM, 0.05 for Random Forest, and 0.6 for MLP. However, the present disclosure is not limited thereto, and the second ensemble learning model may be created by differently weighting each of the XGBoost, CatBoost, LGBM, Random Forest, and MLP models to improve inference performance.

[0109] In one embodiment, the cut-off value in XGBoost may be set to 0.08. The cut-off value in CatBoost may be set to 0.07. The cut-off value in LGBM may be set to 0.06. The cut-off value in Random Forest may be set to 0.08. The cut-off value in the MLP model may be set to 0.17. In this case, the cut-off value may be a value that serves as a criterion for distinguishing whether a patient is predicted to be in a long-term hospital stay or not.

[0110] In one embodiment, the evaluation table (620) illustrated in FIG. 6c may be a table obtained by evaluating predicted results using patient-related information input for actual inference, rather than learning data, to explain the inference performance of the second ensemble learning model.

[0111] Referring to the evaluation table (630) illustrated in FIG. 6d, the evaluation table (630) illustrated in FIG. 6d is evaluated using the results of whether a patient will stay in the hospital for a long time, predicted using data input for actual inference, not learning data used in the second ensemble learning model. However, it is illustrated that a certain score or higher was obtained in each evaluation item.

[0112] Through this, not only the learning performance of the second ensemble learning model learned using patient-related information for learning can be improved, but also the accuracy of the second ensemble learning model's prediction of whether the patient will remain hospitalized for a long time can be improved when making inferences using the second ensemble learning model.

[0113] In one embodiment, the ensemble learning model of the server (100, see FIG. 1) can predict whether a patient has died and whether the patient will remain in the hospital for a long time using the first ensemble learning model and the second ensemble learning model. The server (100) can predict the outcome using at least one of the first ensemble learning model and the second ensemble learning model, depending on the acquired patient-related information and the information to be predicted using the acquired patient-related information.

[0114] FIGS. 7A to 7E are diagrams for explaining features that affect the performance of the first ensemble learning model according to one embodiment of the present disclosure.

[0115] Referring to FIGS. 7A to 7E, in one embodiment, FIGS. 7A to 7E illustrate graphs analyzing the influence of features affecting the performance of predicting whether a patient will die using patient-related information in each of the XGBoost, CatBoost, LGBM, Random Forest, and MLP models combined in the first ensemble learning model.

[0116] In one embodiment, FIGS. 7A to 7E illustrate graphs analyzing the influence of features affecting the performance of predicting whether a patient will die after surgery for each model using the Shapley Additive exPlanations (SHAP) method.

[0117] In one embodiment, FIGS. 7A to 7E illustrate features affecting the performance of predicting whether a patient will die in each of XGBoost, CatBoost, LGBM, Random Forest, and MLP models based on SHAP values, and the influence of each feature. At this time, among the features illustrated in FIGS. 7A to 7E, the higher the feature is positioned on the y-axis of the graph, the greater the influence it has on the performance of the XGBoost model, and the lower the feature is positioned on the y-axis of the graph, the smaller the influence it has on the performance of the XGBoost model. At this time, the influence on the performance of the model may mean that the higher the influence, the higher the probability that the patient will be predicted to die.

[0118] Additionally, based on the x-axis of the graph, the line with the color displayed for each feature may have a greater impact on the performance of the XGBoost model the redder the color, and a smaller impact on the performance of the XGBoost model the bluer the color.

[0119] In one embodiment, based on “Age” among the characteristics shown in FIG. 7a, the color goes to the right (as age increases) from the origin 0 on the x-axis, which means that the probability that the patient is predicted to die increases, and the color goes to the left (as age decreases), which means that the probability that the patient is predicted to die decreases.

[0120] In one embodiment, FIG. 7a illustrates features and the influence of each feature on the performance of predicting whether a patient died in an XGBoost model based on SHAP values.

[0121] In one embodiment, FIG. 7b illustrates features and the influence of each feature on the performance of predicting patient death in a CatBoost model based on SHAP values.

[0122] In one embodiment, FIG. 7c illustrates features and the influence of each feature on the performance of predicting patient death in LGBM based on SHAP values.

[0123] In one embodiment, FIG. 7d illustrates features and the influence of each feature on the performance of predicting whether a patient died in a Random Forest model based on SHAP values.

[0124] In one embodiment, FIG. 7e illustrates features and the influence of each feature on the performance of predicting whether a patient will die in an MLP model based on SHAP values.

[0125] Figures 8a to 8e are diagrams illustrating features that affect the performance of a second ensemble learning model according to one embodiment of the present disclosure. Hereinafter, redundant descriptions of components identical to those described in Figures 7a to 7e will be omitted.

[0126] Referring to FIGS. 8A to 8E, in one embodiment, FIGS. 8A to 8E illustrate graphs analyzing the influence of features affecting the performance of predicting whether a patient will stay in the hospital for a long time using patient-related information in each of the XGBoost, CatBoost, LGBM, Random Forest, and MLP models combined in the second ensemble learning model.

[0127] In one embodiment, FIGS. 8A to 8E illustrate graphs analyzing the influence of features affecting the performance of predicting whether a patient will remain hospitalized long-term after surgery for each model using the Shapley Additive exPlanations (SHAP) method.

[0128] In one embodiment, FIG. 8a illustrates features and the influence of each feature on the performance of predicting whether a patient will stay in the long-term in an XGBoost model based on SHAP values.

[0129] In one embodiment, FIG. 8b illustrates features and the impact of each feature on the performance of predicting long-term hospital stay of a patient in a CatBoost model based on SHAP values.

[0130] In one embodiment, FIG. 8c illustrates features and the impact of each feature on the performance of predicting long-term hospitalization of a patient in LGBM based on SHAP values.

[0131] In one embodiment, FIG. 8d illustrates features and the influence of each feature on the performance of predicting whether a patient will stay in the hospital long-term in a Random Forest model based on SHAP values.

[0132] In one embodiment, FIG. 8e illustrates features and the influence of each feature on the performance of predicting long-term hospitalization of a patient in an MLP model based on SHAP values.

[0133] Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium storing computer-executable instructions. The instructions may be stored in the form of program code, and when executed by a processor, may generate program modules to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

[0134] Computer-readable storage media include all types of storage media that store instructions that can be deciphered by a computer. Examples include read-only memory (ROM), random access memory (RAM), magnetic tape, magnetic disks, flash memory, and optical data storage devices.

[0135] The disclosed embodiments have been described with reference to the attached drawings as described above. Those skilled in the art will understand that the present disclosure can be implemented in forms other than the disclosed embodiments without altering the technical spirit or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

Claims

1. In a server that provides a model for predicting whether a patient will die after surgery and whether the patient will remain in the hospital for a long time, a memory that stores at least one instruction; and At least one processor configured to execute at least one instruction stored in the memory, At least one processor, Obtaining patient-related information including at least one of the patient's clinical information, surgical information, or blood test results, Applying the patient-related information obtained above to an artificial intelligence learning model that predicts whether the patient has died and whether the patient will be hospitalized for a long time, A server that generates a result predicting at least one of whether the patient will die or whether the patient will be hospitalized for a long time through the artificial intelligence learning model.

2. In paragraph 1, The above surgical information includes information on whether the surgery performed on the patient was a regular surgery or an emergency surgery, the surgical time taken for the surgery, or at least one of the departments in which the surgery was performed. The above death and long-term stay are predicted results for the patient on whom the above surgery was performed. The above artificial intelligence learning model is a server model that is trained to predict whether the patient will be hospitalized for a long period of time based on whether the patient is expected to be hospitalized for more than 30 days from the date of hospitalization after the surgery, based on the patient-related information.

3. In paragraph 1, The above server further includes a communication unit, The at least one processor transmits the result of predicting at least one of the patient's death or the patient's long-term presence to an external electronic device through the communication unit, A server in which at least one processor provides a notification to an external device through the communication unit, including information that the patient is a critically ill patient, when the patient is predicted to die without long-term hospitalization.

4. In paragraph 1, The above artificial intelligence learning model is, A first ensemble learning model trained to predict whether the patient will die after the surgery using the patient-related information obtained above; and A server including a second ensemble learning model trained to predict whether the patient will be hospitalized for a long period of time using the patient-related information obtained above.

5. In paragraph 4, The above artificial intelligence learning model is, Divide patient-related information for learning into a training set and a validation set, Using the training set, it is learned to predict whether the patient died and whether the patient was hospitalized for a long time. This is a model that externally verifies the performance of the learned model using the above verification set. The above patient-related information for learning purposes is: A server containing at least one of clinical information of a training patient, surgical information of a training patient, death of a training patient, length of stay of a training patient, or blood tests of a training patient.

6. In paragraph 5, The above artificial intelligence learning model is, It may be a model created by ensembling XGBoost (eXtreme Gradient Boosting), CatBoost, LGBM (Light Gradient Boosting Model), Random Forest, and MLP (Multi-Layer Perceptron) learned using the above learning patient-related information, The above XGBoost (eXtreme Gradient Boosting), the above CatBoost and the above LGBM (Light Gradient Boosting Model) are models learned by replacing missing values ​​included in the learning patient-related information with blanks during learning. The above Random Forest and MLP are models learned by replacing the missing values ​​with the median during learning.

7. In paragraph 6, The first ensemble learning model is trained using first patient-related information including the clinical information, the surgical information of the training patient, the death status of the training patient, and the blood test of the training patient. The second ensemble learning model is trained using second patient-related information including the clinical information, the surgical information of the training patient, the hospital stay period of the training patient, and the blood test of the training patient. The first and second ensemble learning models are ensemble models that are weighted by each of the XGBoost (eXtreme Gradient Boosting), the CatBoost, the LGBM (Light Gradient Boosting Model), the Random Forest, and the MLP (Multi-Layer Perceptron).

8. In the operating method of a server that provides a model for predicting whether a patient will die after surgery and whether the patient will remain in the hospital for a long time, A step of obtaining patient-related information including at least one of clinical information, surgical information, or blood test results of the patient; A step of applying the patient-related information obtained above to an artificial intelligence learning model that predicts whether the patient has died and whether the patient will be hospitalized for a long time; A method of operating a server, comprising a step of generating a result predicting at least one of whether the patient is dead or whether the patient is in the long-term hospital through the artificial intelligence learning model.

9. In paragraph 8, The above surgical information includes information on whether the surgery performed on the patient was a regular surgery or an emergency surgery, the surgical time taken for the surgery, or at least one of the departments in which the surgery was performed. The above death and long-term stay are predicted results for the patient on whom the above surgery was performed. The above artificial intelligence learning model is a model learned to predict whether the patient will be hospitalized for a long period of time based on whether the patient is expected to be hospitalized for more than 30 days from the date of hospitalization after the surgery, based on the patient-related information, and the operating method of the server.

10. In paragraph 8, Further comprising a step of transmitting the result of predicting at least one of the death of the patient or the long-term presence of the patient to an external electronic device through the communication unit of the server, In the step of transmitting the predicted result to the external electronic device, A method of operating a server, wherein a notification including information that the patient is a critically ill patient is provided to the external device through the communication unit, as the patient is not expected to remain in the room for a long period of time and is expected to die.

11. In paragraph 8, The above artificial intelligence learning model is, A first ensemble learning model trained to predict whether the patient will die after the surgery using the patient-related information obtained above; and A method of operating a server including a second ensemble learning model learned to predict whether the patient will be hospitalized for a long period of time using the patient-related information obtained above.

12. In paragraph 11, The above artificial intelligence learning model is, A step of dividing patient-related information for learning into a training set and a validation set; A step of learning to predict whether the patient is dead and whether the patient is in the long-term hospital using the training set; and A model learned through a step of externally verifying the performance of the learned model using the above verification set, The above patient-related information for learning purposes is: A method of operating a server that includes at least one of clinical information of a training patient, surgical information of a training patient, death of a training patient, hospitalization period of a training patient, or blood test of a training patient.

13. In paragraph 12, The above artificial intelligence learning model is, It may be a model created by ensembling XGBoost (eXtreme Gradient Boosting), CatBoost, LGBM (Light Gradient Boosting Model), Random Forest, and MLP (Multi-Layer Perceptron) learned using the above learning patient-related information, The above XGBoost (eXtreme Gradient Boosting), the above CatBoost and the above LGBM (Light Gradient Boosting Model) are models learned by replacing missing values ​​included in the learning patient-related information with blanks during learning. The above Random Forest and the above MLP are models learned by replacing the missing values ​​with the median during learning, and are a method of operating a server.

14. In paragraph 13, The first ensemble learning model is trained using first patient-related information including the clinical information, the surgical information of the training patient, the death status of the training patient, and the blood test of the training patient. The second ensemble learning model is trained using second patient-related information including the clinical information, the surgical information of the training patient, the hospital stay period of the training patient, and the blood test of the training patient. The above first and second ensemble learning models are ensemble models in which weights are applied to the XGBoost (eXtreme Gradient Boosting), the CatBoost, the LGBM, the Random Forest, and the MLP (Multi-Layer Perceptron), respectively, and the server operation method.

15. A computer-readable recording medium, coupled with a computer, storing a program for executing the method of any one of claims 8 to 14.

Images