Method for determining whether a drug has been administered and server using same

Abstract

A server for determining whether a medication has been administered is provided, the server including: a transceiver that receives video recorded by the wearable device; a memory that stores a detection model and a confirmation model, the detection model being trained to output whether each of predetermined targets appears in an image; the confirmation model being trained to output whether a medication has been administered, the predetermined targets including objects related to the medication or medication container and poses related to medication administration; and one or more processors configured to detect the predetermined targets by inputting image frames from the video into the detection model, and determine whether the medication has been administered by inputting confirmation model input data generated based on detection results of the detection model into the confirmation model.

Description

[Technical field]

[0001] The present disclosure relates to a method for determining whether a medication has been administered, and a server that uses the method. [Background technology]

[0002] There has always been an interest in medicine and medically based health care. In recent years, this interest has further increased as life expectancy has increased with the development of medical technology. Such health care needs to be performed by users themselves on a regular basis in their daily lives, such as taking prescribed medicines, periodically injecting medicines into the body, and measuring health indicators using medical devices.

[0003] In order to improve the effectiveness of the above-mentioned healthcare, it is necessary to induce or force users to take actions related to healthcare. To this end, a solution is needed to detect the user's actions related to healthcare in daily life and to confirm whether healthcare is being performed well.

[0004] However, in the past, when monitoring a user's actions related to healthcare, the user had to take the time to operate the monitoring device before administering medicine, record the user's actions, etc. If the user forgets to do this, healthcare-related information will not be collected.

[0005] Therefore, considering user convenience and healthcare accuracy, it is necessary to develop a solution for automatically monitoring user actions for healthcare in daily life without user effort.

[0006] The above is intended only to aid in understanding the background of the present disclosure, and is not intended to imply that the present disclosure is within the scope of relevant art known to those skilled in the art. Summary of the Invention [Problem to be solved by the invention]

[0007] The present disclosure aims to provide a method for accurately determining whether a medication has been administered, and a server that uses this method.

[0008] The present disclosure aims to provide a method for determining whether medication has been administered using video captured by a wearable device such as a smart watch, and a server that employs this method.

[0009] The present disclosure aims to provide a method for handling various types of medication administration situations, and a server employing this method.

[0010] The problems to be solved by the present disclosure are not limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by those skilled in the art from the present disclosure and the accompanying drawings. [Means for solving the problem]

[0011] According to one embodiment, a server for determining whether a medication has been administered may include a transceiver for receiving a video recorded by a wearable device, a memory for storing a detection model and a confirmation model, the detection model being trained to output whether each of predefined targets appears in the image, and the confirmation model being trained to output whether a medication has been administered, the predefined targets including objects related to the medication or the medication container, and postures related to medication administration, and one or more processors configured to detect the predefined targets by inputting image frames of the video into the detection model, and determine whether a medication has been administered by inputting confirmation model input data generated based on the detection results of the detection model into the confirmation model. The posture may include at least one of a posture for holding a medication container or a posture for taking a medication.

[0012] According to one embodiment, a method executed by one or more processors for determining whether a medication has been administered may include receiving a video recorded by a wearable device, detecting one or more predefined targets by inputting image frames of the video into a detection model, the detection model being trained to output whether each of the predefined targets appears in the image, the predefined targets including objects related to the medication or the medication container, and postures related to medication administration, and determining whether a medication has been administered by inputting confirmation model input data generated based on detection results of the detection model into the confirmation model. The postures may include at least one of a posture of holding the medication container or a posture of taking the medication.

[0013] According to one embodiment, a method executed by one or more processors for training a detection model and a confirmation model to determine whether a medication has been administered, the method may include the steps of: acquiring detection model training data including training images and detection model labels corresponding to the training images, each of the detection model labels may have a class of medication or medication container and a class of posture associated with medication administration, and the detection model labels may include a first detection model label corresponding to a first one of the training images and a second detection model label corresponding to a second one of the training images; acquiring output data by inputting the training images into the detection model; updating the detection model by comparing the detection model labels to the output data of the detection model; acquiring confirmation model training data including confirmation model input data and confirmation model labels corresponding to the confirmation model input data, the confirmation model labels may include labels for determining whether a medication has been administered, and the confirmation model input data may be generated based on the output data of the detection model; acquiring output data by inputting the confirmation model input data into the confirmation model; and updating the confirmation model by comparing the confirmation model labels to the output data of the confirmation model. The posture may include at least one of a posture for holding a medicine container or a posture for taking a medicine.

[0014] The technical solutions in the present disclosure are not limited to the above, and other technical solutions not mentioned will be clearly understood by those skilled in the art from the present disclosure and the accompanying drawings. Effect of the Invention

[0015] According to an embodiment of the present disclosure, there is provided a method for determining whether a drug has been administered, and a server for using the method, which can improve the accuracy of determining whether a drug has been administered by taking into account both a drug administration-related posture and a drug administration-related object.

[0016] The effects of the present disclosure are not limited to the effects described above, and other effects not described in this specification will be clearly understood by those skilled in the art from this disclosure and the accompanying drawings. [Brief description of the drawings]

[0017] The above and other objects, features and advantages of the present disclosure will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings. [Figure 1] FIG. 1 illustrates a drug administration process according to one embodiment. [Diagram 2] FIG. 1 illustrates a system for determining whether a medication has been administered, according to one embodiment. [Diagram 3] 1 is a block diagram illustrating a wireless communication device according to one embodiment. [Figure 4] FIG. 1 is a block diagram illustrating a wearable device according to one embodiment. [Diagram 5] FIG. 2 is a block diagram illustrating a server according to one embodiment. [Figure 6-7] FIG. 1 illustrates a method for determining whether a medication has been administered using a supervisory model, according to one embodiment. [Figure 8-9] FIG. 1 illustrates a method for determining whether a medication has been administered using detection and confirmation models, according to one embodiment. [Figure 10] FIG. 1 illustrates detection of medication administration related objects and poses using a detection model, according to one embodiment. [Figure 11] FIG. 2 illustrates several medication administration related objects. [Figure 12] FIG. 1 illustrates several drug administration-related positions. [Figure 13] FIG. 13 is a diagram illustrating an example of detection model output data. [Figure 14] FIG. 2 illustrates a detection model including an object detection model and a pose detection model, according to one embodiment. [Figure 15]FIG. 1 illustrates the use of a confirmation model to determine whether a drug has been administered, according to one embodiment. [Figure 16] FIG. 1 illustrates detection model output data and confirmation model input data according to one embodiment. [Figure 17] FIG. 1 illustrates a method for determining whether a medication has been administered using a classification model, according to one embodiment. [Figure 18-20] FIG. 2 illustrates a classification model according to one embodiment. [Figure 21-22] FIG. 13 illustrates using classification and video derived categories to determine whether a medication has been administered according to one embodiment. [Diagram 23] FIG. 1 illustrates a method for training a model, according to one embodiment. [Figure 24-28] FIG. 1 illustrates detection model training data according to one embodiment. [Figure 29] FIG. 1 illustrates a first example embodiment of a method for determining whether a drug has been administered using a supervisory model. [Figure 30-31] FIG. 13 illustrates a second example embodiment of a method for determining whether a drug has been administered using a detection model and a confirmation model. [Diagram 32] FIG. 13 illustrates a third embodiment of an example method for determining whether a medication has been administered using a classification model. [Figure 33-34] 1A-1C are diagrams illustrating drug administration targeted to the main hand and drug administration targeted to the secondary hand, respectively, according to one embodiment. [Diagram 35] A figure showing the results of determining whether or not a drug has been administered, depending on whether or not a distinction is made between drug administration performed primarily by the main hand and drug administration performed primarily by the sub-hand, in one embodiment. [Diagram 36] FIG. 1 illustrates model training using a video with some image frames removed, according to one embodiment. [Figure 37]FIG. 2 illustrates a method for removing some image frames of a training video according to one embodiment. [Figure 38] FIG. 1 illustrates a method for removing some sub-data of validation model input data for training according to one embodiment. [Figure 39] FIG. 1 illustrates a model trained to account for a omitted portion of the drug administration process, according to one embodiment. [Diagram 40] FIG. 1 illustrates consecutive image frame removal and random image frame removal according to one embodiment. [Diagram 41] FIG. 2 illustrates image frame selection according to one embodiment. [Diagram 42] FIG. 1 illustrates determining drug information when swallowing a pill, according to one embodiment. [Diagram 43] FIG. 13 is a diagram illustrating the provision of a second opinion report as an example of an application to remote medical care. [Diagram 44] FIG. 13 illustrates providing prescription reports as another example of a telemedicine application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] The embodiments described in the present disclosure are intended to clearly explain the scope of the present disclosure to those skilled in the art to which the present disclosure pertains, and are not intended to limit the present disclosure. It is to be understood that the present disclosure includes modifications or alterations within the scope of the present disclosure.

[0019] The terms used in this disclosure are selected from currently widely used general terms, taking into consideration the function of the elements in this disclosure, and may have a meaning that changes according to the intention of a person skilled in the art, the practice of the art, or the emergence of new technology. When a specific term is used in a specific sense, the meaning of the term is specifically explained. Therefore, the terms used in this disclosure should not be interpreted as the simple name of the term, but should be interpreted based on the actual meaning of the term and the entire context of this disclosure.

[0020] The accompanying drawings of the present disclosure are intended to facilitate explanation of the present disclosure, and the shapes of the drawings may be exaggerated to facilitate understanding of the present disclosure, and therefore the present disclosure is not limited to the drawings.

[0021] In this disclosure, if it is determined that a detailed description of known configurations or functions related to the present disclosure would make the gist of the present disclosure unclear, the detailed description will be omitted as necessary.

[0022] According to one embodiment, a server for determining whether a medication has been administered may include a transceiver for receiving a video recorded by a wearable device, a memory for storing a detection model and a confirmation model, the detection model being trained to output whether each of predefined targets appears in the image, and the confirmation model being trained to output whether a medication has been administered, the predefined targets including objects related to the medication or the medication container, and postures related to medication administration, and one or more processors configured to detect the predefined targets by inputting image frames of the video into the detection model, and determine whether a medication has been administered by inputting confirmation model input data generated based on the detection results of the detection model into the confirmation model. The posture may include at least one of a posture for holding a medication container or a posture for taking a medication.

[0023] The detection result may include a plurality of detection model output data, each of the detection model output data corresponding to a respective image frame of the video. The one or more processors may be further configured to generate the confirmation model input data by combining the plurality of detection model output data.

[0024] The confirmation model input data may be generated by aligning a number of detection model output data with a time series of corresponding image frames.

[0025] The detection model may include an object detection model and a pose detection model. The one or more processors may be further configured to detect objects by inputting image frames of the video to the object detection model and detect poses by inputting image frames of the video to the pose detection model.

[0026] The detection models may include a first detection model for detecting one or more pre-defined first type targets and a second detection model for detecting one or more pre-defined second type targets. The memory may store a classification model for classifying a category of the video, and the one or more processors may be further configured to define a category of the video using the video and the classification model, and detect the pre-defined targets by inputting image frames of the video to the first detection model if the category of the video is defined to the first category corresponding to the first type, and detect the pre-defined targets by inputting image frames of the video to the second detection model if the category of the video is defined to a second category corresponding to the second type.

[0027] The validation models may include a first validation model corresponding to the first detection model and a second validation model corresponding to the second detection model. The one or more processors may be further configured to determine whether a medication has been administered by inputting the validation model input data into the first validation model if the category of the video is defined as the first category, and to determine whether a medication has been administered by inputting the validation model input data into the second validation model if the category of the video is defined as the second category.

[0028] The one or more processors may be further configured to estimate a position of the preset target such that the detection results and the confirmation model input data reflect a position of the preset target within the image frames of the video, and determine whether the drug has been administered taking into account the position of the preset target.

[0029] According to one embodiment, a method executed by one or more processors for determining whether a medication has been administered may include receiving a video recorded by a wearable device, detecting one or more predefined targets by inputting image frames of the video into a detection model, the detection model being trained to output whether each of the predefined targets appears in the image, the predefined targets including objects related to the medication or the medication container, and postures related to medication administration, and determining whether a medication has been administered by inputting confirmation model input data generated based on detection results of the detection model into the confirmation model. The postures may include at least one of a posture of holding the medication container or a posture of taking the medication.

[0030] The detection result may include a plurality of detection model output data, each of the detection model output data corresponding to a respective image frame of the video. The method may further include generating the confirmation model input data by combining the plurality of detection model output data.

[0031] The confirmation model input data may be generated by time-sequencing at least a portion of the detection model output data.

[0032] The detection model may include an object detection model and a pose detection model. The detecting step may include detecting objects by inputting image frames of the video into the object detection model, and detecting poses by inputting image frames of the video into the pose detection model.

[0033] The detection models may include a first detection model for detecting one or more pre-defined targets of a first type, and a second detection model for detecting one or more pre-defined targets of a second type. The method may further include defining a category of the video using the video and the classification model. The detecting may include detecting the pre-defined targets by inputting image frames of the video into the first detection model if the category of the video is defined into a first category corresponding to the first type, and detecting the pre-defined targets by inputting image frames of the video into the second detection model if the category of the video is defined into a second category corresponding to the second type.

[0034] The validation model may include a first validation model corresponding to the first detection model and a second validation model corresponding to the second detection model. The determining step may include determining whether a medication has been administered by inputting the validation model input data into the first validation model if the category of the video is defined as the first category, and determining whether a medication has been administered by inputting the validation model input data into the second validation model if the category of the video is defined as the second category.

[0035] The detecting step may include estimating a location of the preset target such that the detection results and the validation model input data reflect a location of the preset target within the image frames of the video. The determining step may include determining whether the drug has been administered given the location of the preset target.

[0036] According to one embodiment, a method executed by one or more processors for training a detection model and a confirmation model to determine whether a medication has been administered may include the steps of: acquiring detection model training data including training images and detection model labels corresponding to the training images, each of the detection model labels may have a class of medication or medication container and a class of posture associated with medication administration, the detection model labels may include a first detection model label corresponding to a first image of the training images and a second detection model label corresponding to a second image of the training images; acquiring output data by inputting the training images into the detection model; updating the detection model by comparing the detection model labels to the detection model output data; acquiring confirmation model training data including confirmation model input data and confirmation model labels corresponding to the confirmation model input data, the confirmation model labels may include labels for determining whether a medication has been administered, the confirmation model input data may be generated based on the detection model output data; acquiring output data by inputting the confirmation model input data into the confirmation model; and updating the confirmation model by comparing the confirmation model labels to the confirmation model output data. The posture may include at least one of a posture of holding a medication container or a posture of taking a medication.

[0037] According to one embodiment, a method executed by one or more processors for training a surveillance model that determines whether a user has administered a medication using video recorded by a wearable device worn on a user's wrist, the method may include steps of acquiring main hand training data including a main hand training video and a first label corresponding to the main hand training video, where the main hand training video was recorded by a first wearable device when a first person wearing the first wearable device on their main hand administered the medication primarily with the main hand, and where the first label indicates administration of the medication; acquiring sub-hand training data including a sub-hand training video and a second label corresponding to the sub-hand training video, where the sub-hand training video was recorded by a second wearable device when a second person wearing the second wearable device on an opposite hand to the sub-hand administered the medication primarily with the sub-hand, and where the second label indicates administration of the medication; updating the surveillance model using one of the main hand training data and the sub-hand training data; and updating the updated surveillance model again using the other of the main hand training data and the sub-hand training data.

[0038] The first label may include a main hand label that displays drug administration primarily performed by the main hand. The second label may include a sub-hand label that displays drug administration primarily performed by the sub-hand. The monitoring model may distinguish between drug administration primarily performed by the main hand and drug administration primarily performed by the sub-hand to determine whether the user has administered the drug.

[0039] The method may further include a step of acquiring no-drug-administered training data including a no-drug-administered training video and a no-drug-administered label corresponding to the no-drug-administered training video, the no-drug-administered training video representing no-drug administration and a no-drug-administered label indicating no-drug administration, and a step of re-updating the updated monitoring model using the no-drug-administered training data.

[0040] According to one embodiment, a method executed by one or more processors for training detection and confirmation models to determine whether a user has administered a medication using a video recorded by a wearable device worn on a user's wrist, the method including the steps of acquiring main hand training data including a main hand training video, a first detection model label, and a first confirmation model label, the main hand training video may be recorded by a first wearable device when a first person wearing the first wearable device on their main hand administered a medication primarily with their main hand, each of the first detection model labels may include a class for detecting an appearance of a medication or a medication container and a class for detecting a posture associated with medication administration, and the first confirmation model label may indicate medication administration; ... The method may include a step of acquiring sub-hand training data including a training video, a second detection model label, and a second confirmation model label, where the sub-hand training video may be recorded by a second wearable device when a second person wearing the second wearable device on an opposite hand to the sub-hand administers a medication primarily via the sub-hand, where each of the second detection model labels may include a class for detecting an appearance of the medication or medication container and a class for detecting a posture associated with medication administration, and where the second confirmation model label may indicate medication administration; updating a judgment model using at least a portion of the main hand training data and at least a portion of the sub-hand training data; and updating a confirmation model using at least a portion of the main hand training data and at least a portion of the sub-hand training data.

[0041] The confirmation model may be updated further using output data of the updated decision model.

[0042] The judgment model may be updated using the main hand training video, the first detection model label, the sub-hand training video, and the second detection model label. The verification model may be updated using the first detection model label, the first verification model label, the second detection model label, and the second verification model label.

[0043] The first confirmation model label may include a main hand label that displays drug administration mainly performed by the main hand. The second confirmation model label may include a sub-hand label that displays drug administration mainly performed by the sub-hand. The confirmation model may distinguish between drug administration mainly performed by the main hand and drug administration mainly performed by the sub-hand to determine whether the user has administered the drug.

[0044] According to one embodiment, a method executed by one or more processors for determining whether a user has administered a medication using video recorded by a wearable device worn on a user's wrist, the method includes the steps of receiving video recorded by the wearable device and determining whether the user has administered a medication by inputting the video into a surveillance model, the surveillance model may be updated using one of the main hand training data and the sub-hand training data and then updated again using the other of the main hand training data and the sub-hand training data, the main hand training data being a main hand training video and a main hand training video. The sub-hand training data may include a first label corresponding to the main hand training video, the main hand training video may be recorded by the first wearable device when a first person wears the first wearable device on their main hand and administers the medication primarily via the main hand, and the first label may indicate medication administration, the sub-hand training data may include the sub-hand training video and a second label corresponding to the sub-hand training video, the sub-hand training video may be recorded by the second wearable device when a second person wears the second wearable device on an opposite hand to the sub-hand and administers the medication primarily via the sub-hand, and the second label may indicate medication administration.

[0045] The first label may include a main hand label that displays drug administration primarily performed by the main hand. The second label may include a sub-hand label that displays drug administration primarily performed by the sub-hand. The monitoring model may distinguish between drug administration primarily performed by the main hand and drug administration primarily performed by the sub-hand to determine whether the user has administered the drug.

[0046] According to one embodiment, a server for determining whether a user has administered a medication using video recorded by a wearable device worn on a user's wrist, the server may include a transceiver that receives the video recorded by the wearable device, a memory that stores a surveillance model for determining whether the user has administered a medication, and one or more processors configured to determine whether the user has administered a medication using the video received via the transceiver and the surveillance model, where the surveillance model may be trained using at least a main hand training video recorded by the first wearable device when a first person wearing the first wearable device on their main hand administered the medication primarily with their main hand, and the one or more processors may be further configured to determine medication administration using the video and the surveillance model, where the video is recorded by the wearable device when a user wearing the wearable device on a hand opposite the sub-hand administered the medication primarily with their sub-hand.

[0047] The one or more processors may be further configured to distinguish between drug administration performed primarily by the main hand and drug administration performed primarily by the sub-hand and to determine whether the user has administered the drug.

[0048] According to one embodiment, a server for determining whether a drug administration has been performed may include a transceiver for receiving a video recorded by the wearable device, a memory for storing a first model and a second model, where the first model may be trained to output whether a drug administration associated with a first category of medical supplies has been performed and the second model may be trained to output whether a drug administration associated with a second category of medical supplies has been performed, and one or more processors further configured to determine whether a drug administration has been performed by inputting the video into the first model or the second model, where the one or more processors may be further configured to define a category of the video based on image frames of the video, and if the category of the video is defined as the first category, determine whether a drug administration corresponding to the first category has been performed by inputting the video into the first model, and if the category of the video is defined as the second category, determine whether a drug administration corresponding to the second category has been performed by inputting the video into the second model.

[0049] The memory may store a classification model for defining a category of the video, where the one or more processors may be further configured to define a category of the video using the classification model and the video.

[0050] The classification model may define a category of the video by considering both the first category of medical supplies and the second category of medical supplies, where the one or more processors may be further configured to: when determining whether a drug administration has been performed using the first model, determine whether a drug administration has been performed by considering the first category of medical supplies and the first category posture associated with drug administration without considering the second category of medical supplies, and when determining whether a drug administration has been performed using the second model, determine whether a drug administration has been performed by considering the second category of medical supplies and the second category posture associated with drug administration without considering the first category of medical supplies.

[0051] The classification model may further consider the first category pose and the second category pose to define a category of the video.

[0052] The video may include a plurality of image frames, where the one or more processors may be further configured to calculate first and second values ​​reflecting a number of image frames in which the first and second categories of medical items appear, respectively, by inputting each of the plurality of image frames into the classification model, and define a category of the video using the first and second values.

[0053] The video may include a plurality of image frames, where the one or more processors may be further configured to calculate first and second values ​​reflecting a number of medical items of a first and second category appearing in the plurality of image frames by inputting each of the plurality of image frames into the classification model, and define a category of the video using the first and second values.

[0054] The transceiver may receive data indicative of a category of the video, where the one or more processors may be further configured to define the category of the video in response to the received data.

[0055] The one or more processors may be further configured to store in the memory information related to whether the user administered the medication as determined by the model, and information related to a category of the video.

[0056] According to an embodiment, a system for determining whether a drug administration has been performed may include a server of the present disclosure and a management server. The management server may be configured to receive information related to whether a user has administered a drug and information related to a category of a video from the server, obtain a drug administration schedule for the user, and determine a user's medication adherence to the drug administration schedule by comparing the information related to whether the user has administered a drug with the information related to the category of the video.

[0057] In this disclosure, medication administration is meant to encompass various types of medication, including oral administration of medication, such as swallowing a medication (e.g., pills, tablets, capsules, powders, and liquids), inhaling a medication using an inhaler, turbuhaler, nasal spray, etc., administering eye drops using an eye dropper, and injecting a medication using a syringe.

[0058] In the present disclosure, determining whether a medication has been administered refers to determining, for example, whether the medication has been swallowed, whether the medication has been inhaled using an inhaler, turbuhaler, nasal spray, etc., whether eye drops have been administered, and whether the medication has been injected using a syringe.

[0059] FIG. 1 is a diagram showing a drug administration process according to an embodiment, showing the entire process of a user administering a drug. Referring to FIG. 1, the entire process of administering a drug may include a user approaching a pill bottle containing pills therein (S10), a user taking out the pills from the pill bottle into a palm (S20), a user picking up the pills in the palm with a finger (S30), and a user holding the pills in a mouth and swallowing the pills (S40). However, the pattern of performing the drug administration process (hereinafter referred to as a "drug administration pattern") may be different between users, and even for the same user, the drug administration pattern may be different from time to time. For example, in some cases, a user may hold a pill bottle directly to a mouth and swallow a pill. Also, FIG. 1 shows a case where a user takes out a pill from a pill bottle and swallows the pill, but when a different type of drug administration is performed, the drug administration process or drug administration pattern may be different. In some embodiments, the drug administration process may include pushing a pill out of a blister pack and swallowing the pill, inhaling the drug using an inhaler, turbuhaler, nasal spray, etc., administering eye drops using an eye dropper, and / or injecting the drug using a syringe. For example, when inhaling a drug using an inhaler, the entire process of administering the drug may include the user approaching the inhaler, opening the cap of the inhaler, holding the inhaler to their mouth, and / or inhaling the drug. Thus, the user's drug administration pattern may also vary.

[0060] In the present disclosure, a method for determining whether a drug has been administered based on a video, a server using the method, and a system including the server are described. In particular, a method for determining whether a drug has been administered, a server using the method, and a system including the server are described. The method is applicable even when performing a drug administration process according to various drug administration patterns as described above.

[0061] 1. System for determining whether a drug has been administered

[0062] According to one embodiment of the present disclosure, a system for determining whether a medication has been administered is disclosed.

[0063] The system for determining whether a drug has been administered according to the embodiment is a system for determining whether a user has administered a drug when the user administers the drug to diagnose, treat, or prevent a disease or disorder, or to promote health.

[0064] Systems according to embodiments may include multiple devices that may cooperate, for example by transmitting and receiving data, to determine whether a medication has been administered.

[0065] 2 is a diagram illustrating a system for determining whether a medication has been administered, according to one embodiment. Referring to FIG. 2, the system may include a wireless communication device 1000, a wearable device 2000, and a server 3000.

[0066] The wireless communication device 1000 may be attached to or placed near a medical item (eg, a medication container such as a medicine bottle, an inhaler, a turbuhaler, a nasal spray, an eye dropper, and a syringe).

[0067] The wireless communication device 1000 may collect data. For example, the wireless communication device 1000 may collect data on the movement of the medical item or the ambient light or both.

[0068] The wireless communication device 1000 may transmit the collected data to an external device such as the wearable device 2000 or the server 3000.

[0069] The wireless communication device 1000 may use the collected data to determine whether the user has started a drug administration action. For example, the wireless communication device 1000 may determine that the user has started a drug administration action when the movement of the medical supplies reaches or exceeds a predetermined level. As another example, the wireless communication device 1000 may determine that the user has started a drug administration action when the ambient light has a predetermined illuminance or higher. As yet another example, the wireless communication device 1000 may determine that the user has started a drug administration action when the movement of the medical supplies reaches or exceeds a predetermined level and the ambient light has a predetermined illuminance or higher.

[0070] The wireless communication device 1000 may transfer a signal instructing activation of the camera unit (hereinafter referred to as an "activation signal") to an external device such as the wearable device 2000 or the server 3000. For example, when it is determined that the user has started a drug administration action, the wireless communication device 1000 may transmit the activation signal to an external device. The camera unit may be the camera unit of the wearable device 2000. Activating the camera unit may mean generating a video through shooting.

[0071] The wireless communication device 1000 may be, but is not limited to, a short-range wireless communication device.

[0072] The wearable device 2000 may refer to a device provided in a form that can be worn by a user on the body. Examples of the wearable device 2000 include, but are not limited to, a smart watch, a smart band, a smart neckband, a smart ring, and a smart necklace.

[0073] The wearable device 2000 may communicate with the wireless communication device 1000. For example, the wearable device 2000 may receive data on the movement of the medical supplies or the ambient light or both from the wireless communication device 1000. As another example, the wearable device 2000 may receive a camera unit activation signal from the wireless communication device 1000.

[0074] The wearable device 2000 may generate images or videos. For example, the wearable device 2000 may generate images or videos of the medication administration.

[0075] The wearable device 2000 may generate an image or video based on data received from the wireless communication device 1000. For example, the wearable device 2000 may generate an image or video based on data of the movement of the medical supplies or the ambient light or both, or based on a wake-up signal. As a more specific example, the wearable device 2000 may generate an image or video when the movement of the medical supplies reaches a predetermined level or more. As another more specific example, the wearable device 2000 may generate an image or video when the ambient light is at a predetermined illuminance or more. As yet another more specific example, the wearable device 2000 may generate an image or video when the movement of the medical supplies reaches a predetermined level or more and the ambient light is at a predetermined illuminance or more. As yet another more specific example, the wearable device 2000 may generate an image or video when the movement of the medical supplies reaches a predetermined level or more and the ambient light is at a predetermined illuminance or more. As yet another more specific example, the wearable device 2000 may generate an image or video when a wake-up signal is received.

[0076] The wearable device 2000 may generate an image or video based on the signal strength of the data received from the wireless communication device 1000. For example, the wearable device 2000 may generate an image or video if the signal strength is within a predetermined range. The signal strength may be represented as, but is not limited to, a received signal strength indicator (RSSI). When the signal strength is represented as an RSSI, examples of the predetermined range include, but are not limited to, a range of -90 dBm or more, a range of -70 dBm or more, and a range from -70 dBm to -50 dBm.

[0077] The wearable device 2000 may generate an image or video by considering the signal strength of the data received from the wireless communication device 1000 and the contents of the data. Here, the signal strength has priority over the contents. For example, the wearable device 2000 does not generate an image or video if the signal strength is not within a predetermined range even if the movement of the medical product reaches a predetermined degree or more. As another example, the wearable device 2000 does not generate an image or video if the signal strength is not within a predetermined range even if the ambient light is at a predetermined illuminance or more. As yet another example, the wearable device 2000 does not generate an image or video if the signal strength is not within a predetermined range even if the movement of the medical product reaches a predetermined degree or more and the ambient light is at a predetermined illuminance or more. As yet another example, the wearable device 2000 does not generate an image or video if the signal strength is not within a predetermined range even if the activation signal is received. That is, the wearable device 2000 may generate an image or video by considering the signal strength of the data received from the wireless communication device 1000 before considering the contents of the data.

[0078] The wearable device 2000 may generate an image or a video by capturing an image of the surroundings. In particular, when the wearable device 2000 is a smart watch or a smart band, the wearable device 2000 may capture an image of the surroundings of a user's hand to generate an image or a video. In this case, the image or the video may include at least a partial area of ​​the user's palm. That is, at least a portion of the image or at least a portion of at least one image frame of the video may include at least a partial area of ​​the user's palm.

[0079] The wearable device 2000 may measure information about the user wearing the wearable device 2000. For example, the wearable device 2000 may measure biological information about the user wearing the wearable device 2000. Examples of the biological information include, but are not limited to, information about exercise such as the number of steps, a heart rate, blood oxygen saturation, and an electrocardiogram. As another example, the wearable device 2000 may measure information about the surroundings of the user wearing the wearable device 2000. As a specific example, the wearable device 2000 may measure light around the user wearing the wearable device 2000.

[0080] The wearable device 2000 may communicate with the server 3000. For example, the wearable device 2000 may transmit generated images or videos to the server 3000. As another example, the wearable device 2000 may transmit information about the measured user to the server 3000.

[0081] The server 3000 may communicate with the wireless communication device 1000 or the wearable device 2000 or both. The server 3000 may receive images or videos from the wearable device 2000. The server 3000 may receive medical item movement or ambient light or both data from the wireless communication device 1000 or the wearable device 2000 or both.

[0082] The server 3000 may determine whether the medicine has been administered. For example, the server 3000 may determine whether the medicine has been administered based on information obtained from the wireless communication device 1000 or the wearable device 2000 or both. As a specific example, the server 3000 may determine whether the medicine has been administered by analyzing an image or video received from the wearable device 2000.

[0083] 1.1 Wireless communication devices

[0084] 3 is a block diagram illustrating a wireless communication device according to an embodiment. Referring to FIG. 3, a wireless communication device 1000 may include a motion sensor 1100, an ambient light sensor 1200, a communication unit 1300, a storage unit 1400, and a control unit 1500.

[0085] The motion sensor 1100 may sense the motion of the wireless communication device 1000. When the wireless communication device 1000 is attached to a medical article, the motion sensor 1100 may sense the motion of the medical article by sensing the motion of the wireless communication device 1000. The motion sensor 1100 may include at least some of an accelerometer 1110, a gyro sensor 1120, and a geomagnetic sensor 1130. The motion sensor 1100 may be, but is not limited to, a 6-axis sensor, a 9-axis sensor, etc.

[0086] The ambient light sensor 1200 may measure the light around the wireless communication device 1000. When the wireless communication device 1000 is attached to or placed near a medical item, the ambient light sensor 1200 may measure the light around the medical item.

[0087] The communication unit 1300 may communicate with an external device such as the wearable device 2000 or the server 3000. The communication unit 1300 may perform bidirectional or unidirectional communication. Examples of the communication unit 1300 include, but are not limited to, a transceiver, a beacon, a Bluetooth module, a WiFi module, a Zigbee module, an RF module, an NFC module, and combinations thereof.

[0088] The communication unit 1300 may transmit the motion or light data or both to an external device. For example, the communication unit 1300 may transmit data measured by the motion sensor 1100 or the ambient light sensor 1200 or both to an external device such as the wearable device 2000 or the server 3000.

[0089] The communication unit 1300 may transmit the wake-up signal to the wearable device 2000 or to an external device such as the server 3000.

[0090] The storage unit 1400 may store various kinds of data and programs required for the operation of the wireless communication device 1000. The storage unit 1400 may store information acquired by the wireless communication device 1000. For example, the storage unit 1400 may store data of motion or light or both. As a specific example, the storage unit 1400 may store data measured by the motion sensor 1100 or the ambient light sensor 1200 or both. As another example, the storage unit 1400 may store a wake-up signal. Examples of the storage unit 1400 include, but are not limited to, non-volatile semiconductor memory, hard disk, flash memory, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), other tangible non-volatile recording media, and combinations thereof.

[0091] The control unit 1500 may perform various types of information processing and manipulation within the wireless communication device 1000. The control unit 1500 may also control other elements that make up the wireless communication device 1000.

[0092] The control unit 1500 may be implemented as a computer or similar device according to hardware, software, or a combination thereof. In hardware, the control unit 1500 may be one or more processors. Alternatively, the control unit 1500 may be provided as physically separate processors cooperating by communication. Examples of the control unit 1500 include, but are not limited to, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a state machine, an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), and combinations thereof. In software, the control unit 1500 may be provided as a program for operating the hardware control unit 1500.

[0093] The control unit 1500 may execute control such that the motion sensor 1100 senses motion or the ambient light sensor 1200 measures light.

[0094] The control unit 1500 may execute control such that data of the motion or the light or both is transmitted to the outside via the communication unit 1300. For example, the control unit 1500 may execute control such that data measured by the motion sensor 1100 or the ambient light sensor 1200 or both is transmitted to the outside, such as the wearable device 2000 or the server 3000, via the communication unit 1300.

[0095] The control unit 1500 may perform control such that data of motion or light or both is stored in the storage unit 1400. For example, the control unit 1500 may perform control such that data measured by the motion sensor 1100 or the ambient light sensor 1200 or both is stored in the storage unit 1400.

[0096] The control unit 1500 may use the data of the movement or the light or both to determine whether the user has started the medicine administration action. For example, the control unit 1500 may determine that the user has started the medicine administration action when the movement reaches a predetermined level or more. As another example, the control unit 1500 may determine that the user has started the medicine administration action when the light is at a predetermined illuminance or more. As yet another example, the control unit 1500 may determine that the user has started the medicine administration action when the movement reaches a predetermined level or more and the light is at a predetermined illuminance or more. The movement data may be data measured by the movement sensor 1100. The light data may be data measured by the ambient light sensor 1200.

[0097] The control unit 1500 may transmit a wake-up signal to an external device, such as the wearable device 2000 or the server 3000, via the communication unit 1300. For example, when it is determined that the user has started a drug administration action, the control unit 1500 may transmit a wake-up signal to an external device, such as the wearable device 2000 or the server 3000, via the communication unit 1300.

[0098] Hereinafter, unless otherwise specified, it may be understood that the operations of the wireless communication device 1000 are performed under the control of the control unit 1500.

[0099] Not all elements shown in Fig. 3 are essential elements of the wireless communication device 1000. The wireless communication device 1000 may further include elements not shown in Fig. 3. Furthermore, at least some of the elements of the wireless communication device 1000 shown in Fig. 3 may be omitted. For example, the wireless communication device 1000 may not include the motion sensor 1100 or the ambient light sensor 1200, or neither of them.

[0100] 1.2 Wearable devices

[0101] 4 is a block diagram illustrating a wearable device according to one embodiment. Referring to FIG. 4, a wearable device 2000 may include a camera unit 2100, a user input unit 2200, an output unit 2300, a communication unit 2400, a storage unit 2500, and a control unit 2600.

[0102] The camera unit 2100 may generate an image or a video. For example, the camera unit 2100 may capture an image of the surroundings of the wearable device 2000 to generate an image or a video. The camera unit 2100 may include one or more camera modules. Examples of camera modules include, but are not limited to, an RGB camera, a thermal imaging camera, and a night vision device.

[0103] The video generated by the camera unit 2100 may include a plurality of image frames. At least some of the plurality of image frames may include at least a partial area of ​​the user's palm. The video may include a predetermined number of the plurality of image frames. The video may be a video captured over a predetermined period of time.

[0104] The camera unit 2100 may be disposed at one point in the wearable device 2000. For example, when the wearable device 2000 is a smart watch or a smart band, the camera unit 2100 may be disposed in a strap portion. Here, the camera unit 2100 may be disposed so that the camera unit 2100 is located inside the wrist when the user wears the wearable device 2000 on the wrist. Also, the camera unit 2100 may be disposed so that the camera unit 2100 faces the direction of the user's hand and captures the surroundings of the user's hand when the user wears the wearable device 2000 on the wrist. The above-described positions and directions for disposing the camera unit 2100 are merely examples and are not limited thereto.

[0105] The user input unit 2200 may perform the function of obtaining information from a user. The user input unit 2200 may receive user input from a user. Examples of user input include, but are not limited to, key input, touch input, voice input, and combinations thereof.

[0106] The user input unit 2200 may be implemented as a commonly used user input device, examples of which include, but are not limited to, a touch sensor for sensing a user touch, a microphone for receiving an audio signal, a gesture sensor for recognizing a user gesture, and combinations thereof.

[0107] The output unit 2300 may perform a function of outputting information so that a user can view the information. The output unit 2300 may output information obtained from a user, or information obtained from and / or processed by an external device. The output of information may be, or may be provided in various forms, including but not limited to, visual, auditory, and / or tactile output.

[0108] The output unit 2300 may be realized as a display for outputting images, a speaker for outputting sounds, a haptic device for generating vibrations, and / or other output means.

[0109] Alternatively, instead of being a device itself that outputs information externally, output unit 2300 may be realized as an output interface (such as a USB port or a PS / 2 port) that connects wearable device 2000 to an external output device that outputs information.

[0110] The output unit 2300 may be integrated with the user input unit 2200. For example, if the output unit 2300 is a display, the output unit 2300 may be a touch display integrated with the touch sensor of the user input unit 2200.

[0111] Similar to the communication unit 1300 of the wireless communication device 1000 described above, the communication unit 2400 may execute a function of communicating with the outside, for example, a function of transmitting and receiving data to and from the outside. For example, the communication unit 2400 may receive motion or light data or both from the wireless communication device 1000. As another example, the communication unit 2400 may receive a wake-up signal from the wireless communication device 1000. As yet another example, the communication unit 2400 may transmit an image or video to the server 3000. As yet another example, the communication unit 2400 may transmit motion or light data or both to the server 3000. The same or similar parts as those of the communication unit 1300 of the wireless communication device 1000 will not be described again.

[0112] Similar to the storage unit 1400 of the wireless communication device 1000 described above, the storage unit 2500 may store various types of data and programs necessary for the wearable device 2000 to operate. The storage unit 2500 may store information acquired by the wearable device 2000. For example, the storage unit 2500 may store images or videos generated by the camera unit 2100. As another example, the storage unit 2500 may store user input. As yet another example, the storage unit 2500 may store information output so that the user can confirm the information. As yet another example, the storage unit 2500 may store motion or light data or both data received from the wireless communication device 1000. As yet another example, the storage unit 2500 may store a wake-up signal. For parts that are the same as or similar to the storage unit 1400 of the wireless communication device 1000, a duplicated description will be omitted.

[0113] Similar to the control unit 1500 of the wireless communication device 1000 described above, the control unit 2600 may process and operate various types of information within the wearable device 2000. The control unit 2600 may control other elements constituting the wearable device 2000. Duplicate descriptions of parts that are the same as or similar to the control unit 1500 of the wireless communication device 1000 will be omitted.

[0114] The control unit 2600 may activate the camera unit 2100. In this case, the control unit 2600 may execute control such that the camera unit 2100 generates an image or a video. For example, the control unit 2600 may execute control such that the camera unit 2100 captures the surroundings and generates an image or a video. The image or video may be an image or a video of drug administration.

[0115] The control unit 2600 may activate the camera unit 2100 based on data received from the wireless communication device 1000. For example, the control unit 2600 may activate the camera unit 2100 based on data of the movement of the medical supplies or the ambient light or both received from the wireless communication device 1000, or based on an activation signal. As a more specific example, the control unit 2600 may activate the camera unit 2100 when the movement of the medical supplies reaches a predetermined level or more. As another more specific example, the control unit 2600 may activate the camera unit 2100 when the ambient light is at a predetermined illuminance or more. As yet another more specific example, the control unit 2600 may activate the camera unit 2100 when the movement of the medical supplies reaches a predetermined level or more and the ambient light is at a predetermined illuminance or more. As yet another more specific example, the control unit 2600 may activate the camera unit 2100 when the movement of the medical supplies reaches a predetermined level or more and the ambient light is at a predetermined illuminance or more. As yet another more specific example, the control unit 2600 may activate the camera unit 2100 when an activation signal is received.

[0116] The control unit 2600 may activate the camera unit 2100 based on the signal strength of the data received from the wireless communication device 1000. For example, the control unit 2600 may activate the camera unit 2100 if the signal strength is within a predetermined range. The signal strength may be represented as, but is not limited to, a received signal strength indicator (RSSI). When the signal strength is represented as RSSI, examples of the predetermined range include, but are not limited to, a range of -90 dBm or more, a range of -70 dBm or more, and a range from -70 dBm to -50 dBm.

[0117] The control unit 2600 may activate the camera unit 2100 taking into consideration the signal strength of the data received from the wireless communication device 1000 rather than the contents of the data. Here, the signal strength has priority over the contents. For example, the control unit 2600 does not activate the camera unit 2100 when the signal strength is not within a predetermined range even if the movement of the medical supplies reaches a predetermined level or more. As another example, the control unit 2600 does not activate the camera unit 2100 when the signal strength is not within a predetermined range even if the ambient light is at a predetermined illuminance or more. As yet another example, the control unit 2600 does not activate the camera unit 2100 when the movement of the medical supplies reaches a predetermined level or more and the ambient light is at a predetermined illuminance or more but the signal strength is not within a predetermined range. As yet another example, the control unit 2600 does not activate the camera unit 2100 when the signal strength is not within a predetermined range even if the movement of the medical supplies reaches a predetermined level or more and the ambient light is at a predetermined illuminance or more. As yet another example, the control unit 2600 does not activate the camera unit 2100 when the signal strength is not within a predetermined range even if an activation signal is received. That is, the control unit 2600 may activate the camera unit 2100 taking into account the signal strength of the data before taking into account the content of the data received from the wireless communication device 1000 .

[0118] The control unit 2600 may perform control such that the user input unit 2200 obtains information from a user. The control unit 2600 may perform control such that the user input unit 2200 receives user input from a user.

[0119] The control unit 2600 may execute control such that the output unit 2300 executes a function of outputting information so that a user can view the information.

[0120] The control unit 2600 may execute control so that the communication unit 2400 executes a function of communicating with the outside, for example, a function of transmitting and receiving data to and from the outside. The control unit 2600 may obtain information from the outside via the communication unit 2400.

[0121] The control unit 2600 may execute control such that the storage unit 2500 stores various types of data and programs necessary for the operation of the wearable device 2000. The control unit 2600 may execute control such that the storage unit 2500 stores information acquired by the wearable device 2000.

[0122] Hereinafter, unless otherwise specified, the operation of the wearable device 2000 may be interpreted as being performed under the control of the control unit 2600.

[0123] Not all of the elements shown in Fig. 4 are essential elements of the wearable device 2000. The wearable device 2000 may further include elements not shown in Fig. 4. For example, the wearable device 2000 may further include at least one of a motion sensor that detects a user's motion, an ambient light sensor that measures ambient light, a heart rate sensor that measures the user's heart rate, a blood oxygen saturation sensor that measures the user's blood oxygen saturation, and an electrocardiogram sensor that measures the user's electrocardiogram. Furthermore, at least some of the elements of the wearable device 2000 shown in Fig. 4 may be omitted.

[0124] 1.3 Server

[0125] 5 is a block diagram illustrating a server according to an embodiment. Referring to FIG. 5, a server 3000 may include a server communication unit 3100, a server storage unit 3200, and a server control unit 3300.

[0126] Similar to the communication unit 1300 of the wireless communication device 1000 described above, the server communication unit 3100 may perform a function of communicating with the outside, such as a function of transmitting and receiving data to and from the outside. For example, the server communication unit 3100 may receive an image or a video from the wearable device 2000. As another example, the server communication unit 3100 may receive motion or light or both data from the wearable device 2000. The motion or light or both data may be data measured by the wearable device 2000. The motion data may incorporate the number of steps, posture, etc. of the user wearing the wearable device 2000. The light data may incorporate the light around the wearable device 2000. As another example, the server communication unit 3100 may receive biometric information of the user wearing the wearable device 2000 from the wearable device 2000. The same or similar parts as the communication unit 1300 of the wireless communication device 1000 will not be described again.

[0127] Similar to the storage unit 1400 of the wireless communication device 1000 described above, the server storage unit 3200 may store various types of data and programs required for the server 3000 to operate. The server storage unit 3200 may store information acquired by the server 3000. For example, the server storage unit 3200 may store images or videos received from the wearable device 2000. As another example, the server storage unit 3200 may store motion or light data or both data received from the wearable device 2000. Duplicate descriptions of parts that are the same as or similar to the storage unit 1400 of the wireless communication device 1000 will be omitted.

[0128] Similar to the control unit 1500 of the wireless communication device 1000 described above, the server control unit 3300 may process and operate various types of information in the server 3000. The server control unit 3300 may control other elements constituting the server 3000. Duplicate descriptions of parts that are the same as or similar to the control unit 1500 of the wireless communication device 1000 will be omitted.

[0129] The server control unit 3300 may execute control so that the server communication unit 3100 executes a function of communicating with the outside, for example, a function of transmitting and receiving data to and from the outside. The server control unit 3300 may obtain information from the outside via the server communication unit 3100.

[0130] The server control unit 3300 may execute control such that the server storage unit 3200 stores various kinds of data and programs required for the operation of the server 3000. The server control unit 3300 may execute control such that the server storage unit 3200 stores information acquired by the server 3000.

[0131] The server control unit 3300 may determine whether the medication has been administered. For example, the server control unit 3300 may determine whether the medication has been administered based on an image or video acquired from the wearable device 2000 via the server communication unit 3100.

[0132] Hereinafter, unless otherwise specified, the operations of the server 3000 may be interpreted as being performed under the control of the server control unit 3300.

[0133] Not all of the elements shown in Fig. 5 are essential elements of the server 3000. The server 3000 may further include elements not shown in Fig. 5. Furthermore, at least some of the elements of the server 3000 shown in Fig. 5 may be omitted.

[0134] The elements of the server 3000 may be physically included in one server, or may be distributed servers distributed by function. For example, the server 3000 may include a first server that executes a first function and a second server that executes a second function. As a specific example, the server 3000 may include a first server that determines whether a drug has been administered, and a second server that manages the result of determining whether a drug has been administered. The first server and the second server may be physically separate specific servers. For example, the first server may be located in a first country such as Korea, and the second server may be located in a second country such as the United States, Europe, Japan, and China.

[0135] 1.4 Additional Elements

[0136] Returning to Fig. 2, not all of the elements shown in Fig. 2 are essential elements of the system. Some of the elements of the system shown in Fig. 2 may be omitted, or elements not shown in Fig. 2 may be further included.

[0137] For example, the system may further include a user terminal. Examples of the user terminal include, but are not limited to, a smartphone, a tablet PC, a laptop computer, and a desktop computer. The user terminal may exchange data with at least some of the wireless communication device, the wearable device, and the server. The user terminal may provide the user, a guardian, or a medical professional with information regarding the user's drug administration, or an application for managing the user's drug administration. The user terminal may include an input unit, an output unit, a communication unit, a storage unit, a control unit, and the like. The configuration or realization method of the elements of the user terminal is the same as that of the wireless communication device 1000 or the wearable device 2000, so the overlapping parts will be omitted.

[0138] The application running on the user terminal may provide a scheduling service, a notification service, etc., to guide the user in taking medication. Here, when executing the application, the user terminal may apply data acquired from the wearable device 2000 or the server 3000.

[0139] 2. Method for determining whether a drug has been administered

[0140] 2.1 Introduction

[0141] A method for determining whether a drug has been administered according to an embodiment will be described below. For convenience of explanation, it is assumed that the method for determining whether a drug has been administered is executed by a server, but this is merely one of various embodiments and does not mean that the method for determining whether a drug has been administered is executed only by a server.

[0142] In other words, the method for determining whether a drug has been administered according to an embodiment is not necessarily performed only by the server, but may be performed by another server, such as a wireless communication device or a wearable device, etc. Furthermore, a part of the method for determining whether a drug has been administered may be performed by the server, and another part may be performed by a device other than the server, such as a wireless communication device or a wearable device.

[0143] In the following description of a method for determining whether a medication has been administered according to one embodiment, unless otherwise stated, operations performed by the server may be interpreted as being performed by the server control unit or by another element of the server controlled by the server control unit.

[0144] The server may use a model to determine whether a medication has been administered, examples of which include, but are not limited to, deep learning models such as artificial neural networks, support vector machines, generalized linear models, decision trees, random forests, gradient boosting machines, and combinations thereof.

[0145] Below, we describe some examples of methods for determining whether a drug has been administered based on a video according to one embodiment.

[0146] 2.2 Methods for Determining Whether a Drug Has Been Administered Using a Surveillance Model

[0147] 6 and 7 are diagrams illustrating a method for using a supervisory model to determine whether a medication has been administered, according to one embodiment.

[0148] In step S110, the server may obtain a video. For example, the server may receive a video from the wearable device, where the video may be obtained by a camera unit of the wearable device and transmitted via a wearable device communication unit. The video may include a plurality of image frames. The number of the plurality of image frames may be 50 or more, 100 or more, 200 or more, 400 or more, 600 or more, 800 or more, or 1000 or more. The entire drug administration process may be captured by video. Alternatively, a large portion of the entire drug administration process may be captured by video.

[0149] In step S120, the server may use the video and the surveillance model to determine whether the medication has been administered. The video may be acquired in step S110.

[0150] Referring to FIG. 7, the server may input the video 10 into the surveillance model and determine whether the medication has been administered. For example, the server control unit may input the video 10 received via the server communication unit into the surveillance model and determine whether the medication has been administered. As another example, the server control unit may store the video 10 received via the server communication unit in the server storage unit, and input the video 10 stored in the server storage unit into the surveillance model, thereby determining whether the medication has been administered. In this case, the input data of the surveillance model may be the video 10, and the output data may be the result of determining whether the medication has been administered.

[0151] The video input to the surveillance model may have a particular format. For example, the video may include a predetermined number of image frames. As another example, the video may have a predetermined resolution. As yet another example, the video may have a predetermined ratio (width to height).

[0152] The server may input the video to the surveillance model without performing pre-processing. For example, the server control unit may input the video received via the server communication unit to the surveillance model without performing pre-processing. That is, the video 10 input to the surveillance model may be the same as the video captured by the camera unit. Alternatively, the server may perform pre-processing on the video and input the pre-processed video to the surveillance model. For example, the server control unit may perform pre-processing on the video received via the server communication unit and input the pre-processed video to the surveillance model. That is, the video 10 input to the surveillance model may be the result of performing pre-processing on the video captured by the camera unit. Examples of pre-processing include, but are not limited to, normalization, resizing, cropping, noise removal, and combinations thereof.

[0153] The result of determining whether a drug has been administered output from the surveillance model may be an index indicative of whether a drug has been administered. That is, the output data of the surveillance model may be an index indicative of whether a drug has been administered. The server may use the surveillance model to obtain an index indicative of whether a drug has been administered for a video input to the surveillance model. As a non-limiting example, the index may be expressed as a numerical value.

[0154] The index, which is the output data of the monitoring model that indicates whether or not a drug has been administered, may have a variety of formats.

[0155] For example, an index that is output data of a monitoring model and indicates whether a drug has been administered may include an index indicating administration of a drug and an index indicating no administration of a drug. Here, the monitoring model may output either an index indicating administration of a drug or an index indicating no administration of a drug. For example, the monitoring model may output a value of 1 when administration of a drug is estimated, or output a value of 0 when no administration of a drug is estimated.

[0156] As another example, the index, which is output data of the monitoring model and indicates whether or not a drug has been administered, may include an index indicating the probability that drug administration is estimated (hereinafter referred to as a "drug administration probability index"), and an index indicating the probability that no drug administration is estimated (hereinafter referred to as a "no drug administration probability index"). Here, the monitoring model may output the drug administration probability index and the no drug administration probability index together. The server may use the drug administration probability index and the no drug administration probability index to determine whether or not a drug has been administered. For example, if the drug administration probability index is greater than the no drug administration probability index, the server may determine that a drug has been administered. If the no drug administration probability index is greater than the drug administration probability index, the server may determine that a drug has not been administered. As another example, the server may determine that a drug has been administered if the drug administration probability index meets a predetermined condition. As an example of meeting the predetermined condition, the drug administration probability index is greater than or equal to a predetermined value, but is not limited thereto.

[0157] Examples of supervision models include, but are not limited to, transformer models with attention mechanisms, recurrent neural network (RNN) models such as long short-term memory (LSTM), hybrid models such as 2D CNN-LSTM, and deep learning models such as 3D ConvNet, Inflated 3D ConvNet (I3D), and SlowFast networks.

[0158] When the server determines whether or not a drug has been administered based on a video, the accuracy of determining whether or not a drug has been administered is improved compared to when the server determines whether or not a drug has been administered based on a single image frame or several image frames. For example, when determining whether or not a drug has been administered using only a single image frame, it is determined that the drug has been administered when the drug is detected in the image frame. As another example, when determining whether or not a drug has been administered using only several image frames, it is determined that the drug has been administered when the drug is detected in a predetermined number or more of the several image frames. However, even in these cases, it is possible that the user has not actually administered the drug. Therefore, it is difficult to accurately determine whether or not a drug has been administered using only a single image frame or several image frames. Conversely, as described above, when determining whether or not a drug has been administered based on a video, it is not determined that the drug has been administered just because the drug has been detected in a specific image frame, but whether or not the drug has been administered is determined taking into account the drug administration process captured in the entire video. Therefore, compared to when determining whether or not a drug has been administered based on a single image frame or several image frames, it is more accurate to determine whether or not a drug has been administered.

[0159] 2.3 Methods for Determining Whether a Drug Has Been Administered Using Detection and Confirmation Models

[0160] 8 and 9 are diagrams illustrating a method for determining whether a medication has been administered using a detection model and a confirmation model, according to one embodiment. A server may use the video, the detection model, and the confirmation model to determine whether a medication has been administered. The server may obtain a video (S210), detect medication administration related objects and poses (hereinafter referred to as medication administration related objects and medication administration related poses) using the video and the detection model (S220), and use the detection results of the medication administration related objects and poses (hereinafter referred to as "detection results") and the confirmation model to determine whether a medication has been administered (S230).

[0161] In step S210, the server may obtain a video. For example, the server may receive a video from the wearable device, where the video may be obtained by a camera unit of the wearable device and transmitted via a wearable device communication unit. The video may include a plurality of image frames. The number of the plurality of image frames may be 50 or more, 100 or more, 200 or more, 400 or more, 600 or more, 800 or more, or 1000 or more. The entire drug administration process may be captured by video. Alternatively, a large portion of the entire drug administration process may be captured by video.

[0162] In step S220, the server may use the video and the detection model to detect drug administration related objects and poses. For example, the server control unit may use the video received via the server communication unit and the detection model to detect drug administration related objects and poses. As another example, the server control unit may store the video received via the server communication unit in the server storage unit, and may use the video and the detection model stored in the server storage unit to detect drug administration related objects and poses. The video may be obtained in step S210.

[0163] FIG. 10 illustrates detection of medication administration related objects and poses using a detection model, according to one embodiment.

[0164] Referring to FIG. 10 , the server may input image frames 41 and 42 of a video 40 into a detection model to detect medication-administration related objects and poses included in image frames 41 and 42. In this case, the input data of the detection model may be image frames 41 and 42. The server may input the video for each image frame into the detection model to detect medication-administration related objects and poses in the image frames. In this case, the server detects medication-administration related objects and poses for all image frames of the video such that medication-administration related objects and poses included in the video are detected. The server may detect medication-administration related objects and poses for only a portion of all image frames of the video.

[0165] The server inputting an image frame into the detection model to detect a drug administration-related object may mean that the server inputs an image frame into the detection model and determines whether or not a drug administration-related object is present in the image frame. The server inputting an image frame into the detection model to detect a drug administration-related posture may mean that the server inputs an image frame into the detection model and determines whether or not a drug administration-related posture is present in the image frame. Here, determining whether or not a drug administration-related posture is present may mean determining whether or not a specific scene corresponding to the drug administration-related posture is present. In this case, the server inputting an image frame into the detection model to detect a drug administration-related posture may mean that the server inputs an image frame into the detection model and determines whether or not a specific scene corresponding to the drug administration-related posture is present in the image frame. Drug administration-related objects and drug administration-related postures will be described in detail later.

[0166] The image frames input to the detection model may have a particular format. For example, the image frames may have a predefined resolution. As another example, the image frames may have a predefined ratio (width to height).

[0167] The server may input the image frames to the detection model without performing pre-processing. For example, the server control unit may input the image frames received via the server communication unit to the detection model without performing pre-processing. That is, the image frames 41 and 42 input to the detection model may be the same as the image frames acquired by the camera unit. Alternatively, the server may perform pre-processing on the image frames and input the pre-processed image frames to the detection model. For example, the server control unit may perform pre-processing on the image frames received via the server communication unit and input the pre-processed image frames to the detection model. That is, the image frames 41 and 42 input to the detection model may be the result of pre-processing on the image frames acquired by the camera unit. Examples of pre-processing include, but are not limited to, normalization, resizing, cropping, noise removal, and combinations thereof.

[0168] The detection model may output data related to medication administration-related objects and postures (hereinafter referred to as "detection model output data"). In this case, the output data of the detection model may be referred to as detection model output data. The server may use the detection model to obtain the detection model output data. If the video includes N image frames, the server inputs the N image frames into the detection model and obtains N detection model output data corresponding to the N image frames. That is, the server may input the video to the detection model for each image frame, and obtain a detection result corresponding to the video, including detection model output data corresponding to each image frame. As a non-limiting example, the detection model output data may be expressed as a numerical value. The detection model output data will be described in detail later.

[0169] Medication-administration related objects and medication-administration related postures are described in more detail below.

[0170] FIG. 11 illustrates several types of medication-related objects. The medication-related objects may include various types of objects related to medication, such as objects used by a user to administer medication. As a non-limiting example, with reference to FIG. 11, the medication-related objects may include a pill bottle 21, an eye dropper 22, a nasal sprayer 23, a turbuhaler 24, a pill 25, an inhaler 26, a blister pack 27, a syringe 28, and the like. Although not illustrated, the medication-related objects may include body parts of a user, such as an arm, a hand, a wrist, a palm, an outer portion of the palm, a face, a mouth, and a nose.

[0171] 12 is a diagram illustrating several types of postures related to medication administration. The postures related to medication administration may include various types of postures related to medication administration, such as positioning of a user for medication administration. As a non-limiting example, referring to FIG. 12, the postures related to medication administration may include postures for holding a medication administration related object, such as a posture 31 for holding a blister pack, a posture 32 for holding an inhaler, a posture 33 for holding a turbuhaler, a posture 34 for holding an eye dropper, a posture 35 for holding a nasal sprayer, and a posture 36 for holding a pill, a posture 37 for swallowing a medication, such as a pill, postures for inhaling a medication using a medication administration related object, such as an inhaler, a turbuhaler, and a nasal sprayer, postures for administering eye drops using a medication administration related object, such as an eye dropper, postures for injecting a medication using a medication administration related object, such as a syringe, and postures for opening a medication administration related object, such as a pill bottle.

[0172] Comparing the medication administration related postures with the medication administration related objects, the medication administration related postures may be considered as specific postures involving one or more medication administration related objects. For example, posture 31 for holding a blister pack may be considered as a posture involving a blister pack and a hand. As another example, posture 37 for swallowing a medication may be considered as a posture involving a medication and a face. That is, the medication administration related posture may include at least a part of a body and at least a part of a medical item. Thus, detecting the medication administration related posture by the server may be detecting a specific posture involving a medication administration related object without individually detecting one or more medication administration related objects.

[0173] In some embodiments, the detection model may detect drug administration-related objects and poses associated with a particular type of drug administration. For example, the detection model may detect drug administration-related objects and poses associated with swallowing a drug from an image frame because the detection model is trained to detect drug administration-related objects and poses associated with swallowing a drug, such as oral administration of a pill from a pill bottle. As a more specific example, when an image frame is input, the detection model may detect a pill bottle, a pill, an action of opening a pill bottle, an action of holding a pill, and an action of swallowing a pill. In this case, the detection model may not be able to detect drug administration-related objects and poses associated with drug administration other than the particular type. For example, a detection model trained to detect drug administration-related objects and poses associated with swallowing a drug, such as oral administration of a pill from a pill bottle, may not detect drug administration-related objects and poses associated with inhaling a drug using an inhaler.

[0174] In some embodiments, the detection model may detect medication-administration related objects and poses associated with multiple types of medication administration, for example, the detection model may detect medication-administration related objects and poses associated with at least two of the following medication administrations from the image frames: swallowing a medication, using an inhaler, using a turbuhaler, using a nasal spray, administering eye drops, and injecting a medication, and thus may detect medication-administration related objects and poses associated with at least two medication administrations from the image frames.

[0175] The detection model output data is described in more detail below.

[0176] As described above, the server may use the detection model to detect detection model output data. The detection model output data may include a plurality of sub-data. Each of the plurality of sub-data may correspond to a particular region in the image frame. The particular region may be at least one of a region corresponding to a medication administration related object, a region corresponding to a medication administration related posture, and a region not related to medication administration.

[0177] The detection model output data includes position information incorporating the location of medication-administration-related objects and poses within the image frame. For example, sub-data of the detection model output data may include position information. In this case, the position information may incorporate the location within the image frame of any one of a predetermined number of regions corresponding to the sub-data. The position information may be represented by a bounding box. Examples of bounding box representations include, but are not limited to, Pascal VOC and COCO.

[0178] The detection model output data may include probability information regarding medication administration related objects and poses. For example, sub-data of the detection model output data may include probability information.

[0179] The probability information may include presence probability information or class probability information or both. The presence probability information may incorporate whether a drug administration related object or a drug administration related pose or both are present. The class probability information may incorporate a probability of a class of a drug administration related object or a drug administration related pose or both.

[0180] In some embodiments, the classes of the drug administration-related objects and drug administration-related poses may refer to types of the drug administration-related objects and drug administration-related poses. For example, if a first drug administration-related object and pose is of a different type than a second drug administration-related object and pose, the first drug administration-related object and pose is of a different class than the second drug administration-related object and pose. As a more specific example, a pill bottle and a pill may be of different classes. As another more specific example, a pill bottle and an inhaler may be of different classes.

[0181] In some embodiments, a class of drug administration-related objects and poses may refer to a group of drug administration-related objects and poses of the same or similar type. For example, if a first drug administration-related object and pose and a second drug administration-related object and pose are of the same or similar type, the first drug administration-related object and pose and the second drug administration-related object and pose are of the same or similar class. As a more specific example, a pill and a medicine bottle may be of the same class. However, even in this case, a medicine bottle and an inhaler may be of different classes.

[0182] The class probability information may include object class probability information corresponding to drug administration related objects, or posture class probability information corresponding to drug administration related postures, or both.

[0183] The probability information may include one or more probability values. The existence probability information may include one or more existence probability values. The class probability information may include one or more class probability values. The number of probability values ​​included in the probability information may vary depending on the case.

[0184] The server may detect a medication-administration related object or a medication-administration related pose or both included in the image frame based on the detection model output data.The server may detect a medication-administration related object or a medication-administration related pose or both included in the image frame based on sub-data of the detection model output data.

[0185] The server may detect a drug administration-related object or a drug administration-related posture or both included in the image frame based on the probability information of the subdata. For example, the server may detect a drug administration-related object or a drug administration-related posture or both by determining whether or not the drug administration-related object or the drug administration-related posture or both exist in a region corresponding to the presence probability information based on the presence probability information. As a more specific example, if the presence probability value is equal to or greater than a predetermined value, the server may determine that the drug administration-related object or the drug administration-related posture or both exist in a region corresponding to the presence probability value, and detect the drug administration-related object or the drug administration-related posture or both. As another example, the server may estimate a class of the drug administration-related object or the drug administration-related posture or both expected to exist in a region corresponding to the class probability information based on the class probability information. As a more specific example, if the class probability value is equal to or greater than a predetermined value, the server may estimate that the class of the drug administration-related object or the drug administration-related posture or both expected to exist in a region corresponding to the class probability value is the class corresponding to the class probability value.

[0186] FIG. 13 is a diagram illustrating an example of detection model output data.

[0187] As an example of detection model output data, FIG. 13 shows a case where the server acquires data of three regions 51, 52, and 53 in an image frame 50. Specifically, the server may acquire data corresponding to a region 51 corresponding to a pill bottle, which is a drug administration-related object, a region 52 corresponding to a posture for holding a pill, which is a drug administration-related posture, and a region 53 not related to the drug administration-related object and posture. As shown in FIG. 13, the predetermined number of regions 51, 52, and 53 acquired by the server may include the regions 51 and 52 corresponding to the drug administration-related object or the drug administration-related posture or both, and the region 53 not related thereto. However, compared with the regions 51 and 52 corresponding to the drug administration-related object or the drug administration-related posture or both, the probability value of the not related region 53 may be low. Therefore, as described above, the server may determine the regions 51 and 52 corresponding to the drug administration-related object or the drug administration-related posture or both among the regions 51, 52, and 53 based on the probability value, and may detect the drug administration-related object or the drug administration-related posture or both.

[0188] FIG. 13(b) shows a case where the detection model output data 61 includes three sub-data 62, 63, and 64 corresponding to the three regions 51, 52, and 53, respectively.

[0189] FIG. 13B illustrates, as an example of the presence probability information, presence probability information 72 incorporating the probability that a drug administration related object or a drug administration related posture or both are present within the bounding boxes 51, 52, and 53. In FIG. 13B, the subdata 62, 63, and 64 each include, but are not limited to, one presence probability value, p1, p2, and p3. FIG. 13B also illustrates, as an example of the class probability information, class probability information 73 incorporating the probability of a class of a drug administration related object or a drug administration related posture or both expected to be present within the bounding boxes 51, 52, and 53. The class probability information 73 includes object class probability information 74 and posture class probability information 75. In (b) of FIG. 13, each of the sub-data 62, 63, and 64 includes, but is not limited to, "a" object class probability values ​​as object class probability information 74, "b" posture class probability values ​​as posture class probability information 75, for a total of "a+b" class probability values.

[0190] The detection model output data may have a specific format. For example, referring to (b) of FIG. 13, the detection model output data 61 may have a 2D matrix format. Here, the rows of the detection model output data 61 may correspond to one of the sub-data 62, 63, and 64, respectively. Also, the columns of the detection model output data 61 may correspond to a specific type of information, such as position information and probability information, included in the sub-data 62, 63, and 64, respectively. However, the format of the detection model output data 61 shown in (b) of FIG. 13 is merely an example and is not limited thereto.

[0191] Although Fig. 13 shows a case where the server acquires data of three regions 51, 52, and 53 in an image frame 50, and the detection model output data 61 includes three sub-data 62, 63, and 64, the number of regions acquired by the server may be different. For example, the server may acquire data of k regions. In this case, the detection model output data may include k sub-data. According to the format shown in Fig. 13(b), the detection model output data may be represented as a 2D matrix with k rows.

[0192] Also, not all of the elements shown in Fig. 13 are necessarily required elements of the detection model output data. The detection model output data may further include elements not shown in Fig. 13. Furthermore, at least some of the elements of the detection model output data shown in Fig. 13 may be omitted. For example, the detection model output data may not include information regarding a bounding box. Alternatively, the detection model output data may not include presence probability information. Alternatively, the detection model output data may not include object class probability information or pose class probability information, or both.

[0193] Examples of detection models include, but are not limited to, deep learning models such as R-CNN, Fast R-CNN, Faster R-CNN, SPPNet, YOLO, and SSD.

[0194] The detection model may include multiple sub-detection models, which may detect at least partially different classes of drug administration related objects and / or drug administration related poses for the model.

[0195] For example, the detection model may include a first detection model for detecting a first type of drug administration related object and a first type of drug administration related posture, and a second detection model for detecting a second type of drug administration related object and a second type of drug administration related posture.

[0196] As another example, the detection model may include a detection model for detecting objects related to medication administration (hereinafter referred to as the "object detection model") and a detection model for detecting postures related to medication administration (hereinafter referred to as the "posture detection model").

[0197] FIG. 14 illustrates a detection model including an object detection model and a pose detection model, according to one embodiment. Referring to FIG. 14, the server may input image frames 81 and 82 of the video 80 into the object detection model to detect drug administration related objects included in the image frames 81 and 82. The server may also input image frames 81 and 82 of the video 80 into a pose detection model to detect drug administration related poses included in the image frames 81 and 82. In this case, the object detection model and the pose detection model are used to detect drug administration related objects and poses in all image frames of the video 80, thereby detecting drug administration related objects and poses included in the video 80. The image frames 81 and 82 shown in FIG. 14 may be input into the same object detection model. The image frames 81 and 82 shown in FIG. 14 may be input into the same pose detection model.

[0198] Examples of sub-detection models include, but are not limited to, deep learning models such as R-CNN, Fast R-CNN, Faster R-CNN, SPPNet, YOLO, and SSD.

[0199] Returning to FIG. 8, in step S230, the server may use the detection result and the confirmation model to determine whether the medication has been administered. For example, the server control unit may use the detection result and the confirmation model to determine whether the medication has been administered. As another example, the server control unit may store the detection result in the server storage unit, and may use the detection result and the confirmation model stored in the server storage unit to determine whether the medication has been administered. The detection result may include detection model output data. The detection result may be obtained in step S220.

[0200] The server may input the detection result directly into the confirmation model. In this case, the input data input into the confirmation model may be the same as the detection result of the detection model. Alternatively, the server may convert the detection result and input the obtained detection result into the confirmation model. In this case, the input data input into the confirmation model may be generated based on the detection result.

[0201] FIG. 15 is a diagram illustrating the use of a confirmation model to determine whether a drug has been administered, according to one embodiment.

[0202] 15, the server may input the confirmation model input data into the confirmation model to determine whether the medication has been administered. In this case, the input data of the confirmation model may be the confirmation model input data, and the output data may be a result of determining whether the medication has been administered.

[0203] The server may generate confirmation model input data based on the detection result. Referring to FIG. 15, the detection result may include detection model output data corresponding to each image frame of the video. If the video includes N image frames, the detection result includes N detection model output data. Here, the server may generate the confirmation model input data by combining the N detection model output data.

[0204] The validation model input data may incorporate whether a particular class of medication-administration related object and pose is present in the video, or alternatively, the validation model input data may incorporate the movement and pose of a particular class of medication-administration related object in the video over time.

[0205] 16 illustrates detection model output data and confirmation model input data according to one embodiment, assuming the detection model outputs data for k regions and the video contains N image frames, where (a) in FIG. 16 illustrates an example of detection model output data and (b) in FIG. 16 illustrates an example of confirmation model input data.

[0206] The confirmation model input data may include sub-data corresponding to each image frame. For example, referring to FIG. 16(b), the confirmation model input data 92 may include N sub-data 93 corresponding to the N image frames, respectively. When comparing the detection model output data and the confirmation model input data, the sub-data of the detection model output data may correspond to a particular region within an image frame, and the sub-data of the confirmation model input data may correspond to a particular image frame.

[0207] The validation model input data may include sub-data arranged in chronological order, for example, the validation model input data may include sub-data arranged in ascending chronological order.

[0208] The sub-data of the confirmation model input data may correspond to the detection model output data. For example, referring to Figure 16, each sub-data of the confirmation model input data 93 may correspond to one detection model output data 91. That is, the detection model output data 91 may only include information about one image frame, while the confirmation model input data 92 may include information about multiple image frames (e.g. all image frames of a video). The sub-data of the confirmation model input data may be generated using the detection model output data corresponding to it.

[0209] The sub-data of the validation model input data may incorporate whether a drug administration related object or a drug administration related posture or both are present in the image frame. For example, referring to FIG. 16, the sub-data of the validation model input data 93 may include information 94 incorporating whether a drug administration related object or a drug administration related posture or both of a particular class are present in the image frame. The information 94 may be generated by combining the class probability information 95 of the detection model output data. Specifically, the information 94 corresponding to a particular class may be generated by combining the particular class probability information 95 of the detection model output data. Examples of generating the information 94 corresponding to a particular class by combining the particular class probability information 95 include adding the probability values ​​of the particular class probability information 95 and determining the sum as the information 94, determining a weighted average of the probability values ​​as the information 94, and determining a maximum value of the probability values ​​as the information 94. However, examples are not limited thereto.

[0210] The confirmation model input data may have a specific format. For example, referring to (b) of FIG. 16, the confirmation model input data 92 may have a 2D matrix format. Here, each row of the confirmation model input data 92 may correspond to one sub-data 93. Also, each column of the confirmation model input data 92 may correspond to a specific type of information, such as probability information, included in the sub-data 93. However, the format of the confirmation model input data 92 shown in (b) of FIG. 16 is merely an example and is not limited thereto.

[0211] Returning to FIG. 15, the result of determining whether or not a drug has been administered, output from the confirmation model, may be an index indicating whether or not a drug has been administered. That is, the output data of the confirmation model may be an index indicating whether or not a drug has been administered. Details of the index indicating whether or not a drug has been administered, which is the output data of the monitoring model, may be applied to the index indicating whether or not a drug has been administered, which is the output data of the confirmation model, and therefore the description thereof will be omitted.

[0212] Examples of validation models include, but are not limited to, deep learning models such as RNN models, i.e., LSTM, and hybrid models, i.e., 2D CNN-LSTM.

[0213] When the server determines whether a drug has been administered using confirmation model input data generated based on a video, accuracy may be improved compared to when the server determines whether a drug has been administered using confirmation model input data generated based on a single image frame or a few image frames. The reason for this is the same as or similar to the reason for the improvement in accuracy when the server determines whether a drug has been administered based on a video, compared to when the server determines whether a drug has been administered based on a single image frame or a few image frames, as described in "2.2 Method for determining whether a drug has been administered using a supervisory model." Therefore, the explanation of the reason is omitted.

[0214] Although the use of drug administration-related objects and drug administration-related postures in determining whether a drug has been administered has been described, whether a drug has been administered may be determined using only one of the drug administration-related objects or the drug administration-related postures. However, compared to using only one of the drug administration-related objects or the drug administration-related postures, when both the drug administration-related objects and the drug administration-related postures are used, the accuracy of determining whether a drug has been administered is improved. For example, compared to when the confirmation model input data includes information on only one of the drug administration-related objects or the drug administration-related postures, when the confirmation model input data includes information on both the drug administration-related objects and the drug administration-related postures, the accuracy of determining whether a drug has been administered using the confirmation model is improved. For this reason, it is more advantageous for the detection model to detect both the drug administration-related objects and the drug administration-related postures than to detect only one of the drug administration-related objects or the drug administration-related postures.

[0215] 2.4 Method for determining whether a drug was administered, given the category of the video

[0216] The server may define a category of the video and determine whether a medication has been administered given the category. For example, the server may use a particular supervision model depending on the category of the video to determine whether a medication has been administered. As another example, the server may use a particular detection model depending on the category of the video to determine whether a medication has been administered. As yet another example, the server may use a particular confirmation model depending on the category of the video to determine whether a medication has been administered.

[0217] Defining a category for a video may refer to determining a type of medication administration that corresponds to the video. For example, the server may define a category for a video by determining to which of two or more categories the video corresponds, the category corresponding to different types of medication administration. In this case, example categories include, but are not limited to, a medication swallowing category, an inhaler use category, a turbuhaler use category, a nasal spray use category, an eye drop administration category, and a medication injection category.

[0218] The server may use a classification model to define a category for the video. In this case, the server may further use the classification model to determine whether a medication has been administered. For example, the server may use the classification model in addition to the supervision model to determine whether a medication has been administered. As another example, the server may use the classification model in addition to the detection model and the confirmation model to determine whether a medication has been administered.

[0219] 17 illustrates a method for determining whether a medication has been administered using a classification model, according to one embodiment. A server may obtain a video (S310), may define categories for the video using the video and a classification model (S320), and may determine whether a medication has been administered using the defined categories and the video (S330).

[0220] In step S310, the server may obtain a video. For example, the server may receive a video from the wearable device, where the video may be obtained by a camera unit of the wearable device and transmitted via a wearable device communication unit. The video may include a plurality of image frames. The number of the plurality of image frames may be 50 or more, 100 or more, 200 or more, 400 or more, 600 or more, 800 or more, or 1000 or more. The entire drug administration process may be captured by video. Alternatively, a large portion of the entire drug administration process may be captured by video.

[0221] In step S320, the server may use the video and the classification model to define a category for the video. For example, the server control unit may use the video received via the server communication unit and the classification model to define a category for the video. As another example, the server control unit may store the video received via the server communication unit in a server storage unit, and may use the video stored in the server storage unit and the classification model to define a category for the video. The video may be obtained in step S310.

[0222] The server using the classification model to define a category of a video may mean that the server uses the classification model to obtain an index (hereinafter referred to as a "category index") that indicates the category of the video. As a non-limiting example, the category index may be expressed as a numerical value.

[0223] 18 to 20 are diagrams illustrating a classification model according to one embodiment.

[0224] Referring to FIG. 18, the server may input the video 100 into the classification model and define the category 100 of the video. For example, the server control unit may input the video 100 received through the server communication unit into the classification model and define the category 100 of the video. As another example, the server control unit may store the video 100 received through the server communication unit in the server storage unit, and may input the video 100 stored in the server storage unit into the classification model and define the category 100 of the video. Hereinafter, the classification model that receives the video and defines the category of the video is referred to as a first type of classification model. In this case, the input data of the first type of classification model may be a video.

[0225] The video input to the first type of classification model may have a particular format. For example, the video may include a predetermined number of image frames. As another example, the video may have a predetermined resolution. As yet another example, the video may have a predetermined ratio (width to height).

[0226] The server may input the video to the first type of classification model without performing pre-processing. For example, the server control unit may input the video received via the server communication unit to the first type of classification model without performing pre-processing. That is, the video input to the first type of classification model may be the same as the video captured by the camera unit. Alternatively, the server may perform pre-processing on the video and input the pre-processed video to the first type of classification model. For example, the server control unit may perform pre-processing on the video received via the server communication unit and input the pre-processed video to the first type of classification model. That is, the video input to the first type of classification model may be the result of performing pre-processing on the video captured by the camera unit. Examples of pre-processing include, but are not limited to, normalization, resizing, cropping, noise removal, and combinations thereof.

[0227] The format of the output data of the first type of classification model may vary.

[0228] For example, the output data of the first type of classification model may include a plurality of indices representing different categories, where the first type of classification model may output one of the plurality of indices, and the server may determine the outputted indices as category indices for the video.

[0229] As another example, the output data of the first type of classification model may include a plurality of indexes indicating the probability that a category is estimated. Hereinafter, the index indicating the probability that a category is estimated is referred to as a category probability index. The plurality of category probability indexes may correspond to different categories. Herein, the first type of classification model may output a plurality of category probability indexes. In this case, the server may define a category of the video using the plurality of category probability indexes. That is, the server may determine a category index of the video using the plurality of category probability indexes. For example, the server may define a category of the video as a category corresponding to a maximum value among the plurality of category probability indexes.

[0230] The first type of classification model may be realized with a structure that is the same as or similar to the structure of the above-mentioned monitoring model. However, even in this case, the first type of classification model and the monitoring model may be the same or similar in structure only, but parameters such as weighting and filter values ​​may be different. Furthermore, the first type of classification model is not necessarily realized with a structure that is the same as or similar to the structure of the monitoring model.

[0231] Referring to FIG. 19, the server may input a video 101 including image frames 102 and 103 into a classification model, detect a drug administration-related object or a drug administration-related posture or both included in the image frames 102 and 103, and define a category 101 of the video based on the detection result. In this case, similar to the above detection model, the classification model shown in FIG. 19 (hereinafter referred to as a "second type classification model") may receive image frames and detect a drug administration-related object or a drug administration-related posture or both included in the image frames. Therefore, the second type classification model may be implemented with a structure that is the same as or similar to that of the above detection model, and the details of the above detection model may be applied to the second type classification model. For example, the second type classification model output data, which is the output data of the second type classification model, and the above detection model output data may have the same or similar format. The following mainly describes the differences between the second type classification model and the detection model.

[0232] The types of medication administration related objects and poses detected by the second type of classification model may include a different type than the types of medication administration related objects and poses detected by the detection model.The types of medication administration related objects and poses detected by the detection model may include a different type than the types of medication administration related objects and poses detected by the second type of classification model.

[0233] The second type of classification model may detect medication-administration related objects, and the detection model may detect medication-administration related objects and medication-administration related postures. That is, the second type of classification model may not detect medication-administration related postures. Here, the type of medication-administration related object detected by the second type of classification model may be different from the type of medication-administration related object detected by the detection model.

[0234] For example, the types of medication-administration related objects detected by the second type of classification model may be more diverse than the types of medication-administration related objects detected by the detection model. As a specific example, the medication-administration related objects detected by the second type of classification model may be related to multiple types of medication administration, and the medication-administration related objects detected by the detection model may be related to a specific type of medication administration among the multiple types. As a more specific example, the second type of classification model may detect medication-administration related objects such as a pill bottle, an inhaler, a turbuhaler, and an eye dropper that are related to multiple types of medication administration, and the detection model may detect a specific medication-administration related object, such as an inhaler, that is related to a specific type of medication administration, such as using an inhaler, among the multiple types. In this case, the detection model may detect one or more additional medication-administration related objects that the second type of classification model does not detect. For example, the detection model may detect a body part in addition to an inhaler.

[0235] The second type of classification model and the detection model may have the same or similar structure, but may have different parameters such as weights and filter values. The second type of classification model is not necessarily realized with the same or similar structure as the detection model.

[0236] The server may define a category for the video using the medication administration related objects and poses detected by the second type of classification model, in which case the server may use the second type of classification model output data to determine a category index for the video.

[0237] Medication administration related objects and poses may have categories corresponding to them.

[0238] The category of the drug administration related object and posture may be determined according to the type of drug administration related to the drug administration related object and posture. Drug administration related objects and postures related to the same or similar types of drug administration may correspond to the same category. Drug administration related objects and postures related to different types of drug administration may correspond to different categories. For example, with reference to Figs. 11 and 12, drug administration related objects and postures such as pill bottle 21, pill 25, and posture 36 for holding the pill may correspond to a category related to swallowing the drug. Drug administration related objects and postures such as inhaler 26 and posture 32 for holding the inhaler may correspond to a category related to using the inhaler. Drug administration related objects and postures such as turbuhaler 24 and posture 33 for holding the turbuhaler may correspond to a category related to using the turbuhaler. Drug administration related objects and postures such as eye dropper 22 and posture 34 for holding the eye dropper may correspond to a category related to administering eye drops. Medication-administration related objects and poses, such as the nasal sprayer 23 and the pose for holding the nasal sprayer 35, may correspond to categories related to using the nasal sprayer. This description of categories of medication-administration related objects and poses is by way of example only and is not intended to be limiting.

[0239] In some embodiments, the server may define a category of the video based on the number of image frames in which the drug administration-related object or the drug administration-related posture or both are detected. In this case, the server may define a category index of the video based on the number of image frames in which the drug administration-related object or the drug administration-related posture or both are detected. For example, the server may calculate the number of image frames in which the drug administration-related object or the drug administration-related posture or both are detected corresponding to a particular category, and may define a category of the video based on the calculated number. Specifically, the server may compare the number of image frames in which the drug administration-related object or the drug administration-related posture or both are detected corresponding to a first category with the number of image frames in which the drug administration-related object or the drug administration-related posture or both are detected corresponding to a second category, and may define the category of the video as either the first category or the second category. Here, the server may define the category of the video as the category with a large number of image frames, but is not limited thereto.

[0240] In some embodiments, the server may define a category of the video based on the number of detected drug administration related objects or drug administration related postures or all of them. In this case, the server may determine a category index of the video based on the number of drug administration related objects or drug administration related postures or all of them. For example, the server may calculate the number of drug administration related objects or drug administration related postures or all of them corresponding to a particular category in all image frames, and may define a category of the video based on the calculated number. Specifically, the server may compare the number of drug administration related objects or drug administration related postures or all of them corresponding to a first category in all image frames with the number of drug administration related objects or drug administration related postures or all of them corresponding to a second category in all image frames, and may define the category of the video as either the first category or the second category. Here, the server may define the category of the video as a category with a large number of objects or postures or all of them, but is not limited thereto. In addition, when the number of drug administration related objects is calculated, different parts of the same object may be considered as one object or different objects. For example, the cap and body of a medicine bottle may be considered as one object and may be counted as one medication administration related object, or may be considered as two objects and may be counted as two medication administration related objects. In particular, when different parts of a medication administration related object are physically separated, for example when the cap and body of a medicine bottle are separated, the parts may be considered and counted as different objects.

[0241] In some embodiments, referring to FIG. 20 , the server may use the second type of classification model output data and the third type of classification model to define a category of the video. For example, the server control unit may use the second type of classification model output data and the third type of classification model to define a category of the video. As another example, the server control unit may store the second type of classification model output data in the server storage unit, and may use the second type of classification model output data and the third type of classification model stored in the server storage unit to define a category of the video. The output data of the third type of classification model may be a category index of the video.

[0242] The server may input the second type of classification model output data directly to the third type of classification model. In this case, the input data input to the third type of classification model may be the same as the second type of classification model output data. Alternatively, the server may convert the second type of classification model output data and input the resulting data to the third type of classification model. In this case, the input data input to the third type of classification model may be generated based on the second type of classification model output data. The details of generating the confirmation model input data based on the detection result of the above-mentioned detection model may also be applied to generating the input data of the third type of classification model based on the second type of classification model output data, and therefore a detailed description thereof will be omitted.

[0243] The third type of classification model may be realized with a structure that is the same as or similar to the structure of the above-mentioned confirmation model. However, even in this case, the third type of classification model and the confirmation model may have the same or similar structure only, but parameters such as weights and filter values ​​may be different. Furthermore, the third type of classification model is not necessarily realized with a structure that is the same as or similar to the structure of the confirmation model.

[0244] 17, in step S330, the server may use the defined categories and the video to determine whether the medication has been administered. The categories may be the result of the classification in step S320. The video may be obtained in step S310.

[0245] 21 and 22 are diagrams illustrating the use of classification and categories derived from a video to determine whether a medication has been administered, according to one embodiment. FIG. 21 illustrates the use of a supervision model, as in "2.2 Method for determining whether a medication has been administered using a supervision model." FIG. 22 illustrates the use of a detection model and a confirmation model, as in "2.3 Method for determining whether a medication has been administered using a detection model and a confirmation model."

[0246] Referring to FIG. 21, the server may use a specific monitoring model for each category of video to determine whether a drug has been administered. In this case, different monitoring models may determine whether a different type of drug administration has been performed. For example, a first monitoring model corresponding to a first category may determine whether a first type of drug administration corresponding to the first category has been performed. A second monitoring model corresponding to a second category may determine whether a second type of drug administration corresponding to the second category has been performed. As a specific example, the first type of drug administration may be any one of swallowing a drug, using an inhaler, using a turbuhaler, using a nasal spray, administering eye drops, and injecting a drug. The second type of drug administration may be any one of swallowing a drug, using an inhaler, using a turbuhaler, using a nasal spray, administering eye drops, and injecting a drug. This is not a limitation. The video input to the monitoring model may be the same as the video input to the classification model, but is not limited thereto.

[0247] Referring to FIG. 22, the server may use a specific detection model and a specific confirmation model for each category of the video to determine whether a drug has been administered. In this case, different detection models may detect different types of drug administration-related objects and postures. For example, a first detection model corresponding to a first category may detect a first type of drug administration-related object and posture corresponding to the first category, and a second detection model corresponding to a second category may detect a second type of drug administration-related object and posture corresponding to the second category. Also, different confirmation models may determine whether different types of drug administration have been performed. For example, a first confirmation model corresponding to a first category may determine whether a first type of drug administration has been performed based on a first type of drug administration-related object and posture corresponding to the first category. A second confirmation model corresponding to a second category may determine whether a second type of drug administration has been performed based on a second type of drug administration-related object and posture corresponding to the second category. That is, the server may use a classification model to define a category of the video, may use a detection model corresponding to the defined category to detect a specific type of drug administration-related object and posture, and may use a confirmation model corresponding to the defined category to determine whether a specific type of drug administration has been performed. The video input to the detection model may be, but is not limited to, the same video input to the classification model.

[0248] Although Figure 22 shows that the detection model and the verification model are different for each category of videos, the detection model or the verification model may be the same. For example, the detection model may be different for each category of videos, but the verification model may be the same. As another example, the detection model may be the same for each category of videos, but the verification model may be different.

[0249] Also, although FIGS. 21 and 22 show a case where the videos are classified into k categories, three or more categories, the videos may be classified into two categories.

[0250] Although the server defines the category of the video using the classification model, the method of defining the category of the video is not limited thereto. For example, the server may define the category of the video based on data indicating the category of the video. Specifically, the server may receive data from an external device such as a wireless communication device or a wearable device, and may define the category of the video. In this case, the server may receive data from an external device such as a wireless communication device or a wearable device, and may determine a category index of the video.

[0251] The above method for determining whether a drug has been administered by combining a monitoring model, a detection model, a confirmation model, and a classification model is merely an example. Whether a drug has been administered may be determined by combining such models other than the above example. Also, whether a drug has been administered may be determined by combining at least some of the above models with other models.

[0252] 2.5 Model training method

[0253] A method for training the above model will be described below, taking the case where the model is a deep learning model as an example, but is not limited to this.

[0254] FIG. 23 is a diagram illustrating a method of training a model according to an embodiment, illustrating supervised learning. The server may use training data to train the model. The training data may include input data and label data. The input data and label data may correspond to each other. The server may input the input data into the untrained model to obtain output data, and compare the output data with the label data to train the model through backpropagation. The server that trains the model may be the same as the server that uses the model to determine whether the drug has been administered, or may be a specific training server for training the model.

[0255] The server may prepare the training data. For example, the server may generate the training data. As a specific example, the server may receive input data and generate label data based on the input data. As another example, the server may obtain the training data from an external source. As a specific example, the server may receive training data generated by an external device or training data generated by a person.

[0256] The model may be trained using a variety of training techniques, such as Optimizer Adam, SGD Scheduler, Random Contrast, Random Hue, Random Brightness, Random Flip left right, Random Crop, and Random Resize.

[0257] 2.5.1 How to train a supervised model

[0258] The server may train the surveillance model. The server may train the surveillance model using the surveillance model training data for training the surveillance model.

[0259] The surveillance model training data may include a training video and surveillance model label data for the training video, the surveillance model label data indicating whether a drug has been administered. The format of the training video may be the same as the format of the video input to the surveillance model when determining whether a drug has been administered. Also, the format of the surveillance model label data may be the same as the format of the index indicating whether a drug has been administered.

[0260] The training videos may include drug administration training videos and no drug administration training videos.

[0261] The surveillance model label data may include a label corresponding to the training video of drug administration and indicating drug administration, and a label corresponding to the training video of no drug administration and indicating no drug administration. Alternatively, the surveillance model label data may include a label indicating the probability that drug administration is estimated, and a label indicating the probability that no drug administration is estimated.

[0262] 23, the input data, label data, and output data shown in FIG. 23 may be a training video, surveillance model label data, and an index indicating whether a drug has been administered, respectively. The server may input the training video to a surveillance model that has not yet completed training to obtain an index indicating whether a drug has been administered, and may compare the index indicating whether a drug has been administered with the surveillance model label data to train the surveillance model through error backpropagation.

[0263] 2.5.2 How to train a detection model

[0264] The server may train the detection model. The server may train the detection model using the detection model training data for training the detection model.

[0265] The detection model training data may include training image frames and detection model label data, the detection model label data relating to medication-administration related objects and / or medication-administration related poses included in the training image frames. The format of the training image frames may be the same as the format of the image frames input to the detection model described above when determining whether medication has been administered. Also, the format of the detection model label data may be the same as the format of the detection model output data described above.

[0266] The detection model label data may include label data of objects related to medication administration (hereinafter referred to as "object label data") and label data of postures related to medication administration (hereinafter referred to as "posture label data").

[0267] The object label data may include labels for various types of medication-related objects used by a user to administer medication. By way of non-limiting example, the object label data may include a medicine bottle label, an eye dropper label, a nasal spray label, a turbuhaler label, a pill label, an inhaler label, a blister pack label, and a syringe label.

[0268] The object label data may include labels for body parts. As a non-limiting example, the object label data may include an arm label, a hand label, a wrist label, a palm label, an outer palm label, a face label, a mouth label, and a nose label.

[0269] The posture label data may include various types of medication administration related postures, such as positioning of a user for medication administration. As a non-limiting example, the posture label data may include posture labels for holding medication administration related objects, such as posture labels for holding a blister pack, posture labels for holding an inhaler, posture labels for holding a turbuhaler, posture labels for holding an eye dropper, posture labels for holding a nasal sprayer, and posture labels for holding a pill, posture labels for swallowing medication, such as a pill, posture labels for inhaling medication using a medication administration related object, such as an inhaler, a turbuhaler, and a nasal sprayer, posture labels for administering eye drops using a medication administration related object, such as an eye dropper, posture labels for injecting medication using a medication administration related object, such as a syringe, and posture labels for opening a medication administration related object, such as a pill bottle.

[0270] A more detailed example of the detection model training data is provided below.

[0271] 24 to 28 are diagrams illustrating detection model training data according to one embodiment, conceptually illustrating detection model label data in training image frames.

[0272] FIG. 24 is a diagram showing an example of detection model training data related to swallowing medicine using a blister pack among medical supplies. Referring to FIG. 24, the object label data may include labels for the body, such as an arm label 110a, a palm label 110b, an outer palm part label 110c, and a face label (or a mouth-nose label) 110d, and labels for blister packs, such as a blister pack top label 110e, a blister pack bottom label 110f, a blister pack side label 110g, and a blister pack box label 110h. The posture label data may include a posture label 110i for holding a blister pack, a posture label 110j for holding a pill, and a posture label 110k for swallowing a pill.

[0273] Fig. 25 is a diagram showing an example of detection model training data related to inhalation of medicine using an inhaler among medical supplies. Referring to Fig. 25, the object label data may include body labels such as an arm label 111a, a wrist label 111b, and a face label (or a mouth-nose label) 111c, and labels for an inhaler such as an inhaler label 111d, an open inhaler label 111e, and an inhaler cap label 111f. The posture label data may include a posture label 111g for holding an inhaler, a posture label 111h for opening an inhaler, and a posture label 111i for inhaling with an inhaler.

[0274] Fig. 26 is a diagram showing an example of detection model training data related to inhalation of medicine using a turbulent, among medical supplies. Referring to Fig. 26, the object label data may include body labels such as an arm label 112a, a wrist label 112b, and a face label (or a mouth-nose label) 112c, and labels for the turbulent, such as a turbulent label 112d, an open turbulent label 112e, and a turbulent cap label 112f. The posture label data may include a posture label 112g for holding the turbulent.

[0275] Fig. 27 is a diagram showing an example of detection model training data related to administering eye drops using an eye dropper, which is one of medical supplies. Referring to Fig. 27, the object label data may include body labels such as an arm label 113a, a wrist label 113b, and a face label (or a mouth-nose label) 113c, and labels for eye droppers such as an eye dropper label 113d, an open eye dropper label 113e, and an eye dropper cap label 113f. The posture label data may include a posture label 113g for holding the eye dropper and a posture label 113h for administering the eye dropper.

[0276] Fig. 28 is a diagram showing an example of detection model training data related to inhalation of medicine using a nasal spray, among medical supplies. Referring to Fig. 28, the object label data may include body labels such as arm label 114a, wrist label 114b, and face label (or mouth-nose label) 114c, and labels for nasal sprayers such as nasal sprayer label 114d, open nasal sprayer label 114e, and nasal sprayer cap label 114f. The posture label data may include posture labels for holding the nasal sprayer 114g and posture labels for spraying with the nasal sprayer 114h.

[0277] 24-28, depending on the type of drug administration or the type of medical supplies, the detection model label data associated therewith may include different labels. For example, the detection model label data associated with swallowing a drug using a blister pack may include labels 110e, 110f, 110g, and 110h for a blister pack, a label 110i for a posture for holding a blister pack, a label 110j for a posture for holding a pill, and a label 110k for a posture for swallowing a pill. The detection model label data associated with inhaling a drug using an inhaler may include labels 111d, 111e, and 111f for an inhaler, a label 111g for a posture for holding an inhaler, a label 111h for a posture for opening an inhaler, and a label 111i for a posture for inhaling with an inhaler.

[0278] Detection model label data associated with different types of medication or medical supplies may include the same labels. For example, all medications shown in Figures 24-28 include arm labels 110a, 111a, 112a, 113a, and 114a, and face labels 110d, 111c, 112c, 113c, and 114c.

[0279] The medication-administration related objects and / or medication-administration related poses detected by the detection model may differ from the detection model training data used when the detection model is trained.

[0280] Depending on the type of drug administration determined by the server, the detection model training data used when training the detection model may be different. For example, in the case of a detection model used to determine whether drug administration using a blister pack has been performed, the server may train the detection model using the detection model training data as shown in FIG. 24. In the case of a detection model used to determine whether drug administration using a blister pack has been performed and whether drug inhalation using an inhaler has been performed, the server needs to train the detection model using the detection model training data as shown in FIG. 24 and the detection model training data as shown in FIG. 25. That is, the more types of drug administration determined, the more types of detection model training data the server uses to train the detection model.

[0281] The detection model training data shown in Figures 24 to 28 are merely examples, and the detection model training data is not limited thereto. The detection model training data may further include labels not shown in Figures 24 to 28. In addition, at least some of the labels of the detection model training data shown in Figures 24 to 28 may be omitted.

[0282] Returning to Fig. 23, the input data, label data, and output data shown in Fig. 23 may be training image frames, detection model label data, and detection model output data, respectively. The server may input training image frames into an untrained detection model to obtain detection model output data, and compare the detection model output data with the detection model label data to train the detection model through error backpropagation.

[0283] 2.5.3 How to train a confirmation model

[0284] The server may train the validation model. The server may train the validation model using the validation model training data for training the validation model.

[0285] The confirmation model training data may include training confirmation model input data and confirmation model label data corresponding to the training confirmation model input data and indicating whether a drug has been administered. The format of the training confirmation model input data may be the format of the confirmation model input data described above. Also, the format of the confirmation model label data may be the same as the format of the index indicating whether a drug has been administered described above.

[0286] The training confirmation model input data may include training confirmation model input data representing drug administration and training confirmation model input data representing drug non-administration.

[0287] The training validation model input data may be generated based on a video. The training validation model input data may incorporate drug administration related objects and / or drug administration related postures contained in the video. The training validation model input data representing drug administration may be generated based on a training video of drug administration. The training validation model input data representing no drug administration may be generated based on a training video of no drug administration.

[0288] The confirmed model label data may include a label indicating administration of a drug and a label indicating no administration of a drug. Alternatively, the confirmed model label data may include a label indicating the probability that administration of a drug is estimated and a label indicating the probability that no administration of a drug is estimated.

[0289] 23, the input data, label data, and output data shown in FIG. 23 may be the training validation model input data, the validation model label data, and an index indicating whether the drug has been administered, respectively. The server may input the training validation model input data to the validation model that has not yet been trained to obtain an index indicating whether the drug has been administered, and may compare the index indicating whether the drug has been administered with the validation model label data to train a supervision model through backpropagation.

[0290] 2.5.4 How to Train a Classification Model

[0291] The server may train the classification model.

[0292] For the first type of classification model, the server may train the first type of classification model using the first type of classification model training data for training the first type of classification model.

[0293] The first type of classification model training data may include a training video and a first type of classification model label data indicating a category of the training video (or indicating a probability that the category is estimated). The format of the training video may be the same as the format of the video input to the first type of classification model described above when determining whether a drug has been administered. Also, the format of the first type of classification model label data may be the same as the format of the index indicating the category (or an index indicating the probability that the category is estimated) described above.

[0294] The training videos may include multiple videos corresponding to different categories. As a non-limiting example, the training videos may include a video corresponding to a category of swallowing medication, a video corresponding to a category of using an inhaler, a video corresponding to a category of using a turbuhaler, a video corresponding to a category of administering eye drops, and a video corresponding to a category of using a nasal spray.

[0295] The training videos may include drug administration training videos and no drug administration training videos.

[0296] The first type of classification model label data may include a plurality of labels corresponding to different categories. For example, the first type of classification model label data may include a label corresponding to a category of swallowing medicine, a label corresponding to a category of using an inhaler, a label corresponding to a category of using a turbuhaler, a label corresponding to a category of administering eye drops, and a label corresponding to a category of using a nasal spray.

[0297] 23, the input data, label data, and output data shown in FIG. 23 may be a training video, a first type of classification model label data, and an index indicating a category (or an index indicating a probability that a category is estimated), respectively. The server may input a training video to a first type of classification model that has not yet completed training to obtain an index indicating a category (or an index indicating a probability that a category is estimated), and may compare the index indicating a category (or an index indicating a probability that a category is estimated) with the first type of classification model label data to train the first type of classification model through error backpropagation.

[0298] For the second type of classification model, the server may train the second type of classification model using the second type of classification model training data for training the second type of classification model.

[0299] The second type of classification model training data may include training image frames and second type of classification model label data, the second type of classification model label data relating to medication-administration-related objects or medication-administration-related postures or both included in the training image frames. The format of the training image frames may be the same as the format of the image frames input to the second type of classification model described above when determining whether medication has been administered. Also, the format of the second type of classification model label data may be the same as the format of the second type of classification model output data described above.

[0300] The second type of classification model label data may include object label data or pose label data, or both, as discussed above.

[0301] The second type of classification model label data may include labels of a different type from the labels of the types included in the detection model label data. The detection model label data may include labels of a different format from the labels included in the second type of classification model label data.

[0302] The second type of classification model label data may include object label data, and the detection model label data may include object label data and posture label data. That is, the second type of classification model label data may not include posture label data. Here, the object label data of the second type of classification model label data may include a label different from the type of label included in the object label data of the detection model label data. For example, the object label data of the second type of classification model label data may include a label related to multiple types of drug administration, and the object label data of the detection model label data may include a label related to a specific type of drug administration among the multiple types. As a specific example, the object label data of the second type of classification model label data may include labels such as a pill bottle label, an inhaler label, a turbulent label, and an eye dropper label, which are related to multiple types of drug administration, and the detection model label data may include an inhaler label, which is a label related to a specific type of drug administration using an inhaler. In this case, the object label data of the detection model label data may include an additional label not included in the object label data of the second type of classification model label data. For example, in addition to the inhaler label, the object label data of the detection model label data may include a label for a body part.

[0303] 23, the input data, label data, and output data shown in FIG. 23 may be training image frames, second type classification model label data, and second type classification model output data, respectively. The server may input the training image frames to the second type classification model that has not yet been trained to obtain the second type classification model output data, and compare the second type classification model output data with the second type classification model label data to train the second type classification model through error backpropagation.

[0304] For the third type of classification model, the server may train the third type of classification model using the third type of classification model training data for training the third type of classification model.

[0305] The third type of classification model training data may include third type of classification model input data for training and third type of classification model label data indicating a category corresponding to the third type of classification model input data for training (indicating a probability that the category is estimated). The format of the third type of classification model input data for training may be the same as the format of the input data input to the above-mentioned third type of classification model when it is determined whether or not a drug has been administered. Also, the format of the third type of classification model label data may be the same as the format of the output data of the above-mentioned third type of classification model.

[0306] The third type of classification model input data for training may include a plurality of input data corresponding to different categories. As a non-limiting example, the third type of classification model input data for training may include input data corresponding to a category of swallowing medicine, input data corresponding to a category of using an inhaler, input data corresponding to a category of using a Turbuhaler, input data corresponding to a category of administering eye drops, and input data corresponding to a category of using a nasal spray.

[0307] The third type of classification model input data for training may include input data representing drug administration and input data representing no drug administration.

[0308] The details of the above first type of classification model label data may be applied to the third type of classification model label data, so duplicated explanations will be omitted.

[0309] 23, the input data, label data, and output data shown in FIG. 23 may be a third type of classification model input data for training, a third type of classification model label data, and an index indicating a category (or an index indicating a probability that a category is estimated), respectively. The server may input the third type of classification model input data for training to a third type of classification model that has not yet been trained to obtain an index indicating a category (or an index indicating a probability that a category is estimated), and may compare the index indicating a category (or an index indicating a probability that a category is estimated) with the third type of classification model label data to train the third type of classification model through error backpropagation.

[0310] The above training methods are merely examples and are not limiting, and the model may be trained in other ways, such as unsupervised learning, reinforcement learning, and imitation learning.

[0311] 2.5.5 Model training sequences

[0312] The above models may be trained in conjunction with each other.

[0313] In some embodiments, when using the detection model and the confirmation model to determine whether a drug has been administered, the server may first train one of the two models and then train the other. For example, the server may first train the detection model and then train the confirmation model. As a specific example, the server may first train the detection model using the detection model training data, then connect the detection model and the confirmation model, and again train using the training image frames of the detection model training data. In this case, the data input to the confirmation model to train the confirmation model may be generated based on the detection model output data. Also, when the confirmation model is trained, the already trained detection model is retrained or not trained.

[0314] In some embodiments, when using a first type of classification model and a monitoring model to determine whether a medication has been administered, if the first type of classification model and the monitoring model are identical in structure, the server trains one of the two models first and then trains the other.

[0315] In some embodiments, when using a second type of classification model, a detection model, and a confirmation model to determine whether a medication has been administered, if the second type of classification model and the detection model are identical in structure, the server trains one of the two models first and then trains the other.

[0316] The above models may be trained independently. For example, when the detection model and the confirmation model are used to determine whether a drug has been administered, the server may train the detection model and the confirmation model independently. As a specific example, the server may train the detection model using the detection model training data, and separately, the server may train the confirmation model using the confirmation model training data. Here, either the detection model or the confirmation model may be trained first.

[0317] 2.6 Implementation and Results

[0318] 2.6.1 First embodiment

[0319] FIG. 29 is a diagram showing a first embodiment, which is an example of a method for determining whether a drug has been administered using a monitoring model, and relates to a result of determining whether a drug has been administered using a monitoring model. In FIG. 29, it was determined whether oral administration of a pill taken out of a pill bottle, which is swallowing of a drug among various types of drug administration, has been performed. As the monitoring model, a ResNet101-based I3D network and a ResNet50-based SlowFast network were used. In FIG. 29, a training video including 100 training image frames was used to train the monitoring model, and each of the training image frames had a resolution of 224x224 pixels. Referring to FIG. 29, when the I3D network was used as the monitoring model, the accuracy of determining whether a drug has been swallowed ranged from 77.8 to 81% depending on the training method. Also, when the SlowFast network was used as the monitoring model, the accuracy of determining whether a drug has been swallowed ranged from 74 to 80% depending on the training method. For the surveillance model, the accuracy was found to be up to about 80%, regardless of the type of network. However, the first embodiment is the result of training using about 1200 training videos. The accuracy shown in FIG. 29 may vary from case to case. For example, the accuracy may improve as training is performed or as the training video is changed.

[0320] 2.6.2 Second embodiment

[0321] 30 and 31 are diagrams showing a second embodiment, which is an example of a method for determining whether a drug has been administered using a detection model and a confirmation model, and relate to the result of determining whether a drug has been administered using the detection model and the confirmation model. In FIG. 30, it is determined whether oral administration of a pill taken out of a pill bottle, which is swallowing of a drug among various types of drug administration, has been performed. Version 2 relates to a case where additional training has been performed using training data supplemented after the initial training (version 1). As the detection model, ResNet101-based Faster R-CNN was used, and as the confirmation model, a combination of a 2D convolution layer and an LSTM was used. Referring to FIG. 31, the structure of the confirmation model will be described in more detail. The confirmation model input data input to the confirmation model is connected after passing through a 2D convolution layer and a maximum pooling layer, and passes through a ReLU, LSTM, and fully connected layer, and is output as a result of determining whether a drug has been administered. However, since the structure of the confirmation model shown in FIG. 31 is exemplary, the structure of the confirmation model is not limited thereto. For example, unlike the verification model in which a 2D convolutional layer, an LSTM, and a fully connected layer are designed in that order as shown in FIG. 31, the verification model may be designed in the order of an LSTM, a 2D convolutional layer, and a fully connected layer.

[0322] Returning to FIG. 30, the accuracy of the detection model was 85.1% for the initial training and 88.8% for the additional training. The accuracy of the confirmation model was 90.5% for the initial training and 94.5% for the additional training. It was found that the accuracy of the model improved when the training data was replenished. It was also found that when the detection model and the confirmation model were used, the accuracy of determining whether or not the drug was administered improved compared to the first embodiment in which the monitoring model was used as shown in FIG. 29. However, the accuracy shown in FIG. 30 may change depending on the case. For example, the accuracy may improve as training is performed, or as the training video is changed.

[0323] 2.6.3 Third embodiment

[0324] FIG. 32 is a diagram showing a third embodiment, which is an example of a method for determining whether a drug has been administered using a classification model, and relates to a result of determining whether a drug has been administered using a classification model and a monitoring model. In FIG. 32, among various types of drug administration, whether a drug has been administered is determined for oral administration of a pill taken out of a medicine bottle, which is swallowing of a drug, inhalation of a drug using an inhaler, and oral administration of a pill pushed out of a blister pack, which is swallowing of a drug. In FIG. 32, the case where the model type is a monitoring model refers to a case where whether a drug has been administered is determined using only a monitoring model without using a classification model for three types of drug administration. The case where the model is a classification model + monitoring model refers to a case where whether a drug has been administered is determined using both a classification model and a monitoring model for three types of drug administration. Here, the monitoring model when the classification model is not used is a model that can determine whether a drug has been administered for three types of drug administration. The monitoring model when the classification model is used is a model that can determine whether a drug has been administered for one specific type of drug administration among the three types of drug administration. That is, when no classification model was used, one supervision model was used, and when no classification model was used, three supervision models were used.

[0325] Referring to FIG. 32, when only the supervision model was used without the classification model, it was found that for the three types of drug administration, whether or not the drug was administered was determined with an accuracy of 60.1%. Conversely, when the classification model and the supervision model were used together, it was found that whether or not the drug was administered was determined with the following accuracy: 77% accuracy for oral administration of a pill taken out of a medicine bottle, which is swallowing of the drug, 78.4% accuracy for inhalation of a drug using an inhaler, and 83.9% accuracy for oral administration of a pill pushed out of a blister pack, which is swallowing of the drug. It was found that whether or not the drug was administered was determined with an average accuracy of 79.8%. When the classification model was used, the supervision model related to the blister pack determined whether or not the drug was administered with high accuracy because the hyperparameters optimized for it were used to train the supervision model related to the medicine bottle and the supervision model related to the inhaler. However, the accuracy shown in FIG. 32 may change depending on the case. For example, the accuracy may improve as training is performed, or the accuracy may improve as the training video is changed.

[0326] 2.7 Additional Examples of Methods for Determining Whether a Drug Has Been Administered

[0327] Below, we describe some additional examples of methods for determining whether a drug has been administered.

[0328] 2.7.1 First Additional Example - Main Hand and Sub Hand

[0329] As described above, the server may determine whether or not the drug has been administered based on the video acquired from the wearable device. Here, when the video is generated by a wearable device (e.g., a smart watch, a smart band, or a smart ring) worn on either the left or right part of the user's body, the video generated when the drug is administered using the body part on which the wearable device is worn and the video generated when the drug is administered using the body part on which the wearable device is not worn may have different aspects. For example, when generating a video with a camera unit provided on a smart watch, the video generated when the drug is administered using the hand on which the smart watch is worn and the video generated when the drug is administered using the hand on which the smart watch is not worn may have different aspects. The following mainly describes the case where the wearable device is a smart watch. The hand on which the wearable device is worn is called the main hand, and the hand on which the wearable device is not worn is called the sub-hand. For example, when the smart watch is worn on the left wrist, the left hand is the main hand and the right hand is the sub-hand.

[0330] As described above, the drug administration pattern may differ between users. For example, some users may administer drugs using the main hand, and other users may administer drugs using the sub-hand, which is different from the drug administration pattern of the some users.

[0331] As described above, the same user may have different drug administration patterns depending on the situation. For example, the user may administer a drug using the main hand this time, but the user may administer a drug using the sub-hand next time, which is different from the drug administration pattern this time.

[0332] On the other hand, the user may administer the medicine using the main hand and the sub-hand together in some cases. For example, the user may hold a pill in the sub-hand, place the pill in the palm of the main hand, move the main hand to the mouth, and swallow the pill. In this case, it may be unclear whether the medicine was administered using the main hand or the sub-hand. Even in this case, the user's drug administration pattern may differ depending on the case. For example, the user swallowed the pill as described above this time, but the user may swallow the pill in a different pattern next time. For example, the user holds a pill in the main hand, places the pill in the palm of the sub-hand, moves the sub-hand to the mouth, and swallows the pill.

[0333] Here, between the main hand and the sub-hand, the drug administration may be considered to have been performed using the hand in which the main step of drug administration was performed. For example, if the main step of drug administration was performed with the main hand, the drug administration is considered to have been performed with the main hand. The main step of drug administration may differ depending on the type of drug administration. For example, in the case of swallowing a pill, the step of moving the pill to the mouth and swallowing the pill may be the main step of drug administration. As another example, in the case of using an inhaler, the step of moving the inhaler to the mouth and inhaling may be the main step of drug administration. As yet another example, in the case of using a turbuhaler, the step of moving the turbuhaler to the mouth and inhaling may be the main step of drug administration. As yet another example, in the case of using a nasal sprayer, the step of moving the nasal sprayer to the nose and inhaling may be the main step of drug administration. As yet another example, in the case of administering eye drops, the step of moving the eye dropper to the eye and administering the eye drops may be the main step of drug administration. As yet another example, in the case of injecting a drug, the step of moving the syringe to a part of the body for injection may be the main step of drug administration. The main steps of drug administration according to the drug administration type described above are merely examples, are not limiting, and may vary from case to case.

[0334] Alternatively, between the main hand and the secondary hand, the drug administration may be considered to have been performed using the hand that performed more steps of the overall drug administration process.

[0335] Hereinafter, drug administration that is deemed to have been performed using the main hand, for example, drug administration in which at least the main steps of drug administration are performed using the main hand or the majority of drug administration is performed using the main hand, will be referred to as drug administration targeted by the main hand. Drug administration that is deemed to have been performed using the sub-hand, for example, drug administration in which at least the main steps of drug administration are performed using the sub-hand or the majority of drug administration is performed using the sub-hand, will be referred to as drug administration targeted by the sub-hand.

[0336] The user's actions may be different depending on whether the user uses the main hand or the sub-hand to administer the medication to the target, and thus different actions may be captured in a video depicting administration of the medication to the main hand (hereinafter referred to as the "main hand video") and a video depicting administration of the medication to the sub-hand (hereinafter referred to as the "sub-hand video").

[0337] 33 and 34 are diagrams illustrating drug administration to the main hand and drug administration to the sub-hand, respectively, according to an embodiment related to swallowing a drug using a blister pack. Figures 33 and 34 are diagrams illustrating image frames of a video generated by a smart watch, a wearable device with a camera unit disposed to be located on the inside of the wrist when the smart watch is worn by a user.

[0338] The same or similar actions may be captured in the main hand video and the secondary hand video. For example, with reference to Figures 33 and 34, both the main hand video and the secondary hand video include the following captured therein: user actions 115a and 116a of holding a blister pack box, user actions 115b and 116b of removing a blister pack from the blister pack box, user actions 115c and 116c of pushing pills out of the blister pack, user actions 115e and 116f of placing a blister pack into the blister pack box, and user actions 115f and 116g of closing the blister pack box.

[0339] However, in some cases, an action captured in one of the main hand video and the sub-hand video may not be captured in the other. For example, referring to Figures 33 and 34, the user's action 115d of swallowing a pill is captured in the main hand video but not in the sub-hand video (action 116e is captured there), and the user's action 116d of holding a pill is captured in the sub-hand video but not in the main hand video. That is, the user's actions captured in the main hand video and the sub-hand video may be different, and even in this situation, the server needs to be able to determine whether or not the medicine has been administered based on the video. Therefore, it is necessary to provide a model trained to take this into account.

[0340] The server may determine whether a drug has been administered using a model trained using both the main hand video and the sub-hand video. For example, the training video of the surveillance model training data may include the main hand video and the sub-hand video. Alternatively, the training image frames of the detection model training data may include image frames of the main hand video and image frames of the sub-hand video. Alternatively, the confirmation model input data for training the confirmation model training data may include input data corresponding to the main hand video and input data corresponding to the sub-hand video. Alternatively, the training video (or training image frames) of the classification model training data may include the main hand video (or image frames of the main hand video) and the sub-hand video (or image frames of the sub-hand video). In this case, the server may determine whether a drug has been administered as drug administered or drug not administered. Even when the server trains a model using both the main hand video and the sub-hand video, the training method described in "2.5 Model Training Method" may be applied in the same or similar manner.

[0341] In addition to using both the main hand video and the sub hand video, the server may determine whether a drug has been administered using a model trained to determine whether a drug has been administered by distinguishing between drug administrations targeted at the main hand and drug administrations targeted at the sub hand, in which case, rather than determining whether a drug has been administered as either drug administered or not administered, the server determines whether a drug has been administered as either a drug administered targeted at the main hand, a drug administered targeted at the sub hand, or not administered.

[0342] For example, when a surveillance model is used to determine whether a medication has been administered, the training video of the surveillance model training data includes a main hand video and a sub-hand video, and the surveillance model label data includes a label corresponding to the main hand video and indicating medication administration by the main hand (hereinafter referred to as a "main hand label"), and a label corresponding to the sub-hand video and indicating medication administration by the sub-hand (hereinafter referred to as a "sub-hand label").

[0343] As another example, when a detection model and a confirmation model are used to determine whether a medication has been administered, the training image frames of the detection model training data may include image frames of a main hand video and image frames of a sub-hand video, and the confirmation model label data of the confirmation model training data may include a main hand label and a sub-hand label.

[0344] Even if the server trains a model to determine whether a drug has been administered, the training method described in "2.5 Model Training Method" may be applied in the same or similar manner by distinguishing between drug administration targeted at the main hand and drug administration targeted at the sub-hand.

[0345] FIG. 35 is a diagram showing the result of determining whether or not a drug has been administered depending on whether or not a drug administration targeted by the main hand is distinguished from a drug administration targeted by the sub-hand according to an embodiment. FIG. 35 shows the determination of whether or not a drug has been administered using a detection model and a confirmation model. In FIG. 35, it was determined whether or not oral administration of a pill taken out of a medicine bottle, which is the swallowing of a drug among various types of drug administration, was performed. As the detection model, ResNet101-based Faster R-CNN was used, and as the confirmation model, a combination of a 2D convolutional layer and LSTM was used (see FIG. 31). Referring to FIG. 35, it was found that the validation accuracy and test accuracy were 98.8% and 96.8%, respectively, when determining whether or not a drug has been administered by distinguishing between a drug administration targeted by the main hand and a drug administration targeted by the sub-hand. The verification accuracy and test accuracy were higher than the 97.7% verification accuracy and 88.8% test accuracy when determining whether a drug was administered without distinguishing between drug administration to the main hand and drug administration to the sub-hand. In other words, when determining whether a drug was administered by distinguishing between drug administration to the main hand and drug administration to the sub-hand, it is possible to accurately determine whether a drug was administered, compared to when determining whether a drug was administered without distinguishing between drug administration to the main hand and drug administration to the sub-hand.

[0346] The server may train the model using either the main hand video or the sub-hand video first, and then train the model using the other. Alternatively, the server may train the model using the main hand video and the sub-hand video alternately. Alternatively, the server may train the model using the main hand video and the sub-hand video without any particular rule of order.

[0347] The server may use a model for determining whether a drug administration targeting the main hand has been fulfilled and a model for determining whether a drug administration targeting the sub-hand has been fulfilled.

[0348] 2.7.2 Second Additional Example - Omission of part of the drug administration process

[0349] The server may determine whether the drug has been administered by checking in real time whether the user has performed a specific action for each step of drug administration. For example, the server may determine whether the drug has been administered by checking through video or images, step by step, whether the user is holding a pill, whether the user has put the pill in his / her mouth, and whether the user has swallowed the pill. In this case, since it is checked whether the user has performed a specific action for each step, it is advantageous to determine whether the drug has been administered without skipping a specific step. However, the user needs to perform an unnatural specific action to determine whether the drug has been administered, which is inconvenient for the user.

[0350] Alternatively, the server may determine whether the drug has been administered without requiring the user to perform a specific action. That is, even if the user administers the drug as usual, the server may analyze the video and determine whether the drug has been administered. However, in this case, since the user does not check in real time whether he or she has performed a specific action for each step, a specific stage of the drug administration may not be captured in the video depending on the user's positioning and the time it takes to administer the drug. Here, the server needs to determine whether the drug has been administered using a video of the drug administration in which a part of the drug administration process is omitted. In particular, when a video is taken for a predetermined time to determine whether the drug has been administered, when the user administers the drug for a time longer than the predetermined time, the later part of the drug administration process is not captured in the video. For example, in the case of oral administration of a pill taken out of a pill bottle, which is swallowing the drug, the earlier part of the drug administration process, the user's action of taking the pill out of the pill bottle or the user's action of holding the pill, may be captured in the video, and the later part of the drug administration process, the user's action of swallowing the pill, may not be captured in the video. Alternatively, if the camera unit is activated after the user starts to administer the medicine, the earlier (later) stages of the medicine administration process are not captured in the video. For example, if the camera unit is activated at the time when the user swallows a pill, the user's action of taking the pill out of the pill bottle or the user's action of holding the pill are not captured in the video. Even in such exceptional cases, the server needs to be able to determine whether the medicine has been administered or not, so a model that takes this into account needs to be provided.

[0351] Figure 36 is a diagram illustrating model training using a video with some image frames removed, according to one embodiment. Referring to Figure 36, the server may train a model using a training video 120 that includes all image frames and a training video 121 with some image frames removed. The model may be a model where the input data is a video, such as a supervision model and a first type of classification model.

[0352] Referring to FIG. 36, a training video 121 with some image frames removed may be generated based on a training video 120 that includes all image frames.

[0353] FIG. 37 illustrates a method for removing some image frames of a training video according to one embodiment.

[0354] With reference to (a) to (c) of FIG. 37, the training videos 122, 124, and 126 from which some image frames have been removed may be training videos from which a predetermined number of consecutive image frames 123, 125, and 127 have been removed from all the image frames of the training videos. Here, the removed consecutive image frames may be the front part 123 of the training video as shown in (a) of FIG. 37, the middle part 125 of the training video as shown in (b) of FIG. 37, or the rear part 127 of the training video as shown in (c) of FIG. 37. When all the image frames of the training video include the entire drug administration process, the training video from which the image frames corresponding to the front part have been removed may be considered as a case in which the front part of the drug administration process is not captured. Similarly, the training video from which the image frames corresponding to the middle part have been removed may be considered as a case in which the middle part of the drug administration process is not captured. The training video from which the image frames corresponding to the rear part have been removed may be considered as a case in which the rear part of the drug administration process is not captured. That is, the server trains the model using a training video in which successive image frames have been removed, so that even if some steps of the drug administration are not captured in the video, the server uses the video to determine whether the drug has been administered.

[0355] Referring to (d) of FIG. 37, the training video 128 from which some image frames have been removed may be a training video from which a predetermined number of image frames 129 have been randomly removed from all image frames of the training video.

[0356] The illustration of removing some image frames of the training video shown in FIG. 37 is by way of example only and is not intended to be limiting.

[0357] Even if some image frames of a training video are removed, the server needs to add data to the removed part because the format of the data input to the model is fixed. In this case, the server may add an image frame having a predetermined pixel value to the position of the removed image frame. For example, all pixels of the added image frame may have the same pixel value, but is not limited to this. That is, the training video from which some image frames are removed may include the same number of image frames as the training video including all image frames, and information about the removed image frames (e.g., information about pixel values) is lost in the training video. Alternatively, the server may add an image frame having a predetermined pixel value to a position different from the position of the removed image frame. For example, as shown in (a) of FIG. 37, even if a front portion 123 of the training video is removed, the server adds an image frame having a predetermined pixel value to a rear portion of the training video. As another example, as shown in (c) of FIG. 37, even if a rear portion 127 of the training video is removed, the server adds an image frame having a predetermined pixel value to a front portion of the training video.

[0358] The server may first train the model using either the training video from which some image frames have been removed or the training video including all image frames, and then train the model using the other. For example, the server may first train the model using the training video from which some image frames have been removed, and then train the model using the training video including all image frames. Alternatively, the server may alternately train the model using the training video from which some image frames have been removed and the training video including all image frames. Alternatively, the server may train the model using the training video from which some image frames have been removed and the training video including all image frames, without any specific rule of order.

[0359] Unlike surveillance models or first type classification models, verification models do not use video as input data, so the following describes the case of verification models.

[0360] 38 illustrates a method for removing some sub-data of the validation model input data for training, according to one embodiment. As described above, the validation model input data includes sub-data, each of which may correspond to an image frame. Thus, some sub-data of the validation model input data for training may be removed to account for situations where parts of the drug administration process are omitted, similar to the case where some image frames of the training video are removed, as shown in FIGS. 36 and 37.

[0361] 38, the training confirmation model input data 131 from which some sub-data has been removed may be training confirmation model input data from which some sub-data 132 has been removed from the training confirmation model input data 130 including all the sub-data. Similar to the case where some image frames of the training video have been removed as shown in FIG. 37, the removed sub-data of the training confirmation model input data may correspond to consecutive image frames or random image frames. Also, when removing sub-data of the training confirmation model input data corresponding to consecutive image frames, sub-data corresponding to image frames in the previous, middle or later parts may be removed.

[0362] Similar to the removal of some image frames of the training video, even if some sub-data of the training verification model input data are removed, the server may need to add data to the removed portion because the format of the data input to the verification model is fixed. In this case, referring to FIG. 38, the server may add sub-data 133 having a predetermined value (e.g., 0 in FIG. 38) to the position of the removed sub-data. Also, the training verification model input data from which some sub-data are removed as described above may include the same number of sub-data as the training verification model input data including all the sub-data, and information about the removed sub-data is lost in the training verification model input data.

[0363] The illustration of removing some sub-data of the training validation model input data shown in FIG. 38 is by way of example only and is not intended to be limiting.

[0364] The server may first train the confirmation model using either the training confirmation model input data from which some sub-data has been removed or the training confirmation model input data including all of the sub-data, and then train the confirmation model using the other. For example, the server may first train the confirmation model using the training confirmation model input data from which some sub-data has been removed, and then train the confirmation model using the training confirmation model input data including all of the sub-data. Alternatively, the server may train the confirmation model alternately using the training confirmation model input data from which some sub-data has been removed and the training confirmation model input data including all of the sub-data. Alternatively, the server may train the confirmation model using the training confirmation model input data from which some sub-data has been removed and the training confirmation model input data including all of the sub-data, without any particular rule of order.

[0365] The model may be trained using a label smoothing technique, in which case the server can more accurately determine whether a drug has been administered, particularly for videos in which parts of the drug administration process are omitted, in which case the server can more accurately determine whether a drug has been administered.

[0366] FIG. 39 is a diagram showing a model trained considering a situation where a part of the drug administration process is omitted according to an embodiment, and relates to the result of determining whether or not the drug has been administered by the detection model and the confirmation model trained using the training confirmation model input data from which some sub-data has been removed. The confirmation model 1 is a confirmation model trained only using the training confirmation model input data including all the sub-data. The confirmation model 2 is a confirmation model trained using both the training confirmation model input data from which some sub-data has been removed and the training confirmation model input data including all the sub-data. Basically, a 20-second video with a frame rate of 30 fps (i.e., the number of all image frames was 600) was used in the model training. That is, the training confirmation model input data with the number of all sub-data being 600 was used. Among them, 200 consecutive sub-data were removed to generate the training confirmation model input data from which some sub-data has been removed. As the detection model, ResNet101-based Faster R-CNN was used, and as the confirmation model, a combination of 2D convolutional layers and LSTM was used for both confirmation models 1 and 2 (see Figure 31).

[0367] Referring to FIG. 39, when the verification model 1 was used for the video of the entire drug administration process, the server determined whether the drug was administered with 96.5% accuracy. When the verification model 2 was used for the video, the server determined whether the drug was administered with 99.8% accuracy. When the verification model 1 was used for the video of drug administration in which a part of the drug administration process was omitted, the server determined whether the drug was administered with 83.7% accuracy. When the verification model 2 was used for the video, the server determined whether the drug was administered with 95.8% accuracy. That is, compared with the case where the verification model trained using only the training verification model input data including all the subdata was used, the accuracy was improved when the verification model trained using both the training verification model input data in which some subdata was removed and the training verification model input data including all the subdata was used. In particular, the accuracy of determining whether the drug was administered was improved for the video of drug administration in which a part of the drug administration process was omitted. In other words, when the model is trained using a training video from which some image frames have been removed, or corresponding training validation model input data from which some sub-data has been removed, the accuracy of determining whether a drug has been administered for a drug administration video in which part of the drug administration process has been omitted is improved.

[0368] FIG. 40 illustrates the removal of consecutive image frames and the removal of random image frames according to one embodiment, and relates to the results of determining whether a drug was administered by a detection model and a confirmation model trained using training confirmation model input data from which some sub-data were removed. Confirmation model 3 is a confirmation model trained using training confirmation model input data from which consecutive sub-data were removed and training confirmation model input data including all sub-data. Confirmation model 4 is a confirmation model trained using training confirmation model input data from which random sub-data were removed and training confirmation model input data including all sub-data. The same structure of the detection model and confirmation model as in FIG. 39 were used. The number of all sub-data and the number of removed sub-data in the training confirmation model input data were the same as in FIG. 39.

[0369] Referring to FIG. 40, when the confirmation model 3 was used for the video of the entire drug administration process, the server determined whether the drug was administered with 98.9% accuracy. When the confirmation model 4 was used for the video, the server determined whether the drug was administered with 99.3% accuracy. When the confirmation model 3 was used for the video of the drug administration in which a part of the drug administration process was omitted, the server determined whether the drug was administered with 93.1% accuracy. When the confirmation model 4 was used for the video, the server determined whether the drug was administered with 90.0% accuracy. That is, regardless of the method of removing the sub-data, whether the drug was administered was determined with similar accuracy for the video of the entire drug administration process. However, for the video of the drug administration in which a part of the drug administration process was omitted, the use of a confirmation model trained using training confirmation model input data from which successive sub-data were removed is advantageous in improving the accuracy of determining whether the drug was administered.

[0370] 2.7.3 Third Additional Example - Changing the Number of Frames

[0371] It has been described to determine whether a drug has been administered based on a video including a predetermined number of image frames or validation model input data including a predetermined number of sub-data. However, in some cases it may be necessary to determine whether a drug has been administered for a video including a different number of image frames or validation model input data including a different number of sub-data than the predetermined number.

[0372] For example, depending on the type of drug administration, the video recording duration may vary, and as a result, it may be necessary to determine whether a drug has been administered for video that includes a different number of image frames than a predetermined number, or for validation model input data that includes a different number of sub-data than a predetermined number.

[0373] As another example, for some users, the video recording duration may vary, and as a result, it may be necessary to determine whether a drug has been administered for videos that include a different number of image frames than a predetermined number, or for validation model input data that includes a different number of sub-data than a predetermined number.

[0374] First, we describe how the server processes videos that contain more than a predetermined number of image frames, or validation model input data that contains more than a predetermined number of sub-data.

[0375] 41 is a diagram illustrating image frame selection according to one embodiment. Referring to FIG. 41, a server may obtain a video 141 including a predetermined number of image frames through frame selection from a video 140 including more than a predetermined number of image frames.

[0376] In some embodiments, frame selection may refer to selecting odd or even image frames. For example, for a video that includes twice the predetermined number of image frames, the server may select odd or even image frames of the video to generate a video with the predetermined number of image frames. Similarly, for a video that includes n times the predetermined number of image frames, the server may select every nth image frame of the video to generate a video with the predetermined number of image frames.

[0377] Alternatively, frame selection may refer to randomly selecting image frames. For example, the server may randomly select a predetermined number of image frames from a video that includes more than a predetermined number of image frames to generate a video that includes the predetermined number of image frames.

[0378] This illustration of frame selection is by way of example only and is not intended to be limiting.

[0379] Similarly, the server may obtain confirmation model input data including a predetermined number of subdata from confirmation model input data including a number of subdata exceeding a predetermined number through subdata selection. Details of frame selection may be applied to subdata selection, and therefore detailed description thereof will be omitted.

[0380] We now describe how the server processes videos that contain fewer than a predetermined number of image frames or validation model input data that contains fewer than a predetermined number of sub-data.

[0381] The server may add image frames to a video that includes less than a predetermined number of image frames such that a video that includes a predetermined number of image frames is generated. For example, the added image frames may be added to an earlier portion of a video that includes less than a predetermined number of image frames. As another example, the added image frames may be added to a later portion of a video that includes less than a predetermined number of image frames. As yet another example, the added image frames may be added to a middle portion of a video that includes less than a predetermined number of image frames. As yet another example, the added image frames may be added at a random position in a video that includes less than a predetermined number of image frames.

[0382] Depending on the position where the image frame is added, the accuracy of determining whether the drug has been administered may differ. Here, the position where the image frame that improves the accuracy of determining whether the drug has been administered is added may differ depending on the type of model. For example, when using an LSTM model or a similar model, adding an image frame to the front of the video can improve the accuracy of determining whether the drug has been administered compared to other cases.

[0383] The added image frame may have a predetermined pixel value, for example, but not limited to, all pixels of the added image frame may have the same pixel value. 3. Method for determining drug administration

[0384] Although the method for determining whether a drug has been administered has been described, in some cases, it may be necessary to determine the content of the drug administration. Here, the content of the drug administration broadly refers to the content of whether a drug has been administered, for example, which pill the user has swallowed, and how many pills the user has swallowed. The following mainly describes the swallowing of pills.

[0385] FIG. 42 illustrates the determination of drug information when swallowing pills according to an embodiment. The drug information may include the number of pills and the type of pills. Referring to FIG. 42, the server may determine the drug information based on an image frame 151 showing a scene in which a pill is placed on a palm among image frames of a video 150. For example, the server may input the image frame 151 showing a scene in which a pill is placed on a palm among image frames of a video 150 into a drug information determination model or obtain the drug information.

[0386] When there are multiple image frames showing the scene where the pill is placed in the palm of the hand, the server may determine the drug information based on the image frame corresponding to the latest time among the multiple image frames, or the server may determine the drug information based on each of the multiple image frames, and may combine the obtained drug information to determine the final drug information.

[0387] The server may use a model capable of detecting the situation where a pill is placed on the palm to find image frames showing a scene where a pill is placed on the palm. For example, the server may input the video into the model for each image frame, and may find image frames showing a scene where a pill is placed on the palm among all image frames. If a detection model is trained to detect the situation where a pill is placed on the palm, the server may check the detection result of the detection model to find image frames showing a scene where a pill is placed on the palm without performing additional calculations using a specific model. In this case, image frames showing a scene where a pill is placed on the palm can be found significantly faster than when performing additional calculations using a specific model. Also, the computational cost can be significantly reduced.

[0388] The model may be implemented the same as or similar to the detection model described above, but is not limited to such.

[0389] The server that determines the drug administration content may be the same as the server that determines whether the drug has been administered, or the server that determines the drug administration content may be a specific server different from the server that determines whether the drug has been administered.

[0390] 4. How to manage information about drug administration

[0391] The server may manage various information related to drug administration. For example, the server may manage the user's current status of drug administration, whether the user is wearing the wearable device, and the video recording time of the wearable device. The current status of drug administration may include the video, the results of analyzing the video (e.g., the results of determining whether the drug was administered, such as information regarding whether the video depicts drug administration or no drug administration), the user's drug administration schedule, the time of the user's drug administration, and whether the user administered the drug again. The current status of drug administration is described in more detail below.

[0392] The server that manages the information related to drug administration may be the same as the server that determines whether the drug has been administered or the server that determines the drug administration content. Alternatively, the server that manages the information related to drug administration may be a specific server for management that is different from the server that determines whether the drug has been administered or the server that determines the drug administration content.

[0393] The method for managing information related to drug administration will now be described in more detail.

[0394] 4.1 Managing the current status of medication administration

[0395] The server may also manage the current status of a user's medication administration.

[0396] The server may store a schedule of medication administration for the user.

[0397] The server may store the time of the user's drug administration. The time of the drug administration may be determined based on the time at which the wearable device generates the video. For example, the time of the drug administration may be the time at which the wearable device generates the video.

[0398] The server may provide notifications to the user based on the user's medication schedule.

[0399] For example, close to the time when the user needs to take the medication according to the medication schedule, the server may provide a notification to the wearable device or user terminal to guide the user in taking the medication.

[0400] As another example, the server may determine whether the user has overdosed on a medication, and may transmit the result of the determination to the wearable device or the user terminal to notify the user that the medication has been administered again. Here, an overdose of a medication means that more than a predetermined number of medications have been administered in a predetermined time, for example, the user has taken two pills when only one pill should have been taken. The server may determine whether the user has overdosed on a medication by comparing the user's medication administration with the medication administration schedule.

[0401] The server may confirm to the user whether the medicine has been administered. The server may confirm to the user whether the medicine has been administered via the wearable device or the user terminal.

[0402] For example, the server may prompt the user whether the medication has been administered based on the medication administration schedule. Close to the time when the user is required to administer the medication according to the medication administration schedule, the server may prompt the user whether the medication has been administered.

[0403] As another example, the server may receive the video and confirm to the user whether the medication has been administered.The server may receive the video from the wearable device and confirm to the user whether the medication has been administered.

[0404] As yet another example, the server may determine whether the user administered the medication and may confirm to the user whether the medication was administered. The server may analyze the video. Upon determining that the user administered the medication, the server confirms with the user whether the user administered the medication. Alternatively, regardless of the results of analyzing the video, the server may analyze the video to determine whether the medication was administered and may confirm with the user whether the medication was administered.

[0405] The server may provide the user's guardian or medical staff with information regarding the current status of the user's medication administration. For example, the server may provide the guardian or medical staff with at least some of the following: a video, a schedule of the user's medication administration, whether the medication was administered, the time of the medication administration, and whether the medication was administered again. In this case, the server storage unit may store an identification of the user of the wearable device in association with an identification of the device of the user's guardian or medical staff.

[0406] If the server determines that the user has administered the medication, the server may provide a parent or medical staff with information regarding the current status of the user's medication administration.

[0407] If the server determines that the user has not administered the medication, the server may provide a parent or medical staff with information regarding the current status of the user's medication administration. For example, if the server determines that the user has not administered the medication for a predetermined period of time, the server may notify a parent or medical staff that the user has not administered the medication.

[0408] 4.2 Wearable Device Management

[0409] The server may manage the user's wearing of the wearable device.

[0410] The server may determine whether the user is wearing the wearable device based on the information collected by the wearable device. For example, the server may analyze the movement of the wearable device through a motion sensor of the wearable device and determine whether the user is wearing the wearable device. Specifically, if the wearable device has not moved for a predetermined period of time, the server may determine that the user is not wearing the wearable device.

[0411] If the server determines that the user is not wearing the wearable device, the server may notify the user via the wearable device or the user terminal to prompt the user to wear the wearable device.

[0412] The server may check whether the user is wearing the wearable device properly. Depending on the direction in which the wearable device is worn, inaccurate video may be obtained. For example, if the wearable device is a smart watch or a smart band, the wearable device needs to be worn with the camera unit facing the user's hand so that the area around the user's hand is captured.

[0413] The server may analyze the video to determine whether the user is wearing the wearable device properly. For example, if the server determines that the drug administration was not administered a predetermined number of times or more consecutively as a result of analyzing the video, the server may determine that the user is not wearing the wearable device correctly. Alternatively, if the server determines that the drug administration-related objects and postures are not detected or are detected a predetermined number or less as a result of analyzing the video, the server may determine that the user is not wearing the wearable device correctly. Alternatively, if the server determines that the area corresponding to the user's arm in the acquired image frame exceeds a predetermined percentage of the total area of ​​the image frame, the server may determine that the user is not wearing the wearable device correctly.

[0414] The server may check whether the camera unit of the wearable device is covered by another object, for example, if the wearable device is a smart watch or a smart band, the server may check whether the camera unit is covered by the user's sleeve.

[0415] The server may analyze the video and determine whether the camera unit of the wearable device is covered by another object. For example, if the server determines that no drug administration has been performed a predetermined number of times or more consecutively as a result of analyzing the video, the server may determine that the camera unit is covered by another object. Alternatively, if the server determines that no drug administration-related objects and postures are detected or a predetermined number or less of them are detected as a result of analyzing the video, the server may determine that the camera unit is covered by another object. Alternatively, the server may determine whether the camera unit is covered by another object based on pixel values ​​of the acquired image frames.

[0416] If the server determines that the user is not wearing the wearable device correctly, the server may notify the user via the wearable device or the user terminal to guide the user to properly wear the wearable device.

[0417] When the wearer wears the wearable device, the wearable device may perform a step of acquiring and transmitting image frames or videos to the server, and the server may then manage the wearer's wearing of the wearable device based on the received image frames or videos.

[0418] 4.3 Video Recording Time Management

[0419] The server may manage the video recording time.

[0420] The server may change the video recording time of the wearable device. For example, the server may increase or decrease the video recording time. The server may change the video recording time to optimize the video recording time for each user.

[0421] The server may analyze the video and determine whether the video recording time is appropriate for the user. For example, the server may determine whether the video recording time currently set on the wearable device is appropriate for the user by determining which last image frame of the video corresponds to a medication administration event.

[0422] If the server determines that the video recording time is inappropriate for a user, the server may change the video recording time. For example, if the server determines that the video recording time is too short for a particular user, the server may increase the video recording time for the particular user. Alternatively, if the server determines that the video recording time is too long for a particular user, the server may decrease the video recording time for the particular user.

[0423] 5. Application to remote medical care

[0424] The details of drug administration described above may also be applied to telemedicine situations.

[0425] Telemedicine is a medical procedure in which a doctor and a patient are remote from each other, and the doctor assesses the patient's condition and provides appropriate treatment through communication means. In this telemedicine situation, it may be difficult for the doctor to assess the patient's condition compared to when the patient meets the doctor face-to-face. Therefore, it may be useful to provide the doctor with information about the patient's condition.

[0426] FIG. 43 is a diagram showing the provision of a second opinion report as an example of an application to telemedicine. The second opinion report is a report on the health condition of a user. A doctor or a guardian may refer to the second opinion report when checking the health condition of the user or creating a healthcare plan for the user. This allows the doctor to check the health condition of the user or create a healthcare plan for the user without meeting the user face-to-face. The server can provide or support a telemedicine service by providing the second opinion report to the doctor.

[0427] The server may generate the second opinion report taking into account the degree of medication adherence of the user, for example, the server may generate the second opinion report taking into account the degree of medication adherence with a previous prescription, where the degree of medication adherence with a previous prescription may be calculated based on a result of determining whether medication administration of the previous prescription was performed.

[0428] The server may determine the health status of the user and generate a second opinion report. The server may determine the health status of the user after the drug is administered and a predetermined time has elapsed, and may generate a second opinion report. The predetermined time may vary depending on at least one of the user, the type of drug, and the type of drug administration.

[0429] To determine the user's health status, the server may send questions about the user's health status to the wearable device, including, but not limited to, side effects of medications, symptoms and feelings after administration of medications, one or more clinical values, and one or more pain values.

[0430] To determine the user's health status after the medication has been administered and a predetermined time has elapsed, the server may send a question regarding the health status to the wearable device after the medication has been administered and a predetermined time has elapsed.

[0431] The wearable device may obtain a user's answer to the question received from the server. The wearable device may transmit the obtained user's answer to the server. The server may determine a health condition of the user based on the obtained user's answer. The server may generate a second opinion report based on the determined health condition of the user. The server may provide the generated second opinion report to a doctor or a guardian.

[0432] The server may further consider the user's biometric information to determine the user's health status or generate a second opinion report. As described above, examples of the biometric information include, but are not limited to, information on exercise such as the number of steps, heart rate, blood oxygen saturation, and electrocardiogram.

[0433] The server may further consider the biological information before drug administration, or the biological information after drug administration, or both to determine the user's health status or generate a second opinion report. For example, the server may determine the user's health status or generate a second opinion report by comparing the biological information before drug administration and the biological information after drug administration.

[0434] The server may obtain the user's biological information through the wearable device. The wearable device may measure the user's biological information. The wearable device may measure the user's biological information through a motion sensor, a heart rate sensor, a blood oxygen saturation sensor, and an electrocardiogram sensor of the wearable device. The wearable device may transmit the measured biological information to the server.

[0435] Although FIG. 43 shows the server determining the user's health condition via the wearable device, the user's health condition may be determined via a user terminal such as a smartphone, a tablet PC, a laptop computer, a desktop computer, etc. In this case, the server may determine the user's health condition by communicating with the user terminal rather than the wearable device. Alternatively, the server may determine the user's health condition via both the wearable device and the user terminal.

[0436] Also, although FIG. 43 shows that the server that determines whether the medication has been administered is the same as the server that generates the second opinion report, the server that generates the second opinion report may be different from the server that determines whether the medication has been administered.

[0437] FIG. 44 is a diagram showing provision of a prescription report as another example of application to remote medical care. A doctor or a pharmacist may provide a prescription to a user by referring to the prescription report. This allows the doctor or the pharmacist to provide a prescription to the user without meeting the user face-to-face. The server can provide or support a remote medical care service by providing the prescription report to the doctor or the pharmacist.

[0438] The server may calculate a degree of medication adherence of the user based on the information about the prescription. For example, the server may calculate a degree of medication adherence of the user with the first prescription based on the information about the first prescription. The degree of medication adherence of the user with the first prescription may be calculated based on a result of determining whether medication administration of the first prescription has been performed.

[0439] The server may generate a prescription report taking into account the degree of medication adherence of the user. For example, the server may generate a prescription report for a second prescription taking into account the degree of medication adherence according to the first prescription. Specifically, if the degree of medication adherence according to the first prescription is equal to or greater than a predetermined standard, the server may generate a prescription report stating that the user is permitted to administer the second prescription. Alternatively, if the degree of medication adherence according to the first prescription is less than a predetermined standard, the server may generate a prescription report stating that the user is not permitted to administer the second prescription.

[0440] The server may generate a prescription report taking into consideration the schedule of drug administration of the user. For example, when it is estimated that the user has completed drug administration of the first prescription according to the schedule of drug administration, the server may generate a prescription report. Alternatively, when there is a request for the second prescription, the server may generate a prescription report. For example, when there is a request for the second prescription from the user, the server may generate a prescription report. Here, as shown in FIG. 44, the server may receive a request for the second prescription from the user via the wearable device, but may also receive a request for the second prescription via another route such as a user terminal. Also, the server may receive a request for the second prescription of the user from a doctor, a pharmacist, or a guardian, rather than from the user.

[0441] The server may provide the generated prescription report to a physician or pharmacist.

[0442] Although FIG. 44 shows the server that determines whether the medication has been administered being the same as the server that generates the prescription report, the server that generates the prescription report may be different from the server that determines whether the medication has been administered.

[0443] 6. Expansion to Healthcare

[0444] Although the drug administration has been mainly described, the details of the drug administration may be extended to health care. Health care relates to management of a user's health in a broad sense in addition to drug administration, and examples of the management of a user's health include intake and measurement of health information by a measurement device, such as measurement of blood pressure by a blood pressure monitor and measurement of blood glucose by a blood glucose meter. That is, the drug administration described in "1. System for determining whether or not a drug has been administered", "2. Method for determining whether or not a drug has been administered", "3. Method for determining details of drug administration", "4. Method for managing information related to drug administration", and "5. Application to remote medical care" may be applied to health care, such as measurement of health information by a measurement device and intake.

[0445] The system for determining whether a drug has been administered may be extended to a system for determining whether a healthcare intervention has been performed. In the following, the details of the system for determining whether a drug has been administered described above will be mainly described as being extended to a system for determining whether a healthcare intervention has been performed.

[0446] The wireless communication device 1000 may be attached to or located near a measurement device (e.g., a blood pressure monitor or a blood glucose meter), or may be attached to or located near an eating utensil (e.g., a spoon, chopsticks, fork, or bowl).

[0447] The wireless communication device 1000 may collect data on the movement of the measuring device, or the ambient light, or both. Alternatively, the wireless communication device 1000 may collect data on the movement of the eating utensil, or the ambient light, or both.

[0448] The wireless communication device 1000 may use the collected data to confirm whether the user has started a healthcare action. For example, the wireless communication device 1000 may confirm whether the user has started a health information measurement action. As another example, the wireless communication device 1000 may confirm whether the user has started an intake action. When it is confirmed that the user has started a healthcare action, the wireless communication device 1000 transmits a wake-up signal to an external device such as the wearable device 2000 or the server 3000.

[0449] Alternatively, the wireless communication device 1000 may transmit the collected data to an external device such as the wearable device 2000 or the server 3000.

[0450] The wearable device 2000 may communicate with the wireless communication device 1000. The wearable device 2000 may receive data on the movement of the measuring device, or the ambient light, or both, from the wireless communication device 1000. Alternatively, the wearable device 2000 may receive data on the movement of the eating utensil, or the ambient light, or both, from the wireless communication device 1000.

[0451] The wearable device 2000 may generate a video. For example, the wearable device 2000 may generate a video related to health care. As a specific example, the wearable device 2000 may generate a video related to measuring health information. As another specific example, the wearable device 2000 may generate a video related to intake.

[0452] The wearable device 2000 may generate the video based on data received from the wireless communication device 1000. For example, the wearable device 2000 may generate the video based on data of movement of a measuring device, or ambient light, or both, or based on a wake-up signal. As another example, the wearable device 2000 may generate the video based on data of movement of an eating utensil, or ambient light, or both, or based on a wake-up signal.

[0453] The server 3000 may receive data on the movement of the measuring device, or the ambient light, or both, from the wireless communication device 1000 or the wearable device 2000 or both. Alternatively, the server 3000 may receive data on the movement of the eating utensil, or the ambient light, or both, from the wireless communication device 1000 or the wearable device 2000 or both.

[0454] The server 3000 may determine whether or not a healthcare action has been performed. For example, the server 3000 may determine whether or not a healthcare action has been performed based on information acquired from the wireless communication device 1000 or the wearable device 2000 or both. As a specific example, the server 3000 may determine whether or not a healthcare action has been performed by analyzing a video received from the wearable device 2000.

[0455] The method for determining whether a drug has been administered may be extended to a method for determining whether a healthcare intervention has been performed. Hereinafter, the details of the above-mentioned method for determining whether a drug has been administered will be mainly described, with a focus on the portion that extends the method for determining whether a healthcare intervention has been performed.

[0456] The server may use the surveillance model to determine whether a healthcare action has been performed. For example, the server may use the surveillance model to determine whether a measurement device has been used. As another example, the server may use the surveillance model to determine whether an intake has been performed.

[0457] The server may input the video into the surveillance model and determine whether the healthcare action has been performed, in which case the output data of the surveillance model may be the result of determining whether the healthcare action has been performed.

[0458] The result of determining whether or not a healthcare action has been performed output from the surveillance model may be an index indicating whether or not a healthcare action has been performed. That is, the output data of the surveillance model may be an index indicating whether or not a healthcare action has been performed. The server may use the surveillance model for the input video to obtain an index indicating whether or not a healthcare action has been performed. As a non-limiting example, the index may be expressed as a numerical value.

[0459] The training videos of the surveillance model may include training videos of performing a healthcare action and training videos of not performing a healthcare action.

[0460] The surveillance model label data may include a label corresponding to a training video of performing a healthcare action and indicating performing the healthcare action, and a label corresponding to a training video of not performing the healthcare action and indicating not performing the healthcare action. Alternatively, the surveillance model label data may include a label indicating a probability that performing the healthcare action is estimated, and a label indicating a probability that not performing the healthcare action is estimated.

[0461] The server may use the detection model and the confirmation model to determine whether a healthcare action has been performed. For example, the server may use the detection model and the confirmation model to determine whether a measurement device has been used. As another example, the server may use the detection model and the confirmation model to determine whether an intake has been performed.

[0462] The server may use the detection model to detect healthcare related objects and poses (hereafter referred to as healthcare related objects and poses). For example, the server may use the detection model to detect objects and poses associated with measurement device use (hereafter referred to as measurement device use objects and poses). As another example, the server may use the detection model to detect ingestion related objects and poses (hereafter referred to as ingestion objects and poses).

[0463] The server may input image frames of the video into a detection model to detect healthcare-related objects and poses contained in the image frames. The detection model may output data related to the healthcare-related objects and poses.

[0464] The healthcare-related objects may include various kinds of healthcare-related objects used by a user for a healthcare action. For example, the measurement device use object may include various kinds of measurement device use objects used by a user during the use of a measurement device. As a specific example, the measurement device use object may include a measurement device such as a blood pressure meter and a blood glucose meter. As another example, the intake object may include various kinds of intake objects used by a user for intake. As a specific example, the intake object may include eating utensils such as a spoon, chopsticks, and a fork.

[0465] The healthcare-related postures may include various kinds of healthcare-related postures, such as positioning of a user for a healthcare action. For example, the measurement device use postures may include various kinds of measurement device use postures, such as positioning of a user during use of a measurement device. As a specific example, the measurement device use postures may include postures for holding a measurement device-using object, such as a posture for holding a measurement device, a posture for using a measurement device, and a posture for opening a measurement device. As another example, the intake postures may include various kinds of intake postures, such as positioning of a user for intake. As a specific example, the intake postures may include a posture for holding food and a posture for taking food.

[0466] The detection model training data may include training image frames and detection model label data contained in the training image frames, the detection model label data relating to healthcare related objects and poses.

[0467] The server may use the detection results and the confirmation model to determine whether a healthcare action has been performed. For example, the server may use the detection results and the confirmation model to determine whether a measurement device has been used. As another example, the server may use the detection results and the confirmation model to determine whether an intake has been performed.

[0468] The input data for the validation model, validation model input data, may incorporate the presence or absence of a particular class of healthcare-related object and pose in the video, or the motion and pose of a particular class of healthcare-related object in the video over time.

[0469] The output data of the validation model may be a result of determining whether a healthcare action has been performed. For example, the output data of the validation model may be a result of determining whether a measurement device has been used. As another example, the output data of the validation model may be a result of determining whether an intake has been performed.

[0470] The result of judging whether or not a healthcare intervention has been performed output from the confirmation model may be an index indicating whether or not a healthcare intervention has been performed. That is, the output data of the confirmation model may be an index indicating whether or not a healthcare intervention has been performed.

[0471] The training validation model input data may include training validation model input data representative of performance of a healthcare action and training validation model input data representative of non-performance of a healthcare action.

[0472] The confirmation model label data may include a label indicating the execution of a healthcare action and a label indicating the non-execution of a healthcare action. Alternatively, the confirmation model label data may include a label indicating the probability that the healthcare action is estimated to be executed and a label indicating the probability that the healthcare action is estimated not to be executed.

[0473] The server may use the classification model to determine whether a healthcare action has been performed. The server may define a category of the video and use the defined category of the video to determine whether a healthcare action has been performed.

[0474] Defining a category for a video may refer to determining a type of healthcare corresponding to the video. For example, the server may define a category for a video by determining to which of two or more categories corresponding to different types of healthcare the video corresponds. In this case, examples of categories include, but are not limited to, a medication administration category, a blood pressure measurement category, a blood glucose measurement category, and an intake category.

[0475] Healthcare-related objects and poses may have categories corresponding to them. The categories of the healthcare-related objects and poses may be determined according to a type of healthcare associated with the healthcare-related objects and poses. Healthcare-related objects and poses associated with the same or similar types of healthcare may correspond to the same category. Healthcare-related objects and poses associated with different types of healthcare may correspond to different categories. For example, healthcare-related objects and poses such as a blood glucose meter and a pose for holding the blood glucose meter may correspond to a category related to using the blood glucose meter. Healthcare-related objects and poses such as a blood pressure meter and a pose for holding the blood pressure meter may correspond to a category related to using the blood pressure meter. Healthcare-related objects and poses such as eating utensils and a pose for holding food may be included in an eating-related category. This description of the categories of healthcare-related objects and poses is by way of example only and is not limiting.

[0476] The server may use a specific monitoring model for each video category to determine whether a healthcare action has been performed. Alternatively, the server may use a specific detection model and a specific confirmation model for each video category to determine whether a healthcare action has been performed.

[0477] The main hand video may be a video of a healthcare action that the main hand is a target of. The sub-hand video may be a video of a healthcare action that the sub-hand is a target of.

[0478] The server may use a model trained using both the main hand video and the sub-hand video to determine whether a healthcare action has been performed.

[0479] In addition to using both the main hand video and the sub hand video, the server may determine whether a healthcare action has been performed using a model trained to determine whether a healthcare action has been performed by distinguishing between a healthcare action performed by the main hand and a healthcare action performed by the sub hand. In this case, the server determines whether a healthcare action has been performed as a healthcare action performed by the main hand, as a healthcare action performed by the sub hand, or as a non-performance of a healthcare action, without determining whether a healthcare action has been performed as a healthcare action performed by the main hand or a healthcare action performed by the sub hand.

[0480] The method for determining the drug administration content may be extended to a method for determining the health care content. Hereinafter, the details of the method for determining the drug administration content described above will be mainly described, with a focus on the portion that is extended to the method for determining the health care content.

[0481] The server may determine the health care content, where the health care content broadly relates to whether a health care action has been performed, such as, for example, what blood pressure or blood sugar level is measured by the user, what food the user has taken, and how much food the user has taken.

[0482] The server may analyze the video and obtain the blood pressure or blood glucose level measured by the user, or the server may receive user input and obtain the blood pressure or blood glucose level measured by the user.

[0483] The server may analyze the video and obtain the type or amount of food consumed by the user, or the server may receive user input and obtain the type or amount of food consumed by the user.

[0484] The server may manage various information related to health care. For example, the server may manage the current health care status of the user. As a specific example, the server may manage the measurement device usage schedule of the user. As another specific example, the server may manage the intake schedule of the user.

[0485] The method according to an embodiment may be implemented as program instructions executable by various computer means and may be stored on a computer readable recording medium. The computer readable recording medium may include, individually or in combination, program instructions, data files, data structures, and the like. The program instructions stored on the computer readable recording medium may be specially designed and constructed for the present disclosure, or may be well known and available to those skilled in the art of computer software. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices such as ROMs, RAMs, and flash memories that are specially configured to store and execute program instructions. Combinations of these are also included, but are not limited to these. Examples of program instructions may include machine language codes produced by a compiler, and high-level language codes that are executable by a computer using an interpreter, or the like.

[0486] The present disclosure has been described with reference to exemplary embodiments thereof, but is not limited thereto. It will be apparent to those skilled in the art that various changes and modifications can be made within the spirit and scope of the present disclosure. It is therefore to be understood that such changes and modifications are intended to fall within the scope of the appended claims.

[0487] As stated in the best mode above, the relevant matters are described.

Claims

1. A server for determining whether a drug has been administered, a transceiver for receiving video recorded by the wearable device; a memory storing a detection model, a confirmation model, and a classification model, the detection model being trained to output whether each of predefined targets appears in an image, the confirmation model being trained to output whether a medication has been administered, and the classification model being trained to classify a category of the video, the detection models including a first detection model for detecting one or more predefined targets of a first type, and a second detection model for detecting one or more predefined targets of a second type, the predefined targets including objects related to medication or medication containers, and poses related to medication administration; one or more processors configured to: define a category of the video using the video and the classification model; if the category of the video is defined as a first category corresponding to the first type, detect the pre-defined target by inputting image frames of the video into the first detection model; if the category of the video is defined as a second category corresponding to the second type, detect the pre-defined target by inputting image frames of the video into the second detection model; and determine whether a drug has been administered by inputting confirmation model input data generated based on a detection result of the first detection model or the second detection model into the confirmation model; The server, wherein the posture includes at least one of a posture of holding a medicine container and a posture of taking medicine.

2. the detection result includes a plurality of detection model output data, each of the detection model output data corresponding to a respective image frame of the video; The server of claim 1 , wherein the one or more processors are further configured to generate the validation model input data by combining the plurality of detection model output data.

3. The server of claim 2 , wherein the confirmation model input data is generated by aligning a plurality of the detection model output data with time series information of corresponding image frames.

4. the detection model includes an object detection model and a pose detection model; 2. The server of claim 1 , wherein the one or more processors are further configured to detect objects by inputting image frames of the video to the object detection model, and to detect the pose by inputting image frames of the video to the pose detection model.

5. the verification models include a first verification model corresponding to the first detection model and a second verification model corresponding to the second detection model; The one or more processors: if the category of the video is defined as the first category, determining whether a medication has been administered by inputting the validation model input data into the first validation model; The server of claim 1, further configured to determine whether a medication has been administered by inputting the validation model input data into the second validation model if the category of the video is defined as the second category.

6. 1. A method, implemented by one or more processors, for determining whether a medication has been administered, comprising: receiving a video recorded by a wearable device; storing a detection model, a confirmation model and a classification model, the detection model being trained to output whether each of predefined targets appears in an image, the predefined targets including objects related to medication or a medication container, and postures related to medication administration, the postures including at least one of a posture of holding a medication container and a posture of taking a medication, the confirmation model being trained to output whether a medication has been administered, and the classification model being trained to classify a category of the video, the detection models including a first detection model for detecting one or more predefined first type targets and a second detection model for detecting one or more predefined second type targets; defining a category of the video using the video and the classification model, where if the category of the video is defined as a first category corresponding to the first type, detecting one or more predefined targets by inputting image frames of the video into the first detection model, and if the category of the video is defined as a second category corresponding to the second type, detecting the predefined targets by inputting image frames of the video into the second detection model; A method comprising: determining whether or not a drug has been administered by inputting confirmation model input data generated based on the detection results of the first detection model or the second detection model into the confirmation model.

7. the detection result includes a plurality of detection model output data, each of the detection model output data corresponding to a respective image frame of the video; The method of claim 6 , further comprising generating confirmation model input data by combining the plurality of detection model output data.

8. 8. The method of claim 7, wherein the confirmation model input data is generated by time-sequencing at least some of the detection model output data.

9. the detection model includes an object detection model and a pose detection model; The detecting step includes: detecting objects by inputting image frames of the video into the object detection model; and detecting pose by inputting image frames of the video into the pose detection model.

10. the verification models include a first verification model corresponding to the first detection model and a second verification model corresponding to the second detection model; The determining step includes: if the category of the video is defined as the first category, determining whether a medication has been administered by inputting the validation model input data into the first validation model; 7. The method of claim 6, further comprising: if the category of the video is defined as the second category, determining whether a medication has been administered by inputting the validation model input data into the second validation model.

11. 1. A method executed by one or more processors for training a detection model and a confirmation model to determine whether a drug has been administered, the method comprising: obtaining detection model training data comprising training images and detection model labels corresponding to the training images, each of the detection model labels having a class of medication or medication container and a class of posture associated with medication administration, the detection model labels including a first detection model label corresponding to a first one of the training images and a second detection model label corresponding to a second one of the training images; obtaining output data by inputting the training images into the detection model; updating the detection model by comparing the detection model labels with output data of the detection model; obtaining confirmatory model training data including confirmatory model input data and confirmatory model labels corresponding to the confirmatory model input data, the confirmatory model labels including a label for determining whether a drug has been administered, the confirmatory model input data being generated based on output data of the detection model; inputting said validation model input data into said validation model to obtain output data; updating the verification model by comparing the verification model labels with the verification model output data; The method, wherein the posture includes at least one of a posture of holding a medication container or a posture of taking a medication.

12. A non-transitory computer readable medium storing a program for causing a computer to execute the method of claim 6.

Images