Wearable electronic device for cooperation with other devices for managing health, and operation method of same
The wearable electronic device automatically monitors medication intake by capturing images of medication actions and using an artificial neural network for analysis, addressing the need for user intervention and improving the accuracy and convenience of health management tracking.
Patent Information
- Application Number
- PCT/KR2024/002307
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2024-01-10
- Filing Date
- 2024-02-22
- Publication Date
- 2024-11-21
AI Technical Summary
Current health management systems require user intervention for monitoring medication intake, leading to incomplete data collection and reduced accuracy in tracking daily health management activities.
A wearable electronic device with a detachable sub-device equipped with a camera and near-field communication circuitry that captures images of medication actions and communicates with a main device using short-range communication, enabling automatic monitoring and analysis of medication intake using an artificial neural network.
Improves user convenience by minimizing intervention in health management monitoring, enhances the accuracy of medication intake tracking, and supports health care services through collaboration with third-party devices, providing a comprehensive life log and disease symptom management.
Smart Images

Figure KR2024002307_21112024_PF_FP_ABST
Abstract
Description
Wearable electronic device for collaboration with other devices for health management, and method of operation thereof
[0001] The present disclosure relates to a wearable electronic device for collaboration with other devices for health management, and a method of operating the same.
[0002] Healthcare, based on medical science, has long been a constant topic of interest. This interest has intensified recently, fueled by advances in medical technology and increased life expectancy. This type of health management requires regular, voluntary engagement within daily life, whether through taking prescribed medications, regularly injecting medications, or using medical devices to measure health indicators.
[0003] In order to improve the effectiveness of the aforementioned health management, it is necessary to induce or enforce user behavior related to health management. To this end, a monitoring system is required that detects user behavior related to health management in daily life and confirms whether health management is being carried out properly.
[0004] Meanwhile, in the past, when monitoring the user's health management behavior, the user was required to operate the monitoring device before taking the medication or record his / her own behavior. This resulted in problems such as the user forgetting his / her behavior, resulting in information on health management not being collected.
[0005] Therefore, considering user convenience and the accuracy of health management, there is a need to develop a system that automatically monitors health management behaviors without the user's actions in daily life.
[0006] Meanwhile, there is a patent for detecting medication status using images captured by a camera, Publication Patent No. 2019-0078752, “Intelligent Information Management System for Medicine Storage and Medication Management for Smart Homes” (published on July 5, 2019).
[0007] The problem that the present invention seeks to solve is to provide a monitoring system that automatically monitors a user's health management behavior in daily life.
[0008] The problem to be solved by the present invention is to provide a monitoring system that detects the user's actions when the user takes a medication, takes a video related to the user taking the medication, and analyzes the video to determine whether the medication has been taken.
[0009] The problem to be solved by the present invention is to provide a wireless communication device that detects a user's action and generates a shooting start signal for shooting an image related to the user's drug administration.
[0010] The problem to be solved by the present invention is to provide a wearable device that can be worn on at least a part of a user's body and, when a shooting start signal is received, opens a closed camera to shoot a video related to the user's drug administration.
[0011] The problem that the present invention seeks to solve is to provide a server that analyzes a user's medication-related video using an artificial neural network to determine whether the user has taken the medication correctly.
[0012] The problem to be solved by the present invention is to provide a sub-wearable device that can support health management services by collaborating with a wearable device manufactured by a third-party manufacturer.
[0013] The problem that the present invention seeks to solve is to provide a server that improves the quality of a user's life log by utilizing the correlation between whether or not the user is taking medication and the life log obtained using a sensor.
[0014] The problem to be solved by the present invention is to provide a server that enables adjustment of the dosage for improving a disease by using the correlation between whether or not a drug is being taken and the values for disease symptoms obtained using a sensor.
[0015] The problem that the present invention seeks to solve is to provide a server that determines whether a user has taken medication correctly based on values sensed from various sensors as well as images.
[0016] The problems to be solved by the present invention are not limited to the problems described above, and problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention pertains from this specification and the attached drawings.
[0017] According to various embodiments, a device for determining medication is provided, comprising: a first detachable structure configured to be detached from a main device; a sub-device that is operated independently of the main device; wherein the sub-device includes: at least one processor; a short-range communication circuit; and a camera disposed on an outer surface of the sub-device; wherein the sub-device is coupled to the main device, and the at least one processor of the sub-device is configured to: establish a communication connection with the main device using the short-range communication circuit; capture an image of a medication action performed by a user using the camera; and provide the captured image, based on which the main device obtains result information regarding the medication action.
[0018] According to various embodiments, an operating method of a device for determining medication taking, including a main device and a sub-device configured to be detachably attached to the main device, may be provided, the operating method including: establishing a communication connection with the main device using the short-range communication circuit of the sub-device; capturing an image of a medication taking action performed by a user using a camera of the sub-device; and obtaining result information on the medication taking action by the main device based on providing the captured image using the camera of the sub-device.
[0019] According to various embodiments, the means for solving the problem are not limited to the above-described means, and the means for solving the problem that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from this specification and the attached drawings.
[0020] According to one example of this specification, user convenience can be improved by automatically monitoring the user's health management and minimizing user intervention.
[0021] According to one example of the present specification, a wearable device is provided that is worn by a user and captures images of the user's medication adherence, thereby facilitating monitoring of the user's health management in daily life, thereby increasing its usability.
[0022] According to one example of the present specification, a wireless communication device is provided that detects a user's motion and generates a signal for recording a video related to drug administration, thereby enabling more accurate and regular monitoring of a user's drug administration.
[0023] According to one example of this specification, the accuracy of determining whether a drug has been administered can be improved by analyzing an image related to drug administration using an artificial neural network.
[0024] According to one example of this specification, convenience for health management of users can be improved by providing a sub-wearable device that can support health management services by collaborating with a wearable device manufactured by a third-party manufacturer.
[0025] According to one example of this specification, a server is provided that improves the quality of a user's life log by utilizing the correlation between whether or not the user has taken medication and the life log obtained using a sensor, thereby enabling accurate medication for health promotion.
[0026] According to one example of this specification, a server is provided that utilizes the correlation between whether medication is taken and the values for disease symptoms obtained using a sensor, thereby enabling adjustment of the dosage for disease improvement.
[0027] According to one example of this specification, convenience for health management of users can be improved by providing a server that determines whether a user has taken medication correctly based on values sensed from various sensors as well as images.
[0028] The effects according to this specification are not limited to the effects described above, and effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from this specification and the attached drawings.
[0029] FIG. 1 is a block diagram illustrating an example of a configuration of a health care system according to various embodiments.
[0030] FIG. 2 is a block diagram illustrating another example of a configuration of a health care system according to various embodiments.
[0031] FIG. 3 is a diagram illustrating an example of a health care action according to various embodiments.
[0032] FIG. 4 is a drawing illustrating another example of a health care action according to various embodiments.
[0033] FIG. 5 is a block diagram illustrating examples of components of a server according to various embodiments.
[0034] FIG. 6 is a diagram illustrating an example of at least one artificial intelligence model for obtaining health management data on medication behavior according to various embodiments.
[0035] FIG. 7 is a drawing for explaining an example of a medication action according to various embodiments.
[0036] FIG. 8 is a diagram illustrating an example of an image of a medication-taking behavior as training data for at least one artificial intelligence model according to various embodiments.
[0037] FIG. 9 is a block diagram illustrating examples of components of a wearable device according to various embodiments.
[0038] FIG. 10 is a block diagram illustrating an example of a component of a user terminal according to various embodiments.
[0039] FIG. 11 is a block diagram illustrating an example of components of a wireless communication device according to various embodiments.
[0040] FIG. 12 is a drawing illustrating an example of a wireless communication device attached to a target object according to various embodiments.
[0041] FIG. 13 is a flowchart illustrating an example of a basic operation for a medication-taking action, which is an example of a health management action of a health management system according to various embodiments.
[0042] FIG. 14A is a perspective view illustrating a wearable device according to some embodiments of the present disclosure.
[0043] FIG. 14b is a perspective view showing the camera and protective cover with the housing of the main strap removed in some embodiments of the present disclosure.
[0044] FIG. 15A is a diagram illustrating an automatic shielding structure in some embodiments of the present disclosure.
[0045] FIG. 15b is a diagram showing the operation sequence of the automatic shielding protection cover and the camera in some embodiments of the present disclosure.
[0046] FIG. 16A is a structural diagram of a passive shielding protection cover according to some embodiments of the present disclosure.
[0047] FIG. 16b is a diagram illustrating the operation of a wearable device according to a closed state of a passive shielding protective cover in some embodiments of the present disclosure.
[0048] FIG. 17 is a diagram illustrating the operation of a wearable device according to an intermediate opening of a passive shielding protection cover in some embodiments of the present disclosure.
[0049] FIG. 18 is a diagram illustrating an AI learning method for a missing section due to intermediate opening in some embodiments of the present disclosure.
[0050] FIG. 19A is a diagram illustrating the operation flow of a wearable device according to the middle opening of a passive shielding protection cover in some embodiments of the present disclosure.
[0051] FIG. 19b is a diagram illustrating the operational flow of a server according to the intermediate opening of a passive shield protection cover in some embodiments of the present disclosure.
[0052] FIG. 20 is a block diagram illustrating an example of a healthcare system further including an add-on wearable device (e.g., a first wearable device), according to various embodiments.
[0053] FIG. 21 is a block diagram illustrating an example of a configuration of a first wearable device and a second wearable device according to various embodiments.
[0054] FIG. 22 is a drawing for explaining examples of a first wearable device and a second wearable device according to various embodiments.
[0055] FIG. 23 is a drawing for explaining an example of a first wearable device implemented in the form of a single strap according to various embodiments.
[0056] FIG. 24 is a diagram illustrating various examples of add-on wearable devices (e.g., a first wearable device) according to various embodiments.
[0057] FIG. 25 is a flowchart illustrating the basic operation of a healthcare system further including an add-on wearable device (e.g., a first wearable device) according to various embodiments.
[0058] FIG. 26 is a diagram illustrating an example of providing a health management service for medication compliance based on a first wearable device according to various embodiments.
[0059] FIG. 27 is a diagram for explaining an example of a communication connection method of a first wearable device according to various embodiments.
[0060] FIG. 28 is a flowchart illustrating an example of a communication connection establishment operation between a first wearable device and a second wearable device according to various embodiments.
[0061] FIG. 29 is a flowchart for explaining a remote control operation of a first wearable device according to various embodiments.
[0062] FIG. 30 is a drawing for explaining an example of remote control of a first wearable device according to various embodiments.
[0063] FIG. 31 is a flowchart illustrating an example of operation of a plurality of modes of an add-on wearable device (e.g., a first wearable device) according to various embodiments.
[0064] FIG. 32 is a diagram illustrating examples of multiple modes of an add-on wearable device (e.g., a first wearable device) according to various embodiments.
[0065] FIG. 33 is a flowchart illustrating an example of an operation of another wearable device when one of the first wearable device and the second wearable device is powered off, according to various embodiments.
[0066] FIG. 34 is a flowchart illustrating an example of operation of a health management system when a first wearable device is lost, according to various embodiments.
[0067] FIG. 35 is a block diagram illustrating an operation of performing medication determination using at least one sensor of a wearable device, according to various embodiments.
[0068] FIG. 36 is a diagram showing activation signal reference conditions for each wearable sensor in some embodiments of the present disclosure.
[0069] FIG. 37 is a flowchart illustrating a process of generating an activation signal using a wearable sensor in some embodiments of the present disclosure.
[0070] FIG. 38 is a flowchart illustrating a process of generating an activation signal using a wearable sensor in some embodiments of the present disclosure.
[0071] FIG. 39 is a diagram illustrating various types of wearable sensors and corresponding medication targets and activation signal reference conditions for each wearable sensor in some embodiments of the present disclosure.
[0072] FIG. 40 is a flowchart illustrating a process of identifying a medication target and generating an activation signal through a wearable sensor and a server in some embodiments of the present disclosure.
[0073] FIG. 41 is a flowchart illustrating a process of identifying a medication target and generating an activation signal through a wearable sensor and a server in some embodiments of the present disclosure.
[0074] FIG. 42 is a flowchart illustrating an example of an operation for providing a medication determination result based on at least one sensor of a wearable device according to various embodiments.
[0075] FIGS. 43 to 44 are diagrams for explaining an operation of obtaining a medication determination result based on a sensor value of a health management system according to various embodiments.
[0076] FIGS. 45 to 47 are diagrams for explaining examples of learning an artificial intelligence model for one sensor-based medication judgment, and a medication judgment operation based on the learned artificial intelligence model, according to various embodiments.
[0077] FIG. 48 is a diagram for explaining an example of learning an artificial intelligence model for medication decision based on multiple sensors according to various embodiments, and a medication decision operation based on the learned artificial intelligence model.
[0078] FIG. 49 is a flowchart illustrating an example of a composite medication determination operation of a wearable device according to various embodiments.
[0079] FIG. 50 is a diagram illustrating an example of a server's composite medication judgment operation according to various embodiments.
[0080] FIG. 51 is a flowchart illustrating an example of a medication decision scenario of a wearable device according to various embodiments.
[0081] FIG. 52 is a diagram for explaining life log linkage operations of a health management system according to various embodiments.
[0082] FIG. 53 is a diagram illustrating an example of a life log identifiable using a sensor of a wearable device according to various embodiments.
[0083] FIG. 54 is a flowchart illustrating an example of a medication correlation providing operation of a health care system according to various embodiments.
[0084] FIG. 55 is a diagram illustrating an example of an operation for identifying a medication correlation of a health care system according to various embodiments.
[0085] FIG. 56 is a flowchart illustrating an example of a linkage correlation providing operation of a health care system according to various embodiments.
[0086] FIG. 57 is a diagram illustrating an example of an operation for identifying a linkage correlation of a health care system according to various embodiments.
[0087] FIG. 58 is a diagram illustrating an example of a linkage correlation diagram according to various embodiments.
[0088] FIG. 59 is a flowchart illustrating an example of a linkage correlation providing operation of a health care system according to various embodiments.
[0089] FIG. 60 is a diagram illustrating examples of operations for providing a service for managing a user's life log based on a correlation of a health care system according to various embodiments.
[0090] FIG. 61 is a diagram illustrating another example of an operation for providing a service for managing a user's life log based on a correlation of a health care system according to various embodiments.
[0091] FIG. 62 is a flowchart illustrating a method of operation of a digital therapeutic in a health care system according to some embodiments of the present disclosure.
[0092] Figure 63 is a drawing showing a method of providing a dosage according to the degree of symptoms after taking the medication in some embodiments of the present disclosure.
[0093] Figure 64 is a drawing showing a method for providing a dosage according to the degree of symptoms after taking the medication in another embodiment of the present disclosure.
[0094] Figure 65 is a diagram illustrating a method for providing a dosage according to side effects after taking a medication in some embodiments of the present disclosure.
[0095] FIG. 66 is a flowchart illustrating an example of an operation for providing medication authentication and rewards to a user of a health care system according to various embodiments.
[0096] FIG. 67 is a diagram illustrating an example for medication authentication and reward granting in a health care system according to various embodiments.
[0097] FIG. 68 is a flowchart illustrating an example of an operation for providing medication authentication and rewards to a user of a health care system according to various embodiments.
[0098] FIG. 69 is a flowchart illustrating an example of an operation for determining the true intent of a medication and providing a reward based thereon in a health care system according to various embodiments.
[0099] FIG. 70 is a diagram illustrating an example for determining the true intent of medication and granting rewards in a health care system according to various embodiments.
[0100]
[0101] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.
[0102] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.
[0103] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).
[0104] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., built-in memory or external memory) readable by a machine (e.g., an electronic device). For example, a processor (e.g., a processor) of the machine (e.g., an electronic device) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.
[0105] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.
[0106] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.
[0107] According to various embodiments, a device for determining medication is provided, comprising: a first detachable structure configured to be detached from a main device; a sub-device that is operated independently of the main device; wherein the sub-device includes: at least one processor; a short-range communication circuit; and a camera disposed on an outer surface of the sub-device; wherein the sub-device is coupled to the main device, and the at least one processor of the sub-device is configured to: establish a communication connection with the main device using the short-range communication circuit; capture an image of a medication action performed by a user using the camera; and provide the captured image, based on which the main device obtains result information regarding the medication action.
[0108] According to various embodiments, a device may be provided in which the sub-device further includes a housing, and a flexible substrate disposed inside the housing, wherein inside the housing, the at least one processor, the at least one short-range communication circuit, and the connection terminal of the camera are electrically connected through the flexible substrate.
[0109] According to various embodiments, a device may be provided in which, when the sub-device and the main device are combined and worn on the user's wrist, the camera is directed to photograph the user's hand.
[0110] According to various embodiments, the at least one processor may be configured to: receive a request for a communication connection from the main device through the short-range communication circuit, and establish a communication connection with the main device based on the received request.
[0111] According to various embodiments, the at least one processor may be configured to: obtain identification information for a wireless communication device physically connected to a target object for medication from the main device through the short-range communication circuit; obtain a signal including activation data broadcast from the wireless communication device based on the identification information; and activate the camera based on the activation data.
[0112] According to various embodiments, the at least one processor may be configured to: establish a communication connection with the wireless communication device through the short-range communication circuitry based on the identification information; and obtain the signal including the activation data based on the communication connection established with the wireless communication device.
[0113] According to various embodiments, a device may be provided that is configured to scan a signal including activation data broadcast from the wireless communication device based on the identification information through the short-range communication circuit, and acquire the signal including the activation data based on the scan.
[0114] According to various embodiments, a device may be provided in which the at least one processor is configured to: transmit the captured image to the outside based on activation of the camera through the short-range communication circuit, and the transmitted image is analyzed based on at least one artificial intelligence model of a server, so that the main device obtains result information on the medication action.
[0115] According to various embodiments, an operating method of a device for determining medication taking, including a main device and a sub-device configured to be detachably attached to the main device, may be provided, the operating method including: establishing a communication connection with the main device using the short-range communication circuit of the sub-device; capturing an image of a medication taking action performed by a user using a camera of the sub-device; and obtaining result information on the medication taking action by the main device based on providing the captured image using the camera of the sub-device.
[0116] According to various embodiments, the sub-device may further include a housing, and a flexible substrate disposed inside the housing; and an operating method may be provided, wherein inside the housing, the at least one processor, the at least one short-range communication circuit, and the connection terminal of the camera are electrically connected through the flexible substrate.
[0117] According to various embodiments, a method of operation may be provided in which the sub-device and the main device are worn on the user's wrist while being combined, and the camera is directed to photograph the user's hand.
[0118] According to various embodiments, an operating method may be provided, further comprising: receiving a request for a communication connection from the main device through the short-range communication circuit of the sub device; and establishing a communication connection between the sub device and the main device based on the received request through the short-range communication circuit of the sub device.
[0119] According to various embodiments, an operating method may be provided, further comprising: obtaining identification information about a wireless communication device physically connected to a target object for medication from the main device through the short-range communication circuit of the sub device; obtaining a signal including activation data broadcast from the wireless communication device based on the identification information through the short-range communication circuit of the sub device; and activating the camera of the sub device based on the activation data.
[0120] According to various embodiments, an operating method may be provided, further comprising: an operation of establishing a communication connection with the wireless communication device based on the identification information through the short-range communication circuit of the sub-device; and an operation of obtaining the signal including the activation data based on the communication connection established with the wireless communication device.
[0121] According to various embodiments, an operating method may be provided, further comprising: an operation of scanning a signal including activation data broadcast from the wireless communication device based on the identification information through the short-range communication circuit of the sub-device; and an operation of acquiring the signal including the activation data based on the scan of the sub-device.
[0122]
[0123] 1. Overview of the Health Care System (1)
[0124] According to various embodiments, a health management system (1) may be implemented to provide the user with health management data collected by automatically recording and analyzing various types of health management behaviors of the user using a wearable device (20) that can be worn by the user. The health management system (1) may not only enable the user to manage health management behaviors without omission by conveniently recording health management behaviors unconsciously without the user having to consciously record the health management behaviors, but may also provide the user with health management data that can provide insight into health management based on analyzing health management behaviors based on an artificial intelligence (AI) model, thereby contributing to the promotion of human health.
[0125]
[0126] 1.1. Composition of the health care system (1)
[0127] FIG. 1 is a block diagram illustrating an example of the configuration of a health management system (1) according to various embodiments. FIG. 2 is a block diagram illustrating another example of the configuration of a health management system (1) according to various embodiments. Below, FIGS. 1 and 2 will be further described with reference to FIGS. 3 and 4.
[0128] FIG. 3 is a diagram illustrating an example of a health care action according to various embodiments. FIG. 4 is a diagram illustrating another example of a health care action according to various embodiments.
[0129] According to various embodiments, referring to FIG. 1, a health management system (1) may include a server (10), a wearable device (20), a user terminal (30), and a wireless communication device (40) that is arranged (or provided) in a form associated with a target object (T) associated with a health management activity (e.g., attached, coupled, etc.). For example, referring to FIGS. 3 and 4, the health management activity analyzed by the health management system (1) may include a medication activity and a diet (or meal plan) management activity, and may include various types of health management activities for promoting the user's health, such as exercise and sleep, without being limited to the examples shown and / or described in FIG. 1. Meanwhile, without being limited to the examples shown and / or described in FIG. 1, the health management system (1) may be implemented to include more components and / or to include fewer components. For example, referring to FIG. 2, the health management system (1) may be implemented to obtain information on the health management activities of a user of the health management system (1) using only a wearable device (20), without implementing the wireless communication device (40). Also, for example, although not illustrated, the health management system (1) may be implemented to provide health management data to a user through another device (e.g., a wearable device (20)), without implementing the user terminal (30).
[0130] According to various embodiments, the server (10) may be implemented to analyze information on health management behaviors and provide services for health management. For example, the server (10) may obtain information on health management behaviors received from the wearable device (20) and provide health management data generated based on the analysis of the information on the health management behaviors to the user. For example, the health management data may be a broad concept that includes not only whether the user has performed the health management behaviors but also the content of the health management behaviors, the quality of the health management behaviors, etc. Based on the health management data provided from the server (10), the user can perform the health management behaviors more efficiently. For example, referring to FIG. 3, the server (10) may obtain information on medication behaviors from the wearable device (20) and provide information on whether the medication behaviors have been performed (medication determination result information) as health management data to the user through the user terminal (30), thereby allowing the user to manage the medication management behaviors. In addition to the above medication judgment result information, information on reliability may also be provided, which will be described later.
[0131] According to various embodiments, the wearable device (20) may be implemented to automatically (e.g., regardless of the user's control) acquire information about a health care activity. For example, the information about the health care activity may include a video (or image) capturing the health care activity, a sensor value generated during the health care activity, etc., and may include various types of information for recording the health care activity when the user performs the health care activity, without being limited to the examples described. For example, referring to FIGS. 3 and 4 , the wearable device (20) may acquire an image of a medication activity performed by the user and / or an image of a diet consumed by the user. Referring to FIG. 3 , the wearable device (20) may be implemented to initiate an operation (e.g., capturing an image using a camera) to acquire information about the health care activity based on activation data received through a wireless communication device (40) attached to a target object (T) for the health care activity. The target object (T) above refers to an object associated with the health management activity, and may vary depending on the type of the health management activity. For example, in the case of a medication activity, the target object (T) may include a medication object (e.g., a medicine bottle), the user's hand, etc., but is not limited to the examples described. Also, for example, in the case of a diet, the target object (T) may include dishes, the user's hand, etc., but is not limited to the examples described. Since the operation of acquiring health management information of the wearable device (30) is automatically triggered based on the wireless communication device (40) attached to the target object (T), not only is the operating burden of the wearable device (30) for health management activities reduced, but also all health management activities can be automatically captured without the user's intervention.Meanwhile, without being limited to the described example, the wearable device (20) may be implemented to acquire activation data based on a user's input (e.g., touch input for taking a picture) and initiate an operation (e.g., taking a picture using a camera) to acquire information about the health management action, without receiving activation data from the wireless communication device (40).
[0132] According to various embodiments, the user terminal (30) may provide health management data to the user. For example, based on the execution of an application installed on the user terminal (30), the user terminal (30) may provide information to the user by displaying an execution screen (300, 400) containing health management data. For example, referring to FIGS. 3 and 4 , the execution screen (300, 400) may include information regarding whether medication has been taken and / or information regarding the composition of a diet. However, the execution screen is not limited to the illustrated and / or described examples and may include various types of information based on the analysis results of health management actions. However, the execution screen is not limited to the illustrated and / or described examples and health management data may be provided to the user in various forms, such as visual content, auditory, tactile, etc.
[0133] According to various embodiments, the wireless communication device (40) may be implemented to activate an operation of acquiring information on a health management activity of the wearable device (20) by providing a wireless signal to the wearable device (20). For example, referring to FIG. 3, the wireless communication device (40) may be attached to a target object (T) for a health management activity, and may be implemented to generate an activation signal for triggering an operation of acquiring information on a health management activity of the wearable device (20) based on a sensor value that is naturally generated according to a change in the target object (T) when the user performs the health management activity, and transmit the generated activation signal to the wearable device (20).
[0134] Meanwhile, for the convenience of explanation, the type of health management behavior managed by the health management system (1) is specifically described as “medication behavior” below, but it is obvious to those skilled in the art that the contents described below can be applied to other types of health management behavior unless otherwise specified.
[0135]
[0136] 1.1.1. Implementation type of health management system (1)
[0137] According to various embodiments, the health management system (1) may be implemented as a server type in which analysis of information on health management behavior for health management services is intensively performed on a server (10), or as an on-device type in which analysis may be performed on a local device (e.g., a wearable device (20) and / or a user terminal (30)) other than the server (10).
[0138] In one embodiment, when the health management system (1) is implemented as a server type, the health management system (1) can be implemented so that information on health management behavior collected by the wearable device (20) is analyzed in the server (10) as described above with reference to FIGS. 1 and 2.
[0139] Unlike the illustrated and / or described embodiments, when the health management system (1) is implemented as an on-device type, all operations performed in the server (10) described below may be implemented to be performed in a local device (e.g., a wearable device (20) and / or a user terminal (30)). For example, since the medication determination module (500) of the server (10) is stored in a local device (e.g., a wearable device (20) and / or a user terminal (30)), analysis of information on medication behavior and determination of whether or not medication has been performed may be implemented to be performed in a local device (e.g., a wearable device (20) and / or a user terminal (30)).
[0140]
[0141] 1.2. Components of the health care system (1)
[0142] Below, examples of components of a health care system (1) according to various embodiments are described.
[0143]
[0144] 1.2.1. Components of the server (10)
[0145] FIG. 5 is a block diagram illustrating an example of components of a server (10) according to various embodiments. Below, FIG. 5 will be further described with reference to FIGS. 6 to 8.
[0146] FIG. 6 is a diagram illustrating an example of at least one artificial intelligence model for obtaining health management data on medication behavior, according to various embodiments. FIG. 7 is a diagram illustrating an example of medication behavior, according to various embodiments. FIG. 8 is a diagram illustrating an example of an image capturing a medication behavior as training data for at least one artificial intelligence model, according to various embodiments.
[0147] According to various embodiments, referring to FIG. 5, the server (10) may include a first processor (501), a first communication circuit (503), and a first storage device (505) storing a database (507) and a medication determination module (500). However, without being limited to the illustrated and / or described examples, the server (10) may be implemented to include more components and / or fewer components.
[0148] According to various embodiments, the first processor (501) may control at least one other component (e.g., a hardware or software component) of the server (10) connected to the first processor (501) by executing, for example, software (e.g., a medication judgment module (500)), and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the first processor (501) may store a command or data received from another component (e.g., the first communication circuit (503)) in a volatile memory, process the command or data stored in the volatile memory, and store result data in a non-volatile memory. According to one embodiment, the first processor (501) may include a main processor (not shown) (e.g., a central processing unit or an application processor) or an auxiliary processor (not shown) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the server (10) includes a main processor (not shown) and an auxiliary processor (not shown), the auxiliary processor (not shown) may be configured to use less power than the main processor (not shown) or to be specialized for a specific function. The auxiliary processor (not shown) may be implemented separately from the main processor (not shown) or as a part of the main processor (not shown).
[0149] The auxiliary processor (not shown) may control at least a part of functions or states related to at least one component (e.g., the first communication circuit (503)) of the server (10), for example, on behalf of the main processor (not shown) while the main processor (not shown) is in an inactive (e.g., sleep) state, or together with the main processor (not shown) while the main processor (not shown) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (not shown) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., the first communication circuit (503)). In one embodiment, the auxiliary processor (not shown) (e.g., a neural network processing device) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. Such learning may be performed, for example, in the server (10) itself where artificial intelligence is performed, or may be performed through a separate server (e.g., a learning server). Learning algorithms may include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, for example. An artificial intelligence model may include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.
[0150] According to various embodiments, the first communication circuit (503) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the server (10) and an external electronic device (e.g., wearable device (20), user terminal (30)), and the performance of communication through the established communication channel. For example, the server (10) may receive a medication performance image captured by the wearable device (20) through the first communication circuit (503). The first communication circuit (503) may operate independently from the first processor (501) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the first communication circuit (503) may include a wireless communication module (not shown) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (e.g., a wearable device (20), a user terminal (30)) via a first network (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as a plurality of separate components (e.g., multiple chips).The wireless communication module can identify or authenticate a server (10) within a communication network, such as a first network or a second network, using subscriber information (e.g., international mobile subscriber identity (IMSI)) stored in the subscriber identification module.
[0151] According to various embodiments, the first storage device (505) may store various data used by at least one component of the server (10) (e.g., the processor (110) or the sensor module (176)). The data may include, for example, input data or output data for software (e.g., a program and instructions related thereto). The first storage device (505) may include volatile memory or non-volatile memory.
[0152] According to various embodiments, the first storage device (505) may be implemented to store the medication judgment module (500) and the database (507). However, without being limited to the illustrated and / or described examples, the first storage device (505) may also be implemented to further store various data, information, and / or software necessary for the operation of the server (10).
[0153] According to various embodiments, the medication-taking determination module (500) may be software (or a computer program, or a computer code, or instructions) implemented to provide health management data based on analysis of information associated with medication-taking performance. Referring to FIG. 7, the medication-taking determination module (500) may include at least one artificial intelligence model. The at least one artificial intelligence model may include at least one of a classification model (610), a detection model (620), or a confirmation model (630), and is not limited to the illustrated example, and at least one of the illustrated artificial intelligence models may be omitted. The at least one artificial intelligence model (e.g., at least one of the classification model (610), the detection model (620), or the confirmation model (630)) may be trained to output specific information based on at least one image (or video) captured of an operation (700) that is sequentially (or time-seriesly) performed when performing medication-taking, as illustrated in FIG. 7. For example, referring to FIG. 8, the sequentially performed operations (700) may include operations for holding drug delivery-related objects such as an operation for holding a blister (71), an operation for holding an inhaler (72), an operation for holding a turbhaler (73), an operation for holding an eye drop bottle (74), an operation for holding a nasal spray (75), an operation for holding a pill (76), an operation for taking a drug such as a pill (77), an operation for inhaling a drug using a drug delivery-related object such as an inhaler, a turbhaler, or a nasal spray, an operation for administering eye drops using a drug delivery-related object such as an eye drop bottle, an operation for injecting a drug using a drug delivery-related object such as a syringe, an operation for opening a drug delivery-related object such as a medicine bottle, etc. Since the learning algorithm for the above learning is the same as described above, a redundant description will be omitted.
[0154] According to various embodiments, the classification model (610) may be implemented to identify a category of at least one image (or video) (600) (hereinafter, “medication performance image”) associated with medication performance captured by the wearable device (20). The category of the medication performance image (600) indicates a type of medication compliance, a medication target, etc., and examples thereof include, but are not limited to, a medication administration category, an inhaler use category, a turbidaler use category, a nasal spray use category, an eye drop administration category, and a drug injection category. The classification model (610) may be implemented to output a category index indicating the category in response to receiving the medication performance image (600) as input. Since the learning algorithm for the learning is the same as described above, a redundant description will be omitted.
[0155] According to various embodiments, based on the category index, the result information output from the detection model (620) described below may be different. In one embodiment, as different detection models (620) are implemented for each category index, information associated with the target object (T) obtained by inputting the medication performance image into the detection model (620) selected by the category index may be different. In addition, in one embodiment, as the detection model (620) is implemented so that the category index can be input together with the medication performance image (600) into the detection model (620), information associated with the target object (T) obtained from the detection model (620) selected according to the category index may be different.
[0156] According to various embodiments, the detection model (620) may be implemented to detect information associated with a target object (T) of a medication-taking image (600). The information associated with the target object (T) may include at least one of probability information on an object associated with medication-taking or probability information on at least one of an action associated with medication-taking. The object associated with medication-taking refers to a product containing a drug to be taken, such as a pill bottle, a PTP (press through package), a pill packaging material, an inhaler, a tube-haler, an eye dropper, and the like, and is not limited to the described examples. In addition, the action associated with medication-taking refers to an action (700) sequentially performed during the aforementioned medication-taking, and / or may refer to a body part associated with an action (700) sequentially performed during medication-taking. For example, if the above-described action is a motion of holding a pill (76), the body part associated with the motion (700) sequentially performed during the medication taking process, as illustrated in FIG. 8, may refer to the palm (BO1). The detection model (620) may be implemented to output, in response to receiving at least one of the medication taking image (600) or the category index, at least one of probability information on an object sequentially (or time-series) associated with the medication taking process or probability information on at least one of the motions associated with the medication taking process.
[0157] According to various embodiments, the detection model (620) may be learned based on training data (800) including sequentially performed operations (700) and images (800) captured for each sequentially performed operation (700), as illustrated in FIG. 8. For example, the images (81, 82, 83, 84, 85, 86, 87) for each operation include an object (MO) to be subjected to medication and / or a body part (BO1, BO2) for medication included in the image for each operation, and the object (MO) and the body part (BO1, BO2) may be labeled for each of the images (81, 82, 83, 84, 85, 86, 87). The server (10) sets the above actions (700) as output data and sets the labeled images (800) corresponding to each of the actions (700) as input data, and as learning proceeds, the detection model (620) can be obtained.
[0158] According to various embodiments, the confirmation model (630) may be implemented to obtain information on whether medication has been taken. For example, the confirmation model (630) may provide result information indicating whether medication has been taken (e.g., medication has been taken, medication has not been taken) in response to receiving information related to the target object (T) output from the detection model (620). The confirmation model (630) may be implemented to additionally provide reliability information on the result information, and the reliability may be provided as a numerical value of 0% or more and 100% or less, but is not limited to the described examples.
[0159] Various embodiments of the present document may be implemented as software (e.g., a program) including one or more instructions stored in a storage medium (e.g., internal memory or external memory) that can be read by a machine. For example, a processor (e.g., a first processor (501)) of a machine (e.g., a server (10)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.
[0160] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.
[0161] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.
[0162]
[0163] 1.2.2. Components of a wearable device (20)
[0164] FIG. 9 is a block diagram illustrating an example of components of a wearable device (20) according to various embodiments.
[0165] According to various embodiments, referring to FIG. 9, the wearable device (20) may include a second processor (901), a second communication circuit (903), a camera (905), a first display (907), a first input device (909), and a second storage device (911). Without being limited to the examples illustrated and / or described, the wearable device (20) may be implemented to include more components and / or fewer components. The second processor (901) may be implemented like the first processor (501) described above, the second communication circuit (903) may be implemented like the first communication circuit (503) described above, and the second storage device (911) may be implemented like the first storage device (505) described above, and therefore, redundant descriptions thereof will be omitted.
[0166] According to various embodiments, the camera (905) may be implemented to capture at least one image (or video). For example, the camera (905) may be implemented to capture a health care activity. The camera (905) may be activated based on the control of the second processor (901). The second processor (901) may be implemented to activate the camera (905) based on an activation signal received from the wireless communication device (40) described above. In this case, the second processor (901) may be further configured to activate the camera (905) when a specific condition is satisfied upon receiving the activation signal, but is not limited thereto. For example, the second processor (901) may be configured to determine that the specific condition is satisfied and activate the camera (905) when the RSSI (received signal strength indication) of the received activation signal is greater than or equal to a specific value.
[0167] According to various embodiments, the first display (907) can visually provide information to an external party (e.g., a user) of the wearable device (20). The display module (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the first display (907) can include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch (e.g., including a touch screen).
[0168] According to various embodiments, the first input device (909) may be implemented to acquire a user input received by the wearable device (20). For example, the user input may include a tactile input (e.g., touch, press), an auditory input (e.g., user's speech), etc. Accordingly, the first input device (909) may include a physical button for acquiring a tactile input (e.g., user's press), a microphone for acquiring an auditory input (e.g., user's speech), etc., but is not limited to the described examples.
[0169] According to various embodiments, the second storage device (911) may store a first application (900). The first application (900) may be controlled to perform at least one function to provide a medication management service through the wearable device (20). For example, the at least one function may include at least one of a function of activating the camera (905) of the wearable device (20) to capture a medication behavior, a function of establishing a communication connection between the wearable device (20) and another external device (e.g., a user terminal (30), a wireless communication device (40)), a function of providing a captured medication performance image (600) to another external device (e.g., a user terminal (30), a wireless communication device (40)), or a function of linking with a second application (1000) installed in the user terminal (30) to be described later. The first application (900) may be downloaded from the server (10) (or a separate app distribution server).
[0170]
[0171] 1.2.3. Components of the user terminal (30)
[0172] FIG. 10 is a block diagram showing an example of components of a user terminal (30) according to various embodiments.
[0173] According to various embodiments, referring to FIG. 10, the user terminal (30) may include a third processor (1001), a third communication circuit (1003), a second display (1005), a second input device (1007), and a third storage device (1009). Without being limited to the examples illustrated and / or described, the user terminal (30) may be implemented to include more components and / or fewer components. The third processor (1001) is implemented like the first processor (501) described above, the third communication circuit (1003) is implemented like the first communication circuit (503) described above, the second display (1005) is implemented like the first display (907) described above, the second input device (1007) is implemented like the first input device (909) described above, and the third storage device (1009) can be implemented like the first storage device (505) described above, so redundant descriptions are omitted.
[0174] According to various embodiments, the third storage device (1009) may store a second application (1000). The second application (1000) may be implemented so that the user terminal (30) provides health management data to the user. For example, the execution screen of the second application (1000) may include information about the health management data. As an example, the execution screen of the second application (1000) may be implemented so as to include the user's medication decision result. The second application (1000) may be downloaded from the server (10) (or a separate app distribution server).
[0175]
[0176] 1.2.4. Components of a wireless communication device (40)
[0177] FIG. 11 is a block diagram illustrating an example of a component of a wireless communication device (40) according to various embodiments. FIG. 12 is a drawing illustrating an example of a wireless communication device (40) attached to a target object (T) according to various embodiments.
[0178] According to various embodiments, referring to FIG. 11, the wireless communication device (40) may include a fourth processor (1101), a fourth communication circuit (1103), and a sensor (1105). Without being limited to the illustrated and / or described examples, the wireless communication device (40) may be implemented to include more components and / or fewer components. For example, referring to FIG. 13, the wireless communication device (40) may be attached to a target object (T) and may further include an adhesive member for attachment. Meanwhile, the fourth processor (1101) may be implemented like the first processor (501) described above, and the fourth communication circuit (1103) may be implemented like the first communication circuit (503) described above, and therefore, redundant descriptions thereof will be omitted.
[0179] According to various embodiments, the wireless communication device (40) may include a housing for contacting the target object (T). The housing may include an upper surface, a lower surface, a side surface connected between the upper surface and the lower surface, and an internal space defined by the upper surface, the lower surface, and the side surface. Referring to (a) of FIG. 12, the housing may be implemented in a cylindrical shape, but is not limited to the illustrated shape and may be implemented in various shapes. As illustrated in (b) of FIG. 12, the above-described adhesive member is provided on the lower surface of the housing so as to be attached to the target object (T) (e.g., a medicine bottle), but is not limited to the illustrated example and the adhesive member may be implemented in various shapes such as a pair of forceps attached in a form of picking up the target object (T).
[0180] According to various embodiments, the sensor (1105) may be implemented to sense the start of a user's medication performance. For example, the sensor (1105) may include at least one of a motion sensor (1210) (e.g., a tilt sensor, an angular velocity sensor) or an illuminance sensor (1220). The sensor (1105) may be provided in the internal space of the housing, and may be provided as an illuminance sensor in which an opening is formed in a portion of the upper surface and / or lower surface of the housing corresponding to the illuminance sensor so that external light is distributed into the internal space of the housing. Accordingly, the fourth processor (1101) may determine whether an activation signal is generated based on at least one of an illuminance value identified by the illuminance sensor or a motion value (e.g., a tilt value, an angular velocity value) identified by the motion sensor. For example, when at least one of the illuminance value or the motion value satisfies a specific value (or is greater than or equal to a specific value), it may be determined that the activation signal is generated. The above activation signal can cause the wearable device (20) to activate the camera (905).
[0181] According to various embodiments, the fourth communication circuit (1103) may establish a communication connection with the external device (e.g., a wearable device (20)). The fourth processor (1101) may transmit the generated activation signal to the wearable device (20) through the fourth communication circuit (1103).
[0182]
[0183] 2.1. Basic operation of the health management system (1)
[0184] FIG. 13 is a flowchart illustrating an example of basic operations for medication-taking, an example of health management operations of a health management system (1), according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed.
[0185] According to various embodiments, the wearable device (20) and the wireless communication device (40) may establish a short-range communication connection in operation 1301. For example, the short-range communication connection may be a communication connection based on Bluetooth, BLE, Wi-Fi direct, and / or NFC.
[0186] According to various embodiments, the wireless communication device (40) may generate activation data in operation 1303. For example, when a user holds a target object (T) (e.g., a medicine bottle) to perform medication, movement of the wireless communication device (40) attached to the target object (T) may occur. Based on the movement, the wireless communication device (40) may obtain at least one sensing value from a sensor (1105) (e.g., a motion sensor (1210), an illuminance sensor (1220)), generate the activation signal when the obtained at least one sensing value satisfies a preset condition, and transmit the generated activation signal to the wearable device (20) based on a short-range communication connection.
[0187] According to various embodiments, the wearable device (20) may receive activation data in operation 1305, perform a photographing operation using the camera (905) in operation 1307, acquire at least one image (or video) in operation 1309, and transmit at least one image to the user terminal (30) in operation 1311. The wearable device (20) may activate the camera (905) based on the activation data, and capture the user's medication behavior for a preset time (hereinafter, a shooting time) based on the activated camera (905), thereby acquiring a medication performance image (600) including at least one image for each sequentially performed medication behavior. Alternatively, without being limited to the described example, if the user does not perform the medication behavior, the acquired medication performance image (600) may not include the medication behavior. The wearable device (20) may, as at least part of the operation of activating the camera (905) based on the activation data, determine whether the RSSI of the activation data is equal to or greater than a preset value, and, if equal to or greater than the preset value, perform an operation of activating the camera (905), but is not limited to the described example. The transmission of the captured medication performance image (600) of the wearable device (20) may be performed in real time simultaneously with the capture of the camera (905), and / or may be performed after the camera (905) is deactivated after the capture time.
[0188] According to various embodiments, the transmission of the medication performance image (600) may be performed (hereinafter, “transfer via”) via the user terminal (30), as in operation 1313. For example, the wearable device (20) may transmit the medication performance image (600) to the user terminal (30) based on a short-range communication connection, and the user terminal (30) may transmit the received medication performance image (600) to the server (10) based on a long-range communication (e.g., Wi-Fi and / or cellular communication). The transmission via may be set to be performed under specific conditions, but is not limited to the described examples. For example, the transmission via may be set to be triggered when the communication connection between the wearable device (20) and the server (10) is released and / or the quality of the communication connection is below a threshold.
[0189] According to various embodiments, the server (10) may obtain medication determination result information based on at least one artificial intelligence model (e.g., classification model (610), detection model (620), and confirmation model (630)) and at least one image in operation 1315, and transmit the medication determination result information to the user terminal (30) in operation 1317. For example, the server (10) may obtain medication determination result information indicating whether or not medication has been taken, along with a reliability thereof, in response to inputting the received at least one image into the at least one artificial intelligence model (e.g., classification model (610), detection model (620), and confirmation model (630)).
[0190] According to various embodiments, the user terminal (30) may provide the received medication determination result information in operation 1319. For example, the user terminal (30) may display an execution screen including the received medication determination result information. When the second application (1000) is executed by the user, the user terminal (30) may display an execution screen of the aforementioned second application (1000) including the medication determination result information, and / or may display a lock screen (or pop-up panel) including the medication determination result information based on the execution of the second application (1000).
[0191]
[0192] 2.2. Camera shielding structure
[0193] FIG. 14a is a perspective view illustrating a wearable device according to some embodiments of the present disclosure, and FIG. 14b is a perspective view illustrating a camera and a protective cover with the housing of the main strap removed according to some embodiments of the present disclosure.
[0194] Referring to FIGS. 14A and 14B , a wearable device (20) includes a main body that visually provides a user interface to a user through a first display (1007) and a strap that is coupled to and worn on the user's body. In various embodiments, the strap may include a main strap (1400) that is detachably coupled to one side of the main body and has a camera (1005) for photographing the user's medication-taking motion, and a sub-strap (not shown) that is detachably coupled to the other side of the main body and includes a length-adjustable portion.
[0195] The main strap (1400) and the sub-strap can be combined with each other and worn on the user's body, and various combination methods can be applied, such as gap fitting, force fitting, or magnetic combination, without being limited to a specific combination method. The camera (1005) provided on one side of the main strap (1400) can easily capture the user's medication-taking motion and can form a predetermined angle so as not to interfere with the user's body when wearing the wearable device (20).
[0196] Meanwhile, as illustrated in FIG. 14a, the main strap (1400) may include a housing (1410) shaped to wrap around the camera (1005) to protect the camera (1005) from impact, and a protective cover (1411) provided on one side of the housing (1410) to prevent foreign substances from getting caught in the camera lens.
[0197] In Fig. 14b, the protective cover (1411) is illustrated as a sliding structure, but may also be implemented as a sliding structure, and as a method of opening and closing the protective cover (1411), not only a manual shielding structure in which a user directly shields the protective cover (1411) through an external force, but also an automatic shielding structure using an electrochromic element, etc. may be used. In the case of the automatic shielding structure, the on / off of the protective cover (1411) may be controlled through a shielding device and a control module connected to the protective cover (1411).
[0198] FIG. 15a is a diagram illustrating an automatic shielding structure according to some embodiments of the present disclosure, and FIG. 15b is a diagram illustrating the operation sequence of an automatic shielding protective cover and a camera according to some embodiments of the present disclosure. Hereinafter, the operation method of the protective cover will be described with reference to FIGS. 15a and 15b.
[0199] As illustrated in FIG. 15a, the main strap (1400) of the automatic shielding structure shields the protective cover (1411) with electrical or mechanical force, and may further include a second processor (1001) and an automatic shielding device (1420).
[0200] When the main strap (1400) receives an activation signal generated from a wearable sensor, the second processor (1001) controls the camera (1005) to be activated and generates and transmits at least one control signal for each of the automatic shielding device (1420) and the camera (1005).
[0201] The control signal of the camera (1005) may include a shooting signal and first time information. Camera shooting may begin based on the shooting signal, and the camera shooting may be maintained for a first time corresponding to the first time information. Additionally, in some embodiments, the second processor (1001) may check the open / closed state of the camera (1005) before generating the control signal, and generate the control signal based on the open / closed state.
[0202] The control signal of the automatic shielding device (1420) may include an opening signal, a closing signal, and second time information. The shielding device operates to shield the protective cover (1411) according to the opening signal and the closing signal, and the second time information may refer to a second time during the period in which the protective cover (1411) is opened between the opening signal and the closing signal.
[0203] Meanwhile, the first time and the second time may differ depending on the type of medication target, and the time lengths of the first time and the second time may also differ from each other. That is, as shown in (a) of Fig. 15b, the first time may be shorter than the second time, as shown in (b) of Fig. 15b, the first time may be longer than the second time, or as shown in (c) of Fig. 15b, the first time and the second time may have the same time length.
[0204] In addition, the start and end times of the first time and the second time may be different from each other. That is, as in (a) of Fig. 15b, the start time of the first time may be later than the start time of the second time, and the end time of the first time may be earlier than the end time of the second time. Similarly, as in (b) of Fig. 15b, the start time of the first start time may be earlier than the start time of the second time, and the end time of the first time may be later than the start time of the second time, or as in (c) of Fig. 15b, the start and end times of the first time may be earlier than the start and end times of the second time.
[0205] In this way, the wearable device (20) of the present disclosure can customize various shooting environments to suit user needs by setting the length, start and end points of the first time, which is the shooting time of the camera, and the second time, which is the shielding time of the protective cover, to be different from each other.
[0206] FIG. 16A is a structural diagram illustrating a passive shielding protective cover according to some embodiments of the present disclosure. FIG. 16A illustrates a main strap (1400) of a passive shielding structure that shields the protective cover by applying an external force from a user.
[0207] In a passive shielding structure, the main strap (1400) may further include a second processor (1001), a passive shielding device (1430) including a protective cover (1411), and a sensor (1431) for detecting the opening / closing status of the protective cover. As in the automatic shielding structure, when the main strap (1400) receives an activation signal generated from a wearable sensor, the second processor (1001) activates the camera (1005).
[0208] Thereafter, the second processor (1001) generates a control signal and transmits it to the camera (1005). The control signal of the camera (1005) may include a shooting signal and first time information, similar to the automatic shielding structure described above. Camera shooting may begin according to the shooting signal, and camera shooting may be maintained for a first time corresponding to the first time information.
[0209] The sensor (1431) detects the shielding status of the protective cover (1411), and generates an open signal or a close signal based on the detection result and transmits the signal to the second processor (1001). In the case of the passive shielding structure, unlike the automatic shielding structure, the second processor (1001) does not provide a control signal to the shielding device, so the camera shooting and the opening of the protective cover (1411) may not occur simultaneously. In other words, in the passive shielding structure, whether the protective cover (1411) is shielded or not is determined by an external force regardless of whether the camera is shooting, so various embodiments may occur due to a discrepancy between the opening time of the protective cover and the camera shooting time.
[0210] FIG. 16B is a diagram illustrating the operation of a wearable device (20) according to the closed state of a passive shielding protective cover in some embodiments of the present disclosure. As illustrated in FIG. 16B, when the main strap (1400) receives an activation signal from the main body (S1600), acquires a sensor value through the sensor (1431) (S1610), and compares the sensor value with a preset reference value to determine the open / closed state of the protective cover (1411) (S1620).
[0211] The main strap (1400) controls the camera to be activated when the open / closed state of the protective cover (1411) is determined to be open, and controls the camera to be deactivated when the open / closed state of the protective cover (1411) is determined to be closed (S1630), and transmits closure information to the main body (S1640). The main body stores the medication history based on the closure information, and can provide a notification to the user through the display that the protective cover (1411) is closed (S1650). Here, the medication history may be a graph related to medication by time zone and record information indicating that the medication action was not recorded.
[0212] In contrast, in various embodiments, the wearable device (20) can generate a control signal according to an input from the user at any point in time, regardless of the open / close state of the protective cover (1411), thereby controlling the camera to be activated and performing a medication-taking motion photographing.
[0213] In this way, the wearable device (20) of the present disclosure can collect and database the user's personal medication information by recording the medication history using the wearable device itself.
[0214] In another embodiment, even if the open / closed state of the protective cover (1411) is determined to be closed, the wearable device (20) may control the camera (1005) to an active state and transmit a control signal to perform a photographing of the user's medication-taking action. However, in this case, even though the camera is performing a photographing, there may be a section where the photographing of the user's medication-taking action is partially omitted due to the closing of the protective cover (1411). Hereinafter, several embodiments will be described regarding the operation of the wearable device (20) when the user opens the protective cover (1411) in the middle.
[0215] FIG. 17 is a diagram illustrating the operation of a wearable device (20) according to the intermediate opening of a passive shielding protection cover in some embodiments of the present disclosure. As illustrated in FIG. 17, when the second processor (1001) of the main strap (1400) receives an activation signal based on sensing of a user's medication-taking motion, it controls the camera to be activated for a first period of time. At this time, the first period of time may be a sufficiently long period of time for the user to complete the medication-taking motion, and as described above, the first period of time may vary depending on the type of medication target.
[0216] When the first time period ends, the second processor (1001) controls the camera to be deactivated. Before the first time period ends, if the user performs an opening operation of the protective cover (1411) by external force, the sensor (1431) detects this and transmits an opening signal.
[0217] In various embodiments, referring to (a) of FIG. 17, the second processor (1001) may generate a control signal to capture a camera image immediately upon receiving an activation signal. In this case, regardless of whether the protective cover (1411) is opened or closed, the camera image is captured first, so a missing section (1710) may occur. In other words, the missing section refers to a section in which the user's medication action is not captured due to the closure of the protective cover (1411), and the second processor (1001) may extend the camera image capture time by granting an additional time corresponding to the missing section.
[0218] In various embodiments, referring to (b) of FIG. 17, the second processor (1001) may generate a control signal to capture images from the camera starting from the point in time when it receives an activation signal and then receives an open signal from the sensor (1431). In this case, as well, since the capture begins after the user begins taking the medication, a missing section (1720) may occur.
[0219] Additionally, in various embodiments, if the protective cover (1411) is opened for a period exceeding a preset time (e.g., 20 seconds) from the time the activation signal is received, the second processor (1001) may not generate a control signal and may not perform camera shooting, as the user's medication action may have already ended.
[0220] Figure 18 is a diagram illustrating an AI learning method for missing sections due to intermediate opening in some embodiments of the present disclosure. For the missing sections, an AI model can be trained using a training data set consisting of groups of image frames of medication taken for each section.
[0221] Referring to (a) of FIG. 18, when the second processor (1001) generates a control signal and takes a picture of the camera immediately after receiving the activation signal, a blank image is taken by the closed protective cover (1411) during the missing section, so the server (10) can train the AI model by inputting a training data set including a blank image (2110) for a time corresponding to the missing section in the group of image frames for each section.
[0222] Referring to (b) of FIG. 18, when the second processor (1001) receives an activation signal and then generates a control signal from the time point at which an open signal is received from the sensor (1431) to take pictures of the camera, since the shooting does not proceed during the missing section, the server (10) can train the AI model by inputting a training data set that does not include an image (2120) for a time corresponding to the missed section in the group of image frames for each section.
[0223] Here, the second processor (1001) can improve the accuracy of medication judgment even when a missing section occurs during camera shooting by training an AI model using a group of medication image frames in which the time corresponding to the missing section is set in various ways, as illustrated in FIG. 18.
[0224] Meanwhile, in various embodiments, the server (10) may use an AI model to determine the user's medication behavior, calculate the reliability of the medication determination result, and, if the reliability is low, guide the user to directly record whether or not he or she has taken the medication through a nudge alarm.
[0225] FIG. 19a is a diagram illustrating an operational flow of a wearable device (20) according to an intermediate opening of a passive shielding protection cover in some embodiments of the present disclosure, and FIG. 19b is a diagram illustrating an operational flow of a server (10) according to an intermediate opening of a passive shielding protection cover in some embodiments of the present disclosure. Hereinafter, the operations of the wearable device (20) and the server (10) will be described in detail with reference to FIGS. 19a and 19b.
[0226] Referring to FIG. 19a, the main strap (1400) of the wearable device (20) receives an activation signal (S1910a), receives an open signal from a sensor (1431) (S1920a), and calculates the time difference between the times at which each signal was received (S1930a).
[0227] Thereafter, the main strap (1400) compares the calculated time difference value with a preset time (S1940a), and if the calculated difference value is less than the preset time, only the captured medication image is transmitted to the server (10) (S1950a), and if the calculated difference value is greater than or equal to the preset time, the captured medication image is transmitted to the server (10) together with the difference value (S1960a).
[0228] Referring to FIG. 19b, when the server (10) receives a medication image and a difference value (S1910b), it inputs information about the medication image and the difference value into the AI model (S1920b). At this time, the server (10) can input the medication image and the difference value itself into the AI model (S1921b), or alternatively, it can input a blank image frame corresponding to the medication image and the difference value into the AI model (S1922b). Thereafter, the server (10) can obtain medication judgment information as the output of the AI model (S1930b).
[0229]
[0230] 3. Add-on wearable devices
[0231] FIG. 20 is a block diagram illustrating an example of a healthcare system (2000) further including an add-on wearable device (e.g., a first wearable device (2000a)), according to various embodiments.
[0232] According to various embodiments, referring to FIG. 20, a health management system (2000) may include a first wearable device (2000a), a second wearable device (2000b) operatively connected to the first wearable device (2000a), a user terminal (30), and a wireless communication device (40) disposed (or provided) (e.g., attached, coupled, etc.) in a form associated with a target object (T) associated with a health management activity. The health management system (2000) illustrated in FIG. 20 may be implemented like the health management system (1) described with reference to FIG. 1, and thus redundant descriptions thereof will be omitted. The first wearable device (2000a) may be defined as a sub-device, the second wearable device (2000b) may be defined as a main device, and the sub-device may support a service for a health management activity (e.g., a medication management service) while being physically connected to the main device.
[0233] According to various embodiments, each of the first wearable device (2000a) and the second wearable device (2000b) may be wearable by a user and implemented to acquire and / or analyze information on medication performance based on collaboration. For example, each of the first wearable device (2000a) and the second wearable device (2000b) may be implemented to perform different functions, thereby enabling collaboration between them. In one embodiment, the first wearable device (2000a) may perform a function of photographing sequential medication implementation actions according to the user's performance of medication, and the second wearable device (2000b) may perform a function of transmitting the photographed medication performance image (600) to an external device (e.g., a server (10) and / or a user terminal (30)). In another embodiment, the second wearable device (2000b) is implemented to acquire other biometric information related to the medication performance, so that a service based on the correlation between the medication performance and the other biometric information (e.g., life log) can be provided, which will be described later in “5.1. Life log linkage.” In another embodiment, the second wearable device (2000b) can perform an operation to assist the photographing operation of the first wearable device (2000a), which will be described later in “4.1 Vision Activation.” As collaboration is performed between the first wearable device (2000a) and the second wearable device (2000b) performing different functions, the operational burden for each device is reduced, and when necessary, a health management service (e.g., medication performance judgment service) can be performed, thereby improving user convenience.
[0234] According to various embodiments, the first wearable device (2000a) and the second wearable device (2000b) may be provided to be operatively connected to each other. For example, the operative connection may mean at least one of a physical connection (or coupling) or a communication connection for exchanging data.
[0235] Meanwhile, without being limited to the examples shown and / or described, the first wearable device (2000a) may establish a direct communication connection with the user terminal (30) to exchange data, thereby performing an operation similar to the operation of collaborating with the second wearable device (2000b) described below together with the user terminal (30).
[0236]
[0237] 3.1. Configuration of add-on wearable devices
[0238] According to various embodiments, the add-on wearable device (e.g., the first wearable device (2000a)) may be implemented in the form of a strap that is attached to the main body of the smart watch. The main body of the smart watch is an electronic product manufactured by a third party (e.g., another smart watch manufacturer), and the add-on wearable device (e.g., the first wearable device (2000a)) according to various embodiments may be implemented to provide a medication management service in cooperation with the electronic product of the other third party.
[0239] FIG. 21 is a block diagram illustrating an example of the configuration of a first wearable device (2000a) and a second wearable device (2000b) according to various embodiments. Below, FIG. 21 will be further described with reference to FIG. 22.
[0240] FIG. 22 is a drawing for explaining examples of a first wearable device (2000a) and a second wearable device (2000b) according to various embodiments.
[0241] According to various embodiments, referring to FIGS. 21 and 22, the first wearable device (2000a) may be implemented with at least one strap (e.g., a first strap (2120a), a second strap (2120b)) that is physically detachable from the main body (e.g., a body (2110)) of the smart watch. The at least one strap (e.g., a first strap (2120a), a second strap (2120b)) may be formed to be physically detachable from the body (2111)), but may not include a component that is directly electrically connected thereto, but may be implemented to be capable of collaboration, thereby improving usability. The first strap (2120a) functions as a main strap for taking a video of the user taking medication and transmitting the taken video of the user taking medication to the body (2110), and the second strap (2120b) functions as a sub strap so that the degree to which it is fastened to the first strap (2120a) can be adjusted, thereby allowing the degree of tightening by the first strap (2120a) and the second strap (2120b) on a part of the user's body (e.g., wrist) to be adjusted.
[0242] According to various embodiments, referring to FIG. 21, the first strap (2120a) may include a first processor (2121), a camera (2122), a first short-range communication circuit (2123), a first battery (2124), a first charging circuit (2125), a first body fastening structure (2126a), and a first strap fastening structure (2127a), and the second strap (2120b) may include a second body fastening structure (2126b) and a second strap fastening structure (2127b). However, at least one strap (e.g., the first strap (2120a, 2120b)) may be implemented to include more or fewer components without being limited to the illustrated and / or described examples. Some electronic components of the first strap (2120a) (e.g., the first processor (2121), the camera (2122), the first short-range communication circuit (2123)) can be implemented like the wearable device (20) described above, so a redundant description will be omitted.
[0243] According to various embodiments, referring to FIG. 22, electronic components (2200, 2201) (e.g., a first processor (2121), a camera (2122), a first short-range communication circuit (2123), a battery (2124), a first charging circuit (2125)) of the first strap (2120a) may be electrically connected to each other by being disposed on a flexible substrate (e.g., an electronic circuit, an electric circuit) (2200) (e.g., a flexible printed circuit board (FPCB)) disposed inside the first strap (2120a). At this time, a terminal of the camera (2122) formed externally may be disposed on the flexible substrate (2200).
[0244] According to various embodiments, the first short-range communication circuit (2123) may be implemented to establish a short-range communication connection with an external device. The short-range communication may be a communication connection based on Bluetooth, BLE, Wi-Fi direct, and / or NFC (near field communication). The first short-range communication circuit (2123) may be implemented to support a function of establishing a short-range communication connection with each of a plurality of external devices. For example, the first short-range communication circuit (2123) may be implemented to establish a first short-range communication connection with the body (2111) (e.g., the second short-range communication circuit (2113)) and to establish a second short-range communication connection with the wireless communication device (40) attached to the target object (T). Accordingly, the first strap (2120a) (e.g., the first processor (2121)) can activate the camera (2122) based on an activation signal received from the wireless communication device (40) and transmit a medication-taking image captured using the activated camera (2122) to the body (2110). Meanwhile, without being limited to the described example, the first short-range communication circuit (2123) may be implemented to directly establish a communication connection with the user terminal (30).
[0245] According to various embodiments, the camera (2122) may be provided inside a camera housing formed to be extended in the vertical direction of the first strap (2120a), as shown in FIG. 22. The camera (2122) may be positioned inside the camera housing so that the camera has a shooting direction in the direction of the user's hand when the first strap (2120a) is worn by the user.
[0246] According to various embodiments, the first battery (2124) may be implemented to store power to be provided to components of the first strap (2120a) (e.g., the first processor (2121), the camera (2122), the first short-range communication circuit (2123)).
[0247] According to various embodiments, the first charging circuit (2125) may be implemented to charge the first battery (2124). However, without being limited to the illustrated and / or described examples, the first battery (2124) may be replaced without implementing the first charging circuit (2125).
[0248] According to various embodiments, the body fastening structure (e.g., the first body fastening structure (2126a) and the second body fastening structure (2126b)) may be implemented to be physically attached to the body (or main body) (2110) of the smart watch (e.g., the fastening structure (2117a)). Referring to FIG. 22, the body fastening structure (e.g., the first body fastening structure (2126a) and the second body fastening structure (2126b)) may be implemented at one end of the strap (e.g., the first strap (2120a) and the second strap (2120b)). The above body fastening structures (e.g., the first body fastening structure (2126a) and the second body fastening structure (2126b)) are implemented so as to be detachably attached to one end of the strap (e.g., the first strap (2120a), the second strap (2120b)), and can be implemented in various types corresponding to the type of the body (2110) of the smart watch. Accordingly, a specific type of the body fastening structure (e.g., the first body fastening structure (2126a) and the second body fastening structure (2126b)) corresponding to the type of the smart watch and / or the size of the body can be coupled to one end of the strap (e.g., the first strap (2120a), the second strap (2120b)). Based on a specific type of the above body fastening structure (e.g., the first body fastening structure (2126a) and the second body fastening structure (2126b)), the strap (e.g., the first strap (2120a), the second strap (2120b)) can be connected to the body (2110) of the smart watch used by the user.
[0249] According to various embodiments, the strap fastening structure (e.g., the first strap fastening structure (2127a), the second strap fastening structure (2127b)) may be formed at the other end of the strap (e.g., the first strap (2120a), the second strap (2120b)), and the two straps (e.g., the first strap (2120a), the second strap (2120b)) may be formed in a structure in which they are fastened to each other. For example, as illustrated in FIG. 22, one strap fastening structure (e.g., the first strap fastening structure (2127a)) may be formed in a structure including a hole, and thus another strap fastening structure (e.g., the second strap fastening structure (2127b)) may be inserted into the hole, thereby implementing the two straps (e.g., the first strap (2120a), the second strap (2120b)) to be fastened to each other, but is not limited to the described example. At this time, referring to FIG. 22, the one strap (e.g., the second The strap (2120b) may be implemented to further include at least one length adjustment member (2128a, 2128b). For example, on the outer surface of the second strap (2120b), detachable adhesive members (e.g., non-woven fabric) (2128a, 2128b) are formed, so that the other end (e.g., second strap fastening structure (2127b)) of the second strap (2120b) may be bent so that the two adhesive members (e.g., non-woven fabric) (2128a, 2128b) come into contact while being fastened by penetrating the hole (e.g., first strap fastening structure (2127a)) of the first strap (2120a). As the position at which the two adhesive members (e.g., non-woven fabric) (2128a, 2128b) come into contact is changed, the second strap (2120b) The length of the other end (e.g., the second strap fastening structure (2127b)) penetrating the hole (e.g., the first strap fastening structure (2127a)) of the first strap (2120a) is adjusted, so that the degree of tightening by the two straps (e.g., the first strap (2120a), the second strap (2120b)) on a part of the user's body (e.g., the wrist) can be adjusted.
[0250] According to various embodiments, referring to FIGS. 21 and 22, the second wearable device (2000b) may be implemented as a main body (e.g., body (2110)) of a smart watch, and may be physically detachable from at least one strap (e.g., first strap (2120a), second strap (2120b)) described above. The body (2110) may be implemented as an electronic product manufactured by a third party (e.g., another smart watch manufacturer) to provide a medication management service in collaboration with at least one strap (e.g., first strap (2120a), second strap (2120b)) described above, but is not limited to the described examples and may also be manufactured by a manufacturer of at least one strap (e.g., first strap (2120a), second strap (2120b)) that is not a third party.
[0251] According to various embodiments, referring to FIG. 21, the body (2110) may include a second processor (2111), a long-range communication circuit (2112), a second short-range communication circuit (2113), a sensor (2114), a second battery (2115), a first charging circuit (2116), and a fastening structure (2117). However, at least one strap (e.g., the first strap (2120a), the second strap (2120b)) is not limited to the illustrated and / or described examples and may be implemented to include more or fewer components. Some electronic components of the above body (2110) (e.g., second processor (2111), long-distance communication circuit (2112), second short-distance communication circuit (2113), sensor (2114), second battery (2115), first charging circuit (2116)) can be implemented like the wearable device (20) described above, so redundant descriptions are omitted.
[0252] According to various embodiments, the remote communication circuit (2112) may be implemented to establish a remote communication connection with the server (10) and / or the user terminal (30). The remote communication may include wireless LAN communication such as Wi-Fi, and / or cellular communication. The body (2110) (e.g., the second processor (2111)) may transmit the received medication performance image (600) to the server (10) and / or the user terminal (30) via the remote communication circuit (2112).
[0253] According to various embodiments, the second short-range communication circuit (2113) may be implemented to establish a short-range communication connection with the first strap (2120a). The body (2110) (e.g., the second processor (2111)) may receive a medication performance image (600) captured by the first strap (2120a) through the short-range communication circuit (2113), and / or transmit a control signal to the first strap (2120a).
[0254] According to various embodiments, the body (2110) may be implemented to be connected to at least one strap (e.g., first strap (2120a), second strap (2120b)) based on the fastening structure (2117).
[0255] FIG. 23 is a drawing for explaining an example of a first wearable device (2000a) implemented in the form of a single strap according to various embodiments.
[0256] Meanwhile, referring to FIG. 23 according to various embodiments, instead of the aforementioned plurality of straps (e.g., the first strap (2120a), the second strap (2120b)), a single strap may be implemented and fastened to the body (2110). The first strap (2120a) may be implemented as a single strap (2300a) including a plurality of body fastening structures (2301a, 2303a) to be fastened to fastening structures formed on both sides of the body (2110) as illustrated in (a) of FIG. 23, and / or may be implemented as a single strap (2300b) including a single body fastening structure (2301b) to be fastened in a form that wraps around the side of the body (2110) as illustrated in (b) of FIG. 23, but may be implemented as a single strap in various forms that are not limited to the illustrated and / or described examples. The configurations arranged inside the above single strap (2300a, 2300b) are the same as those of the first strap (2120a), so a detailed description is omitted.
[0257]
[0258] 3.1.2. Additional examples of add-on wearable devices
[0259] FIG. 24 is a diagram illustrating various examples of add-on wearable devices (e.g., first wearable device (2000a)) according to various embodiments.
[0260] In one embodiment, referring to (a) of FIG. 24, the first wearable device (2000a) is implemented as a wearable ring (2400a) that can be worn on the user's finger, and the wearable ring (2400a) may include a camera (2401a) for acquiring a medication performance image (600). Since the configuration of the wearable ring (2400a) may be implemented similarly to the first strap (2120a) described above, a redundant description will be omitted.
[0261] In one embodiment, referring to (b) of FIG. 24, the first wearable device (2000a) is implemented as a bracelet (2400b) that can be worn on the user's wrist, and the bracelet (2400b) may include a camera (2401b) for acquiring a medication performance image (600). The configuration of the wearable ring (2400a) may be implemented similarly to the first strap (2120a) described above, and thus, a redundant description thereof will be omitted.
[0262] In one embodiment, referring to (c) of FIG. 24, the first wearable device (2000a) is implemented as an additional member (2400c) (e.g., a wrist strap) detachable from the strap of the smart watch, and the additional member (2400c) may include a camera (2401c) for acquiring a medication performance image (600). Since the configuration of the additional member (2400c) may be implemented similarly to the first strap (2120a) described above, a duplicate description will be omitted.
[0263] Hereinafter, for convenience of explanation, an example in which the first wearable device (2000a) is implemented in the form of a strap will be described.
[0264]
[0265] 3.2. How the Health Management System (2000) Works
[0266] 3.2.1. Basic Operation of the Health Management System (2000)
[0267] Below, examples of the basic operation of a health management system (2000) further including an add-on wearable device (e.g., the first wearable device (2000a)) according to various embodiments are described. Descriptions that overlap with the basic operation of the health management system (1) described with reference to FIG. 13 are omitted.
[0268] FIG. 25 is a flowchart illustrating the basic operations of a healthcare system (2000) further including an add-on wearable device (e.g., a first wearable device (2000a)) according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. Below, FIG. 25 will be further described with reference to FIGS. 26 and 27.
[0269] FIG. 26 is a diagram illustrating an example of providing a healthcare service for medication compliance based on a first wearable device (2000a), according to various embodiments. FIG. 27 is a diagram illustrating an example of a communication connection method of a first wearable device (2000a), according to various embodiments.
[0270] According to various embodiments, the first wearable device (2000a) may establish a communication connection with the second wearable device (2000b) in operation 2501, and may establish a short-range communication connection with the wireless communication device (40) in operation 2503. For example, as illustrated in FIG. 26, the first wearable device (2000a) (e.g., a strap) may receive a request for a communication connection from a physically coupled second wearable device (2000b) (e.g., a main body of a smart watch) through the second short-range communication circuit (2123), and may establish a communication connection with the second wearable device (2000b). Also, for example, as illustrated in (a) or (b) of FIG. 27, the first wearable device (2000a) (e.g., a strap) may be implemented to obtain an activation signal broadcast from the wireless communication device (40). The first wearable device (2000a) (e.g., strap) may obtain an activation signal broadcast from the wireless communication device (40) based on identification information of the wireless communication device (40) received from the second wearable device (2000b) based on a short-range communication connection. For example, the first wearable device (2000a) may be implemented to establish a communication connection with the wireless communication device (40) attached to the target object (T) based on the identification information of the wireless communication device (40), or to scan a signal (e.g., an advertisement signal) including activation data broadcast from the wireless communication device (40) without establishing a communication connection based on the identification information of the wireless communication device (40).
[0271] According to various embodiments, the wireless communication device (40) may generate activation data in operation 2505. According to various embodiments, the first wearable device (2000a) may receive activation data in operation 2507, perform a photographing operation using a camera in operation 2509, acquire at least one image in operation 2511, and transmit the at least one image to the second wearable device (2000b) in operation 2513. For example, referring to FIG. 26, when movement of a target object (T) occurs in accordance with the user's medication performance, the wireless communication device (40) may generate activation data and transmit the activation data to the first wearable device (2000a). At this time, the activation data may be transmitted based on the short-range communication connection setup as described above, or the first wearable device (2000a) may scan the activation data included in the advertisement signal broadcast from the wireless communication device (40) regardless of the communication connection setup. The first wearable device (2000a) may activate the camera (2600) based on the activation data to obtain a medication-taking performance image (2610). At this time, instead of transmitting the medication-taking performance image to the second wearable device (2000b) as described above, the first wearable device (2000a) may transmit it to the user terminal (30) based on a direct communication connection setup.
[0272] According to various embodiments, the second wearable device (2000b) may acquire at least one image (e.g., medication performance image (2610)) in operation 2515, and transmit the at least one image (e.g., medication performance image (2610)) to the server (10) in operation 2517. At this time, the transmission of the (e.g., medication performance image (2610)) to the server (10) may be performed via the user terminal (30) as in operation 2519, but is not limited to the described example.
[0273] According to various embodiments, the server (10) may obtain medication determination result information based on at least one artificial intelligence model (e.g., at least one of a classification model (610), a detection model (620), or a confirmation model (630)) and at least one image in operation 2521, and transmit the medication determination result information to the user terminal (30) in operation 2523. According to various embodiments, the user terminal (30) may provide the medication determination result information in operation 2525.
[0274]
[0275] 3.2.1.1. Registering an Add-on Wearable Device
[0276] According to various embodiments, a medication management service can be performed by registering information about a user's add-on wearable device (e.g., a first wearable device (2000a)) to a server (10).
[0277] FIG. 28 is a flowchart illustrating an example of a communication connection establishment operation between a first wearable device (2000a) and a second wearable device (2000b) according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed.
[0278] According to various embodiments, the first wearable device (2000a) may establish a communication connection with the second wearable device (2000b) in operation 2701, and transmit wearable identification information to the second wearable device (2000b) in operation 2703.
[0279] According to various embodiments, the second wearable device (2000b) may register wearable identification information in operation 2705, store the wearable identification information and user identification information (e.g., user ID) in a form that is associated with each other in operation 2707, and transmit the wearable identification information and user identification information (e.g., user ID) to the server (10) in operation 2709.
[0280] According to various embodiments, the server (10) may store wearable identification information and user identification information (e.g., user ID) in a form associated with each other in operation 2711, and transmit medication management information in operation 2713. The server (10) may register the wearable identification information and the user identification information, and then store and manage health management data in a form associated with the wearable identification information and the user identification information. Thereafter, when receiving a request for health management data from the user terminal (30) and / or the second wearable device (2000b), the server may provide the stored health management data corresponding to the wearable identification information or the user identification information to the user terminal (30) and / or the second wearable device (2000b).
[0281]
[0282] 3.2.1.2. Remote Control for Add-on Wearable Devices
[0283] According to various embodiments, the add-on wearable device (e.g., the first wearable device (2000a)) can be controlled by another external device (e.g., the user terminal (30) or the second wearable device (2000b)). Accordingly, since the add-on wearable device (e.g., the first wearable device (2000a)) can be implemented without including electronic components for communicating with a user, such as a display, a touch screen, etc., implementation of the add-on wearable device (e.g., the first wearable device (2000a)) can be facilitated.
[0284] FIG. 29 is a flowchart illustrating remote control operations of a first wearable device (2000a) according to various embodiments. The operations may be performed regardless of the order of the operations illustrated and / or described, and more operations may be performed and / or fewer operations may be performed. Below, FIG. 28 will be further described with reference to FIG. 29.
[0285] FIG. 30 is a drawing for explaining an example of remote control of a first wearable device (2000a) according to various embodiments.
[0286] According to various embodiments, the first wearable device (2000a) may establish a communication connection with an external device (e.g., a second wearable device (2000b) and / or a user terminal (30)) in operation 2901, and receive control information from the external device (e.g., the second wearable device (2000b) and / or the user terminal (30)) in operation 2903.
[0287] According to various embodiments, the first wearable device (2000a) may perform at least one operation based on the control information in operation 2905. In one embodiment, the first wearable device (2000a) may perform an operation of providing information about the first wearable device (2000a) to the external device (e.g., the second wearable device (2000b) and / or the user terminal (30)). For example, as illustrated in (a) of FIG. 30, based on the execution of an application (e.g., the first application (900)) installed in the second wearable device (2000b), the second wearable device (2000b) may provide an execution screen including information (3000a) about the second wearable device (2000b) (e.g., battery information, communication connection status information). At this time, the first wearable device (2000b) may provide information (3000a) (e.g., battery information, communication connection status information) about the second wearable device (2000b) to the second wearable device (2000b) based on a request from the second wearable device (2000b) so that the information is provided on the execution screen. In addition, in one embodiment, the first wearable device (2000a) may perform an operation based on a control signal of the external device (e.g., the second wearable device (2000b) and / or the user terminal (30)). For example, as illustrated in (b) of FIG. 30, the first wearable device (2000a) may perform a photographing operation using the camera (3000b) based on a camera control signal received from the second wearable device (2000b).
[0288]
[0289] 3.2.2. Mode of add-on wearable devices
[0290] According to various embodiments, an add-on wearable device (e.g., a first wearable device (2000a)) may be implemented to be configured to have multiple modes that perform different functions. Based on the multiple modes, the operating burden of the add-on wearable device (e.g., the first wearable device (2000a)) may be reduced, thereby reducing the power consumption of a battery (e.g., the first battery (2124)).
[0291] FIG. 31 is a flowchart illustrating examples of operations for multiple modes of an add-on wearable device (e.g., the first wearable device (2000a)) according to various embodiments. The operations may be performed regardless of the order of the operations illustrated and / or described, and more operations may be performed and / or fewer operations may be performed. Below, FIG. 31 will be further described with reference to FIG. 32.
[0292] FIG. 32 is a diagram illustrating examples of multiple modes of an add-on wearable device (e.g., a first wearable device (2000a)) according to various embodiments.
[0293] According to various embodiments, the first wearable device (2000a) may identify the occurrence of an event in operation 3101. The first wearable device (2000a) may determine whether the event generated in operation 3103 corresponds to an activation event, and if it corresponds to the activation event (3103-Y), may identify the occurrence stage of the activation event in operation 3105. For example, referring to (a) of FIG. 32, the mode (3201) of the first wearable device (2000a) may include a sleep mode and an activation mode, and the activation mode may include a plurality of activation stages (e.g., a first activation stage and a second activation stage).
[0294] According to various embodiments, the degree of operation (or the degree of activation of electronic components) of the first wearable device (2000a) may be different for each mode of the first wearable device (2000a). In one embodiment, the degree of operation (or the degree of activation) may be set to be greater in the activation mode compared to the sleep mode. For example, referring to (b) of FIG. 32, when the mode of the first wearable device (2000a) is set to the sleep mode, only the communication circuit (3205) of the first wearable device (2000a) is activated in a preset first cycle, so that the first wearable device (2000a) can establish a communication connection with an external device (e.g., a second wearable device (2000b), a user terminal (30), and / or a wireless communication device (40)) in the first cycle for a specific period of time. Accordingly, based on the communication connection established for the specific time, the first wearable device (2000a) can intermittently exchange data (or information) with an external device (e.g., a second wearable device (2000b), a user terminal (30), and / or a wireless communication device (40)). For example, referring to (b) of FIG. 32, when the mode of the first wearable device (2000a) is set to the first activation stage, the activation state of the communication circuit (3205) of the first wearable device (2000a) is maintained (or activated in a second cycle shorter than the first cycle), so that the first wearable device (2000a) can continuously maintain a communication connection with an external device (e.g., a second wearable device (2000b), a user terminal (30), and / or a wireless communication device (40)). Accordingly, the first wearable device (2000a) can be controlled to react in real time to data received from the external device based on the communication settings.For example, referring to (b) of FIG. 32, when the mode of the first wearable device (2000a) is set to the second activation stage, the activation state of the communication circuit (3205) and the camera (3203) of the first wearable device (2000a) is maintained, so that the first wearable device (2000a) can transmit an image captured by the camera (3203) to an external device (e.g., the second wearable device (2000b), the user terminal (30), and / or the wireless communication device (40)).
[0295] According to various embodiments, the mode of the first wearable device (2000a) can be switched based on an event that has occurred, and when an activation event has occurred, the mode of the first wearable device (2000a) can be set to a first activation stage or a second activation stage, and when a deactivation event has occurred, the mode of the first wearable device (2000a) can be set to a sleep mode.
[0296] For example, the activation event may include a first activation event for transitioning from a sleep mode to a first activation stage, and the first activation event may occur when an activation signal is received, but is not limited to the described example.
[0297] For example, the activation event may include a second activation event for transitioning from the first activation stage to the second activation stage, and the second activation event may occur when the shielding structure of the camera is opened, but is not limited to the described example.
[0298] For example, the activation event may include a first deactivation event for switching from an activation mode (e.g., a first activation step, and a second activation step) to a sleep mode, and the first deactivation event may occur when a preset period of time has elapsed from the time an activation signal is received, but is not limited to the described examples.
[0299] Meanwhile, without being limited to the described example, an event may occur to transition from the second activation step to the first activation step.
[0300] According to various embodiments, the first wearable device (2000a) may determine whether the generation step of the activation event corresponds to the first step in operation 3107, and if the generation step of the activation event does not correspond to the first step (3107-N), determine whether the generation step of the activation event corresponds to the second step in operation 3109. According to various embodiments, the first wearable device (2000a) may maintain a communication connection state with the second wearable device (2000b) in operation 3109 if the activation step is the first step (3107-Y), and may activate the camera while maintaining a communication connection state with the second wearable device (2000b) in operation 3113 if the activation step is the second step (3111-Y).
[0301]
[0302] 3.2.3. Complementary actions
[0303] According to various embodiments, when one of the first wearable device (2000a) and the second wearable device (2000b) is powered off, the other wearable device can perform an operation in its place.
[0304] FIG. 33 is a flowchart illustrating an example of operations of a wearable device when one of the first wearable device (2000a) and the second wearable device (2000b) is powered off, according to various embodiments. The operations may be performed regardless of the order of the operations illustrated and / or described, and more operations may be performed and / or fewer operations may be performed.
[0305] According to various embodiments, referring to (a) of FIG. 33, when the second wearable device (2000b) is powered off, the first wearable device (2000a) may store and accumulate a medication performance image, and when the second wearable device (2000b) is powered on, the accumulated medication performance image may be transmitted to the second wearable device (2000b). The first wearable device (2000a) and the second wearable device (2000b) may establish a communication connection in operation 3301, and periodically exchange status information in operation 3303. Accordingly, each wearable device (e.g., the first wearable device (2000a) and the second wearable device (2000b)) may periodically identify the power status of the other wearable device based on the status information. When the second wearable device (2000b) is identified as being powered off, the first wearable device (2000a) performs a camera shooting operation based on an activation signal in operation 3305a and accumulates the captured images, and when the second wearable device (2000b) is powered on in operation 3307a, transmits the accumulated images to the second wearable device (2000b), so that a medication judgment can be performed based on the images captured while the second wearable device (2000b) was powered off through the server (10). Meanwhile, without being limited to the illustrated and / or described examples, the first wearable device (2000a) may be implemented to directly establish a communication connection with the server (10) and / or the user terminal (30) and transmit the captured image to the server (10) and / or the user terminal (30) when the second wearable device (2000b) is identified as being powered off.
[0306] According to various embodiments, referring to (b) of FIG. 33, when the first wearable device (2000a) is powered off, the second wearable device (2000b) can obtain information on medication performance using at least one sensor included in the second wearable device (2000b). When the first wearable device (2000a) is identified as being powered off, the second wearable device (2000b) can accumulate information on medication performance using at least one sensor based on receiving activation data from the wireless communication device (40) in operation 3305b (e.g., establishing a communication connection with the wireless communication device (40) or scanning a signal broadcast from the wireless communication device (40), and can perform a medication judgment based on the accumulated information in operation 3307b. In other words, if the second wearable device (2000b) is identified as being powered off from the first wearable device (2000a), it may initiate an operation to acquire information on medication performance using a sensor implemented by itself instead of the first wearable device (2000a). An example of an operation to acquire information on medication performance using a sensor implemented by itself instead of the first wearable device (2000a) is specifically described in “4.2. Non-Vision Algorithm.”
[0307]
[0308] 3.2.4. Privacy Protection Behavior
[0309] According to various embodiments, if the first wearable device (2000a) is lost, the lost first wearable device (2000a) may be used by another user, thereby misusing personal information. As described above, the second wearable device (2000b) can prevent the problem of misuse due to loss by authenticating the first wearable device (2000a) based on registered wearable identification information and user identification information.
[0310] FIG. 34 is a flowchart illustrating an example of the operation of a health management system (2000) when a first wearable device (2000a) is lost, according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed.
[0311] According to various embodiments, the first wearable device (2000a) and the second wearable device (2000b) may establish a communication connection in operation 3401 and exchange identification information (e.g., wearable identification information and user identification information) with each other in operation 3403.
[0312] According to various embodiments, if the identification information of the first wearable device (2000a) (e.g., a strap) is recognized in operation 3405, the second wearable device (2000b) may normally execute a medication management app (e.g., the first application (900)). For example, as described above, the second wearable device (2000b) (or the server (10) or the user terminal (30)) may register (or store) the identification information (e.g., wearable identification information) of the first wearable device (2000a) used by the user and the user identification information in a form that is associated with each other. After the registration, if the identification information of the first wearable device (2000a) is received, the second wearable device (2000a) may be implemented to normally execute the first application (900) and normally provide a medication management service if it is authenticated based on a comparison with the registered information.
[0313]
[0314] 4. Hybrid medication decision operation
[0315] FIG. 35 is a block diagram for explaining an operation of performing medication determination by further utilizing at least one sensor (3803) of a wearable device (20), according to various embodiments.
[0316] According to various embodiments, referring to FIG. 35, the wearable device (20) may activate a camera (3801) to capture an image of medication performance based on a value identified using at least one sensor (3803) without the need for an activation signal received from a wireless communication device (40) (vision activation), and / or may use the acquired information for at least a part of an operation of obtaining information on medication performance and obtaining medication judgment result information using at least one sensor (3803) (non-vision algorithm).
[0317] According to various embodiments, the at least one sensor (3803) may include a motion sensor for sensing the movement of an object, a microphone for sensing sounds generated externally, a vibration sensor for measuring vibration of an object, a magnetic field sensor for measuring a surrounding magnetic field, a proximity sensor for measuring the degree of proximity to an object, a physical information acquisition sensor such as an ultrasonic sensor, a biometric information acquisition sensor such as an electromyogram sensor, and a vision-limited sensor such as a lidar, an IR sensor, or other optical sensors. When a medication-taking image is captured by the camera (3801), a privacy issue may arise as sensitive objects that may cause personal information leakage are captured by the capture. However, the at least one sensor (3803) has an advantage in that personal information leakage can be prevented, unlike the camera (2801).
[0318] According to various embodiments, the at least one sensor (3803) may be a sensor included in a second wearable device (2000b) (e.g., body (2110)) coupled with an add-on wearable device (e.g., first wearable device (2000a)) described in the above-described “3. Add-on wearable”, but is not limited to the described example and may be a sensor included in a single wearable device (20) (e.g., smart watch).
[0319]
[0320] 4.1. Vision Activation Algorithm
[0321] FIG. 36 is a diagram illustrating activation signal reference conditions for each wearable sensor in some embodiments of the present disclosure, FIG. 37 is a flowchart illustrating a process of generating an activation signal using a wearable sensor in some embodiments of the present disclosure, and FIG. 38 is a flowchart illustrating a process of generating an activation signal using a wearable sensor in some embodiments of the present disclosure. Hereinafter, a method of generating an activation signal will be described with reference to FIGS. 36 to 38.
[0322] The wearable sensor is a sensor that is provided on one side of a wearable device (20) and generates an activation signal for a camera module by sensing a medication target (e.g., target device (T)). The wearable sensor may be a sensor of various types that senses a medication target (e.g., target device (T)) using a non-vision method.
[0323] Referring to FIG. 36, the wearable sensor may be a motion sensor that generates an activation signal based on a user's movement, a pressure sensor that generates an activation signal based on pressure data, a sound detection sensor that generates an activation signal based on sound data, a vibration sensor that generates an activation signal based on vibration data, an electromyography (EMG) sensor that generates an activation signal based on electromyography data, a hall sensor that generates an activation signal based on three-dimensional magnetic data, an infrared ray (IR) sensor that generates an activation signal based on electromagnetic waves in the infrared region, a lidar that generates an activation signal based on laser pulses, and a photoplethysmography (PPG) sensor that generates an activation signal based on heart rate data. The medication target may be various types of medicine containers such as a solid pill bottle, a liquid medicine bottle, a syringe, a sachet, an inhaler, and a spray.
[0324] In some embodiments, the medication target (e.g., target target (T)) may be individually corresponding to the function of each wearable sensor. For example, a medication object sensed by a motion sensor (e.g., target object (T)) may be at least one of a solid pill bottle, a liquid pill bottle, a syringe, a sachet, an inhaler, and a spray capable of motion recognition, a medication object sensed by a pressure sensor (e.g., target object (T)) may be at least one of a liquid pill bottle, a sachet, a syringe, and a spray capable of pressure change detection, a medication object sensed by a sound detection sensor (e.g., target object (T)) may be at least one of a solid pill bottle, a sachet, and a spray capable of generating noise when taking medication, a medication object sensed by a vibration sensor (e.g., target object (T)) may be at least one of a solid pill bottle, a sachet, an inhaler, and a spray capable of generating vibration when taking medication, a medication object sensed by an EMG sensor (e.g., target object (T)) may be at least one of a liquid pill bottle, a sachet, a syringe, and a spray capable of acquiring electromyography data, and a medication object sensed by a Hall sensor (e.g., The target object (T)) may be a solid pill bottle, a liquid pill bottle, a syringe, a sachet, an inhaler, and a spray capable of acquiring three-dimensional magnetic data, the medication object (e.g., the target object (T)) sensed by the infrared sensor and the lidar may be at least one of a solid pill bottle, a liquid pill bottle, a syringe, a sachet, an inhaler, and a spray capable of measuring distance, and the medication object (e.g., the target object (T)) sensed by the PPG sensor may be at least one of a solid pill bottle, a liquid pill bottle, a syringe, a sachet, an inhaler, and a spray capable of measuring heart rate.
[0325] Referring to FIGS. 37 and 38, in step S4010, the wearable device (20) sets reference values and reference conditions for generating activation signals for each wearable sensor for various medication subjects (e.g., target subjects (T)) through at least one wearable sensor.
[0326] As described above, each of the multiple medication targets (e.g., target target (T)) corresponds to each other according to the function of each wearable sensor. Since the function of each wearable sensor is different, the set reference values and reference conditions may also vary for each wearable sensor. Below, the reference values and reference conditions set for each wearable sensor will be explained with examples.
[0327] Referring again to Figure 36, the motion sensor detects motion recognition related to the user taking medication, so the wearable sensor can set the occurrence of medication motion based on the user's hand movement as a reference condition. The pressure sensor detects changes in pressure, so the amount of pressure change in the wrist that occurs when taking medication (e.g., 5 Pa) can be set as a reference condition. In the case of a sound detection sensor, the size of the sound generated when taking medication (e.g., 30 dB) can be set as a reference condition, in the case of a vibration sensor, the vibration intensity or vibration frequency (e.g., 3 Hz) due to taking medication can be set as a reference condition, in the case of an EMG sensor, the amount of electromyography change in the wrist when taking medication (e.g., 1 mV) can be set as a reference condition, in the case of a Hall sensor, the displacement value according to the movement of the hand position when taking medication (e.g., 30 cm) can be set as a reference condition, in the case of an infrared sensor or a lidar, the distance (e.g., 10 cm) between the wearable device (20) and the medication subject (e.g., target subject (T)) can be set as a reference condition, and in the case of a PPG sensor, the heart rate (e.g., 100 bpm) when taking medication can be set as a reference condition.
[0328] In step S4020, when a medication event occurs, the wearable device (20) detects the user's medication event through at least one wearable sensor and obtains medication data generated as a sensing result. The medication data may include different information and may be different types of data depending on the wearable sensor. For example, the wearable device (20) may obtain motion data from a motion sensor, pressure data from a pressure sensor, sound data from a sound detection sensor, vibration data from a vibration sensor, electromyography data from an EMG sensor, 3D position data of a medication target (e.g., target target (T)) from a Hall sensor, distance data between the wearable device (20) and the medication target (e.g., target target (T)) (e.g., target device (T)) from an infrared sensor or a lidar, and heart rate data from a PPG sensor.
[0329] The wearable device (20) compares the numerical value of the acquired medication data with a preset reference value (step S4030), determines whether the result of the comparison satisfies the reference condition (step S4040), and if the reference condition is satisfied, generates an activation signal (step S4050). For example, if the medication data is pressure data with a pressure change amount of 5 Pa, the preset reference value is 4 Pa, and the reference condition is 'greater than the reference value', when the pressure change amount of the medication data is compared with the reference value, the reference condition is satisfied, so the wearable device (20) generates an activation signal. The generated activation signal is transmitted to the camera module to control the camera to be activated.
[0330] In this way, according to the embodiments of FIGS. 36 to 38, the wearable device (20) compares medication data acquired through a non-vision wearable sensor with a reference value through a processor within the wearable device (20), and immediately generates an activation signal based on the comparison result, thereby enabling the camera to be quickly activated without a time delay due to data transmission and reception with the server (10).
[0331]
[0332] In contrast, in various embodiments, the wearable device (20) may transmit the acquired medication data to the server (10). That is, the wearable device (20) may identify a medication target (e.g., target target (T)) corresponding to the medication data using an artificial intelligence model, and compare the medication data with preset reference conditions for each type of the identified medication target (e.g., target target (T)).
[0333] In this case, rather than comparing the numerical value of medication data with the reference value based solely on the type of wearable sensor, the type of wearable sensor and the type of medication subject (e.g., target subject (T)) are both considered to compare the medication data with the reference value for each medication subject (e.g., target subject (T)). This not only improves the accuracy of medication event judgment, but also improves the efficiency of activation signal generation.
[0334] FIG. 39 is a diagram showing various types of wearable sensors and a medication target (e.g., target target (T)) corresponding to each wearable sensor and an activation signal reference condition according to some embodiments of the present disclosure, FIG. 40 is a flowchart showing a process of identifying a medication target (e.g., target target (T)) and generating an activation signal through a wearable sensor and a server (10) according to some embodiments of the present disclosure, and FIG. 41 is a flowchart showing a process of identifying a medication target (e.g., target target (T)) and generating an activation signal through a wearable sensor and a server (10) according to some embodiments of the present disclosure. Hereinafter, a method of generating an activation signal will be described with reference to FIGS. 39 to 41.
[0335] First, referring to FIGS. 40 and 41, in step S4310, the wearable device (20) sets a reference value and a reference condition for generating an activation signal for each wearable sensor for various medication subjects (e.g., target subjects (T)) through at least one wearable sensor. In addition, the wearable device (20) can set a reference value and a reference condition for generating an activation signal for each medication subject (e.g., target subjects (T)) as well as for each wearable sensor based on the wearable sensors sensing different medication subjects (e.g., target subjects (T)).
[0336] In various embodiments, as described above, each of the plurality of medication subjects (e.g., target subjects (T)) corresponds to each other according to the function of each wearable sensor, and as the function of each wearable sensor differs, the set reference values and reference conditions may also differ for each wearable sensor. Hereinafter, the set reference values and reference conditions for each wearable sensor and medication subject (e.g., target subjects (T)) will be described with examples.
[0337] Referring to Fig. 39, the medication object (e.g., target object (T)) sensed by the motion sensor may be a solid pill bottle, a liquid pill bottle, a syringe, a sachet, an inhaler, and a spray capable of motion recognition. At this time, the reference value (reference motion) for the motion sensor may be a different motion depending on the type of the medication object (e.g., target object (T)), such as a motion of spreading the palm upward to receive a pill discharged from a solid pill bottle, a hand motion of holding a liquid pill bottle, a motion of spreading the thumb and index / ring finger to inject the syringe into the body, a motion of tearing the sachet of a sachet, and a motion of bringing the medication object (e.g., target object (T)) such as an inhaler and spray to the face area (e.g., eyes, nose, mouth) for using the inhaler and spray. The reference condition may be that each motion according to the medication object (e.g., target object (T)) continues for a specific period of time or longer, or that the reliability calculated from each continuous motion is higher than a predetermined reliability level.
[0338] The medication object sensed by the pressure sensor (e.g., target object (T)) may be at least one of a pill bottle, a sachet, a syringe, and a spray capable of detecting pressure changes. In this case, the reference value for the pressure sensor may be a different wrist pressure change value according to muscle contraction in various actions, such as an action of holding a solid pill bottle or a liquid pill bottle, an action of injecting a drug through a syringe, an action of tearing a sachet from a sachet, and an action of pressing a spray firing button. The reference condition for the pressure sensor may be that the calculated wrist pressure change value is greater than the reference value for each action.
[0339] The medication object sensed by the sound detection sensor (e.g., target object (T)) may be at least one of a solid pill bottle, a sachet, and a spray that generates noise when the medication is taken. At this time, the reference value for the sound detection sensor may be the size, frequency, and waveform of various sounds typically generated from each medication object (e.g., target object (T)), such as the sound of pills colliding in a solid pill bottle, the sound of opening the lid of the pill bottle, the sound of tearing a sachet, and the sound of spray being sprayed. The reference condition for the sound detection sensor may be that the size of the acquired noise is higher than the reference value for each medication object (e.g., target object (T)).
[0340] The medication object sensed by the vibration sensor (e.g., target object (T)) may be at least one of a solid pill bottle, a packet of pills, an inhaler, and a spray that generates vibrations when taking the pills. At this time, the reference value for the vibration sensor may be the waveform, intensity, and frequency of various vibrations generated by each medication object (e.g., target object (T)), such as the vibration generated when taking a pill out of a solid pill bottle, the vibration generated when shaking a packet to align the pills inside the packet before tearing the packet, and the vibration generated by the shaking motion before using the inhaler or spray. The reference condition for the vibration sensor may be that the vibration waveform, intensity, and frequency of the acquired vibration data are equal to or higher than the reference value for each medication object (e.g., target object (T)).
[0341] The medication subject sensed by the EMG sensor (e.g., target subject (T)) may be at least one of a pill bottle, a sachet, a syringe, and a spray from which electromyography data can be acquired. At this time, the reference value for the EMG sensor may be a wrist electromyography change value according to muscle contraction in various actions, such as a motion of holding a solid pill bottle or a liquid pill bottle, a motion of injecting a drug through a syringe, a motion of tearing a sachet from a pill bag, and a motion of pressing a spray firing button, and the reference condition for the EMG sensor may be that the wrist electromyography change value of the acquired electromyography data is equal to or higher than the reference value for each medication subject (e.g., target subject (T)).
[0342] The Hall sensor measures the displacement value according to the position movement of the wrist, and senses the medication object (e.g., target object (T)) based on the displacement value. The medication object (e.g., target object (T)) sensed by the Hall sensor may be a solid pill bottle, a liquid pill bottle, a syringe, a sachet, an inhaler, and a spray, which can acquire three-dimensional magnetic data. At this time, the reference value for the Hall sensor may be a displacement value. Specifically, the displacement value of the solid pill bottle is small because the position movement occurs around the hand, the displacement value of the syringe is medium because the position movement occurs around the wrist, etc., and the displacement value of the inhaler, spray, and liquid may be high because the position movement occurs in the face area. In addition, the reference condition for the Hall sensor may be that the displacement value of the acquired three-dimensional magnetic data is higher than the reference value for each medication object (e.g., target object (T)).
[0343] The medication target (e.g., target target (T)) sensed by the infrared sensor and the lidar may be at least one of a solid pill bottle, a liquid pill bottle, a syringe, a sachet, an inhaler, and a spray capable of measuring distance. In this case, the reference value for the infrared sensor and the lidar may be a preset distance between the wearable device (20) and the medication target (e.g., target target (T)), and the reference condition for the infrared sensor and the lidar may be that the determined distance is greater than or equal to the reference value.
[0344] The medication object sensed by the PPG sensor (e.g., target object (T)) may be at least one of a solid pill bottle, a liquid pill bottle, a syringe, a sachet, an inhaler, and a spray. In this case, the reference value for the PPG sensor may be a heart rate, and the reference range may be that the acquired heart rate is greater than or equal to the reference value.
[0345] In step S4320, the wearable device (20) detects a user's medication event using at least one wearable sensor, and when the medication data is obtained, in step S4330, the obtained medication data is transmitted to a server to identify a medication target (e.g., target target (T)) through an artificial intelligence (AI) model of the server.
[0346] Next, the server (10) compares the numerical value included in the medication data with a reference value corresponding to the identified medication subject (e.g., target subject (T)) to determine whether a medication event has occurred and the type of medication (step S4340), and if the reference condition is satisfied (step S4350), generates an activation signal by transmitting an activation signal generation request to the wearable device (20) (step S4360).
[0347] Unlike the previously described embodiment, in this embodiment, since medication data analysis is performed through the server's artificial intelligence model, the identification of the medication subject (e.g., target subject (T)) is possible. Therefore, the reference value and reference conditions for generating an activation signal are set not only for each wearable sensor, but also for each medication subject (e.g., target subject (T)). This allows for confirmation of not only the user's medication timing but also the type of restrictions being applied, thereby improving the efficiency of activation signal generation.
[0348]
[0349] 4.2. Non-vision algorithms
[0350] 4.2.1. Single medication decision operation
[0351] According to various embodiments, the health management system (1) can obtain information on medication performance by using at least one sensor (3803) of the wearable device (20), and provide a medication judgment result based on the obtained information on medication performance. The health management system (1) can obtain at least one artificial intelligence model by performing learning using sensor values and medication judgment result information corresponding to the sensor values as training data, and perform a medication judgment operation based on the at least one artificial intelligence model, which will be described later with reference to FIGS. 42 to 48.
[0352] FIG. 42 is a flowchart illustrating an example of operations for providing medication determination results based on at least one sensor (4620) of a wearable device (20), according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed.
[0353] FIGS. 43 to 44 are drawings for explaining an operation of obtaining a medication judgment result based on a sensor value of a health management system (1) according to various embodiments.
[0354] According to various embodiments, the wearable device (20) may obtain activation data in operation 4501. In one embodiment, the wearable device (20) may receive, from the wireless communication device (40), activation data generated by the wireless communication device (40) based on the movement of the target object (T) when the user takes medication. In addition, in one embodiment, the wearable device (20) may obtain activation data by using at least one sensor (4620) (e.g., at least one sensor (3803)), as described in “4.2.2. Sensor-based medication determination operation”.
[0355] According to various embodiments, the wearable device (20) may activate at least one sensor (4620) having a pre-set authority for medication determination among a plurality of sensors in operation 4503, and obtain at least one value using at least one sensor (4620) of the wearable device in operation 4505. For example, the wearable device (20) may activate at least one sensor (4620) (e.g., a physical information acquisition sensor, a biometric information acquisition sensor, a vision information limitation sensor) set to obtain information on medication performance among a plurality of sensors of the wearable device (20), and obtain at least one sensor value using the activated at least one sensor (3803). An application (e.g., first application (900)) running on the wearable device (20) may be set to have permission to use at least one sensor (4620) (e.g., physical information acquisition sensor, biometric information acquisition sensor, vision information limitation sensor) based on the consent of the user of the wearable device (20).
[0356] According to various embodiments, the wearable device (20) may be implemented to acquire non-vision information using at least one sensor (4620), as illustrated in FIG. 43. For example, as illustrated in (a) of FIG. 44, the wearable device (20) may acquire movement (or vibration) generated by the user's medication administration during medication administration using a physical information acquisition sensor (e.g., a motion sensor, a vibration sensor, etc.), and / or may acquire the user's biometric information (e.g., electromyography information) that changes due to the user's medication administration using a biometric information sensor. In addition, as illustrated in (b) of FIG. 44, the wearable device (20) may acquire sound generated by the user's medication administration during medication administration through a microphone. Although not illustrated, the wearable device (20) may also acquire information about medication administration using a limited vision sensor (e.g., a lidar, an IR sensor, etc.).
[0357] According to various embodiments, as described above, the at least one sensor (4620) may be a sensor included in a second wearable device (2000b) (e.g., body (2110)) coupled with an add-on wearable device (e.g., first wearable device (2000a)) described in “3. Table of Contents” described above, but is not limited to the described example and may be a sensor included in a single wearable device (20) (e.g., smart watch).
[0358] According to various embodiments, the wearable device (20) may control, in operation 4507, to transmit at least one value to an external source (e.g., a server (10), a user terminal (30)) so that information on a medication-taking decision result is acquired based on at least one artificial intelligence model (e.g., at least one of a classification model (610), a detection model (620), or a confirmation model (630)). For example, the server (10) may receive at least one value acquired by the wearable device (20), and may acquire the medication-taking decision result information in response to inputting the received at least one value into at least one pre-trained artificial intelligence model (e.g., at least one of a classification model (610), a detection model (620), or a confirmation model (630)) to acquire a medication-taking decision result based on non-vision information. The medication-taking decision result information may be provided through the user terminal (30) and / or the wearable device (20) as described above.
[0359] Below, examples of learning and / or using at least one artificial intelligence model to provide medication decision results based on non-vision information of a server (10) according to various embodiments are described.
[0360]
[0361] 4.2.2. Medication decision-making based on daily sensors
[0362] FIGS. 45 to 48 are diagrams for explaining examples of learning an artificial intelligence model for one sensor-based medication judgment, and a medication judgment operation based on the learned artificial intelligence model, according to various embodiments.
[0363] According to various embodiments, the server (10) may learn at least one artificial intelligence model (e.g., a classification model (620), a confirmation model (630), or a single artificial intelligence model (not shown)) based on sensor values received from sensors included in the wearable device (20) (e.g., a motion sensor (e.g., an acceleration sensor), an electromyography sensor, a microphone, a magnetic field sensor, or other sensors (e.g., a vibration sensor, an ultrasonic sensor) not shown). Based on the at least one artificial intelligence model, a medication determination operation based on the A identified from the sensor may be performed.
[0364]
[0365] 4.2.2.1. Multiple Artificial Intelligence Model Examples
[0366] Below, the description is given with reference to FIGS. 45 to 47.
[0367] According to various embodiments, the server (10) may sequentially (or time-seriesly) generate at least one artificial intelligence model based on the feature values of sensor values from the sensors (e.g., motion sensors (e.g., acceleration sensors), electromyography sensors, microphones, magnetic field sensors, or other sensors (e.g., vibration sensors, ultrasonic sensors)) of the wearable device (20) as the user takes medication while wearing the wearable device (20). For example, referring to FIGS. 45 and 46, as the user takes medication while wearing the wearable device (20), sensor values (e.g., angular velocity values for each axis, electromyography values, sound values, magnetic field values) may be sequentially (or time-seriesly) obtained from the sensors (e.g., motion sensors (e.g., acceleration sensors), electromyography sensors, microphones, magnetic field sensors, or other sensors (e.g., vibration sensors, ultrasonic sensors)) of the wearable device (20). Feature information (C1, C2, C3, C4) can be extracted (or labeled) for each time frame (TF1, TF2) of the sequential sensor values obtained above. The feature information (C1, C2, C3, C4) can represent information that is a feature of the sensor value, such as a characteristic value of the sensor value, an inflection point of the sensor value, and a slope of the sensor value. In one embodiment, as illustrated in FIG. 47, the server (10) can generate a classification model (610) by performing learning using the extracted (or labeled) feature values (C1, C2, C3, C4) of the sensor value as input data and a category index indicating a category of a medication behavior actually performed by the user as output data. The classification model (610) can be implemented to output the category index based on the sequential sensor data obtained by the wearable device (20) being input.In another embodiment, the server (10) may generate a detection model (620) by performing learning using the feature values (C1, C2, C3, C4) for each of the time frames (TF1, TF2) as input data and the corresponding medication administration actions (O1, O2) performed by the user for each of the time frames (TF1, TF2) as output data. For example, if the category of medication administration is an action of taking a pill from a medicine bottle, the first time frame (TF1) may correspond to the first medication administration action (O1) (e.g., an action of taking a pill), and the second time frame (TF2) may correspond to the second medication administration action (O2) (e.g., an action of picking up a pill). That is, the feature information (C1, C2, C3, C4) for each of the time frames (TF1, TF2) may represent the medication administration actions (O1, O2). The above detection model (620) can be implemented to output probability information of drug delivery motion for each time frame based on the sequential motion data acquired by the wearable device (20) as shown in FIG. 47.
[0368] According to various embodiments, the result information output from the detection model may vary based on the category index. As described above, since different detection models are implemented and selected for each category index, and / or the detection model (610) is implemented so that the category index can be input together with motion data, the probability information on drug implementation motion obtained from the detection model (620) may vary depending on the category index.
[0369] According to various embodiments, the server (10) may learn a confirmation model (630) implemented to output medication judgment result information indicating whether or not a specific category of medication has been taken, in response to inputting probability information of a medication implementation operation, as illustrated in FIG. 47.
[0370] Consequently, according to various embodiments, the server (10) can obtain medication decision result information based on inputting sequential sensor values received from the wearable device (20) into at least some of the plurality of artificial intelligence models (e.g., classification model (610), detection model (620), and confirmation model (630)).
[0371]
[0372] 4.2.2.2. Single AI Model Example
[0373] Meanwhile, without being limited to the above-described example and not shown, the server (10) may perform learning by using the characteristic information (C1, C2, C3, C4) for each time frame (TF1, TF2) of the sensor value as input data and setting the medication judgment result as output data, thereby implementing a single artificial intelligence model (not shown) that outputs the medication judgment result in response to inputting motion data. In this case as well, a classification model may be separately implemented, and the artificial intelligence model (not shown) may be implemented to provide different results depending on the identification information (e.g., category index) for the category of the medication judgment, but is not limited to the described example.
[0374]
[0375] 4.2.3. Medication Decision Action Based on Multiple Sensors
[0376] FIG. 49 is a diagram for explaining an example of learning an artificial intelligence model for medication decision based on multiple sensors according to various embodiments, and a medication decision operation based on the learned artificial intelligence model.
[0377] According to various embodiments, as illustrated in FIG. 49, the server (10) may perform an operation of learning at least one artificial intelligence model for providing a medication decision result based on sensor values acquired from a plurality of sensors (e.g., motion sensor (4801), electromyography sensor (4803)) of a wearable device (20), and acquiring medication decision result information based on the at least one pre-learned artificial intelligence model. At this time, learning of the at least one artificial intelligence model may be performed as described above in “4.2.2. One sensor-based medication decision operation,” and therefore, a redundant description thereof will be omitted.
[0378] According to various embodiments, referring to FIG. 49, the server (10) may, in comparison with “4.2.2. One-sensor based medication decision operation”, when learning an artificial intelligence model, use different types of sensor values (e.g., motion data and electromyography data) simultaneously acquired from each of a plurality of sensors (e.g., motion sensor (4801), electromyography sensor (4803)) when the user takes medication as input data. For each sensor value, feature information (C11, C12, C21, C22, C13, C14, C23, C24) may be extracted for each time frame (TF1, TF2). In addition, extraction (labeling) of feature values and learning of a classification model and / or a detection model based on the corresponding medication implementation operation may be performed in the same manner as in “4.2.2. One-sensor based medication decision operation”, and therefore, redundant descriptions thereof will be omitted.
[0379]
[0380] 4.2.4. Compound medication judgment operation
[0381] According to various embodiments, the server (10) can obtain medication judgment result information based on a medication performance image captured by a camera (4610) of a wearable device (20) and a sensor value obtained from the aforementioned sensor (4620).
[0382] FIG. 49 is a flowchart illustrating an example of a composite medication determination operation of a wearable device (20) according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. Below, FIG. 49 will be further described with reference to FIG. 50.
[0383] FIG. 50 is a drawing for explaining an example of a composite medication judgment operation of a server (10) according to various embodiments.
[0384] According to various embodiments, the wearable device (20) may obtain activation data in operation 5201.
[0385] According to various embodiments, the wearable device (20) may obtain a sensor value for each time frame using at least one sensor (4620) of the wearable device (20) in operation 5203, and may obtain at least one image for each time frame using the camera (4610) of the wearable device (20) in operation 5205. For example, as illustrated in FIG. 50, the wearable device (20) may simultaneously obtain a medication performance image and a sensor value based on activating the camera (4610) and the at least one sensor (4620) based on receiving the activation data.
[0386] According to various embodiments, the wearable device (20) may, in operation 5207, transmit the sensor value and the at least one image to an external device (e.g., a server (10) and / or a user terminal (30)) so as to obtain information on a medication decision result based on at least one artificial intelligence model.
[0387]
[0388] 4.2.4.1. Judgment of this time
[0389] According to various embodiments, the server (10) may be implemented to utilize different artificial intelligence models for the sensor values and at least one image acquired by the wearable device (20). For example, the server (10) may obtain final medication decision result information based on result information acquired in response to inputting the sensor values or at least one image to the artificial intelligence model, as shown in [Table 1] below.
[0390] Classification Classification Model Detection Model Example 1 Specific sensor value Medication performance image Example 2 Medication performance image Specific sensor value
[0391] As an example, in the case of the first embodiment of [Table 1], the server (10) selects a detection model based on a category index output based on the input of the specific sensor value into the classification model, and inputs information about the target object (T) output based on the input of the medication performance image into the detection model into the confirmation model, thereby obtaining medication judgment result information as a result. As another example, in the case of the second embodiment of [Table 1], the server (10) selects a detection model based on a category index output based on the input of the medication performance image into the classification model, and inputs probability information about the medication implementation operation output based on the input of the sensor value into the detection model into the confirmation model, thereby obtaining medication judgment result information as a result.
[0392] 4.2.4.2. Simultaneous judgment
[0393] According to various embodiments, the server (10) may be implemented to use the sensor value and the at least one image acquired by the wearable device (20) by inputting them together into a specific artificial intelligence model. To this end, each of the classification model and the detection model may use the medication performance image and the corresponding sensor value as input data. At this time, learning of the classification model and the detection model may be performed based on the labeling of the medication performance image for each time frame and the feature values of the sensor value for each time frame. Since the learning operation for each model may be performed as described above, a detailed description thereof will be omitted.
[0394]
[0395] 4.2.5. Medication Decision Scenario
[0396] FIG. 51 is a flowchart illustrating an example of a medication judgment scenario of a wearable device (20) according to various embodiments. The operations may be performed regardless of the order of the operations illustrated and / or described, and more operations may be performed and / or fewer operations may be performed.
[0397] According to various embodiments, the wearable device (20) may obtain activation data in operation 5401.
[0398] According to various embodiments, the wearable device (20) determines whether a camera activation condition is satisfied in operation 5403, and if the camera activation condition is not satisfied (5403-N), activates at least one sensor pre-configured (authorized) for medication determination among a plurality of sensors in operation 5405, acquires at least one value using at least one sensor of the wearable device in operation 5407, and transmits at least one value to the outside in operation 5409, so that information on a medication determination result is obtained based on at least one artificial intelligence model. That is, in a situation where it is difficult to use the camera (4610), the wearable device (20) may be set to acquire a value of a sensor (4620), and perform an operation so that medication determination result information is obtained based on the value of the sensor (4620).
[0399] According to various embodiments, the wearable device (20) determines whether a composite judgment condition is satisfied in operation 5411 when a camera activation condition is satisfied (5403-Y), and when the composite judgment condition is satisfied (5411-Y), in operation 5411, the wearable device obtains a sensor value for each time frame using at least one sensor of the wearable device, in operation 5413, obtains at least one image for each time frame using a camera of the wearable device, and in operation 5415, transmits the sensor value and the at least one image to the outside, so that information on a medication judgment result is obtained based on at least one artificial intelligence model. That is, in a situation where the wearable device (20) can use the camera (4610), but the accuracy of the medication judgment is below the threshold and cooperation of the sensor (4620) is required, the wearable device (20) is set to acquire the value of the sensor (4620) together with the medication performance image using the camera (4610), and performs an operation so that the medication judgment result information is acquired based on the medication performance image and the value of the sensor (4620).
[0400] According to various embodiments, the wearable device (20) may control the camera of the wearable device (4610) to acquire at least one image per time frame in operation 5419 when the composite judgment condition is not satisfied (5411-N), and to transmit the at least one image to the outside in operation 5421 so that information on the result of the medication judgment is acquired based on at least one artificial intelligence model. That is, the wearable device (20) may be set to acquire an image of medication performance using the camera (4610) in a situation where the camera (4610) is available and the accuracy of the medication judgment is higher than a threshold value, and may perform an operation so that information on the result of the medication judgment is acquired based on the image of the medication performance.
[0401]
[0402] 5. User Scenario
[0403] According to various embodiments, the health management system (1) can be implemented to not only provide a service that basically improves the accuracy of medication compliance of a user using the health management system (1), but also provide various types of services to ultimately improve the overall quality of life of the user by linking medication judgment result information with other biometric information.
[0404]
[0405] 5.1. Life Log Linkage
[0406] FIG. 52 is a drawing for explaining the life log linkage operation of the health management system (1) according to various embodiments.
[0407] According to various embodiments, the health management system (1) may be implemented to provide a service that links various types of life logs (LF1) (e.g., biometric information, activity information, nutritional information, communication) acquired by sensors of an existing wearable device (20) with medication behaviors (LF2) identified by the health management system (1). Accordingly, medication behaviors may be expanded to be managed by the life log.
[0408] At this time, referring to “3. Add-on wearable device”, the health management activity (LF2) may be identified by the first wearable device (2000a), and the life log (LF1) may be identified by the second wearable device (2000b), but is not limited to the described examples and may all be identified by a single wearable device (20).
[0409]
[0410] 5.1.1. Example of Life Log Information
[0411] FIG. 53 is a drawing for explaining an example of a life log (LF1) identifiable using a sensor (6400) of a wearable device (20) according to various embodiments.
[0412] According to various embodiments, referring to FIG. 53, the sensor (6400) of the wearable device (20) may include at least one of a motion sensor, a PPG sensor, a heart rate sensor, a BIA sensor, an ECG sensor, or a skin temperature sensor, and may include more sensors and / or fewer sensors, without being limited to the examples illustrated and / or described. Hereinafter, an example of a life log (LF) obtained based on a value identified using the sensor (6400) according to various embodiments will be described.
[0413] According to various embodiments, referring to FIG. 53, the wearable device (20) may obtain information on cardiovascular health, such as blood pressure, heart rate, heart rhythm, and oxygen saturation, and / or dietary information, such as blood sugar, weight, and BMI, based on at least one of a PPG value identified using a PPG sensor (6420) or a BIA value identified using a BIA sensor.
[0414] According to various embodiments, referring to FIG. 53, the wearable device (20) can obtain information about skin health, such as skin spectrum, based on PPG values identified using the PPG sensor (6420).
[0415] According to various embodiments, referring to FIG. 53, the wearable device (20) can obtain physiological information, such as ovulation cycle, based on a PPG value identified using a PPG sensor (6420) or a skin temperature value identified using a skin temperature sensor (6460).
[0416] According to various embodiments, referring to FIG. 53, the wearable device (20) may obtain information on the type of exercise and / or the intensity of exercise based on at least one of a PPG value identified using a PPG sensor (6420) or a motion value identified using a motion sensor (6410).
[0417] According to various embodiments, referring to FIG. 53, the wearable device (20) may obtain information on sleep health, such as sleep patterns and sleep quality, and / or information on emotions, such as depression and stress, based on at least one of a motion value identified using a motion sensor (6410) or a heart rate value identified using a heart rate sensor (6430).
[0418]
[0419] 5.1.2. Action to provide medication correlation chart
[0420] According to various embodiments, a health management system (1) (e.g., server (10)) may obtain a direct correlation between a lifelog (e.g., lifelog (LF) of FIG. 52) identified using a wearable device (20) and medication decision result information, and provide information on the obtained correlation.
[0421] Figure 54 is a flowchart illustrating an example of a medication correlation chart provision operation of a healthcare system (1) according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. Below, Figure 54 will be further described with reference to Figure 55.
[0422] FIG. 55 is a diagram illustrating an example of an operation for identifying a medication correlation of a health management system (1) according to various embodiments. 6600a of FIG. 55 may represent a medication subject performed over time, and 6600b may represent a value of a specific life log (e.g., blood pressure) measured over time. However, it is apparent to those skilled in the art that the operation for measuring medication correlation may be performed for various types of life logs without being limited to the illustrated example.
[0423] According to various embodiments, the health management system (1) (e.g., server (10)) may, in operation 6501, identify a first value of a specific life log using a sensor (6400) of a wearable device (20). For example, referring to FIG. 55, the server (10) may obtain a first value for a specific life log (e.g., blood pressure) identified using a sensor (6400) of the wearable device (20) from the wearable device (20) before the user takes medication (ta, tb).
[0424] According to various embodiments, the health management system (1) (e.g., server (10)) may obtain information on medication in operation 6503. For example, referring to FIG. 54, the server (10) may receive a medication performance image received from the wearable device (20) at the time of the user's medication performance (ta, tb), and, based on analysis of the medication performance image, identify information on medication (e.g., type of medication, number of times of medication, etc.) together with medication judgment result information.
[0425] According to various embodiments, the health management system (1) (e.g., server (10)) may, in operation 6505, identify a second value of the life log using a sensor (6400) of a wearable device (20). For example, referring to FIG. 55, the server (10) may obtain a second value for a specific life log (e.g., blood pressure) identified using the sensor (6400) of the wearable device (20) from the wearable device (20) after the user's medication performance time (ta, tb).
[0426] According to various embodiments, the health management system (1) (e.g., server (10)) may, in operation 6507, identify a correlation between medication information and a life log based on a difference between the medication information and the life log. For example, the server (10) may identify a difference value between the first value and the second value for the specific life log, and measure a correlation between the identified difference value and information on the medication (e.g., type of medication (or medication target), medication amount). For example, referring to [Mathematical Formula 1] below, the server (10) may measure a correlation between a specific type of medication and a specific life log based on dividing the difference value by the medication amount; however, without being limited to the described example, if the medication amount is not identified, a result value obtained by reflecting a predetermined coefficient to the difference value may be identified as a medication correlation for the specific type of medication.
[0427]
[0428] Based on the above medication correlation, when the user later takes a specific type of medication, the value of the specific type of life log (e.g., blood pressure) after taking the medication can be predicted based on the medication correlation. At this time, as illustrated in FIG. 55, when the second value is greater than the first value, the specific type of medication (e.g., medication A) is identified as having a first-direction (e.g., positive) medication correlation with respect to the specific life log (e.g., blood pressure), and when the second value is less than the first value, the specific type of medication (e.g., medication B) is identified as having a second-direction (e.g., negative) medication correlation with respect to the specific life log (e.g., blood pressure).
[0429] Meanwhile, the server (10) may further identify the medication amount for the medication subject as at least part of the operation of identifying the medication correlation between the medication subject (e.g., medication type) and the life log. In other words, the server (10) may calculate the medication correlation between the life log and the medication subject for each medication amount of the medication subject. Depending on the medication amount, the medication correlation may be calculated differently, and accordingly, the medication correlation between the specific medication and the life log may have a positive or negative value.
[0430] According to various embodiments, the wearable device (20) may be configured to sense a first life log using at least one sensor at the time the activation signal is received, based on receiving the activation signal, and to sense a second life log using at least one sensor after a preset time has elapsed from the time the activation signal is received (or the time when image capture by the camera is terminated).
[0431] According to various embodiments, a health management system (1) (e.g., server (10)) may be implemented to measure a plurality of medication-taking correlations over a preset period of time and identify a final medication-taking correlation that may be a representative value based on the measured plurality of medication-taking correlations. For example, the preset period of time may be determined by a user or preset by the server (10). The preset period of time may also be determined according to the type of medication for which the medication-taking correlation is to be measured and / or the type of life log. The server (10) may calculate an average of the values of the plurality of medication-taking correlations collected over the preset period of time, or may calculate a representative value for the plurality of medication-taking correlations based on various types of algorithms that are not limited to the examples described. Accordingly, a personalized medication-taking correlation for the user of the wearable device (20) may be identified. That is, the medication correlation of the first user calculated by the server (10) and the medication correlation of the second user may be calculated differently.
[0432] According to various embodiments, the health management system (1) (e.g., server (10)) may, in operation 6509, identify correlations by accumulating them over a specific period of time and provide a final correlation based on the identified correlations. For example, the server (10) may be implemented to measure a plurality of medication-taking correlations over a preset period of time and identify a final medication-taking correlation as a representative value based on the measured plurality of medication-taking correlations. For example, the preset period of time may be determined by a user or may be preset by the server (10). The preset period of time may also be determined according to the type of medication for which the medication-taking correlation is to be measured and / or the type of lifelog. The server (10) may calculate an average of the values of the plurality of medication-taking correlations collected over the preset period of time, or may calculate a representative value for the plurality of medication-taking correlations based on various types of algorithms that are not limited to the examples described. Accordingly, a personalized medication correlation for the user of the wearable device (20) can be identified. That is, the medication correlation for the user of the first wearable device calculated by the server (10) and the medication correlation for the user of the second wearable device can be calculated differently.
[0433]
[0434] 5.1.3. Linked Correlation Provision Behavior
[0435] According to various embodiments, a health management system (1) (e.g., server (10)) may obtain a first correlation between different lifelogs (e.g., lifelog (LF) of FIG. 52) identified using a wearable device (20), and a second correlation between the lifelog and a medication determination result, and provide information on a linked correlation based on the first correlation and the second correlation.
[0436] FIG. 56 is a flowchart illustrating an example of a linkage correlation providing operation of a health care system (1) according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. Below, FIG. 56 will be further described with reference to FIGS. 57 and 58.
[0437] FIG. 57 is a diagram illustrating an example of an operation for identifying a linkage correlation of a health management system (1) according to various embodiments. 6800a of FIG. 57 represents a subject taking medication over time, and 6800b and 6099b may represent values of different life logs (e.g., blood pressure, sleep quality) measured over time, but are not limited to the types of life logs illustrated. FIG. 58 is a diagram illustrating an example of a linkage correlation according to various embodiments.
[0438] According to various embodiments, the health management system (1) (e.g., server (10)) may, in operation 6701, identify a change amount of a first life log and a change amount of a second life log using a wearable sensor (e.g., sensor (6400) of a wearable device (20)), and, in operation 6703, identify a cross-correlation between the first life log and the second life log. For example, referring to (a) of FIG. 58, the cross-correlation may represent a correlation between different types of life logs (e.g., blood pressure, sleep quality). For example, referring to FIG. 57, the server (10) may obtain a plurality of life logs (e.g., blood pressure, sleep quality) based on values identified using at least one sensor (6400) of the wearable device (20). The server (10) can identify changes in values of a plurality of acquired life logs (e.g., blood pressure, sleep quality) and identify a correlation based on changes in the identified values. For example, the server (10) can obtain a first value of another life log (e.g., sleep quality) before the time point (ta1, tb1) at which the value of a first life log (e.g., blood pressure) changes (or is changed) and a second value of a second life log (e.g., sleep quality) after the time point (ta1, tb1), and identify a difference value (d1) between the first value and the second value. The server (10) can calculate a correlation between the first life log (e.g., blood pressure) and the second life log (e.g., sleep quality) based on the difference value (d1) between the amount of change (d2) of the first life log (e.g., blood pressure) and the second life log (e.g., sleep quality), as described in [Mathematical Formula 2] below.
[0439]
[0440] Not limited to what is described in the above [Mathematical Formula 2], the above [Mathematical Formula 2] may further include certain coefficients, but a detailed description thereof is omitted. Meanwhile, the correlation of the first life log with the second life log (e.g., sleep quality) can be inferred as d2 / d1.
[0441] According to various embodiments, the health management system (1) (e.g., server (10)) may, in operation 6705, identify a medication correlation between a specific medication target (e.g., medication A, medication B) and a first life log (e.g., blood pressure) using a wearable sensor (e.g., sensor (6400) of a wearable device (20)). For example, referring to FIG. 57, the server (10) may obtain a medication correlation between a medication type (or medication target) and a life log described in “5.1.2. Medication Correlation Providing Operation” based on a medication determination result (e.g., medication A, and medication B) identified based on a medication performance image captured using a camera of the wearable device (20) and a value of a life log (e.g., blood pressure) obtained using the sensor (6400). Meanwhile, without being limited to the described example, the server (10) may also obtain a medication correlation between the medication judgment result (e.g., medication A, and medication B) and the second life log (e.g., quality of sleep).
[0442] According to various embodiments, the health management system (1) (e.g., server (10)) may, in operation 6707, identify a chain correlation between a specific medication subject and the second life log based on the medication correlation and the cross-correlation. For example, the server (10) may identify a chain correlation between the medication subject and the second life log based on the medication correlation between the aforementioned medication subject (e.g., medication A, medication B) and the first life log and the cross-correlation between the first life log and the second life log.
[0443] According to various embodiments, the health management system (1) (e.g., server (10)) may, in operation 6709, identify correlations cumulatively over a specific period of time and provide a final correlation based on the identified correlations. For example, the server (10) may measure a plurality of medication correlations and a plurality of cross-correlations over a preset period of time, and identify a plurality of chain correlations based on the measured correlations (e.g., medication correlations, cross-correlations). The server (10) may be implemented to identify a final medication correlation as a representative value among the correlations (e.g., medication correlations, cross-correlations, chain correlations) identified over the preset period of time. Since the operation of identifying the representative value may be performed in the same manner as the operation 6509 described above, a redundant description thereof will be omitted.
[0444]
[0445] 5.1.4. Correlation-based service provision behavior
[0446] According to various embodiments, the health management system (1) (e.g., server (10)) may provide at least one service for managing a life log based on medication, based on the aforementioned correlations (e.g., medication correlation, cross-correlation, chain correlation).
[0447] FIG. 59 is a flowchart illustrating an example of a linkage correlation providing operation of a health care system (1) according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. Below, FIG. 59 will be further described with reference to FIGS. 60 and 61.
[0448] FIG. 60 is a diagram illustrating examples of operations for providing a service for managing a user's life log based on a correlation of a health management system (1) according to various embodiments. FIG. 61 is a diagram illustrating another example of operations for providing a service for managing a user's life log based on a correlation of a health management system (1) according to various embodiments.
[0449] According to various embodiments, a health management system (e.g., server (10)) may identify a change amount of a first life log and a change amount of a second life log using a wearable sensor (e.g., sensor (6400) of a wearable device (20)) in operation 7001, identify a cross-correlation between the first life log and the second life log in operation 7003, identify a medication correlation between a specific medication subject and the first life log using a wearable sensor (e.g., sensor (6400) of a wearable device (20)) in operation 7005, and identify a chain correlation between the specific medication subject and the second life log based on the medication correlation and the cross-correlation in operation 7007.
[0450] According to various embodiments, a health management system (e.g., server (10)) may, in operation 7009, provide information on a life log corresponding to a user's medication status based on at least one of the medication correlation, the cross-correlation, or the chain correlation.
[0451] In one embodiment, the server (10) may be implemented to provide at least one expected (or predicted) change in life logs when performing a specific medication based on at least one of the aforementioned correlations (e.g., the medication correlation, the cross-correlation, or the chain correlation). For example, as illustrated in (a) of FIG. 60, the server (10) may control an execution screen including expected changes in life logs when performing a specific medication (e.g., blood pressure medication) so as to be displayed on a local device (e.g., a wearable device (20), a user terminal (30)). For example, the server (10) may provide an execution screen for receiving information about a specific medication to a local device (e.g., a wearable device (20), a user terminal (30)), and may receive input for performing the specific medication (e.g., blood pressure medication) through the execution screen. The server (10) may control an execution screen to be displayed on a local device (e.g., a wearable device (20), a user terminal (30)) including a change amount in a first life log (e.g., blood pressure) calculated based on a medication correlation for a first life log (e.g., blood pressure) associated with a specific medication (e.g., blood pressure medication), and a change amount in a second life log (e.g., sleep quality) identified based on a cross-correlation between the first life log (e.g., blood pressure) and the second life log (e.g., sleep quality). At this time, without being limited to the described example, the change amount in the second life log (e.g., sleep quality) may be implemented to be provided based on a chain correlation calculated between the specific medication target and the second life log.
[0452] In another embodiment, the server (10) may be implemented to provide a change amount in the life log based on other medication-related information (e.g., medication amount, medication cycle, etc.) input together with the medication target (or medication type). For example, as illustrated in (b) of FIG. 60, the server (10) may control an execution screen including a change amount between a specific medication target (e.g., medication A) and at least one life log to be displayed on a local device (e.g., a wearable device (20), a user terminal (30)). At this time, the server (10) may provide a change amount in the life log based on a medication correlation between the specific medication target (e.g., medication A) and at least one life log, and a medication amount of the specific medication target (e.g., medication A). For example, the amount of change in the life log may be proportional to the amount of medication taken, but is not limited to the described example, and may not be proportional because the correlation between the life log and the medication correlation is set to an opposite value (e.g., changing from a negative value to a positive value) in the case of a specific amount of medication taken. In this case, the server (10) may control the value of the life log, which is sequentially changed according to the change in the amount of medication taken of the specific medication subject (e.g., medication A), to be provided on the execution screen. In addition, the server (10) may calculate the amount of change in the life log according to not only the amount of medication taken of the specific medication subject (e.g., medication A), but also the amount of medication taken of another additional medication subject (e.g., medication B), to provide a composite amount of change in the life log.
[0453] In another embodiment, the server (10) may identify a target medication amount for a specific medication based on accumulated correlations (e.g., medication correlations, cross-correlations, chain correlations) for a plurality of users, and provide the identified target medication amount to the user. For example, as illustrated in 7200b of FIG. 61, the server (10) may identify an average medication correlation (e.g., 8% increase in blood pressure when taking the medication) based on the medication correlations between a specific medication (e.g., medication A) and a life log (e.g., blood pressure) for a plurality of users. As illustrated in 7200a of FIG. 61, if there is a difference between the medication correlation between a specific medication (e.g., medication A) and a life log (e.g., blood pressure) of a specific user (e.g., user A) and the average medication correlation, the server (10) may provide a medication amount adjustment guide. For example, if the medication correlation of a specific user (e.g., user A) has a higher positive value than the average medication correlation, a guidance for reducing medication dosage may be provided, and if the medication correlation of a specific user (e.g., user A) has a lower positive value than the average medication correlation, a guidance for increasing medication dosage may be provided. It will be apparent to those skilled in the art that the opposite guidance may be provided if the medication correlations are all negative values.
[0454]
[0455] 5.1.5. Digital Therapeutics
[0456] According to various embodiments, the health management system (1) can be utilized as a digital therapeutic agent that uses a wearable device (20) to determine a disease currently present in a user's body and provides a software treatment solution for the determined disease.
[0457] Diseases that can be diagnosed through the health management system (1) include, for example, Parkinson's disease, insomnia, and schizophrenia. Below, the aforementioned diseases will be used as examples to explain how they can be utilized as digital therapeutics. However, the diseases listed above are merely examples and are not necessarily limited to them, and any disease that can be diagnosed through a wearable device can be targeted.
[0458] First, Parkinson's disease is characterized by decreased cognitive and problem-solving abilities, hallucinations, and hand tremors. Insomnia is characterized by sleep deprivation or difficulty falling into deep sleep due to excessive stress, and schizophrenia is a biological disorder caused by brain abnormalities, manifesting as hallucinations, delusions, and language disorders. In other words, each disease presents with a variety of symptoms and characteristics, and these characteristics can be utilized to diagnose the disease.
[0459] FIG. 62 is a flowchart illustrating an operation method of a digital therapeutic in a health management system according to some embodiments of the present disclosure. Referring to FIG. 62, the health management system (1) determines a first symptom severity for a user's disease-related symptoms before taking medication (S7310), and determines a second symptom severity for a disease-related symptom after taking medication (S7320). Here, the first symptom severity and the second symptom severity may each be symptom severity measured based on an image of the user's hand acquired using a camera of a wearable device (20), and the related symptoms may include representative symptoms, which are typical characteristics of each of various diseases, and side effects, which are additional symptoms that appear separately from the representative symptoms.
[0460] Thereafter, the health management system (1) compares the first and second symptom levels to obtain a difference in symptom levels and calculates a correlation (S7330). The calculated correlation is compared with a preset correlation reference value (S7340), and the health management system (1) determines a recommended dosage based on the comparison results (S7350) and informs the user of the determined recommended dosage, thereby providing a digital treatment solution.
[0461] FIG. 63 is a drawing showing a method for providing a dosage according to the degree of symptoms after taking medication in some embodiments of the present disclosure, and FIG. 64 is a drawing showing a method for providing a dosage according to the degree of symptoms after taking medication in another embodiment of the present disclosure.
[0462] First, referring to Figure 63, the health management system (1) determines the first symptom severity for each disease-specific representative symptom, and then determines the second symptom severity for each representative symptom after medication administration. Here, the representative symptoms that appear are medical and physical characteristics that frequently appear in the corresponding disease, and as described above, may vary by disease. In addition, the first and second symptom severity for each representative symptom may differ depending on the drug absorbed into the body after medication administration.
[0463] For example, in the case of Parkinson's disease, since hand tremor is a representative symptom, the severity of the representative symptom could be the number of hand tremors that occurred over a certain period of time (e.g., 120 times per minute). In the case of insomnia, since a slow heart rate due to lack of sleep is a representative symptom, the severity of the representative symptom could be the heart rate over a certain period of time. In the case of schizophrenia, since a speech disorder is a representative symptom, the severity of the representative symptom could be the speed of response to personalized questions (speech rate).
[0464] Thereafter, the health management system (1) calculates a first medication-taking correlation coefficient based on the difference between the first and second symptom degrees and the ratio of the medication dose. The first medication-taking correlation coefficient is a correlation between the taking of a specific drug and the symptoms resulting from the medication, and can be calculated using the following [Mathematical Formula 3].
[0465]
[0466] For example, in Parkinson's disease, if the first symptom severity, which is the number of tremor times per minute, the second symptom severity, is 60 times per minute, and the medication dosage is 60 mg, the first medication-taking correlation can be 1 (k1 is a proportional constant, assuming k1=1). Here, the first medication-taking correlation has a higher value as the difference between the first symptom severity and the second symptom severity increases, and has a higher value as the medication dosage decreases. In other words, the greater the difference between the first symptom severity and the second symptom severity, the higher the medication effect, and the smaller the medication dosage, the higher the effect even with a lower drug dose. Therefore, the first medication-taking correlation is proportional to the difference between the first symptom severity and the second symptom severity, and inversely proportional to the medication dosage.
[0467] The health management system (1) determines a recommended medication dosage by comparing the calculated first medication-taking correlation with a preset first reference correlation (w). Here, the preset first reference correlation (w) may be the average medication-taking correlation of patients with the disease in question or an arbitrarily set target medication-taking correlation.
[0468] In various embodiments, the health management system (1) may determine a recommended dosage as an increased dosage compared to the current dosage if the calculated first dosage correlation is less than the first reference correlation (w), determine a recommended dosage as the same dosage as the current dosage if the calculated first dosage correlation is equal to the first reference correlation (w), and determine a recommended dosage as a decreased dosage compared to the current dosage if the calculated first dosage correlation is greater than the first reference correlation (w).
[0469] Additionally, in various embodiments, the health management system (1) can specifically determine the increase or decrease in the recommended medication amount based on the first medication correlation, the preset first reference correlation (w), and the current medication amount.
[0470] The increase or decrease in the recommended dosage can be calculated by the following equation [Mathematical Formula 4].
[0471]
[0472] For example, in Parkinson's disease, if the preset first baseline correlation (w) is 15, the calculated first medication correlation of the user is 10, and the current medication amount is 5 (k2 is a proportional constant, assuming k2 = 1), the increase or decrease may be (15-10) / 5 = 1. Therefore, the final recommended medication amount may be 6, which is an increase of 1 from the current medication amount of 5.
[0473] In this way, the health management system (1) of the present disclosure can provide a digital treatment solution for a specific disease by individually determining the degree of symptoms of the user's disease, calculating a first medication correlation, and providing a recommended medication amount.
[0474] Meanwhile, in various embodiments, the severity of symptoms can be determined based on the number of symptoms manifested for each disease. Referring to Figure 64, possible symptoms of Parkinson's disease include tremors, rigidity, bradykinesia (slow movement), postural abnormalities, autonomic nervous system abnormalities, and sensory abnormalities. The severity of symptoms can be determined based on the number of symptoms manifested.
[0475] The occurrence of symptoms can be determined by comparing them with the standard values for each symptom. For example, in the case of tremor, if it is 7 Hz or higher, it is determined that tremor symptoms have occurred. In the case of rigidity, if there is no movement for a certain period of time (e.g., 10 seconds), it is determined that rigidity symptoms have occurred. In the case of bradykinesia, if the speaking or walking speed is below a certain speed, it is determined that bradykinesia symptoms have occurred. In the case of postural abnormality, it can be determined that postural abnormality symptoms have occurred based on the angle of the head or waist.
[0476] The health management system (1) counts the number of symptoms occurring before taking medication to determine the severity of the first symptom, and counts the number of symptoms occurring after taking medication to determine the severity of the second symptom. At this time, the severity of symptoms can be determined by applying weights to representative symptoms of each disease (e.g., counting a symptom twice when it occurs).
[0477] The health management system (1) calculates a second medication-compliance correlation, which is a correlation between medication compliance and the number of symptoms manifested, using the first and second symptom severity levels. The second medication-compliance correlation can be calculated based on the ratio of the first and second symptom severity levels, and can be specifically calculated using the following [Mathematical Formula 5].
[0478]
[0479] Here, k3 represents a proportional constant, and the second medication correlation coefficient has a high value when the number of second-onset symptoms is small compared to the number of first-onset symptoms, so the higher the value of the second medication correlation coefficient, the better the efficacy of the corresponding medication.
[0480] Thereafter, the health management system (1) compares the calculated second medication correlation with a preset second reference correlation (r) to determine a recommended medication dosage. The preset second reference correlation (r) may be an arbitrarily set target reference ratio.
[0481] For example, if the second medication correlation is less than the preset second reference correlation, it means that the efficacy of the medication being taken is low or does not work, and therefore the health management system (1) can determine the recommended medication amount by increasing the medication cycle, dosage, and number of times taken. Conversely, if the second medication correlation is significantly greater than the preset second reference correlation, it can be determined to recommend replacing the currently taken medication with a different medication.
[0482] If the second medication correlation is the same as the preset second reference correlation, it means that the efficacy of the medication being taken is working appropriately, so the health management system (1) can determine the recommended medication amount by maintaining the medication cycle, medication dosage, and number of times the medication is taken.
[0483] If the second medication correlation is greater than the preset second reference correlation, it means that the efficacy of the medication being taken is greater, so the health management system (1) can determine the recommended medication amount by reducing the medication cycle, medication dosage, and number of times the medication is taken.
[0484] Figure 65 is a diagram illustrating a method for providing a dosage according to side effects after taking a medication in some embodiments of the present disclosure.
[0485] In various embodiments of the present disclosure, as illustrated in FIG. 65, the degree of symptoms may be determined based on the degree of side effects experienced by the user for each disease. For example, possible side effects in Parkinson's disease include indigestion, possible side effects in insomnia include memory impairment, and possible side effects in schizophrenia include drowsiness or weight gain. The degree of symptoms may be determined based on at least one of the degree of occurrence of the side effects, the number of occurrences, and the duration of the side effects, and the health management system (1) may include various sensors (e.g., a body fat detection sensor, a sensor capsule, a drowsiness detection sensor, a heartbeat detection sensor, etc.) that can sense various symptoms and output them as quantitative values, thereby determining the degree of symptoms for each disease.
[0486] The health management system (1) determines the severity of side effects for each medication-taking behavior by assessing multiple medication-taking sessions by the user. Specifically, the health management system (1) can determine the severity of the third symptom, which is the severity of the disease-specific side effects following the first medication, after the first medication is taken, and can determine the severity of the fourth symptom, which is the severity of the disease-specific side effects following the second medication, after the second medication is taken. In this case, the second medication-taking time may be a time point after a considerable amount of time has elapsed since the first medication is taken, sufficient to fully manifest the efficacy of the first medication.
[0487] The health management system (1) calculates a third medication correlation coefficient, which is a correlation coefficient between medication compliance and side effects of a specific medication, based on the third symptom severity, the fourth symptom severity, the first medication dosage, and the second medication dosage. Specifically, the third medication correlation coefficient can be calculated using the following [Mathematical Formula 6].
[0488]
[0489] For example, in schizophrenia, if the third symptom severity is 110 beats per minute, the fourth symptom severity is 130 beats per minute, and the first dosage is 20 mg and the second dosage is 30 mg, then the third dosage correlation can be 2 (k4 is a proportional constant, assuming k4=1). Here, the third dosage correlation has a higher value as the difference between the third and fourth symptom severity increases, and the value is higher as the difference between the first and second dosages decreases. In other words, the greater the difference between the third and fourth symptom severity, and the smaller the difference between the first and second dosages, the higher the third dosage correlation, which means that the possibility of side effects due to taking the corresponding medication increases.
[0490] The health management system (1) determines the recommended dosage by comparing the calculated third medication correlation with a preset third reference correlation (u). Here, the third reference correlation (u) may be a medication correlation that includes the maximum tolerable degree of side effect symptoms for the disease in question, or an arbitrarily set target side effect correlation.
[0491] In the embodiment, the health management system (1) may determine a dosage that is increased from the current dosage as the recommended dosage if the calculated third dosage correlation is greater than the third reference correlation (u), determine a dosage that is the same as the current dosage as the recommended dosage if the calculated third dosage correlation is equal to the third reference correlation (u), and determine a dosage that is decreased from the current dosage as the recommended dosage if the calculated third dosage correlation is less than the third reference correlation (u).
[0492] Meanwhile, in various embodiments, the health management system (1) may calculate a comprehensive medication correlation by considering both the symptom severity for the representative symptoms of each disease and the symptom severity for the side effect symptoms, as in the above-described embodiment, and determine a recommended medication dosage based on the calculated comprehensive medication correlation. The calculation of the comprehensive medication correlation based on the representative symptoms and side effect symptoms and the determination of the recommended medication dosage can be combined in various ways and forms, and a detailed description thereof is omitted here.
[0493] Although the present invention has been described with reference to the drawings exemplified above, it is to be understood that the present invention is not limited to the embodiments and drawings disclosed herein, and that various modifications may be made by those skilled in the art within the scope of the technical idea of the present invention. Furthermore, even if the operational effects according to the configuration of the present invention were not explicitly described and explained while describing the embodiments of the present invention, it is natural that the effects predictable by the corresponding configuration should also be acknowledged.
[0494]
[0495] 6. Medical incentives
[0496] 6.1. Basic Operation
[0497] FIG. 66 is a flowchart illustrating an example of operations for providing medication certification and rewards to users of a healthcare system (1), according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed, and / or fewer operations may be performed. Below, FIG. 66 will be further described with reference to FIG. 67.
[0498] FIG. 67 is a drawing for explaining an example for medication authentication and reward granting of a health management system (1) according to various embodiments.
[0499] According to various embodiments, the health management system (1) can set medication conditions in operation 8201, determine medication based on an artificial intelligence model in operation 8205 when photography is performed in operation 8203, and determine whether the reliability of the medication determination result is greater than a threshold (e) in operation 8207.
[0500] According to various embodiments, the health management system (1) may, if the reliability is greater than the threshold (e), grant authentication in operation 8209 and provide a reward in operation 8211. For example, referring to (a) and (b) of FIG. 67, if the reliability of the medication determination result is greater than the threshold, the server (10) may identify a specific user as having performed medication management using the health management system (1) and provide a reward along with granting a medication certification mark to the specific user. In other words, a reward may be provided for a medication determination result for which the aforementioned medication certification mark is granted. The reward may be used to utilize a service provided by the health management system (1) (e.g., life log management) and / or to purchase a product sold (e.g., health functional food).
[0501] According to various embodiments, the health management system (1) may not proceed with the medication record in operation 8213 if the reliability is less than the threshold (e). Meanwhile, the failure to proceed with the medication record means that the recording of the medication judgment result is performed, but insufficient information on the medication is recorded. Accordingly, the medication record may be performed, but the medication certification mark may not be granted as described above.
[0502] Meanwhile, it is obvious to those skilled in the art that incentives as described above can be provided for various health management activities other than medication management (e.g., diet management, etc.).
[0503]
[0504] 6.2. Alarm provision behavior
[0505] Figure 68 is a flowchart illustrating an example of operations for providing medication certification and rewards to users of a healthcare system (1) according to various embodiments. The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed.
[0506] According to various embodiments, the health management system (1) may set a medication condition in operation 8401, receive activation data in operation 8403, determine whether to perform photography in operation 8405, determine whether to take the medication based on an artificial intelligence model in case photography is performed (8405-y), and determine whether the reliability of the medication determination result is greater than a threshold (e) in operation 8407.
[0507] According to various embodiments, the health management system (1) may, when the reliability is greater than the threshold (e) (8409-Y), provide a medication record and medication certification mark in operation 8411 and provide a reward in operation 8413.
[0508] According to various embodiments, the health management system (1) may, if no photographing is performed (8405-N) or the reliability of medication judgment is below a threshold (8409-N), provide an alarm in operation 8415, manually record medication based on a specified template in operation 8417, perform medication recording in operation 8419 but without assigning an authentication mark, and not provide a reward in operation 8421. That is, the server (10) may not assign a reward for a medication judgment result for which an authentication mark is not assigned, thereby increasing the utilization rate of medication management using the health management system (1) by the user of the health management system (1). Meanwhile, without being limited to the described example, even if medication recording is performed based on a separate manual input of the user without photographing using a wearable device (20), a medication authentication mark may not be assigned and a reward may not be provided.
[0509]
[0510] 6.3. Rewards granted based on the patient's medication decision
[0511] Figure 69 is a flowchart illustrating an example of operations for determining the true intent of medication and providing rewards based on the determination, according to various embodiments, of a health care system (1). The operations may be performed regardless of the order of the illustrated and / or described operations, and more operations may be performed and / or fewer operations may be performed. Below, Figure 69 will be further described with reference to Figure 70.
[0512] FIG. 70 is a drawing for explaining an example of a health management system (1) for determining whether a patient is taking medication and providing rewards according to various embodiments.
[0513] According to various embodiments, the health management system (1) can set medication conditions in operation 8601, determine medication based on an artificial intelligence model in operation 8605 when photography is performed in operation 8603, and determine whether the reliability of the medication determination result is greater than a threshold (e) in operation 8607.
[0514] According to various embodiments, the health management system (1) may determine whether the medication was taken by a genuine user in operation 8609 if the reliability is greater than the threshold (e). For example, when a medication determination result is obtained, the wearable device (20) (or user terminal (30)) may determine whether the medication was taken by a genuine user based on the information obtained for user authentication. For example, referring to FIG. 86, when the wearable device (20) obtains the medication determination result information, the wearable device (20) may request acquisition of fingerprint information for user authentication and may obtain the fingerprint information to identify whether the user is a genuine user. At this time, if the authentication information is not received within a preset time from the time of requesting the information for authentication, or an abnormal gesture such as taking off and then putting on the wearable device (20) is detected, the wearable device (20) may determine that the user is not a genuine user.
[0515] According to various embodiments, the health management system (1) may grant authentication in operation 8611 and provide a reward in operation 8613 when it is determined that the user has taken the medication in good faith (8609-Y). In addition, according to various embodiments, the health management system (1) may not proceed with the medication record in operation 8615 when the reliability is below the threshold (e) (8607-N) or when the user is not a good faith user (8609-N). Meanwhile, without being limited to the described example, the medication record may be performed but the medication certification mark may not be granted as described above.
Claims
1. As a device for determining medication, A first detachable structure configured to be detached from a main device, and a sub-device driven independently of the main device; wherein the sub-device: At least one processor; short-range communication circuit; and A camera disposed on the outer surface of the sub-device; The sub-device is coupled to the main device, and the at least one processor of the sub-device: Establishing a communication connection with the main device using the above short-range communication circuit, Using the above camera, a video is taken of the user taking medication, Based on the provision of the above-described captured image, the main device is set to obtain result information on the medication action. device.
2. In the first paragraph, the sub-device further includes a housing and a flexible substrate disposed inside the housing; Inside the housing, the at least one processor, the at least one short-range communication circuit, and the connection terminal of the camera are electrically connected through the flexible substrate. device.
3. In paragraph 2, When the sub-device and the main device are combined and worn on the user's wrist, the camera is directed to photograph the user's hand. device.
4. In paragraph 1, At least one processor of the above: Through the above short-range communication circuit, a request for communication connection is received from the main device, Set to establish a communication connection with the main device based on the received request, device.
5. In paragraph 4, At least one processor of the above: Through the above short-range communication circuit, identification information for a wireless communication device physically connected to a target object for medication is obtained from the main device, Based on the above identification information, a signal including activation data broadcast from the wireless communication device is obtained, Set to activate the camera based on the above activation data, device.
6. In paragraph 5, At least one processor of the above: Through the short-range communication circuit, based on the identification information, a communication connection is established with the wireless communication device, Based on the communication connection established with the wireless communication device, the signal including the activation data is set to be acquired. device.
7. In paragraph 5, Scanning a signal including activation data broadcast from the wireless communication device based on the identification information through the short-range communication circuit; Based on the above scan, the signal including the activation data is set to be acquired, device.
8. In paragraph 5, At least one processor of the above: Through the above short-range communication circuit, the captured image is transmitted to the outside based on the activation of the camera, Based on the above transmitted image being analyzed based on at least one artificial intelligence model of the server, the main device is set to obtain result information on the medication action. device.
9. A method of operating a device for determining medication, including a main device and a sub-device implemented to be detachable from the main device, An operation of establishing a communication connection with the main device using the short-range communication circuit of the sub device; An action of taking a video of a user's medication-taking behavior using a camera of the above sub-device; and An operation in which the main device obtains result information on the medication action based on providing the captured image using the camera of the sub device; How it works.
10. In the 9th paragraph, the sub-device further includes a housing and a flexible substrate disposed inside the housing; Inside the housing, the at least one processor, the at least one short-range communication circuit, and the connection terminal of the camera are electrically connected through the flexible substrate. How it works.
11. In paragraph 10, When the sub-device and the main device are combined and worn on the user's wrist, the camera is directed to photograph the user's hand. How it works.
12. In paragraph 9, An operation of receiving a request for a communication connection from the main device through the short-range communication circuit of the sub device; and Further comprising an operation of establishing a communication connection between the sub-device and the main device based on the received request through the short-range communication circuit of the sub-device; How it works.
13. In paragraph 12, An operation of obtaining identification information about a wireless communication device physically connected to a target object for medication from the main device through the short-range communication circuit of the sub device; An operation of obtaining a signal including activation data broadcast from the wireless communication device based on the identification information through the short-range communication circuit of the sub-device; and further comprising an operation of activating the camera of the sub-device based on the activation data; How it works.
14. In paragraph 13, An operation of establishing a communication connection with the wireless communication device based on the identification information through the short-range communication circuit of the sub-device; and Further comprising an operation of obtaining the signal including the activation data based on a communication connection established with the wireless communication device; How it works.
15. In paragraph 13, An operation of scanning a signal including activation data broadcast from the wireless communication device based on the identification information through the short-range communication circuit of the sub-device; and further comprising an operation of acquiring the signal including the activation data based on the scan of the sub-device; How it works.
Citation Information
Patent Citations
Medication management support system
JP2014174786A
Composition for inhibiting biofilm formation and antibacterial against Gram-positive bacteria and fungi comprising indole derivatives
KR1020230106223A
Floating structure
KR1020230120831A
Line management system for Optical Fiber Distruction
KR1020240114467A
Emergency supervisory system for 2-stage low temperature water absorption chiller
KR1020240137233A
