System for calculating estimated spray inhaled dose using turbidity image and photoplethysmogram measurement technique and control method therefor

Abstract

The present disclosure relates to: a system for calculating an estimated spray inhaled dose by using a turbidity image and a photoplethysmogram (PPG) measurement technique; and a control method therefor. According to the present disclosure, an inhaled drug dose can be estimated by using a change in pulse waves measured by a PPG measurement technique and the turbidity of an image captured of a drug that cannot be properly inhaled by a patient and thus disappears into the air in a mist without being administered, and feedback and notification on the inhaled drug dose can be given such that a user can easily know the inhaled dose of the drug relative to the total amount of the drug, and thus user convenience can be improved.

Description

A system for calculating spray inhalation dose estimates using turbidity images and photoplethysmography techniques and a control method thereof

[0001] The present disclosure relates to a system for calculating a nebulized inhaled drug dose estimate. More specifically, the present disclosure relates to a system for calculating a nebulized inhaled drug dose estimate using turbidity images and photoplethysmography techniques, and a control method thereof.

[0002] In recent years, air pollution has become increasingly severe, and the number of asthma patients is on the rise. Asthma is a chronic respiratory disease whose prevalence is steadily increasing worldwide. Korea is also seeing a steady increase, with the proportion of severe asthma cases increasing significantly.

[0003] Asthma is a disease in which the airways become severely narrowed due to inflammation following exposure to certain substances. It is accompanied by symptoms such as shortness of breath, chest tightness, coughing, and wheezing (a whistling sound when breathing). The burden of asthma is also high in Korea. In 2018, approximately 1.459 million people visited hospitals for asthma treatment, resulting in medical expenses reaching 158.29 billion won. Therefore, the need for ongoing medical management for effective asthma control is growing.

[0004] Inhalers, essential for asthma patients, are difficult to use, and many asthmatics often use them incorrectly. Medications used to treat asthma are primarily reliever, controller, and emergency medications, administered using medical inhalers.

[0005] The most common inhaler is the L-shaped metered-dose inhaler, which dispenses medication in a spray when the top of the inhaler is pressed. These devices have revolutionized respiratory care for the past 50 years by delivering medication directly to the lungs and avoiding the side effects of other forms of medication.

[0006] Pulse waves are waveforms that indicate changes in blood vessel volume as the heart pumps blood, and a detector that monitors these volume changes is called a pulse sensor. Photoplethysmography (PPG), a representative non-invasive pulse wave measurement method, measures changes in blood flow according to the heartbeat by irradiating infrared and red light on the surface of the human body, such as the fingertips or earlobes. More specifically, PPG can measure pulse waves according to changes in blood flow by measuring the amount of change in light penetrating the body.

[0007] Conventional PPG signal analysis methods can be broadly categorized into three categories. First, heart rate (HR) can be measured by monitoring the periodicity of waveform fluctuations obtained through sensor measurements. Second, arterial blood oxygen saturation (SpO2) can be measured using two wavelengths: infrared and red light. Third, heart rate variability (HRV) can be analyzed using PPG signals to assess stress levels.

[0008] However, in the case of conventional technology, there were many concerns about whether the patient was correctly administering the correct amount of medication, and it was difficult to know how much of the inhaled spray medication was actually administered into the body, and it was difficult to confirm whether the exact amount of medication was administered, which led to drug misuse or difficulty in expecting effective treatment results, resulting in user inconvenience.

[0009] The purpose of the embodiment disclosed in the present disclosure is to provide a system for calculating an estimated dose of a spray inhaled drug that estimates the dose of an inhaled drug by using the turbidity of an image taken of a drug that is not properly inhaled by a patient and disappears into the air as a cloud and a pulse wave change measured by a photoplethysmography (PPG) measurement technique.

[0010] The embodiment disclosed in the present disclosure aims to provide a spray inhalation drug dose estimation calculation system that more accurately estimates the inhalation drug dose using image opacity and photoplethysmography technology.

[0011] The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0012] In order to achieve the above-described technical task, a system for calculating a nebulized inhaled drug dose estimate using a turbidity image and a photoplethysmography technique according to the present disclosure comprises: a camera for capturing an image in front; a nebulizer for spraying a drug so that a user inhales the drug; a server for generating a drug inhalation amount prediction model; and a wearable device for measuring the user's respiratory rate data, wherein the server comprises: a communication module for transmitting and receiving data with an external device; a memory for storing at least one process for performing an operation and storing user input and data; A processor for performing a control method according to the above process is included, wherein the processor controls the camera to capture a first image including an image of a drug that has been sprayed by the nebulizer but has not been inhaled by the user, labels the captured first image according to a dosage state, learns based on the labeled first image, generates the drug inhalation amount prediction model based on the learned result, receives a second image captured by the camera, predicts the first dosage state of the user from the second image using the drug inhalation amount prediction model, estimates the second dosage state based on the user's respiratory rate data measured through the wearable device, determines a final dosage state by combining the first dosage state and the second dosage state, and outputs the determined final dosage state.

[0013] At this time, the wearable device can measure respiratory rate data by pulse wave changes using photoplethysmography (PPG) measurement technology.

[0014] Additionally, the processor can estimate the second medication state by multiplying the respiratory rate obtained through the wearable device by the tidal volume.

[0015] Additionally, the processor can obtain a PPG waveform of the drug through the wearable device and estimate a third medication state based on the obtained PPG waveform of the drug.

[0016] Additionally, the PPG waveform may form different patterns depending on the type of drug.

[0017] Additionally, the processor can obtain a first PPG waveform before inhalation of the drug through the wearable device, obtain a second PPG waveform after inhalation of the drug through the wearable device, and estimate a third medication state by comparing the first PPG waveform and the second PPG waveform.

[0018] Additionally, the processor can derive a similarity between the first PPG waveform and the second PPG waveform, and if the derived similarity is greater than or equal to a preset threshold value, the third medication state can be determined as abnormal.

[0019] Additionally, the processor can derive a similarity between the first PPG waveform and the second PPG waveform, and if the derived similarity is less than a preset threshold value, the third medication state can be determined as normal.

[0020] In addition, the processor may predict the first medication state of the user from the second image using the drug inhalation amount prediction model, estimate the second medication state based on respiratory rate data measured through the wearable device, estimate the third medication state based on the PPG waveform of the drug measured through the wearable device, and determine a final weight by combining the first weight, the second weight, and the third weight in an ensemble (voting format), wherein the first weight may correspond to the first medication state, the second weight may correspond to the second medication state, and the third weight may correspond to the third medication state.

[0021] Additionally, the processor can control the communication module to transmit data related to calculating the acquired spray inhalation drug dose estimate to an external server.

[0022] In addition, the present disclosure for achieving the above-described technical problem is a method for calculating an estimated nebulized inhaled drug dose using a turbidity image and photoplethysmography technology, which is performed by a system including a camera, a nebulizer, a server, and a wearable device, the method comprising: a step in which the server controls the camera to capture a first image including an image of a drug not inhaled by the user among drugs sprayed by the nebulizer; a step in which the server labels the captured first image according to a dosage state; a step in which the server learns based on the labeled first image; a step in which the server generates a drug inhalation amount prediction model based on the learned result; a step in which the server receives a second image captured by the camera; a step in which the server predicts a first dosage state of the user from the second image using the drug inhalation amount prediction model; a step in which the server estimates a second dosage state based on respiratory rate data of the user measured through the wearable device; a step in which the server determines a final dosage state by combining the first dosage state and the second dosage state; And the server may include a step of outputting the determined final dosage status.

[0023] According to the present disclosure, the inhaled drug dosage can be estimated and fed back using the image opacity of a drug that is not properly inhaled by a patient and disappears into the air as cloudy and the pulse wave change measured using photoplethysmography (PPG) measurement technology, so that the user can easily know the inhaled drug dosage among the total drugs, thereby improving user convenience.

[0024] According to the present disclosure, the inhaled drug dose can be estimated more accurately using image opacity and photoplethysmography technology, so that the user can know the exact dosage and reflect it in treatment, thereby improving user convenience.

[0025] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0026] Figure 1 is a configuration diagram of a system for calculating a spray inhalation drug dose estimate and providing real-time medication guidance according to the present disclosure.

[0027] FIG. 2 is a flowchart illustrating a method for calculating a spray inhalation drug dose estimate and controlling real-time dosing guidance according to the present disclosure.

[0028] FIG. 3 is a drawing illustrating an embodiment of the present invention according to the present disclosure.

[0029] FIG. 4 is a drawing illustrating a specific configuration of a nebulizer according to the present disclosure.

[0030] FIG. 5 is a drawing illustrating an example of a patient suction guide using a user terminal according to the present disclosure.

[0031] FIG. 6 is a drawing illustrating an example of a patient suction guide using a user terminal according to the present disclosure.

[0032] FIG. 7 is a drawing illustrating an embodiment of adding a blindfold to a nebulizer to cover the patient and the spray smoke according to the present disclosure.

[0033] FIG. 8 is a diagram illustrating an example of data preprocessing by adjusting image contrast according to the present disclosure.

[0034] FIG. 9 is a diagram illustrating an example of data preprocessing by labeling an image according to color according to the present disclosure.

[0035] FIG. 10 is a diagram illustrating an example of calculating a drug intake amount using deep learning according to the present disclosure.

[0036] FIG. 11 is a drawing illustrating the overall concept of the present invention according to the present disclosure.

[0037] Figure 12 is a drawing illustrating a commercialization model of the present invention according to the present disclosure.

[0038] FIG. 13 is a diagram illustrating a wearable device and a photoplethysmography waveform according to the present disclosure.

[0039] Figure 14 is a diagram showing a photoplethysmographic waveform before and after drug inhalation according to the present disclosure.

[0040] Figure 15 is a diagram illustrating the results of drug inhalation and inhaled drug estimation by a patient according to the present disclosure.

[0041] FIG. 16 and FIG. 17 are drawings illustrating a commercialization model of the present invention according to the present disclosure.

[0042] Figure 18 is a diagram illustrating a configuration of a spray inhalation drug dose estimation calculation system according to the present disclosure.

[0043] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and any content that is common in the technical field to which this disclosure pertains or that overlaps between embodiments is omitted. The terms "part, module, element, block" used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple "parts, modules, elements, blocks" may be implemented as a single component, or a single "part, module, element, block" may include multiple components.

[0044] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0045] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0046] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0047] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0048] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0049] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0050] The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

[0051] The present invention can be implemented not only in a server system but also in various devices capable of performing computational processing and providing results to a user. For example, the present invention can include a computer, a server device, and a mobile terminal, or can be implemented in any one of these forms.

[0052] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0053] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0054] The above portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD).

[0055] The artificial intelligence-related functions according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU or a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0056] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned by a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed in the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0057] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0058] The processor can create a neural network, train (or learn) a neural network, perform computations based on received input data, and generate information signals based on the results of the computations, or retrain the neural network.

[0059] Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), Generative Adversarial Network (GAN), Liquid State Machine (LSM), Extreme Learning Machine (ELM), It will be understood by those skilled in the art that any neural network may be included, including but not limited to ESN (Echo State Network), DRN (Deep Residual Network), DNC (Differentiable Neural Computer), NTM (Neural Turning Machine), CN (Capsule Network), KN (Kohonen Network), and AN (Attention Network).

[0060] According to an exemplary embodiment of the present disclosure, the processor may be configured to perform a process for generating a CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, Region with Convolution Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restrcted Boltzman Machine (RBM), Fully Convolutional Network, Long Short-Term Memory (LSTM) Network, Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA for natural language processing, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, ResNet for data intelligence, Anomaly Detection, Prediction, Time-Series Forecasting, Various artificial intelligence structures and algorithms, including optimization, recommendation, and data creation, can be utilized, but are not limited thereto. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0061] Figure 1 is a configuration diagram of a spray inhalation drug dose estimation calculation system according to the present disclosure.

[0062] Referring to FIG. 1, the nebulized inhalation drug dose estimation calculation system (1000) includes a server (100), a camera (200), a nebulizer (300), a user terminal (400), and a wearable device (500).

[0063] The server (100) generates a drug intake amount prediction model.

[0064] The camera (200) captures an image of the front according to a control command from the server (100).

[0065] The nebulizer (300) sprays the drug so that the user inhales the drug.

[0066] The nebulizer (300) can emit light at a specific part of the main body according to a control command from the server. A detailed description of this is provided in Fig. 4.

[0067] The user terminal (400) outputs medication guidance content according to a control command from the server (100).

[0068] The wearable device (500) measures the user's respiratory rate data.

[0069] A detailed explanation of this is given in Fig. 13.

[0070] The server (100) includes an input module (110), a sensor module (120), a processor (130), a display module (140), a memory (150), a communication module (160), and a camera module (170).

[0071] The input module (110) receives user input.

[0072] The sensor module (120) senses user movement.

[0073] The processor (130) performs a method for calculating an estimated dose of a spray inhalation drug according to a process.

[0074] The processor (130) controls the camera (200) to capture a first image including an image of a drug that the user did not inhale among the drugs sprayed by the nebulizer (300), labels the captured first image according to the medication status, learns based on the labeled first image, creates the drug inhalation amount prediction model based on the learning result, receives a second image captured by the camera, predicts the first medication status of the user from the second image using the drug inhalation amount prediction model, and estimates the second medication status based on respiratory rate data measured through the wearable device.

[0075] The first dosage state and the second dosage state are combined to determine the final dosage state, and the determined final dosage state is output.

[0076] The display module (140) displays a graphic image according to a control command from the processor (130).

[0077] The memory (150) stores at least one process for performing an operation and stores user input and data.

[0078] The communication module (160) transmits and receives data with an external device (200).

[0079] Here, the external device (200) includes an external device such as a smartphone, a PC, a laptop, a tablet PC, etc. The external device (200) includes a camera (200), a nebulizer (300), a user terminal (400), and a wearable device (500).

[0080] The camera module (170) captures images of the front.

[0081] The camera module (170) photographs a subject in front according to a control command from the processor (130).

[0082] The processor (130) estimates the second medication state by multiplying the respiratory rate obtained through the wearable device (500) by the tidal volume. A detailed description of this is provided in FIG. 13.

[0083] The processor (130) obtains the PPG waveform of the drug through the wearable device and estimates the third medication state based on the obtained PPG waveform of the drug.

[0084] The processor (130) obtains a first PPG waveform before inhalation of the drug through the wearable device (500), obtains a second PPG waveform after inhalation of the drug through the wearable device, and estimates a third medication state by comparing the first PPG waveform and the second PPG waveform.

[0085] The processor (130) derives a similarity between the first PPG waveform and the second PPG waveform, and if the derived similarity is greater than or equal to a preset threshold value, determines the third medication state as abnormal.

[0086] The processor (130) derives a similarity between the first PPG waveform and the second PPG waveform, and if the derived similarity is less than a preset threshold value, determines the third medication state as normal. A detailed description thereof is provided in Fig. 14.

[0087] The processor (130) predicts the first medication state of the user from the second image using the drug inhalation amount prediction model, estimates the second medication state based on the respiratory rate data measured through the wearable device, estimates the third medication state based on the PPG waveform of the drug measured through the wearable device, and determines the final weight by combining the first weight w1, the second weight w2, and the third weight w3 in an ensemble (Voting format).

[0088] Here, the first weight w1 corresponds to the first dosage state. The second weight w2 corresponds to the second dosage state. The third weight w3 corresponds to the third dosage state. A detailed description of this is provided in Fig. 15.

[0089] However, the components illustrated in FIG. 1 are not essential for implementing the present invention according to the present disclosure, and thus the present invention described in this specification may have more or fewer components than the components listed above.

[0090] The communication module (160) may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0091] The input module (110) is for inputting image information (or signal), audio information (or signal), data, or information input from a user, and may include at least one camera, at least one microphone, and at least one user input unit. Voice data or image data collected by the input module (110) may be analyzed and processed into a user control command.

[0092] The display module (140) displays (outputs) information processed in the present invention. For example, the present invention can display execution screen information of a running application program (e.g., an application), or UI (User Interface) or GUI (Graphical User Interface) information based on such execution screen information.

[0093] The memory (150) can store data supporting various functions of the present invention, programs for the operation of the control unit, input / output data (e.g., music files, still images, videos, etc.), and a plurality of application programs (or applications) driven by the artificial intelligence-based user behavior pattern analysis device (100), data for the operation of the device, and commands. At least some of these application programs can be downloaded from an external server via wireless communication.

[0094] The memory (150) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory (150) may be a database connected by wire or wirelessly, although separate from the present invention, and may be implemented as a database system.

[0095] The processor (130) may be implemented as at least one core, a memory storing data for an algorithm for controlling the operation of components within the present invention or a program reproducing the algorithm, and at least one processor (not shown) that performs the aforementioned operations using the data stored in the memory. In this case, the memory and the processor may be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

[0096] In addition, the processor (130) can control any one or a combination of the components discussed above to implement various embodiments according to the present disclosure described in FIGS. 2 to 18 below.

[0097] At least one component may be added or deleted to correspond to the performance of the components illustrated in Figure 1. Furthermore, it will be readily apparent to those skilled in the art that the relative positions of the components may be altered to correspond to the performance or structure of the system.

[0098] Meanwhile, each component illustrated in FIG. 1 refers to software and / or hardware components such as a Field Programmable Gate Array (FPGA) and an Application Specific Integrated Circuit (ASIC).

[0099] FIG. 2 is a flowchart illustrating a method for calculating a spray inhalation drug dose estimate according to the present disclosure. The present invention is performed by a spray inhalation drug dose estimate calculation system (1000) or a processor (130) of a server (100).

[0100] Referring to FIG. 2, the processor (130) controls the camera (200) to capture a first image including an image of a drug that was not inhaled by the user among the drugs sprayed by the nebulizer (300) (S210).

[0101] The processor (130) labels the photographed first image according to the medication status (S220).

[0102] The processor (130) learns based on the labeled first image (S230).

[0103] The processor (130) generates a drug intake amount prediction model based on the learning results (S240).

[0104] The processor (130) receives a second image captured by the camera (200) (S250).

[0105] The processor (130) predicts the user's first medication status from the second image using a drug intake amount prediction model (S260).

[0106] The processor (130) estimates the second medication state based on the respiratory rate data measured through the wearable device (500) (S270).

[0107] The processor (130) determines the final dosage state by combining the first dosage state and the second dosage state (S280).

[0108] The processor (130) outputs the determined final dosage state (S290).

[0109] The processor (130) estimates the second medication state by multiplying the respiratory rate obtained through the wearable device (500) by the tidal volume.

[0110] The processor (130) obtains the PPG waveform of the drug through the wearable device (500) and estimates the third medication state based on the obtained PPG waveform of the drug.

[0111] The processor (130) obtains a first PPG waveform before inhalation of the drug through the wearable device (500), obtains a second PPG waveform after inhalation of the drug through the wearable device (500), and estimates a third medication state by comparing the first PPG waveform and the second PPG waveform.

[0112] FIG. 3 is a drawing illustrating an embodiment of the present invention according to the present disclosure.

[0113] As shown in (310) of Fig. 3, when a patient (10) inhales a drug sprayed from a nebulizer (300), the concept of the present invention is to photograph the drug (50) that is not properly inhaled by the patient and disappears into the air as cloud without being administered into the body, and estimate the inhaled drug dose by deep learning the turbidity of the photographed image and provide feedback to the patient.

[0114] FIG. 4 is a drawing illustrating a specific configuration of a nebulizer according to the present disclosure.

[0115] As shown in (410) of FIG. 4, the nebulizer (300) includes a first part (310) and a second part (320).

[0116] The first part (310) corresponds to the user's nose.

[0117] The second part (320) corresponds to the user's mouth.

[0118] The first part (310) and the second part (320) each include a light emitting diode.

[0119] The processor (130) of the server (100) controls the color of the first part (310) and at least one of the colors of the second part (320) to change according to the user's breathing state.

[0120] For example, to induce a user to inhale through the nose and exhale through the mouth when inhaling a drug, when the user inhales the drug through the nose, the first part (310) lights up at predetermined time intervals. When the user exhales through the mouth, the second part (320) lights up at predetermined time intervals.

[0121] According to the present invention, when a user inhales a drug through the nose, the part corresponding to the nose lights up, and when the user exhales through the mouth, the part corresponding to the nose lights up, so that the user has the advantage of easily knowing what action he or she should take.

[0122] According to the present invention, when the first part (310) operates, the second part (320) becomes stationary, and when the second part (320) operates, the first part (310) becomes stationary, so that the user can easily know what action to take by looking at the light bulb.

[0123] FIG. 5 is a drawing illustrating an example of a patient suction guide using a user terminal according to the present disclosure.

[0124] As shown in (510) of FIG. 5, the processor (130) controls the user terminal (400) to play the pre-stored content.

[0125] For example, while the user is wearing the nebulizer (300), the user terminal (400) can output audio to guide medication administration.

[0126] When the user runs the Doctor Panda app on the user terminal (400) while wearing the nebulizer (300), and the nebulizer (300) sprays the drug, the user terminal (400) outputs an audio voice such as “Inhale through your mouth and exhale through your nose.”

[0127] According to an embodiment of the present invention, different audio may be output depending on the age group of the user.

[0128] The processor (130) controls the user terminal (400) to output audio corresponding to the user's age based on the user's age information.

[0129] If the user is a child (0 to 7 years old), the user terminal (400) outputs audio based on the nursery rhyme.

[0130] If the user is an adult (20 to 35 years old), the user terminal (400) outputs audio based on pop songs.

[0131] If the user is a senior citizen (65 years of age or older), the terminal (400) outputs audio based on adult music.

[0132] According to the present invention, a user can obtain a sense of peace of mind while watching a panda video output to a user terminal (400).

[0133] According to the present invention, the breathing rhythm of elderly or child patients can be pleasantly guided and friendly animal images can be provided to provide a sense of psychological stability and reduce loneliness by providing a feeling of semi-medical staff staying by the patient's side during medication, thereby helping to provide comfortable treatment, thereby improving user convenience.

[0134] FIG. 6 is a drawing illustrating an example of a patient suction guide using a user terminal according to the present disclosure.

[0135] As shown in (610) of FIG. 6, the processor (130) controls the user terminal (400) to output guide content to the patient based on the medication status.

[0136] The processor (130) controls the user terminal (400) to hold the guide when the medication status is above a preset threshold value, and controls the user terminal (400) to output guide content (encouragement message, medication guidance message, medication guidance message) when the medication status is below the preset threshold value.

[0137] According to the present invention, in the case of a patient whose medication status is good based on the predicted medication status, separate guidance is not provided, and in the case of a patient whose medication status is not good, a message of encouragement, etc. is output, so that the guide content can be operated more efficiently.

[0138] FIG. 7 is a drawing illustrating an embodiment of adding a blindfold to a nebulizer to cover the patient and the spray smoke according to the present disclosure.

[0139] As illustrated in (710) of Fig. 7, the mouthpiece portion of the nebulizer (300) further includes a shield (330) that covers the user's face and indicates the amount of drug that has not been inhaled. Here, the color of the shield (330) may be black.

[0140] The above shield (330) is attached to the mouthpiece portion of the nebulizer (300) and can be adjusted in angle and length.

[0141] The screen (330) can be extended in length in all directions.

[0142] The screen (330) is configured in the form of a rollable display and can be extended toward the user.

[0143] According to the present invention, when the color of the shield is black, there is an advantage in that the camera can more easily take pictures of drugs that the user is not inhaling.

[0144] According to the present invention, by adding a shield between the spray smoke and the patient, the patient can focus more on his / her breathing without being concerned about the surroundings, thereby improving user convenience.

[0145] FIG. 8 is a diagram illustrating an example of data preprocessing by adjusting image contrast according to the present disclosure.

[0146] As shown in FIG. 8, the processor (130) enhances the contrast of the first image (810) at a preset ratio.

[0147] For example, the processor (130) enhances the contrast of the first image (810) by about 10%. Here, the enhancement ratio is not fixed at 10% and may vary depending on the surrounding circumstances.

[0148] According to the present invention, when the contrast of the first image (810) is enhanced at a predetermined ratio, there is an advantage in that learning can be facilitated.

[0149] The processor (130) excludes some images from learning if the brightness of some of the first images (810) is below a threshold value. Here, brightness includes brightness.

[0150] For example, if the luminance of some images (812) among the first images (810) is less than a threshold value, the processor (130) excludes those images from learning.

[0151] Additionally, the processor (1300) includes a specific image (811) among the first images (810) in learning if the luminance of the specific image (811) is greater than or equal to a threshold value.

[0152] The processor (130) divides the first image (810) into a plurality of regions, and if the brightness of some of the regions is less than a threshold value, the processor excludes those regions from learning.

[0153] According to the present invention, since smoke that the user has not inhaled is brightly colored, only the image corresponding to smoke among the entire image is used for learning, so there is an advantage in that parts of the entire image that are not related to learning can be controlled and learned.

[0154] The processor (130) controls the camera (200) to take multiple pictures during a specific period of time.

[0155] For example, the processor (130) controls the camera to capture 300 images at predetermined time intervals for 15 minutes. Here, the images include turbidity images. A turbidity image refers to an image indicating the degree of turbidity when a drug is sprayed into the air.

[0156] FIG. 9 is a diagram illustrating an example of data preprocessing by labeling an image according to color according to the present disclosure.

[0157] As shown in (910) of FIG. 9, the processor (130) labels the captured first image with different colors depending on the medication status.

[0158] The first image may be taken differently depending on the user's medication status.

[0159] For example, the following description is based on an example in which a patient inhales 3 mL of drug.

[0160] For example, if a patient inhales all of the medication, the remaining medication will be 0%. In this case, the captured image will be black.

[0161] If the patient inhales 50% of the drug, the remaining drug will be 50%, or 1.5 mL. In this case, the captured image will be closer to gray.

[0162] If the patient does not inhale the drug, the remaining drug will be 100%, or 3 mL. In this case, the captured image will be close to white.

[0163] If the first image captured is black (state 1), the remaining drug is 0%, and the color matches black (911).

[0164] If the first image captured is close to gray (second state), the remaining drug is 50%, and the color matches gray (912).

[0165] If the first image captured is close to white (state 3), the remaining drug is 100%, and the color matches white (913).

[0166] That is, when the captured first image changes from the first state to the third state, the matching remaining drug can change from 0% to 100%, and the matching color can match from black to white.

[0167] The processor (130) can perform data preprocessing by matching the captured image to the remaining drug ratio and color.

[0168] The processor (130) performs data preprocessing by labeling the captured image according to color.

[0169] FIG. 10 is a diagram illustrating an example of calculating a drug intake amount using deep learning according to the present disclosure.

[0170] As shown in (1010) of Figure 10, the drug intake amount can be calculated using the following mathematical formula 1.

[0171]

[0172] Here, the drug intake amount can have a value of 0 to 1.

[0173] If the drug intake is 0, the sum of the proportions of remaining drug corresponding to the images generated at the time of capture is 1.

[0174] This means that the captured image is completely white, meaning that the patient did not inhale all of the medication sprayed by the nebulizer.

[0175] If the drug intake amount is 1, the sum of the proportions of the remaining drug corresponding to the images generated at the time of capture is 0.

[0176] That is, the captured image becomes completely black, meaning that the patient has inhaled all the medication sprayed by the nebulizer.

[0177] The processor (130) performs deep learning based on the amount of drug inhaled.

[0178] FIG. 11 is a drawing illustrating the overall concept of the present invention according to the present disclosure.

[0179] As illustrated in (1110) of FIG. 11, the overall concept of the present invention is to photograph a drug that is not properly inhaled by a patient and is not administered into the body but disappears into the air as cloudy matter, and to estimate the inhaled drug dosage by deep learning the turbidity of the photographed image and to provide feedback on the same.

[0180] The detailed concept is explained in the previous drawing.

[0181] Figure 12 is a drawing illustrating a commercialization model of the present invention according to the present disclosure.

[0182] As shown in (1210) of Figure 12, the customer (nebulizer-administered patient) pays the treatment fee to the hospital and medical staff and receives treatment, nursing plan setting, and guidance services.

[0183] FIG. 13 is a diagram illustrating a wearable device and a photoplethysmography waveform according to the present disclosure.

[0184] Figure 13 includes Figures 13(a) and 13(b).

[0185] (1310) of Fig. 13(a) is a drawing showing the structure of a wearable device.

[0186] (1320) in Fig. 13(b) is a diagram illustrating a photoplethysmographic waveform.

[0187] As shown in (1310) of FIG. 13(a), the wearable device (500) includes a light emitting unit (510) and a light receiving unit (520).

[0188] The light emitting unit (510) includes a light source.

[0189] The light receiving unit (520) receives scattered light emitted from the light source and scattered by a specific material.

[0190] The wearable device (500) measures the user's respiratory rate data by pulse wave changes using photoplethysmography (PPG) measurement technology.

[0191] The processor (130) estimates the second medication state by multiplying the respiratory rate obtained through the wearable device (500) by the tidal volume.

[0192] As shown in (1320) of Fig. 13(b), the photoplethysmographic waveform is plotted on a two-dimensional plane including the x-axis and the y-axis. The x-axis represents time, and the y-axis represents the voltage intensity of the signal.

[0193] As shown in Fig. 13(b), the processor (130) can calculate the respiratory rate by pulse wave change using photoplethysmography (PPG) measurement technology, and then estimate the value by multiplying the respiratory rate by tidal volume.

[0194] For example, if the drug dose is 10 mL, the second dosage state estimated by multiplying the tidal volume by the respiratory rate could be 5.5 mL.

[0195] In this case, it can be seen that the patient inhaled only 5.5 mL and sprayed 4.5 mL into the air.

[0196] The processor (130) monitors and analyzes the internal action of a sympathetic beta 2 agonist, which is a bronchodilator, or a sympathetic alpha 1 agonist, which is a decongestant, because the PPG waveform changes due to drugs that induce autonomic nervous system responses.

[0197] According to the present invention, the turbidity image capturing and calculation method can be used for measuring air pollution or fine dust in the future, and the PPG measurement technology has the advantage of being applied as a simple and excellent technology for monitoring conditions and drugs by converting waveform change data into data for monitoring conditions such as breathing patterns, breathing patterns of anesthetized or unconscious patients, pulse changes, etc., or for monitoring whether drugs are absorbed into the body and their effects.

[0198] Photoplethysmography (PPG) is a pulse measurement method that estimates cardiac activity by measuring the amount of blood flowing through blood vessels using the optical properties of biological tissue.

[0199] Pulse is a pulsatile waveform that appears as blood is pumped out of the heart. It can be measured by observing changes in blood flow and the resulting changes in blood vessel volume due to the relaxation and contraction of the heart. Photovolumetric pulse uses light to observe the characteristics of biological tissue, such as reflectivity, absorption, and transmittance, that appear when blood vessel volume changes, and pulse is measured through these changes.

[0200] This method is widely used as a non-invasive biosignal measurement method, and has the advantages of miniaturization of the measurement device and ease of use, making it easy to develop wearable biosignal detection sensors.

[0201] Figure 14 is a diagram showing a photoplethysmographic waveform before and after drug inhalation according to the present disclosure.

[0202] Figure 14 includes Figures 14(a) and 14(b).

[0203] (1410) of Figure 14(a) is a diagram showing a photoplethysmographic waveform before drug inhalation.

[0204] (1420) in Fig. 14(b) is a diagram showing a photoplethysmographic waveform after drug inhalation.

[0205] The processor (130) obtains the PPG waveform of the drug through the wearable device (500) and estimates the third medication state based on the obtained PPG waveform of the drug.

[0206] Here, the PPG waveform forms different patterns depending on the type of drug.

[0207] The processor (130) obtains a first PPG waveform before inhalation of the drug through the wearable device, obtains a second PPG waveform after inhalation of the drug through the wearable device, and estimates a third medication state by comparing the first PPG waveform and the second PPG waveform.

[0208] For example, if the drug dose is 10 mL, the third dosage state estimated by comparing the first and second PPG waveforms may be 5.5 mL.

[0209] In this case, it can be seen that the patient inhaled only 5.5 mL and sprayed 4.5 mL into the air.

[0210] This paper describes a method for determining whether a medication status is normal or abnormal by similarity.

[0211] The processor (130) derives a similarity between the first PPG waveform (1410) and the second PPG waveform (1420), and if the derived similarity is greater than a preset threshold value, determines the medication status as abnormal.

[0212] For example, if the similarity between the first PPG waveform (1410) and the second PPG waveform (1420) is 80% or more, it means that the drug was administered in very small amounts, as the drug had no effect even when administered.

[0213] Therefore, the third medication status is determined to be abnormal.

[0214] The processor (130) derives a similarity between the first PPG waveform and the second PPG waveform, and if the derived similarity is less than a preset threshold value, determines the medication status as normal.

[0215] For example, if the similarity between the first PPG waveform (1410) and the second PPG waveform (1420) is less than 80%, it means that the drug has been administered and is effective, and thus, the drug has been administered in an appropriate amount.

[0216] Therefore, the medication status is determined to be normal.

[0217] Figure 15 is a diagram illustrating the results of drug inhalation and inhaled drug estimation by a patient according to the present disclosure.

[0218] Figure 15 includes Figures 15(a) and 15(b).

[0219] (1510) of Figure 15(a) is a drawing illustrating a patient's drug inhalation.

[0220] (1520) of Fig. 15(b) is a diagram showing the results of inhalation drug estimation.

[0221] As shown in (1510) of Figure 15(a), the patient inhales the drug through a nebulizer.

[0222] As shown in (1520) of FIG. 15(b), the mobile device (400) outputs the inhalation drug estimation result.

[0223] For example, the inhaled dose of the drug is 55% and the non-inhaled dose is 45%.

[0224] If the drug dose is 10 mL, it can be seen that the patient inhaled only 5.5 mL and atomized 4.5 mL into the air.

[0225] Explains weights.

[0226] As illustrated in (1510) of FIG. 15(a), the processor (130) predicts the first medication state of the user from the second image using the drug inhalation amount prediction model, estimates the second medication state based on the respiratory rate data measured through the wearable device, estimates the third medication state based on the PPG waveform of the drug measured through the wearable device, and determines the final weight by combining the first weight w1, the second weight w2, and the third weight w3 in an ensemble (Voting format).

[0227] Here, the first weight w1 corresponds to the first dosage state. The second weight w2 corresponds to the second dosage state. The third weight w3 corresponds to the third dosage state.

[0228] w1 + w2 + w3 = 1.

[0229] For example, the weight w1 of the turbidity image may be the largest, the weight w2 of the respiratory rate data measured through a wearable device may be the next largest, and the weight w3 of the drug waveform measured through a wearable device may be the smallest.

[0230] w1 = 0.5, w2 = 0.3, w3 = 0.2, and the processor (130) combines these into an ensemble and reflects them to determine the final weight.

[0231] FIG. 16 and FIG. 17 are drawings illustrating a commercialization model of the present invention according to the present disclosure.

[0232] First, as shown in (1610) of Fig. 16, the customer (nebulizer-administered patient) pays the treatment fee to the hospital and medical staff, and receives treatment, nursing plan setting, and guidance services.

[0233] The processor (130) of the server (100) controls the communication module (160) to transmit turbidity image data and suction amount data to an external server.

[0234] Next, as shown in (1710) of Fig. 17, the customer (nebulizer-administered patient) pays the treatment fee to the hospital and medical staff, and receives treatment, nursing plan setting, and guidance services.

[0235] The processor (130) of the server (100) controls the communication module (160) to transmit PPG data and drug intake data obtained through the wearable device to an external server.

[0236] Figure 18 is a diagram illustrating a configuration of a spray inhalation drug dose estimation calculation system according to the present disclosure.

[0237] Referring to FIG. 18, the present invention includes a device (1600). The device (1600) may include a memory (1602), a processor (1603), a transceiver (1604), and a peripheral device (1601). In addition, as an example, the device (1600) may further include other configurations and is not limited to the above-described embodiment.

[0238] More specifically, the device (1600) of FIG. 18 may be an exemplary hardware / software architecture, such as an NDN device, an NDN server, a content router, etc. In this case, as an example, the memory (1602) may be a non-removable memory or a removable memory. In addition, as an example, the peripheral device (1601) may include a display, GPS, or other peripheral devices, and is not limited to the above-described embodiment.

[0239] In addition, as an example, the above-described device (1600) may include a communication circuit such as the transceiver (1604), and may perform communication with an external device based thereon.

[0240] Additionally, as an example, the processor (1603) may be at least one of a general-purpose processor, a digital signal processor (DSP), a DSP core, a controller, a microcontroller, ASICs (Application Specific Integrated Circuits), FPGA (Field Programmable Gate Array) circuits, any other type of integrated circuit (IC), and one or more microprocessors associated with a state machine. In other words, it may be a hardware / software configuration that performs a control role for controlling the above-described device (1600).

[0241] At this time, the processor (1603) may execute computer-executable instructions stored in the memory (1602) to perform various essential functions of the present invention. For example, the processor (1603) may control at least one of signal coding, data processing, power control, input / output processing, and communication operations. In addition, the processor (1603) may control the physical layer, the MAC layer, and the application layers. In addition, for example, the processor (1603) may perform authentication and security procedures in the access layer and / or the application layer, and is not limited to the above-described embodiment.

[0242] For example, the processor (1603) can communicate with other devices via the transceiver (1604). For example, the processor (1603) can control a node to communicate with other nodes via a network through the execution of computer-executable instructions. That is, the communication performed in the present invention can be controlled. For example, the other nodes can be NDN servers, content routers, and other devices. For example, the transceiver (1604) can transmit RF signals via an antenna and transmit signals based on various communication networks.

[0243] In addition, as an example, MIMO technology, beamforming, etc. can be applied as antenna technology, and are not limited to the above-described embodiment. In addition, the signal transmitted and received through the transceiver (1604) can be modulated and demodulated and controlled by the processor (1603), and are not limited to the above-described embodiment.

[0244] The method according to one embodiment of the present disclosure described above can be implemented as a program (or application) and stored in a medium to be executed in combination with a hardware server.

[0245] The above-described program may include codes coded in a computer language, such as C, C++, JAVA, or machine language, that can be read by the processor (CPU) of the computer through the device interface of the computer, so that the computer reads the program and executes the methods implemented as a program. Such codes may include functional codes related to functions that define functions necessary for executing the methods, and may include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such codes may further include memory reference-related codes regarding which location (address address) of the internal or external memory of the computer should reference additional information or media necessary for the processor of the computer to execute the functions. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module (120) of the computer, what information or media to send and receive during communication, etc.

[0246] The above storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program can be stored in various recording media on various servers that the computer can access or in various recording media on the user's computer. In addition, the medium can be distributed across network-connected computer systems, so that computer-readable code can be stored in a distributed manner.

[0247] The steps of a method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present disclosure pertains.

[0248] While the embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will appreciate that the present disclosure can be implemented in other specific forms without altering the technical spirit or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. In a system for calculating the estimated dose of atomized inhaled drug using turbidity images and photoplethysmography technology, A camera that captures images of the front; A nebulizer that sprays the drug so that the user inhales the drug; A server that generates a drug intake prediction model; and Including a wearable device that measures the user's respiratory rate data, The above server, A communication module that transmits and receives data with external devices; Memory in which at least one process is stored for performing actions and for storing user input and data; Including a processor that performs a control method according to the above process, The above processor, Controlling the camera to capture a first image including an image of a drug that the user did not inhale among the drugs sprayed by the nebulizer; The first image captured above is labeled according to the medication status, Learning based on the above labeled first image, Based on the learned results above, a drug intake amount prediction model is created, Receive a second image captured by the above camera, Predicting the user's first medication status from the second image using the drug intake prediction model, Estimating the second medication state based on the user's respiratory rate data measured through the wearable device, The final dosage state is determined by combining the first dosage state and the second dosage state, Outputting the final dosage status determined above, A system for calculating the estimated dose of inhaled aerosol drugs using turbidity images and photoplethysmography technology.

2. In the first paragraph, the wearable device, Measures respiratory rate data from pulse wave changes using photoplethysmography (PPG) measurement technology. A system for calculating the estimated dose of inhaled aerosol drugs using turbidity images and photoplethysmography technology.

3. In the first paragraph, the processor, The second medication state is estimated by multiplying the tidal volume by the respiratory rate obtained through the wearable device. A system for calculating the estimated dose of inhaled aerosol drugs using turbidity images and photoplethysmography technology.

4. In the first paragraph, the processor, Obtaining the PPG waveform of the drug through the wearable device, Estimating the third medication state based on the PPG waveform of the drug obtained above, A system for calculating the estimated dose of inhaled aerosol drugs using turbidity images and photoplethysmography technology.

5. In paragraph 4, The above PPG waveform forms different patterns depending on the type of drug. A system for calculating the estimated dose of inhaled aerosol drugs using turbidity images and photoplethysmography technology.

6. In the fourth paragraph, the processor, Obtaining the first PPG waveform before inhalation of the drug through the wearable device, Obtaining a second PPG waveform after inhalation of the drug through the wearable device; Estimating a third medication state by comparing the first PPG waveform and the second PPG waveform, A system for calculating the estimated dose of inhaled aerosol drugs using turbidity images and photoplethysmography technology.

7. In the 6th paragraph, the processor, Deriving the similarity between the first PPG waveform and the second PPG waveform, If the similarity derived above is greater than a preset threshold, the third medication state is determined as abnormal. A system for calculating the estimated dose of inhaled aerosol drugs using turbidity images and photoplethysmography technology.

8. In the 6th paragraph, the processor, Deriving the similarity between the first PPG waveform and the second PPG waveform, If the similarity derived above is less than the preset threshold, the third medication state is determined as normal. A system for calculating the estimated dose of inhaled aerosol drugs using turbidity images and photoplethysmography technology.

9. In the first paragraph, the processor, Predicting the user's first medication status from the second image using the drug intake prediction model, The second medication state is estimated based on the respiratory rate data measured through the above wearable device, Estimating the third medication state based on the PPG waveform of the drug measured through the wearable device, The final weight is determined by combining the first, second, and third weights as an ensemble (in voting format). The above first weight corresponds to the above first dosage state, The second weight corresponds to the second dosage state, The third weight corresponds to the third dosage state, A system for calculating the estimated dose of inhaled aerosol drugs using turbidity images and photoplethysmography technology.

10. In the first paragraph, the processor, Controlling the above communication module to transmit the data related to the calculation of the acquired spray inhalation drug dose estimate to an external server, A system for calculating the estimated dose of inhaled aerosol drugs using turbidity images and photoplethysmography technology.

11. A method for calculating an estimated dose of a nebulized inhaled drug using turbidity images and photoplethysmography techniques, performed by a system including a camera, a nebulizer, a server, and a wearable device, A step in which the server controls the camera to capture a first image including an image of a drug that the user did not inhale among the drugs sprayed by the nebulizer; A step in which the server labels the photographed first image according to the medication status; A step in which the server learns based on the labeled first image; A step in which the server generates a drug intake amount prediction model based on the learned results; A step in which the server receives a second image captured by the camera; A step in which the server predicts the user's first medication status from the second image using the drug intake amount prediction model; A step in which the server estimates a second medication state based on the user's respiratory rate data measured through the wearable device; The step of the server determining the final dosage state by combining the first dosage state and the second dosage state; and The server comprises a step of outputting the final medication status determined above, A method for calculating the estimated dose of aerosolized inhaled drugs using turbidity imaging and photoplethysmography techniques.

12. In the 11th paragraph, the wearable device, Measures respiratory rate data from pulse wave changes using photoplethysmography (PPG) measurement technology. A method for calculating the estimated dose of aerosolized inhaled drugs using turbidity imaging and photoplethysmography techniques.

13. In paragraph 11, Further comprising a step of estimating the second medication state by multiplying the tidal volume by the respiratory rate obtained through the wearable device. A method for calculating the estimated dose of aerosolized inhaled drugs using turbidity imaging and photoplethysmography techniques.

14. In paragraph 11, A step of obtaining a PPG waveform of the drug through the wearable device; and Further comprising a step of estimating a third dosage state based on the PPG waveform of the drug obtained above. A method for calculating the estimated dose of aerosolized inhaled drugs using turbidity imaging and photoplethysmography techniques.

15. In paragraph 14, The above PPG waveform forms different patterns depending on the type of drug. A method for calculating the estimated dose of aerosolized inhaled drugs using turbidity imaging and photoplethysmography techniques.

Images