Spray inhalation drug dose estimate calculation and real-time medication guidance system using turbidity imaging and deep learning, and control method thereof

Abstract

The present disclosure relates to a spray inhalation drug dose estimate calculation and real-time medication guidance system using turbidity imaging and deep learning, and a control method thereof. The system captures an image of a drug that is not properly inhaled by a patient and is not administered into the body but disappears cloudily into the air, estimates an inhalation drug dose by performing deep learning on the turbidity of the captured image, and provides feedback to notify a user, so that the user can easily determine the inhalation drug dose among all drugs, thereby improving user convenience.

Description

A system for calculating nebulized inhaled drug dose estimates using turbidity image capture and deep learning, and a real-time dosing guidance system and its control method

[0001] The present disclosure relates to a system for calculating nebulized inhaled drug dose estimates and providing real-time medication guidance. More specifically, the present disclosure relates to a system for calculating nebulized inhaled drug dose estimates and providing real-time medication guidance using turbidity imaging and deep learning, and a control method thereof.

[0002] In recent years, air pollution has become increasingly severe, and the number of asthma patients is on the rise. Asthma is a chronic respiratory disease whose prevalence is steadily increasing worldwide. Korea is also seeing a steady increase, with the proportion of severe asthma cases increasing significantly.

[0003] Asthma is a disease in which the airways become severely narrowed due to inflammation following exposure to certain substances. It is accompanied by symptoms such as shortness of breath, chest tightness, coughing, and wheezing (a whistling sound when breathing). The burden of asthma is also high in Korea. In 2018, approximately 1.459 million people visited hospitals for asthma treatment, resulting in medical expenses reaching 158.29 billion won. Therefore, the need for ongoing medical management for effective asthma control is growing.

[0004] Inhalers, essential for asthma patients, are difficult to use, and many asthmatics often use them incorrectly. Medications used to treat asthma are primarily reliever, controller, and emergency medications, administered using medical inhalers.

[0005] The most common inhaler is the "L"-shaped metered-dose inhaler, which dispenses medication in a spray when the top of the inhaler is pressed. These devices have revolutionized respiratory care for the past 50 years by delivering medication directly to the lungs and avoiding the side effects of other forms of medication.

[0006] However, when using conventional inhalers, there were many concerns about whether patients were properly administering the correct amount of medication, there were many cases where medication was not actually administered, but it was impossible to estimate this, the medication dosage could be misused, and medication that was not properly inhaled could spread throughout the hospital room, causing discomfort to other patients, etc., so there were problems that made users feel uncomfortable.

[0007] The embodiment disclosed in the present disclosure aims to provide a system for calculating an estimated dose of a spray inhaled drug and providing real-time medication guidance, which photographs a drug that is not properly inhaled by a patient and disappears into the air as cloudy matter without being administered into the body, estimates the dose of the inhaled drug by deep learning the turbidity of the photographed image, and provides feedback on the estimated dose.

[0008] The embodiment disclosed in the present disclosure aims to provide a system for calculating estimated doses of a spray inhalation drug and providing real-time medication guidance, in which the light bulbs of the eyes and nose change at regular intervals so that the user can inhale through the mouth and exhale through the nose when inhaling the drug, and voice-guided medication guidance and medication administration guidance are possible.

[0009] The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0010] In order to achieve the above-described technical task, the present disclosure provides a system for capturing turbidity images and calculating a nebulized inhalation drug dose estimate using deep learning and for real-time medication guidance, comprising: a camera for capturing an image in front; a nebulizer for spraying a drug so that a user inhales the drug; a server for generating a drug inhalation amount prediction model; and a user terminal for outputting medication guidance content according to a control command from the server, wherein the server comprises: a communication module for transmitting and receiving data with an external device; a memory for storing at least one process for performing an operation and storing user input and data; And a processor for performing a control method according to the above process, wherein the processor controls the camera to capture a first image including an image of a drug that the user did not inhale among the drugs sprayed by the nebulizer, labels the captured first image according to the medication status, learns based on the labeled first image, creates the drug inhalation amount prediction model based on the learned result, receives a second image captured by the camera, predicts the user's medication status from the second image using the drug inhalation amount prediction model, and outputs the predicted medication status.

[0011] At this time, the nebulizer includes a first part corresponding to the user's nose and a second part corresponding to the user's mouth, and the first part and the second part each include a light-emitting diode, and the processor of the server can control the color of the first part and the color of at least one of the second part to change at predetermined time intervals.

[0012] Additionally, the processor can control the user terminal to output guide content when the medication status is below a preset threshold.

[0013] Additionally, the mouthpiece portion of the nebulizer according to the present disclosure may further include a shield that covers the user's face and indicates the amount of drug that has not been inhaled.

[0014] Additionally, the processor can enhance the contrast of the first image by a preset ratio.

[0015] Additionally, the processor may exclude some of the first images from learning if the luminance of the images is below a threshold.

[0016] Additionally, the processor can label the captured first image with different colors depending on the medication status.

[0017] In addition, the present disclosure provides a method for estimating a nebulized inhaled drug dose and providing real-time medication guidance using turbidity image capturing and deep learning, which is performed by a system including a camera, a nebulizer, and a server, the method comprising: a step in which the server controls the camera to capture a first image including an image of a drug not inhaled by the user among drugs sprayed by the nebulizer; a step in which the server labels the captured first image according to a medication status; a step in which the server learns based on the labeled first image; a step in which the server generates a drug inhalation amount prediction model based on the learned result; a step in which the server receives a second image captured by the camera; a step in which the server predicts the user's medication status from the second image using the drug inhalation amount prediction model; and a step in which the server outputs the predicted medication status.

[0018] According to the present invention, a drug that is not properly inhaled by a patient and is not administered into the body but disappears into the air as cloudy matter is photographed, and the turbidity of the photographed image is deep learned to estimate the inhaled drug dosage and provide feedback, so that the user can easily know the inhaled drug dosage among the total number of drugs, thereby improving user convenience.

[0019] According to the present invention, the light bulbs in the eyes and nose change at regular intervals so that the patient can inhale through the mouth and exhale through the nose when inhaling the drug, and the user can easily inhale the drug under the medication guidance, thereby improving user convenience.

[0020] According to the present invention, the breathing rhythm of elderly or child patients can be pleasantly guided and friendly animal images can be provided to provide a sense of psychological stability and reduce loneliness by providing a feeling of semi-medical staff staying by the patient's side during medication, thereby helping to provide comfortable treatment, thereby improving user convenience.

[0021] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0022] Figure 1 is a configuration diagram of a system for calculating a spray inhalation drug dose estimate and providing real-time medication guidance according to the present disclosure.

[0023] FIG. 2 is a flowchart illustrating a method for calculating a spray inhalation drug dose estimate and controlling real-time dosing guidance according to the present disclosure.

[0024] FIG. 3 is a drawing illustrating an embodiment of the present invention according to the present disclosure.

[0025] FIG. 4 is a drawing illustrating a specific configuration of a nebulizer according to the present disclosure.

[0026] FIG. 5 is a drawing illustrating an example of a patient suction guide using a user terminal according to the present disclosure.

[0027] FIG. 6 is a drawing illustrating an example of a patient suction guide using a user terminal according to the present disclosure.

[0028] FIG. 7 is a drawing illustrating an embodiment of adding a blindfold to a nebulizer to cover the patient and the spray smoke according to the present disclosure.

[0029] FIG. 8 is a diagram illustrating an example of data preprocessing by adjusting image contrast according to the present disclosure.

[0030] FIG. 9 is a diagram illustrating an example of data preprocessing by labeling an image according to color according to the present disclosure.

[0031] FIG. 10 is a diagram illustrating an example of calculating a drug intake amount using deep learning according to the present disclosure.

[0032] FIG. 11 is a drawing illustrating the overall concept of the present invention according to the present disclosure.

[0033] Figure 12 is a drawing illustrating a commercialization model of the present invention according to the present disclosure.

[0034] FIG. 13 is a diagram illustrating a configuration of a system for calculating a spray inhalation drug dose estimate and real-time medication guidance according to the present disclosure.

[0035] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and any content that is common in the technical field to which this disclosure pertains or that overlaps between embodiments is omitted. The terms "part, module, element, block" used in the specification may be implemented in software or hardware, and depending on the embodiments, multiple "parts, modules, elements, blocks" may be implemented as a single component, or a single "part, module, element, block" may include multiple components.

[0036] Throughout the specification, when a part is said to be "connected" to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection via a wireless communication network.

[0037] Additionally, when a part is said to "include" a component, this does not mean that it excludes other components, but rather that it may include other components, unless otherwise specifically stated.

[0038] Throughout the specification, when we say that an element is "on" another element, this includes not only cases where the element is in contact with the other element, but also cases where another element exists between the two elements.

[0039] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0040] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0041] The identification codes for each step are used for convenience of explanation and do not describe the order of each step. Each step may be performed in a different order than specified unless the context clearly indicates a specific order.

[0042] The operating principle and embodiments of the present disclosure are described below with reference to the attached drawings.

[0043] The present invention can be implemented not only in a server system but also in various devices capable of performing computational processing and providing results to a user. For example, the present invention can include a computer, a server device, and a mobile terminal, or can be implemented in any one of these forms.

[0044] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0045] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0046] The above portable terminal may include, for example, a wireless communication device that ensures portability and mobility, and may include all kinds of handheld-based wireless communication devices such as a PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, a smart phone, and a wearable device such as a watch, a ring, a bracelet, an anklet, a necklace, glasses, contact lenses, or a head-mounted device (HMD).

[0047] The artificial intelligence-related functions according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, one or more processors may be a general-purpose processor such as a CPU, an AP, a DSP (Digital Signal Processor), a graphics-only processor such as a GPU or a VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU. One or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if one or more processors are artificial intelligence-only processors, the artificial intelligence-only processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0048] The predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned by a learning algorithm using a plurality of learning data, thereby creating a predefined operation rules or artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed in the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0049] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and performs neural network operations through operations between the operation results of the previous layer and the multiple weights. The multiple weights of the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model is reduced or minimized during the learning process. The artificial neural network may include a deep neural network (DNN), and examples thereof include, but are not limited to, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), or deep Q-networks.

[0050] The processor can create a neural network, train (or learn) a neural network, perform computations based on received input data, and generate information signals based on the results of the computations, or retrain the neural network.

[0051] Neural networks include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated Recurrent Unit), AE (Auto Encoder), VAE (Variational Auto) Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics Network), Generative Adversarial Network (GAN), Liquid State Machine (LSM), Extreme Learning Machine (ELM), It will be understood by those skilled in the art that any neural network may be included, including but not limited to ESN (Echo State Network), DRN (Deep Residual Network), DNC (Differentiable Neural Computer), NTM (Neural Turning Machine), CN (Capsule Network), KN (Kohonen Network), and AN (Attention Network).

[0052] According to an exemplary embodiment of the present disclosure, the processor may be configured to perform a process for processing a CNN (Convolution Neural Network) such as GoogleNet, AlexNet, VGG Network, etc., a R-CNN (Region with Convolution Neural Network), a RPN (Region Proposal Network), a RNN (Recurrent Neural Network), a S-DNN (Stacking-based deep Neural Network), a S-SDNN (State-Space Dynamic Neural Network), a Deconvolution Network, a DBN (Deep Belief Network), a RBM (Restricted Boltzman Machine), a Fully Convolutional Network, an LSTM (Long Short-Term Memory) Network, a Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA for natural language processing, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics for vision processing, Visual Understanding, Video Synthesis, ResNet for data intelligence, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, Various artificial intelligence structures and algorithms, including Recommendation and Data Creation, can be utilized, but are not limited thereto. Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0053] Figure 1 is a configuration diagram of a system for calculating a spray inhalation drug dose estimate and providing real-time medication guidance according to the present disclosure.

[0054] Referring to FIG. 1, the nebulized inhalation drug dose estimation calculation and real-time medication guidance system (1000) includes a server (100), a camera (200), a nebulizer (300), and a user terminal (400).

[0055] The server (100) generates a drug intake amount prediction model.

[0056] The camera (200) captures an image of the front according to a control command from the server (100).

[0057] The nebulizer (300) sprays the drug so that the user inhales the drug.

[0058] The nebulizer (300) can emit light at a specific part of the main body according to a control command from the server. A detailed description of this is provided in Fig. 4.

[0059] The user terminal (400) outputs medication guidance content according to a control command from the server (100).

[0060] The server (100) includes an input module (110), a sensor module (120), a processor (130), a display module (140), a memory (150), a communication module (160), and a camera module (170).

[0061] The input module (110) receives user input.

[0062] The sensor module (120) senses user movement.

[0063] The processor (130) performs a method for calculating a spray inhalation drug dose estimate and controlling real-time dosing guidance according to the process.

[0064] The processor (130) controls the camera (200) to capture a first image including an image of a drug that the user did not inhale among the drugs sprayed by the nebulizer (300), labels the captured first image according to the medication status, learns based on the labeled first image, creates the drug inhalation amount prediction model based on the learning result, receives a second image captured by the camera, predicts the user's medication status from the second image using the drug inhalation amount prediction model, and outputs the predicted medication status.

[0065] The display module (140) displays a graphic image according to a control command from the processor (130).

[0066] The memory (150) stores at least one process for performing an operation and stores user input and data.

[0067] The communication module (160) transmits and receives data with an external device (200).

[0068] Here, the external device (200) includes an external device such as a smartphone, a PC, a laptop, a tablet PC, etc. The external device (200) includes a camera (200), a nebulizer (300), and a user terminal (400).

[0069] The camera module (170) captures images of the front.

[0070] The camera module (170) photographs a subject in front according to a control command from the processor (130).

[0071] The processor (130) controls the user terminal (400) to play pre-stored content. A detailed description of this is provided in FIG. 5.

[0072] The processor (130) controls the user terminal (400) to output guide content to the patient based on the medication status.

[0073] The processor (130) controls the user terminal (400) to hold the guide if the medication status is above a preset threshold value, and controls the user terminal (400) to output guide content (encouragement message) if the medication status is below the preset threshold value. A detailed description thereof is provided in Fig. 6.

[0074] The mouthpiece portion of the nebulizer (300) further includes a shield (330) that covers the user's face. A detailed description thereof is provided in Fig. 7.

[0075] The processor (130) enhances the contrast of the first image at a preset ratio. A detailed description thereof is provided in FIG. 8.

[0076] The processor (130) excludes some of the first images from learning if the brightness of some of the images is below a threshold value. A detailed description of this is provided in Fig. 8.

[0077] The processor (130) controls the camera (200) to take multiple pictures over a specific period of time. A detailed description of this is provided in Fig. 8.

[0078] The processor (130) labels the captured first image with different colors depending on the medication status. A detailed description of this is provided in Fig. 9.

[0079] However, the components illustrated in FIG. 1 are not essential for implementing the present invention according to the present disclosure, and thus the present invention described in this specification may have more or fewer components than the components listed above.

[0080] The communication module (160) may include one or more components that enable communication with an external device, and may include, for example, at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

[0081] The input module (110) is for inputting image information (or signal), audio information (or signal), data, or information input from a user, and may include at least one camera, at least one microphone, and at least one user input unit. Voice data or image data collected by the input module (110) may be analyzed and processed into a user control command.

[0082] The display module (140) displays (outputs) information processed in the present invention. For example, the present invention can display execution screen information of a running application program (e.g., an application), or UI (User Interface) or GUI (Graphical User Interface) information based on such execution screen information.

[0083] The memory (150) can store data supporting various functions of the present invention, programs for the operation of the control unit, input / output data (e.g., music files, still images, videos, etc.), and a plurality of application programs (or applications) driven by the artificial intelligence-based user behavior pattern analysis device (100), data for the operation of the device, and commands. At least some of these application programs can be downloaded from an external server via wireless communication.

[0084] The memory (150) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. In addition, the memory (150) may be a database connected by wire or wirelessly, although separate from the present invention, and may be implemented as a database system.

[0085] The processor (130) may be implemented as at least one core, a memory storing data for an algorithm for controlling the operation of components within the present invention or a program reproducing the algorithm, and at least one processor (not shown) that performs the aforementioned operations using the data stored in the memory. In this case, the memory and the processor may be implemented as separate chips. Alternatively, the memory and the processor may be implemented as a single chip.

[0086] In addition, the processor (130) can control any one or a combination of the components discussed above to implement various embodiments according to the present disclosure described in FIGS. 2 to 13 below.

[0087] At least one component may be added or deleted to correspond to the performance of the components illustrated in Figure 1. Furthermore, it will be readily apparent to those skilled in the art that the relative positions of the components may be altered to correspond to the performance or structure of the system.

[0088] Meanwhile, each component illustrated in FIG. 1 refers to software and / or hardware components such as a Field Programmable Gate Array (FPGA) and an Application Specific Integrated Circuit (ASIC).

[0089] FIG. 2 is a flowchart illustrating a method for calculating a spray inhalation drug dose estimate and controlling real-time medication guidance according to the present disclosure. The present invention is performed by a processor (130) of a spray inhalation drug dose estimate calculation and real-time medication guidance system (1000) or a server (100).

[0090] Referring to FIG. 2, the processor (130) controls the camera (200) to capture a first image including an image of a drug that was not inhaled by the user among the drugs sprayed by the nebulizer (300) (S210).

[0091] The processor (130) labels the photographed first image according to the medication status (S220).

[0092] The processor (130) learns based on the labeled first image (S230).

[0093] The processor (130) generates a drug intake amount prediction model based on the learning results (S240).

[0094] The processor (130) receives a second image captured by the camera (200) (S250).

[0095] The processor (130) predicts the user's medication status from the second image using a drug intake prediction model (S260).

[0096] The processor (130) outputs the predicted medication status (S270).

[0097] The processor (130) controls the user terminal (400) to play the pre-stored content.

[0098] The processor (130) controls the user terminal (400) to output guide content to the patient based on the medication status.

[0099] The processor (130) controls the user terminal (400) to hold the guide if the medication status is above a preset threshold value, and controls the user terminal (400) to output guide content if the medication status is below the preset threshold value.

[0100] The processor (130) enhances the contrast of the first image at a preset ratio.

[0101] The processor (130) excludes some of the images from learning when the brightness of some of the first images is below a threshold value.

[0102] The processor (130) controls the camera (200) to take multiple pictures during a specific period of time.

[0103] The processor (130) labels the photographed first image with different colors depending on the medication status.

[0104] FIG. 3 is a drawing illustrating an embodiment of the present invention according to the present disclosure.

[0105] As shown in (310) of Fig. 3, when a patient (10) inhales a drug sprayed from a nebulizer (300), the concept of the present invention is to photograph the drug (50) that is not properly inhaled by the patient and disappears into the air as cloud without being administered into the body, and estimate the inhaled drug dose by deep learning the turbidity of the photographed image and provide feedback to the patient.

[0106] FIG. 4 is a drawing illustrating a specific configuration of a nebulizer according to the present disclosure.

[0107] As shown in (410) of FIG. 4, the nebulizer (300) includes a first part (310) and a second part (320).

[0108] The first part (310) corresponds to the user's nose.

[0109] The second part (320) corresponds to the user's mouth.

[0110] The first part (310) and the second part (320) each include a light emitting diode.

[0111] The processor (130) of the server (100) controls the color of the first part (310) and at least one of the colors of the second part (320) to change according to the user's breathing state.

[0112] For example, to induce a user to inhale through the nose and exhale through the mouth when inhaling a drug, when the user inhales the drug through the nose, the first part (310) lights up at predetermined time intervals. When the user exhales through the mouth, the second part (320) lights up at predetermined time intervals.

[0113] According to the present invention, when a user inhales a drug through the nose, the part corresponding to the nose lights up, and when the user exhales through the mouth, the part corresponding to the nose lights up, so that the user has the advantage of easily knowing what action he or she should take.

[0114] According to the present invention, when the first part (310) operates, the second part (320) becomes stationary, and when the second part (320) operates, the first part (310) becomes stationary, so that the user can easily know what action to take by looking at the light bulb.

[0115] FIG. 5 is a drawing illustrating an example of a patient suction guide using a user terminal according to the present disclosure.

[0116] As shown in (510) of FIG. 5, the processor (130) controls the user terminal (400) to play the pre-stored content.

[0117] For example, while the user is wearing the nebulizer (300), the user terminal (400) can output audio to guide medication administration.

[0118] When the user runs the Doctor Panda app on the user terminal (400) while wearing the nebulizer (300), and the nebulizer (300) sprays the drug, the user terminal (400) outputs an audio voice such as “Inhale through your mouth and exhale through your nose.”

[0119] According to an embodiment of the present invention, different audio may be output depending on the age group of the user.

[0120] The processor (130) controls the user terminal (400) to output audio corresponding to the user's age based on the user's age information.

[0121] If the user is a child (0 to 7 years old), the user terminal (400) outputs audio based on the nursery rhyme.

[0122] If the user is an adult (20 to 35 years old), the user terminal (400) outputs audio based on pop songs.

[0123] If the user is a senior citizen (65 years of age or older), the terminal (400) outputs audio based on adult music.

[0124] According to the present invention, a user can obtain a sense of peace of mind while watching a panda video output to a user terminal (400).

[0125] According to the present invention, the breathing rhythm of elderly or child patients can be pleasantly guided and friendly animal images can be provided to provide a sense of psychological stability and reduce loneliness by providing a feeling of semi-medical staff staying by the patient's side during medication, thereby helping to provide comfortable treatment, thereby improving user convenience.

[0126] FIG. 6 is a drawing illustrating an example of a patient suction guide using a user terminal according to the present disclosure.

[0127] As shown in (610) of FIG. 6, the processor (130) controls the user terminal (400) to output guide content to the patient based on the medication status.

[0128] The processor (130) controls the user terminal (400) to hold the guide when the medication status is above a preset threshold value, and controls the user terminal (400) to output guide content (encouragement message, medication guidance message, medication guidance message) when the medication status is below the preset threshold value.

[0129] According to the present invention, in the case of a patient whose medication status is good based on the predicted medication status, separate guidance is not provided, and in the case of a patient whose medication status is not good, a message of encouragement, etc. is output, so that the guide content can be operated more efficiently.

[0130] FIG. 7 is a drawing illustrating an embodiment of adding a blindfold to a nebulizer to cover the patient and the spray smoke according to the present disclosure.

[0131] As illustrated in (710) of Fig. 7, the mouthpiece portion of the nebulizer (300) further includes a shield (330) that covers the user's face and indicates the amount of drug that has not been inhaled. Here, the color of the shield (330) may be black.

[0132] The above shield (330) is attached to the mouthpiece portion of the nebulizer (300) and can be adjusted in angle and length.

[0133] The screen (330) can be extended in length in all directions.

[0134] The screen (330) is configured in the form of a rollable display and can be extended toward the user.

[0135] According to the present invention, when the color of the shield is black, there is an advantage in that the camera can more easily take pictures of drugs that the user is not inhaling.

[0136] According to the present invention, by adding a shield between the spray smoke and the patient, the patient can focus more on his / her breathing without being concerned about the surroundings, thereby improving user convenience.

[0137] FIG. 8 is a diagram illustrating an example of data preprocessing by adjusting image contrast according to the present disclosure.

[0138] As shown in FIG. 8, the processor (130) enhances the contrast of the first image (810) at a preset ratio.

[0139] For example, the processor (130) enhances the contrast of the first image (810) by about 10%. Here, the enhancement ratio is not fixed at 10% and may vary depending on the surrounding circumstances.

[0140] According to the present invention, when the contrast of the first image (810) is enhanced at a predetermined ratio, there is an advantage in that learning can be facilitated.

[0141] The processor (130) excludes some images from learning if the brightness of some of the first images (810) is below a threshold value. Here, brightness includes brightness.

[0142] For example, if the luminance of some images (812) among the first images (810) is less than a threshold value, the processor (130) excludes those images from learning.

[0143] Additionally, the processor (1300) includes a specific image (811) among the first images (810) in learning if the luminance of the specific image (811) is greater than or equal to a threshold value.

[0144] The processor (130) divides the first image (810) into a plurality of regions, and if the luminance of some of the regions is less than a threshold value, the processor excludes those regions from learning.

[0145] According to the present invention, since smoke that the user has not inhaled is bright in color, only the image corresponding to smoke among the entire image is used for learning, so there is an advantage in that parts of the entire image that are not related to learning can be controlled and learned.

[0146] The processor (130) controls the camera (200) to take multiple pictures during a specific period of time.

[0147] For example, the processor (130) controls the camera to capture 300 images at predetermined time intervals for 15 minutes. Here, the images include turbidity images. A turbidity image refers to an image indicating the degree of turbidity when a drug is sprayed into the air.

[0148] FIG. 9 is a diagram illustrating an example of data preprocessing by labeling an image according to color according to the present disclosure.

[0149] As shown in (910) of FIG. 9, the processor (130) labels the captured first image with different colors depending on the medication status.

[0150] The first image may be taken differently depending on the user's medication status.

[0151] For example, the following description is based on an example in which a patient inhales 3 mL of drug.

[0152] For example, if a patient inhales all of the medication, the remaining medication will be 0%. In this case, the captured image will be black.

[0153] If the patient inhales 50% of the drug, the remaining drug will be 50%, or 1.5 mL. In this case, the captured image will be closer to gray.

[0154] If the patient does not inhale the drug, the remaining drug will be 100%, or 3 mL. In this case, the captured image will be close to white.

[0155] If the first image captured is black (state 1), the remaining drug is 0%, and the color matches black (911).

[0156] If the first image captured is close to gray (second state), the remaining drug is 50%, and the color matches gray (912).

[0157] If the first image captured is close to white (state 3), the remaining drug is 100%, and the color matches white (913).

[0158] That is, when the captured first image changes from the first state to the third state, the matching remaining drug can change from 0% to 100%, and the matching color can match from black to white.

[0159] The processor (130) can perform data preprocessing by matching the captured image to the remaining drug ratio and color.

[0160] The processor (130) performs data preprocessing by labeling the captured image according to color.

[0161] FIG. 10 is a diagram illustrating an example of calculating a drug intake amount using deep learning according to the present disclosure.

[0162] As shown in (1010) of Figure 10, the drug intake amount can be calculated using the following mathematical formula 1.

[0163]

[0164] Here, the drug intake amount can have a value of 0 to 1.

[0165] If the drug intake is 0, the sum of the proportions of remaining drug corresponding to the images generated at the time of capture is 1.

[0166] This means that the captured image is completely white, meaning that the patient did not inhale all of the medication sprayed by the nebulizer.

[0167] If the drug intake amount is 1, the sum of the proportions of the remaining drug corresponding to the images generated at the time of capture is 0.

[0168] That is, the captured image becomes completely black, meaning that the patient has inhaled all the medication sprayed by the nebulizer.

[0169] The processor (130) performs deep learning based on the amount of drug inhaled.

[0170] FIG. 11 is a drawing illustrating the overall concept of the present invention according to the present disclosure.

[0171] As illustrated in (1110) of FIG. 11, the overall concept of the present invention is to photograph a drug that is not properly inhaled by a patient and is not administered into the body but disappears into the air as cloudy matter, and to estimate the inhaled drug dosage by deep learning the turbidity of the photographed image and to provide feedback on the same.

[0172] The detailed concept is explained in the previous drawing.

[0173] Figure 12 is a drawing illustrating a commercialization model of the present invention according to the present disclosure.

[0174] As shown in (1210) of Figure 12, the customer (nebulizer-administered patient) pays the treatment fee to the hospital and medical staff and receives treatment, nursing plan setting, and guidance services.

[0175] FIG. 13 is a diagram illustrating a configuration of a system for calculating a spray inhalation drug dose estimate and real-time medication guidance according to the present disclosure.

[0176] Referring to FIG. 13, the present invention includes a device (1600). The device (1600) may include a memory (1602), a processor (1603), a transceiver (1604), and a peripheral device (1601). In addition, as an example, the device (1600) may further include other configurations and is not limited to the above-described embodiment.

[0177] More specifically, the device (1600) of FIG. 13 may be an exemplary hardware / software architecture, such as an NDN device, an NDN server, a content router, etc. In this case, as an example, the memory (1602) may be a non-removable memory or a removable memory. In addition, as an example, the peripheral device (1601) may include a display, GPS, or other peripheral devices, and is not limited to the above-described embodiment.

[0178] In addition, as an example, the above-described device (1600) may include a communication circuit such as the transceiver (1604), and may perform communication with an external device based thereon.

[0179] Additionally, as an example, the processor (1603) may be at least one of a general-purpose processor, a digital signal processor (DSP), a DSP core, a controller, a microcontroller, ASICs (Application Specific Integrated Circuits), FPGA (Field Programmable Gate Array) circuits, any other type of integrated circuit (IC), and one or more microprocessors associated with a state machine. In other words, it may be a hardware / software configuration that performs a control role for controlling the above-described device (1600).

[0180] At this time, the processor (1603) may execute computer-executable instructions stored in the memory (1602) to perform various essential functions of the present invention. For example, the processor (1603) may control at least one of signal coding, data processing, power control, input / output processing, and communication operations. In addition, the processor (1603) may control the physical layer, the MAC layer, and the application layers. In addition, for example, the processor (1603) may perform authentication and security procedures in the access layer and / or the application layer, and is not limited to the above-described embodiment.

[0181] For example, the processor (1603) can communicate with other devices via the transceiver (1604). For example, the processor (1603) can control a node to communicate with other nodes via a network through the execution of computer-executable instructions. That is, the communication performed in the present invention can be controlled. For example, the other nodes can be NDN servers, content routers, and other devices. For example, the transceiver (1604) can transmit RF signals via an antenna and transmit signals based on various communication networks.

[0182] In addition, as an example, MIMO technology, beamforming, etc. can be applied as antenna technology, and are not limited to the above-described embodiment. In addition, the signal transmitted and received through the transceiver (1604) can be modulated and demodulated and controlled by the processor (1603), and are not limited to the above-described embodiment.

[0183] The method according to one embodiment of the present disclosure described above can be implemented as a program (or application) and stored in a medium to be executed in combination with a hardware server.

[0184] The above-described program may include codes coded in a computer language, such as C, C++, JAVA, or machine language, that can be read by the processor (CPU) of the computer through the device interface of the computer, so that the computer reads the program and executes the methods implemented as a program. Such codes may include functional codes related to functions that define functions necessary for executing the methods, and may include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such codes may further include memory reference-related codes regarding which location (address address) of the internal or external memory of the computer should reference additional information or media necessary for the processor of the computer to execute the functions. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module (120) of the computer, what information or media to send and receive during communication, etc.

[0185] The above storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program can be stored in various recording media on various servers that the computer can access or in various recording media on the user's computer. In addition, the medium can be distributed across network-connected computer systems, so that computer-readable code can be stored in a distributed manner.

[0186] The steps of a method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or implemented by a combination thereof. The software module may reside in a random access memory (RAM), a read only memory (ROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable recording medium well known in the art to which the present disclosure pertains.

[0187] While the embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will appreciate that the present disclosure can be implemented in other specific forms without altering the technical spirit or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. A camera that captures images of the front; A nebulizer that sprays the drug so that the user inhales the drug; A server that generates a drug intake prediction model; and Including a user terminal that outputs medication guidance content according to a control command from the above server, The above server, A communication module that transmits and receives data with external devices; Memory in which at least one process is stored for performing actions and for storing user input and data; Including a processor that performs a control method according to the above process, The above processor, Controlling the camera to capture a first image including an image of a drug that the user did not inhale among the drugs sprayed by the nebulizer; The first image captured above is labeled according to the medication status, Learning based on the above labeled first image, Based on the learned results above, a drug intake amount prediction model is created, Receive a second image captured by the above camera, Predicting the user's medication status from the second image using the above drug intake prediction model, Outputting the predicted dosage status above, A system for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

2. In the first paragraph, the nebulizer, A first part corresponding to the nose of the user, a second part corresponding to the user's mouth, The first part and the second part each include a light-emitting diode, The processor of the above server is, Controlling the color of the first part and at least one color of the second part to change at predetermined time intervals, A system for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

3. In the first paragraph, the processor, If the above medication status is below a preset threshold, the user terminal is controlled to output guide content. A system for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

4. In paragraph 1, Further comprising a shield covering the user's face in the mouthpiece portion of the nebulizer to indicate the amount of drug not inhaled. A system for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

5. In the first paragraph, the processor, Enhancing the contrast of the first image by a preset ratio, A system for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

6. In the first paragraph, the processor, If the luminance of some of the first images is below a threshold, some of the images are excluded from learning. A system for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

7. In the first paragraph, the processor, The first image captured above is labeled with different colors depending on the dosage status. A system for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

8. In a method for estimating the nebulized inhalation drug dose and guiding real-time dosing using deep learning and turbidity image capturing performed by a system including a camera, a nebulizer, and a server, A step in which the server controls the camera to capture a first image including an image of a drug that the user did not inhale among the drugs sprayed by the nebulizer; A step in which the server labels the photographed first image according to the medication status; A step in which the server learns based on the labeled first image; A step in which the server generates a drug intake amount prediction model based on the learned results; A step in which the server receives a second image captured by the camera; The step of the server predicting the user's medication status from the second image using the drug intake prediction model; and The server comprises a step of outputting the predicted medication status, A method for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

9. In paragraph 8, the nebulizer, A first part corresponding to the nose of the user, a second part corresponding to the user's mouth, The first part and the second part each include a light-emitting diode, The above server, Controlling the color of the first part and at least one color of the second part to change at predetermined time intervals, A method for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

10. In paragraph 8, The server further includes a step of controlling the user's terminal to output guide content when the medication status is below a preset threshold. A method for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

11. In paragraph 8, Further comprising a shield covering the user's face in the mouthpiece portion of the nebulizer to indicate the amount of drug not inhaled. A method for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

12. In paragraph 8, Further comprising a step of enhancing the contrast of the above 1 image at a preset ratio, A method for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

13. In paragraph 8, Further comprising a step of excluding some of the images from learning when the luminance of some of the first images is below a threshold value. Method for calculating nebulized inhaled drug dose estimates and controlling real-time dosing guidance.

14. In paragraph 8, Further comprising a step of labeling the photographed first image with different colors according to the dosage status. A method for calculating nebulized inhaled drug dose estimates and real-time dosing guidance using turbidity imaging and deep learning.

Images