Inhaler training system and method

Through a portable computer device, the processing of audio and video signals is identified and the deviation of the inhaler type and patient operation are provided, and real-time feedback is solved, which is the problem of lack of personalization and portability of the inhaler training method in the prior art, and the accuracy of the inhaler use is improved and therapeutic effects.

CN111886654BActive Publication Date: 2024-11-08SAVINO HEALTH CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN201880083273.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2017-12-21
Filing Date
2018-12-21
Publication Date
2024-11-08
Estimated Expiration
2039-03-13

AI Technical Summary

Technical Problem

The existing inhalation device training methods lack personalization and are inconvenient to carry, and cannot effectively guide patients to use the inhaler correctly, resulting in the failure to release the drugs as required or improper inhalation, affecting the treatment effect.

Method used

Through a portable computer device, digital audio and video signal processing is used to identify the type of inhaler and the patient's inhalation movement deviation, providing real-time feedback messages, and guiding patients to use the inhaler correctly. The device includes a data processing system, a camera, microphone, display and speaker, uses an artificial neural network to analyze signals, identify inhaler types and deviation types in real time, and indicates correct operation through audio and video feedback.

Benefits of technology

It realizes real-time identification of inhaler type and patient operation deviation on handheld devices, provides personalized feedback guidance, improves the accuracy and treatment effect of patients using inhaler, reduces misuse, and is suitable for various inhaler types.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN111886654B_ABST
    Figure CN111886654B_ABST
Patent Text Reader

Abstract

The present invention relates to a method and system for providing feedback to a patient using a specific type of inhaler to inhale a therapeutic aerosol according to instructions for use provided to the patient together with the inhaler. The method comprises the following steps: obtaining a sequence of digital audio signals and a sequence of digital video signals emitted by the patient when using the inhaler; identifying the type of the inhaler by processing the video signals; identifying whether the patient deviates from the instructions by processing the audio and video signals; and identifying the type of the deviation; and presenting one or more feedback messages to the patient, the feedback messages indicating: whether a deviation is identified; and if so, indicating the type of deviation that occurred; and optionally the inhalation action in the absence of the deviation.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] Respiratory patients receive most of their medication via an inhalation device. In this way, the medication reaches the lungs directly, the target area to be treated. A known problem is that the market for inhalation devices is growing and therefore diverse. There are at least 4 different categories of devices that support the inhalation of medication into the lungs: dry powder inhalers, nebulizers, metered dose inhalers (MDIs) and soft mist inhalers. They are all intended for patients to self-administer medication at home.

[0002] When a drug is administered in tablet form, only adherence to the medication is crucial to the success of the treatment. For inhaled drugs, not only is it necessary to use the drug regularly, but also the correct operation of the inhalation device and the inhalation action are crucial. If the inhalation is not performed correctly, the amount of drug administered to the lungs is too low for a successful treatment. Chronic diseases such as asthma or COPD are more likely to be unstable, and patients are unnecessarily prescribed higher and higher doses, or end up in hospital more and more frequently. Therefore, patients need guidance to use the inhalation system correctly.

[0003] For example, in an MDIS, the release of the drug needs to be consistent with the start of inhalation, or inhalation through a dry powder inhaler must be rapid and forceful. In order to release the full amount of drug, some devices must be kept upright or at another defined angle.

[0004] According to the literature, misuse of inhaled therapy is common but can be avoided through proper and repeated education and training of patients. Such training methods include providing patients with videos and other instructional manuals, as well as additional auxiliary equipment and devices to measure the effect during the use of inhalers. These devices constitute specific structures, such as sensors coupled to the patient and related feedback mechanisms processed by computers.

[0005] For example, in US 2015 / 0339953 A1, a self-learning system is provided, which provides guidance for an inhaler system using a portable computer device and a camera device. The system includes guidance for multiple stages of real-time learning, such as watching a teaching video and a real-time video of the patient at the same time.

[0006] In US 2013 / 0063579 A1, a training method is described, which can train a patient to use a predetermined inhalation system. The system is adapted specifically for each patient and inhaler. A universally applicable system would be more efficient.

[0007] Existing methods for inhalation training either lack personal interaction with the patient or are coupled with the need for additional equipment and are therefore not portable. Therefore, there is a need for a personalized and convenient training method for patients using an inhalation device. Summary of the invention

[0008] In a first aspect, the present invention relates to a computer-implemented method for providing feedback to a patient using a particular type of inhaler to inhale a therapeutic aerosol according to instructions for use of the inhaler or provided to the patient with the inhaler. The method comprises the following steps: (a) obtaining a sequence of digital audio signals and a sequence of digital video signals emitted by the patient when using the inhaler; (b) identifying the type of the inhaler by processing the video signals; (c) identifying whether the patient deviates from the instructions by processing the audio and video signals; (d) if a deviation is identified in step (c), identifying the type of the deviation; and (e) presenting one or more feedback messages to the patient, the feedback messages indicating (i) whether a deviation is identified; (ii) if a deviation is identified, indicating the type of the deviation; and optionally (iii) an inhalation action in the absence of the deviation.

[0009] In another aspect, the invention provides a data processing system comprising a processor adapted to perform the method described herein.

[0010] In yet another aspect, the invention provides a computer program comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method described herein.

[0011] According to yet another aspect, the present invention provides a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A simplified diagram illustrating the steps of the method of the invention according to a first embodiment is shown.

[0013] Figure 2 A simplified diagram showing the steps of the method of the present invention according to another alternative embodiment is shown. DETAILED DESCRIPTION

[0014] In a first aspect, the invention provides a computer implemented method as defined in claim 1 for providing feedback to a patient using a particular type of inhaler to inhale a therapeutic aerosol in accordance with instructions for use provided to the patient with the inhaler.

[0015] In one embodiment, the method comprises the following steps:

[0016] (a) obtaining a sequence of digital audio signals and a sequence of digital video signals emitted from the patient while using the inhaler;

[0017] (b) identifying the type of the inhaler by processing the video signal;

[0018] (c) identifying whether the patient deviates from the instructions by processing the audio and video signals;

[0019] (d) if a deviation is identified in step (c), identifying the type of the deviation; and

[0020] (e) presenting one or more feedback messages to the patient, the feedback messages indicating

[0021] (i) Whether the deviation is identified;

[0022] (ii) if a deviation is identified, indicating the type of deviation; and optionally

[0023] (iii) an inhalation action without said deviation;

[0024] Wherein, preferably, step b) and step c) are performed by analyzing the video signal and the audio signal using an artificial neural network, or by a method for detecting regular motion, wherein the artificial neural network is trained to recognize the inhaler type and to recognize deviations;

[0025] Wherein, steps (a) to (e) are performed by a handheld portable feedback device, the device comprising

[0026] - data processing systems;

[0027] - data storage;

[0028] -camera;

[0029] -microphone;

[0030] - a display; and

[0031] -speaker.

[0032] Preferably, steps b) and c) comprise performing the analysis of the video signal and the audio signal using an artificial neural network trained to recognize the inhaler type and to recognize deviations, and step c) further comprises using a method for detecting regular movements.

[0033] As used herein, inhaler is an inhalation device that can send therapeutic aerosol so that a user or patient can inhale the aerosol.Aerosol is the dispersion of solid phase, semi-solid phase or liquid phase in a continuous gas phase, so it comprises for example powder aerosol (pharmaceutically called inhaled powder aerosol) and atomized aerosol.Usually the inhalation device for conveying powder aerosol is described as powder inhaler.Atomized liquid is used by means of various inhalation devices (comprising nebulizer, pressurized metered dose inhaler and soft mist inhaler).In one embodiment, method of the present invention uses powder inhaler, nebulizer, metered dose inhaler or soft mist inhaler to provide feedback to the patient.

[0034] A major advantage of the method according to the invention is that the method can be performed on a handheld device. A suitable handheld device may be a smartphone. The method can generally be performed on any processor, even on a relatively slow processor such as a smartphone or a raspberry pie. The method can be performed in real time and does not require access to a server with a large amount of computing power, i.e. the method can be performed offline.

[0035] Inhaler type refers to a specific inhaler model or model series. In this case, the model series refers to those model variants that are used by patients in essentially the same way, especially with respect to the breathing action. For illustration, examples of specific types of nebulizers are the PARI eFlow rapid nebulizer system, which is an electronic nebulizer that incorporates vibrating mesh technology; examples of specific types of powder inhalers are the ADVAIR sold by GSK An example of a specific type of soft mist inhaler is sold by Boehringer Ingelheim An example of a particular type of metered dose inhaler is sold by GSK HFA.

[0036] Each inhaler is provided to the patient along with instructions for use, which are usually printed on a patient leaflet, but may also be provided to the patient over the Internet or through verbal and / or visual information received from a health care provider (such as a physician, pharmacist, or nurse).

[0037] The patient using the inhaler is a human subject who needs or will benefit from the therapeutic aerosol administered by the inhaler. The use may be for any medical purpose, including the prevention or treatment of any disease, disorder or symptom for which an aerosol may be beneficial.

[0038] An advantage of the method of the present invention is that the method identifies the inhaler type. Prior art methods such as described in US 2013 / 0063579 A1 may be suitable for training patients, but require prior adaptation to a specific inhaler.

[0039] A pre-trained artificial neural network can be used to identify the inhaler. In this way, the method can be performed on a handheld device with limited computing power (eg, a smartphone).

[0040] The method of the present invention is computer-implemented. Any suitable computer or other device including a data processing system, a data storage device, a device for obtaining an audio signal and a video signal, and a device for presenting an audio feedback message and a video feedback message can be used to perform the method, for example by installing the following software: the software, when executed by the data processing system of the device, causes the device to perform the method steps as described herein. In one of the preferred embodiments, the device is a portable feedback device. An example of a suitable device that is portable and can be configured to perform the steps of the present invention by means of a computer program is a smart phone.

[0041] The method comprises the following steps: obtaining a sequence of digital audio signals and a sequence of digital video signals emitted from the patient while using the inhaler. The audio signals may be obtained from a microphone and the video signals may be obtained from a camera; the microphone and the camera may be combined in a portable feedback device. Relevant signals are emitted from the patient while using the inhaler, the signals including signals emitted from the inhaler itself during use by the patient.

[0042] In one embodiment, the audio signal sequence covers the time during which the patient uses an inhaler to perform a breathing action or is collected within the time. In another embodiment, obtain or collect the audio signal from two or more continuous breathing actions. As used herein, breathing action includes an inhalation phase and an exhalation phase, in which the patient breathes in or inhales an aerosol emitted from the inhaler; in the exhalation phase, the patient exhales or exhales. In some cases, the breathing action further includes the stage during which the patient holds his breath and neither inhales nor exhales.

[0043] Similarly, the video signal can cover the time during which the patient uses the inhaler to perform one breathing action, and optionally the time during which the patient performs two or more consecutive breathing actions, or be collected during said time. In one of the preferred embodiments, the video signal includes signals from the patient's face and the inhaler.

[0044] In one embodiment, the audio signal sequence and the video signal sequence are obtained simultaneously.

[0045] The method further comprises the step of identifying the type of inhaler, which step is performed by processing the video signal in the previous step. This step can be performed by using frames from the video signal that are suspected of capturing inhaler A as input to a classifier based on a pre-trained neural network. The classifier outputs x different values, where each value corresponds to the detection of inhaler A. i The probability of, and i is a number from 1 to x, corresponding to the inhaler type.

[0046] Said method further comprises the step of identifying whether the patient deviates from the explanation, and said step is performed by processing the audio signal and the video signal that obtain in the previous step. In this case, departing from and the patient uses or has used the inhaler that is used to inhale therapeutic aerosol and obtains or collects the mode of audio signal and video signal is relevant. This step can be performed after the step of identifying the type of inhaler. To a certain extent, this step also can be performed simultaneously with identifying the inhaler type.

[0047] Deviations from the instructions may be related, for example, to the patient's posture, the position or orientation of the inhaler, or the patient's breathing maneuver. With regard to the breathing maneuver, for example, deviations from the instructions may be indicated, for example, by too fast or too slow inhalation, too short or too long inhalation phases, the presence or absence of breath-holding phases, incorrect inhaler actuation timing, incorrect exhalation (e.g., through the inhaler mouthpiece, or vice versa, where the instructions specify otherwise), etc.

[0048] Another deviation may be improper preparation of the inhaler. Some inhalers, particularly suspension aerosols in the form of metered dose inhalers, require shaking in order to provide an optimized suspension to be inhaled. For other inhalers or combinations of inhalers and pharmaceutical products, slow stirring of the (optionally reconstituted) active ingredient may be required.

[0049] The step of identifying whether the patient deviates from the instructions may be performed by processing audio and video signals emanating from the patient in order to calculate the likelihood that the breathing maneuvers performed by the patient conform (or do not conform) to the breathing maneuvers correctly performed according to the instructions applicable to the corresponding inhaler.

[0050] The inventors were able to develop a new method to identify whether an inhaler is correctly primed. Prior art methods are based on optical flow analysis, which requires a lot of computing power and cannot be performed in real time on smartphones or low-power processors. The inventors developed a new method that allows for the identification of regular movements, i.e. shaking or stirring, in real time on power-limited processors, such as smartphone processors.

[0051] The method is generally based on motion recognition. Any method suitable for the selected device can be used. In particular, the method is a method for detecting regular motion as defined below.

[0052] If a deviation has been identified, the method further comprises the step of identifying the type of deviation that has occurred.This step also involves processing of the audio and video signals obtained in the previous step.

[0053] In a further step, one or more feedback messages are presented to the patient indicating whether a deviation was identified and, if so, the type of deviation that occurred. Optionally, the feedback includes information about an inhalation maneuver without a corresponding deviation and how the patient should perform the inhalation maneuver.

[0054] According to the present invention, the feedback message can be presented to the patient in an audio and / or video format, for example, by a speaker and / or display, which can be part of a portable feedback device, such as a speaker and display of a smart phone configured to perform the method of the present invention. In one embodiment, the feedback message or at least one of the feedback messages is presented to the patient in an audio-visual format. In addition to audio and / or video formats, other formats, such as tactile feedback, can also be used. In one embodiment, the feedback message is presented using augmented reality.

[0055] For example, a feedback message in an audio format with an indication that a deviation has occurred may be presented as a specific sound with a "negative" meaning, such as an alarm sound, or it may also be presented as a spoken language message. In a visual format, the feedback may include displaying a symbol, color, or text message indicating that a deviation exists. Optionally, a video clip may be displayed to convey this information.

[0056] Similarly, the indication of the deviation type can be presented as a feedback message in an audio format, a visual format or a combined audio-visual format. In one of the preferred embodiments, and due to the higher complexity of such indications, the format of this feedback is audio-visual.

[0057] Optional feedback indicating the correct use of the inhaler and especially the correct breathing action according to the corresponding instructions can also be presented in any of the above-mentioned formats. In a preferred embodiment, the format is audio-visual. In a specific preferred embodiment, the format is audio-visual using augmented reality.

[0058] As mentioned, steps (a) to (e) of the method can be performed by a handheld portable feedback device. Such a device comprises at least a data processing system, a data storage, a camera, a microphone, a display and a speaker. In the context of the present invention, the combination of "portable" and "handheld" means that the feedback device can be held in the patient's hand while performing the method. The device can be customized or universal, such as a smartphone. In order for the device to become a feedback device according to the present invention, it must be adapted to perform the method steps, the adaptation being achieved by a computer program executed on the device, i.e. by the data processing system of the device, so that when the program is executed, method steps (a) to (e) are also performed.

[0059] The computer program may be stored in a data storage device of the portable feedback device. The sequence of digital audio signals emanating from the patient may be obtained in step (a) by means of a microphone, and the digital video signals may be obtained by means of a camera of the device. The one or more feedback messages presented to the patient according to step (e) may be displayed on a display of the device, audibly via a speaker, or by using a combination of these output devices.

[0060] The identification of the inhaler type according to step (b) is based on processing of the video signal, which processing may be performed by a data processing system of the device. 1 To A x can be stored in a database in the data memory of the feedback device. Alternatively, parameter A 1 To A x The data may be stored in a database stored in a (e.g. cloud-based) remote memory or data storage device, which is preferably accessible to the data processing system via wireless communication. Therefore, preferably, the feedback device includes means for wirelessly communicating with such an external data memory or data storage device, optionally via the Internet.

[0061] Similarly, the identification of the presence and type of deviations of the patient from the instructions when using the inhaler according to steps (c) and (d) may be performed by a data processing system of the device on which a computer program is executed. Likewise, the pre-existing data on which the data processing system identifies the deviations in the patient's breathing maneuvers may be stored in a data memory of the feedback device or may be obtained from an external data memory or data storage device, which is preferably accessed by the feedback device by means of wireless communication.

[0062] As described above, one or more feedback messages are presented to the patient, the feedback message indicating whether a deviation is identified; and if this is the case, indicating the type of deviation that occurs. Optionally, the feedback includes information about the inhalation action in the absence of a corresponding deviation and how the patient should perform the inhalation action. The feedback message can be presented to the patient in an audio and / or video format, for example, by a speaker and / or display, which can be part of a portable feedback device, such as a speaker and display of a smart phone configured to perform the method of the present invention.

[0063] In one embodiment, the feedback message is presented using augmented reality.

[0064] Optionally, the method comprises a further step of recognizing the identity of the patient based on the video signal. This step may also be performed by a data processing system of the portable feedback device, the data processing system being configured to obtain the video signals from the camera, to process these signals and to compare them with stored data corresponding to a particular face. Likewise, the database in which such data is stored may be located in a data memory of the portable feedback device, or may be located in an external data memory or data storage device, which is preferably accessed by the feedback device by means of wireless communication.

[0065] The timing of this step is more flexible. Its performance requires the availability of the digital video signal sequence sent from the patient, but this is not necessarily necessary when using an inhaler. Therefore, even before step (a) or after step (e), the identification of the patient's identity can be carried out. However, alternatively, as defined according to step (a), the digital video signal sequence sent from the patient is identified by processing when using an inhaler, and the identification itself can be performed at least in part in parallel with step (a), or performed simultaneously with steps (b), (c) and / or (d).

[0066] If the patient's identity is identified, the method may include a further optional step of presenting a feedback message to the patient indicating the fact that the patient has been identified or recognized, or indicating the identity of the patient that has been identified. Like other feedback messages, such a message regarding the patient's identity may be presented in an audio and / or video format, for example, via a speaker and / or display, which may be part of a portable feedback device, such as a speaker and display of a smartphone configured to perform the method of the present invention. For example, the patient's name may be displayed in written form, or a voice message indicating the name may be issued via a speaker, or an image of the patient's face may be displayed, or any combination thereof.

[0067] In one embodiment, a feedback message indicating the fact that the patient has been identified or recognized, or indicating that the patient's identity has been identified, is presented prior to presenting one or more feedback messages indicating whether a deviation has been identified and the type of deviation, if any. In another embodiment, the feedback message regarding the patient's identity is presented at least partially in parallel (or simultaneously) with one or more other feedback messages of step (e).

[0068] In addition, the method may optionally include the steps of identifying whether there is a detrimental condition that can adversely affect the quality of the audio signal or the video signal, and optionally, identifying the nature and extent of such detrimental condition. As used herein, detrimental conditions should be broadly interpreted to include environmental conditions and patient manipulation of the portable feedback device, or a combination of the two.

[0069] In one embodiment, the harmful condition is an environmental condition. For example, when the patient will use an inhaler, this environmental condition may be relevant with a dominant particularly low or high luminous intensity. The harmful environmental condition (for example, by the mode of the patient holding the inhaler and / or portable feedback device) that is jointly affected by the patient is the luminous contrast in the video signal sent from the patient. If this luminous contrast is very high, then the execution of the optional step of step (b), (c) and / or (d) and identifying the patient's identity based on the video signal may be difficult or impossible in extreme cases.

[0070] Another example of a harmful environmental condition is the presence of acoustic noise (i.e., unwanted sounds unrelated to the patient's use of the inhaler). For example, the average acoustic background noise should preferably be below about 50 dB, and more preferably below about 40 dB, or below about 35 dB. In one of the particularly preferred embodiments, the threshold is about 30 dB, above which the presence of a harmful environmental condition is identified.

[0071] An example of an adverse condition caused by the patient is shaking of the portable feedback device. Shaking frequencies of about 1 Hz or more, or about 2 Hz or more, or in particular about 3 Hz or more may make it difficult to obtain a video signal of sufficient quality to allow steps (b) to (d), or identification of the patient. Each of these shaking frequencies can be used as a threshold above which the presence of an adverse condition is identified.

[0072] If an adverse condition that can adversely affect the quality of the audio signal or the video signal is identified, the method preferably includes the further step of presenting a feedback message to the patient indicating the presence and optionally the nature of the adverse condition. With regard to options and preferences relating to feedback messages, their format and the manner in which they are presented, reference is made to the above sections of this specification which describe the presentation of other feedback messages, which should apply similarly. For example, feedback messages such as "too dark", "too much noise" or "please avoid shaking" may be displayed on a display or sounded via a speaker of a portable feedback device.

[0073] The method of the present invention may further comprise the steps of: while the feedback device is working and held so that the camera and microphone are pointed to the patient's face, instructing the patient to use an inhaler to inhale the therapeutic aerosol according to the instructions for use. This step may be performed before step (a), and the corresponding instructions may be presented to the patient in various optional forms. For example, if the device is a custom device, the instructions may be printed in the instruction manual or manually distributed with the portable feedback device; or if the device is general (such as a smart phone), the instructions may be with a computer program and configured to perform the method of the present invention by means of such software. Alternatively or in addition, the instructions may be displayed on a display or sounded by a speaker of a portable feedback device in the same manner as one of the feedback messages as described above.

[0074] In another aspect of the invention, there is provided a data processing system comprising a processor adapted to perform the method described herein. As mentioned, the data processing system may be of a general purpose nature, such as a data processing system of a smartphone, adapted or configured to perform the method steps by means of a computer program loaded and executed on the data processing system. However, whether the device incorporating the data processing system is general purpose or custom made, it is preferred that the device is portable and suitable for handheld use, and further comprises a data storage, a camera, a microphone, a display and a speaker.

[0075] In a further aspect, the invention provides a computer program comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to the present invention.Again, the computer preferably comprises a data processing system as defined above.

[0076] In yet another aspect, the present invention provides a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method of the present invention. The computer-readable storage medium may be any type of data storage medium, i.e., optical-based, magnetic-based, semiconductor-based or any other class of media. Preferably, it is a non-volatile storage medium, such as a flash memory, a digital optical disk, a hard disk drive or a solid state drive.

[0077] In another aspect, the present invention relates to a computer-implemented method for detecting and / or analyzing regular movements, the method comprising the following steps:

[0078] (a) obtaining a sequence of digital video signals emitted from the regular motion;

[0079] (b) in the video signal, performing the following steps on a series of frames of the video signal:

[0080] (i) selecting a portion of the frame through which the regular motion passes;

[0081] (ii) calculating an average brightness value within said portion of said frame;

[0082] (iii) storing the average brightness value of the portion of the frame in a buffer;

[0083] (c) repeating steps (i) to (iii) for a frame sequence comprising a plurality of frames, wherein the portion of the frame is identical in each frame, thereby creating the average brightness value buffer;

[0084] (d) calculating the average of the average brightness of all values ​​determined and stored in the brightness value buffer;

[0085] (e) subtracting the determined average brightness mean from each value in the buffer to generate a zero-centered buffer;

[0086] (f) Calculate the zero-crossing rate of the zero-centered buffer.

[0087] The new method developed by the inventors has advantages over prior art methods because it allows real-time analysis even on slower, less powerful processors, such as smartphone processors or Raspberry Pi.

[0088] Preferably, the method steps (a) to (f) are performed by a computing system comprising at least the following:

[0089] - data processing systems;

[0090] -Data storage.

[0091] In some embodiments, the device further comprises a camera.

[0092] In some embodiments, the method steps (a) to (f) are performed by a computing system comprising:

[0093] - data processing systems;

[0094] - data storage;

[0095] -camera;

[0096] -microphone;

[0097] - a display; and

[0098] -speaker.

[0099] In more preferred embodiments, the computing system is portable, more preferably portable and handheld. In some embodiments, in particular embodiments, the computing system is a smartphone.

[0100] Steps (i) to (iii) should be performed over a plurality of frames so as to cover at least 1 second of motion. Preferably, the steps are repeated for at least about 24 frames, or preferably at least about 48 frames.

[0101] The method requires selecting a portion of a frame. The frame should cover a portion of the motion, i.e. the motion is through the device but not entirely performed within the portion of the frame. The portion of the frame should be the same for the motion analysis. Preferably, the portion of the frame is exactly the same in each frame analyzed, preferably the position relative to the regular movement is exactly the same. Preferably, if the camera or video capture device is, for example, a handheld device, the portion of the frame is corrected for movement of the camera.

[0102] In some embodiments, the selection of a portion of a frame is adapted to adjust and correct for movement of a video capture device.

[0103] As already pointed out, the method is particularly suitable for computing systems with limited computing power, such as smartphones.In order to improve the performance of the method, some further adjustments can be made to the method.

[0104] The method utilizes and stores the average brightness value of the portion of the frame. Preferably, the brightness value is

[0105] In some embodiments of the present invention, the buffer used to store average brightness values ​​is a circular buffer.

[0106] If the video signal is converted to grayscale rather than full color, the brightness value may be easier to calculate. Thus, in some embodiments, the method includes in step (b) a further step before step (i):

[0107] (o) Convert the color information of the frame into grayscale.

[0108] The method provides a zero crossing rate which is proportional to the intensity or speed of the regular motion. In this way, the speed or intensity of the regular motion can be determined by comparison with a reference value.

[0109] Therefore, in one embodiment, the present invention relates to a method as defined above, further comprising the step (g):

[0110] (g) comparing the determined zero-crossing rate with a predetermined reference value to obtain an estimate of the speed or intensity of the regular motion.

[0111] Further embodiments and optional features will be apparent to those skilled in the art.

Claims

1. A computer-implemented method for providing feedback to a patient using a particular type of inhaler to inhale a therapeutic aerosol according to instructions for use for or provided to the patient with the inhaler, the method comprising the steps of: (a) obtaining a sequence of digital audio signals and a sequence of digital video signals emitted from the patient while using the inhaler; (b) identifying the type of the inhaler by processing the video signal; (c) identifying whether the patient deviates from the instructions by processing the audio signal and the video signal; (d) if a deviation is identified in step (c), identifying the type of the deviation; as well as (e) presenting one or more feedback messages to the patient, the feedback messages indicating (i) Whether the deviation is identified; as well as (ii) if a deviation is identified, indicating the type of deviation; wherein step b) is performed by analyzing the video signal and the audio signal using an artificial neural network, the artificial neural network being trained to recognize the type of inhaler, and step c) is performed by analyzing the video signal and the audio signal using a method for detecting regular motion, the method for detecting regular motion comprising the following steps: (I) performing the following steps on a series of frames of the video signal of the patient using the inhaler: (i) selecting a portion of the frame through which the regular motion passes; (ii) calculating an average brightness value within said portion of said frame; (iii) storing the average brightness value of the portion of the frame in a buffer; (II) repeating steps (i) to (iii) for a frame sequence comprising a plurality of frames, wherein the portion of the frame is identical in each frame, thereby creating a buffer of luminance values; (III) calculating an average of the average brightness of all values ​​determined and stored in the brightness value buffer; (IV) subtracting the determined average brightness mean from each value in the buffer to generate a zero-centered buffer; (V) calculating the zero-crossing rate of the zero-center buffer; (VI) comparing the determined zero-crossing rate with a predetermined reference value to obtain an estimate of the speed or intensity of the regular motion; The regular motion is shaking; Wherein, steps (a) to (e) are performed by a handheld portable feedback device, the device comprising - data processing systems; - data storage; -camera; -microphone; - a display; and -speaker.

2. The method according to claim 1, wherein: In step (e), the feedback message indicates (i) Whether the deviation is identified; (ii) if a deviation is identified, indicating the type of deviation; as well as (iii) Inhalation action without said deviation.

3. The method according to claim 1, wherein: In step (I), the following steps are performed on a series of frames of the video signal of the patient using the inhaler: (i) converting the color information of the frame into grayscale; (ii) selecting a portion of the frame through which the regular motion passes; (iii) calculating an average brightness value within said portion of said frame; (iv) storing the average brightness value of the portion of the frame in a buffer; In step (II), steps (i) to (iv) are repeated for a frame sequence including a plurality of frames.

4. The method according to claim 1, wherein: Step c) is performed in real time on the portable feedback device and the feedback message is immediately presented in step e).

5. The method according to any one of claims 1 to 4, wherein: In step e), the feedback message is presented using augmented reality.

6. The method according to any one of claims 1 to 4, wherein: The inhaler is a powder inhaler, a nebulizer, a metered dose inhaler or a soft mist inhaler.

7. The method according to any one of claims 1-4, further comprising the step of recognizing the identity of the patient based on the video signal.

8. The method according to any one of claims 1-4, further comprising the step of identifying whether there is a detrimental condition that can adversely affect the quality of the audio signal or the video signal.

9. The method of claim 8, wherein the harmful condition is an environmental condition or a patient activity.

10. The method of claim 9, wherein the environmental condition is luminous contrast or acoustic noise.

11. The method of claim 9, wherein the patient activity is shaking of the portable feedback device.

12. The method of claim 8, further comprising the step of presenting to the patient a feedback message indicating the presence and nature of the adverse condition.

13. The method according to any one of claims 1 to 4, further comprising: The step of instructing the patient to use the inhaler to inhale the therapeutic aerosol in accordance with the instructions for use while the feedback device is operating and held so that the camera and the microphone are directed toward the patient's face.

14. A data processing system comprising: A processor adapted to perform a method according to any one of claims 1 to 13, wherein the data processing system comprises a handheld portable feedback device, the feedback device comprising - data processing systems; - data storage; -camera; -microphone; -monitor; as well as -speaker.

15. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 13, wherein the computer comprises a handheld portable feedback device, the feedback device comprising - data processing systems; - data storage; -camera; -microphone; -monitor; as well as -speaker.

16. The computer program product of claim 15, wherein: The computer comprises a data processing system according to claim 14.

17. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 13, wherein the computer comprises a handheld portable feedback device, the feedback device comprising - data processing systems; - data storage; -camera; -microphone; -monitor; as well as -speaker.