Intelligent training method for inhalation device and related device

Abstract

The invention discloses an intelligent training method for an inhalation device and a related device. The method comprises the following steps: establishing a chronic obstructive pulmonary disease knowledge base; selecting a corresponding teaching video according to the type of the inhalation device used by the patient; when a patient plays a teaching video, correct inhalation device use operation is synchronously displayed in real time through the AI digital person in a mirror image simulation mode, and key operation nodes of the AI digital person are subjected to highlight labeling and slow action playback; recording the process of using the inhalation device by the patient through a camera of the patient client, performing real-time analysis on a recorded video uploaded by the patient, and identifying an operation key node of the patient; scoring according to the operation key node of the patient and the operation key node of the AI digital human to obtain a scoring assessment result; and setting a virtual reward according to the daily medicine inhalation condition of the patient. According to the invention, the operation key nodes of the patient are identified and compared with the AI digital human standard operation for scoring, so that the education and training effect of using the inhalation device by the patient is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of medical device training, and particularly relates to an intelligent training method and device for an inhalation device, and a computing device. Background Art

[0002] Chronic obstructive pulmonary disease is a common, preventable and treatable disease in respiratory diseases. Inhalation therapy is the main treatment method for chronic obstructive pulmonary disease. The drugs used in inhalation therapy can directly act on the lungs, and have the advantages of rapid onset, good efficacy and safety. Research shows that the incidence rate of improper use of the device is as high as 65%-88%, and there is an obvious relationship between the improper use of the inhalation device and the symptom control of chronic obstructive pulmonary disease. The Global Initiative for Chronic Obstructive Lung Disease emphasizes the importance of inhalation management, pointing out that the correct use skills and evaluation of inhalers can improve the treatment prognosis of patients with chronic obstructive pulmonary disease, and suggesting paying attention to the individualized selection of inhalation devices and the education and training of patients on the use of inhalation devices. At present, the training of inhalation devices in clinics is generally face-to-face teaching by medical staff, which is restricted by time, place, etc.

[0003] To solve the above problems, the present invention proposes an intelligent training method and related device for an inhalation device to improve the education and training effect of patients on the use of inhalation devices. Summary of the Invention

[0004] In view of the above problems, the present invention provides an intelligent training method and device for an inhalation device, and a computing device.

[0005] According to one aspect of the present invention, there is provided an intelligent training method for an inhalation device, including:

[0006] Establishing a knowledge base for chronic obstructive pulmonary disease, wherein the knowledge base includes pathological knowledge, treatment plans, drug action mechanisms, inhalation device teaching videos, and daily precautions for patients; retrieving the knowledge base according to the natural language input by the patient client, and dynamically adjusting the push frequency and content of the knowledge base according to the duration and click times of the patient reading the knowledge base; and selecting corresponding teaching videos according to the type of inhalation device used by the patient;

[0007] While the patient is playing the teaching video, the correct operation of using the inhalation device is synchronously and real-timely displayed in the form of mirror simulation by an AI digital human, and the key nodes of the AI digital human operation are highlighted and played back in slow motion to guide the patient to learn; wherein the key nodes of the AI digital human operation include the posture of the inhalation device, the pressing / starting timing, the inhalation duration, and the breath-holding duration; the posture of the inhalation device includes the bending angle of the wrist, the bending angle of the finger, and the posture of the lips;

[0008] Record the process of the patient using the inhalation device through the camera of the patient client, perform real-time analysis on the recorded video uploaded by the patient, and identify the key operation nodes of the patient; score according to the key operation nodes of the patient and the key operation nodes of the AI digital human to obtain the scoring assessment result, where the scoring assessment result includes fail, pass, and effective.

[0009] Set virtual rewards according to the patient's daily inhalation medication situation and display them through a gamified interface to encourage the patient to continue taking medicine.

[0010] In an alternative way, the high-lighting and slow-motion playback of the key operation nodes of the AI digital human to guide the patient to learn further includes:

[0011] Identify the key pose points of the AI digital human through the AlphaPose model, and identify the action sequence of the key pose points of the AI digital human through the RNN recurrent neural network;

[0012] Identify the edge points of the key pose points according to the Sobel operator, perform color replacement on the identified edge points, change them to a highlighted color of yellow or red, and superimpose the image of the key pose points with the highlighted color on the original video frame;

[0013] Increase the frame rate of the original video through the frame interpolation method to achieve a slow-motion effect.

[0014] In an alternative way, the scoring according to the key operation nodes of the patient and the key operation nodes of the AI digital human to obtain the scoring assessment result further includes:

[0015] Traverse each key node of the patient's operation and match the corresponding key node in the key nodes of the AI digital human; among them, if no corresponding key node of the AI digital human is found within the preset time window for the patient's operation, it is determined that the key node is missing;

[0016] Calculate the matching degree of each matched key node according to the Manhattan distance formula and perform tolerance scaling to obtain the node score;

[0017] Measure the degree of order deviation of each matched key node according to the sequence alignment algorithm and convert it into an operation order score;

[0018] Sum up the node scores and operation order scores of all key nodes to obtain the total operation score, and obtain the scoring assessment result according to the total operation score and the set score threshold.

[0019] In an alternative way, the increasing the frame rate of the original video through the frame interpolation method to achieve a slow-motion effect further includes:

[0020] Calculate the motion vector between adjacent frames through the optical flow estimation method; where the optical flow estimation formula is:

[0021]

[0022] Where, is the image intensity gradient; u is the motion vector; is the change in image intensity over time;

[0023] Interpolate each frame of the motion vector to achieve a slow-motion effect; where the interpolation calculation formula is:

[0024]

[0025] Where, f desired is the target frame rate; f original is the original frame rate; represents rounding down.

[0026] In an optional manner, the scaling formula for the tolerance is:

[0027]

[0028] Where, ManhtDistance is the Manhattan distance, a i and b i respectively represent the eigenvalue of the key nodes of the patient and the AI digital human, n is the feature dimension; T is the tolerance threshold.

[0029] In an optional manner, the further steps of measuring the order deviation degree of each matched key node according to the sequence alignment algorithm and converting it into an operation order score include:

[0030] Use the dynamic programming algorithm for sequence alignment and recursively obtain the score matrix; where the recurrence formula of the score matrix is:

[0031]

[0032] Where, [M[i][j] represents the highest score for the matching of the first i key nodes of the patient's operation and the first j key nodes of the AI digital human's operation; match_score(i,j) is the matching score; gap_penalty is the penalty score for insertion or deletion;

[0033] The calculation formula of the operation order score is:

[0034]

[0035] Among them, n and m are the total number of key nodes of the patient operation and the AI digital human operation respectively; M[n][m] is the final score of the score matrix.

[0036] In an alternative manner, the calculation formula of the matching score is:

[0037]

[0038] Among them, D ij is the spatial distance; Δt ij is the time difference; D max , Δt max are the maximum tolerance values of the preset spatial distance and time difference respectively.

[0039] In an alternative manner, the method further includes:

[0040] Statistically analyze the correct operations of each key node type based on historical operation data to obtain the distribution characteristics of the spatial distance and time difference of each key node type;

[0041] Set the initial values of the maximum tolerance values of the preset spatial distance and time difference for each key node type according to the distribution characteristics of the spatial distance and time difference of each key node type.

[0042] According to another aspect of the present invention, there is provided an intelligent training device for an inhalation device, including:

[0043] A knowledge base push module for establishing a knowledge base of chronic obstructive pulmonary disease, where the knowledge base includes pathological knowledge, treatment plans, drug action mechanisms, inhalation device teaching videos, and daily precautions for patients; retrieving the knowledge base according to the natural language input by the patient client, dynamically adjusting the push frequency and content of the knowledge base according to the duration and click times of the patient reading the knowledge base; and selecting corresponding teaching videos according to the type of inhalation device used by the patient;

[0044] A teaching simulation module for, while the patient plays the teaching video, in real-time synchronously displaying the correct operation of using the inhalation device in the form of mirror simulation through an AI digital human, highlighting and slow-motion replaying the key nodes of the AI digital human operation to guide the patient to learn; wherein the key nodes of the AI digital human operation include the posture of the inhalation device, the pressing / starting timing, the inhalation duration, and the breath-holding duration; the posture of the inhalation device includes the bending angle of the wrist, the bending angle of the finger, and the posture of the lips;

[0045] An operation evaluation module, which is used to record the process of a patient using an inhalation device through the camera of the patient's client, perform real-time analysis on the recorded video uploaded by the patient, and identify the key operation nodes of the patient; score according to the key operation nodes of the patient and the key operation nodes of the AI digital human to obtain a scoring assessment result, where the scoring assessment result includes fail, pass, and effective;

[0046] A patient incentive module, which is used to set virtual rewards according to the patient's daily inhalation medication situation and display them through a gamified interface to encourage the patient to continue taking medication.

[0047] According to another aspect of the present invention, there is provided a computing device, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete communication with each other through the communication bus;

[0048] The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the operations corresponding to the above-mentioned intelligent training method for inhalation devices.

[0049] According to the solution provided by the present invention, a knowledge base for chronic obstructive pulmonary disease is established. The knowledge base includes pathological knowledge, treatment plans, drug action mechanisms, teaching videos of inhalation devices, and daily precautions for patients. Retrieve the knowledge base according to the natural language input by the patient client, and dynamically adjust the push frequency and content of the knowledge base according to the duration and click times of the patient reading the knowledge base. In addition, select the corresponding teaching video according to the type of inhalation device used by the patient. While the patient plays the teaching video, the correct operation of the inhalation device is displayed in real time and synchronously in the form of mirror simulation by an AI digital human. Highlight the key nodes of the AI digital human operation and play them back in slow motion to guide the patient to learn. The key nodes of the AI digital human operation include the posture of the inhalation device, the timing of pressing / starting, the duration of inhalation, and the duration of breath holding. The posture of the inhalation device includes the bending angle of the wrist, the bending angle of the fingers, and the posture of the lips. Record the process of the patient using the inhalation device through the camera of the patient client, and perform real-time analysis on the recorded video uploaded by the patient to identify the key nodes of the patient's operation. Score according to the key nodes of the patient's operation and the key nodes of the AI digital human operation to obtain a scoring assessment result. The scoring assessment result includes unqualified, qualified, and effective. Set virtual rewards according to the patient's daily inhalation medication situation and display them through a gamified interface to motivate the patient to continue taking the medication. The present invention identifies the key nodes of the patient's operation and compares and scores them with the standard operation of the AI digital human, improving the education and training effect of the patient's use of the inhalation device. Specifically, using the AI digital human for mirror simulation teaching not only demonstrates the correct operation of the inhalation device, but also makes the learning process more intuitive and understandable by highlighting and playing back the key nodes in slow motion, significantly improving the patient's mastery of the operation skills. By recording and real-time analyzing the patient's operation through the camera and comparing and scoring it with the standard operation of the AI digital human, the patient can immediately obtain feedback on the operation result, which helps to correct mistakes in a timely manner. Combining the patient's medication situation with gamified rewards and displaying them through a gamified interface stimulates the patient's enthusiasm for participation and medication compliance.

[0050] The above description is only an overview of the technical solution of the present invention. In order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable, the following specifically describes the embodiments of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. And throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0052] Figure 1 The flow schematic diagram of the intelligent training method for the inhalation device according to the embodiment of the present invention is shown;

[0053] Figures 2A to 2B The operation schematic diagram of the inhalation device according to the embodiment of the present invention is shown;

[0054] Figure 3 The comparison schematic diagram of the key operation nodes and the standard operation of the AI digital human according to the embodiment of the present invention is shown;

[0055] Figure 4 The framework schematic diagram of the intelligent training device for the inhalation device according to the embodiment of the present invention is shown;

[0056] Figure 5 The structural schematic diagram of the computing device according to the embodiment of the present invention is shown. Detailed implementation manners

[0057] Hereinafter, the exemplary embodiments of the present invention will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present invention can be more thoroughly understood and the scope of the present invention can be fully conveyed to those skilled in the art.

[0058] Figure 1 The flow schematic diagram of the intelligent training method for the inhalation device according to the embodiment of the present invention is shown. Specifically, as Figure 1 shown, the following steps are included:

[0059] Step S101, establish a knowledge base for chronic obstructive pulmonary disease, wherein the knowledge base includes pathological knowledge, treatment plans, drug action mechanisms, inhalation device teaching videos, and daily precautions for patients; retrieve the knowledge base according to the natural language input by the patient client, dynamically adjust the push frequency and content of the knowledge base according to the duration and click times of the patient reading the knowledge base; and select the corresponding teaching video according to the type of inhalation device used by the patient.

[0060] In this embodiment, the knowledge base covers multiple aspects such as pathological knowledge, treatment plans, and drug action mechanisms of chronic obstructive pulmonary disease, and also includes teaching videos of inhalation devices and daily precautions for patients, facilitating patients to obtain comprehensive information in one stop. Among them, the operation steps of the inhalation device are as Figures 2A to 2B shown.

[0061] Step S102, while the patient is playing the teaching video, the AI digital human synchronously displays the correct operation of the inhalation device in real time in the form of mirror simulation, highlights and replays the key nodes of the AI digital human's operation in slow motion to guide the patient to learn; wherein, the key nodes of the AI digital human's operation include the posture of the inhalation device, the timing of pressing / starting, the inhalation duration, and the breath-holding duration; the posture of the inhalation device includes the bending angle of the wrist, the bending angle of the fingers, and the posture of the lips.

[0062] In this embodiment, the AI digital human is synchronized with the teaching video to display the correct operation steps of the inhalation device in real time, highlights and replays the key nodes during the operation to help the patient grasp the essence of the operation and make it easier to understand and imitate. Specifically, as Figure 3 shown, an AI digital human model suitable for simulating the operation of the inhalation device can be a digital model of a real human or a virtual character designed specifically for this purpose. Use motion capture technology to record the correct usage actions of the inhalation device to ensure that the actions of the AI digital human are consistent with the actual operation. Establish an action library containing various usage postures of the inhalation device, covering different types of devices and operation steps. When the video is played, trigger the AI digital human to perform corresponding action simulations. Highlight these key nodes during the AI digital human simulation to highlight the important operation details, and insert slow motion replays at the key nodes to help the patient observe and understand the operation points in detail.

[0063] In an alternative way, the highlighting and slow motion replay of the key nodes of the AI digital human's operation to guide the patient to learn further includes:

[0064] Identify the key pose points of the AI digital human through the AlphaPose model, and identify the action sequence of the key pose points of the AI digital human through the RNN recurrent neural network;

[0065] Identify the edge points of the key pose points according to the Sobel operator, replace the color of the identified edge points with a highlighted color such as yellow or red, and overlay the image of the key pose points with the highlighted color on the original video frame;

[0066] Increase the frame rate of the original video through the frame interpolation method to achieve the slow motion effect.

[0067] Specifically, load the pre-trained AlphaPose model, input the video frames of the AI digital human into the AlphaPose model, and obtain the coordinate information of the key pose points in each frame. Track the coordinate information of the pose points in consecutive frames to ensure that each pose point has a corresponding trajectory throughout the video.

[0068] Arrange the pose point coordinate information output by the AlphaPose model in chronological order to form time series data. Use the pose point data containing different operation action sequences to train the RNN recurrent neural network so that it can recognize different action sequences. Input the pose point sequence of the AI digital human into the trained RNN model to recognize the start, progress, and end of actions, such as pressing, inhaling, and holding the breath of an inhaler, etc.

[0069] For the identified key pose points, use the Sobel operator to detect their edge information, and replace the color of the detected edge point pixels with a highlight color such as yellow or red. Superimpose the pose point image with the highlight color on the original video frame to form a highlight display effect.

[0070] Increase the frame rate of the original video through frame interpolation methods such as linear interpolation or optical flow interpolation, and play the video with the increased frame rate to achieve the slow motion effect.

[0071] In an alternative manner, the increasing of the frame rate of the original video through the frame interpolation method to achieve the slow motion effect further includes:

[0072] Calculate the motion vector between adjacent frames through the optical flow estimation method; where the optical flow estimation formula is:

[0073]

[0074] Where, is the image intensity gradient; u is the motion vector; is the change in image intensity over time;

[0075] Interpolate each frame of the motion vector to achieve the slow motion effect; where the interpolation calculation formula is:

[0076]

[0077] Where, f desired is the target frame rate; f original is the original frame rate; represents rounding down.

[0078] In this embodiment, calculating the motion vector between adjacent frames through the optical flow estimation method can more accurately simulate the motion between frames, thereby providing a smoother and more natural slow motion effect. Calculating the motion vector between frames through interpolation reduces the visual artifacts generated by the interpolated frames.

[0079] Step S103: Record the process of the patient using the inhalation device through the camera of the patient client, perform real-time analysis on the recorded video uploaded by the patient, and identify the key operation nodes of the patient; score the key operation nodes of the patient and the key operation nodes of the AI digital human to obtain a scoring assessment result, where the scoring assessment result includes fail, pass, and effective.

[0080] In this embodiment, the key operation nodes of the patient are compared and scored with the key operation nodes of the AI digital human to obtain an objective and quantitative assessment result. According to the scoring assessment result, the weak links of the patient are explained and demonstrated emphatically to improve the teaching efficiency.

[0081] In an alternative manner, the scoring the key operation nodes of the patient and the key operation nodes of the AI digital human to obtain a scoring assessment result further includes:

[0082] Traverse each key node of the patient's operation and match the corresponding key node in the key nodes of the AI digital human; where, if no corresponding key node of the AI digital human is found within the preset time window for the patient's operation, it is determined that this key node is missing;

[0083] Calculate the matching degree of each matched key node according to the Manhattan distance formula and perform tolerance scaling to obtain a node score;

[0084] Measure the degree of order deviation of each matched key node according to the sequence alignment algorithm and convert it into an operation order score;

[0085] Sum up the node scores and operation order scores of all key nodes to obtain an operation total score, and obtain the scoring assessment result according to the operation total score and the set score threshold.

[0086] In this embodiment, not only the operation result is concerned, but also each key node in the operation process is analyzed in depth. By traversing the key nodes of the patient's operation and the key nodes of the AI digital human, the deficiencies and errors in the patient's operation are accurately identified. The node matching degree is calculated by the Manhattan distance and the operation order is measured by the sequence alignment algorithm, converting the subjective operation evaluation into an objective quantitative index. The evaluation result is more transparent. The tolerance scaling mechanism allows a certain degree of deviation, avoiding the distortion of the evaluation result due to minor operation differences. At the same time, the omission of operations is identified through the preset time window to avoid missing evaluation. Among them, the sequence alignment algorithm (such as the Needleman-Wunsch algorithm) measures the degree of deviation between the order of the key nodes of the patient's operation and the order of the AI digital human's operation. The degree of deviation is converted into an operation order score, and the greater the deviation, the lower the score.

[0087] In an alternative approach, the step of measuring the order deviation degree of each matched key node according to the sequence alignment algorithm and converting it into an operation order score further includes:

[0088] Use the dynamic programming algorithm for sequence alignment to recursively obtain the score matrix; where the recurrence formula of the score matrix is:

[0089]

[0090] where, [M[i][j] represents the highest score for the match between the first i key nodes of the patient's operation and the first j key nodes of the AI digital human's operation; match_score(i,j) is the match score; gap_penalty is the penalty score for insertion or deletion;

[0091] The calculation formula for the operation order score is:

[0092]

[0093] where, n and m are the total numbers of key nodes of the patient's operation and the AI digital human's operation respectively; M[n][m] is the final score of the score matrix.

[0094] In this embodiment, by using dynamic programming for sequence alignment to calculate the order deviation between the patient's operation and the AI digital human's operation, the subtle differences in the operation order can be captured.

[0095] For example, match_score is 20 (the score for each matched node), and gap_penalty is 10 (the penalty score for each insertion or deletion). The patient's operation has 4 key nodes, and the AI digital human has 5 key nodes. The score matrix M:

[0096] M[0][0] = 0

[0097] M[1][0] = -10

[0098] M[2][0] = -20 ...

[0100] M[4][5] = 60

[0101] where, M[4][5] = 60 because: "Action 1" and "Action 2" are matched, with a score of 20*2 = 40. "Action 2" is skipped, so there is 1 deletion, -10 points. "Action 4" is matched, +20 points. "Action 5" is matched, +20 points.

[0102] Calculate the operation order score:

[0103] O score=(M[4][5]) / max(4,5)*100=(60 / 5)*100 = 1200

[0104] / 5 = 72%, that is, the order score of the patient's operation is 72 points.

[0105] In an alternative manner, the calculation formula for the matching score is:

[0106]

[0107] where D ij is the spatial distance; Δt ij is the time difference; D max , Δt max are respectively the maximum tolerance values of the preset spatial distance and time difference.

[0108] In this embodiment, by comprehensively scoring in two dimensions of space and time, the quality of the operation can be more accurately reflected.

[0109] In an alternative manner, the method further includes:

[0110] Statistically analyze the correct operations of each key node type based on historical operation data to obtain the distribution characteristics of the spatial distance and time difference of each key node type;

[0111] Set the initial values of the maximum tolerance values of the preset spatial distance and time difference for each key node type according to the distribution characteristics of the spatial distance and time difference of each key node type.

[0112] In this embodiment, by analyzing the distribution characteristics of key nodes determined from historical operation data, it can better adapt to the actual operation situations of different patients. By using a data-driven method, the subjectivity of artificially setting the tolerance can be reduced. For example, if the spatial distance distribution of a certain key node is relatively concentrated (with a small standard deviation), the preset spatial distance can be set to a smaller value, indicating a lower tolerance for the spatial deviation of this node.

[0113] In an alternative manner, the scaling formula for the tolerance is:

[0114]

[0115] where ManhtDistance is the Manhattan distance, a i 、b i respectively represent the characteristic values of the key nodes of the patient and the AI digital human, n is the number of characteristic dimensions; T is the tolerance threshold.

[0116] For example, the patient has 5 key features, a = [10, 20, 30, 40, 50], which are the feature values of the AI digital human, and b = [12, 25, 28, 45, 50].

[0117] Manhattan distance = |10 - 12| + |20 - 25| + |30 - 28| + |40 - 45| + |50 - 50| = 2 + 5 + 2 + 5 + 0 = 14.

[0118] Tolerance = 1 - min(14 / 30, 1) = 1 - 0.4667 ≈ 0.5333.

[0119] In step S104, set virtual rewards according to the patient's daily inhalation medication situation and display them through a gamified interface to motivate the patient to continue taking medication.

[0120] In this embodiment, by combining the patient's daily inhalation medication behavior with gamification elements, the medication compliance of the patient is effectively improved. The gamified interface provides timely feedback, rewards, and goals, stimulating the enthusiasm and participation of the patient.

[0121] According to the solution provided by the present invention, a knowledge base for chronic obstructive pulmonary disease is established. The knowledge base includes pathological knowledge, treatment plans, drug action mechanisms, inhalation device teaching videos, and daily precautions for patients. The knowledge base is retrieved according to the natural language input by the patient client, and the push frequency and content of the knowledge base are dynamically adjusted according to the duration and click times of the patient reading the knowledge base. In addition, according to the type of inhalation device used by the patient, the corresponding teaching video is selected. While the patient plays the teaching video, the correct operation of the inhalation device is synchronously displayed in real time in the form of mirror simulation by an AI digital human. The key nodes of the AI digital human operation are highlighted and slow-motion replayed to guide the patient to learn. The key nodes of the AI digital human operation include the posture of the inhalation device, the pressing / starting timing, the inhalation duration, and the breath-holding duration. The posture of the inhalation device includes the bending angle of the wrist, the bending angle of the fingers, and the posture of the lips. The process of the patient using the inhalation device is recorded by the camera of the patient client, and the recorded video uploaded by the patient is analyzed in real time to identify the key nodes of the patient's operation. The key nodes of the patient's operation are scored against the key nodes of the AI digital human operation to obtain a scoring and assessment result, where the scoring and assessment result includes fail, pass, and effective. Virtual rewards are set according to the patient's daily inhalation medication situation and displayed through a gamified interface to motivate the patient to continue taking medication. The present invention identifies the key nodes of the patient's operation and compares and scores them with the standard operation of the AI digital human, improving the education and training effect of the patient's use of the inhalation device. Specifically, using the AI digital human for mirror simulation teaching not only shows the correct operation of the inhalation device but also makes the learning process more intuitive and understandable by highlighting and slow-motion replaying the key nodes, significantly improving the patient's mastery of the operation skills. By recording and analyzing the patient's operation in real time through the camera and comparing and scoring it with the standard operation of the AI digital human, the patient can immediately obtain feedback on the operation result, which helps to correct mistakes in a timely manner. Combining the patient's medication situation with gamified rewards and displaying them through a gamified interface stimulates the patient's enthusiasm for participation and medication compliance.

[0122] Figure 4 Fig. shows a schematic framework diagram of an intelligent training device for an inhalation device according to an embodiment of the present invention. The intelligent training device for an inhalation device includes:

[0123] A knowledge base push module 410, configured to establish a knowledge base for chronic obstructive pulmonary disease, where the knowledge base includes pathological knowledge, treatment plans, drug action mechanisms, inhalation device teaching videos, and daily precautions for patients; retrieve the knowledge base according to the natural language input by the patient client, and dynamically adjust the push frequency and content of the knowledge base according to the duration and click times of the patient reading the knowledge base; and select the corresponding teaching video according to the type of inhalation device used by the patient;

[0124] A teaching simulation module 420, which is used to, while a patient plays a teaching video, synchronously and in real time display the correct operation of using an inhalation device in the form of mirror simulation by an AI digital human, highlight and slow-motion replay the key nodes of the AI digital human's operation to guide the patient to learn. Among them, the key nodes of the AI digital human's operation include the posture of the inhalation device, the pressing / starting timing, the inhalation duration, and the breath-holding duration. The posture of the inhalation device includes the bending angle of the wrist, the bending angle of the fingers, and the posture of the lips.

[0125] An operation evaluation module 430, which is used to record the process of a patient using an inhalation device through the camera of the patient client, perform real-time analysis on the recorded video uploaded by the patient, and identify the key nodes of the patient's operation. Score according to the key nodes of the patient's operation and the key nodes of the AI digital human's operation to obtain a scoring assessment result. Among them, the scoring assessment result includes failed, qualified, and effective.

[0126] A patient incentive module 440, which is used to set virtual rewards according to the patient's daily inhalation medication situation and display them through a gamified interface to encourage the patient to continue taking medication.

[0127] Figure 5 The structural schematic diagram of an embodiment of the computing device of the present invention is shown. The specific implementation of the present invention does not limit the specific implementation of the computing device.

[0128] As Figure 5 shown, the computing device may include: a processor 502, a communications interface 504, a memory 506, and a communication bus 508.

[0129] Among them: The processor 502, the communications interface 504, and the memory 506 communicate with each other through the communication bus 508. The communications interface 504 is used to communicate with network elements of other devices such as clients or other servers. The processor 502 is used to execute a program 510, and specifically can execute the relevant steps in the embodiment of the intelligent training method for the inhalation device described above.

[0130] Specifically, the program 510 may include program codes, and the program codes include computer operation instructions.

[0131] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. One or more processors included in the computing device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0132] A memory 506 for storing a program 510. The memory 506 may include high-speed RAM memory and may also include non-volatile memory, such as at least one magnetic disk memory.

[0133] According to the solution provided by the present invention, a knowledge base for chronic obstructive pulmonary disease is established. The knowledge base includes pathological knowledge, treatment plans, drug action mechanisms, inhalation device teaching videos, and daily precautions for patients. Retrieve the knowledge base according to the natural language input by the patient client, and dynamically adjust the push frequency and content of the knowledge base according to the duration and click times of the patient reading the knowledge base. Also, select the corresponding teaching video according to the type of inhalation device used by the patient. While the patient is playing the teaching video, the correct inhalation device usage operation is displayed in real-time synchronization in the form of mirror simulation by an AI digital human, and the key nodes of the AI digital human operation are highlighted and slow-motion replayed to guide the patient to learn. Among them, the key nodes of the AI digital human operation include the posture of the inhalation device, the timing of pressing / starting, the duration of inhalation, and the duration of breath holding. The posture of the inhalation device includes the bending angle of the wrist, the bending angle of the fingers, and the posture of the lips. Record the process of the patient using the inhalation device through the camera of the patient client, and perform real-time analysis on the recorded video uploaded by the patient to identify the key nodes of the patient's operation. Score according to the key nodes of the patient's operation and the key nodes of the AI digital human operation to obtain a scoring assessment result. The scoring assessment result includes fail, pass, and effective. Set virtual rewards according to the patient's daily inhalation medication situation and display them through a gamified interface to encourage the patient to continue taking medicine. The present invention identifies the key nodes of the patient's operation and compares and scores them with the standard operation of the AI digital human, improving the education and training effect of the patient's use of the inhalation device. Specifically, using the AI digital human for mirror simulation teaching not only shows the correct inhalation device usage operation, but also makes the learning process more intuitive and understandable by highlighting and slow-motion replaying the key nodes, significantly improving the patient's mastery of operation skills. By recording and real-time analyzing the patient's operation through the camera and comparing and scoring it with the standard operation of the AI digital human, the patient can immediately obtain feedback on the operation result, which helps to correct errors in a timely manner. Combining the patient's medication situation with gamified rewards and displaying them through a gamified interface stimulates the patient's enthusiasm for participation and medication compliance.

[0134] Those skilled in the art can understand that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all the features disclosed in this specification (including the accompanying claims, abstract and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose. In addition, those skilled in the art can understand that although some of the embodiments herein include certain features included in other embodiments rather than other features, the combination of the features of different embodiments means that it is within the scope of the present invention and forms different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination. The present invention can be implemented by means of hardware including several different elements and by means of a properly programmed computer. In the unit claims listing several devices, several of these devices can be embodied by the same hardware item. The steps in the above embodiments, unless otherwise specified, should not be construed as a limitation on the execution order.

Claims

1. An intelligent training method for an inhalation device, characterized in that: include: Establish a knowledge base for chronic obstructive pulmonary disease, wherein the knowledge base includes pathological knowledge, treatment plans, drug mechanisms of action, inhalation device teaching videos, and daily precautions for patients; retrieve the knowledge base based on natural language input by the patient client, dynamically adjust the push frequency and content of the knowledge base based on the patient's reading time and click count; and select corresponding teaching videos based on the type of inhalation device used by the patient; While the patient is playing the teaching video, the AI ​​digital human will demonstrate the correct operation of the inhalation device in real time in the form of mirror simulation, highlight the key nodes of the AI ​​digital human operation and play it back in slow motion to guide the patient's learning; the key nodes of the AI ​​digital human operation include the posture of the inhalation device, the timing of pressing / starting, the duration of inhalation and the duration of breath holding; the posture of the inhalation device includes the bending angle of the wrist, the bending angle of the fingers and the posture of the lips; The process of the patient using the inhalation device is recorded through the camera of the patient client, and the recorded video uploaded by the patient is analyzed in real time to identify the patient's key operation nodes; the patient's key operation nodes and the AI ​​digital human's key operation nodes are scored to obtain the scoring and assessment results, among which the scoring and assessment results include failure, qualification and validity; Set virtual rewards based on the patient's daily inhalation medication use and display them through a gamified interface to encourage patients to continue taking medication.

2. The intelligent training method for an inhalation device according to claim 1, characterized in that: The highlighting and slow-motion playback of the key nodes of the AI ​​digital human operation and guiding the patient to learn further include: The AlphaPose model is used to identify the key posture points of the AI ​​digital human, and the RNN recurrent neural network is used to identify the action sequence of the key posture points of the AI ​​digital human; Identify edge points of key pose points according to the Sobel operator, replace the color of the identified edge points, change them into yellow or red highlight colors, and superimpose the key pose point images with highlight colors on the original video frame; The frame rate of the original video is increased through frame interpolation method to achieve slow motion effect.

3. The intelligent training method for an inhalation device according to claim 1, characterized in that: Scoring the patient's key operation nodes and the AI ​​digital human's key operation nodes to obtain the scoring and assessment results further includes: Traverse each key node of the patient's operation and match the corresponding key node in the AI ​​digital human key nodes; if the patient's operation does not find the corresponding AI digital human key node within the preset time window, determine that the key node is missing; The matching degree of each matching key node is calculated according to the Manhattan distance formula and the tolerance is scaled to obtain the node score; The degree of order deviation of each matched key node is measured according to the sequence alignment algorithm and converted into an operation order score; The node scores and operation sequence scores of all key nodes are summed up to get the total operation score, and the scoring and assessment results are obtained based on the total operation score and the set score threshold.

4. The intelligent training method for an inhalation device according to claim 2, characterized in that: The step of increasing the frame rate of the original video by using a frame interpolation method to achieve a slow motion effect further includes: The motion vector between adjacent frames is calculated by the optical flow estimation method; wherein the optical flow estimation formula is: in, is the image intensity gradient; u is the motion vector; is the image intensity variation over time; The motion vector is interpolated for each frame to achieve a slow motion effect; the interpolation calculation formula is: Among them, f desired is the target frame rate; f original is the original frame rate; Indicates rounding down.

5. The intelligent training method for an inhalation device according to claim 3, characterized in that: The scaling formula for tolerance is: Among them, ManhtDistance is the Manhattan distance, a i 、b i They represent the characteristic values ​​of the key nodes of the patient and the AI ​​digital human respectively, n is the characteristic dimension; T is the tolerance threshold.

6. The intelligent training method for an inhalation device according to claim 3, characterized in that: The step of measuring the order deviation degree of each matched key node according to the sequence alignment algorithm and converting it into an operation order score further includes: The dynamic programming algorithm is used to perform sequence alignment and recursively obtain the score matrix; the recursive formula of the score matrix is: Where [M[i][j] represents the highest score of the match between the first i key nodes operated by the patient and the first j key nodes operated by the AI ​​digital human; match_score(i,j) is the matching score; gap_penalty is the penalty score for insertion or deletion; The calculation formula for the operation order score is: Among them, n and m are the total number of key nodes of patient operation and AI digital human operation respectively; M[n][m] is the final score of the scoring matrix.

7. The intelligent training method for an inhalation device according to claim 6, characterized in that: The matching score is calculated as: Among them, D ij is the spatial distance; Δt ij is the time difference; D max , Δt max They are the maximum tolerance values ​​of preset spatial distance and time difference respectively.

8. The intelligent training method for an inhalation device according to claim 7, characterized in that: The method further comprises: According to the historical operation data, the correct operation of each key node type is counted to obtain the distribution characteristics of the spatial distance and time difference of each key node type; According to the distribution characteristics of the spatial distance and time difference of each key node type, the initial value of the maximum tolerance value of the preset spatial distance and time difference of each key node type is set.

9. An intelligent training device for an inhalation device, characterized in that: include: A knowledge base push module is used to establish a knowledge base for chronic obstructive pulmonary disease, wherein the knowledge base includes pathological knowledge, treatment plans, drug mechanisms of action, inhalation device teaching videos, and daily precautions for patients; retrieve the knowledge base according to the natural language input by the patient client, dynamically adjust the push frequency and content of the knowledge base according to the patient's reading time and click times of the knowledge base; and select the corresponding teaching video according to the type of inhalation device used by the patient; The teaching simulation module is used to show the correct operation of the inhalation device in real time and synchronously in the form of mirror simulation through the AI ​​digital human while the patient is playing the teaching video, highlight the key nodes of the AI ​​digital human operation and play it back in slow motion to guide the patient's learning; wherein, the key nodes of the AI ​​digital human operation include the posture of the inhalation device, the timing of pressing / starting, the duration of inhalation and the duration of breath holding; the posture of the inhalation device includes the bending angle of the wrist, the bending angle of the fingers and the posture of the lips; The operation evaluation module is used to record the process of the patient using the inhalation device through the camera of the patient client, conduct real-time analysis on the recorded video uploaded by the patient, and identify the patient's key operation nodes; score the patient's key operation nodes and the AI ​​digital human's key operation nodes to obtain the scoring and assessment results, where the scoring and assessment results include failure, qualification, and validity; The patient incentive module is used to set virtual rewards based on the patient's daily inhalation medication use, and to display them through a gamified interface to encourage patients to continue taking medication.

10. A computing device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned intelligent training method for the inhalation device.

Images