Old person medicine taking detection method and system based on human key point detection and transformer model

Abstract

The application relates to the fields of machine learning and wisdom pension services, and relates to an old person medicine taking detection method and system based on human body key point detection and a Transformer model, which comprises the following steps: collecting spatial data overall data and human body face area data through a video device respectively to obtain human body key point features, face key point features and medicine features; performing logical judgment according to the human body key point features, the face key point features and the medicine features to obtain a first result R1 of whether correct medicine taking is completed; performing frame-by-frame modeling on the human body key point features through a Transformer time sequence model, classifying an input action classifier containing global information feature sets of each frame of action of the video, and obtaining a second result R2 of whether the action is a medicine taking action; and judging and outputting a result of whether correct medicine taking is completed according to the result R1 and the result R2. The application can improve the distinguishing degree of similar actions such as medicine taking and water drinking, and accurately detects whether an old person correctly completes medicine taking according to the amount.

Description

Technical Field

[0001] This invention relates to the fields of machine learning and smart elderly care services, specifically to a method and system for detecting medication use in the elderly based on human key point detection and the Transformer model. Background Technology

[0002] As the birth rate gradually declines, my country is entering an aging society. According to the seventh national census, the proportion of the population aged 65 and above in my country exceeds 13.50%, and the degree of population aging has surpassed the global average. my country's aging population problem is becoming increasingly serious, and ensuring that everyone is "cared for in their old age" has become a pressing issue we must address. Many elderly people suffer from multiple chronic diseases, requiring them to take medication correctly and on time. However, the elderly often have poor memory, and some even suffer from amnesia, frequently confusing the timing, frequency, type, and quantity of medications. Their children cannot always be by their side to remind them, making it difficult to guarantee that the elderly will take their medication correctly and in the correct dosage. This is not only detrimental to their condition but may also cause adverse drug reactions. Therefore, ensuring that they take their medication correctly and in the correct dosage is crucial. Current related products can only guarantee that the medication is correctly dispensed, but cannot guarantee that forgetful elderly people will take the correct dosage, and they cannot provide real-time panoramic monitoring. Summary of the Invention

[0003] To address the technical problems existing in the prior art, this invention provides a method and system for detecting whether elderly people are taking medication based on human body key point detection and Transformer model. The method detects whether elderly people are taking medication by using key points of various parts of the human body, and the Transformer time series model can improve the differentiation of similar actions such as taking medication and drinking water, and accurately detect whether elderly people are taking medication correctly and in the correct dosage.

[0004] The first objective of this invention is to provide a method for detecting medication use in the elderly based on human key point detection and a Transformer model.

[0005] The second objective of this invention is to provide a medication detection system for the elderly based on human key point detection and a Transformer model.

[0006] The first objective of this invention can be achieved by adopting the following technical solution:

[0007] A method for detecting medication use in the elderly based on human keypoint detection and Transformer model, the method comprising the following steps:

[0008] The overall spatial data and facial region data of the human body are collected separately using video equipment to obtain a video set;

[0009] The video set was used to detect and extract human body key point set, mouth key point set and drug information set through different neural network models. The human body key point set was used as human body key point feature, the mouth key point set was used as facial key point feature, and the drug information set was used as drug feature.

[0010] Logical judgments are made based on the relative positions of key points in the human body key point features, the relative positions of key points in the facial key point features, and the type and quantity of drugs in the drug features to obtain the first result R1 of whether the correct medication has been taken.

[0011] The key features of the human body are modeled frame by frame using the Transformer temporal model to obtain a feature set containing global information of the action in each frame of the video. The feature set containing global information of the action in each frame of the video is then input into the action classifier for classification to obtain a second result R2 of whether the action is medication.

[0012] Based on the first result R1 and the second result R2, determine and output whether the medication was taken correctly.

[0013] Preferably, the step of making logical judgments based on the relative positions of key points in the human body key point features, the relative positions of key points in the facial key point features, and the type and quantity of drugs in the drug features to obtain the first result R1 of whether the correct medication has been taken includes:

[0014] When the acute angle between the line connecting the elbow key point and the corresponding shoulder on the side of the body and the body's midline is greater than 30 degrees, and the wrist key point is higher than the shoulder and lower than the nose, the arm is considered raised, and the result is set to R. 11 It is true.

[0015] When the area enclosed by the mouth contour points first increases and then decreases, it is considered as a drug-taking action, and the result R is set. 12 It is true.

[0016] When the type and quantity of the drug in the drug characteristics match the preset data in the database, it is considered that the medication has been taken correctly, and the result R is set. 13 If true, otherwise it is considered inaccurate or incorrect medication; set the result R. 13 No;

[0017] The result R 11 Result R 12 and result R 13 Perform a logical judgment, when the result R 11 Result R 12 and result R 13 If all values ​​are true, then the result R1 is true, which is considered as completing the correct medication.

[0018] Preferably, the Transformer temporal model includes a Transformer submodule, which models the obtained human keypoint features frame by frame using a self-attention mechanism.

[0019]

[0020]

[0021]

[0022]

[0023] Where Q, K, and V represent the Query matrix, Key matrix, and Value matrix in the attention mechanism, respectively; W q W k W v These are all vectors used to abstract features to form the Query matrix, Key matrix, and Value matrix mentioned above. Each element in J1 represents a key human feature. The feature is the global motion information for each frame of the video, where * denotes matrix multiplication;

[0024]

[0025] Where, x i For matrix operations Q*K T Each element in the resulting matrix, where N is the total number of elements in the resulting matrix. With the natural logarithm e as the base, x i This refers to the power operation of the exponent.

[0026] The second objective of this invention can be achieved by adopting the following technical solution:

[0027] A system for detecting medication use in the elderly based on human keypoint detection and Transformer models, the system comprising:

[0028] The video acquisition module is used to acquire overall spatial data and facial region data of the human body through video equipment to obtain a video set.

[0029] The feature extraction and detection module is used to extract human body key point sets, mouth key point sets, and drug information sets from the video set through different neural network models. The human body key point set is used as human body key point features, the mouth key point set is used as facial key point features, and the drug information set is used as drug features.

[0030] The logic judgment module is used to make logical judgments based on the relative positions of key points in the human body key point features, the relative positions of key points in the facial key point features, and the type and quantity of drugs in the drug features, to obtain a first result R1 indicating whether the correct medication has been taken. The human body key point features are modeled frame by frame using the Transformer temporal model to obtain a feature set containing global action information for each frame of the video. This feature set is then input into an action classifier for classification to obtain a second result R2 indicating whether the action is medication taking. Based on results R1 and R2, the module judges and outputs the result indicating whether the correct medication has been taken.

[0031] The interactive display module shows the result information of whether the medication was taken correctly through text or voice broadcast.

[0032] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0033] This invention provides a method and system for detecting medication use in the elderly based on human keypoint detection and Transformer models. By utilizing human keypoint detection technology, it determines whether the elderly have swallowed the medication correctly by examining key points around the mouth. The YOLO classification algorithm is used to determine the type and quantity of the medication, thus detecting whether the elderly have taken the medication correctly. In the temporal action features constructed from human keypoints, the Transformer framework in temporal modeling can effectively utilize the temporal information implied in the actions generated by human keypoint modeling to distinguish similar actions such as taking medication and drinking water, accurately detecting whether the elderly have taken the medication correctly and in the correct dosage. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0035] Figure 1 This is a flowchart of the method for detecting medication use in the elderly according to an embodiment of the present invention;

[0036] Figure 2 This is a logic block diagram of the method for detecting medication use in the elderly according to an embodiment of the present invention. Detailed Implementation

[0037] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments, and the implementation of the present invention is not limited thereto. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1:

[0039] To address the problem of elderly people being unable to take medication on time and in the correct dosage due to forgetfulness, this invention provides a medication detection method for the elderly based on human keypoint detection and a Transformer model, combining deep learning methods. This invention utilizes human keypoint detection technology to determine whether the elderly person has swallowed the medication correctly by using key points around the mouth. The YOLO classification algorithm is used to determine the type and quantity of medication, thus detecting whether the elderly person has taken the medication correctly. In the temporal action features constructed from human keypoints, the Transformer framework in temporal modeling effectively utilizes the temporal information implicit in the actions generated by human keypoint modeling to distinguish similar actions such as taking medication and drinking water. Global human keypoints, facial keypoints, and medication information are used as conditional criteria. The temporal human keypoints are then input into the Transformer temporal model for action classification, improving the distinguishability of similar actions such as taking medication and drinking water, thereby reducing the false positive rate and accurately detecting whether the elderly person has taken medication on time and in the correct dosage.

[0040] like Figure 1-2 As shown, the method for detecting medication use in the elderly based on human key point detection and Transformer model according to the present invention includes the following steps:

[0041] S1. Collect overall spatial data and facial region data using video equipment to obtain a video set v. N .

[0042] In this embodiment, the video device uses two RGB-D cameras (depth + color cameras) or two ordinary cameras to collect overall spatial data and facial region data of the elderly person, respectively, to obtain a video set v. N .

[0043] S2, combine the video collection v N Different neural network models are input separately to detect and extract sets of human body key points, mouth key points, and drug information. The human body key point set is used as human body key point feature J1, the mouth key point set in the facial key points is used as facial key point feature J2, and the drug information set is used as drug feature J3.

[0044] In this embodiment, the video collection vN Make three copies to get three identical video sets. Video collection and video collection This allows subsequent steps to be computed in parallel, speeding up the processing.

[0045] Based on video collection A neural network model trained using the open-source PP-TinyPose algorithm was used to obtain a set of human body keypoints, which was then used as the human body keypoint feature J1. The human body keypoints consist of 17 points: nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right groin, left and right knees, and left and right ankles.

[0046] Based on video collection A neural network model trained using the open-source Face_Landmark algorithm was used to obtain a set of mouth keypoints in the facial keypoints, and this set of mouth keypoints was used as the facial keypoint feature J2. The facial keypoints include 68 keypoints, covering the facial contour, eyes, nose, and mouth, with keypoints involving the mouth being numbered 49 to 68.

[0047] Based on video collection A neural network model trained using the open-source YOLOv7 algorithm is used to obtain a set of drug information consisting of the quantity and type of drugs predicted by the model. This set of drug information is used as drug feature J3, which represents the quantity and type of drugs the elderly person is currently preparing to take. The feature set J = {J1, J2, J3} is then defined.

[0048] A model trained using the open-source PP-TinyPose algorithm is used to obtain a set of human keypoints, which are then used as features J1. This open-source algorithm boasts advantages such as speed and high accuracy, and performs well in detecting small targets. Furthermore, the algorithm is lightweight and suitable for rapid deployment on various devices. The algorithm employs a top-down approach, first detecting the human bounding box and then detecting the corresponding keypoints, thus ensuring high accuracy in keypoint detection. Simultaneously, AID and UDP strategies are incorporated in preprocessing and postprocessing, with DARK strategies added in postprocessing to guarantee high model performance. The AID strategy, Augmentation by Information Dropping, improves the model's ability to locate keypoints through selective information loss. UDP, Unbiased Data Processing, improves model accuracy by encoding and decoding unbiased data. DARK, Distribution-Aware Coordinate Representation of Keypoints, improves the model's accuracy under low-resolution heatmaps by introducing distribution-aware coordinates.

[0049] The model trained using the open-source Face_Landmark algorithm obtains the set of mouth key points in the facial key points and uses them as features J2. The algorithm adopts a multi-task learning method and uses Wing Loss as the loss function to make the algorithm have better fitting. The algorithm can quickly detect 68 key points of the human face.

[0050] A model trained using the open-source YOLOv7 algorithm is used to obtain a set consisting of the quantity and type of drugs, which is then used as feature J3. The algorithm extracts features through convolution operations and predicts object bounding boxes based on these features. Then, non-maximum suppression (NMS) is used to remove redundant windows to obtain detection boxes. Based on this, the number of objects to be detected can be obtained by counting the detection boxes.

[0051] S3. Based on the relative positions of key points in human body key point feature J1, the relative positions of key points in facial key point feature J2, and the type and quantity of medicine in medicine feature J3, a logical judgment is made to obtain the first result R1 of whether the correct medication has been taken. The human body key point feature J1 is then modeled frame-by-frame using a Transformer temporal model to obtain a feature set containing global action information for each frame of the video. This feature set is then input into an action classifier for classification to obtain the second result R2 of whether the medication has been taken. Based on the first result R1 and the second result R2, the result of whether the correct medication has been taken is determined and output.

[0052] S31. Based on the relative positions of each key point in human body key point feature J1 and facial key point feature J2, and the type and quantity of medicine in medicine feature J3, a conditional judgment is made to obtain the first result R1 of whether the correct medication has been taken, specifically including:

[0053] (1) For the overall key features of the human body, when the acute angle between the line connecting the elbow (regardless of left or right elbow) key point and the corresponding shoulder and the body's midline is greater than 30 degrees, and the wrist (regardless of left or right wrist) key point is higher than the shoulder and lower than the nose, it is considered that the arm is raised, i.e., result R 11 It is true.

[0054] (2) For local facial key features, when the area enclosed by the mouth contour points first increases and then decreases, it is considered as a drug-taking action, i.e., result R 12 It is true.

[0055] (3) For drug information, the obtained type, quantity, and current medication time are compared with preset data in the database. If there is no error, it means the medication was taken correctly. That is, R 13 If R is true, then it is false. Otherwise, it is false. 13 If not, the system will issue a voice alarm, "Beep beep beep, medication information error." The preset data in the database can be set by doctors or family members based on the doctor's treatment recommendations.

[0056] (4) The result R 11 Result R 12 and result R 13 Perform a logical judgment, when the result R 11 Result R 12 Result R 13 If both are true, then the result R1 is true, which is considered as completing the correct medication.

[0057] S32. The human body key point features J1 are modeled frame by frame using the Transformer temporal model to obtain a feature set containing global information of the action in each frame of the video. The feature set containing global information of the action in each frame of the video is input into the action classifier for classification to obtain the classification result R2.

[0058] Specifically, the Transformer temporal model includes at least one Transformer submodule, which models the obtained human keypoint features J1 frame by frame using a self-attention mechanism:

[0059]

[0060]

[0061]

[0062]

[0063] Where Q, K, and V represent the Query matrix, Key matrix, and Value matrix in the attention mechanism, respectively; W q W k W v These are all vectors used to abstract features to form the Query matrix, Key matrix, and Value matrix mentioned above. Each element in J1 represents a key human feature. The feature is the global motion information for each frame of the video, where * denotes matrix multiplication;

[0064]

[0065] Where, x i For matrix operations Q*K T Each element in the resulting matrix, where N is the total number of elements in the resulting matrix. With the natural logarithm e as the base, x i This refers to the exponentiation operation. The significance of this operation is to normalize the resulting matrix, using the weight assigned to each element as its new value. In this way, we obtain a feature set containing global motion information for each frame of the video, after abstract representation.

[0066] The self-attention mechanism specifically includes: first, traversing each part of the overall human keypoint features, calculating the degree of association between the part and each part (including itself) that makes up the whole, and then assigning a series of weight parameters, namely attention coefficients, to each part (including itself) according to the degree of association. Multiplying the attention coefficients by the original values ​​of each part yields a new feature representation.

[0067] Furthermore, the first Transformer submodule generates a set of keypoint features containing global motion information for each frame of the video. The input is then passed to the next Transformer submodule to continue modeling:

[0068]

[0069]

[0070]

[0071]

[0072] This process continues until the last Transformer submodule completes its modeling. The final output is a set of keypoint features that, after being abstracted, contains global motion information for each frame of the video.

[0073] To quickly aggregate temporal information from videos, the Transformer temporal model can employ a multi-layer soft-attention mechanism, and it can also utilize a multi-head mechanism in terms of width.

[0074] In soft-attention, the attention coefficient is a decimal between [0, 1], which makes the attention more refined.

[0075] In the multi-head mechanism, multiple self-attention modules are parallelized to obtain a multi-faceted degree of correlation, thereby improving the feature representation capability.

[0076] A set of keypoint features containing global motion information for each frame of the video. The input action classifier is used for classification and the results are normalized by softmax to obtain the result R2. When R2 is the action of taking medicine, it is considered true (i.e., the medication is taken). If R1 and R2 are both true (i.e., the medication is taken), it is considered that the action of taking medicine is completed.

[0077] The action classifier includes a score regressor and a classifier. The score regressor and the classifier are implemented through two fully connected layers. The score regressor is used to calculate the score of each possible action in each frame of the video and perform softmax normalization on the score of each action in the video. The classifier is used to classify and select the action with the largest action score as the classification result R2.

[0078] The fully connected layer is:

[0079]

[0080] This allows us to calculate the score for each type of action, i.e. The scores for each of the n distinct action classes are assigned, and then softmax normalized. The action corresponding to the largest number among the n numbers from a1 to an is taken as the classification result R2. If the classification result R2 corresponds to the action of taking medicine, then R2 is considered true (i.e., the medicine was taken); otherwise, R2 is considered false. For example, the result of R2 could be a specific action from multiple classes, such as drinking water, taking medicine, or eating. If the score for drinking water is 0.3, the score for taking medicine is 0.5, and the score for eating is 0.2, then the score for taking medicine is the highest, and R2 is considered true. If the score for drinking water is 0.5, the score for taking medicine is 0.2, and the score for eating is 0.3, then the score for drinking water is the highest, and R2 is considered true. If R2 does not correspond to the action of taking medicine, then R2 is considered false.

[0081] S33: When both the logical judgment result R1 and the classification result R2 are true, the action of taking the medicine is considered to be completed; otherwise, the action of taking the medicine is considered to be incomplete.

[0082] S34. Display the result information of whether the medication was taken correctly through text or through voice broadcast.

[0083] In this embodiment, the result information of whether the medication was taken correctly can be passed to the interactive display module, and the "medication taken is complete" information can be displayed on the app client or prompted by voice broadcast.

[0084] This embodiment employs a deep learning approach to provide a method for detecting medication use in the elderly based on human keypoint detection and Transformer technology. It utilizes human keypoint detection to determine the medication-taking action, uses facial keypoints (especially the mouth) to determine if the elderly person has swallowed the medication correctly, and uses the YOLO classification algorithm to determine the type and quantity of medication to determine if the elderly person has taken the medication correctly. By employing the Transformer framework in temporal modeling, it can effectively utilize the temporal information implied in the actions generated by human keypoint modeling to distinguish similar actions such as taking medication and drinking water, thereby solving the problem of elderly people being unable to take medication on time and in the correct dosage due to forgetfulness.

[0085] Compared with existing technologies, this invention does not require additional equipment; only two cameras are needed to collect information. These cameras can also be used for monitoring, making them reusable. It can effectively model the temporal information contained in actions, fully mining useful information. The temporal action features constructed from human key points, after being processed by a transformer model, can distinguish similar behaviors such as drinking water and taking medicine, avoiding misjudgments. The key point criteria can be replaced with other actions to complete the detection of all physical behaviors of the elderly, demonstrating excellent scalability.

[0086] Example 2:

[0087] This embodiment provides a medication detection system for the elderly based on human keypoint detection and the Transformer model. It includes a video acquisition module, a feature extraction and detection module, a logical judgment module, and an interactive display module. The specific functions of each module are as follows:

[0088] The video acquisition module is used to acquire overall spatial data and facial region data of the human body through video equipment to obtain a video set.

[0089] The feature extraction and detection module is used to detect and extract human keypoint sets, mouth keypoint sets from facial keypoints, and drug information sets from video sets using different neural network models. The human keypoint set is used as human keypoint features, the mouth keypoint set is used as facial keypoint features, and the drug information set is used as drug features.

[0090] The logic judgment module is used to make logical judgments based on the relative positions of key points in the human body key point features, the relative positions of key points in the facial key point features, and the type and quantity of drugs in the drug features, to obtain a first result R1 indicating whether the correct medication has been taken. The human body key point features are modeled frame by frame using the Transformer temporal model to obtain a feature set containing global action information for each frame of the video. This feature set is then input into an action classifier for classification to obtain a second result R2 indicating whether the action is medication taking. Based on the first result R1 and the second result R2, the module judges and outputs the result indicating whether the correct medication has been taken.

[0091] The interactive display module shows the result information of whether the medication was taken correctly through text or voice broadcast.

[0092] Preferably, the step of detecting and extracting the human body key point set, the mouth key point set from the facial key points, and the drug information set from the video set using different neural network models includes: copying the video set three times to obtain three identical video sets. Video collection Video collection Based on video collection A set of human key points was obtained by detecting and extracting them using a neural network model trained with the open-source PP-TinyPose algorithm; and based on the video set... A set of mouth keypoints was obtained from facial keypoints using a neural network model trained with the open-source Face Landmark algorithm; based on the video set... A neural network model trained using the open-source YOLOv7 algorithm was used to detect and extract a set of drug information consisting of the quantity and type of drugs.

[0093] The method for detecting medication use in the elderly based on human key point detection and the Transformer model is characterized by the following step: Logical judgment is made based on the relative positions of key points in the human key point features, the relative positions of key points in the facial key point features, and the type and quantity of medication in the medication features to obtain a first result R1 indicating whether correct medication has been taken. This includes:

[0094] When the acute angle between the line connecting the elbow key point and the corresponding shoulder on the side of the body and the body's midline is greater than 30 degrees, and the wrist key point is higher than the shoulder and lower than the nose, the arm is considered raised, and the result is set to R. 11 It is true.

[0095] When the area enclosed by the mouth contour points first increases and then decreases, it is considered as a drug-taking action, and the result R is set. 12 It is true.

[0096] When the type and quantity of the drug in the drug characteristics match the preset data in the database, it is considered that the medication has been taken correctly, and the result R is set. 13 If true, otherwise it is considered inaccurate or incorrect medication; set the result R. 13 No;

[0097] The result R 11 Result R 12 and result R 13 Perform a logical judgment, when the result R 11 Result R 12 and result R 13 If all values ​​are true, then the result R1 is true, which is considered as completing the correct medication.

[0098] The Transformer temporal model includes a Transformer submodule, which models the obtained human keypoint features frame by frame using a self-attention mechanism.

[0099]

[0100]

[0101]

[0102]

[0103] Where Q, K, and V represent the Query matrix, Key matrix, and Value matrix in the attention mechanism, respectively; W q W k W v These are all vectors used to abstract features to form the Query matrix, Key matrix, and Value matrix mentioned above. Each element in J1 represents a key human feature. The feature is the global motion information for each frame of the video, where * denotes matrix multiplication;

[0104]

[0105] Where, x i For matrix operations Q*K T Each element in the resulting matrix, where N is the total number of elements in the resulting matrix. With the natural logarithm e as the base, x i This refers to the power operation of the exponent.

[0106] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for detecting medication use in the elderly based on human key point detection and Transformer model, characterized in that, Includes the following steps: The overall spatial data and facial region data of the human body are collected separately using video equipment to obtain a video set; The video set is processed using different neural network models to extract human body keypoint sets, mouth keypoint sets, and drug information sets. The human body keypoint sets are used as human body keypoint features, the mouth keypoint sets as facial keypoint features, and the drug information sets as drug features. The process includes the following steps: copying the video set three times to obtain three identical video sets. Video collection Video collection According to the video collection The set of human body key points is obtained by detecting and extracting them using a neural network model trained with the open-source PP-TinyPose algorithm; based on the video set... A set of mouth keypoints was obtained from facial keypoints using a neural network model trained with the open-source Face_Landmark algorithm; based on the video set... The neural network model trained by the open-source YOLOv7 algorithm is used to detect and extract a set of drug information consisting of the quantity and type of drugs. Based on the relative positions of key points in the human body's key features, the relative positions of key points in the facial key features, and the type and quantity of drugs in the drug features, a logical judgment is made to obtain the first result of whether the correct medication was taken. , This includes: when the acute angle between the line connecting the elbow key point and the corresponding shoulder on the side of the body and the body's midline is greater than 30 degrees, and the wrist key point is higher than the shoulder and lower than the nose, the arm is considered raised. (Setting the result) True; When the area enclosed by the mouth contour points first increases and then decreases, it is considered as a drug-taking action, and the result is set. True; When the type and quantity of the drug in the drug characteristics match the preset data in the database, it is considered that the medication has been taken correctly, and the result is set. If true, otherwise it is considered inaccurate or incorrect medication; set the result. No; The result ,result and results Perform logical judgments, when the result ,result and results If all are true, then the result is... If true, it is considered that the medication has been taken correctly; Human body key points are analyzed using the Transformer time series model. Frame-by-frame modeling is performed to obtain a feature set containing global action information for each frame of the video. This feature set is then input into an action classifier for classification to obtain a second result indicating whether the action represents medication administration. ; According to the first result Second result Determine and output the result of whether the medication was taken correctly.

2. The method for detecting medication use in the elderly based on human key point detection and Transformer model according to claim 1, characterized in that, The video device consists of two RGB-D cameras.

3. The method for detecting medication use in the elderly based on human key point detection and Transformer model according to claim 1, characterized in that, The Transformer temporal model includes a Transformer submodule, which models the obtained human keypoint features frame by frame using a self-attention mechanism. ； ； ； ； Where Q, K, and V represent the Query matrix, Key matrix, and Value matrix in the attention mechanism, respectively; These are all vectors used to abstract features to form the Query matrix, Key matrix, and Value matrix mentioned above. Represents key features of the human body Each element in The feature is the global motion information for each frame of the video, where * denotes matrix multiplication; ； in, For matrix operations Each element in the resulting matrix, where N is the total number of elements in the resulting matrix. It is based on the natural logarithm e. This refers to the power operation of the exponent.

4. The method for detecting medication use in the elderly based on human key point detection and Transformer model according to claim 3, characterized in that, The Transformer temporal model includes at least two Transformer sub-modules. The keypoint feature set generated by the first Transformer sub-module, which contains global motion information for each frame of the video, is input into the next Transformer sub-module to continue modeling. This process is repeated until the last Transformer sub-module completes its modeling, and finally outputs a keypoint feature set containing global motion information for each frame of the video.

5. The method for detecting medication use in the elderly based on human key point detection and Transformer model according to claim 3, characterized in that, The action classifier includes a score regressor and a classifier, each implemented through two fully connected layers. The score regressor calculates the score for each possible action in each frame of the video and assigns a score to each action. Normalization, the classifier is used to classify actions, and the action with the highest score is the classification result. .

6. The method for detecting medication use in the elderly based on human key point detection and Transformer model according to claim 1, characterized in that, The process includes displaying the results of whether the medication was taken correctly via text or voice announcement.

7. A medication detection system for the elderly based on human key point detection and Transformer model, characterized in that, The system includes: The video acquisition module is used to acquire overall spatial data and facial region data of the human body through video equipment to obtain a video set; The feature extraction and detection module is used to detect and extract human key point sets, mouth key point sets in facial key points and drug information sets from video sets through different neural network models. The human key point set is used as human key point features, the mouth key point set is used as facial key point features, and the drug information set is used as drug features. The process of extracting human body key point sets, mouth key point sets, and drug information sets from the video set using different neural network models includes the following steps: Duplicate the video set three times to get three identical video sets. Video collection Video collection According to the video collection The set of human body key points is obtained by detecting and extracting them using a neural network model trained with the open-source PP-TinyPose algorithm; based on the video set... A set of mouth keypoints was obtained from facial keypoints using a neural network model trained with the open-source Face_Landmark algorithm; based on the video set... The neural network model trained by the open-source YOLOv7 algorithm is used to detect and extract a set of drug information consisting of the quantity and type of drugs. The logic judgment module is used to make logical judgments based on the relative positions of key points in the human body key point features, the relative positions of key points in the facial key point features, and the type and quantity of drugs in the drug features, to obtain the first result of whether the correct medication has been taken. ,include: When the acute angle between the line connecting the elbow key point and the corresponding shoulder on the side of the body and the body's midline is greater than 30 degrees, and the wrist key point is higher than the shoulder and lower than the nose, the arm is considered raised, and the result is set. True; When the area enclosed by the mouth contour points first increases and then decreases, it is considered as a drug-taking action, and the result is set. True; When the type and quantity of the drug in the drug characteristics match the preset data in the database, it is considered that the medication has been taken correctly, and the result is set. If true, otherwise it is considered inaccurate or incorrect medication; set the result. No; The result ,result and results Perform logical judgments, when the result ,result and results If all are true, then the result is... If true, it is considered that the medication has been taken correctly; The Transformer temporal model is used to model human keypoint features frame by frame, resulting in a feature set containing global action information for each frame of the video. This feature set is then input into an action classifier for classification, yielding a second result indicating whether the action represents medication administration. According to the first result Second result Determine and output the result of whether the medication was taken correctly; The interactive display module shows the result of whether the medication was taken correctly through text or voice announcement.

8. The elderly medication detection system based on human key point detection and Transformer model according to claim 7, characterized in that, The Transformer temporal model includes a Transformer submodule, which models the obtained human keypoint features frame by frame using a self-attention mechanism. ； ； ； ； Where Q, K, and V represent the Query matrix, Key matrix, and Value matrix in the attention mechanism, respectively; These are all vectors used to abstract features to form the Query matrix, Key matrix, and Value matrix mentioned above. Represents key features of the human body Each element in The feature is the global motion information for each frame of the video, where * denotes matrix multiplication; ； in, For matrix operations Each element in the resulting matrix, where N is the total number of elements in the resulting matrix. It is based on the natural logarithm e. This refers to the power operation of the exponent.

Images