Medicine taking auditing and liver injury early warning system and method based on privacy protection

By introducing a privacy-protected drug review and liver injury warning system in the management model of tuberculosis patients, using facial privacy protection algorithms and intelligent audit models, the problems of irregular medication and privacy leakage are solved, management efficiency and security are improved, and management is ensured. Patient health management while protecting privacy.

CN118969319APending Publication Date: 2024-11-15SHENZHEN NANSHAN DISTRICT CHRONIC DISEASE CONTROL CENT (SHENZHEN NANSHAN DISTRICT MENTAL HEALTH CENT) +1
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
CN202411001709.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

In the existing management model of tuberculosis patients, irregular medication and untimely monitoring of adverse drug reactions, and the remote video supervision medication model has a risk of human resources shortage and privacy leakage, affecting the cure rate and privacy protection.

Method used

A drug review and liver injury warning system based on privacy protection was designed, including video upload module, facial privacy protection module, intelligent drug review module and drug-induced liver injury warning module, through facial privacy protection algorithm, intelligent audit model and image processing Technology to ensure patient privacy and automatically identify medication actions and signs of liver damage.

Benefits of technology

It improves work efficiency, reduces human resource consumption, ensures the security of patient information, improves the accuracy of medication review and liver injury warning, and achieves a balance between health management and privacy protection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118969319A_ABST
    Figure CN118969319A_ABST
Patent Text Reader

Abstract

The invention discloses a medicine taking auditing and liver injury early warning system and method based on privacy protection, and relates to the technical field of disease management, and the method comprises the steps that a video uploading module receives a video shot by a patient; the face privacy protection module covers the face in the video through a face privacy protection algorithm; the intelligent medicine taking auditing module analyzes the video after the face coverage through an intelligent auditing model, and automatically identifies and audits the medicine taking action and the medicine information; and the drug-induced liver injury early warning module analyzes the difference change of the sclera color of the patient in the video by using an image processing technology and a machine learning algorithm, and carries out early warning on the drug-induced liver injury. An encryption technology and privacy protection measures are adopted, and the safety of patient information is ensured. And the accuracy of medicine taking auditing and liver injury early warning is improved by using a deep learning technology. Medicine taking videos can be uploaded anytime and anywhere, and medical staff can timely acquire and process information. Potential health risks are found in time, and better health management is provided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of disease management, and more specifically to a medication review and liver injury early warning system and method based on privacy protection. Background Art

[0002] Tuberculosis, as a global health challenge, has not only attracted much attention due to its chronic and infectious characteristics, but its global spread has also constituted a major public health issue. In order to meet this challenge, a public health service model with DOTS (Directly Observed Treatment, Short-course) as the core has been adopted. This model mainly relies on patients to go to community health centers or professional institutions for treatment, supplemented by door-to-door services and telephone follow-up by medical staff. Although this model has achieved certain results in the early stage, its limitations have gradually emerged over time, such as the low actual implementation rate of DOTS, irregular medication of patients, and untimely monitoring of adverse drug reactions, which have affected the cure rate and may lead to the development of drug resistance.

[0003] To solve these problems, remote video-observed therapy (VOT) has gradually been introduced as a new model for the management of tuberculosis patients. VOT uses video technology to supervise patients' medication, which reduces the workload of medical staff and improves management efficiency. However, the shortage of human resources and the untimely review of medication videos are still key factors affecting the effectiveness of VOT.

[0004] The rapid development of artificial intelligence technology has provided new possibilities for the improvement of the VOT model. Using AI technology for intelligent video review can achieve more efficient and accurate medication management. In addition, AI technology has also shown great potential in identifying anti-tuberculosis drug-induced liver injury (ATLI). By analyzing the patient's facial images, AI can help detect signs of drug-induced liver injury in a timely manner, thereby providing decision support for clinicians and reducing the risk of severe ATLI. Similarly, when implementing the VOT management model, the issue of patient personal privacy protection cannot be ignored. The collection and storage of sensitive information such as the patient's facial images and shooting environment may lead to the risk of privacy leakage.

[0005] Therefore, how to propose a medication review and liver injury early warning system and method based on privacy protection, protect personal privacy while ensuring patient health management, and realize a solution that can be used for health management and protect patient privacy, and meet the long-term sustainable development of the VOT management model is an urgent problem that technical personnel in this field need to solve. Summary of the invention

[0006] In view of this, the present invention provides a privacy-protected medication review and liver injury early warning system and method, which protects personal privacy while ensuring patient health management, realizes a solution that can be used for health management and protect patient privacy, and meets the long-term sustainable development of the VOT management model. In order to achieve the above purpose, the present invention adopts the following technical solutions:

[0007] A medication review and liver injury early warning system based on privacy protection, comprising: a video upload module, a facial privacy protection module, an intelligent medication review module and a drug-induced liver injury early warning module connected in sequence;

[0008] The video uploading module is used to receive the video taken by the patient;

[0009] The facial privacy protection module is used to cover the face of the person in the video with a mask through a facial privacy protection algorithm;

[0010] The intelligent medication review module is used to analyze the video of the face covered with the mask through the intelligent review model, and automatically identify and review the medication action and medication information;

[0011] The drug-induced liver injury early warning module is used to analyze the difference in sclera color of patients in the video using image processing technology and machine learning algorithms to issue early warnings for drug-induced liver injury.

[0012] Optionally, it also includes: a user identity authentication module, which is used to authenticate the video received by the video upload module, identify and match the patient's facial information through an integrated face recognition algorithm, and ensure that the video is uploaded by the patient himself.

[0013] Optionally, it also includes: a video encryption and storage module, which is used to encrypt the video using the SM4 encryption algorithm after the video verification and upload is successful; the video data stored in the video encryption and storage module is automatically deleted after being stored for half a year.

[0014] Optionally, analyzing the video with the face covered by the mask by using the intelligent audit model includes:

[0015] Construct a drug detection model based on the target detection network yolov8, and obtain drug detection results through the drug detection model;

[0016] A medication action recognition model is constructed through a convolutional neural network, and the medication action recognition result is obtained through the medication action recognition model;

[0017] The drug detection results and the medication action recognition results are coupled and judged. When the drug is detected and the complete medication action is recognized, the patient's medication video will be reviewed and confirmed as effective medication.

[0018] Optionally, the constructing a medication action recognition model by using a convolutional neural network, and obtaining a medication action recognition result by using the medication action recognition model includes:

[0019] Video frame sampling module: used to use sparse sampling method to evenly group videos and randomly collect one frame in each group to form a video frame sequence;

[0020] Image feature extraction module: used to process the obtained video frame sequence through the image feature extraction module to extract image features;

[0021] Skeleton feature extraction module: used to pass the video frame sequence through the human key point detection module to obtain the human skeleton features;

[0022] Feature fusion module: used to fuse image features and skeleton features through the feature fusion module to form features for classification;

[0023] Classification module: used to process the fused features through the classification head to obtain the results of medication action recognition.

[0024] Optionally, the feature fusion module includes: image and skeleton features F pose After a combination of a fully connected layer and a nonlinear activation, the features F' of the same dimension are obtained. image and F' pose :

[0025] F image =σ(F image w image +b image );

[0026] F pose =σ(F pose w pose +b pose );

[0027] Where σ is a nonlinear activation function, w pose and w image are the weights of the linear layer, b pose and b image are the biases of the linear layer, respectively;

[0028] Then F' image and F' pose After weighted addition, the fusion feature F is obtained fuse ;

[0029] F fuse =α pose F' pose +α image F' image ;

[0030] α pose and α image are the weight coefficients of weighted fusion, where

[0031] α image =softmax(F' pose w+b);

[0032] α pose =softmax(F' image w+b);

[0033] Among them, w is the weight of the linear layer and b is the bias of the linear layer.

[0034] Optionally, the skeleton feature extraction module includes:

[0035] Extract 2D keypoint human skeleton sequences from video frames and reformulate them into 3D heatmaps before feeding them into the skeleton feature extraction module;

[0036] According to the key point coordinates, a Gaussian kernel is used to generate a K×H×W heat map for each frame;

[0037] Stack T frames to get the 3D heat map of the entire sequence, with dimensions of K×T×H×W, where K is the number of skeleton points, T is the number of frames, and H and W are the height and width respectively;

[0038] The 3D heat map is passed through the skeleton feature extraction module to obtain the skeleton features.

[0039] Optionally, the early warning of drug-induced liver injury includes:

[0040] Color calibration module: used to calibrate the image using the dynamic threshold white balance method;

[0041] Sclera ROI extraction module: used to extract the sclera ROI of the color-corrected image and obtain the facial key points using the face landmarks detection in the mediapipe;

[0042] Early warning module: used to compare the video uploaded by the patient with the last uploaded video searched in the database, and issue an early warning based on the comparison results.

[0043] Optionally, the early warning module includes:

[0044] For videos uploaded by patients, the last uploaded video is searched in the database;

[0045] After color calibration, frame sampling and sclera ROI extraction, there are two paths. The first is to obtain the difference image, which is then extracted through a convolutional neural network to obtain image difference features. The second is to obtain the differences in the three color spaces of RGB, HSV, and Lab. After normalization, they are fused with the above image difference features as features, and then the warning results are obtained through the classification head. A threshold is set. If the network output score is greater than the set threshold, it means that the sclera color has changed significantly, and feedback is given.

[0046] Optionally, a privacy-protected medication review and liver injury early warning method includes:

[0047] Receive videos taken by patients;

[0048] Use facial privacy protection algorithms to cover faces in the video with masks;

[0049] The intelligent review model analyzes the video of the face covered with a mask, automatically identifying and reviewing the medication action and drug information;

[0050] Image processing technology and machine learning algorithms are used to analyze the differences in the color of the patients' sclera in the video and to provide early warning for drug-induced liver injury.

[0051] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a medication review and liver injury early warning system and method based on privacy protection, which has the following beneficial effects:

[0052] The present invention proposes a medication review and liver injury early warning system based on privacy protection, comprising: a video upload module, a facial privacy protection module, an intelligent medication review module and a drug-induced liver injury early warning module connected in sequence; the video upload module is used to receive videos shot by patients; the facial privacy protection module is used to cover the face in the video with a mask through a facial privacy protection algorithm; the intelligent medication review module is used to analyze the video after the face is covered with a mask through an intelligent review model, automatically identify and review medication actions and drug information; the drug-induced liver injury early warning module is used to analyze the difference in sclera color of patients in the video using image processing technology and machine learning algorithms, and warn of drug-induced liver injury. The present invention is based on the VOT remote management system, and adds intelligent medication review, privacy protection, and drug-induced liver injury early warning. Through the automated review process, work efficiency is greatly improved and human resource consumption is reduced. Advanced encryption technology and privacy protection measures are used to ensure the security of patient information. Deep learning technology is used to improve the accuracy of medication review and liver injury early warning. Patients can upload medication videos anytime and anywhere, and medical personnel can also obtain and process information in a timely manner. The system can detect potential health risks in a timely manner and provide patients with better health management. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0054] Figure 1 This is a structural framework diagram of a medication review and liver injury early warning system based on privacy protection provided by the present invention.

[0055] Figure 2 This is a schematic diagram of the structure of the drug-induced liver injury early warning module provided by the present invention.

[0056] Figure 3 This is a schematic diagram of the structure of the intelligent medication review module provided by the present invention.

[0057] Figure 4 This is a schematic diagram of the facial privacy protection module structure provided by the present invention. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The embodiment of the present invention discloses a privacy-protection-based medication review and liver injury early warning system, comprising: a video upload module, a facial privacy protection module, an intelligent medication review module, and a drug-induced liver injury early warning module connected in sequence;

[0060] The video uploading module is used to receive the video taken by the patient;

[0061] The facial privacy protection module is used to cover the face of the person in the video with a mask through a facial privacy protection algorithm;

[0062] The intelligent medication review module is used to analyze the video of the face covered with the mask through the intelligent review model, and automatically identify and review the medication action and medication information;

[0063] The drug-induced liver injury early warning module is used to analyze the difference in sclera color of patients in the video using image processing technology and machine learning algorithms to issue early warnings for drug-induced liver injury.

[0064] Furthermore, it also includes: a user identity authentication module, which is used to authenticate the video received by the video upload module, and identify and match the patient's face information through an integrated face recognition algorithm to ensure that the video is uploaded by the patient himself. This not only improves the security of the system, but also avoids the risk of others impersonating the patient to upload inappropriate content.

[0065] Furthermore, it also includes: a video encryption and storage module, which is used to encrypt the video using the SM4 encryption algorithm after the video verification and upload is successful; the video is encrypted using the SM4 encryption algorithm to ensure the security of the video data during transmission and storage. The SM4 algorithm is a symmetric encryption algorithm independently developed by China, which has high security and efficiency. In addition, the system sets a mechanism for automatic deletion of video data after being stored for half a year, which not only complies with relevant regulations on medical data protection, but also reduces the security risks that may be caused by long-term storage of data.

[0066] Furthermore, the facial privacy protection module is used to cover the face of the person in the video with a mask using a facial privacy protection algorithm, such as Figure 4 As shown, including:

[0067] In order to protect the facial privacy of patients, the system automatically applies a facial privacy protection algorithm when storing and processing videos, and covers the faces in the video with masks. This processing method protects the privacy of patients without affecting the effectiveness of video review. The system strictly limits the access rights to the original video, and only medical personnel or managers with the highest authority can access the original video when necessary.

[0068] The specific methods of wearing a mask are as follows:

[0069] 1) Key point detection: detecting key points on the face in the video and the predetermined mask image. The key points are the basis for subsequent facial coverage.

[0070] For facial key points in the video: FaceMesh algorithm in mediapipe is used to detect facial key points. FaceMesh algorithm detects 468 facial key points, and 75 of them are selected as the final detection targets. Since the medication video may contain rotation and occlusion of the character, the facial key point detection may not be accurate enough and is prone to omissions or errors. In order to solve this problem, a human posture detection algorithm is introduced to assist the detection of facial key points by using the relative position information of other parts of the human body, thereby reducing omissions and errors. The human posture estimation adopts the DWPose method. The facial key points obtained by human posture detection may not completely match the set standard facial key points. Therefore, these key points need to be linearly transformed to match the predetermined facial key points.

[0071] For mask key points: Because the mask is fixed, the 75 selected key points are manually annotated to obtain the mask key points.

[0072] 2) Delaunay triangulation: The Delaunay triangulation algorithm is used to connect the key points of the face in the mask and the video frame in the same way. This algorithm can ensure the optimal connection method, so that the mask and the face fit more naturally.

[0073] 3) Triangle affine transformation: Use triangle affine transformation technology to map the mask triangle to the face triangle in the video. Adjust the size, shape and position of the mask so that it accurately covers the face.

[0074] Furthermore, the analysis of the video with the face covered by the mask by the intelligent audit model includes:

[0075] Construct a drug detection model based on the target detection network yolov8, and obtain drug detection results through the drug detection model;

[0076] A medication action recognition model is constructed through a convolutional neural network, and the medication action recognition result is obtained through the medication action recognition model;

[0077] The drug detection results and the medication action recognition results are coupled and judged. When the drug is detected and the complete medication action is recognized, the patient's medication video will be reviewed and confirmed as effective medication.

[0078] In a specific embodiment, analyzing the video of the face covered with a mask by the intelligent audit model specifically includes:

[0079] The system uses an intelligent audit model trained by deep learning technology to analyze videos of patients taking medication while wearing masks, and automatically identifies the correctness of medication actions and information such as the type and quantity of medication. Intelligent auditing can not only improve auditing efficiency, but also reduce human errors, ensuring that patients take medication correctly and safely.

[0080] The smart audit method is as follows:

[0081] 1) Drug testing:

[0082] For the video of patients taking medicine, the system will detect the medicine frame by frame to ensure that no possible part of the medicine display is missed. The drug detection is achieved through a dedicated drug detection module, which is built based on the object detection network yolov8.

[0083] 2) Medication Action Recognition

[0084] 2.1) Video frame sampling: Since the lengths of medication videos taken by patients vary, the system uses a sparse sampling method to evenly divide the videos into 32 groups and randomly sample one frame in each group as a representative, forming a 32-frame sequence as the input for subsequent analysis.

[0085] 2.2) Image feature extraction: The sampled 32-frame sequence is processed by the image feature extraction module, which is built based on a 3D convolutional neural network to extract image features and obtain features with a dimension of 2048 after average pooling.

[0086] 2.3) Skeleton feature extraction: The video frame sequence also passes through the human key point detection module, which uses hrnet and coco 17 key point models to obtain the human skeleton sequence. The 2D key point human skeleton sequence is extracted from the video frame and needs to be re-expressed as a 3D heat map before inputting it into the skeleton feature extraction module. According to the key point coordinates, a Gaussian kernel is used to generate a K×H×W heat map for each frame, which is calculated as follows:

[0087] J kij =exp(-[(ix k ) 2+(jy k ) 2 ] / (2*σ 2 ))*c k ;

[0088] Where (x k ,y k ) and c k are the kth key point coordinates and confidence respectively, and σ is the variance of the control Gaussian graph.

[0089] Then, T frames are stacked to obtain a 3D heat map of the entire sequence with dimensions of K×T×H×W, where K is the number of skeleton points, T is the number of frames, H and W are the height and width respectively. In this method, K is 17, T is 32, H and W are 56, and the 3D heat map dimensions are 17×32×56×56.

[0090] The 3D heat map is passed through the skeleton feature extraction module to obtain skeleton features. The skeleton feature extraction module is similar to the image feature extraction module and is also a 3D convolutional neural network, which obtains skeleton features with a dimension of 512.

[0091] 2.4) Feature fusion: Image features and skeleton features are fused through the feature fusion module to form features for classification.

[0092] The feature fusion module is calculated as follows: first, the image feature F image and skeleton features F pose After a combination of a fully connected layer and a nonlinear activation, the features F'image and F'pose with dimensions of 256 are obtained:

[0093] F' image =σ(F image w image +b image );

[0094] F' pose =σ(F pose w pose +b pose );

[0095] Where σ is a nonlinear activation function, w pose and w image are the weights of the linear layer, b pose and b image are the biases of the linear layer respectively. Then F' image and F' pose After weighted addition, the fusion feature F is obtained fuse .

[0096] F fuse =α pose F' pose+α image F' image ;

[0097] α pose and α image are the weight coefficients of weighted fusion, α pose and α image The calculation is as follows,

[0098] α image =softmax(F' pose w+b);

[0099] α pose =softmax(F' image w+b); where w is the weight of the linear layer and b is the bias of the linear layer. The weight coefficient here is calculated based on the other party's features and can encode the relationship between different modes.

[0100] 2.5) Classification: The fused features are processed by a classification head, such as a multi-layer perceptron, to obtain the result of medication action recognition, that is, a two-dimensional vector representing probability, where the category subscript with a larger probability represents the recognition result.

[0101] 3) Intelligent medication review module, such as Figure 3 As shown, the review result combination system combines the drug detection results with the medication action recognition results. Only when both conditions are met, that is, the drug is detected and the complete medication action is identified, the patient's medication video will be reviewed and confirmed as effective medication.

[0102] In a specific implementation, a privacy-protected medication review and liver injury early warning method includes:

[0103] Receive videos taken by patients;

[0104] Use facial privacy protection algorithms to cover faces in the video with masks;

[0105] The intelligent review model analyzes the video of the face covered with a mask, automatically identifying and reviewing the medication action and drug information;

[0106] Image processing technology and machine learning algorithms are used to analyze the differences in the color of the patients' sclera in the video and to provide early warning for drug-induced liver injury.

[0107] In a specific embodiment, the early warning of drug-induced liver injury specifically includes: using image processing technology and machine learning algorithms to analyze the difference in the color of the patient's sclera to warn of drug-induced liver injury. The change in sclera color is closely related to the state of liver function. The system can detect abnormal conditions in time and notify medical personnel, so as to take intervention measures in advance to prevent or reduce the occurrence of drug-induced liver injury.

[0108] The early warning method for drug-induced liver injury is as follows:

[0109] (1) Color calibration module. The early warning of drug-induced liver injury is performed based on the change in the scleral color. Therefore, color calibration processing needs to be performed on the image to reduce the influence of the illumination environment during video shooting. The dynamic threshold white balance method is used for color calibration processing, and the calculation steps are as follows:

[0110] (1) White point detection

[0111] 1.1) Convert the original image with size w×h from the RGB color space to the YCrCb color space.

[0112] 1.2) Calculate the average values Mr and Mb of the Cr and Cb components.

[0113] 1.3) Calculate the variances Dr and Db of the Cr and Cb components according to Mr and Mb.

[0114] Dr = ∑ i,j (|Cr(i,j) - Mr|) / N;

[0115] Db = ∑ i,j (|Cb(i,j) - Mb|) / N;

[0116] 1.4) Determine the near-white area. The discrimination criterion is:

[0117] Cb(i,j) - (Mb + Db×sign(Mb) < 1.5×Db;

[0118] Cr(i,j) - (Mr + Dr×sign(Mr) < 1.5×Dr;

[0119] where sign is the sign function, that is, it returns 1 for positive numbers and 0 for negative numbers.

[0120] 1.5) Set a luminance matrix RL of a "reference white point" with size w×h.

[0121] 1.6) If it meets the discriminant formula, it is used as the "reference white point", and the luminance (Y component) of this point (i,j) is assigned to RL(i,j). If it does not meet, the value of RL(i,j) at this point is 0.

[0122] (2) White point adjustment

[0123] 2.1) Select the 10% of the largest luminance (Y component) values in the "reference white point" and select the minimum value Lu_min among them.

[0124] 2.2) Adjust RL. If RL(i,j) < Lu_min, RL(i,j) = 0; otherwise, RL(i,j) = 1.

[0125] 2.3) Multiply R, G, B by RL to get R2, G2, B2. Calculate the average values ​​of R2, G2, B2, Rav, Gav, Bav respectively.

[0126] 2.4) Get the adjustment gain: Define Ymax = double(max(max(Y))), then

[0127] Rgain = Ymax / Rav;

[0128] Ggain = Ymax / Gav;

[0129] Bgain=Ymax / Bav.

[0130] Adjust the original image: Ro = R*Rgain; Go = G*Ggain; Bo = B*Bgain.

[0131] (ii) Sclera ROI extraction module. The color-corrected image needs to be subjected to sclera ROI extraction before being input into the network. The extraction method adopts a method based on facelandmarks detection. The facelandmarks detection in mediapipe is used to obtain 478 key points of the face, and the iris key points and inner eyelid key points are separated. The drawContours function in opencv is used to obtain the eye contour and iris contour respectively using the inner eyelid key points and iris key points. The area difference between the two contours is calculated, that is, the part of the eye contour minus the iris contour. This part is the sclera area, and the sclera ROI is obtained.

[0132] (III) Drug-induced liver injury early warning module, such as Figure 2 As shown in the figure, for the video uploaded by the patient, the last uploaded video is searched in the database. After color correction, frame sampling and sclera ROI extraction, there are two paths. First, the sclera ROI extracted from the current video sampling frame is subtracted from the sclera ROI extracted from the last uploaded video sampling frame to obtain a difference image, and the convolutional neural network is used to extract features to obtain image difference features; second, the difference in the three color spaces of RGB, HSV, and Lab is obtained, and after normalization, it is fused with the above image difference features as a feature, and then the warning result is obtained through the classification head. The threshold is set to 0.8. If the network output score is greater than 0.8, it means that the sclera color has changed greatly, and feedback is given to the doctor to pay attention to drug-induced liver injury. If the patient is using the VOT system for the first time, this module calculation is not performed.

[0133] In a specific implementation, a privacy-protected medication video intelligent review and drug-induced liver injury early warning system, such as Figure 1As shown, it includes: 1) When the patient uploads the video, a face recognition algorithm is added to determine whether it is uploaded by the patient himself; 2) The SM4 encryption algorithm is used to store the video uploaded by the patient, and the video data is automatically cleared after half a year; 3) The face privacy protection algorithm covers the face in the video with a mask, and only the highest authority can access the original video; 4) Intelligent medication review based on the masked medication video; 5) Drug-induced liver injury warning based on the difference in sclera color. The system operation process is as follows: When the patient uploads the video to VOT, the system will confirm the user's identity to ensure that it is the patient's own medication video. If the verification fails, it will be fed back to the patient client. The uploaded video will be stored using the SM4 encryption algorithm, and the video will be protected for facial privacy, and a mask will be put on the face in the video. Then, based on the masked video, intelligent medication review and drug-induced liver injury warning will be performed, and abnormal processing results will be fed back to the doctor, who will then confirm and feed back to the patient.

[0134] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0135] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A privacy-protected medication review and liver injury early warning system, characterized in that: include: The video upload module, facial privacy protection module, intelligent medication review module and drug-induced liver injury early warning module are connected in sequence; The video uploading module is used to receive the video taken by the patient; The facial privacy protection module is used to cover the face of the person in the video with a mask through a facial privacy protection algorithm; The intelligent medication review module is used to analyze the video of the face covered with the mask through the intelligent review model, and automatically identify and review the medication action and medication information; The drug-induced liver injury early warning module is used to analyze the difference in sclera color of patients in the video using image processing technology and machine learning algorithms to issue early warnings for drug-induced liver injury.

2. According to claim 1, a privacy-protected medication review and liver injury early warning system is characterized in that: Also includes: The user identity authentication module is used to authenticate the video received by the video upload module, and identify and match the patient's facial information through an integrated face recognition algorithm to ensure that the video is uploaded by the patient himself.

3. A privacy-protected medication review and liver injury early warning system according to claim 2, characterized in that: Also includes: A video encryption and storage module, wherein the video encryption and storage module is used to encrypt the video using the SM4 encryption algorithm after the video verification upload is successful; The video data stored in the video encryption and storage module will be automatically deleted after six months.

4. According to the privacy protection-based medication review and liver injury early warning system of claim 1, it is characterized in that: The analysis of the video with the face covered by the mask by the intelligent audit model includes: Construct a drug detection model based on the target detection network yolov8, and obtain drug detection results through the drug detection model; A medication action recognition model is constructed through a convolutional neural network, and the medication action recognition result is obtained through the medication action recognition model; The drug detection results and the medication action recognition results are coupled and judged. When the drug is detected and the complete medication action is recognized, the patient's medication video will be reviewed and confirmed as effective medication.

5. A privacy-protected medication review and liver injury early warning system according to claim 4, characterized in that: The method of constructing a medication action recognition model through a convolutional neural network and obtaining a medication action recognition result through the medication action recognition model includes: Video frame sampling module: used to use sparse sampling method to evenly group videos and randomly collect one frame in each group to form a video frame sequence; Image feature extraction module: used to process the obtained video frame sequence through the image feature extraction module to extract image features; Skeleton feature extraction module: used to pass the video frame sequence through the human key point detection module to obtain the human skeleton features; Feature fusion module: used to fuse image features and skeleton features through the feature fusion module to form features for classification; Classification module: used to process the fused features through the classification head to obtain the results of medication action recognition.

6. A privacy-protected medication review and liver injury early warning system according to claim 5, characterized in that: The feature fusion module includes: image and skeleton features F pose After a combination of a fully connected layer and a nonlinear activation, the features F' of the same dimension are obtained. image and F′ pose : F′ image =σ(F image w image +b image ); F' pose =σ(F pose w pose +b pose ); Where σ is a nonlinear activation function, w pose and w image are the weights of the linear layer, b pose and b image are the biases of the linear layer, respectively; Then F' image and F' pose After weighted addition, the fusion feature F is obtained fuse ; F fuse =a pose F′ pose +a image F′ image ; α pose and α image are the weight coefficients of weighted fusion, where a image =softmax(F' pose w+b); a pose =softmax(F′ image w+b); Among them, w is the weight of the linear layer and b is the bias of the linear layer.

7. A privacy-protected medication review and liver injury early warning system according to claim 5, characterized in that: The skeleton feature extraction module comprises: Extract 2D keypoint human skeleton sequences from video frames and reformulate them into 3D heatmaps before feeding them into the skeleton feature extraction module; According to the key point coordinates, a Gaussian kernel is used to generate a K×H×W heat map for each frame; Stack T frames to get the 3D heat map of the entire sequence, with dimensions of K×T×H×W, where K is the number of skeleton points, T is the number of frames, and H and W are the height and width respectively; The 3D heat map is passed through the skeleton feature extraction module to obtain the skeleton features.

8. The privacy-protected medication review and liver injury early warning system according to claim 1, characterized in that: The early warning of drug-induced liver injury includes: Color calibration module: used to calibrate the image using the dynamic threshold white balance method; Sclera ROI extraction module: used to extract the sclera ROI from the color-corrected image, and use the facelandmarks detection in mediapipe to obtain facial key points; Early warning module: used to compare the video uploaded by the patient with the last uploaded video searched in the database, and issue an early warning based on the comparison results.

9. A privacy-protected medication review and liver injury early warning system according to claim 8, characterized in that: The early warning module comprises: For videos uploaded by patients, the last uploaded video is searched in the database; After color calibration, frame sampling and sclera ROI extraction, there are two paths. The first is to obtain the difference image, which is then extracted through a convolutional neural network to obtain image difference features. The second is to obtain the differences in the three color spaces of RGB, HSV, and Lab. After normalization, they are fused with the above image difference features as features, and then the warning results are obtained through the classification head. A threshold is set. If the network output score is greater than the set threshold, it means that the sclera color has changed significantly, and feedback is given.

10. A privacy-protected medication review and liver injury early warning method, characterized in that: include: Receive videos taken by patients; Use facial privacy protection algorithms to cover faces in the video with masks; The intelligent review model analyzes the video of the face covered with a mask, automatically identifying and reviewing the medication action and drug information; Image processing technology and machine learning algorithms are used to analyze the differences in the color of the patients' sclera in the video and to provide early warning for drug-induced liver injury.

Citation Information

Patent Citations

  • Health condition detection method and device and equipment

    CN106021945A

  • Video human body action recognition system and method based on deep learning and privacy protection

    CN116895098A

  • Intelligent medicine taking video auditing method and system

    CN118038321A

  • Prefab moving toy block

    KR102250615B1