A visual perception-based AD patient emotion and safety monitoring system and method

Abstract

This invention provides a visual perception-based system and method for monitoring the emotions and safety of Alzheimer's disease (AD) patients, belonging to the field of intelligent monitoring technology based on computer vision. Specifically, it includes: periodically acquiring video streams of AD patients, performing image enhancement and invalid frame removal to obtain a stable and usable image sequence; extracting facial AU4 anxiety features and behavioral features such as body posture and hip center point using MediaPipe; jointly identifying anxious wandering through sliding windows, motion path entropy, and dynamic thresholds; simultaneously detecting behavioral states such as falls, abnormal stillness, and wandering off based on body posture features; outputting a comprehensive status and providing tiered alarms to achieve seamless, accurate, and low-false-alarm real-time monitoring. This invention achieves accurate identification of "anxious wandering," is adapted to the characteristics of AD patients, has strong environmental adaptability, is low-cost, and requires no patient-worn devices, resulting in good patient compliance.

Description

Technical Field

[0001] This invention belongs to the field of intelligent monitoring technology based on computer vision, and particularly relates to a visual perception-based system and method for monitoring the emotions and safety of AD patients. Background Technology

[0002] Alzheimer's disease (AD) is a progressive neurodegenerative disease, often accompanied by symptoms such as impaired language function, emotional expression disorders, and behavioral abnormalities. According to the seventh national census, my country has 264 million people aged 60 and above, making the care needs of AD patients increasingly urgent. In elderly care facilities, AD patients are often unable to effectively express their inner states, and their emotions such as anxiety, irritability, and apathy, as well as behaviors like aimless wandering, are easily overlooked, placing enormous pressure on caregivers.

[0003] Clinical observations show that anxiety and wandering often occur simultaneously and influence each other in AD patients: anxiety drives wandering behavior, and wandering behavior in turn exacerbates anxiety. This complex state is one of the most difficult to manage in care, but current technology cannot effectively identify it.

[0004] Existing AD patient monitoring technologies have the following main shortcomings: 1. Wearable device monitoring: This method collects physiological data such as heart rate and steps from patients using devices such as smart bracelets and chest straps. However, this method requires patients to actively wear the devices, and AD patients often refuse to wear them or remove them themselves due to cognitive impairment, resulting in poor compliance. Furthermore, it cannot capture nonverbal emotional signals such as facial expressions. 2. Environmental sensor monitoring method: Infrared sensors, door magnets, pressure pads, etc., are deployed in the room to monitor physical events such as the patient getting out of bed and falling. However, this method can only identify single behaviors and cannot determine complex emotional and behavioral states such as "anxious wandering". 3. Current visual monitoring research is largely confined to the laboratory stage, using expensive equipment (such as depth cameras) or relying on multimodal sensors. General facial expression recognition models (such as OpenFace) show decreased accuracy in AD patients because AD patients have smaller facial expression ranges and greater asymmetry. Furthermore, current technologies can only identify emotions or behaviors individually, failing to capture the coupling relationship between the two simultaneously.

[0005] Meanwhile, existing technology research shows that due to uneven lighting and numerous obstructions in nursing homes, ordinary algorithms are prone to losing feature points. Real-world testing shows that the key point loss rate is approximately 8.3% in simulated elderly care scenarios. Furthermore, the “normal” state varies greatly among different AD patients, and fixed thresholds can easily lead to false alarms or missed alarms.

[0006] In summary, existing technologies cannot solve the problem of automatically identifying complex states such as "anxious wandering" in AD patients, and there is an urgent need for a low-cost, highly reliable intelligent monitoring solution that can be integrated with caregiver workflows. Summary of the Invention

[0007] To address the aforementioned problems, the first aspect of this invention provides a method for monitoring the emotion and safety of AD patients based on visual perception, comprising the following steps: S1, acquire video stream data of AD patients, preprocess each frame of the video stream, remove invalid frames to obtain a continuous and valid single-frame image sequence; S2, based on the MediaPipe FaceMesh framework, extracts facial key points from a single frame image and calculates the glabella tightness based on the coordinates of the inner points of the left and right eyebrows and the root of the nose. Anxiety feature intensity; Based on the MediaPipePose framework, human pose key points are extracted from the same single-frame image, and the coordinates of the hip center point and body pose features of the corresponding single-frame image are calculated. S3, at preset time intervals, extract the most recent consecutive multiple frames of images to form a time-series sliding judgment window, and count within the window. The percentage of frames with intensity exceeding the threshold is used to calculate the motion path entropy based on the hip center point coordinate sequence, and compared with the path entropy threshold to jointly determine whether the patient is in an anxious wandering state. S4, based on body posture characteristics, simultaneously identifies and determines abnormal states such as falling, abnormal stillness, and wandering off the boundary; S5 outputs a comprehensive assessment of emotional and safety status, including anxious wandering, falling, abnormal stillness, and wandering off the boundary.

[0008] Preferably, in step S1, the preprocessing of the video stream image specifically includes: performing CLAHE adaptive histogram equalization, Gaussian filtering for noise reduction, and Laplacian sharpening on each frame of the image in sequence; The invalid frame rejection rule is as follows: reject image frames with blurriness higher than a preset threshold or MediaPipe key point detection confidence lower than 0.5, and retain only continuous valid single-frame image sequences.

[0009] Preferably, in step S2, facial key points are extracted and calculated based on the MediaPipe FaceMesh framework. The specific steps for strength testing are as follows: S21, using the MediaPipe FaceMesh framework, extracted 468 key points on the face and located the inner point of the left eyebrow. Right eyebrow inner point Root of the nose ; S22, calculate the distance between the eyebrows, the formula is: ; in: The distance between the eyebrows, in pixels, represents the horizontal distance between the inner points of the left and right eyebrows. The coordinates of the point inside the left eyebrow; The coordinates of the point inside the right eyebrow; |·| is the absolute value operator; S23, calculate the vertical displacement between the eyebrows, the formula is: ; in: The vertical displacement between the eyebrows is expressed in pixels, representing the degree of lifting / tightening of the eyebrows relative to the root of the nose. , These are the ordinates of the inner points of the left and right eyebrows, respectively; The ordinate of the nasal root point; S24, based on the distance between eyebrows and vertical displacement, outputs the original data through the MediaPipe FaceMesh framework. Intensity value, the original output range is 0-5, the higher the value, the higher the degree of brow furrowing and the more significant the anxiety; S25, the original The intensity value is linearly normalized to the [0,1] interval to obtain the standardized AU4 intensity, which is used for subsequent anxiety state determination.

[0010] Preferably, in S2, The intensity of anxiety features is obtained through multi-scale time window weighted fusion, and the specific calculation process is as follows: Take the AU4 intensity values ​​of multiple consecutive frames within three time windows (1 second, 3 seconds, and 5 seconds) ending at the current frame, and calculate the arithmetic mean of the values ​​within each window. , , ; The enhanced AU4 anxiety feature intensity is obtained by weighting and summing the values ​​with weighting coefficients of 0.5, 0.3, and 0.2. The calculation formula is: ; in: The intensity of the enhanced AU4 anxiety feature after multi-scale fusion; Within a 1-second time window The average intensity is used to capture momentary emotional fluctuations; Within a 3-second time window The average intensity is used to characterize short-term emotional states; Within a 5-second time window The average intensity is used to reflect the sustained level of emotion; 0.5, 0.3, and 0.2 are the weighting coefficients for the corresponding time windows.

[0011] Preferably, in step S2, the specific steps for extracting key points of human posture and calculating the coordinates of the hip center point based on the MediaPipe Pose framework are as follows: Using the MediaPipe Pose framework, 33 pose key points of the human body were extracted, and the left hip key point was located. Key points of the right hip ; The coordinates of the hip center point were calculated using the coordinate mean. The calculation formula is: ; in: Coordinates of the center point of the hip The unit is pixels, used to represent the patient's real-time location and movement trajectory; Coordinates of key points on the left hip; These are the coordinates of the key points on the right hip.

[0012] Preferably, in step S2, after extracting key points, the process further includes feature completion and confidence weighting steps, specifically: When the confidence score of a keypoint output by MediaPipe is lower than 0.5, it is determined that the keypoint is missing. For cases where the number of consecutive lost frames is ≤5, linear interpolation is used to fill in the missing keypoint coordinates; For cases where more than 5 frames are lost consecutively, Kalman filtering is used to predict and complete the key point coordinates. The completed key points are labeled with confidence levels. In subsequent feature calculations, the feature weights of key points with confidence levels below 0.7 are reduced by 50% to reduce the interference of low-confidence features on the judgment results.

[0013] Preferably, in step S3, the specific calculation process of the motion path entropy is as follows: The monitoring area was divided into M=100 equal-sized grids (10 rows x 10 columns). The center point of the hip was counted within the time-series sliding judgment window. Calculate the probability of occurrence of each grid cell by counting the number of times it appears in each cell. ; Calculate motion path entropy based on the information entropy formula. This is used to quantify the degree of disorder in a patient's wandering behavior, and the calculation formula is: ; in: The entropy of the movement path is expressed in bits. The larger the entropy value, the more disordered the patient's movement trajectory and the more significant the pathological characteristics of wandering. Let be the probability that the center point of the hip appears in the i-th grid within the time window, satisfying that the sum of the probabilities of all grids is 1; It is a logarithmic function with base 2; M=100 represents the total number of grids in the monitored area.

[0014] Preferably, in S3, The intensity threshold and path entropy threshold are adjusted using a dynamic adaptive threshold adjustment method based on the patient's historical baseline, specifically as follows: AU4 intensity data and path entropy data of patients were collected over the previous 24 hours. Their mean μ and standard deviation σ were calculated respectively, and a threshold was set. ; in: For dynamic adaptive thresholds; This represents the historical 24-hour average of the corresponding feature; The historical 24-hour standard deviation of the corresponding feature; automatically updated every 24 hours. For newly admitted patients, when there is no historical baseline data, the population mean and standard deviation of AD patients of the same age and disease grade are used as the initial baseline to generate the initial threshold.

[0015] Preferably, in step S3, the specific parameters of the timing sliding decision window are: The window length N = 300 frames, corresponding to 10 seconds of video data, with a video frame rate of 30fps; The sliding step size is 1 second, that is, the most recent 300 consecutive frames of images are captured every 1 second to form a new time sequence judgment window, so as to realize real-time and continuous monitoring of the patient's status.

[0016] Preferably, in step S3, the specific rule for determining the anxious wandering state is as follows: Within a single time-series sliding decision window, statistics The percentage of frames exceeding the threshold ; If satisfied ≥ 0.5, and motion path entropy Path entropy threshold This decision window is recorded as a valid anxiety window. When K consecutive judgment windows are all valid anxiety windows and the corresponding total duration is ≥3 seconds, the AD patient is determined to be in an anxious wandering state, avoiding misjudgment caused by momentary emotional fluctuations or brief disordered movements.

[0017] Preferably, in step S3, the specific rules for determining the various types of abnormal security behaviors are as follows: Fall detection: Based on body posture characteristics, calculate the inclination angle θ of the line connecting the shoulder key point and the hip key point, and the vertical descent velocity of the hip center point. Body posture, width-to-height ratio r; when satisfied A fall is defined as a situation where the speed is 1.5 m / s, θ > 45°, and r > 1.2. Abnormal stillness determination: When the cumulative distance of movement of the center point of the hip is less than 0.5m within 30 consecutive seconds, it is determined to be abnormal stillness, which is used to identify patients who are immobile for a long time, have potential coma or discomfort. Outbound Loss Detection: Based on a preset electronic fence, when the coordinates of the center point of the hip exceed the boundary of the electronic fence, it is determined as outbound loss, which is used to prevent patients from leaving the monitoring area without authorization.

[0018] A second aspect of the present invention provides a visual perception-based emotion and safety monitoring system for Alzheimer's disease (AD) patients, applied to the AD patient emotion and safety monitoring method described in the first aspect, comprising: The image processing module is used to acquire video stream data of AD patients, preprocess each frame of the video stream, remove invalid frames, and obtain a continuous and valid single-frame image sequence. The feature extraction module, based on the MediaPipe FaceMesh framework, extracts facial key points from a single frame image. It calculates the glabella tightness based on the coordinates of the inner points of the left and right eyebrows and the root of the nose. Anxiety feature intensity; Based on the MediaPipe Pose framework, human pose key points are extracted from the same single-frame image, and the coordinates of the hip center point and body pose features of the corresponding single-frame image are calculated. The first judgment module, at preset time intervals, extracts the most recent consecutive multiple frames of images to form a time-series sliding judgment window, and counts within the window. The percentage of frames with intensity exceeding the threshold is used to calculate the motion path entropy based on the hip center point coordinate sequence, and compared with the path entropy threshold to jointly determine whether the patient is in an anxious wandering state. The second judgment module, based on body posture characteristics, simultaneously identifies and judges abnormal states such as falling, abnormal stillness, and wandering off the boundary. The output module is used to output a comprehensive assessment of emotional and safety status, including anxious wandering, falling, abnormal stillness, and wandering off the boundary.

[0019] Compared with the prior art, the present invention has the following innovative features and beneficial effects.

[0020] 1. Identifying Complex States: For the first time, sentiment indicators ( By integrating this data with behavioral indicators (path entropy), accurate identification of "anxious wandering" was achieved. Experiments show that a single... The accuracy was 68.5%, the single-path entropy was 71.2%, and after fusion it increased to 84.7%, an improvement of 13.5 percentage points, proving the non-obviousness of "1+1>2".

[0021] 2. Adapted to AD Patient Characteristics: A multi-scale AU-weighted method was designed to address the limited facial expression range characteristic of AD patients. Experiments show that in videos with small facial expressions, the traditional method achieves a capture rate of 45.2%, while the method of this invention improves this to 78.6%, an increase of 33.4 percentage points, significantly reducing the false negative rate.

[0022] 3. Strong environmental adaptability: A hierarchical feature completion method was designed to address the occlusion problem in nursing homes. Experiments show that the feature point loss rate decreased from 8.3% to 1.2%, ensuring stability in complex environments.

[0023] 4. Personalized Adaptation: To address individual patient differences, a dynamic threshold method based on historical baselines was designed, reducing the false alarm rate from 18.5% to 9.2% and the false negative rate from 12.3% to 8.1%.

[0024] 5. High efficiency of human-machine collaboration: The hierarchical early warning mechanism reduces invalid alarms, with the invalid alarm rate dropping from 35.2% to 12.5%; the caregiver feedback mechanism supports continuous model optimization, with the caregiver acceptance score increasing from 2.8 to 4.2 (out of 5).

[0025] 6. Low cost: Only a regular RGB camera is required, without the need for depth cameras or multimodal sensors, making it suitable for large-scale deployment.

[0026] 7. Non-contact monitoring: Patients do not need to wear any devices, resulting in good compliance. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating the overall technical route of the present invention.

[0028] Figure 2 This is a structural diagram of the feature extraction module of the present invention (illustration of MediaPipe key point extraction).

[0029] Figure 3 This is a flowchart of the dual-modal fusion determination process of the present invention.

[0030] Figure 4 This invention provides a confusion matrix for identifying both emotion and security status.

[0031] Figure 5 This is a time series graph showing the changes in anxiety intensity of AU4 according to the present invention. Detailed Implementation

[0032] This invention mainly provides a method for monitoring the emotions and safety of AD patients based on visual perception. The overall technical approach is as follows: Figure 1 As shown, the process includes the following: S1, Video stream acquisition, preprocessing and invalid frame removal: Every preset time period, video stream data of AD patients is acquired, each frame of the video stream is preprocessed, and invalid frames are removed to obtain a continuous and valid single-frame image sequence. S2, Facial AU4 Feature and Human Pose Feature Extraction: Based on the MediaPipe FaceMesh framework, facial key points are extracted from a single frame image. The glabella tightness is calculated based on the coordinates of the inner left eyebrow point, the inner right eyebrow point, and the root of the nose. Anxiety feature intensity; Based on the MediaPipe Pose framework, human pose key points are extracted from the same single-frame image, and the coordinates of the hip center point and body pose features of the corresponding single-frame image are calculated. S3, Temporal sliding window judgment, joint judgment of anxiety wandering and security anomaly identification: Every preset time period, the most recent consecutive multiple frames of images are extracted to form a temporal sliding judgment window, and statistics are performed within the window. The percentage of frames with intensity exceeding the first threshold is used to calculate the motion path entropy based on the hip center point coordinate sequence and compare it with the path entropy threshold. Combined with continuous multi-window temporal constraints, this is used to jointly determine whether the patient is in an anxious wandering state. Based on body posture features, S4 can simultaneously identify multiple types of abnormal safety behaviors, such as falls, abnormal stillness, and wandering off the boundary. S5, Output of comprehensive emotion and safety status assessment results: The output includes comprehensive emotion and safety status assessment results including anxious wandering, falling, abnormal stillness, and wandering out of boundaries.

[0033] The invention will be further described below with reference to specific embodiments.

[0034] Example 1: I. Video Stream Acquisition, Preprocessing, and Invalid Frame Removal To achieve stable monitoring of the emotional and safety status of AD patients, this embodiment deploys a regular RGB camera in an actual elderly care facility environment. The camera is installed at a height of 2.2 meters, with a downward angle of 15°, covering the main activity area. The video frame rate is fixed at 30fps, and the image resolution is 1280×720 pixels.

[0035] Video stream data of AD patients is acquired at preset time intervals. The following preprocessing operations are performed sequentially on each frame of the video stream: 1. CLAHE Adaptive Histogram Equalization: Convert the image to the LAB color space, apply the CLAHE algorithm only to the L channel, limit the contrast magnification factor to 2.0, and the block size to 8×8 pixels to enhance local contrast and avoid the effects of uneven lighting. 2. Gaussian filtering for noise reduction: A 5×5 Gaussian kernel with a standard deviation σ=1.5 is used to suppress sensor noise and slight motion blur; 3. Laplacian sharpening: The image is sharpened using a 3×3 Laplacian kernel (center value of 5, neighborhood value of -1) to enhance facial texture and edge information.

[0036] Invalid frame removal rules: Remove image frames with blur levels higher than a preset threshold or MediaPipe keypoint detection confidence levels lower than 0.5, retaining only consecutive valid single-frame image sequences to ensure the accuracy and stability of subsequent feature extraction.

[0037] II. Face Feature and Human Posture Feature Extraction like Figure 2 As shown, based on the MediaPipe FaceMesh framework, facial key points are extracted from a single frame image. The glabella tightness is calculated based on the coordinates of the inner points of the left and right eyebrows and the root of the nose. Anxiety feature intensity; Based on the MediaPipe Pose framework, human posture key points are extracted from the same single-frame image, and the coordinates of the hip center point and body posture features of the corresponding single-frame image are calculated.

[0038] 1. Based on MediaPipe FaceMesh Anxiety feature intensity extraction was performed using the MediaPipe FaceMesh framework, extracting 468 key points on the face and locating the inner point of the left eyebrow. Right eyebrow inner point Root of the nose .

[0039] The formula for calculating the distance between the eyebrows is: ; in: The distance between the eyebrows, in pixels, represents the horizontal distance between the inner points of the left and right eyebrows. The coordinates of the point inside the left eyebrow; Here are the coordinates of the inner point of the right eyebrow; |·| is the absolute value operator.

[0040] The formula for calculating the vertical displacement between the eyebrows is: ; in: The vertical displacement between the eyebrows is expressed in pixels, representing the degree of lifting / tightening of the eyebrows relative to the root of the nose. , These are the ordinates of the inner points of the left and right eyebrows, respectively; The vertical coordinate of the nasal root point is given.

[0041] Based on the distance between eyebrows and vertical displacement, the original data is output through the MediaPipe FaceMesh framework. The intensity value, originally ranging from 0 to 5, indicates a higher degree of brow furrowing and more pronounced anxiety. The intensity values ​​are linearly normalized to the [0,1] interval to obtain the standardized values. Intensity, used for subsequent assessment of anxiety state.

[0042] Multi-scale time window weighted fusion: Take multiple consecutive frames within three time windows of 1 second, 3 seconds, and 5 seconds, with the current frame as the endpoint. Intensity value, calculate the arithmetic mean within each window. , , The enhanced AU4 anxiety feature intensity was obtained by weighting and summing the values ​​according to weight coefficients of 0.5, 0.3, and 0.2. The calculation formula is: ; in: Enhanced version after multi-scale fusion Intensity of anxiety characteristics; Within a 1-second time window The average intensity is used to capture momentary emotional fluctuations; Within a 3-second time window The average intensity is used to characterize short-term emotional states; Within a 5-second time window The average intensity is used to reflect the sustained emotional level; 0.5, 0.3, and 0.2 are the weighting coefficients for the corresponding time windows, which highlight the instantaneous emotional characteristics while taking into account the persistence of emotions and weakening the noise interference of single frames.

[0043] like Figure 5The figure shows a typical curve illustrating the change in AU4 anxiety intensity over time. As can be seen from the graph, under normal conditions, the AU4 intensity remains at a low level between 0.25 and 0.35; when a patient enters an anxious state, the AU4 intensity significantly rises to above 0.55 and lasts for a longer period. This invention, by setting a dynamic threshold (shown by the dotted line in the figure), can effectively identify the onset and duration of anxiety, providing reliable emotional feature input for subsequent joint determination of anxiety lingering.

[0044] 2. Human pose feature extraction based on MediaPipe Pose: Using the MediaPipe Pose framework, 33 human pose key points were extracted, and the left hip key point was located. Key points of the right hip The coordinates of the hip center point were calculated using the coordinate mean. The calculation formula is: ; in: Coordinates of the center point of the hip The unit is pixels, used to represent the patient's real-time location and movement trajectory; Coordinates of key points on the left hip; These are the coordinates of the key points on the right hip.

[0045] Simultaneously, the shoulder center point is extracted to calculate the shoulder-hip line inclination angle θ, the body posture width-to-height ratio r, and the vertical descent velocity of the hip center point. This is used for subsequent determination of abnormal security behavior.

[0046] 3. Feature completion When the confidence score of a keypoint output by MediaPipe is below 0.5, it is considered a missing keypoint. For cases where the number of consecutive lost frames is ≤5, linear interpolation is used to complete the missing keypoint coordinates; for cases where the number of consecutive lost frames is >5, Kalman filtering is used to predict and complete the keypoint coordinates. The completed keypoints are labeled with a confidence score. In subsequent feature calculations, the feature weight of keypoints with a confidence score below 0.7 is reduced by 50% to minimize the interference of low-confidence features on the judgment results.

[0047] III. Timing Sliding Window Decision, Joint Decision on Anxious Hesitation, and Identification of Security Anomalies like Figure 3 As shown, at preset time intervals, the most recent consecutive frames are captured to form a time-series sliding judgment window, and statistics are calculated within the window. The percentage of frames with intensity exceeding the first threshold is used to calculate the motion path entropy based on the hip center point coordinate sequence. Combined with continuous multi-window temporal constraints, this is used to jointly determine whether the patient is in an anxious wandering state.

[0048] Simultaneously, based on body posture characteristics, it can identify multiple types of abnormal safety behaviors, such as falls, abnormal stillness, and wandering off the line.

[0049] 1. Timing Sliding Decision Window Configuration In this embodiment, the window length N = 300 frames, corresponding to 10 seconds of video data, with a video frame rate of 30fps. The sliding step is 1 second, meaning that the most recent 300 consecutive frames are captured every 1 second to form a new time-series judgment window, enabling real-time and continuous monitoring of the patient's status.

[0050] 2. Motion path entropy calculation divides the monitoring area into 10 rows × 10 columns, totaling M = 100 equally sized grids. Within the statistical time-series sliding judgment window, the center point of the hip is... Calculate the probability of occurrence of each grid cell by counting the number of times it appears in each cell. Calculate motion path entropy based on the information entropy formula. This is used to quantify the degree of disorder in a patient's wandering behavior, and the calculation formula is: ; in: The entropy of the movement path is expressed in bits. The larger the entropy value, the more disordered the patient's movement trajectory and the more significant the pathological characteristics of wandering. Let be the probability that the center point of the hip appears in the i-th grid within the time window, satisfying that the sum of the probabilities of all grids is 1; It is a logarithmic function with base 2; M=100 is the total number of grids in the monitoring area.

[0051] 3. Dynamic adaptive threshold setting Collect the patient's data 24 hours prior to the event For both intensity data and motion path entropy data, calculate their respective mean μ and standard deviation σ, and set thresholds: ; in: For dynamic adaptive thresholds; For corresponding features ( Historical 24-hour average of intensity data and motion path entropy data; The historical 24-hour standard deviation of the corresponding feature; automatically updated every 24 hours.

[0052] For newly admitted patients, when there is no historical baseline data, the population mean and standard deviation of AD patients of the same age and disease grade are used as the initial baseline to balance individual adaptability and monitoring universality.

[0053] 4. Joint assessment of anxious wandering state Within a single time-series sliding decision window, statistics Percentage of frames with intensity exceeding the first threshold .

[0054] If satisfied ≥ 0.5, and motion path entropy Path entropy threshold This decision window is recorded as a valid anxiety window.

[0055] When K consecutive judgment windows are all valid anxiety windows and the corresponding total duration is ≥3 seconds, the AD patient is determined to be in an anxious wandering state, avoiding misjudgment caused by momentary emotional fluctuations or brief disordered movements.

[0056] 5. Simultaneous identification of multiple types of abnormal security behaviors (1) Fall detection: Based on body posture characteristics, calculate the inclination angle θ of the line connecting the shoulder key point and the hip key point, and the vertical descent velocity of the hip center point. Body posture, width-to-height ratio r. When satisfied... A fall is defined as a movement of >1.5 m / s, θ>45°, and r>1.2. (2) Abnormal stillness judgment: When the cumulative movement distance of the center point of the hip is less than 0.5m within 30 consecutive seconds, it is judged as abnormal stillness, which is used to identify patients who are immobile for a long time, have potential coma or discomfort. (3) Boundary crossing and missing determination: Based on the preset electronic fence (custom closed polygonal safety area), when the coordinates of the center point of the hip exceed the boundary of the electronic fence, it is determined as a boundary crossing and missing, which is used to prevent the patient from leaving the monitoring area without authorization.

[0057] IV. Output of Comprehensive Emotion and Safety Status Assessment Results The output includes a comprehensive assessment of emotional and safety status, including anxious wandering, falling, abnormal stillness, and wandering off the boundary.

[0058] This embodiment outputs a comprehensive status report containing the following fields: Timestamp; Anxious wandering state (yes / no); Fall status (yes / no); Abnormal static state (Yes / No); Status of wandering out of boundaries (Yes / No); It also supports tiered early warning systems: Level 1 Warning: The intensity exceeded the limit, but the path entropy did not; only logs were recorded. Level 2 alert: If a single window meets the conditions for anxious wandering, the interface will highlight the alert. Level 3 alert: If multiple windows continuously detect anxious wandering, or identify a fall / crossing the boundary and getting lost, immediately push a pop-up window and sound alarm.

[0059] Example 2: To comprehensively and objectively verify the effectiveness of the visual perception-based method for monitoring the emotions and safety of AD patients proposed in this invention, this embodiment designed and implemented a series of detailed simulation experiments. The experiments aimed to evaluate the improvements of this invention compared to existing methods in terms of accuracy in identifying anxious wandering, ability to capture subtle facial expressions, environmental robustness, personalized adaptability, and human-machine collaboration efficiency.

[0060] This experiment deployed standard RGB cameras in a simulated nursing home environment. The cameras were installed at a height of 2.2 meters, a downward angle of 15°, covering a 5m × 5m rectangular area, with a video frame rate of 30fps and a resolution of 1280 × 720 pixels. A total of 20 Alzheimer's disease (AD) patients (CDR score 1-2, age 65-85 years, half male and half female) were recruited to participate in data collection. Video data was collected continuously for 7 days, totaling approximately 336 hours. The consistency between the on-site records of caregivers and the retrospective annotations of three experts was used as the gold standard.

[0061] The method of the present invention was compared with the following four representative baseline methods: (1) Single Judgment method: using only (1) Intensity features: using a fixed threshold to determine anxiety state, without integrating path entropy information; (2) Single path entropy determination method: using only motion path entropy features, using a fixed threshold to determine wandering behavior, without integrating... Information; (3) Traditional single frame Extraction method: Instead of using multi-scale temporal window weighted fusion, single-frame normalization is used directly. Intensity value; (4) Fixed threshold method: Instead of using dynamic adaptive threshold, an empirical fixed threshold is used for judgment.

[0062] The comparative experimental results of various methods in terms of accuracy in identifying anxious wandering are shown in Table 1. The experimental data demonstrate that the method of this invention significantly outperforms all the compared methods in terms of accuracy in identifying anxious wandering. (Traditional single-frame...) Extraction accuracy (fixed threshold) was 65.3%, single The accuracy rate of the judgment (multi-scale fusion + fixed threshold) is 68.5%, the accuracy rate of the single path entropy judgment (fixed threshold) is 71.2%, while the accuracy rate of the joint judgment (multi-scale fusion + dynamic threshold) of this invention is improved to 84.7%, which is 13.5 percentage points higher than the better-performing single path entropy judgment, and higher than the traditional single frame. The extraction rate was improved by 19.4 percentage points, demonstrating the necessity and superiority of multi-scale weighted fusion and dynamic adaptive threshold.

[0063] Table 1 Comparison of accuracy rates for identifying anxious wandering among different methods

[0064] like Figure 4 The figure shows the confusion matrix of the method of the present invention for five states (normal / other, anxious wandering, falling, abnormal stillness, and wandering out of bounds). As can be seen from the figure, the method of the present invention achieves excellent results in the identification of all states: the accuracy rate for normal / other is 89.0%, the accuracy rate for anxious wandering is 84.8%, the accuracy rate for falling is 92.0%, the accuracy rate for abnormal stillness is 89.6%, the accuracy rate for wandering out of bounds is 90.0%, and the overall accuracy rate reaches 89.1%.

[0065] To address the limited facial expression range characteristic of AD patients, this invention designs a multi-scale temporal window weighted fusion method. Experiments were conducted on video clips featuring small facial expressions, and the results are shown in Table 2. Traditional single-frame... The capture rate of the extraction method was 45.2%, while the multi-scale weighted fusion method of the present invention improved the capture rate to 78.6%, an increase of 33.4 percentage points, which significantly reduced the false negative rate caused by small facial expression amplitude.

[0066] Table 2 Comparison of AU4 capture rate for minor facial expressions.

[0067] To address interference issues such as occlusion, rapid movement, and uneven lighting in nursing home environments, this invention designs a hierarchical completion strategy combining linear interpolation and Kalman filtering. Experiments were conducted in a simulated nursing home environment, and the results are shown in Table 3. Without completion, the feature point loss rate was 8.3%, while the hierarchical completion strategy of this invention reduced the feature point loss rate to 1.2%, ensuring the stability of feature extraction in complex environments.

[0068] Table 3 Comparison of Feature Point Loss Rate

[0069] To address individual differences among AD patients, this invention designs a dynamic adaptive thresholding method based on historical baselines. The false alarm rate and false negative rate of fixed thresholds versus dynamic thresholds were compared experimentally, and the results are shown in Table 4. The false alarm rate of the fixed threshold was 18.5%, and the false negative rate was 12.3%, while the false alarm rate of the dynamic threshold of this invention was reduced to 9.2%, and the false negative rate was reduced to 8.1%, effectively reducing the false alarm and false negative problems caused by fixed thresholds.

[0070] Table 4 Comparison of effects between dynamic threshold and fixed threshold

[0071] To address the problem of excessive invalid alarms in caregiver work, this invention designs a tiered early warning mechanism (prompt / attention / emergency). An experiment comparing the invalid alarm rate and caregiver acceptance with no tiered early warning and the tiered early warning system of this invention, the results are shown in Table 5. Without tiered early warning, the invalid alarm rate was 35.2%, and the caregiver acceptance score was 2.8 out of 5. After implementing the tiered early warning system of this invention, the invalid alarm rate decreased to 12.5%, and the caregiver acceptance score increased to 4.2, effectively reducing the interference of invalid alarms on caregiver work and achieving good human-machine collaboration.

[0072] Table 5 Comparison of the effects of tiered early warning systems

[0073] In summary, the experimental results verify that the visual perception-based AD patient emotion and safety monitoring method proposed in this invention has significant advantages in complex state recognition, AD patient adaptability, environmental robustness, personalized adaptation, and human-machine collaboration. This system requires only a standard RGB camera and does not require patients to wear any devices, achieving low-cost, unobtrusive, and highly reliable AD patient emotion and safety monitoring.

[0074] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0075] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A visual perception based AD patient emotion and safety monitoring method, characterized in that, Includes the following processes: S1, acquire video stream data of AD patients, preprocess each frame of the video stream, remove invalid frames to obtain a continuous and valid single-frame image sequence; S2, based on the MediaPipe FaceMesh framework, facial key points are extracted from a single frame image, and the inter-brow contraction degree is calculated according to the coordinates of the left inner eyebrow point, the right inner eyebrow point and the nose root point, to obtain anxiety feature intensity; based on the MediaPipe Pose framework, human body posture key points are extracted from the same single frame image, and the hip center point coordinates and body posture features of the corresponding single frame image are calculated. S3, every other preset time period, intercept the recent continuous multiple frames of images to constitute a time sequence sliding determination window, count the number of frames whose intensity exceeds the threshold value in the window The number of frames whose intensity exceeds the threshold value accounts for a proportion, the motion path entropy is calculated according to the sequence of hip center point coordinates, and is compared with a path entropy threshold value, to jointly determine whether the patient is in an anxious wandering state. S4, based on body posture characteristics, simultaneously identifies and determines abnormal states such as falling, abnormal stillness, and wandering off the boundary; S5 outputs a comprehensive assessment of emotional and safety status, including anxious wandering, falling, abnormal stillness, and wandering off the boundary.

2. The method for monitoring the emotion and safety of AD patients based on visual perception as described in claim 1, characterized in that: In step S2, facial key points are extracted and calculated based on the MediaPipe FaceMesh framework. The specific steps for strength testing are as follows: S21, using the MediaPipe FaceMesh framework, extracted 468 key points on the face and located the inner point of the left eyebrow. Right eyebrow inner point Root of the nose ; S22, calculate the distance between the eyebrows, the formula is: ； in: The distance between the eyebrows, in pixels, represents the horizontal distance between the inner points of the left and right eyebrows. The coordinates of the point inside the left eyebrow; The coordinates of the inner point of the right eyebrow are given; |·| is the absolute value operator. S23, calculate the vertical displacement between the eyebrows, the formula is: ； in: The vertical displacement between the eyebrows is expressed in pixels, representing the degree of lifting / tightening of the eyebrows relative to the root of the nose. , These are the ordinates of the inner points of the left and right eyebrows, respectively; The ordinate of the nasal root point; S24, based on the distance between eyebrows and vertical displacement, outputs the original data through the MediaPipe FaceMesh framework. Intensity value, the original output range is 0-5, the higher the value, the higher the degree of brow furrowing and the more significant the anxiety; S25, the original The intensity value is linearly normalized to the [0,1] interval to obtain the standardized AU4 intensity, which is used for subsequent anxiety state determination.

3. The method for monitoring the emotion and safety of AD patients based on visual perception as described in claim 2, characterized in that: In S2, The intensity of anxiety features is obtained through multi-scale time window weighted fusion, and the specific calculation process is as follows: Take the AU4 intensity values ​​of multiple consecutive frames within three time windows (1 second, 3 seconds, and 5 seconds) ending at the current frame, and calculate the arithmetic mean of the values ​​within each window. , , ; The enhanced AU4 anxiety feature intensity is obtained by weighting and summing the values ​​with weighting coefficients of 0.5, 0.3, and 0.

2. The calculation formula is: ； in: The intensity of the enhanced AU4 anxiety feature after multi-scale fusion; Within a 1-second time window The average intensity is used to capture momentary emotional fluctuations; Within a 3-second time window The average intensity is used to characterize short-term emotional states; Within a 5-second time window The average intensity is used to reflect the sustained level of emotion; 0.5, 0.3, and 0.2 are the weighting coefficients for the corresponding time windows.

4. The method for monitoring the emotion and safety of AD patients based on visual perception as described in claim 1, characterized in that: In step S2, the specific steps for extracting key points of human posture and calculating the coordinates of the hip center point based on the MediaPipe Pose framework are as follows: Using the MediaPipe Pose framework, 33 pose key points of the human body were extracted, and the left hip key point was located. Key points of the right hip ; The coordinates of the hip center point were calculated using the coordinate mean. The calculation formula is: ； in: Coordinates of the center point of the hip The unit is pixels, used to represent the patient's real-time location and movement trajectory; Coordinates of the key points on the left hip; The coordinates are for the key points on the right hip.

5. The method for monitoring the emotion and safety of AD patients based on visual perception as described in claim 1, characterized in that: After extracting key points, the process also includes feature completion and confidence weighting steps, specifically: When the confidence score of a keypoint output by MediaPipe is lower than 0.5, it is determined that the keypoint is missing. For cases where the number of consecutive lost frames is ≤5, linear interpolation is used to fill in the missing keypoint coordinates; For cases where more than 5 frames are lost consecutively, Kalman filtering is used to predict and complete the key point coordinates. The completed key points are labeled with confidence levels. In subsequent feature calculations, the feature weights of key points with confidence levels below 0.7 are reduced by 50% to reduce the interference of low-confidence features on the judgment results.

6. The method for monitoring the emotion and safety of AD patients based on visual perception as described in claim 1, characterized in that: In S3, the specific calculation process of the motion path entropy is as follows: The monitoring area was divided into M=100 equal-sized grids of 10 rows × 10 columns. The center point of the hip was counted within the time-series sliding judgment window. Calculate the probability of occurrence of each grid cell by counting the number of times it appears in each cell. ; Calculate motion path entropy based on the information entropy formula. This is used to quantify the degree of disorder in a patient's wandering behavior, and the calculation formula is: ； in: The entropy of the movement path is expressed in bits. The larger the entropy value, the more disordered the patient's movement trajectory and the more significant the pathological characteristics of wandering. Let be the probability that the center point of the hip appears in the i-th grid within the time window, satisfying that the sum of the probabilities of all grids is 1; It is a logarithmic function with base 2; M=100 represents the total number of grids in the monitored area.

7. The method for monitoring the emotion and safety of AD patients based on visual perception as described in claim 1, characterized in that: In S3, The intensity threshold and path entropy threshold are adjusted using a dynamic adaptive threshold adjustment method based on the patient's historical baseline, specifically as follows: AU4 intensity data and path entropy data of patients were collected over the previous 24 hours. Their mean μ and standard deviation σ were calculated respectively, and a threshold was set. ； in: For dynamic adaptive thresholds; This represents the historical 24-hour average of the corresponding feature; The historical 24-hour standard deviation of the corresponding feature; automatically updated every 24 hours. For newly admitted patients, when there is no historical baseline data, the population mean and standard deviation of AD patients of the same age and disease grade are used as the initial baseline to generate the initial threshold.

8. The method for monitoring the emotion and safety of AD patients based on visual perception as described in claim 1, characterized in that: In S3, the specific rules for determining the anxious wandering state are as follows: Within a single time-series sliding decision window, statistics The percentage of frames exceeding the threshold ; If satisfied ≥ 0.5, and motion path entropy Path entropy threshold This decision window is recorded as a valid anxiety window. When K consecutive judgment windows are all valid anxiety windows and the corresponding total duration is ≥3 seconds, the AD patient is determined to be in an anxious wandering state, avoiding misjudgment caused by momentary emotional fluctuations or brief disordered movements.

9. The method for monitoring the emotion and safety of AD patients based on visual perception as described in claim 1, characterized in that: In S3, the specific rules for determining various types of abnormal security behaviors are as follows: Fall detection: Based on body posture characteristics, calculate the inclination angle θ of the line connecting the shoulder key point and the hip key point, and the vertical descent velocity of the hip center point. Body posture, width-to-height ratio r; when satisfied A fall is defined as a situation where the speed is 1.5 m / s, θ > 45°, and r > 1.

2. Abnormal stillness determination: When the cumulative distance of movement of the center point of the hip is less than 0.5m within 30 consecutive seconds, it is determined to be abnormal stillness, which is used to identify patients who are immobile for a long time, have potential coma or discomfort. Outbound Loss Detection: Based on a preset electronic fence, when the coordinates of the center point of the hip exceed the boundary of the electronic fence, it is determined as outbound loss, which is used to prevent patients from leaving the monitoring area without authorization.

10. A visual perception-based AD patient emotion and safety monitoring system, applied to the AD patient emotion and safety monitoring method according to any one of claims 1 to 9, characterized in that, include: The image processing module is used to acquire video stream data of AD patients, preprocess each frame of the video stream, and remove invalid frames to obtain a continuous and valid single-frame image sequence. The feature extraction module, based on the MediaPipe FaceMesh framework, extracts facial key points from a single frame image. It calculates the glabella tightness based on the coordinates of the inner points of the left and right eyebrows and the root of the nose. Anxiety feature intensity; Based on the MediaPipe Pose framework, human pose key points are extracted from the same single-frame image, and the coordinates of the hip center point and body pose features of the corresponding single-frame image are calculated. The first judgment module, at preset time intervals, extracts the most recent consecutive multiple frames of images to form a time-series sliding judgment window, and counts within the window. The percentage of frames with intensity exceeding the threshold is used to calculate the motion path entropy based on the hip center point coordinate sequence, and compared with the path entropy threshold to jointly determine whether the patient is in an anxious wandering state. The second judgment module, based on body posture characteristics, simultaneously identifies and judges abnormal states such as falling, abnormal stillness, and wandering off the boundary. The output module is used to output a comprehensive assessment of emotional and safety status, including anxious wandering, falling, abnormal stillness, and wandering off the boundary.

Images