A medicine taking behavior recognition method and system based on time sequence action detection

Abstract

The application provides a medicine-taking behavior recognition method and system based on time sequence action detection, and relates to the technical field of image recognition.The method comprises the following steps: positioning a human head region and an upper limb region in a continuous video stream, and intercepting image frames around a mouth in a continuous preset time length interval in the continuous video stream; frame by frame, the density of reflected highlight pixels of the image frames around the mouth is counted; baseline drift compensation is performed on the density of reflected highlight pixels, so that a time sequence highlight density series of the image frames around the mouth is obtained; the time sequence highlight density series is subjected to response integrity analysis, so that a saliva secretion response identifier of the time sequence highlight density series is obtained; according to the saliva secretion response identifier, time sequence causal consistency determination of a medicine-taking event of a target object is performed, so that a behavior recognition result of the medicine-taking event is obtained; and the application can improve the efficiency of the medicine-taking behavior recognition based on time sequence action detection.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method and system for recognizing medication behavior based on temporal action detection. Background Technology

[0002] For visual recognition of medication-taking behavior in continuous video streams, existing technologies typically rely on spatial overlap criteria between hand trajectories and the mouth area for action trigger detection. This spatial overlap criterion is prone to feature ambiguity when head posture changes or the hand briefly covers an occluded area, leading to significant deviations in the localization of the behavior trigger moment. Simultaneously, the physiological saliva secretion accompanying medication-taking causes dynamic changes in the specular reflection properties of the skin surface around the mouth. However, the amplitude of this reflected high-gloss signal is weak and superimposed with baseline drift components caused by gradual changes in ambient lighting conditions. Conventional temporal thresholding methods cannot effectively decouple the actual secretion response signal from the lighting drift noise, resulting in a weak correlation between the statistical results of single-frame high-gloss density and the actual physiological process of medication-taking.

[0003] At the temporal dynamic analysis level, existing solutions generally rely solely on the amplitude transition of high optical density values ​​as the response activation criterion, lacking a self-consistent verification mechanism for the integrity of the rise and fall segments of the response waveform. This makes it difficult to effectively eliminate occasional high-light pulse interference triggered by non-drug-taking actions. More critically, existing recognition frameworks typically neglect the temporal causal constraint between the hand-to-mouth action and the subsequent peak of salivary secretion response. They neither include the moment of action occurrence and the moment of physiological response peak in a unified time axis for sequential comparison, nor set a physiological time window to check the interval attribution of the two time spans. This results in a temporal logical disconnect between the action trigger signal and the physiological feedback signal, failing to form a strict causal chain. Therefore, improving the recognition efficiency of drug-taking behavior has become an urgent problem to be solved. Summary of the Invention

[0004] This invention provides a method and system for recognizing medication behavior based on temporal action detection, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a method for recognizing medication behavior based on temporal action detection, comprising:

[0006] S01. Locate the head and upper limb regions of the human body in the continuous video stream, and extract the image frames around the mouth within a continuous preset time interval in the continuous video stream.

[0007] S02. Calculate the reflective highlight pixel density of the image frames around the mouth frame by frame;

[0008] S03. Perform baseline drift compensation on the reflected specular pixel density to obtain the temporal specular density sequence of the image frames around the mouth;

[0009] S04. Perform response integrity analysis on the time-series high optical density sequence to obtain the salivary secretion response identifier of the time-series high optical density sequence;

[0010] S05. Based on the salivary secretion response identifier, perform a temporal causal consistency determination on the medication event of the target object to obtain the behavioral identification result of the medication event.

[0011] In a preferred embodiment, locating the human head region and upper limb region in a continuous video stream, and extracting image frames around the mouth within a continuous preset time interval in the continuous video stream, includes:

[0012] Histogram spatial clustering is performed on the facial skin color pixel cluster and the hand skin color pixel cluster of the continuous video stream according to the brightness gradient direction to obtain the facial region envelope and hand region envelope of the continuous video stream.

[0013] The state transition moment of the distance between the facial region envelope and the hand region envelope is captured instantaneously, and the state transition moment is taken as the moment when the hand is withdrawn from the mouth;

[0014] Taking the moment when the hand is withdrawn from the mouth as the starting zero point, the continuous video stream within a preset time interval after the starting zero point is extracted to obtain image frames around the mouth.

[0015] In a preferred embodiment, the state transition moment of instantaneously capturing the distance between the facial region envelope and the hand region envelope, and using the state transition moment as the moment when the hand is withdrawn from the mouth, includes:

[0016] The spatial Euclidean distance between the center points of the facial region envelope and the hand region envelope is quantified frame by frame to obtain the instantaneous value of the mouth-hand distance.

[0017] Using each frame of the instantaneous distance between the mouth and hand as the geometric center of the window, a sliding observation window is constructed by taking a preset number of frames forward and backward, and the mean distance on the left and the mean distance on the right of the geometric center of the window are calculated respectively.

[0018] The absolute value of the difference between the mean distance on the left and the mean distance on the right is traversed window by window, and the geometric center frame of the window to which the peak value of the absolute value of the difference belongs is taken as the position of the dramatic transformation.

[0019] Using the position of drastic change as the retrieval center, when the left half of the sliding observation window is completely below the low position and enters the marker, and the right half of the window is completely above the high position and leaves the marker, the verification is successful, and the moment of the drastic change position that has been verified is confirmed as the moment when the hand leaves the mouth.

[0020] In a preferred embodiment, the step of calculating the reflective specular pixel density of the image frames surrounding the mouth frame by frame includes:

[0021] The red, green, and blue primary color channel values ​​of pixels in the image frame around the mouth are mapped to hue angle values, saturation percentage values, and brightness percentage values;

[0022] Pixels with a saturation percentage lower than the average saturation value of the entire frame and a brightness percentage higher than the average brightness value of the entire frame are labeled as specular reflection pixels.

[0023] The number of specular reflection pixels is counted and totaled, and the normalized ratio of the total number of pixels to the total number of pixels in the image frame around the mouth is used as the instantaneous value of the reflective specular pixel density of the image frame around the mouth.

[0024] In a preferred embodiment, the baseline drift compensation of the reflected specular pixel density to obtain the temporal specular density sequence of the image frames around the mouth includes:

[0025] Using the moment when the target object's hand leaves its mouth as the time segmentation marker, the density of the reflected high-light pixels is divided into a reference segment before the marker and a response segment after the marker.

[0026] Within the reference section before the boundary marker, deploy a sliding analytical window with equal span, search for the local minimum value of the instantaneous value within the sliding analytical window, and then take the lower envelope baseline trajectory of the local minimum value as the reference contour.

[0027] Based on the reference contour, the instantaneous values ​​of the response segment after the landmark are stripped by difference to obtain the temporal high light density sequence of the image frames around the mouth.

[0028] In a preferred embodiment, the step of performing interpolation stripping on the instantaneous values ​​of the post-landmark response segment based on the reference contour to obtain the temporal hyperluminescence density sequence of the image frames surrounding the mouth includes:

[0029] At the same time phase, the difference between the original instantaneous value of the response segment after the landmark and the reference instantaneous value of the reference profile is used as the original stripping allowance.

[0030] Calculate the nonlinear enhancement factor, wherein the formula for calculating the nonlinear enhancement factor is:

[0031] ;

[0032] In the formula, For time phase The nonlinear enhancement factor at the location, For the current time phase index, The window width of the preset frame number window is set for the starting end of the response segment after the boundary marker. The frame sequence number within the preset frame number window at the starting end. The starting time phase of the response segment after the landmark. The sequence number of the last frame within the preset frame number window. Frame number The original instantaneous value at that point, Frame number The reference instantaneous value at that location, It is the regularization stability constant. For time phase The original instantaneous value at that point, For time phase The reference instantaneous value at that location, This is a function to find the maximum value.

[0033] Based on the nonlinear enhancement factor, the original stripping margin is reshaped by weighted amplitude to obtain the enhanced stripping instantaneous value of the original stripping margin.

[0034] The instantaneous enhanced stripping values ​​below zero are set to zero to obtain optimized instantaneous enhanced stripping values. The optimized instantaneous enhanced stripping values ​​are then arranged in a longitudinal series according to the original time order to obtain the temporal hyperluminescence density sequence of the image frames around the mouth.

[0035] In a preferred embodiment, the step of performing response integrity analysis on the time-series high optical density sequence to obtain the salivary secretion response identifier of the time-series high optical density sequence includes:

[0036] Starting from the frame after the moment the target object’s hand leaves its mouth, the search begins. Along the time-increasing direction, the positive difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame. The position of the first frame of a continuous positive difference frame segment is taken as the climbing start phase.

[0037] Based on the climbing start phase, the negative difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame along the time increment direction, and the position of the last frame before the continuous negative difference frame segment appears is taken as the peak phase.

[0038] Based on the climbing start phase and the peak phase, the phase-time integrity self-consistency verification of the time-series high optical density series is performed to obtain the salivary secretion response confirmation identifier of the time-series high optical density series.

[0039] In a preferred embodiment, the step of performing a phase-time integrity self-consistency check on the temporal high optical density series based on the rise initiation phase and the peak apex phase to obtain a salivary secretion response confirmation identifier for the temporal high optical density series includes:

[0040] Starting from the climbing start phase, the positive difference between the current frame amplitude and the previous frame amplitude is compared frame by frame along the time increment direction, and the first and last frames of the continuous positive difference frame segments are anchored as the climbing segment start boundary and climbing segment end boundary.

[0041] Starting from the peak phase, the negative difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame along the time-increasing direction, and the first and last frames of the continuous negative difference frame segments are anchored as the starting boundary and ending boundary of the attenuation segment.

[0042] When the start boundary of the climbing segment is earlier than the end boundary of the climbing segment, the end boundary of the climbing segment is not later than the start boundary of the decay segment, and the start boundary of the decay segment is earlier than the end boundary of the decay segment, a saliva secretion response confirmation identifier is generated for the time-series high optical density sequence.

[0043] In a preferred embodiment, the step of determining the temporal causal consistency of the medication event of the target subject based on the salivary secretion response identifier to obtain the behavioral identification result of the medication event includes:

[0044] The time of the hand leaving the mouth of the target object is compared with the time of the response peak carried in the saliva secretion response confirmation identifier to obtain a Boolean value of causal order in which the time of the hand leaving the mouth is earlier than the time of the response peak.

[0045] Based on a preset physiological time window, the time span between the moment when the hand is withdrawn from the mouth and the moment when the response peaks is checked for interval attribution, and a window matching Boolean value is obtained in which the time span falls within the physiological time window.

[0046] Logical conjunction and merging are performed on the causal order Boolean value and the window matching Boolean value. When both are true, a temporal causal pass result is generated, and the temporal causal pass result is used as the behavior identification result of the medication event.

[0047] To address the aforementioned problems, the present invention also provides a medication behavior recognition system based on temporal action detection, the system comprising:

[0048] The region positioning and interception module is used to locate the head region and upper limb region of a human body in a continuous video stream, and to intercept the image frames around the mouth within a continuous preset time interval in the continuous video stream.

[0049] The high light density statistics module is used to count the reflected high light pixel density of the image frames around the mouth frame by frame;

[0050] The baseline compensation module is used to perform baseline drift compensation on the reflected specular pixel density to obtain the temporal specular density sequence of the image frames around the mouth.

[0051] The response integrity parsing module is used to perform response integrity parsing on the time-series high optical density sequence to obtain the salivary secretion response identifier of the time-series high optical density sequence;

[0052] The temporal causality determination module is used to determine the temporal causality consistency of the medication event of the target object based on the salivary secretion response identifier, and obtain the behavioral identification result of the medication event.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. This method, by precisely locating the moment the hand is withdrawn from the mouth and extracting the temporal sequence of image frames around the mouth, transforms the spatial criterion for action triggering into a starting zero point on the time axis, establishing a causally correlated temporal benchmark for subsequent physiological response signal analysis. Based on this, the reflective highlight pixel density of the image frames around the mouth is statistically analyzed frame by frame, and a baseline drift compensation mechanism is used to differentially remove the slowly varying components of the illumination environment. This ensures that the extracted temporal highlight density series can truly reflect the dynamic evolution of specular reflection caused by saliva secretion, thereby effectively enhancing the purity and resolvability of the physiological feedback signal on the time series.

[0055] 2. This method further analyzes the response integrity of the temporal high-density sequence, using phase self-consistency verification between the start and end boundaries of the ascent and decay phases to ensure that the identified salivary secretion response has a complete waveform structure, avoiding contamination of the response identifier by non-physiological pulse interference. Simultaneously, the moment the hand leaves the mouth and the peak response are compared sequentially along a unified time axis, and the time span of the two is checked using a preset physiological time window. Through temporal causal consistency determination, a strict logical link between action triggering and physiological feedback is constructed, ensuring that the behavioral recognition results of the medication event possess temporal self-consistency and causal closure, significantly improving the reliability and robustness of the recognition results. Attached Figure Description

[0056] Figure 1This is a flowchart illustrating a method for recognizing medication behavior based on temporal action detection, provided in an embodiment of the present invention.

[0057] Figure 2 This is a functional block diagram of a medication behavior recognition system based on temporal action detection, provided in an embodiment of the present invention.

[0058] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0059] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0060] This application provides a method for recognizing medication use based on temporal action detection. The execution entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method for recognizing medication use based on temporal action detection can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0061] Reference Figure 1 The diagram shown is a flowchart illustrating a medication behavior recognition method based on temporal action detection according to an embodiment of the present invention. In this embodiment, the medication behavior recognition method based on temporal action detection includes:

[0062] S01. Locate the head and upper limb regions of the human body in the continuous video stream, and extract the image frames around the mouth within a continuous preset time interval in the continuous video stream.

[0063] In this embodiment of the invention, locating the human head region and upper limb region in a continuous video stream, and extracting image frames around the mouth within a continuous preset time interval in the continuous video stream, includes:

[0064] Histogram spatial clustering is performed on the facial skin color pixel cluster and the hand skin color pixel cluster of the continuous video stream according to the brightness gradient direction to obtain the facial region envelope and hand region envelope of the continuous video stream.

[0065] The state transition moment of the distance between the facial region envelope and the hand region envelope is captured instantaneously, and the state transition moment is taken as the moment when the hand is withdrawn from the mouth;

[0066] Taking the moment when the hand is withdrawn from the mouth as the starting zero point, the continuous video stream within a preset time interval after the starting zero point is extracted to obtain image frames around the mouth.

[0067] The instantaneous state transition moment of capturing the distance between the facial region envelope and the hand region envelope, and using this state transition moment as the moment when the hand is withdrawn from the mouth, includes:

[0068] The spatial Euclidean distance between the center points of the facial region envelope and the hand region envelope is quantified frame by frame to obtain the instantaneous value of the mouth-hand distance.

[0069] Using each frame of the instantaneous distance between the mouth and hand as the geometric center of the window, a sliding observation window is constructed by taking a preset number of frames forward and backward, and the mean distance on the left and the mean distance on the right of the geometric center of the window are calculated respectively.

[0070] The absolute value of the difference between the mean distance on the left and the mean distance on the right is traversed window by window, and the geometric center frame of the window to which the peak value of the absolute value of the difference belongs is taken as the position of the dramatic transformation.

[0071] Using the position of drastic change as the retrieval center, when the left half of the sliding observation window is completely below the low position and enters the marker, and the right half of the window is completely above the high position and leaves the marker, the verification is successful, and the moment of the drastic change position that has been verified is confirmed as the moment when the hand leaves the mouth.

[0072] Skin color attribute determination is performed on all pixels in each frame of the continuous video stream. Pixels matching the facial skin color value range are grouped into a facial skin color pixel cluster, and pixels matching the hand skin color value range are grouped into a hand skin color pixel cluster. For the spatial area occupied by the facial skin color pixel cluster, histogram spatial merging is performed according to the brightness gradient direction. The brightness gradient direction refers to the trend of pixel brightness values ​​in the horizontal and vertical directions. Histogram spatial merging involves counting the pixel distribution frequency along this trend and connecting adjacent high-density intervals into a closed contour. After this merging, the facial region envelope is obtained. Similarly, the spatial area occupied by the hand skin color pixel cluster is also merged according to the brightness gradient direction using histogram spatial merging to obtain the hand region envelope.

[0073] The facial region envelope has a geometric center point, which is obtained by taking the median value of the spatial coordinates of all pixels within the envelope. The hand region envelope also has a geometric center point. The spatial straight-line distance between the geometric center points of the facial and hand regions on the image plane is calculated frame by frame. The calculated distance values ​​are recorded as instantaneous mouth-hand distance values, with one instantaneous mouth-hand distance value corresponding to each frame. Using each frame as the geometric center of the window, a predetermined number of consecutive frames are taken forward as the left half of the window, and the same predetermined number of consecutive frames are taken backward as the right half. The left and right halves of the window together constitute the sliding observation window corresponding to that frame. For each sliding observation window, the arithmetic mean of the instantaneous mouth-hand distance values ​​carried by each frame within the left half of the window is calculated as the left distance mean. Then, the arithmetic mean of the instantaneous mouth-hand distance values ​​carried by each frame within the right half of the window is calculated as the right distance mean.

[0074] The absolute value of the difference between the mean distance on the left and the mean distance on the right is calculated window by window. After traversing all windows, a set of absolute difference values ​​is obtained, with each absolute difference value corresponding to a geometric center frame of a window. The peak value is located in the entire set of absolute difference values, and the geometric center frame of the window to which the peak value belongs is marked as the location of a sharp transition. Using the frame corresponding to the location of a sharp transition as the retrieval center, the sliding observation window containing it is divided into the left half and the right half of the window. The instantaneous mouth-hand distance values ​​of each frame in the left half and the right half of the window are read respectively. A low-position entry threshold is set as one distance threshold, and a high-position exit threshold is set as another distance threshold. The value of the high-position exit threshold is greater than the value of the low-position entry threshold. When the instantaneous mouth-hand distance values ​​of all frames in the left half of the window are lower than the low-position entry threshold, and the instantaneous mouth-hand distance values ​​of all frames in the right half of the window are higher than the high-position exit threshold, the location of a sharp transition is determined to have passed the verification. The frame time corresponding to the verified drastic transition position is confirmed as the time when the hand is withdrawn from the mouth.

[0075] Starting from the frame where the hand is withdrawn from the mouth, a continuous video stream segment of a preset duration is extracted along the time-increasing direction. This preset duration interval extends from the starting point to a predetermined end time. Within this preset duration interval, a local area image around the mouth is cropped from each frame. The cropping method is to use the mouth positioning point in the facial region envelope as the center and expand outwards by a fixed pixel range to obtain a rectangular cropping frame. Image frames around the mouth are extracted from each frame according to this cropping frame. All image frames around the mouth are arranged sequentially according to the original time order to form a sequence of image frames around the mouth. The spatial size of each frame in this sequence is consistent, and the time interval is the same as the frame interval of the original video stream. Each frame in the sequence of image frames around the mouth retains the original red, green, and blue primary color channel values ​​for subsequent frame-by-frame statistical analysis of reflective highlight pixel density. The starting frame of the sequence of image frames around the mouth is strictly aligned with the time when the hand is withdrawn from the mouth, ensuring that the time base of all subsequent time-series analyses is unified at the time when the hand is withdrawn from the mouth. After completing the above extraction, a complete image frame sequence is obtained for subsequent high reflectance density statistics.

[0076] The beneficial effect is that by spatially merging the histograms of facial and hand skin color pixel clusters according to the brightness gradient direction, the envelopes of the facial and hand regions can be accurately obtained, providing a stable spatial reference for subsequent frame-by-frame quantization of mouth-hand distance. The spatial straight-line distance between the geometric center points of the two envelopes is calculated frame-by-frame, and a sliding observation window is constructed. The absolute value of the difference between the mean distance on the left and right sides of the window is used to locate the location of abrupt transitions. This is further verified by combining the dual conditions that the left half of the window is entirely below the low-level entry marker and the right half is entirely above the high-level exit marker. This ensures that the determination of the moment when the hand leaves the mouth has temporal consistency with the abrupt change in spatial distance, effectively eliminating spurious transition interference caused by minor hand tremors or head rotation. Using the verified moment when the hand leaves the mouth as the starting zero point, the video stream within a subsequent preset time interval is extracted and the image frames around the mouth are cropped. This ensures that the time origin of the resulting image frame sequence around the mouth is strictly aligned with the moment when the action is completed, establishing a unified and accurate time reference for the subsequent timing statistics of high reflectivity, and avoiding the problem of misaligned physiological response signals caused by the positioning deviation of the action trigger moment.

[0077] S02. Calculate the reflective highlight pixel density of the image frames around the mouth frame by frame;

[0078] In this embodiment of the invention, the step of statistically analyzing the reflective specular pixel density of the image frames surrounding the mouth frame by frame includes:

[0079] The red, green, and blue primary color channel values ​​of pixels in the image frame around the mouth are mapped to hue angle values, saturation percentage values, and brightness percentage values;

[0080] Pixels with a saturation percentage lower than the average saturation value of the entire frame and a brightness percentage higher than the average brightness value of the entire frame are labeled as specular reflection pixels.

[0081] The number of specular reflection pixels is counted and totaled, and the normalized ratio of the total number of pixels to the total number of pixels in the image frame around the mouth is used as the instantaneous value of the reflective specular pixel density of the image frame around the mouth.

[0082] For each pixel in each frame of the image surrounding the mouth, the values ​​carried by its red, green, and blue primary color channels are read. The red, green, and blue channel values ​​are then combined in a transformation operation. This operation first extracts the maximum and minimum values ​​from the three channels and calculates the difference between them. Using the maximum and minimum values ​​and this difference, the hue angle value corresponding to the pixel is determined by identifying the channel type of the maximum value and the relative levels of the channel values. Simultaneously, the saturation percentage value corresponding to the pixel is obtained by dividing the difference between the maximum and minimum values ​​by the maximum value. Finally, the brightness percentage value corresponding to the pixel is obtained by dividing the maximum value by the maximum possible value achievable by the pixel's channel values. After this transformation, each pixel possesses all three attributes: hue angle value, saturation percentage, and brightness percentage.

[0083] For the currently processed image frame around the mouth, iterate through all pixels within the frame and read the saturation percentage of each pixel. Sum the saturation percentages of all pixels and divide by the total number of pixels in the frame to obtain the average saturation value for the entire frame. Then, iterate through all pixels within the frame again, reading the brightness percentage of each pixel. Sum the brightness percentages of all pixels and divide by the total number of pixels in the frame to obtain the average brightness value for the entire frame. The average saturation and brightness values ​​for the entire frame are calculated only for the pixel set of the current frame; they are calculated independently for different frames and do not borrow from each other.

[0084] For each pixel within a frame, the relationship between its saturation percentage and the average saturation of the entire frame, as well as the relationship between its brightness percentage and the average brightness of the entire frame, are examined. If a pixel's saturation percentage is lower than the average saturation of the entire frame, but its brightness percentage is higher than the average brightness of the entire frame, then that pixel is marked as a specular pixel. If a pixel's saturation percentage is not lower than the average saturation of the entire frame, or its brightness percentage is not higher than the average brightness of the entire frame, then that pixel is not marked as a specular pixel. All pixels marked as specular pixels in each frame constitute the set of specular pixels for that frame.

[0085] Count all pixels in the frame that have been marked as specular reflection pixels to obtain the total number of pixels. Read the total number of pixels in the frame. Divide the total number of pixels by the total number of pixels in the frame to obtain a normalized ratio. Use this normalized ratio as the instantaneous value of the specular reflection pixel density of the frame. This instantaneous value of the specular reflection pixel density represents the proportion of pixels in the current frame that meet the two conditions of saturation lower than the average of the entire frame and brightness higher than the average of the entire frame. Perform the above complete operation from mapping to marking to obtaining the normalized ratio for each frame of the image around the mouth, and obtain the corresponding instantaneous value of the specular reflection pixel density for each frame. All the image frames around the mouth correspond to a specular reflection pixel density instantaneous value according to the original time sequence, forming an instantaneous value sequence that corresponds one-to-one with the sequence of image frames around the mouth in time.

[0086] The beneficial effects are that the average saturation and brightness of each frame are independently calculated frame by frame, ensuring that the judgment criterion for each frame is entirely based on its own pixel distribution, eliminating crosstalk between different frames caused by changes in lighting or shooting angles. Using a saturation percentage lower than the average saturation and brightness percentage higher than the average brightness as the judgment criterion for specular reflection pixels, the visual attribute of specular reflection areas—lower color saturation and higher brightness—accurately separates the moist reflective area around the mouth caused by saliva secretion from ordinary skin areas and shadow areas. The count of specular reflection pixels is summed and normalized to the total number of pixels in the frame, ensuring the instantaneous values ​​of reflective highlight pixel density are comparable across frames, eliminating numerical fluctuations caused by differences in image content. Each frame independently obtains the corresponding instantaneous value of reflective highlight pixel density and arranges it in its original chronological order. The resulting instantaneous value sequence corresponds strictly one-to-one with the image frame sequence around the mouth in time, providing time-aligned input data for subsequent baseline drift compensation operations.

[0087] S03. Perform baseline drift compensation on the reflected specular pixel density to obtain the temporal specular density sequence of the image frames around the mouth;

[0088] In this embodiment of the invention, the step of performing baseline drift compensation on the reflected specular pixel density to obtain the temporal specular density sequence of the image frames around the mouth includes:

[0089] Using the moment when the target object's hand leaves its mouth as the time segmentation marker, the density of the reflected high-light pixels is divided into a reference segment before the marker and a response segment after the marker.

[0090] Within the reference section before the boundary marker, deploy a sliding analytical window with equal span, search for the local minimum value of the instantaneous value within the sliding analytical window, and then take the lower envelope baseline trajectory of the local minimum value as the reference contour.

[0091] Based on the reference contour, the instantaneous values ​​of the response segment after the landmark are stripped by difference to obtain the temporal high light density sequence of the image frames around the mouth.

[0092] The step of performing difference stripping on the instantaneous values ​​of the post-landmark response segment based on the reference contour to obtain the temporal hyperluminescence density sequence of the image frames around the mouth includes:

[0093] At the same time phase, the difference between the original instantaneous value of the response segment after the landmark and the reference instantaneous value of the reference profile is used as the original stripping allowance.

[0094] Calculate the nonlinear enhancement factor, wherein the formula for calculating the nonlinear enhancement factor is:

[0095] ;

[0096] In the formula, For time phase The nonlinear enhancement factor at the location, For the current time phase index, The window width of the preset frame number window is set for the starting end of the response segment after the boundary marker. The frame sequence number within the preset frame number window at the starting end. The starting time phase of the response segment after the landmark. The sequence number of the last frame within the preset frame number window. Frame number The original instantaneous value at that point, Frame number The reference instantaneous value at that location, It is the regularization stability constant. For time phase The original instantaneous value at that point, For time phase The reference instantaneous value at that location, This is a function to find the maximum value.

[0097] Based on the nonlinear enhancement factor, the original stripping margin is reshaped by weighted amplitude to obtain the enhanced stripping instantaneous value of the original stripping margin.

[0098] The instantaneous enhanced stripping values ​​below zero are set to zero to obtain optimized instantaneous enhanced stripping values. The optimized instantaneous enhanced stripping values ​​are then arranged in a longitudinal series according to the original time order to obtain the temporal hyperluminescence density sequence of the image frames around the mouth.

[0099] Using the moment when the hand leaves the mouth as the time segmentation marker, the instantaneous values ​​of the reflected specular pixel density of all frames before this moment are classified as the pre-mark reference segment, and the instantaneous values ​​of the reflected specular pixel density of all frames after this moment are classified as the post-mark response segment. The two segments are separated by the frame in which the hand leaves the mouth. Each frame in the pre-mark reference segment corresponds to an instantaneous value of reflected specular pixel density, and each frame in the post-mark response segment also corresponds to an instantaneous value of reflected specular pixel density. The two segments are continuous in time and do not overlap.

[0100] Within the reference section before the boundary marker, starting from the first frame of this section, sliding analytical windows of equal span are deployed along the time-increasing direction. Each window covers the same number of consecutive frames, and the starting frames of adjacent windows are separated by a fixed number of frames. This allows the windows to gradually move backward within the reference section before the boundary marker until they cover the entire section. For each deployed sliding analytical window, the instantaneous values ​​of the reflected specular pixel density carried by each frame within the window are traversed. The instantaneous value with the lowest value is found and recorded as the local minimum of that window. After all windows have been traversed, each window contributes a local minimum. These local minimums are arranged sequentially according to the window's chronological order on the time axis. Connecting the points corresponding to these local minimums forms a broken line that varies with the frame number. This broken line is used as the lower envelope baseline trajectory, and this lower envelope baseline trajectory is used as the reference contour. Each frame position of the reference contour has a reference instantaneous value, which is provided by the local minimum of the corresponding window in the reference section before the marker, and is obtained by extrapolating the lower envelope baseline trajectory along the time direction in the response section after the marker.

[0101] Based on the reference profile, a differential stripping operation is performed on each frame of the post-marker response segment. At the same time phase, i.e., at the same frame number position in the reference profile within the post-marker response segment, the original instantaneous value of that frame is read. This original instantaneous value is the instantaneous value of the reflected specular pixel density of that frame. Simultaneously, the reference instantaneous value of the reference profile at that frame position is read. The original instantaneous value is subtracted from the reference instantaneous value, and the difference is used as the original stripping margin for that frame. Each frame within the post-marker response segment receives its own original stripping margin.

[0102] For each frame within the post-marker response segment, the original stripping margin is further used to determine the corresponding nonlinear enhancement factor. The determination process for this nonlinear enhancement factor is as follows: All frames within a preset frame window at the beginning of the post-marker response segment are taken. For each frame within this window, the difference between its original instantaneous value and the reference instantaneous value is calculated, resulting in a sequence of differences for each frame within the window. The squares of all differences within the window are then averaged, and the square root of this average is taken to obtain a value reflecting the degree of fluctuation in the differences within the window. Simultaneously, for the currently processed frame, its original instantaneous value and the reference instantaneous value are read. The original instantaneous value is divided by the larger of the original instantaneous value and the reference instantaneous value, yielding a ratio. This ratio is then divided by the aforementioned value reflecting the degree of fluctuation in the differences within the window, along with a preset regularization stability constant, to obtain an intermediate quotient. The difference between the original instantaneous value and the reference instantaneous value of the current frame is multiplied by this intermediate quotient, and then multiplied by the aforementioned ratio to obtain the nonlinear enhancement factor for that frame. Each frame within the post-marker response segment independently completes the above determination process, and each frame obtains its corresponding nonlinear enhancement factor.

[0103] After obtaining the nonlinear enhancement factor for each frame, the original stripping margin of that frame is multiplied by the nonlinear enhancement factor to obtain the instantaneous enhanced stripping value of that frame. This multiplication operation reshapes the weighted amplitude of the original stripping margin of that frame. Within the post-marker response segment, the original stripping margin of each frame is multiplied by its corresponding nonlinear enhancement factor, and each frame obtains its corresponding instantaneous enhanced stripping value.

[0104] The instantaneous enhancement stripping value of each frame is numerically inspected. If the instantaneous enhancement stripping value of a frame is lower than zero, it is reset to zero. If the instantaneous enhancement stripping value of a frame is equal to or higher than zero, it remains unchanged. After this zeroing operation, the instantaneous enhancement stripping values ​​of all frames within the post-marker response segment are non-negative. These zeroed-value instantaneous enhancement stripping values ​​are the optimized instantaneous enhancement stripping values.

[0105] The optimized enhanced stripping instantaneous values ​​of all frames within the post-marker response segment are vertically concatenated from front to back according to their original chronological order. Specifically, the optimized enhanced stripping instantaneous value of the first frame is placed at the beginning of the sequence, the second frame at the second, and so on until the last frame. This concatenated numerical sequence is used as the temporal hyperspectral density sequence of the images around the mouth. Each value in this temporal hyperspectral density sequence corresponds one-to-one with each frame within the post-marker response segment in time. The starting position of this sequence is aligned with the first frame after the hand is removed from the mouth, and the length of this sequence is equal to the number of frames within the post-marker response segment. After completing all the above operations, the baseline-drift-compensated temporal hyperspectral density sequence is obtained.

[0106] The window width of the preset frame number window at the start of the post-marker response segment is a fixed frame number value, determined before operation and remaining constant throughout the processing. This preset frame number window covers several consecutive frames starting from the initial frame of the post-marker response segment. Each frame carries an original instantaneous value and a reference instantaneous value. The original instantaneous value comes from the instantaneous value of the reflected specular pixel density of the corresponding frame within the post-marker response segment, and the reference instantaneous value comes from the reference instantaneous value at the same frame number position in the reference reference contour. The regularization stabilization constant is a pre-set positive number, determined before operation and remaining constant throughout the processing. For a currently being processed frame, located within the post-marker response segment, the original instantaneous value is directly read from the instantaneous value of the reflected specular pixel density of that frame, and the reference instantaneous value is read from the frame number position in the reference reference contour. The larger of the original and reference instantaneous values ​​for the current frame is selected as the maximum value.

[0107] The window width is determined based on the video frame rate. When the video frame rate is 25 frames per second, the window width is an integer value between 10 and 20 frames. This range ensures that the window covers a time length of 0.4 to 0.8 seconds, which matches the typical duration of the rising edge of the saliva secretion response. The operator can select a fixed integer value within this range based on the actual frame rate, and the selected value will remain unchanged.

[0108] The value of the regularization stability constant is of the same magnitude as the instantaneous value of the reflective specular pixel density. This instantaneous value is the normalization ratio, which ranges from zero to one, and the square root of the mean of the difference is also within this range.

[0109] This formula is used to determine the nonlinear enhancement factor corresponding to each frame within the response segment after the boundary. The first operation in the formula reflects the direction and degree of deviation of the original instantaneous value of the current frame from the reference instantaneous value. The deviation is positive when the original instantaneous value of the current frame is higher than the reference instantaneous value, and negative when the original instantaneous value of the current frame is lower than the reference instantaneous value. The second operation is used for normalization. Its denominator is obtained by averaging the squared differences between the original instantaneous value and the reference instantaneous value of each frame within the preset number of frames at the beginning, and then taking the square root. This yields a value that measures the overall difference level within the window. This value is added to the regularization stability constant and used as the denominator. The larger the denominator, the more the overall enhancement amplitude is suppressed; the smaller the denominator, the more the overall enhancement amplitude is released. The third operation uses the proportion of the current frame's original instantaneous value in the total of the original instantaneous value and the reference instantaneous value as a weight. When the current frame's original instantaneous value is higher than the reference instantaneous value, this weight approaches one; when the current frame's original instantaneous value is lower than the reference instantaneous value, this weight is less than one, and the weight decreases as the reference instantaneous value is higher than the original instantaneous value. The product obtained after continuously multiplying the three operations is the nonlinear enhancement factor. This factor takes a value of zero when the original instantaneous value is equal to the reference instantaneous value, a positive value when the original instantaneous value is higher than the reference instantaneous value, and a negative value when the original instantaneous value is lower than the reference instantaneous value.

[0110] The nonlinear enhancement factor exhibits a specific trend as the relationship between the original instantaneous value and the reference instantaneous value of the current frame changes. When the original instantaneous value of the current frame is higher than the reference instantaneous value, the first operation is positive, and the weight given by the third operation is close to one. At this time, the nonlinear enhancement factor is positive overall, and the larger the difference between the original instantaneous value and the reference instantaneous value, the larger the positive value obtained by the first operation. At the same time, the denominator of the second operation is adjusted accordingly due to the change in the mean square of the difference within the window. The larger the denominator, the more the enhancement factor is suppressed; the smaller the denominator, the more the enhancement factor is amplified. Therefore, the direction of change of the enhancement factor with the difference in the region where the original instantaneous value exceeds the reference instantaneous value is determined by the relative changes in the numerator and denominator. When the original instantaneous value of the current frame is lower than the reference instantaneous value, the first operation is negative, and the weight given by the third operation is less than one and further decreases as the ratio of the reference instantaneous value to the original instantaneous value increases. At this time, the nonlinear enhancement factor is negative, and the absolute magnitude of the negative value is limited by both the negative magnitude of the difference and the degree of weight reduction. When the original instantaneous value of the current frame equals the reference instantaneous value, the first operation is zero, therefore the nonlinear enhancement factor is zero. This factor assigns a larger enhancement magnitude when the overall difference level within the window is low, and a smaller enhancement magnitude when the overall difference level within the window is high, thus achieving adaptive nonlinear weighting of the original stripping margin.

[0111] The beneficial effect is that the moment the hand leaves the mouth is used as a time segmentation marker, dividing the reflected highlight pixel density into a reference segment before the marker and a response segment after the marker. This separates baseline estimation and response analysis on the time axis, preventing them from interfering with each other. Within the reference segment before the marker, equal-span sliding analytical windows are deployed, and local minima of the instantaneous values ​​within each window are extracted. All local minima are connected to form the lower envelope baseline trajectory as a reference profile. This ensures that the reference profile accurately reflects the changing trend of the gradually varying components of the illumination environment on the time axis, rather than the mean or peak trajectory, thus guaranteeing that the subsequent difference stripping operation removes the true baseline drift component. The original stripping margin is obtained by subtracting the reference instantaneous value from the original instantaneous value at the same time phase, achieving numerical separation between the baseline drift component and the response signal. A nonlinear enhancement factor is independently determined for each frame, and the original stripping margin is reshaped with a weighted amplitude. This appropriately amplifies the stripping margin in the weak response segment while maintaining the original amplitude ratio in the strong response segment, enhancing the dynamic resolution of the response signal. After setting the instantaneous values ​​of the enhancement stripping below zero to zero, they are arranged in a longitudinal sequence according to the original time order to obtain a temporal high optical density sequence. Each value in this sequence corresponds one-to-one with each frame in the response segment after the boundary, providing a sequence of input values ​​with no negative values, no baseline drift and optimized amplitude for subsequent response integrity analysis.

[0112] S04. Perform response integrity analysis on the time-series high optical density sequence to obtain the salivary secretion response identifier of the time-series high optical density sequence;

[0113] In this embodiment of the invention, the step of performing response integrity analysis on the time-series high optical density sequence to obtain the salivary secretion response identifier of the time-series high optical density sequence includes:

[0114] Starting from the frame after the moment the target object’s hand leaves its mouth, the search begins. Along the time-increasing direction, the positive difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame. The position of the first frame of a continuous positive difference frame segment is taken as the climbing start phase.

[0115] Based on the climbing start phase, the negative difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame along the time increment direction, and the position of the last frame before the continuous negative difference frame segment appears is taken as the peak phase.

[0116] Based on the climbing start phase and the peak phase, the phase-time integrity self-consistency verification of the time-series high optical density series is performed to obtain the salivary secretion response confirmation identifier of the time-series high optical density series.

[0117] The step of performing a phase-time consistency verification on the temporal high optical density series based on the climbing initiation phase and the peak vertex phase, and obtaining a salivary secretion response confirmation identifier for the temporal high optical density series, includes:

[0118] Starting from the climbing start phase, the positive difference between the current frame amplitude and the previous frame amplitude is compared frame by frame along the time increment direction, and the first and last frames of the continuous positive difference frame segments are anchored as the climbing segment start boundary and climbing segment end boundary.

[0119] Starting from the peak phase, the negative difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame along the time-increasing direction, and the first and last frames of the continuous negative difference frame segments are anchored as the starting boundary and ending boundary of the attenuation segment.

[0120] When the start boundary of the climbing segment is earlier than the end boundary of the climbing segment, the end boundary of the climbing segment is not later than the start boundary of the decay segment, and the start boundary of the decay segment is earlier than the end boundary of the decay segment, a saliva secretion response confirmation identifier is generated for the time-series high optical density sequence.

[0121] The starting frame after the hand is removed from the mouth is used as the retrieval starting point. This starting frame is the first frame in the temporal hyperspectral density sequence after the hand is removed from the mouth. Starting from this starting frame, the amplitude carried by each frame in the temporal hyperspectral density sequence is read frame by frame in ascending time. For the currently processed frame, its amplitude is compared with the amplitude of the previous frame to determine whether the amplitude of the current frame is higher than that of the previous frame. The case where the amplitude of the current frame is higher than that of the previous frame is recorded as a positive difference. The above frame-by-frame comparison is continued in ascending time, and the positive difference of each frame is recorded. Frame segments in which multiple consecutive frames show positive differences are identified as continuous positive difference frame segments. The frame number of the first frame in the continuous positive difference frame segment is used as the climbing start phase.

[0122] Starting from the rising initiation phase, the amplitude of each frame in the temporal high optical density sequence is read sequentially along the time-increasing direction, beginning with the frame containing that phase. For the currently processed frame, its amplitude is compared with that of the previous frame to determine if the current frame's amplitude is lower than that of the previous frame. Cases where the current frame's amplitude is lower than the previous frame's amplitude are recorded as negative differences. This frame-by-frame comparison is continued along the time-increasing direction, recording the negative differences for each frame. Frame segments exhibiting multiple consecutive negative differences are identified as continuous negative difference frame segments. The starting position of the first continuous negative difference frame segment is located on the time axis, and the frame number of the last frame before the occurrence of this continuous negative difference frame segment is taken as the peak phase. This peak phase, located after the rising initiation phase, marks the turning point where the temporal high optical density sequence transitions from a continuous increase to a continuous decrease.

[0123] Starting from the initiation phase of the climb, the amplitude carried by each frame in the temporal hyperspectral density sequence is recursively read in ascending order of time, beginning from the frame containing that phase. For the currently processed frame, its amplitude is compared with that of the previous frame to determine whether the current frame's amplitude is higher than that of the previous frame. This frame-by-frame comparison is continued, and frames in which multiple consecutive frames show an amplitude higher than that of the previous frame are identified as continuous positive difference frame segments. The frame number of the first frame in this continuous positive difference frame segment is anchored as the start boundary of the climb segment, and the frame number of the last frame in this continuous positive difference frame segment is anchored as the end boundary of the climb segment.

[0124] Starting with the peak phase, the amplitude carried by each frame in the temporal hyperspectral density sequence is recursively read in ascending order of time, beginning from the frame containing that phase. For the currently processed frame, its amplitude is compared with that of the previous frame to determine if the current frame's amplitude is lower than that of the previous frame. This frame-by-frame comparison is continued, and frames in which multiple consecutive frames show an amplitude lower than that of the previous frame are identified as consecutive negative difference frame segments. The frame number of the first frame in this consecutive negative difference frame segment is anchored as the starting boundary of the attenuation segment, and the frame number of the last frame in this consecutive negative difference frame segment is anchored as the ending boundary of the attenuation segment.

[0125] The frame numbers of the start and end boundaries of the ascending segment are compared temporally to determine if the start boundary is earlier than the end boundary. The frame numbers of the end and end boundaries of the ascending segment are then compared temporally to determine if the end boundary is no later than the start boundary. Finally, the frame numbers of the start and end boundaries of the attenuation segment are compared temporally to determine if the start boundary is earlier than the end boundary. When all three temporal conditions are met, a saliva secretion response confirmation identifier for the temporal high optical density sequence is generated. This identifier carries four phase position information: the start and end boundaries of the ascending and attenuating segments, indicating a complete waveform structure of first rising and then falling within the temporal high optical density sequence. After these operations, a saliva secretion response identifier for subsequent temporal causality consistency determination is obtained. This identifier includes the ascending start phase, the peak phase, and the saliva secretion response confirmation identifier.

[0126] The beneficial effect is that, starting from the frame after the hand is removed from the mouth, the positive difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame. This makes the positioning of the climbing start phase dependent on the continuity of the monotonically increasing relationship between adjacent frames, eliminating the positioning deviation caused by a fixed threshold. Starting from the climbing start phase, negative differences are compared frame by frame. The last frame before the first consecutive negative difference frame segment is taken as the peak phase, so that this phase is accurately anchored at the unique turning point between the end of climbing and the beginning of decay. Starting from the climbing start phase and the peak phase respectively, the positive and negative difference amounts are recursively calculated frame by frame to anchor the four boundary positions: the start boundary of the climbing segment, the end boundary of the climbing segment, the start boundary of the decay segment, and the end boundary of the decay segment. By jointly verifying three timing conditions—the start boundary of the climbing segment being earlier than the end boundary of the climbing segment, the end boundary of the climbing segment not later than the start boundary of the decay segment, and the start boundary of the decay segment being earlier than the end boundary of the decay segment—the phase timing integrity self-consistency verification of the waveform structure is performed. Only when all three conditions are met is a salivary secretion response confirmation identifier generated. This identifier carries complete boundary location information, indicating that there is a complete waveform structure of rising and falling in the time series of high optical density, providing complete input information for subsequent time series causal consistency determination.

[0127] S05. Based on the salivary secretion response identifier, perform a temporal causal consistency determination on the medication event of the target object to obtain the behavioral identification result of the medication event;

[0128] In this embodiment of the invention, the step of determining the temporal causal consistency of the medication event of the target object based on the salivary secretion response identifier to obtain the behavioral identification result of the medication event includes:

[0129] The time of the hand leaving the mouth of the target object is compared with the time of the response peak carried in the saliva secretion response confirmation identifier to obtain a Boolean value of causal order in which the time of the hand leaving the mouth is earlier than the time of the response peak.

[0130] Based on a preset physiological time window, the time span between the moment when the hand is withdrawn from the mouth and the moment when the response peaks is checked for interval attribution, and a window matching Boolean value is obtained in which the time span falls within the physiological time window.

[0131] Logical conjunction and merging are performed on the causal order Boolean value and the window matching Boolean value. When both are true, a temporal causal pass result is generated, and the temporal causal pass result is used as the behavior identification result of the medication event.

[0132] Extract the peak response time from the salivary secretion response confirmation marker; this peak response time is the frame time corresponding to the peak phase. Compare the hand withdrawal time from the mouth with this peak response time on the timeline to determine if the hand withdrawal time is earlier than the peak response time. If the hand withdrawal time is earlier than the peak response time on the timeline, assign a true Boolean value to the causal order; otherwise, assign a false Boolean value.

[0133] Read a pre-defined physiological time window, which has an earliest boundary time and a latest boundary time, covering the entire time range from the earliest to the latest boundary. Calculate the time span between the moment the hand is withdrawn from the mouth and the peak response time. This time span is equal to the peak response time minus the moment the hand is withdrawn from the mouth. Perform an interval assignment check on this time span and the earliest and latest boundary times of the physiological time window. Determine if the time span is greater than or equal to the earliest boundary time and less than or equal to the latest boundary time. If the time span falls within the coverage of the physiological time window, set the window matching Boolean value to true; otherwise, set it to false.

[0134] A logical conjunction and merging process is performed on the causal order Boolean value and the window matching Boolean value. This involves simultaneously reading the true / false status of both Boolean values. If both the causal order Boolean value and the window matching Boolean value are true, the conjunction and merging result is considered true. If either the causal order Boolean value or the window matching Boolean value is false, or both are false, the conjunction and merging result is considered false. If the conjunction and merging result is true, a temporal causal pass result is generated; if the result is false, no temporal causal pass result is generated. This temporal causal pass result is used as the behavioral identification result of the medication event. This behavioral identification result indicates that the target object's medication event has passed the dual temporal causal consistency test, where the hand withdrawal from the mouth occurs earlier than the peak time of salivary secretion response, and the time span of both occurs within a preset physiological time window.

[0135] The beneficial effect is that by comparing the time when the hand is withdrawn from the mouth with the peak time of the response in the salivary secretion response confirmation marker on a timeline, a Boolean value representing the causal order of the action preceding the response is obtained. Based on a preset physiological time window, an interval check is performed on the time span between the two times to obtain a window-matching Boolean value representing the span falling within the window. The two Boolean values ​​are logically combined; only when both are true is a temporal causal pass result generated, ensuring that the output of the behavior recognition result simultaneously satisfies the dual conditions of correct causal direction and reasonable span. This pass result is directly used as the behavior recognition result for the medication event. Its determination simultaneously covers the temporal direction constraint that the action time is earlier than the peak response time and the duration constraint that the time span of the two is within the physiologically feasible range, thus ensuring that the recognition result has causal self-consistency and physiological rationality in temporal logic, improving the reliability of medication event recognition.

[0136] like Figure 2 The diagram shown is a functional block diagram of a medication behavior recognition system based on time-series action detection provided in an embodiment of the present invention.

[0137] The medication behavior recognition system 10 based on temporal action detection described in this invention can be installed in an electronic device. Depending on the functions implemented, the medication behavior recognition system 10 may include a region localization and interception module 11, a high optical density statistics module 12, a baseline compensation module 13, a response integrity analysis module 14, and a temporal causality determination module 15. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, and are stored in the memory of the electronic device.

[0138] In this embodiment, the functions of each module / unit are as follows:

[0139] The region positioning and interception module 11 is used to locate the human head region and upper limb region in the continuous video stream, and to intercept the image frames around the mouth within a continuous preset time interval in the continuous video stream.

[0140] The high light density statistics module 12 is used to count the reflected high light pixel density of the image frames around the mouth frame by frame.

[0141] The baseline compensation module 13 is used to perform baseline drift compensation on the reflected specular pixel density to obtain the temporal specular density sequence of the image frames around the mouth.

[0142] The response integrity parsing module 14 is used to perform response integrity parsing on the time-series high optical density sequence to obtain the salivary secretion response identifier of the time-series high optical density sequence.

[0143] The temporal causality determination module 15 is used to determine the temporal causality consistency of the medication event of the target object based on the saliva secretion response identifier, and obtain the behavioral identification result of the medication event.

[0144] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0145] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0146] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0147] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0148] This application embodiment can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for recognizing medication-taking behavior based on temporal action detection, characterized in that, The method includes: S01. Locate the head and upper limb regions of the human body in the continuous video stream, and extract the image frames around the mouth within a continuous preset time interval in the continuous video stream. S02. Calculate the reflective highlight pixel density of the image frames around the mouth frame by frame; S03. Perform baseline drift compensation on the reflected specular pixel density to obtain a temporal specular density sequence of the image frames around the mouth, including: Using the moment when the target object's hand leaves its mouth as the time segmentation marker, the density of the reflected high-light pixels is divided into a reference segment before the marker and a response segment after the marker. Within the reference section before the boundary marker, deploy a sliding analytical window with equal span, search for the local minimum value of the instantaneous value within the sliding analytical window, and then take the lower envelope baseline trajectory of the local minimum value as the reference contour. Based on the reference contour, the instantaneous values ​​of the response segment after the landmark are stripped by difference to obtain the temporal high light density sequence of the image frames around the mouth. S04. Perform response integrity analysis on the time-series high optical density sequence to obtain the salivary secretion response identifier of the time-series high optical density sequence, including: Starting from the frame after the moment the target object’s hand leaves its mouth, the search begins. Along the time-increasing direction, the positive difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame. The position of the first frame of a continuous positive difference frame segment is taken as the climbing start phase. Based on the climbing start phase, the negative difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame along the time increment direction, and the position of the last frame before the continuous negative difference frame segment appears is taken as the peak phase. Based on the climbing start phase and the peak phase, the phase-time integrity self-consistency verification of the time-series high optical density series is performed to obtain the salivary secretion response confirmation identifier of the time-series high optical density series. S05. Based on the salivary secretion response identifier, perform a temporal causal consistency determination on the medication event of the target object to obtain the behavioral identification result of the medication event.

2. The method for recognizing medication behavior based on temporal action detection as described in claim 1, characterized in that, The process of locating the human head and upper limb regions in a continuous video stream and extracting image frames around the mouth within a continuous preset time interval from the continuous video stream includes: Histogram spatial clustering is performed on the facial skin color pixel cluster and the hand skin color pixel cluster of the continuous video stream according to the brightness gradient direction to obtain the facial region envelope and hand region envelope of the continuous video stream. The state transition moment of the distance between the facial region envelope and the hand region envelope is captured instantaneously, and the state transition moment is taken as the moment when the hand is withdrawn from the mouth; Taking the moment when the hand is withdrawn from the mouth as the starting zero point, the continuous video stream within a preset time interval after the starting zero point is extracted to obtain image frames around the mouth.

3. The method for recognizing medication behavior based on temporal action detection as described in claim 2, characterized in that, The instantaneous state transition moment of capturing the distance between the facial region envelope and the hand region envelope, and using this state transition moment as the moment when the hand is withdrawn from the mouth, includes: The spatial Euclidean distance between the center points of the facial region envelope and the hand region envelope is quantified frame by frame to obtain the instantaneous value of the mouth-hand distance. Using each frame of the instantaneous distance between the mouth and hand as the geometric center of the window, a sliding observation window is constructed by taking a preset number of frames forward and backward, and the mean distance on the left and the mean distance on the right of the geometric center of the window are calculated respectively. The absolute value of the difference between the mean distance on the left and the mean distance on the right is traversed window by window, and the geometric center frame of the window to which the peak value of the absolute value of the difference belongs is taken as the position of the dramatic transformation. Using the location of the dramatic transition as the retrieval center, when the entire left half of the sliding observation window is below the low-level entry marker and the entire right half of the window is above the high-level exit marker, the verification is successful, and the moment of the dramatic transition that has been verified is confirmed as the moment when the hand leaves the mouth. Here, the low-level entry marker is set as one distance threshold, and the high-level exit marker is set as another distance threshold. The value of the high-level exit marker is greater than the value of the low-level entry marker.

4. The method for recognizing medication behavior based on temporal action detection as described in claim 1, characterized in that, The step of calculating the reflective highlight pixel density of the image frames surrounding the mouth frame by frame includes: The red, green, and blue primary color channel values ​​of pixels in the image frame around the mouth are mapped to hue angle values, saturation percentage values, and brightness percentage values; Pixels with a saturation percentage lower than the average saturation of the entire frame and a brightness percentage higher than the average brightness of the entire frame are labeled as specular reflection pixels. The number of specular reflection pixels is counted and totaled, and the normalized ratio of the total number of pixels to the total number of pixels in the image frame around the mouth is used as the instantaneous value of the reflective specular pixel density of the image frame around the mouth.

5. The method for recognizing medication behavior based on temporal action detection as described in claim 1, characterized in that, The step of performing difference stripping on the instantaneous values ​​of the post-landmark response segment based on the reference contour to obtain the temporal hyperluminescence density sequence of the image frames around the mouth includes: At the same time phase, the difference between the original instantaneous value of the response segment after the landmark and the reference instantaneous value of the reference profile is used as the original stripping allowance. Calculate the nonlinear enhancement factor, wherein the formula for calculating the nonlinear enhancement factor is: ； In the formula, For time phase The nonlinear enhancement factor at the location, For the current time phase index, The window width of the preset frame number window is set for the starting end of the response segment after the boundary marker. The frame sequence number within the preset frame number window at the starting end. The starting time phase of the response segment after the landmark. The sequence number of the last frame within the preset frame number window. Frame number The original instantaneous value at that point, Frame number The reference instantaneous value at that location, It is the regularization stability constant. For time phase The original instantaneous value at that point, For time phase The reference instantaneous value at that location, This is a function to find the maximum value. Based on the nonlinear enhancement factor, the original stripping margin is reshaped by weighted amplitude to obtain the enhanced stripping instantaneous value of the original stripping margin. The instantaneous enhanced stripping values ​​below zero are set to zero to obtain optimized instantaneous enhanced stripping values. The optimized instantaneous enhanced stripping values ​​are then arranged in a longitudinal series according to the original time order to obtain the temporal hyperluminescence density sequence of the image frames around the mouth.

6. The method for recognizing medication behavior based on temporal action detection as described in claim 1, characterized in that, The step of performing a phase-time consistency verification on the temporal high optical density series based on the climbing initiation phase and the peak vertex phase, and obtaining a salivary secretion response confirmation identifier for the temporal high optical density series, includes: Starting from the climbing start phase, the positive difference between the current frame amplitude and the previous frame amplitude is compared frame by frame along the time increment direction, and the first and last frames of the continuous positive difference frame segments are anchored as the climbing segment start boundary and climbing segment end boundary. Starting from the peak phase, the negative difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame along the time-increasing direction, and the first and last frames of the continuous negative difference frame segments are anchored as the starting boundary and ending boundary of the attenuation segment. When the start boundary of the climbing segment is earlier than the end boundary of the climbing segment, the end boundary of the climbing segment is not later than the start boundary of the decay segment, and the start boundary of the decay segment is earlier than the end boundary of the decay segment, a saliva secretion response confirmation identifier is generated for the time-series high optical density sequence.

7. The method for recognizing medication behavior based on temporal action detection as described in claim 1, characterized in that, The step of determining the temporal causal consistency of the medication event based on the salivary secretion response identifier to obtain the behavioral identification result of the medication event includes: The time of the hand leaving the mouth of the target object is compared with the time of the response peak carried in the saliva secretion response confirmation identifier to obtain a Boolean value of causal order in which the time of the hand leaving the mouth is earlier than the time of the response peak. Based on a preset physiological time window, the time span between the moment when the hand is withdrawn from the mouth and the moment when the response peaks is checked for interval attribution, and a window matching Boolean value is obtained in which the time span falls within the physiological time window. Logical conjunction and merging are performed on the causal order Boolean value and the window matching Boolean value. When both are true, a temporal causal pass result is generated, and the temporal causal pass result is used as the behavior identification result of the medication event.

8. A medication behavior recognition system based on temporal action detection, characterized in that, The system is used to implement the medication behavior recognition method based on temporal action detection as described in claim 1, the system comprising: The region positioning and interception module is used to locate the head region and upper limb region of a human body in a continuous video stream, and to intercept the image frames around the mouth within a continuous preset time interval in the continuous video stream. The high light density statistics module is used to count the reflected high light pixel density of the image frames around the mouth frame by frame; The baseline compensation module is used to perform baseline drift compensation on the reflected specular pixel density to obtain a temporal specular density sequence of the image frames around the mouth, including: Using the moment when the target object's hand leaves its mouth as the time segmentation marker, the density of the reflected high-light pixels is divided into a reference segment before the marker and a response segment after the marker. Within the reference section before the boundary marker, deploy a sliding analytical window with equal span, search for the local minimum value of the instantaneous value within the sliding analytical window, and then take the lower envelope baseline trajectory of the local minimum value as the reference contour. Based on the reference contour, the instantaneous values ​​of the response segment after the landmark are stripped by difference to obtain the temporal high light density sequence of the image frames around the mouth. A response integrity parsing module is used to perform response integrity parsing on the time-series high optical density series to obtain the salivary secretion response identifier of the time-series high optical density series, including: Starting from the frame after the moment the target object’s hand leaves its mouth, the search begins. Along the time-increasing direction, the positive difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame. The position of the first frame of a continuous positive difference frame segment is taken as the climbing start phase. Based on the climbing start phase, the negative difference between the amplitude of the current frame and the amplitude of the previous frame is compared frame by frame along the time increment direction, and the position of the last frame before the continuous negative difference frame segment appears is taken as the peak phase. Based on the climbing start phase and the peak phase, the phase-time integrity self-consistency verification of the time-series high optical density series is performed to obtain the salivary secretion response confirmation identifier of the time-series high optical density series. The temporal causality determination module is used to determine the temporal causality consistency of the medication event of the target object based on the salivary secretion response identifier, and obtain the behavioral identification result of the medication event.

Images