An eating and drinking frequency automatic counting method and system based on first-person video

Abstract

The application discloses a kind of based on first person video's eating and drinking frequency automatic counting method and system, belong to computer vision and intelligent wearing technical field, comprising: the collected data is input into sliding window type frame buffer queue, and the candidate video segment of suspected eating or drinking is screened out;Candidate video segment is handled to generate video tensor, and it is input into timing feature extraction network, and output frame-by-frame space-time feature sequence;Through action boundary detection head, frame-by-frame space-time feature sequence is predicted, and time boundary is determined;Global feature vector is obtained based on frame-by-frame space-time feature sequence, and it is input into action classification head to output the classification result of each action instance;The classification result of each action instance is processed by deduplication and merging;Eating class and drinking class action after deduplication and merging are respectively accumulated and counted, and the eating and drinking frequency is obtained.The application realizes classification counting, does not need additional sensor hardware assistance, realizes the distinction of eating action and drinking action.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and smart wearable technology, specifically relating to a method and system for automatically counting the frequency of eating and drinking based on first-person video. Background Technology

[0002] Currently, the automated monitoring of dietary behavior mainly employs the following technical approaches:

[0003] (1) Inertial Measurement Unit (IMU) based scheme. Represented by the Bite Counter from Clemson University, this scheme uses an accelerometer worn on the wrist to detect wrist twisting motions, identify and count "hand-to-mouth" movement patterns. This scheme has the following shortcomings: it cannot distinguish between "eating" and "drinking" actions, and can only count them; the accuracy drops significantly for eating methods with small wrist rotation amplitudes, such as using a spoon or straw; the interval between two actions needs to be more than 8 seconds to be identified separately, resulting in serious missed detections for rapid and continuous eating; and it cannot eliminate false alarms caused by non-eating "hand-to-mouth" actions such as scratching the face or wiping the mouth.

[0004] (2) Solutions based on smartwatch cameras. For example, Medibotics' patent "US9254099B2 - Smart watch and food-imaging member for monitoring food consumption" published on November 27, 2014, uses a smartwatch's integrated camera to capture images of food. This solution focuses on food type identification and does not involve counting the frequency of eating and drinking.

[0005] (3) Neck / ear sensor-based solution. This solution detects chewing sounds or swallowing vibrations by attaching sensors to the throat or ear. While this solution can detect chewing events, it is not comfortable to wear, cannot distinguish between eating and drinking, and is not suitable for long-term wear for 24 hours.

[0006] In summary, existing technologies share the following common problems: 1. They cannot simultaneously distinguish between eating and drinking, with most solutions treating the two as one or focusing on only one; 2. They rely on additional sensors, requiring additional hardware such as wrist IMUs and throat patches, increasing system complexity; 3. Their counting accuracy is insufficient, making them prone to missing detections of rapid, continuous, and small-amplitude movements; 4. They cannot obtain visual context: pure sensor solutions cannot identify handheld objects (tableware or cups), lacking auxiliary judgment criteria. Summary of the Invention

[0007] To address the problems of existing technologies that cannot simultaneously distinguish between eating and drinking, rely on additional sensors, have insufficient counting accuracy, and lack visual context, this invention provides an automatic counting method for eating and drinking frequencies based on first-person video. It utilizes a first-person perspective video stream captured by a wearable camera, obtains a frame-by-frame spatiotemporal feature sequence through a temporal feature extraction network, and then outputs the classification result for each action instance through an action boundary detection head and an action classification head. Finally, after deduplication and merging processing based on temporal overlap and interval, the eating and drinking action instances are cumulatively counted, achieving accurate classification and counting without the need for additional sensor hardware assistance. This method distinguishes between eating and drinking actions and can identify handheld objects, thus solving the problem of lacking visual context.

[0008] According to one aspect of the present invention, an automatic counting method for the frequency of eating and drinking based on first-person video is provided, comprising:

[0009] S1. Input the collected first-person video data into a sliding window frame buffer queue, and filter out candidate video segments that are suspected of eating or drinking through joint judgment.

[0010] S2. After preprocessing, the candidate video segments are input into the temporal feature extraction network, which outputs a frame-by-frame spatiotemporal feature sequence.

[0011] S3. Based on the frame-by-frame spatiotemporal feature sequence, the temporal boundary of each action instance is determined by the action boundary detection head;

[0012] S4. After compressing the frame-by-frame spatiotemporal feature sequence in the temporal dimension, input it into the action classification head, and output the classification result of each action instance in combination with the temporal boundary of each action instance.

[0013] S5. Perform deduplication and merging processing on the classification results of each action instance based on time overlap and interval;

[0014] S6. Count the cumulative counts of the eating and drinking action instances after deduplication and merging to obtain the eating and drinking frequencies.

[0015] Furthermore, through joint judgment, candidate video clips suspected of involving eating or drinking were selected, including:

[0016] Inter-frame difference: Calculate the pixel difference between adjacent frames in the sliding window frame buffer queue. If the area of ​​the pixel difference exceeds the preset proportion threshold of the video frame, it is determined that there is activity.

[0017] Coarse detection of the hand region: Detect the presence of a hand or a held object in the lower half of the video frame using an existing lightweight model;

[0018] When both inter-frame difference and coarse detection of the hand region meet the conditions, candidate video segments suspected of eating or drinking are selected.

[0019] Further, S3 includes:

[0020] The action boundary detection head predicts each frame of the frame-by-frame spatiotemporal feature sequence: a one-dimensional temporal convolution is applied to the frame-by-frame feature sequence to output the three probabilities of action start, action end, and background for each frame.

[0021] Peak detection is performed on the start and end probabilities of an action, and the nearest peak pair is matched to determine the time boundary of each action instance.

[0022] Furthermore, after determining the time boundary of each action instance, action instances with abnormal durations are filtered out by the time boundary; where duration refers to the action end time minus the action start time.

[0023] Further, S4 includes:

[0024] The global feature vector is obtained by performing global average pooling on the frame-by-frame spatiotemporal feature sequence. The global feature vector is then input into the action classification head, and the classification result of each action instance is output by combining the temporal boundary of each action instance.

[0025] Furthermore, the classification results for each action instance undergo deduplication and merging based on temporal overlap and interval, including:

[0026] For similar action instances where the overlap rate of time intervals exceeds a preset threshold, select action instances with higher confidence.

[0027] Merge similar action instances with a time interval less than a preset merging threshold into one action;

[0028] For eating-related actions, if multiple very short intervals of action are detected consecutively, they are considered as continuous chewing in one action and are counted as one.

[0029] Furthermore, the screening in S1 runs on the wearable device's low-power processor. When no candidate video clips suspected of eating or drinking are detected, the device enters a low-power standby mode, maintaining only the inter-frame difference.

[0030] Furthermore, when a candidate video segment suspected of being related to eating or drinking is detected, the candidate video segment is uploaded to a cloud server for processing.

[0031] Furthermore, based on the joint judgment results, the wearable device dynamically switches between low-power standby mode and active upload mode; when no suspected activity is detected, the device is in low-power standby mode and only maintains inter-frame difference operation; when suspected activity is detected, the device switches to active upload mode and uploads candidate video clips to the cloud.

[0032] According to one aspect of the present invention, an automatic counting system for the frequency of eating and drinking based on first-person video is provided, comprising:

[0033] Wearable acquisition device is used to pre-screen the acquired first-person video stream in real time and output the selected candidate video clips that are suspected of eating or drinking.

[0034] The cloud processing terminal communicates with the wearable acquisition terminal to perform temporal action detection and classification on the received candidate video segments, so as to identify and count the eating action instances and drinking action instances, and generate the eating and drinking frequencies.

[0035] Compared with the prior art, the beneficial effects of the present invention are:

[0036] 1. This invention obtains a frame-by-frame spatiotemporal feature sequence through a temporal feature extraction network, and then outputs the classification result of each action instance through an action boundary detection head and an action classification head, thereby distinguishing between eating and drinking actions. Existing IMU solutions and existing throat sensor solutions cannot distinguish between them.

[0037] 2. This invention achieves short-interval detection by continuously acquiring first-person video data and performing subsequent processing, supporting 0.3 seconds. In contrast, existing IMU solutions require an 8-second interval for detection and have poor adaptability to faster movements. Existing throat sensor solutions can only partially detect rapid and continuous movements.

[0038] 3. This invention performs deduplication and merging processing on the classification results of each action instance based on time overlap and interval, and performs cumulative counting. No additional sensors are required. Existing IMU solutions require wearing an IMU on the wrist, and existing throat sensor solutions require a patch on the throat.

[0039] 4. This invention uses object recognition to assist in filtering false alarms, while existing IMU solutions rely solely on motion features, and existing throat sensor solutions rely solely on vibration features. Furthermore, this invention estimates an F1-score of 90%+; existing IMU solutions have an F1-score of 75-80%, and existing throat sensor solutions have an F1-score of 70-80%. This invention also offers high wearing comfort, compared to medium comfort for existing IMU solutions and low comfort for existing throat sensor solutions. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart of an embodiment of the present invention for an automatic counting method of eating and drinking frequency based on first-person video. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] like Figure 1 As shown in the figure, this invention proposes an automatic counting method for the frequency of eating and drinking based on first-person video, including: S1, inputting the collected first-person video data into a sliding window frame buffer queue, and filtering out candidate video segments suspected of eating or drinking through joint judgment; S2, inputting the candidate video segments into a temporal feature extraction network after preprocessing, and outputting a frame-by-frame spatiotemporal feature sequence; S3, determining the temporal boundary of each action instance based on the frame-by-frame spatiotemporal feature sequence through an action boundary detection head; S4, inputting the frame-by-frame spatiotemporal feature sequence into an action classification head after temporal dimension compression, and outputting the classification result of each action instance in combination with the temporal boundary of each action instance; S5, performing deduplication and merging processing on the classification result of each action instance based on temporal overlap and interval; S6, cumulatively counting the deduplicated and merged eating and drinking action instances respectively to obtain the frequency of eating and drinking.

[0044] Specifically, this invention provides the following content regarding first-person video stream acquisition and frame buffering: A wearable camera continuously acquires a first-person perspective video stream at a set frame rate (e.g., 15-30fps). A sliding window-style frame buffer queue is maintained on the device (wearable camera), with a queue length of N frames (e.g., N=64, corresponding to approximately 2-4 seconds of video). The frame buffer queue employs a first-in, first-out (FIFO) strategy and is continuously updated.

[0045] Specifically, this invention provides the content of end-side pre-screening—active area detection:

[0046] To reduce power consumption and bandwidth usage, a lightweight pre-screening module runs on the device. This module determines whether the current video footage contains any suspected eating / drinking activity based on the following strategies:

[0047] Inter-frame difference method: Calculate the pixel difference between adjacent frames. If the difference area exceeds the preset proportion threshold θ1 of the image (such as 15%), it is determined that there is significant motion.

[0048] Coarse detection of the hand region: Use a lightweight model (such as MobileNet-SSD) to detect the presence of a hand or a held object in the lower half of the image.

[0049] Comprehensive judgment: When the inter-frame difference and coarse detection of the hand region meet the conditions at the same time, it is judged as a candidate video segment suspected of eating or drinking, and subsequent processing is triggered.

[0050] By using end-side pre-screening—active area detection, approximately 70%-80% of inactive periods can be filtered out, significantly reducing the cloud processing load.

[0051] Specifically, this invention provides a method for extracting candidate video segments: extracting the most recent T frames (e.g., T=48 frames, approximately 1.6-3.2 seconds) from the frame buffer queue as a candidate video segment. To ensure the integrity of the action, the extraction range is extended forward by T_pre frames (e.g., 8 frames) and backward by T_post frames (e.g., 8 frames), for a total extraction of T_pre+T+T_post frames.

[0052] Specifically, the present invention provides a method for uniform temporal sampling and preprocessing of candidate video segments suspected of being eaten or drunk, specifically as follows:

[0053] The input is an original video segment consisting of frames from the T_pre, T, and T_post stages. Uniform sampling is performed on the original segment, selecting 16 frames. These 16 frames are then spatially resized to 256×256. The resized content is then center-cropped to 224×224. Pixel normalization is performed on the cropped content using mean / variance normalization. The final output is a fixed-dimensional video tensor with dimensions of 16×3×224×224. The sampling strategy is uniform temporal sampling, extracting 16 frames at equal intervals from all frames to ensure the model input covers the complete action time span.

[0054] Specifically, this invention provides the content of temporal feature extraction:

[0055] The preprocessed video tensor is input into a temporal feature extraction network, which employs a video understanding backbone network (such as an architecture based on Video Transformer or 3D convolutional networks) to extract spatiotemporal features. The processing includes:

[0056] Input phase: Receive video data with dimensions of 16 frames × 3 (RGB channels) × 244 (height) × 224 (width) (16 frames represent a continuous frame sequence in the time dimension, 3 is the number of image channels, and 244 × 224 is the spatial resolution of a single frame image).

[0057] Patch Embedding: Each frame of the input video is divided into multiple image patches, and each patch is embedded into a low-dimensional vector through linear transformation and other methods to complete the mapping from "image patch to feature vector".

[0058] Perform positional encoding: Add spatial positional encoding (capturing the positional information of image patches within a single frame) and temporal positional encoding (capturing the temporal positional information between different frames) to the features, enabling the model to perceive the spatial and temporal relative positions of the features.

[0059] Input spatiotemporal attention block (repeated L layers): By stacking L layers of "spatiotemporal attention" modules, attention calculations are performed on both the spatial dimension (feature distribution within a single frame) and the temporal dimension (feature associations between multiple frames) to mine the dependencies in the spatiotemporal dimension and further refine the features.

[0060] Output frame-by-frame spatiotemporal feature sequence: After the above processing, a feature sequence with a shape of 16×D is obtained (16 corresponds to the number of frames in the time dimension, and D is the dimension of the feature in each frame).

[0061] Based on frame-by-frame spatiotemporal feature sequences, the process is divided into two branches:

[0062] (1) Action boundary detection head: Directly utilizes frame-by-frame feature sequences to predict whether each frame is an action boundary (start frame or end frame). The specific steps are as follows:

[0063] Apply a one-dimensional temporal convolution to the frame-by-frame feature sequence to output three probabilities for each frame: P_start (action start), P_end (action end), and P_background (background).

[0064] Peak detection is performed on the probability sequences P_start and P_end. The nearest start-end peak pairs are matched to determine the time boundary [t_start, t_end] of each action instance.

[0065] Filter out abnormal detection results with durations that are too short (<0.3 seconds) or too long (>5 seconds). The duration is t_end - t_start. The filtering rules are as follows: if the duration is <0.3 seconds, filter (too short, possibly detection noise, such as a hand quickly swiping across the screen); if the duration is >5.0 seconds, filter (too long, does not conform to the normal duration of a single bite of food / drink, possibly due to boundary detection errors causing multiple actions to be combined into one).

[0066] (2) Global average pooling + action classification head: First, aggregate all frame features in the time dimension through "global average pooling" to obtain a global feature vector of shape D (compressing the time dimension while preserving the overall semantics); then input the global feature vector into the action classification head to complete the classification of the overall action category of the video (requires global semantic support).

[0067] Specifically, this invention provides a method for classifying action instances:

[0068] Features corresponding to the time boundaries of each action instance are extracted and input into the action classification head for classification, resulting in three-class classification information, as shown in Table 1. Among them, eating actions exhibit a reciprocating motion pattern, while drinking actions exhibit a single arc motion pattern.

[0069] Table 1. Three-category information

[0070] Specifically, the three-class classification requires three feature dimensions: spatiotemporal motion features, handheld object features, and action temporal patterns. Spatiotemporal motion features are directly extracted by the temporal feature extraction network, encoding the trajectory and speed information of hand movements. After determining the action boundary, the feature subsequence of the frame corresponding to the action boundary is pooled into a D-dimensional vector, which encodes the overall motion semantics of the action segment. Handheld object features are obtained by performing object detection on keyframes of the action segment using a newly added object detection submodule, identifying the category of the handheld object (tableware, cup, bottle, food, etc.). This object detection submodule is independent and separate from the temporal feature extraction network. Action temporal patterns are obtained through newly added signal analysis, analyzing the periodicity of the hand region motion vector within the action segment. Eating actions typically exhibit a "reciprocating" pattern (the hand repeatedly moves between the mouth and the plate), while drinking actions exhibit a "single arc" pattern.

[0071] Specifically, embodiments of the present invention provide content on action deduplication and merging:

[0072] Due to the sliding window mechanism, the same actual action may be detected repeatedly in adjacent candidate segments. This step performs deduplication and merging using the following rules:

[0073] Rule 1 (Time Overlap Deduplication): If the time interval overlap rate (IoU) of two instances of the same type of action exceeds θ_iou (e.g., 0.5), the one with the higher confidence level is retained.

[0074] Rule 2 (Interval Merging): If the interval between two similar actions is less than θ_merge (e.g., 1.5 seconds) and they belong to the same category, they are merged into one action and counted as one.

[0075] Rule 3 (Continuous Chewing Filter): For the eating category, if multiple consecutive very short intervals (<0.5 seconds) of action segments are detected, they are considered as continuous chewing in a single "picking up food and putting it to the mouth" action and are counted as one instance.

[0076] Specifically, embodiments of the present invention provide the content of count accumulation and log recording:

[0077] For similar action instances with an overlap rate exceeding a preset threshold, those with higher confidence are retained; for similar action instances with a time interval less than a preset merging threshold, they are merged into a single action. The deduplicated and merged action list is cumulatively counted, and a structured log is generated. The encoding is as follows:

[0078] Plain Text Output data structure: { "session_id": "2026-02-05_12:30:00", "eating_count": 42, / / Frequency of eating "drinking_count": 8, / / Frequency of drinking water "events": [ { "type": "eating", "timestamp": "12:31:05", "confidence": 0.94, "duration_sec": 1.2 }, { "type": "drinking", "timestamp": "12:33:12", "confidence": 0.89, "duration_sec": 2.1 }, ... ], "meal_duration_sec": 960, / / Total meal duration "avg_bite_interval_sec": 22.8 / / Average interval per bit }

[0079] Specifically, the embodiments of the present invention also provide a terminology description, as shown in Table 2.

[0080] Table 2 Terminology Explanation

[0081] Based on the above embodiments, the present invention provides an automatic counting system for the frequency of eating and drinking based on first-person video. This system is used to execute the automatic counting method for the frequency of eating and drinking based on first-person video in the above method embodiments.

[0082] The system includes: a wearable acquisition terminal, used to pre-screen the acquired first-person video stream in real time and output the selected candidate video segments that are suspected of eating or drinking; and a cloud processing terminal, which communicates with the wearable acquisition terminal and is used to perform temporal action detection and classification on the received candidate video segments to identify and count the eating and drinking action instances and generate the eating and drinking frequencies.

[0083] The automatic counting system for eating and drinking frequency based on first-person video provided in this invention addresses the problems of existing technologies, such as the inability to simultaneously distinguish between eating and drinking, reliance on additional sensors, insufficient counting accuracy, and inability to obtain visual context. It employs several modules, utilizing a first-person perspective video stream captured by a wearable camera. A temporal feature extraction network is used to obtain a frame-by-frame spatiotemporal feature sequence. Then, an action boundary detection head and an action classification head output the classification result for each action instance. Finally, after deduplication and merging processing based on temporal overlap and interval, the system cumulatively counts eating and drinking action instances, achieving accurate classification and counting without the need for additional sensor hardware assistance.

[0084] Specifically, this invention also provides a combined application of a necklace-style camera and cloud computing. A magnetically attached camera module is attached to a necklace carrier, with the camera facing upwards and forwards to capture a first-person perspective view from the chest up to the face. The device continuously captures a video stream at 15fps, and an edge-side pre-screening module runs inter-frame differential detection on a low-power processor. When a hand is detected entering the upper half of the frame, 64 candidate frames are extracted and uploaded to the user's mobile phone via Bluetooth / Wi-Fi, and then forwarded by the phone to the cloud server. A complete temporal action detection and classification pipeline runs in the cloud, and the identified eating and drinking frequencies are pushed to the user via an app.

[0085] Specifically, embodiments of the present invention also provide a combined application of glasses-style cameras and cloud computing. By wearing smart glasses with an integrated miniature camera located on the side of the glasses frame, a first-person view is captured, essentially aligned with the user's line of sight. Since the glasses' perspective allows direct observation of the interaction between the hand and mouth, the action features are more pronounced from this viewpoint. The cloud model performs specialized parameter adaptation for the glasses' perspective, assigning higher weight to the "hand-to-mouth" distance feature during the classification stage.

[0086] In summary, this invention provides an automatic counting method for the frequency of eating and drinking based on first-person video. It continuously acquires first-person view video streams using a wearable camera and maintains a sliding window frame buffer queue. The video frames in the frame buffer queue are pre-screened at the edge, and candidate video segments suspected of eating or drinking are selected based on a joint judgment of inter-frame difference and coarse detection of the hand region. The candidate video segments are then subjected to uniform temporal sampling and spatial preprocessing to generate a fixed-dimensional video tensor. This video tensor is input into a temporal feature extraction network to extract frame-by-frame spatiotemporal feature sequences and a global feature vector. Based on the frame-by-frame spatiotemporal feature sequences, an action boundary detection head predicts the start and end probabilities of actions in each frame, determining the temporal boundary of each action instance. Based on the global feature vector and the features within the corresponding time range of the action instance, an action classification head classifies each action instance into three categories: "eating," "drinking," or "irrelevant actions." The classified action instances are then deduplicated and merged based on temporal overlap and interval. Finally, the deduplicated and merged eating and drinking action instances are cumulatively counted, and the eating and drinking frequencies are output.

[0087] Finally, it should be noted that the above specific embodiments are merely representative examples of the present invention. Obviously, the present invention is not limited to the above specific embodiments and many variations are possible. Any simple modifications, equivalent changes, and alterations made to the above specific embodiments based on the technical essence of the present invention should be considered within the protection scope of the present invention.

Claims

1. A method for automatically counting the frequency of eating and drinking based on first-person video, characterized in that, include: S1. Input the collected first-person video data into a sliding window frame buffer queue, and filter out candidate video segments that are suspected of eating or drinking through joint judgment. S2. After preprocessing, the candidate video segments are input into the temporal feature extraction network, which outputs a frame-by-frame spatiotemporal feature sequence. S3. Based on the frame-by-frame spatiotemporal feature sequence, the temporal boundary of each action instance is determined by the action boundary detection head; S4. After compressing the frame-by-frame spatiotemporal feature sequence in the temporal dimension, input it into the action classification head, and output the classification result of each action instance in combination with the temporal boundary of each action instance. S5. Perform deduplication and merging processing on the classification results of each action instance based on time overlap and interval; S6. Count the cumulative counts of the eating and drinking action instances after deduplication and merging to obtain the eating and drinking frequencies.

2. The method for automatically counting the frequency of eating and drinking based on first-person video according to claim 1, characterized in that, Candidate video clips suspected of involving eating or drinking were identified through joint judgment, including: Inter-frame difference: Calculate the pixel difference between adjacent frames in the sliding window frame buffer queue. If the area of ​​the pixel difference exceeds the preset proportion threshold of the video frame, it is determined that there is significant activity. Coarse detection of the hand region: Detect the presence of a hand or a held object in the lower half of the video frame using an existing lightweight model; When both inter-frame difference and coarse detection of the hand region meet the conditions, candidate video segments suspected of eating or drinking are selected.

3. The method for automatically counting the frequency of eating and drinking based on first-person video according to claim 1, characterized in that, The S3 includes: The action boundary detection head predicts each frame of the frame-by-frame spatiotemporal feature sequence: a one-dimensional temporal convolution is applied to the frame-by-frame feature sequence to output the three probabilities of action start, action end, and background for each frame. Peak detection is performed on the start and end probabilities of an action, and the nearest peak pair is matched to determine the time boundary of each action instance.

4. The method for automatically counting the frequency of eating and drinking based on first-person video according to claim 3, characterized in that, After determining the time boundary of each action instance, action instances with abnormal durations are filtered out by the time boundary; where duration refers to the action end time minus the action start time.

5. The method for automatically counting the frequency of eating and drinking based on first-person video according to claim 1, characterized in that, The S4 includes: The global feature vector is obtained by performing global average pooling on the frame-by-frame spatiotemporal feature sequence. The global feature vector is then input into the action classification head, and the classification result of each action instance is output by combining the temporal boundary of each action instance.

6. The method for automatically counting the frequency of eating and drinking based on first-person video according to claim 1, characterized in that, The classification results for each action instance are subjected to deduplication and merging based on time overlap and interval, including: For similar action instances where the overlap rate of time intervals exceeds a preset threshold, select action instances with higher confidence. Merge similar action instances with a time interval less than a preset merging threshold into one action; For eating-related actions, if multiple very short intervals of action are detected consecutively, they are considered as continuous chewing in one action and are counted as one.

7. The method for automatically counting the frequency of eating and drinking based on first-person video according to claim 2, characterized in that, The filtering in S1 runs on the wearable device’s low-power processor. When no candidate video clips suspected of eating or drinking are detected, the device enters a low-power standby mode, maintaining only the inter-frame difference.

8. The method for automatically counting the frequency of eating and drinking based on first-person video according to claim 7, characterized in that, When a candidate video clip that appears to indicate eating or drinking is detected, the candidate video clip is uploaded to a cloud server for processing.

9. The method for automatically counting the frequency of eating and drinking based on first-person video according to claim 2, characterized in that, Based on the joint judgment results, the wearable device dynamically switches between low-power standby mode and active upload mode; when no suspected activity is detected, the device is in low-power standby mode and only maintains inter-frame differential operation. When suspected activity is detected, the device switches to active upload mode and uploads candidate video clips to the cloud.

10. An automatic counting system for the frequency of eating and drinking based on first-person video, characterized in that, include: Wearable acquisition device is used to pre-screen the acquired first-person video stream in real time and output the selected candidate video clips that are suspected of eating or drinking. The cloud processing terminal communicates with the wearable acquisition terminal to perform temporal action detection and classification on the received candidate video segments, so as to identify and count the eating action instances and drinking action instances, and generate the eating and drinking frequencies.

Images