A label-driven long-term intent recognition method and system for online learning
Patent Information
- Application Number
- CN202611395655.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-09
- Publication Date
- 2026-11-03
Smart Images
Figure CN123033204A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and educational informatization technology, and relates to a label-driven long-term intent recognition method and system for online learning. Background Technology
[0002] In online education, the physical separation of teachers and students makes it difficult for teachers to monitor students' long-term learning status (e.g., focus or lack of focus) in real time. Students' long-term learning intentions are a temporal combination of short-term learning behaviors over a period of time, forming an important basis for evaluating the quality of online teaching. However, existing video temporal behavior analysis and intention inference methods are significantly inadequate for this scenario.
[0003] Existing technologies typically employ processing methods based on single-frame feature extraction, frame-by-frame temporal combination, and global feature aggregation. These methods first independently extract features and recognize actions from each frame of the video, then concatenate the recognition results in chronological order, and finally aggregate them using pooling or recurrent neural networks to generate a global feature vector for long-term intent classification and inference. This end-to-end video temporal reasoning model is effective for action recognition tasks with relatively fixed behavioral patterns, but its direct application to online learning scenarios faces numerous challenges.
[0004] However, in practical applications, existing technologies mainly suffer from the following problems: First, learning behaviors are non-stationary and fluctuate. For example, behaviors such as "listening to the lecture" and "lying on the table" alternate without a fixed pattern, making temporal modeling difficult. Frame-by-frame modeling is easily affected by short-term noise, and single-frame errors are propagated and accumulated during the temporal aggregation process. Second, the feature space is not uniform in multi-view scenarios. The randomness of camera placement and posture in online learning leads to the shift in feature distribution between consecutive frames. Existing technologies cannot effectively eliminate the impact of this shift, resulting in poor robustness of long-term intention inference. Finally, existing models lack interpretability, only outputting the final classification label and failing to provide traceable judgment criteria. Teachers cannot understand the support for the model's decisions, making it difficult to apply them in actual teaching scenarios. Summary of the Invention
[0005] The purpose of this invention is to provide a label-driven long-term intent recognition method and system for online learning, in order to solve the above-mentioned problems.
[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a label-driven long-term intent recognition method for online learning, comprising the following steps: Acquire video data and extract multiple frames of images at a preset frame rate; identify each frame of image and extract behavior and pose labels; Based on behavior labels and pose labels, multiple frames of images are grouped, consecutive frames with consistent labels are merged, and short-term noise segments are filtered out to generate multiple behavior segments. A relationship graph is constructed using each behavior segment as a node. The edges of the relationship graph include temporally adjacent edges that represent the temporal sequence relationship and semantically related edges that represent the semantic consistency of the behavior. The node features are aggregated through a graph convolutional network to generate global temporal features of the video. The system outputs long-term intent classification results based on global temporal features of the video, and calculates dynamic engagement scores based on the behavior weight, pose coefficient, duration, and total number of frames of each behavior segment.
[0007] Furthermore, the behavior tags include multiple categories such as reading, taking notes, listening to lectures, typing, playing on a mobile phone, playing on a tablet, getting up, leaning on a table, and daydreaming; the posture tags include multiple categories such as upright, leaning forward, resting chin on hand, tilting to the side, leaning back, standing, and lying down.
[0008] Furthermore, the process of merging consecutive frames with consistent tags and filtering out short-term noise segments to generate multiple behavioral segments specifically involves: Traverse each frame and merge consecutive frames in which the behavior label and the pose label are consistent into a candidate segment; when the duration of the candidate segment is less than the preset minimum segment threshold, merge the candidate segment into the previous stable segment; otherwise, treat it as an independent behavior segment.
[0009] Furthermore, after generating the behavior segments, feature fusion is performed on the frame-level features within each behavior segment to generate segment-level features of fixed dimensions, specifically as follows: The features of each frame within the behavior segment are mapped to a preset dimension via linear projection; dynamic position encoding based on the position index within the segment is added to each frame; the features with added position encoding are input into a single-layer Transformer encoder for feature fusion; the features output by the Transformer encoder are globally averaged along the frame dimension to obtain the segment-level features of the fixed dimension.
[0010] Furthermore, the dynamic position encoding adopts sinusoidal position encoding, and its calculation formula is as follows:
[0011]
[0012] in, This is the position index of the frame within the segment. Indexed by feature dimensions, For preset dimensions.
[0013] Furthermore, the aggregation of node features through a graph convolutional network is implemented using a two-layer relational graph convolutional network, and its node feature update formula is as follows:
[0014] in, For the first The node feature matrix of the layer, initial input It consists of fragment-level features of all behavioral segments; To incorporate the adjacency matrix of self-loops, The adjacency matrix represents the temporally adjacent edges and the semantically related edges; for The corresponding degree matrix; The weight matrix is a learnable matrix; It is a non-linear activation function.
[0015] Furthermore, the calculation of the dynamic participation score specifically involves: Set an initial engagement score; for each behavioral segment, determine whether its corresponding learning state weight is positive; if so, update the dynamic engagement score using a positive accumulation method; if not, update the dynamic engagement score using a negative decay method; wherein, the learning state weight is obtained by multiplying the behavioral weight by the posture coefficient, and the calculation of the positive accumulation and negative decay are both related to the duration of the behavioral segment and the total number of video frames.
[0016] Furthermore, the formula for calculating the positive accumulation is as follows:
[0017] The formula for calculating the negative attenuation is:
[0018] in, and The first Engagement scores at the beginning and end of each behavioral segment. For the first The learning state weights of each segment, For the first The number of frames per segment. This represents the total number of frames in the video. This is the growth rate adjustment coefficient.
[0019] Secondly, the present invention provides a label-driven long-term intent recognition system for online learning, comprising: The data acquisition module is used to acquire video data, extract multiple frames of images at a preset frame rate, and identify each frame of image to extract behavior labels and pose labels. The behavior fragment generation module is used to group multiple frames of images based on behavior labels and pose labels, merge consecutive frames with consistent labels, filter short-term noise fragments, and generate multiple behavior fragments. The global temporal feature generation module is used to construct a relationship graph with each behavior segment as a node. The edges of the relationship graph include temporally adjacent edges that represent temporal sequence relationships and semantically related edges that represent semantic consistency of behaviors. The node features are aggregated through a graph convolutional network to generate global temporal features of the video. The output module is used to output long-term intent classification results based on global temporal features of the video, and to calculate dynamic engagement scores based on the behavior weight, pose coefficient, duration and total number of frames of each behavior segment.
[0020] Thirdly, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the label-driven long-term intent recognition method for online learning.
[0021] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the label-driven long-term intent recognition method for online learning.
[0022] Compared with the prior art, the present invention has the following technical effects: This invention extracts behavior and pose labels frame by frame, providing semantically consistent and explicit operational objects for subsequent grouping, avoiding uninterpretable divisions of raw pixels or abstract features. By dynamically grouping based on behavior and pose labels, merging consecutive frames with consistent labels, and filtering short-term noise fragments, it transforms original non-stationary, noisy frame sequences into semantically homogeneous and stable behavior fragments, effectively solving the problem of interference from irregular behavior switching on temporal modeling in online learning. By constructing a relationship graph containing both temporally adjacent edges and semantically related edges using behavior fragments as nodes, and aggregating node features using a graph convolutional network, it can simultaneously capture the sequential order and long-range semantic consistency between fragments. This not only models the dynamic evolution of behavior but also strengthens the intrinsic connection of the same behavior semantics at different time points, significantly improving feature robustness across perspectives and the accuracy of long-term intent inference.
[0023] By classifying based on global temporal features of videos and combining behavioral weights, posture coefficients, and duration to calculate dynamic engagement scores, the output is no longer just an isolated classification label, but a quantifiable and traceable dynamic process of learning engagement, meeting the core requirement of interpretability in educational scenarios. In summary, through the synergistic effect of the above steps, this invention achieves accurate, robust, and interpretable reasoning about students' long-term intentions in online learning scenarios, providing reliable technical support for online education learning analysis and teaching intervention. Attached Figure Description
[0024] Figure 1 This is a logic block diagram of a tag-driven long-term intent recognition method for online learning in one embodiment of the present invention.
[0025] Figure 2 This is a schematic diagram of the adaptive dynamic Transformer module in one embodiment of the present invention.
[0026] Figure 3 This is a schematic diagram of the structure of a relational graph convolutional network in one embodiment of the present invention.
[0027] Figure 4 This is an example diagram illustrating the visualization of dynamic participation scoring in one embodiment of the present invention.
[0028] Figure 5 This is a flowchart of the present invention. Detailed Implementation
[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0031] The present invention will now be described in further detail with reference to the accompanying drawings: See Figure 1 In one embodiment of the present invention, a label-driven long-term intent recognition method for online learning is provided to address the problems of insufficient accuracy and robustness of long-term intent recognition caused by non-stationary fluctuations in learning behavior and inconsistencies in multi-perspective features in online learning scenarios, as well as the lack of interpretability of recognition results required for educational scenarios.
[0032] Specifically, the label-driven long-term intent recognition method for online learning of this invention includes the following steps: S1: Acquire video data, extract multiple frames of images at a preset frame rate, identify each frame of image, and extract behavior and pose labels.
[0033] S2: Based on behavior labels and pose labels, group multiple frames of images, merge consecutive frames with consistent labels, filter short-term noise segments, and generate multiple behavior segments.
[0034] S3: Construct a relation graph using each behavior segment as a node. The edges of the relation graph include temporally adjacent edges that represent the temporal sequence relationship and semantically related edges that represent the semantic consistency of the behavior. Aggregate node features through a graph convolutional network to generate global temporal features of the video.
[0035] S4: Output long-term intent classification results based on global temporal features of the video, and calculate dynamic engagement score based on the behavior weight, pose coefficient, duration and total number of frames of each behavior segment.
[0036] Step S1 is used to acquire and standardize input data from the raw video stream, providing a structured analysis object with semantic information for the subsequent entire process. First, continuous video data from the online learning scenario is acquired. This video data is typically captured by the camera on the learner's terminal device, with a uniform resolution of 1280×720 pixels. Then, the video is processed by frame extraction at a preset frame rate, generating an image sequence composed of multiple consecutive images. The preset frame rate is designed to balance the sufficiency of temporal information with the consumption of computational resources, capturing dynamic changes in behavior while avoiding data redundancy. Finally, each frame in the sequence is recognized by a trained short-term intent recognition model. This short-term intent recognition model is a deep learning model that integrates multimodal features such as skeleton, face, and hands, and possesses viewpoint invariance. The recognition output of each frame includes a global fusion feature vector and its corresponding behavior label and pose label. The behavior label describes the learning-related action the student is performing in that frame, and the pose label describes the student's body posture in that frame. Step S1 transforms unstructured video data into a semantic label stream consisting of behavior label sequences and pose label sequences. This transformation enables subsequent processing to be performed directly at the semantic level, rather than in the ambiguous pixel or low-level feature space that is susceptible to noise interference. This is the foundation for the interpretability and robustness of the entire method.
[0037] Given the application context of online education, students' learning states are primarily manifested as a series of observable behaviors. These behaviors may switch frequently within a short period, lacking a strict sequential logic, and their timing and duration are unpredictable. For example, a student might be in a "listening" state for a few minutes, then briefly operate their phone before putting it down and resuming the "listening" state. Frame-by-frame behavior recognition results contain a large number of such short-term switching and instantaneous recognition errors. If long-term intent analysis is performed directly based on frame-by-frame results, these noises and fluctuations will severely interfere with the accuracy of temporal modeling. Step S2 dynamically groups the behavior label sequence and posture label sequence generated in step S1. Its core purpose is not simply smoothing filtering, but to extract semantically and behaviorally uniform "homogeneous behavior fragments" from non-stationary and fluctuating temporal data. It forms candidate fragments by merging adjacent, identically labeled consecutive frames, aggregating continuous and consistent behaviors into a meaningful atomic unit. Simultaneously, by filtering out noise fragments with excessively short durations and merging them into previously stable fragments, it effectively suppresses fragmentation caused by instantaneous misidentification or irrelevant minor actions. The original long and unstable frame sequence is compressed and integrated into a smaller sequence of behavioral segments with clearer semantics and more stable behavior. Each behavioral segment has a clear start frame, end frame, consistent behavior label and pose label, thus transforming the non-stationary temporal modeling problem into a modeling problem of the relationship between homogeneous segments, which greatly reduces the difficulty and uncertainty of temporal relationship modeling in subsequent steps.
[0038] The core task of step S3 is to globally model the multiple behavioral segments generated in step S2 to generate global temporal features that can represent the long-term intent of the entire video. This step first creatively treats each behavioral segment as an independent node, thus constructing a relationship graph. This relationship graph is not a simple linear sequence graph; it contains two key edge types: temporally adjacent edges, which establish directed connections between temporally adjacent segments, representing the sequential evolution of behaviors; and semantically related edges, which establish undirected connections between segments with the same behavioral label, representing the inherent semantic consistency between the same behaviors occurring at different points in the video. Then, step S3 uses a graph convolutional network to perform feature propagation and aggregation on this relationship graph. The graph convolutional network can pass the information of each segment node to its neighboring nodes and semantically related nodes along the edges in the graph. After multiple layers of propagation, the features of each node are integrated with its local context and global semantic information. Finally, pooling operations are used to aggregate all updated node features, generating a fixed-dimensional global temporal feature for the video. Compared to traditional methods that directly encode frame-by-frame features or segments in a simple temporal sequence, this video global temporal feature modeled based on relational graph convolutional networks integrates the temporal order of behavior and semantic structure. It is more robust to local feature changes caused by factors such as viewpoint changes and occlusion, and can more robustly represent the long-term learning intent embodied in the entire video.
[0039] Step S4 inputs the global temporal features of the video generated in Step S3 into a classifier, outputting a clear long-term intent classification result, such as "very focused" or "not focused," providing teachers with a direct overall evaluation. However, more importantly, classification results alone are far from sufficient for decision-making interventions like those in education. Step S4 introduces a dynamic engagement scoring mechanism. The core idea of this mechanism is that students' learning engagement is not static but dynamically changes over time and is interpretable. It accumulates and calculates the behavioral weights, posture coefficients, and durations of each behavioral segment according to a pre-defined mathematical model, ultimately generating a continuously changing dynamic engagement score sequence across the entire video timeline. Positive behaviors (such as listening attentively) have a positive cumulative effect on engagement based on their duration, while negative behaviors (such as using a mobile phone) have a negative decay effect. This scoring sequence can be visualized as a dynamically fluctuating curve, which itself is the most intuitive and traceable basis for judging the long-term intent classification result. Teachers can not only see the final conclusion of "focus" or "lack of focus", but also know through the curve which time period of the video the students are highly engaged and which time period the attention drops, as well as the specific behaviors that cause this change. This perfectly meets the core explainability requirements of teachers in the educational context: understandability, intervention capability, and traceability of results.
[0040] This invention presents a label-driven long-term intent recognition method for online learning. Through the interconnected and synergistic effects of steps S1 to S4, a complete and efficient technical loop is formed. Step S1 provides a semantic input foundation, elevating the problem from the pixel space to the label space. Based on this, step S2 creatively transforms unstable frame-by-frame data into stable homogeneous behavioral fragments through label-driven dynamic grouping, effectively filtering temporal noise and paving the way for stable modeling. Step S3 constructs an information propagation graph structure that combines temporal and semantic relationships, and utilizes the powerful relational modeling capabilities of graph convolutional networks to aggregate discrete behavioral fragments into video global temporal features rich in context and global semantics, overcoming the interference of perspective changes. Finally, step S4 not only outputs accurate classifications but also provides a crucial interpretability link by calculating dynamic engagement scores based on fragment attributes. The entire method, from data processing, feature engineering, model building to output application, is designed to address the pain points of existing technologies at each stage, progressing step by step to achieve accurate, cross-perspective robust, and natively interpretable reasoning for students' long-term intent in online learning scenarios.
[0041] Example 2: This invention provides a tag-driven long-term intent recognition method for online learning, comprising: 1) Input a 10-second video from an online learning scenario. The video resolution is uniformly 1280×720. Use ffmpeg to extract frames at 5 frames / second to generate a continuous image of T=50 frames, denoted as T. .
[0042] 2) Call the trained short-term intent recognition model and process the image frame by frame: Output 512-dimensional global fusion features It integrates skeletal, facial, and hand features and possesses viewpoint invariance; Output behavior labels 0-8 correspond to reading, taking notes, listening to lectures, typing, using a mobile phone, using a tablet, getting up, slumping over a table, and daydreaming. Output attitude labels 0-6 correspond to upright, leaning forward, resting chin on hand, tilting to the side, leaning back, standing, and lying down.
[0043] Generate feature sequence F and behavior label sequence Attitude label sequence Enter the subsequent module.
[0044] 3) Set the sliding window step size to 1 and the minimum segment threshold. Execute the grouping algorithm: The algorithm first traverses the label sequence frame by frame. If two adjacent frames have the same behavior label and the number of consecutive frames with the same label is greater than the minimum segment threshold τ_min, then they are merged into the same behavior segment.
[0045] Iterate through frames 2-50. If the current frame label matches the current label, continue iterating. When the label changes, the duration of the current segment is calculated. If the segment length reaches the set threshold τ_min, it is considered a valid new behavior segment and stored, recording the segment's behavior category, pose category, start and end time range, and feature set. For segments shorter than τ_min, they are considered random noise caused by recognition instability. These features are merged into the previous stable segment, and the end time of the previous segment is updated. This eliminates unstable jumps and ensures that subsequent Transformer processing maintains semantically consistent temporal features.
[0046] 4) For the feature set of the i-th behavior segment First, the features are mapped to the input dimension of the Transformer through a linear projection layer. The projected feature matrix is obtained. Add dynamic sine position coding to each frame within a segment, and recount within each segment. This ensures that the same relative position of segments of different lengths generates a consistent code, and the encoding formulas are shown in Formula 4-1 and Formula 4-2.
[0047]
[0048]
[0049] In the formula: For the position index within the segment where a certain frame is located, Indexed by feature dimensions.
[0050] The positional encoding is added to the projected features to obtain the final Transformer input features. Then, a single-layer Transformer encoder is used for feature fusion, and the number of multi-head attention heads is set to 8. The output of the self-attention layer is normalized and connected to the residuals before being input into the feedforward network layer. The feedforward network consists of two linear layers and a ReLU activation function, and the calculation process is shown in Equation 4-3.
[0051]
[0052] In the formula: These are the input features of the feedforward network, which are the outputs of the self-attention layer after layer normalization and residual connections. and It is a learnable weight matrix in the network. and These are learnable bias parameters in the network.
[0053] Finally, the segment feature matrix output by the Transformer encoder is subjected to global average pooling along the frame dimension to obtain fixed-dimensional segment-level batch features. The pooling calculation process is shown in Formula 4-4.
[0054]
[0055] In the formula: It is the feature transformation operation of a single-layer Transformer encoder. It is a global average pooling operation along the frame dimension.
[0056] Through the above process, regardless of the duration of the behavioral segment... Regardless of the quantity, a fixed-dimensional batch processing feature of 256 dimensions can be obtained, and the batch processing features of all segments constitute a sequence. This serves as the input for subsequent graph convolution modules.
[0057] 5) The number of behavioral segments for different videos To address the issue of variable length behavior sequences, this module constructs a temporal relationship graph of behavior segments and uses a relational graph convolutional network (R-GCN) to model the temporal dependencies and semantic associations between segments. This aggregates the global temporal features of the entire video, solving the challenge of global modeling of variable-length behavior sequences.
[0058] First, construct a behavior sequence diagram with each behavior segment as a node. , where the set of nodes Batch processing features corresponding to M segments, edge set It includes the following two types of edges.
[0059] 1) Temporally adjacent edges: Directed edges are established between adjacent behavioral segments to represent the temporal sequence of behaviors, i.e., for any... Establish from arrive The directed edge.
[0060] 2) Semantic association edges: Undirected edges are established between segments with the same behavior label to represent the semantic consistency of the behavior, that is, for any... ,like ,Establish and Undirected edges between them.
[0061] Based on the graph structure described above, construct a relational adjacency matrix. The rules for defining matrix elements are shown in Formula 4-5.
[0062]
[0063] A two-layer relational graph convolutional network is used to update and aggregate node features. The calculation process of the standard R-GCN layer is shown in Equation 4-6.
[0064]
[0065] In the formula: The node feature matrix of the l-th layer is the initial input. . To incorporate the adjacency matrix of self-loops, for The corresponding degree matrix. This is a learnable weight matrix. It is the ReLU nonlinear activation function.
[0066] After two layers of R-GCN, the updated node feature matrix is obtained. Node features are weighted and pooled, with weights determined by the duration of the segment. The decision is that the longer the duration of a behavior segment, the greater its contribution to the long-term state determination. The weighted pooling process is shown in Equation 4-7, and the final video global temporal features are obtained. .
[0067]
[0068] In the formula: For the output feature of the i-th node, This represents the total number of frames in the video. This is the duration weight of the segment.
[0069] 6) Categorized Output and Dynamic Participation Scoring First, based on behavioral labels, short-term intentions are divided into focused (positive participation behavior) and unfocused (negative derailment behavior), with a basic weight for each behavior. The division rules are shown in Table 1.
[0070] Table 1 Semantic Classification of Learning Behaviors by Focus
[0071] An attitude correction weight was also set. This represents the factor by which the current body posture increases or decreases the current learning behavior. For focused actions, the closer the posture conforms to standard learning postures such as uprightness, the greater the correction is (greater than 1); the further the posture deviates from standard learning postures, the less the correction is (less than 1). For unfocused actions, the more unstable the posture, the greater the correction is (greater than 1); the more stable the posture, the less the correction is (less than 1). The classification rules are shown in Table 2.
[0072] Based on behavior and posture, the learning state weights of a classification module can be calculated. The ES value is calculated using an incremental cumulative method. The initial ES value is set to 5. For the ... For each classification group, the ES value at the end of the classification is calculated using both positive cumulative and negative decay methods. When, as shown in Formula 4-10, when When, as shown in Formula 4-11.
[0073] (1-8) (1-9) In the formula: For the first The ES value at the beginning of each behavior segment This is the ES value at the end of the segment. This indicates the weight corresponding to this behavior. Indicates the first The number of frames sustained for each action segment. This indicates the total number of video frames. This is the growth rate adjustment coefficient, with a default value of 5.
[0074] Table 2. Semantic Classification of Attention in Learning Postures
[0075] Implementation Cases This implementation method was tested on a self-built OL-LII60 dataset (3582 10-second videos, including frontal / left / right viewpoints): The overall accuracy is 83.3%, and the macro F1 score is 82.1%, which is better than C3D (64.2%) and TimeSformer (76.1%). The accuracy difference between perspectives is 1.6%, which shows strong robustness to multiple perspectives. The dynamic engagement curve can trace the impact of each behavior on the learning state.
[0076] The OL-LII60 dataset contains 60 learners, 1440 frontal views, 1071 left-side views, and 1071 right-side views, totaling 3582 videos. Each video is 10 seconds long, covering the three common online learning perspectives: frontal, left-side, and right-side. There are four long-term learning intention labels: very focused, focused, rarely focused, and unfocused, with a sample weighting of 1:1:1:1.
[0077] The experiment compared the proposed method and the C3D model
[70] proposed in 2015. It is a classic 3D convolutional video behavior recognition model. In the convolution operation, it extracts features from the spatial and temporal dimensions of the video frame at the same time and learns the appearance features and temporal features of the video sequence end-to-end. The TimeSformer model is a video temporal modeling model based on Transformer. It splits the input continuous frames into tokens in the spatiotemporal dimension and designs a spatiotemporal separation self-attention mechanism, which can capture the temporal relationship between frames, reduce the computational complexity of self-attention and realize efficient modeling of long videos. The InternVideo2 model is based on the unified ViT architecture and uses progressive training to pre-train on large-scale multi-scene video data, which solves the problems of weak cross-scene ability and insufficient long temporal modeling ability of previous video models. The DejaVid model is an encoder-independent temporal matching method. It does not require modification or retraining of the video encoder. It achieves temporal feature aggregation of variable-length videos by learning temporal weights and alignment strategies. The experimental results are shown in Table 3. It can be seen that the method proposed in this invention can simultaneously achieve better performance in both human interaction detection and out-of-distribution detection.
[0078] Table 3
[0079] In another embodiment of the present invention, a tag-driven long-term intent recognition system for online learning is provided, which can be used to implement the above-mentioned tag-driven long-term intent recognition method for online learning. Specifically, the tag-driven long-term intent recognition system for online learning includes: The data acquisition module is used to acquire video data, extract multiple frames of images at a preset frame rate, and identify each frame of image to extract behavior labels and pose labels. The behavior segment generation module is used to group multiple frames of images based on behavior labels and pose labels, merge consecutive frames with consistent labels and filter short-term noise segments to generate multiple behavior segments. The global temporal feature generation module is used to construct a relation graph with each behavior segment as a node. The edges of this relation graph include temporally adjacent edges that represent temporal sequence relationships and semantically related edges that represent semantic consistency of behaviors. The node features are aggregated through a graph convolutional network to generate global temporal features of the video. The output module is used to output long-term intent classification results based on the global temporal features of the video, and to calculate dynamic engagement scores based on the behavior weight, pose coefficient, duration and total number of frames of each behavior segment.
[0080] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.
[0081] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a tag-driven long-term intent recognition method for online learning.
[0082] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the tag-driven long-term intent recognition method for online learning in the above embodiments.
[0083] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0084] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0085] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0086] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A label-driven long-term intent recognition method for online learning, characterized in that, Includes the following steps: Acquire video data and extract multiple frames of images at a preset frame rate; Each frame of the image is identified, and behavior and pose labels are extracted. Based on behavior labels and pose labels, multiple frames of images are grouped, consecutive frames with consistent labels are merged, and short-term noise segments are filtered out to generate multiple behavior segments. A relationship graph is constructed using each behavior segment as a node. The edges of the relationship graph include temporally adjacent edges that represent the temporal sequence relationship and semantically related edges that represent the semantic consistency of the behavior. The node features are aggregated through a graph convolutional network to generate global temporal features of the video. The system outputs long-term intent classification results based on global temporal features of the video, and calculates dynamic engagement scores based on the behavior weight, pose coefficient, duration, and total number of frames of each behavior segment.
2. The label-driven long-term intent recognition method for online learning according to claim 1, characterized in that, The behavior tags include multiple categories such as reading, note-taking, listening, typing, using a mobile phone, using a tablet, getting up, leaning on a table, and daydreaming; the posture tags include multiple categories such as upright, leaning forward, resting chin on hand, tilting to the side, leaning back, standing, and lying down; the process of merging consecutive frames with consistent tags and filtering out short-term noise segments to generate multiple behavior segments is as follows: Traverse each frame and merge consecutive frames in which the behavior label and the pose label are consistent into a candidate segment; when the duration of the candidate segment is less than the preset minimum segment threshold, merge the candidate segment into the previous stable segment; otherwise, treat it as an independent behavior segment.
3. The label-driven long-term intent recognition method for online learning according to claim 1, characterized in that, After generating the behavior segments, feature fusion is performed on the frame-level features within each behavior segment to generate segment-level features of fixed dimensions, specifically: The features of each frame within the behavior segment are mapped to a preset dimension through linear projection; dynamic position encoding based on the position index within the segment is added to each frame; the features with added position encoding are input into a single-layer Transformer encoder for feature fusion; The features output by the Transformer encoder are subjected to global average pooling along the frame dimension to obtain the segment-level features of the fixed dimension.
4. The label-driven long-term intent recognition method for online learning according to claim 3, characterized in that, The dynamic position coding adopts sinusoidal position coding, and its calculation formula is as follows: in, This is the position index of the frame within the segment. Indexed by feature dimensions, For preset dimensions.
5. The label-driven long-term intent recognition method for online learning according to claim 1, characterized in that, The aggregation of node features through a graph convolutional network is implemented using a two-layer relational graph convolutional network, and its node feature update formula is as follows: in, For the first The node feature matrix of the layer, initial input It consists of fragment-level features of all behavioral segments; To incorporate the adjacency matrix of self-loops, The adjacency matrix represents the temporally adjacent edges and the semantically related edges; for The corresponding degree matrix; The weight matrix is a learnable matrix; It is a non-linear activation function.
6. The label-driven long-term intent recognition method for online learning according to claim 1, characterized in that, The calculation of the dynamic participation score is specifically as follows: Set an initial engagement score; for each behavioral segment, determine whether its corresponding learning state weight is positive; if so, update the dynamic engagement score using a positive accumulation method; if not, update the dynamic engagement score using a negative decay method; wherein, the learning state weight is obtained by multiplying the behavioral weight by the posture coefficient, and the calculation of the positive accumulation and negative decay are both related to the duration of the behavioral segment and the total number of video frames.
7. A label-driven long-term intent recognition method for online learning according to claim 6, characterized in that, The formula for calculating the positive accumulation is: The formula for calculating the negative attenuation is: in, and The first Engagement scores at the beginning and end of each behavioral segment; For the first The learning state weights of each segment, For the first The number of frames per segment. This represents the total number of frames in the video. This is the growth rate adjustment coefficient.
8. A label-driven long-term intent recognition system for online learning, characterized in that, include: The data acquisition module is used to acquire video data and extract multiple frames of images at a preset frame rate. Each frame of the image is identified, and behavior and pose labels are extracted. The behavior fragment generation module is used to group multiple frames of images based on behavior labels and pose labels, merge consecutive frames with consistent labels, filter short-term noise fragments, and generate multiple behavior fragments. The global temporal feature generation module is used to construct a relationship graph with each behavior segment as a node. The edges of the relationship graph include temporally adjacent edges that represent temporal sequence relationships and semantically related edges that represent semantic consistency of behaviors. The node features are aggregated through a graph convolutional network to generate global temporal features of the video. The output module is used to output long-term intent classification results based on global temporal features of the video, and to calculate dynamic engagement scores based on the behavior weight, pose coefficient, duration and total number of frames of each behavior segment.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the label-driven long-term intent recognition method for online learning as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of a label-driven long-term intent recognition method for online learning as described in any one of claims 1 to 7.