Classroom behavior detection method based on multi-modal large model

Abstract

This invention discloses a classroom behavior detection method based on a multimodal large model. Specifically, it includes using video files captured by a camera as input data. These video files can be directly input into GroundingDINO for target detection, or face recognition can be performed first. Using the Sub-Center ArcFace model in InsightFace face recognition, after recognizing the face of a specified object, the face region image or target bounding box can be provided as a cue to the TrackAnything model for video target tracking. That is, the target region of the object is found in each frame of the video, and then this region is individually sent to GroundingDINO for target detection to determine whether it is the target of the behavior to be detected. This invention can obtain the behavior detection results for each individual, and then perform statistical analysis of individual or group behavior.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, and mainly to a classroom behavior detection method based on a multimodal large model. Background Technology

[0002] Previous classroom behavior detection methods were mostly based on traditional closed-set object detection methods, primarily the YOLO series. These methods required collecting a large amount of classroom data for object detection task labeling and using the labeled data to train the model. While this method could achieve a good average detection accuracy in a fixed environment, its scalability and generalization would be severely challenged once the environment changed or the target type was updated. Usually, it was necessary to expand the dataset and retrain the model to meet the new application requirements.

[0003] Current classroom behavior monitoring does not provide individualized statistical analysis of the behavior of each student and teacher. It merely summarizes different categories of behavior based on the results of target detection, such as how many students are listening to the lesson and how many are using their phones. It cannot provide personalized statistical results such as the level of concentration of a particular student in this lesson.

[0004] Existing methods lack sufficient synergy between multi-target tracking and behavior recognition, resulting in insufficient granularity and low recognition accuracy in behavior analysis results. They are limited to the detection of a limited number of fixed categories of behavior, lack generalization ability, and require data collection, annotation, and model training, making it difficult to meet the needs of continuous tracking and analysis of specific individual behaviors. Therefore, there is an urgent need for an efficient behavior analysis method that combines face recognition, video tracking, and target detection to achieve accurate identification and statistical analysis of individual behaviors. Summary of the Invention

[0005] To address the above shortcomings, according to a first aspect of the present invention, a classroom behavior detection method based on a multimodal large model is proposed, the specific steps of which are as follows:

[0006] The original video is acquired through a camera, and the original video is read frame by frame using a frame rate control strategy.

[0007] In each frame, the InsightFace model is invoked to recognize all faces and compare them with the feature vectors of specific targets to obtain the target object;

[0008] Identify the target face in the target object, extract the region where the target face is located, and use it as input prompts for the TrackAnything model. The TrackAnything model generates a high-precision target mask based on Meta AI's SAM framework.

[0009] The Track Anything model automatically tracks the target region in subsequent video frames based on the input prompts and outputs the bounding box of the target in each frame;

[0010] The target region image in each frame is cropped separately and input into the GroundingDINO model. The model detects the target based on preset behavioral keywords and outputs the target detection results. A dynamic template update strategy is adopted. When the target is partially occluded or the viewpoint changes, the tracking continuity is maintained by fusing features from historical frames.

[0011] Based on the target detection results, the behavior labels of each frame are stored in the classroom behavior record sequence, and the behavior analysis module generates statistical analysis of individual student behavior or classroom group behavior.

[0012] Furthermore, the specific steps for calling the InsightFace model to recognize all faces include:

[0013] The face detection module provided by the InsightFace model is used to quickly locate all face regions in the current frame image and output the bounding box coordinates and key point information of each face.

[0014] For each detected face region, the Sub-Center ArcFace model in the InsightFace model is called to extract feature vectors, and the feature vectors are normalized to generate face vectors representing identity features.

[0015] The face vector is compared with the registered target identity feature vector. If the similarity is higher than the threshold, the currently detected face is determined to be the target object.

[0016] Furthermore, the implementation process of Track Anything specifically includes: initialization and refinement using the strong single-frame segmentation capability of the SAM framework, handling inter-frame continuity using the temporal correlation capability of the XMem model, and complementary processing through an interactive correction mechanism.

[0017] Furthermore, the XMem model is responsible for time series tracking, constructs spatiotemporal correlation based on the target mask, and rapidly propagates target information in subsequent frames, solving the temporal problems of scale changes and motion blur.

[0018] Furthermore, the prompts input to the SAM framework include point prompts and box prompts; after receiving point prompts and box prompts, the SAM framework generates a preliminary target mask, and uses post-processing techniques to optimize the preliminary target mask to obtain a high-precision target mask.

[0019] Furthermore, the specific steps of the Track Anything model automatically tracking the target region in subsequent video frames based on the input prompts also include: the Track Anything model uses the high-precision target mask to track the target, updates the target position in each frame and regenerates the mask, and uses optical flow estimation to predict the target motion during the tracking process.

[0020] Furthermore, the GroundingDINO model includes: an image feature extraction module, a text feature extraction module, a feature enhancement module, a language-guided query selection module, and a cross-modal decoder module. After performing target detection on the target region image, it outputs preliminary target detection results, maps the detection boxes back to the original image coordinate system, and filters out targets with confidence scores higher than a set threshold to obtain target detection results.

[0021] Furthermore, the dynamic template update strategy specifically includes: dynamically maintaining and updating the template library based on the appearance changes of the target during the tracking process, so as to avoid tracking errors caused by fixed templates.

[0022] According to a second aspect of the present invention, a computer program product is provided, on which one or more computer programs are stored, which, when executed by a computer processor, implement the method described above.

[0023] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0024] (1) Combining language prompts to detect behaviors of unknown categories improves the generalization of the detection system. Summary of the Invention

[0025] (2) The target detection module does not require data collection, labeling and model training.

[0026] (3) Combine facial recognition and target tracking to achieve statistical analysis of individual behavior. Attached Figure Description

[0027] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0028] Figure 1 A schematic diagram of a classroom behavior detection method based on a multimodal large model according to an embodiment of the present invention is shown.

[0029] Figure 2A schematic diagram of Track Anything video target tracking according to an embodiment of the present invention is shown.

[0030] Figure 3 A schematic diagram of the structure of a computer system implementing an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0031] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0033] Figure 1 A schematic diagram of a classroom behavior detection method based on a multimodal large model according to an embodiment of the present invention is shown, as follows: Figure 1 As shown:

[0034] S1. Acquire the original video through the camera, and read the original video frame by frame using a frame rate control strategy;

[0035] The system acquires real-time surveillance video from cameras deployed at the target location, or imports pre-saved raw video files as input. It employs a frame rate control strategy to perform frame-by-frame parsing of the raw video, specifically including:

[0036] The video stream is sampled according to a set frame rate (e.g., 5 frames per second, 10 frames per second, or adjusted as needed); each sampled frame is numbered and timestamped to ensure that subsequent recognition and detection processing has temporal continuity and synchronization; each frame is used as input for subsequent modules such as face recognition, target tracking, and behavior detection.

[0037] S2. In each frame, the InsightFace model is called to recognize all faces and compare them with the feature vectors of specific targets to obtain the target object;

[0038] Using the face detection module provided by InsightFace (such as RetinaFace or YOLO-face), all face regions in the current frame image are quickly located, and the bounding box coordinates and key point information (such as eyes, nose tip, mouth corners, etc.) of each face are output.

[0039] For each detected face region, the Sub-Center ArcFace model (a pre-trained face recognition model) in InsightFace is called to extract feature vectors.

[0040] The model structure is based on ResNet or MobileFaceNet, and outputs a normalized face vector (usually 512-dimensional) representing the identity features of the face.

[0041] Each extracted face vector is compared with the registered target identity vector for similarity. The similarity is measured by cosine similarity, with a threshold T (e.g., T = 0.4~0.6).

[0042] The similarity is calculated using the cosine similarity method, and the formula is shown below:

[0043] ,

[0044] in, This represents the normalized eigenvector. This represents the registered target identity feature vector.

[0045] If the similarity value is higher than the threshold, the face is determined to be the target object.

[0046] For faces identified as target objects, their position information (i.e., bounding box coordinates) and corresponding frame number in the current frame are recorded as initialization prompts for subsequent target tracking modules.

[0047] S3. Identify the target face in the target object, extract the region where the target face is located, and use it as input prompts for the TrackAnything model. The TrackAnything model generates a high-precision target mask based on Meta AI's SAM framework.

[0048] The high-precision target mask is input into the XMem model. XMem uses the memory representation of historical frames to perform a coarse mask estimation for the current frame, generates a coarse segmentation mask for the current frame, and judges the mask quality. If the mask quality meets the requirements, it directly enters the GroundingDINO model; otherwise, it performs fine segmentation. The fine segmentation requires calling SAM again to generate a high-precision target mask using the coarse mask and hints of the current frame.

[0049] The SAM framework model requires cue information to generate target masks. The cue information includes point cue and bounding box cue. Point cue is a set of two-dimensional coordinate points that represent key locations of the target (such as the center of a face or feature points). Bounding box cue is an extended form of the initial bounding box, used to assist in segmentation.

[0050] After receiving point cues and box cues, the SAM model generates a preliminary target mask, which may contain noise or inaccurate boundaries. Post-processing techniques (such as Conditional Random Fields (CRF) or morphological operations) are used to optimize the preliminary mask to obtain a high-precision mask.

[0051] Based on the SAM framework model and XMem model mentioned above, the specific division of labor is as follows: SAM is responsible for accurate segmentation of a single frame (such as initialization, refinement, and correction), and supports users to generate high-quality masks through weak prompts such as clicking, so as to solve the problem of detailed segmentation in complex structures or occluded scenes.

[0052] The XMem model is responsible for time series tracking. It builds spatiotemporal correlations based on the initial mask, rapidly propagates target information in subsequent frames, and handles temporal challenges such as scale changes and motion blur.

[0053] S4. The Track Anything model automatically tracks the target region in subsequent video frames based on the input prompts and outputs the bounding box of the target in each frame;

[0054] like Figure 2 The diagram illustrates Track Anything video target tracking. In a video sequence, the TrackAnything model uses a target mask for target tracking. For each frame z, the target position is updated as follows: and regenerate the mask. Optical flow estimation is used to predict target motion during tracking; the final high-precision target mask is the result of multi-frame optimization.

[0055] Optical flow estimation is a technique used to estimate the motion of pixels in an image sequence. It infers the motion vector of the target by analyzing the changes in pixel intensity between adjacent frames. The specific implementation process is as follows:

[0056] Assuming I(x,y,t) represents the brightness value of the image at position (x,y) and time t, the core of optical flow estimation lies in solving the motion vector v(u,v) of the pixel, where u represents the velocity component in the horizontal direction and v represents the velocity component in the vertical direction.

[0057] Based on the assumption of constant brightness, the model equation is: ,in, = Represents the gradient in the x-direction. = This represents the gradient in the y-direction. = The rate of change over time t is represented by the Lucas-Kanad method or the Horn-Schunck method to solve for constraints, and finally the motion vector of each pixel is obtained, thereby predicting the target's motion trajectory.

[0058] S5. The target region image in each frame is cropped separately and input into the GroundingDINO model. The target detection results are output according to the preset behavioral keywords. A dynamic template update strategy is adopted. When the target is partially occluded or the viewpoint changes, the continuity of tracking is maintained by fusing features from historical frames.

[0059] The GroundingDINO model is an open-set object detection model based on multimodal fusion, with its core capability being support for object detection driven by natural language text prompts. It can accurately align objects described by humans in natural language (such as "a student wearing a red shirt" or "a laptop on the podium") with visual regions in images / videos, outputting the object's location (bounding box), category, and confidence score. Unlike traditional closed-set detection models (which can only identify fixed categories defined in the training data), the GroundingDINO model fuses visual features and textual semantics through a cross-modal attention mechanism, possessing zero-shot detection capability. It can detect objects or behaviors not present in the training data (such as "raising a hand to ask a question" or "looking down at a phone") without requiring retraining for new object categories, making it suitable for scenarios requiring flexible expansion of the detection range.

[0060] The model includes the following modules: image feature extraction module, text feature extraction module, feature enhancement module, language-guided query selection module, and cross-modal decoder module. After performing target detection on the target region image, it outputs preliminary target detection results, maps the detection boxes back to the original image coordinate system, and filters out targets with confidence scores higher than a set threshold to obtain the target detection results.

[0061] The model's core features include: 1. Text-visual alignment: Combining the DINO object detection framework with grounding technology, it aligns text descriptions with visual features through contrastive learning, achieving "text-based image search" detection. 2. Zero-shot generalization: It eliminates the need for retraining for specific categories, directly detecting unknown targets based on user-input text keywords, making it suitable for flexible scenarios (such as custom behavioral keyword detection). 3. High-precision localization: Inheriting the end-to-end detection capabilities of the DETR series, it outputs target bounding boxes and supports multimodal input (text + image).

[0062] Furthermore, in target tracking scenarios, a "template" typically refers to a representation of the target's appearance features (such as a visual template or feature vector). The core of the dynamic template update strategy is to adaptively update the target template during tracking based on real-time changes in the target's appearance and historical frame information, in order to address challenges such as occlusion, changes in viewpoint, and changes in illumination. The specific mechanism is as follows:

[0063] 1) Strategic Objectives:

[0064] Solving tracking drift: When the target is partially occluded or the viewpoint changes, relying solely on the features of the current frame may result in the loss of the target's true appearance (e.g., occlusion causing feature incompleteness). By fusing historical frame templates, the integrity and continuity of the features are maintained.

[0065] Balance old and new information: avoid outdated templates (long-term lack of updates, target appearance has changed) or introduce noise (frequent updates, including occlusion / incorrect features).

[0066] 2) Core Mechanism

[0067] ① Template representation:

[0068] Feature extraction: For each cropped target region (ROI) in each frame, use GroundingDINO or additional feature extractors (such as ResNet, Transformer) to generate visual feature vectors (such as RGB features, semantic features) as temporary templates for the current frame.

[0069] Multimodal fusion: When combined with textual prompts (such as behavioral keywords), the template may contain a fusion representation of textual embedding features (such as BERT encoding) and visual features to enhance semantic constraints.

[0070] ② Dynamic update rules:

[0071] Weighted fusion: The historical template and the current frame template are fused according to their weights. Example formula: New template == α Current frame features + (1- ) Historical templates. Among them... This indicates the update coefficients (e.g., dynamically adjusted according to tracking confidence: reduced during occlusion). (to reduce the impact of current noise characteristics).

[0072] Sliding window update: Maintain a fixed-length queue of historical frames (e.g., the most recent 5 frames). Each time an update is performed, the oldest old template is discarded and the current new template is included, thus avoiding the accumulation of too much outdated information in the template.

[0073] Confidence-driven: When the tracking confidence (such as detection score, IOU matching degree) is lower than the threshold (indicating possible occlusion or loss), template updates are paused, and tracking continues by relying on historical reliable templates to avoid polluting the template library with erroneous features.

[0074] ③ Handling of occlusion and changes in viewing angle

[0075] Partial occlusion: When the target features in the current frame are incomplete, they are supplemented by features from the unoccluded parts in the historical template (e.g., feature weighting by channel, retaining features from historically clear areas).

[0076] Viewpoint variation: Fuse target features from different viewpoints (such as front, side, and back) to construct a multi-view robust template (similar to a "dictionary" of target appearance), and retrieve the sub-template that is closest to the current viewpoint during matching.

[0077] ④ Anti-drift mechanism

[0078] Template verification: Before each update, reverse verification is performed in adjacent frames using the current template (e.g., predict the previous frame with the new template and check for consistency) to eliminate erroneous templates introduced by false detection.

[0079] Background suppression: Explicitly encodes background features around the target in the template, and distinguishes the target from similar distractors through contrastive learning (such as metric learning), reducing the risk of drift.

[0080] 3) Strategic advantages

[0081] Robustness: It compensates for the defects of the current frame by using historical frame information, and maintains tracking continuity in scenarios such as occlusion and blur.

[0082] Adaptability: Dynamically adjust the template update frequency and weight to balance "memorizing history" and "adapting to new changes," avoiding the rigidity of static templates or the accumulation of noise in dynamic templates.

[0083] High efficiency: Feature extraction and template updating are performed only on the cropped target region (ROI), without processing the entire image, thus reducing computational costs.

[0084] S6. Based on the target detection results, store the behavior label of each frame into the classroom behavior record sequence, and generate behavioral statistical analysis of individual students or classroom groups through the behavior analysis module.

[0085] In a specific embodiment, the camera video stream data input is processed in two ways. One way is for full-scene detection: the data is directly input into the GroundingDINO model and combined with text prompts (such as "Detect all people playing with their phones") to perform global target localization. The other way is for individual detection: the face target bounding box is first output by InsightFace face recognition, the face target bounding box is input into TrackAnything to track individual regions, and the individual regions frame by frame are input into the GroundingDINO model and combined with text prompts (such as "Is student A paying attention to the lecture?") to perform local behavior judgment.

[0086] Global detection results: All target locations and categories that match the text prompts (e.g., the coordinates of all students "using mobile phones" in the classroom).

[0087] Individual detection results: Frame-by-frame behavioral labels for each student / teacher (e.g., "Student B raises their hand at the 3rd minute and lowers their head at the 5th minute"), supporting the generation of individual attention curves, behavior frequency statistics, and other analytical data.

[0088] The following is for reference. Figure 3 It shows a schematic diagram of the structure of a computer system 300 suitable for implementing electronic devices according to embodiments of the present application. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0089] like Figure 3 As shown, the computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 302 or programs loaded from storage section 308 into random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the system 300. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0090] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a liquid crystal display (LCD) and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card and a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. A removable medium 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 310 as needed so that computer programs read from it can be installed into storage section 308 as needed.

[0091] Specifically, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the functions defined in the methods of this application. It should be noted that the computer-readable storage medium of this application can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Program code contained on a computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0092] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0094] The modules described in the embodiments of this application can be implemented in software or in hardware.

[0095] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium carries one or more programs. When the electronic device executes one or more of these programs, the electronic device causes the following actions: It acquires raw video through a camera and reads the raw video frame by frame using a frame rate control strategy; in each frame, it calls the InsightFace model to recognize all faces and compares them with specific target identity feature vectors to obtain the target object; it identifies the target face within the target object, extracts the region where the target face is located, and uses it as input prompts for the Track Anything model, which generates a high-precision target mask based on MetaAI's SAM framework; the Track Anything model automatically tracks the target region in subsequent video frames based on the input prompts and outputs the bounding box of the target in each frame; it separately crops the target region image in each frame and inputs it into the GroundingDINO model, performs detection based on preset behavioral keywords, and outputs the target detection results, employing a dynamic template update strategy. When the target is partially occluded or the viewing angle changes, it maintains the continuity of tracking through historical frame feature fusion; based on the target detection results, it stores the behavioral labels of each frame in a classroom behavior record sequence and generates behavioral statistical analysis of individual students or classroom groups through a behavior analysis module.

[0096] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A classroom behavior detection method based on a multimodal large model, characterized in that, include: The original video is acquired through a camera, and the original video is read frame by frame using a frame rate control strategy. In each frame, the InsightFace model is invoked to recognize all faces and compare them with the feature vectors of specific targets to obtain the target object; Identify the target face in the target object, extract the region where the target face is located, and use it as input prompts for the Track Anything model. The Track Anything model generates a high-precision target mask based on Meta AI's SAM framework. The Track Anything model automatically tracks the target region in subsequent video frames based on the input prompts, outputting the target's bounding box in each frame. Specifically, the Track Anything model uses the high-precision target mask for target tracking, updates the target position in each frame and regenerates the mask, and uses optical flow estimation to predict target motion during the tracking process; for each frame z, the target position is updated to B. z And regenerate the mask M z The optical flow estimation infers the target's motion vector by analyzing the changes in pixel intensity between adjacent frames. The model equation is: I x u+I y v+I t = 0, where I(x,y,t) represents the brightness value of the image at position (x,y) and time t. This represents the gradient in the x-direction of the image. This represents the gradient in the y-direction of the image. The velocity component represents the rate of change over time t, u represents the horizontal velocity component, and v represents the vertical velocity component. The target region image in each frame is cropped separately and input into the GroundingDINO model. The model detects the target based on preset behavioral keywords and outputs the target detection results. A dynamic template update strategy is adopted. When the target is partially occluded or the viewpoint changes, the tracking continuity is maintained by fusing features from historical frames. The dynamic template update strategy specifically includes: For each frame, GroundingDINO is used to generate a visual feature vector for the target region after cropping, which serves as a temporary template for the current frame. The formula for the new template is: New template = α * Current frame features + (1-α) * Historical template, where α represents the update coefficient. A sliding window update is used, maintaining a fixed-length queue of historical frames. Each time an update is performed, the oldest old template is discarded and the current new template is included. When the tracking confidence level falls below the threshold, template updates are paused. When the target features in the current frame are incomplete, they are supplemented using features from the unoccluded parts of the historical template; By integrating target features from different perspectives, a multi-view robust template is constructed, and the sub-template closest to the current perspective is retrieved during matching. Before each update, perform reverse verification in adjacent frames using the current template; The background features around the target are explicitly encoded in the template, and the target is distinguished from similar interference objects through contrastive learning. Based on the target detection results, the behavior labels of each frame are stored in the classroom behavior record sequence, and the behavior analysis module generates statistical analysis of individual student behavior or classroom group behavior.

2. The classroom behavior detection method according to claim 1, characterized in that, The specific steps for calling the InsightFace model to recognize all faces include: The face detection module provided by the InsightFace model is used to quickly locate all face regions in the current frame image and output the bounding box coordinates and key point information of each face. For each detected face region, the Sub-Center ArcFace model in the InsightFace model is called to extract feature vectors, and the feature vectors are normalized to generate face vectors representing identity features. The face vector is compared with the registered target identity feature vector. If the similarity is higher than the threshold, the currently detected face is determined to be the target object.

3. The classroom behavior detection method according to claim 1, characterized in that, The implementation process of Track Anything specifically includes: initialization and refinement using the strong single-frame segmentation capability of the SAM framework, handling inter-frame continuity using the temporal correlation capability of the XMem model, and complementary processing through an interactive correction mechanism.

4. The classroom behavior detection method according to claim 3, characterized in that, The XMem model is responsible for time series tracking. It constructs spatiotemporal correlations based on target masks and rapidly propagates target information in subsequent frames, solving the temporal problems of scale changes and motion blur.

5. The classroom behavior detection method according to claim 1, characterized in that, The prompts input to the SAM framework include point prompts and box prompts. After receiving point prompts and box prompts, the SAM framework generates a preliminary target mask and uses post-processing techniques to optimize the preliminary target mask to obtain a high-precision target mask.

6. The classroom behavior detection method according to claim 1, characterized in that, The GroundingDINO model includes: an image feature extraction module, a text feature extraction module, a feature enhancement module, a language-guided query selection module, and a cross-modal decoder module. After performing target detection on the target region image, it outputs preliminary target detection results, maps the detection boxes back to the original image coordinate system, and filters out targets with confidence scores higher than a set threshold to obtain the target detection results.

7. The classroom behavior detection method according to claim 1, characterized in that, The dynamic template update strategy specifically includes: dynamically maintaining and updating the template library based on the appearance changes of the target during the tracking process, so as to avoid tracking errors caused by fixed templates.

8. A computer program product, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-7.

9. A computing system, characterized in that, It includes a processor and a memory, the processor being configured to perform the method as described in any one of claims 1-7.

Images