A classroom multi-modal behavior analysis method fusing target detection and large language model

Abstract

The application belongs to the technical field of artificial intelligence, and discloses a classroom multi-modal behavior analysis method combining target detection and a large language model, comprising the following steps: step one, perception layer data acquisition and processing: a behavior recognition model is used for real-time target detection to generate structured data with time sequence labels; step two, cognitive and reasoning layer analysis: a large language model is integrated in the cognitive and reasoning layer to process the structured data output by the perception layer; step three, calling the large language model for interactive layer response: based on a natural language question and answer interface, the system calls the large language model to understand the question raised by the user to the system in natural language, extracts information from a database and analysis results, combines an education knowledge base retrieval result, and organizes a natural language answer; and step four, application and display layer output through a visual interface. The application combines real-time perception capability and deep understanding and reasoning capability of the large language model to realize intelligent detection and deep analysis of classroom behavior.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a classroom multimodal behavior analysis method that integrates object detection and large language models. Background Technology

[0002] In recent years, computer vision-based object detection technology has been introduced into the field of classroom behavior recognition. Real-time object detection models, represented by the YOLO (YouOnly Look Once) series of algorithms, can quickly detect and locate teachers and students from video streams and identify their basic behavioral states. However, existing YOLO-based classroom behavior recognition systems mainly remain at the "perception" level, that is, they can only identify "who is doing what," lacking the ability to deeply understand and analyze the educational significance behind the behavior.

[0003] Existing technologies struggle to integrate behavior recognition with cognitive teaching contexts due to the following shortcomings: First, behavior recognition results consist only of discrete label data; the behavior labeling system is incomplete, the coordinate system is undefined, and the data lacks practical analytical value. Second, the lack of integration with educational knowledge prevents the formation of coherent teaching context descriptions and in-depth analysis of behavioral patterns, hindering teachers from obtaining substantial guidance for teaching improvement. Furthermore, the reasoning latency of large language models is on the order of seconds, a significant speed gap compared to YOLO's millisecond-level detection. The lack of systematic latency reduction measures makes real-time, flexible query analysis using natural language difficult. Summary of the Invention

[0004] The purpose of this invention is to provide a classroom multimodal behavior analysis method that integrates object detection and large language models, in order to solve the technical problem that existing technologies are unable to combine the real-time perception capabilities of computer vision with the deep understanding and logical reasoning capabilities of large language models to address the characteristics of classroom scenarios, thus making it impossible to construct a method that can achieve intelligent detection and deep analysis of classroom behavior.

[0005] The aforementioned method for classroom multimodal behavior analysis integrating object detection and large language models includes the following steps: Step 1: Data Acquisition and Processing at the Perception Layer: A behavior recognition model is used to perform real-time target detection on classroom surveillance video, detect the location and identity of teachers and students, identify their basic behavioral states, and generate structured data with time-series labels. Step Two: Cognitive and Reasoning Layer Analysis.

[0006] This step integrates a large language model at the cognitive and reasoning layers, processes the structured data output from the perception layer, obtains retrieval results from the knowledge graph and document library through a hybrid retrieval strategy and merges them to form the input of the large language model, and obtains the corresponding output through the large language model; Step 3: Call the large language model for interaction layer response. Based on the natural language question answering interface, the system calls the large language model to understand the questions asked by the user in natural language, extracts information from the database and analysis results, and combines the results retrieved from the educational knowledge base to organize them into a natural language answer. Step 4: Output the application and presentation layers through a visual interface.

[0007] Preferably, in step one, the behavior recognition model is an improved YOLO model, including: The backbone network adopts the lightweight CSPDarknet as the backbone network, introduces the Ghost module to replace some C3 modules, and introduces deformable convolution modules at the end of the backbone. In the neck region, a bidirectional feature pyramid network is used to replace the FPN and PAN structures, and learnable feature fusion weights are introduced to achieve weighted fusion of multi-scale features. The head adopts a decoupled head design, separating the classification and regression tasks into independent convolutional branches, and the output categories include two dimensions: person type and behavior category; The loss function of the behavior recognition model uses a combination of complete intersection-union loss and focus loss as boundary regression loss and classification loss, while introducing an additional scale loss term for small object detection.

[0008] Preferably, in step one, the method of applying identity ID binding and multi-target tracking is adopted, specifically using a three-level fusion mechanism that combines face recognition, personnel re-identification, and multi-target tracking technologies to achieve stable binding of unique identity IDs, including the following steps: (1) Initial identity registration: Before class, each teacher and student entering the classroom is registered using a facial recognition model to establish a facial feature database; (2) Real-time multi-target tracking: The BoT-SORT algorithm is used as the basic tracking algorithm. In each frame, the position of the existing target is predicted by the motion model and then matched with the detection result. The matching degree takes into account the IoU overlap and the cosine distance of the ReID feature. When the target reappears after being occluded, the system prioritizes ReID feature matching for re-association. (3) ID drift correction mechanism: A three-level ID drift correction strategy is designed, including: Level 1: When the ID output by the tracker is inconsistent with the face recognition result, the face recognition result shall be used for correction; Level 2: When the face is not visible, ReID feature matching shall be used for identity verification; Level 3: When the above two levels cannot confirm, probabilistic inference shall be made based on the continuity of the motion trajectory and the prior information of the seat, and the verification shall be continuously performed in subsequent frames.

[0009] Preferably, in step one, a complete classroom behavior labeling system is defined, thereby determining the behavior labels that can be used for complete description to facilitate subsequent cognitive reasoning based on the identified behavior status. The classroom behavior labeling system is divided into two major categories: teacher behavior and student behavior. Each category contains three levels of labels and corresponding labeling rules.

[0010] Preferably, in step two, a dual-carrier educational knowledge base is pre-constructed, comprising a structured knowledge graph and an unstructured document library. The knowledge graph is constructed using the Neo4j graph database and includes entity nodes and relation edges. The entity types corresponding to the entity nodes include teacher behavior, student behavior, teaching strategies, evaluation indicators, educational theories, classroom situations, and teaching suggestions. The relation types include behavior indicators, indicator anomaly suggestion strategies, strategies supported by theories, strategies generating specific suggestions, behaviors occurring in specific situations, and teacher-student behavior associations. The document library uses a vector database to store documents related to education. The documents are segmented, deduplicated, and vectorized before being stored in the vector database, supporting semantic retrieval.

[0011] Preferably, in step two, the hybrid retrieval strategy includes: structured retrieval using the Cypher query language to retrieve teaching strategies, evaluation criteria, and teaching suggestions related to the current behavioral indicators from the knowledge graph; semantic retrieval using vector similarity to retrieve professional literature and teaching cases related to the current analysis scenario from the document library; and merging the two retrieval results after relevance reordering as the input context for the large language model.

[0012] Preferably, in step two, the rules for merging after relevance reordering include: (1) Re-ranking of graph retrieval results: The re-ranking of graph retrieval results adopts a multi-factor weighted scoring mechanism, which comprehensively considers four dimensions: relationship strength, path length, entity importance, and context matching degree. The calculation formula is as follows: Score_graph=w1*R_strength+w2*R_path+w3*R_importance+w4*R_context, Here, Score_graph represents the re-ranking score of the graph retrieval results. R_strength, R_path, R_importance, and R_context represent the scores for relation strength, path length, entity importance, and context matching, respectively. w1, w2, w3, and w4 represent the weights of relation strength, path length, entity importance, and context matching, respectively. The relation strength score is calculated based on the weight attribute of relations in the graph, and for multi-hop paths, the geometric mean of the relation strengths on the path is taken. The path length score is calculated using the decay function: R_path = 1 / (1+a*(hop_count-1)), where a is the coefficient of the multi-hop path and hop_count is the hop count. The entity importance score is calculated as R_importance = degree(node) / max_degree, where degree(node) represents the degree of the node, and max_degree is the maximum degree of the node in the same network. The entity importance score is calculated based on degree centrality. R_context is divided into three levels according to the degree of context matching, with a larger value for higher matching degree. The matching degree levels are: complete match, partial match, and no context information.

[0013] Preferably, the rules for merging after relevance reordering also include: (2) Vector retrieval results re-ranking: Vector retrieval uses the Milvus vector database. Initial ranking is performed based on the cosine similarity between the query vector and the document vector. Two adjustment factors are then introduced, calculated as: Score_vector = cos_sim * T_decay * A_source, where Score_vector represents the re-ranking score of the vector retrieval results, and cos_sim is the cosine similarity between the query vector and the document vector. The document timeliness factor is calculated as follows: ),in For the corresponding coefficients, A_source is the difference between the year the document was published and the current year; A_source is the source authority factor, which is weighted according to the publication channel, with the weighting decreasing in the following order: core journals, Ministry of Education guidance documents, conference papers, textbooks, and online resources.

[0014] Preferably, the rules for merging after relevance reordering also include: (3) The rules for merging mixed search results include the following steps: The first step is weighted fusion: the re-ranking scores of the graph retrieval results and the re-ranking scores of the vector retrieval results are first normalized and then weighted and summed. The calculation formula is: Score_final = a * Score_graph + (1-a) * Score_vector, where a is the graph retrieval weight, (1-a) is the vector retrieval weight, and Score_final is the fusion score. The graph retrieval weight is higher than the vector retrieval weight. The second step is deduplication: when graph retrieval and vector retrieval return semantically similar knowledge items, the items with higher scores are retained. The semantic similarity judgment is based on the threshold judgment of text vector cosine similarity. The third step is truncation: the top K results after merging and sorting by score are taken as the final search results.

[0015] Preferably, in step two, the merged results after relevance reordering are injected into the prompt word template of the large language model according to a specific format. The graph knowledge injection area adopts a chain-structured format; the document knowledge injection area adopts a summary format, taking the first few words of the vector retrieval results; the behavioral data injection area contains detected behavioral statistics; the prompt word template includes: Context fusion templates are used to transform discrete behavioral events into coherent narrative descriptions; In-depth analysis templates are used for in-depth interpretation of situational descriptions from an educational perspective; Natural language question answering templates are used to process users' natural language queries.

[0016] The technical advantages of this invention are as follows: This invention organically combines the real-time perception capability of computer vision with the deep understanding and reasoning capability of a large language model. The object detection model is responsible for millisecond-level real-time perception, while the large language model is responsible for deep understanding and reasoning. The two complement each other, breaking through the limitations of existing technologies that only focus on the perception level. To this end, this invention defines a complete labeling system and coordinate system. Through a three-level behavioral labeling system and a unified physical coordinate system, the data possesses practical analytical value, supporting the entire data flow from pixel-level detection to semantic-level analysis. Correspondingly, this invention effectively injects educational knowledge into the large model. Specifically, it achieves precise injection of domain knowledge through a dual-carrier knowledge base and a hybrid retrieval strategy, achieving a retrieval recall rate of 92%, thus giving the analysis results of the large model educational professionalism. Regarding the latency and asynchrony problem between the two model engines, this invention uses optimization methods such as model quantization, edge deployment, KV-Cache, and asynchronous pipelines to control the overall end-to-end response time of the system to within 10 seconds, meeting the timeliness requirements of actual teaching scenarios.

[0017] Furthermore, this invention also improves the entire YOLO pipeline for classroom scenarios. From the backbone, neck, and head to the loss function and data augmentation, each step is optimized to address classroom-specific challenges such as occlusion, multiple targets, and small targets, resulting in a 7.5% improvement in average mAP for behavior recognition. It also achieves stable teacher and student identity tracking. This invention uses a three-level fusion identity binding mechanism to control the ID drift rate below 5%, ensuring the accuracy of time-series behavioral data and providing a reliable data foundation for subsequent in-depth analysis. Attached Figure Description

[0018] Figure 1 This is a basic flowchart of a classroom multimodal behavior analysis method that integrates object detection and large language model according to the present invention; Figure 2 A schematic diagram of the system architecture for applying the present invention; Figure 3 This is a flowchart of the process of the present invention; Figure 4 This is a schematic diagram of the model structure of the behavior recognition model in this invention; Figure 5 This is a flowchart of identity binding and multi-target tracking in this invention; Figure 6 This is a schematic diagram of the data structure output in step one of the present invention; Figure 7 This is a schematic diagram of the hybrid retrieval process based on an educational knowledge base (including knowledge graphs and document databases) in this invention; Figure 8 This is a flowchart of the deep analysis of the large language model formed in steps two and three of this invention; Figure 9 This is a flowchart illustrating the process from user asking a question to the system answering it, as described in this invention. Detailed Implementation

[0019] The following detailed description of the embodiments, with reference to the accompanying drawings, will further illustrate the specific implementation of the present invention, in order to help those skilled in the art to have a more complete, accurate, and in-depth understanding of the inventive concept and technical solution of the present invention.

[0020] like Figures 1-9 As shown, this invention provides a classroom multimodal behavior analysis method that integrates object detection and large language modeling, comprising the following steps: Step 1: Data Acquisition and Processing at the Perception Layer: A behavior recognition model is used to perform real-time target detection on classroom surveillance videos, detecting the location and identity of teachers and students, identifying their basic behavioral states, and generating structured data with time-series labels.

[0021] 1. The behavior recognition model is an improved YOLO model, based on the YOLOv8 or YOLOv10 architecture, with the following improvements made to meet the specific needs of classroom scenarios (occlusion, multiple targets, small targets, dynamic tracking): (1) Backbone optimization: The lightweight CSPDarknet is used as the backbone network, and the Ghost module is introduced to replace part of the C3 module, which reduces the computational cost by about 40% while maintaining the feature extraction capability. At the same time, a Deformable Convolution module is introduced at the end of the backbone to enhance the perception of partially visible targets in occluded scenes, so that the model can adaptively adjust the receptive field and better detect students occluded by desks.

[0022] (2) Neck optimization: The original FPN+PAN structure is replaced by a Bidirectional Feature Pyramid Network (BiFPN), which introduces learnable feature fusion weights to achieve weighted fusion of multi-scale features. This design can significantly improve the detection accuracy of small targets (such as the hand movements of students in the distance) while maintaining the detection performance of large targets (such as the whole body of a teacher), thus solving the problem of large differences in target scale in classroom scenarios.

[0023] (3) Head optimization: A decoupled head design is adopted, separating the classification and regression tasks into independent convolutional branches. A dedicated behavior recognition branch is added for classroom scenarios, and the output category includes two dimensions: personnel type (teacher / student) and behavior category, avoiding feature interference problems when a single detection head processes multiple tasks at the same time.

[0024] (4) Loss Function Adjustment: A combination of CIoU Loss (Complete Intersection over Union Loss) and Focal Loss (Focus Loss) is used as the boundary regression loss and classification loss. The parameters of Focal Loss are adjusted according to the class imbalance characteristics of the classroom scenario (e.g., there are far more "listening" class samples than "raising hands" class samples), and dynamic class weights are set to improve the detection recall rate of classes with fewer samples. An additional scale loss term for small target detection is also introduced, adding gradient penalty for targets smaller than 32x32 pixels to improve the detection accuracy of students at a distance.

[0025] (5) Dataset adaptation: A dedicated classroom behavior dataset was constructed, and a video acquisition strategy involving multiple angles, multiple lighting conditions, and multiple classroom environments was adopted. Data augmentation strategies included: random occlusion simulation (simulating desk occlusion), motion blur enhancement (simulating rapid movement), random brightness / contrast adjustment (adapting to different lighting conditions), Mosaic hybrid enhancement (improving the ability to detect small objects), and Copy-Paste enhancement (increasing the sample size of few-sample behaviors).

[0026] 2. This step also applies an identity ID binding and multi-target tracking method, specifically employing a three-level fusion mechanism combining facial recognition, personnel re-identification, and multi-target tracking technologies to achieve stable binding of unique identity IDs for teachers and students. The specific steps are as follows: (1) Initial identity registration: Before class, each teacher and student entering the classroom is registered using a face recognition model (trained based on ArcFace or CosFace) to establish a face feature database. Each teacher and student is assigned a unique ID (teacher ID format is T001, T002..., student ID format is S001, S002...), and their face feature vector and visual appearance feature vector are stored (for ReID).

[0027] (2) Real-time multi-target tracking: BoT-SORT (Bag-of-Tricks for SORT) is used as the basic tracking algorithm, which integrates motion model prediction, camera motion compensation, and ReID feature matching. In each frame, the position of existing targets is first predicted by the motion model, and then matched with the YOLO detection results. The matching degree takes into account both IoU overlap and ReID feature cosine distance. When a target reappears after being occluded, the system prioritizes ReID feature matching for re-association, rather than relying solely on spatial location, thereby effectively solving the ID drift problem.

[0028] (3) ID drift correction mechanism: A three-level ID drift correction strategy is designed. Level 1: When the ID output by the tracker is inconsistent with the face recognition result, the face recognition result shall be used for correction; Level 2: When the face is not visible (facing away from the camera, looking down, etc.), ReID feature matching is used for identity verification; Level 3: When the above two levels cannot confirm, probabilistic inference is made based on the continuity of the motion trajectory and the prior information of the seat, and continuous verification is performed in subsequent frames. This mechanism can reduce the ID drift rate in classroom multi-person scenarios to below 5%.

[0029] 3. This invention defines a complete classroom behavior labeling system, thereby determining behavioral labels that can be used for complete description and subsequent cognitive reasoning based on the identified behavioral states in this step. The labeling system is divided into two main categories: teacher behavior and student behavior, each containing three levels of labels. Examples of the specific labeling system in this embodiment are shown in Tables 1 and 2.

[0030] Table 1 Teacher Behavior Labeling System

[0031] Table 2 Student Behavior Labeling System

[0032] 4. Establish the classroom physical coordinate system and implement pixel coordinate transformation. This step defines a unified classroom physical coordinate system. The conversion process from pixel coordinates to physical coordinates is as follows: First, the pixel coordinates (u,v) are converted to normalized camera coordinates using the camera intrinsic parameter matrix and distortion parameters. Second, the bottom center point of the target detection box is used as the target's foot position, assuming the target is standing on the ground (Y=0), and the physical coordinates are calculated using the homography construction method. Finally, through translation and rotation transformations of the camera's installation position, the physical coordinates in the camera coordinate system are converted to the unified classroom coordinate system. The positioning accuracy of this method can reach ±0.3m in an indoor environment, meeting the spatial positioning requirements of behavior analysis.

[0033] In this embodiment, the center of the front wall of the classroom is taken as the origin (0,0), the X-axis points horizontally to the right, the Y-axis points vertically to the ground, and the Z-axis points to the rear of the classroom (student direction). The unit is meters (m), and the coordinate accuracy is 0.1m. Each classroom is calibrated once during system deployment, recording the camera's intrinsic parameter matrix, distortion parameters, and installation position parameters.

[0034] The structured data output format for Step 1 is: {timestamp, person_id, person_type, behavior_label, confidence, pixel_x, pixel_y, pixel_w, pixel_h, physical_x, physical_y, physical_z}. Here, timestamp is a millisecond-level timestamp, person_id is a unique identifier, person_type is the person type (teacher / student), behavior_label is a three-level behavior label, confidence is the detection confidence score, pixel_* are pixel coordinates, and physical_* are physical coordinates.

[0035] Step Two: Cognitive and Reasoning Layer Analysis.

[0036] This step integrates a large language model (such as DeepSeek-V3 or DeepSeek-R1) at the cognitive and reasoning layers, processes the structured data output from the perception layer, obtains retrieval results from the knowledge graph and document library through a hybrid retrieval strategy, merges them to form the input of the large language model, and obtains the corresponding output through the large language model.

[0037] 1. This invention constructs a dual-carrier educational knowledge base, comprising a structured knowledge graph and an unstructured document library, as detailed below: (1) Educational Knowledge Graph: Constructed using the Neo4j graph database, it contains the following entity types and relationships. The knowledge graph contains approximately 200 entity nodes and 500 relationship edges, as shown in Tables 3 and 4.

[0038] Table 3 List of Entity Types in Classroom Education Knowledge Graph

[0039] Table 4. List of Relationship Types in Classroom Education Knowledge Graph

[0040] (2) Document Repository: A vector database (Milvus) is used to store professional documents, curriculum standards, teaching cases, and other documents in the fields of education, educational psychology, and teaching methodology. Documents are segmented, deduplicated, and vectorized (using the BGE-large-zh model) before being stored in the vector database, supporting semantic retrieval. The document repository contains approximately 500 professional documents in the field of education, 30 curriculum standard documents, and 100 excellent teaching cases.

[0041] 2. This step utilizes a hybrid retrieval strategy to obtain search results from the knowledge graph and document database. This invention employs a hybrid retrieval strategy combining structured retrieval and semantic retrieval: structured retrieval uses the Cypher query language to retrieve teaching strategies, evaluation criteria, and teaching suggestions related to the current behavioral indicators from the knowledge graph; semantic retrieval uses vector similarity to retrieve professional literature and teaching cases related to the current analysis scenario from the document database. The two retrieval results are merged after relevance reordering and used as the input context for the large language model. The retrieval recall rate reaches 92%, and the relevance of the top 5 results is over 85%.

[0042] Search Example: In-depth Analysis of Student Distraction Behavior—The following Cypher query corresponds to a typical scenario: When the system detects a high proportion of students being distracted, it retrieves relevant evaluation indicators, teaching strategies, educational theories, and specific suggestions through a knowledge graph, providing structured knowledge support for generating an in-depth analysis report for the large language model. The query performs multi-hop searches along two paths: "behavior, indicators, strategies, theories" and "strategies, suggestions." See Tables 5 and 6 for details.

[0043] Table 5. Typical Cypher search examples: In-depth analysis of students' inattentive behavior MATCH (sb:StudentBehavior (levell: 'Distracted'}) -[:INDICATES]->(ei:EvaluationIndicator) -[:SUGGESTS]->(ts:TeachingStrategy) -[:SUPPORTED_BY]->(et:EducationalTheory) MATCH (ts)-[:GENERATES]->(sg:TeachingSuggestion) WHERE sb.durationThreshold<= 60 AND ei.referenceRange IS NOT NULL RETURN sb.level2AS (distracted type) sb.level3AS subdivides behavior. ei.nameAS related metrics ei.referenceRange AS reference range ts.name AS suggested strategy et.name AS theoretical basis, sg.content AS provides specific recommendations. sg.priorityAS priority ORDER BY sg.priority DESC, ei.referenceRange ASC Table 6 Example of query results:

[0044] 3. This step involves reordering and merging the search results after the initial retrieval. The corresponding rules are as follows.

[0045] The mixed search results need to be reordered and merged before they can be injected into the prompts of the large language model. This invention designs a multi-factor weighted scoring system for graph retrieval, a timeliness and source adjustment system for vector retrieval, and a three-step merging strategy.

[0046] (1) Reorder the map search results.

[0047] The re-ranking of graph retrieval results employs a multi-factor weighted scoring mechanism, comprehensively considering four dimensions: relation strength, path length, entity importance, and contextual fit. Score_graph=w1*R_strength+w2*R_path+w3*R_importance+w4*R_context, Where: w1=0.35 (relationship strength), w2=0.25 (path length), w3=0.25 (entity importance), w4=0.15 (contextual fit). The relation strength score R_strength is calculated based on the weight attribute of relations in the graph, and for multi-hop paths, it is the geometric mean of the strengths of all relations on the path. The path length score R_path uses a decay function: R_path = 1 / (1+a*(hop_count-1)), where a=0.5, 1 hop score is 1.0, 2 hop score is 0.67, and 3 hop score is 0.5, referring to the graph path decay theory of Page et al. (1999). The entity importance score R_importance is calculated based on degree centrality: R_importance = degree(node) / max_degree, referring to the degree centrality theory of Freeman (1978). The context matching score R_context is divided into three levels according to the degree of context matching: perfect match 1.0, partial match 0.6, and no context information 0.3.

[0048] (2) Reordering of vector retrieval results.

[0049] Vector retrieval uses the Milvus vector database, initially ranking the query vectors based on the cosine similarity between the query vectors and document vectors, and then introducing two adjustment factors: Score_vector = cos_sim * T_decay * A_source. Among these, the document timeliness factor... ), =0.1, This is the difference between the document's publication year and the current year; the source authority factor A_source is weighted according to the publication channel: core journals 1.0, Ministry of Education guidance documents 0.95, conference papers 0.8, textbooks 0.7, online resources 0.5.

[0050] (3) Rules for merging mixed search results The merging of mixed search results adopts a three-step strategy of "weighted fusion + deduplication + truncation".

[0051] The first step is weighted fusion: the normalized scores of graph retrieval and vector retrieval are weighted and summed. The calculation formula is: Score_final = a * Score_graph + (1-a) * Score_vector, where a = 0.6 is the graph retrieval weight. The graph retrieval weight is higher because structured knowledge has stronger logical connections and interpretability, which is more important for scenarios requiring precise reasoning, such as classroom behavior analysis. Normalization uses the Min-Max method: Score_normalized = (Score - Score_min) / (Score_max - Score_min).

[0052] The second step is deduplication: When graph retrieval and vector retrieval return semantically similar knowledge entries, the entry with the higher score is retained. Semantic similarity is determined using text vector cosine similarity, with a threshold set to 0.85.

[0053] The third step is truncation: take the top K results after fusion and sorting by score as the final search results. The default value of K is 8, which can be dynamically adjusted to 5-12 according to the number of behavior types and the complexity of the context.

[0054] 4. Inject the search results into the prompt word rules to form the template input content.

[0055] This step injects the merged search results into the prompt word template of the large language model according to a specific format. The graph knowledge injection area adopts a chain-structured format, such as "[Behavior type: Distraction][Evaluation index: Student participation][Suggested strategy: Question chain teaching][Theoretical basis: Constructivism][Specific suggestion: Increase the question waiting time to 3-5 seconds]"; the document knowledge injection area adopts a summary format, taking the first 200 words of the vector search results; the behavior data injection area adopts JSON format, containing the detected behavior statistics.

[0056] This invention designs three types of prompt word templates to guide large language models in professional analysis of educational scenarios. The specific templates are as follows: (1) Context Fusion Template: Used to transform discrete behavioral events into coherent narrative descriptions. The template structure is: "You are a classroom observation and analysis expert. The following is classroom behavior data within a time window: {behavior_data}. Please transform these discrete behavioral events into a coherent narrative of the teaching context, focusing on the temporal and causal relationships between behaviors." Through this template, this step can perform context fusion on temporal behavioral events, organizing discrete behavioral events into a coherent narrative description.

[0057] (2) In-depth analysis template: used for in-depth interpretation of the situation description from an educational perspective. The template structure is: "You are an education expert, skilled in classroom observation and analysis. The following is a classroom situation description: {narrative}. Relevant educational theories are referenced: {retrieved_knowledge}. Please analyze from the following perspectives: ① quality of teacher-student interaction; ② student participation and attention; ③ teaching pace and time allocation; ④ potential problems and improvement suggestions." (3) Natural Language Question Answering Template: Used to process users' natural language queries. The template structure is: "You are a classroom analysis assistant. User question: {user_question}. Current classroom analysis data: {analysis_data}. Relevant educational knowledge: {retrieved_knowledge}. Please answer the user question in natural language based on the data and knowledge. If data is involved, please provide specific values." Step 3: Call the large language model to respond to the interaction layer.

[0058] Based on a natural language question-answering interface, this step allows users to ask questions to the system in natural language. The system then uses a large language model to understand the question, extracts information from the database and analysis results, and combines this information with the results retrieved from the educational knowledge base to form a natural language answer. The system supports multi-turn dialogues, maintains contextual semantics, and allows for follow-up questions and refined queries. For example, a user can ask, "How was the overall student participation in the first half of this lesson?", and the system will return an answer containing specific data and professional interpretation. Users do not need to master any professional operation interface or query language to flexibly conduct classroom analysis queries, thus improving the user experience.

[0059] Step 4: Output the application and presentation layers through a visual interface.

[0060] The dashboard displays real-time monitoring data, behavioral statistics charts, and teacher movement diagrams, and stores raw monitoring data, analysis reports, and generated analytical conclusions, supporting historical review and comparative studies. Developed using a B / S architecture, the dashboard supports real-time data updates and multi-dimensional data drill-down, including data filtering and display by time period, individual student, and behavioral category. Analysis reports are automatically generated, containing statistical data, textual interpretation, and improvement suggestions.

[0061] To address the speed gap between the millisecond-level detection of the YOLO model and the second-level inference of the DeepSeek large language model in this invention, this invention also designs a system latency optimization and deployment strategy to solve this problem. The system latency optimization and deployment strategy specifically includes the following: (1) Model quantization: INT8 quantization (based on the PyTorch framework) was used for the YOLO model, reducing the inference latency from about 15ms / frame to about 8ms / frame, reducing the model size by about 75%, and decreasing mAP by only about 0.5%. INT4 / INT8 hybrid quantization (based on the GPTQ algorithm) was used for the DeepSeek model, reducing the inference latency from about 5 seconds to about 1.5 seconds, compressing the model size from 70B parameters to about 18B equivalent parameters, and decreasing the analysis quality by only about 3%.

[0062] (2) Edge Deployment Solution: A layered deployment architecture combining edge awareness and cloud inference is adopted. The YOLO model is deployed on the NVIDIA Jetson edge device on the classroom to realize real-time processing of the video stream and avoid network transmission latency of video data; the DeepSeek model is deployed on the cloud GPU server to be responsible for deep analysis and report generation. Only structured data (about 10KB per minute) is transmitted between the edge device and the cloud, rather than the raw video stream, which greatly reduces the network bandwidth requirements.

[0063] (3) KV-Cache Optimization: A KV-Cache optimization strategy is implemented for the inference process of the DeepSeek model. In the context fusion stage, repeated queries within the same class time window reuse the already calculated Key-Value cache to avoid duplicate encoding; in multi-turn dialogues, only newly added user inputs are incrementally encoded, and the KV cache of the historical context is directly reused. This strategy can reduce the average response time of multi-turn dialogues by about 40%.

[0064] (4) Asynchronous Pipeline Design: The perception layer and the cognition layer are designed as asynchronous pipeline architectures. YOLO detection runs continuously at the video frame rate, writing structured data to a message queue; the cognition layer consumes data from the queue in configurable time windows (default 5 minutes) for batch analysis. The two layers run independently without blocking each other, and the real-time performance of the perception layer is not affected by the inference latency of the cognition layer. The overall end-to-end response time of the system (from video input to analysis report generation) is controlled within 10 seconds.

[0065] To verify the effectiveness of the technical solution of this invention, an 8-week experimental test was conducted in three different types of classrooms (a large lecture hall, a medium-sized standard classroom, and a small seminar room), collecting video data from a total of 120 lessons, involving 6 teachers and approximately 300 students. The key experimental results are as follows: Experiment 1: Comparison of Behavior Recognition Accuracy The improved YOLO model of this invention was compared with the original YOLOv8 and YOLOv10. The test set contained 5000 labeled images, covering 9 categories of teacher behavior and 11 categories of student behavior. The evaluation metric was mAP@0.5 (mean accuracy). See Table 7 for details.

[0066] Table 7. Detection and evaluation indicators for Experiment 1

[0067] Experimental results show that the improved version of this invention improves the mAP by 8.3% and 7.1% in detecting teacher behavior and student behavior, respectively. At the same time, the inference speed is increased from 13.8ms / frame to 8.1ms / frame through INT8 quantization, which meets the requirements of real-time detection.

[0068] Experiment 2: Performance Comparison of Identity Tracking The identity tracking performance was tested in a classroom setting with more than 30 students. The evaluation metrics included ID switch rate and tracking accuracy, as shown in Table 8.

[0069] Table 8 Tracking performance metrics of Experiment 2

[0070] Experimental results show that the three-level fusion mechanism of the present invention reduces the ID drift rate from 12.8% to 4.2%, improves the tracking accuracy by 10.3 percentage points, and improves the occlusion recovery rate by 23.3 percentage points, effectively solving the identity tracking problem in classroom scenarios with multiple participants.

[0071] Experiment 3: Comparison of Reasoning Delay The inference latency of the system at each stage was tested under different optimization strategies. The test environment was: NVIDIA Jetson AGX Orin at the edge and NVIDIA A100 GPU in the cloud. As shown in Table 9.

[0072] Table 9. Detection and inference delay index of Experiment 3

[0073] Experimental results show that, through systematic latency reduction optimization, the LLM inference latency was reduced from 5.2 seconds to 0.9 seconds, and the overall end-to-end response time of the system was controlled within 10 seconds, meeting the timeliness requirements of actual teaching scenarios.

[0074] Experiment 4: Analytical Validity Assessment Five education experts were invited to conduct blind reviews of the classroom analysis reports generated by the system. The evaluation dimensions included: accuracy of analysis, relevance of suggestions, educational professionalism, naturalness of language, and overall practicality, using a 1-5 point scoring system. See Table 10 for an example.

[0075] Table 10 Analytical validity indicators for Experiment 4

[0076] Experimental results show that the complete system of this invention is significantly better than the comparison scheme in all evaluation dimensions, especially in the aspect of educational professionalism, which improves by 3.3 points (compared to YOLO statistics alone), verifying the key role of educational knowledge base and prompt word template in the quality of analysis.

[0077] Experiment 5: Ablation Experiment The independent contribution of each module was verified by gradually removing them. The overall analytical validity score (out of 5) was used as the evaluation metric, as shown in Table 11.

[0078] Table 11 Ablation test indicators of Experiment 5

[0079] Ablation experiments show that each technical module contributes positively to the overall system performance. Among them, the identity tracking module contributed the most (the score decreased by 1.1 points after removal), verifying the crucial role of the accuracy of time-series data in deep analysis; the prompt word template contributed the second most (a decrease of 0.9 points), indicating that specialized prompt design is crucial to the output quality of large models.

[0080] The present invention has been described above by way of example with reference to the accompanying drawings. Obviously, the specific implementation of the present invention is not limited to the above-described manner. Any non-substantial improvements made using the inventive concept and technical solution of the present invention, or the direct application of the inventive concept and technical solution of the present invention to other occasions without modification, are all within the protection scope of the present invention.

Claims

1. A method for classroom multimodal behavior analysis integrating object detection and large language model, characterized in that, Includes the following steps: Step 1: Data Acquisition and Processing at the Perception Layer: A behavior recognition model is used to perform real-time target detection on classroom surveillance video, detect the location and identity of teachers and students, identify their basic behavioral states, and generate structured data with time-series labels. Step 2: Cognitive and Reasoning Layer Analysis. This step integrates a large language model at the cognitive and reasoning layer, processes the structured data output from the perception layer, and uses a hybrid retrieval strategy to obtain retrieval results from the knowledge graph and document library, merging them to form the input of the large language model, and obtaining the corresponding output through the large language model. Step 3: Call the large language model for interaction layer response. Based on the natural language question answering interface, the system calls the large language model to understand the questions asked by the user in natural language, extracts information from the database and analysis results, and combines the results retrieved from the educational knowledge base to organize them into a natural language answer. Step 4: Output the application and presentation layers through a visual interface.

2. The classroom multimodal behavior analysis method integrating target detection and large language model as described in claim 1, characterized in that, In step one, the behavior recognition model is an improved YOLO model, including: The backbone network adopts the lightweight CSPDarknet as the backbone network, introduces the Ghost module to replace some C3 modules, and introduces deformable convolution modules at the end of the backbone. In the neck region, a bidirectional feature pyramid network is used to replace the FPN and PAN structures, and learnable feature fusion weights are introduced to achieve weighted fusion of multi-scale features. The head adopts a decoupled head design, separating the classification and regression tasks into independent convolutional branches, and the output categories include two dimensions: person type and behavior category; The loss function of the behavior recognition model uses a combination of complete intersection-union loss and focus loss as boundary regression loss and classification loss, while introducing an additional scale loss term for small object detection.

3. The classroom multimodal behavior analysis method integrating target detection and large language model as described in claim 1, characterized in that, Step one involves applying an identity ID binding and multi-target tracking method. Specifically, it employs a three-level fusion mechanism combining face recognition, people re-identification, and multi-target tracking technologies to achieve stable binding of unique identity IDs. This includes the following steps: (1) Initial identity registration: Before class, each teacher and student entering the classroom is registered using a facial recognition model to establish a facial feature database; (2) Real-time multi-target tracking: The BoT-SORT algorithm is used as the basic tracking algorithm. In each frame, the position of the existing target is predicted by the motion model and then matched with the detection result. The matching degree takes into account the IoU overlap and the cosine distance of the ReID feature. When a target reappears after being occluded, the system prioritizes using ReID feature matching for re-association. (3) ID drift correction mechanism: A three-level ID drift correction strategy is designed, including: Level 1: When the ID output by the tracker is inconsistent with the face recognition result, the face recognition result shall be used for correction; Level 2: When the face is not visible, ReID feature matching shall be used for identity verification; Level 3: When the above two levels cannot confirm, probabilistic inference shall be made based on the continuity of the motion trajectory and the prior information of the seat, and the verification shall be continuously performed in subsequent frames.

4. The classroom multimodal behavior analysis method integrating target detection and large language model as described in claim 1, characterized in that, In step one, a complete classroom behavior labeling system is defined, thereby determining the behavior labels that can be used for complete description based on the identified behavior status to facilitate subsequent cognitive reasoning. The classroom behavior labeling system is divided into two main categories: teacher behavior and student behavior. Each category contains three levels of labels and corresponding labeling rules.

5. The classroom multimodal behavior analysis method integrating target detection and large language model as described in claim 1, characterized in that, In step two, a dual-carrier educational knowledge base is pre-constructed, which includes a structured knowledge graph and an unstructured document library. The knowledge graph is constructed using the Neo4j graph database and includes entity nodes and relation edges. The entity types corresponding to the entity nodes include teacher behavior, student behavior, teaching strategies, evaluation indicators, educational theories, classroom situations, and teaching suggestions. The relation types include behavior indicators, indicator anomaly suggestion strategies, strategies supported by theories, strategies generating specific suggestions, behaviors occurring in specific situations, and teacher-student behavior associations. The document library uses a vector database to store documents related to education. The documents are segmented, deduplicated, and vectorized before being stored in the vector database, supporting semantic retrieval.

6. The classroom multimodal behavior analysis method integrating target detection and large language model according to claim 1, characterized in that, In step two, the hybrid retrieval strategy includes: structured retrieval using the Cypher query language to retrieve teaching strategies, evaluation criteria, and teaching suggestions related to the current behavioral indicators from the knowledge graph; semantic retrieval using vector similarity to retrieve professional literature and teaching cases related to the current analysis scenario from the document library; and merging the two retrieval results after relevance reordering as the input context for the large language model.

7. The classroom multimodal behavior analysis method integrating target detection and large language model as described in claim 6, characterized in that, In step two, the rules for merging after relevance reordering include: (1) Re-ranking of graph retrieval results: The re-ranking of graph retrieval results adopts a multi-factor weighted scoring mechanism, which comprehensively considers four dimensions: relationship strength, path length, entity importance, and context matching degree. The calculation formula is as follows: Score_graph=w1*R_strength+w2*R_path+w3*R_importance+w4*R_context, Here, Score_graph represents the re-ranking score of the graph retrieval results. R_strength, R_path, R_importance, and R_context represent the scores for relation strength, path length, entity importance, and context matching, respectively. w1, w2, w3, and w4 represent the weights of relation strength, path length, entity importance, and context matching, respectively. The relation strength score is calculated based on the weight attribute of relations in the graph, and for multi-hop paths, the geometric mean of the relation strengths on the path is taken. The path length score is calculated using the decay function: R_path = 1 / (1+a*(hop_count-1)), where a is the coefficient of the multi-hop path and hop_count is the hop count. The entity importance score is calculated as R_importance = degree(node) / max_degree, where degree(node) represents the degree of the node, and max_degree is the maximum degree of the node in the same network. The entity importance score is calculated based on degree centrality. R_context is divided into three levels according to the degree of context matching, with a larger value for higher matching degree. The matching degree levels are: complete match, partial match, and no context information.

8. The classroom multimodal behavior analysis method integrating target detection and large language model according to claim 7, characterized in that, The rules for merging after relevance reordering also include: (2) Vector retrieval results re-ranking: Vector retrieval uses the Milvus vector database. Initial ranking is performed based on the cosine similarity between the query vector and the document vector. Two adjustment factors are then introduced, calculated as: Score_vector = cos_sim * T_decay * A_source, where Score_vector represents the re-ranking score of the vector retrieval results, and cos_sim is the cosine similarity between the query vector and the document vector. The document timeliness factor is calculated as follows: ),in For the corresponding coefficients, A_source is the difference between the year the document was published and the current year; A_source is the source authority factor, which is weighted according to the publication channel, with the weighting decreasing in the following order: core journals, Ministry of Education guidance documents, conference papers, textbooks, and online resources.

9. The classroom multimodal behavior analysis method integrating target detection and large language model as described in claim 8, characterized in that, The rules for merging after relevance reordering also include: (3) The rules for merging mixed search results include the following steps: The first step is weighted fusion: the re-ranking scores of the graph retrieval results and the re-ranking scores of the vector retrieval results are first normalized and then weighted and summed. The calculation formula is: Score_final = a * Score_graph + (1-a) * Score_vector, where a is the graph retrieval weight, (1-a) is the vector retrieval weight, and Score_final is the fusion score. The graph retrieval weight is higher than the vector retrieval weight. The second step is deduplication: when graph retrieval and vector retrieval return semantically similar knowledge items, the items with higher scores are retained. The semantic similarity judgment is based on the threshold judgment of text vector cosine similarity. The third step is truncation: take the top K results after merging and sorting by score as the final search results.

10. The classroom multimodal behavior analysis method integrating target detection and large language model according to claim 9, characterized in that, In step two, the merged results after relevance reordering are injected into the prompt word template of the large language model in a specific format. The graph knowledge injection area adopts a chain-structured format; the document knowledge injection area adopts a summary format, taking the first few words of the vector retrieval results. The behavior data injection area contains detected behavior statistics; the prompt word templates include: Context fusion templates are used to transform discrete behavioral events into coherent narrative descriptions; In-depth analysis templates are used to provide in-depth interpretations of situational descriptions from an educational perspective. Natural language question answering templates are used to process users' natural language queries.

Images