Multi-target tracking driven student class participation degree evaluation method and system

Abstract

The invention relates to the technical field of target tracking, and provides a multi-target tracking driven student class participation degree evaluation method and system, and the method comprises the steps: firstly obtaining a class video, carrying out the target detection of students in the video, and determining a target candidate box of each student target; thirdly, continuously tracking student targets corresponding to the target candidate boxes by using a multi-target tracking algorithm, and generating a behavior trajectory sequence; then, matching the behavior track sequence with pre-stored student identity information, and establishing a corresponding relationship between identities and tracks; determining behavior frequency data of each student target according to the track sequence and a preset behavior category division rule, and calculating classroom score information according to the behavior frequency data; and finally, in combination with the scoring information and the target evaluation conditions, generating evaluation information of each student target. According to the embodiment of the invention, individualized tracking of students and quantitative evaluation of the classroom participation degree of the students are realized through a multi-target tracking technology.

Description

Technical Field

[0001] The present disclosure relates to the field of target tracking technology, and in particular to a student-level classroom participation evaluation method and system driven by multi-target tracking. Background Art

[0002] With the rapid development of information technologies such as big data, artificial intelligence, and the Internet of Things, how to use educational informatization to accelerate educational reform has become one of the research hotspots in recent years. The analysis of students' fine-grained behavior and classroom participation in the classroom is a very important part of the education evaluation system, especially the individualized evaluation and analysis of each student, which is of great significance to further improve the quality of education and ensure the healthy growth of students.

[0003] In related technologies, the main reliance is on recording the classroom process, usually through manual review of videos after class for statistical analysis, which cannot achieve real-time automatic acquisition of student participation. In recent years, although some studies have tried to use algorithms to automatically identify student behavior, most algorithms can only analyze group-level behavior and cannot distinguish individual differences among students, resulting in the inability to obtain accurate classroom participation assessments. Some studies have introduced multi-target tracking technology to improve the recognition and tracking accuracy of individual student behavior, but due to problems in real classroom environments, such as a large number of students, occlusion, small target scale, and large movement amplitude, these technologies are prone to target loss and identity jumps during the tracking process, affecting the accuracy of the recognition results.

[0004] In addition, even after the behavior identification results are obtained, most classroom engagement assessment methods only score by simply counting the positive or negative impact of different behaviors, lacking multi-dimensional analysis. Summary of the invention

[0005] The embodiments of the present disclosure at least provide a method and system for evaluating student-level classroom participation driven by multi-target tracking, which realizes individualized tracking of students and quantitative evaluation of students' classroom participation through multi-target tracking technology.

[0006] The present disclosure provides a method for evaluating student-level classroom participation driven by multi-target tracking, including: Obtaining a classroom video, and performing target detection on multiple students in the classroom video to determine a target candidate box corresponding to each student target; Based on the multi-target tracking algorithm, the student targets corresponding to each detected target candidate frame are continuously tracked to generate a behavior trajectory sequence corresponding to each student target; For each student goal, matching the behavior trajectory sequence corresponding to the student goal with the pre-stored student identity information, and establishing a corresponding relationship between the student identity information and the behavior trajectory sequence based on the matching result; Determine the behavior frequency data of each student target according to the behavior trajectory sequence corresponding to the student target and the preset behavior category classification rule; and determine the classroom score information corresponding to the student target based on the behavior frequency data of each student target; Based on the class score information corresponding to each student goal and the goal evaluation condition, evaluation information corresponding to each student goal is generated.

[0007] The present disclosure provides a multi-target tracking driven student-level classroom participation evaluation system, including: The behavior recognition module is used to obtain the behavior trajectory sequence of each student target through target detection and multi-target tracking technology; The identity association module is used to match the behavior trajectory sequence of each student target with the pre-stored student identity information; A status evaluation module, used to determine classroom score information corresponding to each student goal based on the behavior frequency data of each student goal; The classroom feedback module is used to generate evaluation information corresponding to each student goal based on the classroom scoring information and goal evaluation conditions corresponding to each student goal.

[0008] The present disclosure provides a multi-target tracking driven student-level classroom participation evaluation device, comprising: The target detection module is used to obtain a classroom video, and perform target detection on multiple students in the classroom video to determine a target candidate box corresponding to each student target; A sequence generation module is used to continuously track the student targets corresponding to each detected target candidate frame based on a multi-target tracking algorithm, and generate a behavior trajectory sequence corresponding to each student target; An identity matching module is used to match the behavior trajectory sequence corresponding to each student target with the pre-stored student identity information, and establish a corresponding relationship between the student identity information and the behavior trajectory sequence based on the matching result; A scoring determination module, for determining the behavior frequency data of each student target according to the behavior trajectory sequence corresponding to the student target and the preset behavior category classification rule; and determining the classroom scoring information corresponding to the student target based on the behavior frequency data of each student target; The student evaluation module is used to generate evaluation information corresponding to each student goal based on the classroom scoring information and the goal evaluation conditions corresponding to each student goal.

[0009] An embodiment of the present disclosure provides a computer device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the student-level classroom participation assessment method driven by multi-target tracking as described in any possible implementation manner described above is performed.

[0010] An embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for evaluating student-level classroom participation driven by multi-target tracking as described in any possible implementation manner described above is implemented.

[0011] The multi-target tracking-driven student-level classroom participation assessment method and system provided in the embodiments of the present disclosure first obtains a classroom video, performs target detection on the students in the video, and determines the target candidate frame of each student target; then, using a multi-target tracking algorithm, continuously tracks the student targets corresponding to each target candidate frame to generate a behavior trajectory sequence; then, matches the behavior trajectory sequence with pre-stored student identity information to establish a corresponding relationship between identity and trajectory; according to the trajectory sequence and preset behavior category classification rules, determines the behavior frequency data of each student target, and calculates the classroom scoring information accordingly; finally, combines the scoring information and the target evaluation conditions to generate the evaluation information of each student target.

[0012] In this way, the disclosed embodiment achieves a comprehensive, objective and quantitative evaluation of students' classroom participation by individualizing students through multi-target tracking technology, which helps teachers understand students' classroom performance more accurately, promote personalized teaching and improve classroom management.

[0013] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required to be cited in the embodiments. The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used together with the specification to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0015] Figure 1A flowchart of a method for evaluating student-level classroom participation driven by multi-target tracking provided by an embodiment of the present disclosure is shown; Figure 2 A flow chart of a method for determining a target candidate frame provided by an embodiment of the present disclosure is shown; Figure 3 A flow chart of a student identity information matching method provided by an embodiment of the present disclosure is shown; Figure 4 A flow chart showing a method for determining classroom score information corresponding to a student goal provided by an embodiment of the present disclosure; Figure 5 A schematic diagram of the structure of a student-level classroom participation evaluation system driven by multi-target tracking provided by an embodiment of the present disclosure is shown; Figure 6 A schematic diagram of the structure of a multi-target tracking driven student-level classroom participation evaluation device provided by an embodiment of the present disclosure is shown; Figure 7 A schematic diagram of the structure of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical scheme and advantages of the embodiments of the present disclosure clearer, the technical scheme in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the present disclosure for protection, but merely represents the selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present disclosure.

[0017] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0018] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent the following three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0019] With the rapid development of information technologies such as big data, artificial intelligence, and the Internet of Things, how to use educational informatization to accelerate educational reform has become one of the research hotspots in recent years. Among them, individualized evaluation and analysis of each student's fine-grained behavior and classroom participation is of great significance for in-depth understanding of students' learning status, discovering problems in the teaching process, and formulating targeted teaching strategies, thereby improving the quality of education and ensuring the healthy growth of students.

[0020] The study found that the traditional smart classroom has made some progress in recording classroom behavior, but there are still many limitations in the analysis of students' classroom participation. The traditional smart classroom mainly focuses on simple recording of the classroom process. If you want to obtain the classroom participation of each student, you can usually only conduct statistical analysis by manually viewing the video after class. This method not only consumes a lot of manpower and time, but also makes it difficult to ensure the timeliness and accuracy of the analysis results.

[0021] In recent years, some studies have begun to explore the use of algorithms to automatically identify student behaviors in order to address the shortcomings of traditional methods. However, most of these studies focus on group-level behavior analysis, which means that they can only obtain the classroom participation of the entire class, but cannot distinguish individual students, making it difficult to meet the needs of personalized education. A few studies have further introduced multi-target tracking technology to try to solve the problem of accurate identification and tracking of individual student behaviors, and provide technical support for intelligent analysis and evaluation of the teaching process. However, in practical applications, due to the complex and changeable real classroom environment, the recorded videos usually have problems such as a large number of students, mutual occlusion, small target scale, and large movement amplitude. During the tracking process, it is very easy for the target to lose track and change its identity, which seriously affects the accuracy of individual behavior recognition.

[0022] In addition, after obtaining the results of automatic identification of student behaviors, in order to further obtain the evaluation results of classroom participation, it is necessary to use statistical methods to quantify these behaviors. However, when evaluating classroom participation, existing studies usually simply consider the positive or negative effects of different behaviors themselves, lacking a multi-dimensional and comprehensive evaluation. In other words, it does not fully consider that the same individual behavior may have different effects in different overall behavioral contexts, resulting in inaccurate evaluation results of classroom participation, which makes it difficult to truly reflect the students' classroom participation status.

[0023] Based on the above research, a method and system for evaluating student-level classroom participation driven by multi-target tracking is provided in the embodiments of the present disclosure. First, a classroom video is obtained, and target detection is performed on the students in the video to determine the target candidate frame of each student target. Then, a multi-target tracking algorithm is used to continuously track the student targets corresponding to each target candidate frame to generate a behavior trajectory sequence. Then, the behavior trajectory sequence is matched with the pre-stored student identity information to establish a corresponding relationship between the identity and the trajectory. According to the trajectory sequence and the preset behavior category classification rules, the behavior frequency data of each student target is determined, and the classroom scoring information is calculated accordingly. Finally, the scoring information and the target evaluation conditions are combined to generate the evaluation information of each student target.

[0024] In the disclosed embodiment, multi-target tracking technology is used to track students individually, thereby achieving a comprehensive, objective and quantitative evaluation of students' classroom participation, which helps teachers understand students' classroom performance more accurately, promote personalized teaching and improve classroom management.

[0025] To facilitate understanding of this embodiment, the executor of the multi-target tracking-driven student-level classroom participation evaluation method provided in the embodiment of the present disclosure is first introduced in detail. The executor of the multi-target tracking-driven student-level classroom participation evaluation method provided in the embodiment of the present disclosure is a computer device. The computer device may be a terminal device or a server. Among them, the terminal device may also be a mobile device, a user terminal, a terminal, a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data and artificial intelligence platforms. Optionally, the method may also be applied to an implementation environment consisting of a computer device and a server.

[0026] The following is a detailed description of the multi-target tracking driven student-level classroom participation evaluation method provided in the embodiment of the present application in conjunction with the accompanying drawings. Figure 1 As shown, it is a flowchart of a method for evaluating student-level classroom participation driven by multi-target tracking provided by an embodiment of the present disclosure, and the method includes the following S101-S105: S101, obtaining a classroom video, and performing target detection on multiple students in the classroom video to determine a target candidate box corresponding to each student target.

[0027] It is understandable that the classroom video can be recorded in real time in the classroom through a camera, or obtained through sources such as the school's online learning platform. Object detection is a technology in computer vision that aims to identify specific targets or objects from images or videos. In this solution, object detection is mainly used to identify multiple students in the video. Specifically, by utilizing object detection algorithms (such as YOLO, Faster R-CNN and other deep learning models), each student in the classroom video can be identified and a target candidate box can be generated for each student. The target candidate box is a rectangular box that represents the position and size of the student in the video. For example, assuming that five students are detected in the classroom video, a rectangular box can be generated for each student for subsequent tracking and analysis.

[0028] Exemplarily, when determining to perform target detection on each student, it can include: performing multi-scale feature extraction on each frame image in the classroom video, and determining multiple student targets corresponding to each frame image and initial candidate boxes corresponding to each student target based on the multi-scale feature extraction results.

[0029] Here, due to the large differences in the posture, size, position, etc. of students in the classroom scene, a single-scale feature extraction method may be difficult to meet the needs of accurate detection, so the present disclosure proposes to adopt a multi-scale feature extraction technology to capture key information in the image from different scales. Among them, the multi-scale feature extraction technology extracts rich feature information by processing the image at different resolutions or different receptive fields. For example, different layers in a convolutional neural network (CNN) can be used to obtain feature maps of different scales. The shallow feature map has a high spatial resolution and can capture detailed information, such as the edges and contours of the students; the deep feature map has high semantic information and can identify the category, posture, etc. of the students. Further, based on the results of multi-scale feature extraction, multiple student targets corresponding to each frame of the image and the initial candidate boxes corresponding to each student target can be determined. The initial candidate box is a rough area (generally a rectangular area) surrounding the student target. There may be multiple initial candidate boxes for each student target, and their positions and sizes may not be the same.

[0030] In some possible embodiments, when performing target detection on each student target in the classroom video, high-precision target behavior detection in a dense seating scenario can be achieved based on the YOLOv11s model to obtain the initial candidate frames corresponding to each student target. Here, the YOLOv11s model is mainly composed of four parts: input, backbone network, neck network, and detection head network. The input is responsible for adjusting the image size of each frame in the classroom video to a fixed size (such as 640X640) and inputting it into the model; the backbone network is responsible for image feature extraction. Compared with the traditional cubic convolution module, more skip layer connections are introduced to improve feature expression capabilities; the neck network is used to process and fuse the extracted features, and the required computing resources are reduced and the speed of generating target frames is increased by fusing multi-scale features; the detection head network detects the target position and category based on multi-scale features.

[0031] For example, after obtaining multiple student targets corresponding to each frame image in the classroom video and the initial candidate frames corresponding to each student target, since there may be target overlap between different candidate frames in the detection results, after obtaining the initial candidate frames, they can be further optimized to obtain more accurate target candidate frames. Figure 2 As shown, when determining the target candidate frame corresponding to each student target, any one of the multiple student targets corresponding to each frame image can be used as the student target to be determined, and then the following steps S201 to S203 are performed: S201, calculating the confidences of the initial candidate boxes corresponding to the to-be-determined student target, and sorting the initial candidate boxes according to their confidences, and taking the initial candidate box with the highest confidence as the target candidate box corresponding to the to-be-determined student target.

[0032] Here, confidence is a value between 0 and 1, which refers to the probability that the model predicts that a candidate box contains a target. The higher the confidence, the greater the possibility that the candidate box contains the target. The confidence can be calculated based on a variety of methods, such as deep learning-based classifiers, regression models, etc. The initial candidate boxes are sorted according to their confidences, and the initial candidate box with the highest confidence is used as the target candidate box corresponding to the student target to be determined. The target candidate box is a precise area around the student target, and its position and size are more accurate; at the same time, the target candidate box is the basis for the subsequent deletion and merging of candidate boxes.

[0033] S202, selecting an initial candidate box having the largest overlapping area with the target candidate box from other initial candidate boxes except the target candidate box; if the overlapping area of ​​the initial candidate box having the largest overlapping area with the target candidate box and the target candidate box is greater than a preset threshold, deleting the initial candidate box having the largest overlapping area with the target candidate box.

[0034] Specifically, after obtaining the target candidate frame, select the initial candidate frame with the largest overlapping area with the target candidate frame from other initial candidate frames except the target candidate frame. Here, the overlapping area refers to the area occupied by the overlapping parts of the two candidate frames on the image. If the overlapping area between the initial candidate frame and the target candidate frame is greater than a preset threshold (set according to actual needs, such as 0.5 or 0.7), it is considered that the two candidate frames are highly overlapping and may represent the same student target. At this time, the initial candidate frame can be deleted to avoid repeated detection.

[0035] S203, return to the above step S202, until the overlapping areas of the other initial candidate boxes except the target candidate box and the target candidate box are less than the preset threshold and / or there are no other initial candidate boxes except the target candidate box remaining, take any one of the remaining student targets as a new student target to be determined, and return to step S201.

[0036] Specifically, the above step S202 is repeatedly performed until the overlapping areas of the other initial candidate frames except the target candidate frame and the target candidate frame are all less than the preset threshold and / or no other initial candidate frames remain. At this time, any one of the remaining student targets can be used as a new student target to be determined, and the process returns to step S201 until each student target in the image has its corresponding target candidate frame determined, so as to complete the task of determining the target candidate frames of each student target in the image.

[0037] The disclosed embodiment can filter out redundant frames whose overlapping area with the target frame is greater than a threshold through the above method, so as to solve the overlapping problem between different candidate frames and realize the rapid identification and positioning of students' classroom behaviors in dense scenes.

[0038] S102, based on the multi-target tracking algorithm, continuously track the student targets corresponding to the detected target candidate frames to generate a behavior trajectory sequence corresponding to each student target.

[0039] Here, the multi-target tracking algorithm is a method for simultaneously tracking multiple targets in a video or image sequence, and is used to locate and maintain the identity consistency of multiple moving targets in a video or image sequence. The behavior trajectory sequence refers to the various types of behavior data recorded in sequence over a period of time by the student target. Each behavior will be clearly recorded in time, and each behavior will be marked with the student's specific performance in class. These behaviors may include but are not limited to: raising hands, answering questions, walking, reading, interacting with classmates, etc.

[0040] Specifically, when continuously tracking the student targets corresponding to the detected target candidate frames based on the multi-target tracking algorithm, the following steps (1) to (2) may be included: (1) For each target candidate frame in each frame image, predict the next moment state information of the student target corresponding to the target candidate frame based on the multi-target tracking algorithm and the historical information about the target candidate frame, and determine the short-term state vector corresponding to the student target based on the prediction result; (2) For each target candidate frame in each frame image, determine the motion consistency metric and appearance similarity metric corresponding to the student target based on the matching function, the short-term state vector corresponding to the student target, and the actual state vector of the target candidate frame; and adjust the information of the student target corresponding to the target candidate frame based on the motion consistency metric and appearance similarity metric corresponding to the student target; and update the target candidate frame corresponding to the student target based on the target candidate frame after the information adjustment, so as to achieve continuous tracking of the student target.

[0041] It is understandable that in the multi-target tracking process, for the target candidate box in each frame of the image, the state of the student target corresponding to the target candidate box at the next moment can be predicted based on the historical information of the target candidate box. Here, the position and motion features of the next moment can be predicted based on the historical information of the target candidate box (such as the target candidate box information of the past 5 frames) through Kalman filtering or LSTM network (Long Short-Term Memory) to form a short-term state vector. Among them, the target candidate box information can include the selection box position coordinates, category label (such as the candidate box sequence number) and confidence; the short-term state vector is a vector that describes the current state of the target, usually containing information such as position, size, motion characteristics, etc., which can help better understand the behavior dynamics of the target, so as to track and predict the student target more accurately.

[0042] In some possible embodiments, after performing target detection on each frame of image, the candidate box position coordinates, category labels (such as candidate box serial numbers) and confidence scores obtained by target detection of each frame can also be stored in a dynamic temporary list according to the frame sequence to form a trajectory temporary queue containing historical detection data, thereby realizing the storage of historical information of the target candidate box.

[0043] Specifically, after obtaining the short-term state vector corresponding to the student target, the predicted short-term state vector of the target candidate frame at the next moment and the actual state vector of the target candidate frame at the next moment can be matched through a matching function to obtain the motion consistency metric and appearance similarity metric corresponding to the student target. Among them, the motion consistency metric is used to measure the consistency of the student target's motion between different frames. For example, if a student walks, their motion trajectory should remain consistent between multiple consecutive frames. By calculating the position change of the target in consecutive frames, it is possible to evaluate whether the target's motion is stable and make corresponding adjustments. The appearance similarity metric refers to the similarity of the target's appearance features (such as color, shape, texture, etc.) in different frames to confirm whether a target is still the same student.

[0044] Furthermore, the information of the student target corresponding to the target candidate frame can be adjusted according to the calculated motion consistency metric and appearance similarity metric to ensure continuous and accurate tracking of the target. If the target candidate frame is consistent with the student target of the original target candidate frame in appearance and motion characteristics, the tracking state is maintained. Otherwise, the target candidate frame is adjusted and the student target corresponding to it is re-identified or corrected to ensure the continuity and accuracy of tracking.

[0045] In some possible embodiments, a matching cost matrix can also be generated based on the motion consistency metric and the appearance similarity metric. This matrix can further achieve the execution of the optimal match by quantifying the degree of match between each candidate frame and the student target. Specifically, the successfully matched detection frame will inherit the original student target trajectory, and those unmatched detection frames will be assigned to new student targets that match it. For those trajectories that have not been matched for a long time, they are terminated to ensure the efficiency and accuracy of tracking. Finally, by correcting the information of the student target corresponding to the target candidate frame and updating it to the trajectory of the corresponding student target, continuous and stable tracking of the student target can be achieved, ensuring that the target can be consistently and accurately tracked even in complex environments.

[0046] S103, for each student target, matching the behavior trajectory sequence corresponding to the student target with the pre-stored student identity information, and establishing a corresponding relationship between the student identity information and the behavior trajectory sequence based on the matching result.

[0047] It is understandable that after completing the tracking of student targets and obtaining the behavioral trajectory sequence corresponding to each student target, the behavioral trajectory of each student target can be identified. Here, the pre-stored student identity information refers to multiple student identity information pre-entered in the system database, which may include basic information such as the student's front photo, name, student number, class, etc. By matching with the student identity information, it can be determined which student performed each behavioral trajectory, thereby completing the identification of the student identity, and then establishing a corresponding relationship between the behavioral trajectory and the student identity. For example, assuming that student A raises his hand to speak many times in class, by associating the student's behavioral trajectory with his personal information, the student's classroom participation can be identified.

[0048] Specifically, refer to Figure 3 As shown, when matching the behavior trajectory sequence corresponding to the student target with the pre-stored student identity information, the following S301 to S303 may be included: S301, determining an initial image of the student target based on the classroom video and the behavior trajectory sequence corresponding to the student target.

[0049] Here, since each frame image in the classroom video may present different postures or facial features of the student target, a clear frontal image of the student target at a specific moment can be selected from the classroom video through the timeline of the behavior trajectory sequence corresponding to the student target as the initial image of the student target, so as to achieve correct identity matching of the student target.

[0050] S302, performing global feature extraction on the initial image to obtain initial global features; and, performing multi-scale local feature extraction on the initial image to obtain initial multi-scale local features; and, determining the information to be matched corresponding to the student target based on a multi-scale feature aggregation strategy, the initial global features and the initial multi-scale local features.

[0051] Specifically, after obtaining the initial image of the student target, image feature extraction will be performed on it for better identity matching. First, through global feature extraction, the overall features of the entire image are analyzed, including obvious features such as face, posture, and clothing. Then, multi-scale local feature extraction is performed, that is, the details of different areas in the image are analyzed. For example, students' facial expressions, gestures, eye direction, etc., these local features can provide more accurate information to help distinguish similar students. By extracting local features at different scales, we can have a more comprehensive understanding of students' specific behaviors in the classroom. Finally, a multi-scale feature aggregation strategy is used to combine global features with local features to form a comprehensive feature representation, that is, the information to be matched corresponding to the student target, which is used for the next matching process.

[0052] In some possible embodiments, a hierarchical multi-granularity network (MGN) may be used, and its branch structure may be used to extract features from an input image (ie, an initial image) at different scales and granularities, thereby achieving a more detailed description of the target.

[0053] Specifically, the initial image can be preprocessed, including image enhancement, noise suppression, and size normalization, to ensure the stability and robustness of subsequent feature extraction. After preprocessing, the image is input into the MGN network, which extracts deep features through a multi-level, branched structure and adopts a multi-scale feature aggregation strategy to fuse local and global features from different network branches to capture the fine-grained information and global semantic information of the target, and obtain the information to be matched corresponding to the student's target.

[0054] S303, respectively determining the similarity values ​​between the student identity information corresponding to each student target in the pre-stored student identity information and the information to be matched, and determining the student identity information corresponding to the student target with the largest similarity value as the target student identity information corresponding to the student target.

[0055] It is understandable that, during the matching process, the similarity values ​​between the student identity information corresponding to each student target in the pre-stored student identity information (usually also including its corresponding image feature information) and the information to be matched can be calculated respectively, and then the student identity information corresponding to the student target with the largest similarity value is selected as the target student identity information corresponding to the current student target. The similarity value can be obtained by calculating the cosine similarity, which is defined as the dot product between two normalized feature vectors.

[0056] For example, assuming that the pre-stored student identity information includes information of student B and student C, the similarity between the image feature information of student B and the information to be matched of student A, and the similarity between the image feature information of student C and the information to be matched of student A are calculated respectively. If the calculation result shows that the similarity value of student B is 0.85 and the similarity value of student C is 0.72, then the system will determine that the target student identity information corresponding to student A is the information of student B.

[0057] S104, determining the behavior frequency data of each student goal according to the behavior trajectory sequence corresponding to the student goal and the preset behavior category classification rule; and determining the classroom score information corresponding to the student goal based on the behavior frequency data of each student goal.

[0058] Here, the behavior classification rules are used to classify students' behavior trajectory data in class. The behavior classification rules are usually set according to the needs of classroom management and teaching objectives, and can help to divide students' behaviors into several representative categories. Based on the behavior trajectory sequence corresponding to each student target and the preset behavior classification rules, the behavior frequency data of each student target can be obtained, that is, the number of times the student target performs a certain behavior in a specific time period.

[0059] For example, the present disclosure summarizes the behavior categories into seven behaviors: raising hands, reading, writing, listening, playing with mobile phones, bowing heads, and lying on the table. In some other embodiments, the behavior categories may also include interacting with others, tidying up desktop items, yawning, stretching, thinking with chin in hand, etc., which are not specifically limited here.

[0060] Specifically, after obtaining the behavior frequency data of each student goal, the classroom scoring information corresponding to the student goal can be determined based on the behavior frequency data of each student goal, so as to achieve a quantitative evaluation of each student's classroom behavior. By determining the classroom scoring information corresponding to the student goal, each student's classroom behavior can be more intuitively understood, and corresponding teaching measures can be taken according to each student's classroom scoring information. For example, if a student's behavior frequency data shows that he frequently raises his hand in class and actively participates in discussions, he may get a higher classroom score; while another student who spends most of his time playing with his mobile phone or looking down may get a lower score. At the same time, teachers can provide personalized tutoring to students based on the scoring information, provide additional support to students with weaker performance, and help them improve their classroom participation; for students with outstanding performance, they can give praise or encouragement to further stimulate their interest in learning.

[0061] It is understandable that, referring to Figure 4 As shown, in order to comprehensively evaluate the performance of students in class, the present disclosure proposes that when determining the class score information corresponding to each student goal based on the behavior frequency data of each student goal, the following steps S401-S403 may be included: S401, constructing a standard for classifying the importance of classroom behaviors based on preset behavior category classification rules.

[0062] Specifically, in a real classroom environment, the importance of the same behavior is not the same at different times. For example, when the teacher asks a simpler question, there are usually more students raising their hands, and the significance of a single hand-raising behavior is relatively low; when asking difficult questions, the number of students raising their hands decreases significantly, and the importance of each hand-raising behavior increases significantly. Therefore, the present disclosure further compares the frequency of classroom behavior with the overall behavior proportion, and constructs a classroom behavior importance classification standard based on a preset behavior category classification rule.

[0063] Among them, the standard for classifying the importance of classroom behavior is an evaluation system for the relative importance of different behavior categories in classroom situations. This disclosure divides seven types of classroom behaviors into two categories: active learning behaviors and passive learning behaviors. Active learning behaviors include raising hands, reading, writing, and listening, while passive learning behaviors include playing with mobile phones, lowering heads, and lying on tables. The importance of each behavior is different. For example, positive behaviors such as raising hands, when the frequency of simultaneous occurrence is low, indicate that students are more focused in class, so the higher the importance; and reading and writing usually have more people occurring at the same time, which means that the overall state of the class is better, because this reflects that students are actively engaged in learning, otherwise it may mean that students are distracted or scattered; and the lower the importance of negative behaviors, the lower the students' classroom participation and learning enthusiasm, such as lying on the table, even if the frequency of occurrence is low, it also reflects that students are not actively participating in classroom learning. Through such a classification standard, students' performance in class can be measured more accurately.

[0064] S402, determining a principal component analysis weight coefficient matrix based on a principal component analysis method and the behavior frequency data of each of the student targets; and determining a factor analysis weight coefficient matrix based on a factor analysis method and the behavior frequency data of each of the student targets.

[0065] It is understandable that in order to more accurately evaluate students' classroom performance, the present disclosure introduces two statistical analysis methods, principal component analysis (PCA) and factor analysis (FA). Both methods can be used to extract the potential structure in the data, reduce the data dimension, thereby simplifying the complexity and improving the analysis efficiency and accuracy.

[0066] Here, the core idea of ​​PCA is to reduce the dimensionality of multidimensional data and extract a few important principal components, thereby achieving information compression and simplification. In classroom behavior analysis, PCA can help convert different types of classroom behavior data (such as hand-raising frequency, writing frequency, etc.) into fewer principal components, which can maximize the retention of variance information of the original data and help determine the relative importance of each behavior. Similar to PCA, factor analysis also aims to extract a few factors from a large number of variables, but it focuses more on analyzing the potential relationship between variables. In classroom behavior analysis, factor analysis can help identify potential factors that affect student performance, such as students' emotional state, classroom environment, etc. Among them, the implementation steps of principal component analysis and factor analysis are similar, including data standardization, covariance matrix calculation, eigenvalue decomposition, etc.

[0067] Specifically, the students' behavior frequency data can be standardized to eliminate the dimensional differences of different behavior characteristics and ensure the comparability of the data. The standardized formula can be expressed as: ; in, It is the standardized data; It is the original data; is the mean of the feature.

[0068] Secondly, the covariance matrix of the standardized data is calculated to measure the correlation between different behaviors. The formula is as follows: ; in, is the covariance matrix; n is the number of samples; is a standardized matrix The transposed matrix of Each element of Represents the covariance between feature i and feature j: ; in, is the value of the i-th feature of the m-th sample, is the mean of the i-th feature; is the value of the jth feature of the mth sample, is the mean of the jth feature.

[0069] Then, by performing eigenvalue decomposition on the covariance matrix, the eigenvalues ​​and eigenvectors are obtained. The specific formula is as follows: ; in, is the covariance matrix The characteristic value of is the corresponding eigenvector. The size of the eigenvalue measures the contribution of the component. Arranging the corresponding eigenvectors in descending order of eigenvalues ​​gives the eigenvector matrix , called the principal component matrix.

[0070] Furthermore, based on the extracted principal components, the eigenvector matrix is ​​extracted The first k columns of , we can get the dimension reduction matrix Each column of the matrix is ​​a direction vector, from left to right, they are the first principal component, the second principal component, ..., the kth principal component. Then, the relationship equation between the principal component and the research item is further established, the formula is as follows: ; in, represents the i-th column of the linear combination coefficient matrix, represents the extracted i-th principal component vector, Represents the eigenvalue corresponding to the i-th principal component vector.

[0071] Furthermore, the linear combination coefficients corresponding to different principal component vectors are weighted according to the variance explanation rate to obtain the comprehensive score coefficient of each feature. The formula is as follows: ; in, It represents the variance explanation rate corresponding to the i-th principal component, and l represents the total number of eigenvalues.

[0072] Here, the variance explanation rate is the proportion of the eigenvalue in the sum of all eigenvalues, and the calculation formula is as follows: ; in, represents the comprehensive score coefficient matrix, and k represents the first k principal components extracted.

[0073] Finally, by normalizing the score coefficients, we get the weight coefficient matrix of each behavior. The formula is as follows: ; in, is the principal component analysis weight coefficient matrix of each feature. Similarly, the factor analysis weight coefficient matrix can be obtained , is the total number of features, It is the value of the comprehensive score coefficient matrix in the i-th column.

[0074] S403, determining the classroom scoring information corresponding to each student goal based on the classroom scoring formula, the principal component analysis weight coefficient matrix, the factor analysis weight coefficient matrix, the classroom behavior importance classification standard, and the behavior frequency data of each student goal.

[0075] Specifically, after obtaining the principal component analysis weight coefficient matrix, the factor analysis weight coefficient matrix and the classroom behavior importance classification standard, the classroom scoring information corresponding to each student goal can be determined based on the classroom scoring formula and the behavior frequency data of individual student goals.

[0076] Among them, the classroom scoring formula can be expressed as: ; ; in, They represent classroom behavior i and classroom behavior j respectively; t represents classroom time; n represents the set of classroom behaviors; represents the comprehensive weight of classroom behavior i at time t; represents the principal component analysis weight of classroom behavior i; represents the factor analysis weight of classroom behavior i; represents the principal component analysis weight of classroom behavior j; represents the factor analysis weight of classroom behavior j; represents the importance coefficient of classroom behavior i at time t in the standard of class behavior importance classification; L represents the classroom score information of student goals; m represents the total class time; represents the frequency of behavior i at time t.

[0077] S105, generating evaluation information corresponding to each student goal based on the classroom scoring information corresponding to each student goal and the goal evaluation condition.

[0078] It is understandable that the evaluation information may include not only the student's participation score in class, but also specific behavior analysis, such as which links the student is active and which links the student is indifferent. The evaluation information can be provided to teachers for further teaching improvements or as part of the student's personal development report. For example, student A may get a higher participation score because he frequently raises his hand and speaks actively in class, while student B may get a lower score because of less participation.

[0079] Here, the evaluation information obtained by the present disclosure can be used as a basis for teachers to improve their teaching. Teachers can adjust teaching methods and strategies according to the student behavior characteristics reflected in the evaluation information, formulate personalized teaching plans for different students' learning situations, and improve teaching effectiveness. On the other hand, the evaluation information can also be used as part of the student's personal development report to provide students with comprehensive learning feedback, help students recognize their strengths and weaknesses, and promote students' self-improvement and development.

[0080] In some possible embodiments, the evaluation information may also include student behavior warning information. Specifically, after obtaining the classroom score information corresponding to each student goal, it is compared with the warning threshold. When the student's classroom score information is lower than the warning threshold, a warning message is generated and notified to relevant personnel through a pop-up window on the teacher side and text messages to the students' parents.

[0081] In order to achieve intuitive display and convenient interaction of student behavior analysis results, this disclosure also proposes a visual display window. This window is designed to clearly display the analysis results and the realization of user interaction. By building an intuitive and easy-to-use interactive page, teachers and researchers can easily view and interpret the results of student behavior analysis, and quickly grasp the classroom dynamics, so as to make timely intervention and adjustment measures to optimize classroom teaching effects.

[0082] Specifically, after generating the assessment information corresponding to each student's goal, the following (a) to (c) may also be included: (a) Determine the overall class situation information based on the classroom score information corresponding to each student’s goal; (b) for each student goal, determining personal information of the student goal based on classroom score information and evaluation information corresponding to the student goal; (c) Displaying the overall situation information of the class and the personal situation information of each student goal in a visualization window.

[0083] It is understandable that by collecting statistics and analyzing the classroom score information of all students in the class, the overall learning situation of the class can be obtained. For example, the average participation score, excellent rate, pass rate and other indicators of the class can be calculated to understand the overall learning level and academic performance of the class. At the same time, the distribution of class participation in different teaching links can also be analyzed to find out the advantages and weaknesses of class learning, and provide reference for teachers' teaching decisions.

[0084] In addition to the overall class situation information, personal situation information also needs to be determined for each student's goal. Personal situation information includes students' participation scores, behavior analysis results, early warning information, etc., which can comprehensively reflect students' performance in class.

[0085] It is understandable that after obtaining the overall information of the class and the personal information of each student target, it can be displayed through the visualization window. Specifically, the visualization window can display the overall information of the class and the personal information of the students in the form of intuitive charts, graphics and text. For example, a bar chart is used to show the distribution of the number of students in different score ranges in the class, a line chart is used to show the trend of the class participation score over time, and a radar chart is used to show the performance of students in different behavioral dimensions. At the same time, the window also provides interactive functions, and teachers can filter, sort and query student information as needed, and conduct in-depth data analysis and research. Through the visualization display window, teachers and researchers can understand students' learning situation more intuitively, find problems in time and take effective measures to solve them, and improve the quality and efficiency of classroom teaching.

[0086] In some possible embodiments, the visualization display window can also be divided into two functional sub-modules: statistical analysis of the overall situation of the class and statistical analysis of individual student situations. The statistical analysis module for individual student situations allows users to select different students for in-depth comparative analysis based on specific student numbers, and can also draw a variety of different charts, including a comparison chart of student classroom participation, a classroom behavior heat map, a student status time series change chart, and an individual behavior tracking frequency chart. The classroom participation of individual students is presented in the form of a bar chart or a line chart, showing the changes in the participation of students in different classes over a period of time, so that teachers can conduct in-depth analysis of students' classroom participation. At the same time, the distribution of individual student behaviors in the classroom is also presented through a heat map, helping teachers analyze the peaks and troughs of students' learning status, and making teaching adjustments through this information. Similarly, the statistical analysis module for the overall situation of the class can be used to provide classroom behavior and participation statistics for the entire class, helping teachers quickly understand the overall situation of the class. This module displays the class participation of all students through interval distribution charts, bar charts showing the frequency of various behaviors of students in class, and heat maps showing the distribution of behaviors during class time, helping teachers to intuitively see the concentrated and dispersed periods of student behaviors. In addition, the system also provides class participation statistics, showing students' class participation in different time periods in the form of line charts or pie charts, and supports users to filter data for specific time periods through drop-down boxes and date selectors, so that teachers can accurately understand the classroom atmosphere.

[0087] The multi-target tracking-driven student-level classroom engagement assessment method and system provided in the embodiments of the present disclosure achieves a comprehensive, objective and quantitative assessment of student classroom engagement through individualized tracking of students through multi-target tracking technology, which helps teachers understand students' classroom performance more accurately, promote personalized teaching and improve classroom management.

[0088] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0089] Based on the same inventive concept, the disclosed embodiment also provides a multi-target tracking-driven student-level classroom participation evaluation system corresponding to the multi-target tracking-driven student-level classroom participation evaluation method. Since the principle of problem solving by the system in the disclosed embodiment is similar to the above-mentioned multi-target tracking-driven student-level classroom participation evaluation method in the disclosed embodiment, the implementation of the system can refer to the implementation of the method, and the repeated parts will not be repeated.

[0090] Reference Figure 5FIG. 1 is a schematic diagram of a multi-target tracking driven student-level classroom participation evaluation system provided by an embodiment of the present disclosure, wherein the system includes: The behavior recognition module is used to obtain the behavior trajectory sequence of each student target through target detection and multi-target tracking technology; The identity association module is used to match the behavior trajectory sequence of each student target with the pre-stored student identity information; A status evaluation module, used to determine classroom score information corresponding to each student goal based on the behavior frequency data of each student goal; The classroom feedback module is used to generate evaluation information corresponding to each student goal based on the classroom scoring information and goal evaluation conditions corresponding to each student goal.

[0091] Based on the same inventive concept, the presently disclosed embodiment also provides a multi-target tracking-driven student-level classroom participation evaluation device corresponding to the multi-target tracking-driven student-level classroom participation evaluation method. Since the principle of solving the problem by the device in the presently disclosed embodiment is similar to the above-mentioned multi-target tracking-driven student-level classroom participation evaluation method in the presently disclosed embodiment, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0092] Reference Figure 6 FIG. 6 is a schematic diagram of a multi-target tracking driven student-level classroom participation evaluation device 600 provided in an embodiment of the present disclosure, wherein the device comprises: The target detection module 601 is used to obtain a classroom video, and perform target detection on multiple students in the classroom video to determine a target candidate box corresponding to each student target; A sequence generation module 602 is used to continuously track the student targets corresponding to each detected target candidate frame based on a multi-target tracking algorithm to generate a behavior trajectory sequence corresponding to each student target; The identity matching module 603 is used to match the behavior trajectory sequence corresponding to each student target with the pre-stored student identity information, and establish a corresponding relationship between the student identity information and the behavior trajectory sequence based on the matching result; The scoring determination module 604 is used to determine the behavior frequency data of each student target according to the behavior trajectory sequence corresponding to the student target and the preset behavior category classification rule; and determine the classroom scoring information corresponding to the student target based on the behavior frequency data of each student target; The student evaluation module 605 is used to generate evaluation information corresponding to each student goal based on the classroom scoring information and the goal evaluation conditions corresponding to each student goal.

[0093] In some possible embodiments, the target detection module 601 is specifically used for: For each frame of the classroom video, multi-scale feature extraction is performed, and based on the multi-scale feature extraction result, a plurality of student targets corresponding to each frame of the image and an initial candidate frame corresponding to each student target are determined; Any one of the multiple student targets corresponding to each frame of the image is used as the student target to be determined, and the following steps are performed: Step 1, calculating the confidence of each initial candidate frame corresponding to the student target to be determined, and sorting the initial candidate frames according to the confidence of each initial candidate frame, and taking the initial candidate frame with the highest confidence as the target candidate frame corresponding to the student target to be determined; Step 2, selecting an initial candidate frame having the largest overlapping area with the target candidate frame from other initial candidate frames except the target candidate frame; if the overlapping area of ​​the initial candidate frame having the largest overlapping area with the target candidate frame and the target candidate frame is greater than a preset threshold, deleting the initial candidate frame having the largest overlapping area with the target candidate frame; Step 3, return to the above step 2 until the overlapping areas of the other initial candidate boxes except the target candidate box and the target candidate box are less than the preset threshold and / or there are no other initial candidate boxes except the target candidate box remaining, take any one of the remaining student targets as a new student target to be determined, and return to step 1.

[0094] In some possible embodiments, the sequence generation module 602 is specifically used to: For each target candidate frame in each frame image, predict the next moment state information of the student target corresponding to the target candidate frame based on the multi-target tracking algorithm and the historical information about the target candidate frame, and determine the short-term state vector corresponding to the student target based on the prediction result; For each target candidate frame in each frame image, the motion consistency measure and the appearance similarity measure corresponding to the student target are determined based on the matching function, the short-time state vector corresponding to the student target and the actual state vector of the target candidate frame; and the information of the student target corresponding to the target candidate frame is adjusted based on the motion consistency measure and the appearance similarity measure corresponding to the student target; and the target candidate frame corresponding to the student target is updated based on the target candidate frame with adjusted information to achieve continuous tracking of the student target.

[0095] In some possible embodiments, the pre-stored student identity information includes student identity information corresponding to multiple student targets; the identity matching module 603 is specifically used to: Determining an initial image of the student target based on the classroom video and the behavior trajectory sequence corresponding to the student target; Performing global feature extraction on the initial image to obtain initial global features; and performing multi-scale local feature extraction on the initial image to obtain initial multi-scale local features; and determining to-be-matched information corresponding to the student target based on a multi-scale feature aggregation strategy, the initial global features, and the initial multi-scale local features; The similarity values ​​between the student identity information corresponding to each student target in the pre-stored student identity information and the information to be matched are determined respectively, and the student identity information corresponding to the student target with the largest similarity value is determined as the target student identity information corresponding to the student target.

[0096] In some possible embodiments, the score determination module 604 is specifically used to: Constructing a standard for classifying the importance of classroom behaviors based on the pre-set rules for classifying behavioral categories; Determine a principal component analysis weight coefficient matrix based on a principal component analysis method and the behavior frequency data of each of the student targets; and determine a factor analysis weight coefficient matrix based on a factor analysis method and the behavior frequency data of each of the student targets; Determine classroom scoring information corresponding to each student goal based on the classroom scoring formula, the principal component analysis weight coefficient matrix, the factor analysis weight coefficient matrix, the classroom behavior importance classification standard, and the behavior frequency data of each student goal; The classroom scoring formula is expressed as: ; ; in, They represent classroom behavior i and classroom behavior j respectively; t represents classroom time; n represents the set of classroom behaviors; represents the comprehensive weight of classroom behavior i at time t; represents the principal component analysis weight of classroom behavior i; represents the factor analysis weight of classroom behavior i; represents the principal component analysis weight of classroom behavior j; represents the factor analysis weight of classroom behavior j; represents the importance coefficient of classroom behavior i at time t in the standard of class behavior importance classification; L represents the classroom score information of student goals; m represents the total class time; represents the frequency of behavior i at time t.

[0097] In some possible embodiments, the student evaluation module 605 is further used to: Determine the overall class situation information based on the classroom score information corresponding to each student's goal; For each student goal, determining personal information of the student goal based on the classroom score information and evaluation information corresponding to the student goal; The overall situation information of the class and the personal situation information of each student goal are displayed in the visualization window.

[0098] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 7 , which is a schematic diagram of the structure of a computer device 700 provided in an embodiment of the present disclosure, including a processor 701, a memory 702, and a bus 703. The memory 702 is used to store execution instructions, including a memory 7021 and an external memory 7022; the memory 7021 is also called an internal memory, which is used to temporarily store the operation data in the processor 701 and the data exchanged with the external memory 7022 such as a hard disk. The processor 701 exchanges data with the external memory 7022 through the memory 7021.

[0099] In the embodiment of the present application, the memory 702 is specifically used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 701. That is, when the computer device 700 is running, the processor 701 communicates with the memory 702 through the bus 703, so that the processor 701 executes the application code stored in the memory 702, and then executes the method described in any of the above embodiments.

[0100] Among them, the memory 702 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0101] Processor 701 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0102] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the computer device 700. In other embodiments of the present application, the computer device 700 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0103] The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the multi-target tracking driven student-level classroom participation evaluation method described in the above method embodiment are executed. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0104] The presently disclosed embodiments also provide a computer program product that carries a program code. The program code includes instructions that can be used to execute the steps of the multi-target tracking-driven student-level classroom participation assessment method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

[0105] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0106] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0108] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0109] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0110] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present disclosure, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A multi-target tracking driven student-level classroom engagement assessment method, characterized in that: include: Obtaining a classroom video, and performing target detection on multiple students in the classroom video to determine a target candidate box corresponding to each student target; Based on the multi-target tracking algorithm, the student targets corresponding to each detected target candidate frame are continuously tracked to generate a behavior trajectory sequence corresponding to each student target; For each student goal, matching the behavior trajectory sequence corresponding to the student goal with the pre-stored student identity information, and establishing a corresponding relationship between the student identity information and the behavior trajectory sequence based on the matching result; Determine the behavior frequency data of each student target according to the behavior trajectory sequence corresponding to the student target and the preset behavior category classification rule; and determine the classroom score information corresponding to the student target based on the behavior frequency data of each student target; Based on the class score information corresponding to each student goal and the goal evaluation condition, evaluation information corresponding to each student goal is generated.

2. The method according to claim 1, characterized in that The performing target detection on the multiple students in the classroom video and determining a target candidate frame corresponding to each student target includes: For each frame of the classroom video, multi-scale feature extraction is performed, and based on the multi-scale feature extraction result, a plurality of student targets corresponding to each frame of the image and an initial candidate frame corresponding to each student target are determined; Any one of the multiple student targets corresponding to each frame of the image is used as the student target to be determined, and the following steps are performed: Step 1, calculating the confidence of each initial candidate frame corresponding to the student target to be determined, and sorting the initial candidate frames according to the confidence of each initial candidate frame, and taking the initial candidate frame with the highest confidence as the target candidate frame corresponding to the student target to be determined; Step 2, selecting an initial candidate frame having the largest overlapping area with the target candidate frame from other initial candidate frames except the target candidate frame; if the overlapping area of ​​the initial candidate frame having the largest overlapping area with the target candidate frame and the target candidate frame is greater than a preset threshold, deleting the initial candidate frame having the largest overlapping area with the target candidate frame; Step 3, return to the above step 2 until the overlapping areas of the other initial candidate boxes except the target candidate box and the target candidate box are less than the preset threshold and / or there are no other initial candidate boxes except the target candidate box remaining, take any one of the remaining student targets as a new student target to be determined, and return to step 1.

3. The method according to claim 2, characterized in that The method of continuously tracking the student targets corresponding to the detected target candidate frames based on the multi-target tracking algorithm includes: For each target candidate frame in each frame image, predict the next moment state information of the student target corresponding to the target candidate frame based on the multi-target tracking algorithm and the historical information about the target candidate frame, and determine the short-term state vector corresponding to the student target based on the prediction result; For each target candidate frame in each frame image, the motion consistency measure and the appearance similarity measure corresponding to the student target are determined based on the matching function, the short-term state vector corresponding to the student target and the actual state vector of the target candidate frame; and the information of the student target corresponding to the target candidate frame is adjusted based on the motion consistency measure and the appearance similarity measure corresponding to the student target; and the target candidate frame corresponding to the student target is updated based on the target candidate frame with adjusted information to achieve continuous tracking of the student target.

4. The method according to claim 1, characterized in that The pre-stored student identity information includes student identity information corresponding to a plurality of student targets; The matching of the behavior trajectory sequence corresponding to the student target with the pre-stored student identity information includes: Determining an initial image of the student target based on the classroom video and the behavior trajectory sequence corresponding to the student target; Performing global feature extraction on the initial image to obtain initial global features; and performing multi-scale local feature extraction on the initial image to obtain initial multi-scale local features; and determining to-be-matched information corresponding to the student target based on a multi-scale feature aggregation strategy, the initial global features, and the initial multi-scale local features; The similarity values ​​between the student identity information corresponding to each student target in the pre-stored student identity information and the information to be matched are determined respectively, and the student identity information corresponding to the student target with the largest similarity value is determined as the target student identity information corresponding to the student target.

5. The method according to claim 1, characterized in that The determining of classroom score information corresponding to the student goal based on the behavior frequency data of each student goal includes: Constructing a standard for classifying the importance of classroom behaviors based on the pre-set rules for classifying behavioral categories; Determine a principal component analysis weight coefficient matrix based on a principal component analysis method and the behavior frequency data of each of the student targets; and determine a factor analysis weight coefficient matrix based on a factor analysis method and the behavior frequency data of each of the student targets; Determine classroom scoring information corresponding to each student goal based on the classroom scoring formula, the principal component analysis weight coefficient matrix, the factor analysis weight coefficient matrix, the classroom behavior importance classification standard, and the behavior frequency data of each student goal; The classroom scoring formula is expressed as: ； ； in, They represent classroom behavior i and classroom behavior j respectively; t represents classroom time; n represents the set of classroom behaviors; represents the comprehensive weight of classroom behavior i at time t; represents the principal component analysis weight of classroom behavior i; represents the factor analysis weight of classroom behavior i; represents the principal component analysis weight of classroom behavior j; represents the factor analysis weight of classroom behavior j; represents the importance coefficient of classroom behavior i at time t in the standard of class behavior importance classification; L represents the classroom score information of student goals; m represents the total class time; represents the frequency of behavior i at time t.

6. The method according to any one of claims 1 to 5, characterized in that After generating the evaluation information corresponding to each student goal, the method further includes: Determine the overall class situation information based on the classroom score information corresponding to each student's goal; For each student goal, determining personal information of the student goal based on the classroom score information and evaluation information corresponding to the student goal; The overall situation information of the class and the personal situation information of each student goal are displayed in the visualization window.

7. A multi-target tracking driven student-level classroom participation evaluation system, characterized in that: include: The behavior recognition module is used to obtain the behavior trajectory sequence of each student target through target detection and multi-target tracking technology; The identity association module is used to match the behavior trajectory sequence of each student target with the pre-stored student identity information; A status evaluation module, used to determine classroom score information corresponding to each student goal based on the behavior frequency data of each student goal; The classroom feedback module is used to generate evaluation information corresponding to each student goal based on the classroom scoring information and goal evaluation conditions corresponding to each student goal.

8. A multi-target tracking driven student-level classroom participation assessment device, characterized in that: include: The target detection module is used to obtain a classroom video, and perform target detection on multiple students in the classroom video to determine a target candidate box corresponding to each student target; A sequence generation module is used to continuously track the student targets corresponding to each detected target candidate frame based on a multi-target tracking algorithm, and generate a behavior trajectory sequence corresponding to each student target; An identity matching module is used to match the behavior trajectory sequence corresponding to each student target with the pre-stored student identity information, and establish a corresponding relationship between the student identity information and the behavior trajectory sequence based on the matching result; A scoring determination module, for determining the behavior frequency data of each student target according to the behavior trajectory sequence corresponding to the student target and the preset behavior category classification rule; and determining the classroom scoring information corresponding to the student target based on the behavior frequency data of each student target; The student evaluation module is used to generate evaluation information corresponding to each student goal based on the classroom scoring information and the goal evaluation conditions corresponding to each student goal.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images