A method for detecting hand raising in a classroom question and answer session

Abstract

The present application relates to the technical field of classroom image processing, and particularly relates to a classroom question and answer hand raising detection method.A VLM model identifies the content of a real-time intercepted courseware image based on a preset prompt word, triggers a detection process when a classroom question and answer task is detected, obtains a panoramic high-definition clear original image based on the collected image, and obtains a marked image with student and seat positioning information through an optimized yolo26 network.The VLM model performs occlusion detection, and if a student is occluded, steps S2 and S3 are repeatedly executed after adjustment until all students are unoccluded.After the question and answer is completed, the VLM model identifies the answer options corresponding to the gestures of the students in the latest marked image one by one based on the student area and dynamic prompt words.Finally, visual statistical results are generated.The present application can quickly and conveniently perform accurate classroom question and answer detection and analysis, guide the attention of students in learning, and achieve low-cost deployment.

Description

Technical Field

[0001] This invention relates to the field of classroom image processing technology, and in particular to a method for detecting hand raising during classroom question and answer. Background Technology

[0002] With the continuous development of society and technology, students need to have a solid grasp of knowledge and improve their learning outcomes. The classroom is one of the most crucial parts of student learning, requiring quick understanding and application of knowledge. However, while teachers lecture throughout the class, students cannot always keep up with the pace. Even slight deviations can lead to disconnected learning, resulting in an inability to systematically grasp knowledge points and severely impacting learning effectiveness. Compared to the traditional teacher-centered lecture and question-and-answer format, there is a pressing need for more effective methods. Ensuring students learn key knowledge points in the classroom, appropriately designing question-and-answer sessions, and stimulating their thinking and interest are extremely important for student learning.

[0003] Traditional oral questioning methods can only cover a small number of students, failing to grasp the overall learning situation of the entire class. Furthermore, answering in front of others can cause students psychological discomfort, inhibiting genuine thinking and affecting knowledge absorption. Existing hardware-software combined classroom question-and-answer detection methods suffer from cumbersome or inaccurate answer recognition. For example, a method and device for rapid review of answers in primary and secondary school classrooms based on deep learning (publication number CN114140282A) requires students to wear physical answer cards and uses OCR to recognize the answers, resulting in high deployment and maintenance costs. Additionally, this solution is tied to student identity, preventing anonymous answering and failing to alleviate students' psychological burden. Another example is a classroom hand-raising behavior detection method (publication number CN114332927A), which can only determine whether a student raised their hand, but cannot identify different hand gestures corresponding to different answer options, nor can it perform option statistics and learning analysis for multiple-choice questions. Furthermore, it is not optimized for scenarios with large classroom depth, small target size for students in the back rows, and easy obstruction, resulting in low detection accuracy for back rows and limited practical effectiveness. Summary of the Invention

[0004] The purpose of this invention is to propose a method and system for detecting hand-raising in classroom question-and-answer sessions, which can quickly and conveniently perform accurate detection and analysis of classroom question-and-answer sessions, guide students' attention in learning, and achieve low-cost deployment.

[0005] To achieve this objective, the present invention adopts the following technical solution:

[0006] A method for detecting hand-raising during classroom questioning includes the following steps:

[0007] S1: Real-time capture of courseware images. The VLM model identifies the content of the current courseware image based on preset prompt words. When a classroom Q&A task is detected, the detection process is triggered.

[0008] S2: The image acquisition device acquires images in the classroom, and performs panoramic depth stitching on the acquired images to obtain a clear panoramic depth original image;

[0009] S3: Based on the optimized YOLO26 network, locate the student area and seat area in the clear panoramic depth original image, and output a marked image with location information;

[0010] S4: The VLM model performs occlusion detection on the marked image based on the preset occlusion rules. If a student is detected to be occluded, the model outputs an adjustment prompt for the student. Then, steps S2 and S3 are repeated until all students are unoccluded.

[0011] S5: After the question and answer session, the VLM model identifies the answer options corresponding to gestures in the student regions of the latest labeled image one by one, based on the student regions and dynamic prompts. The dynamic prompts include the gesture option correspondence rules updated according to the courseware images.

[0012] S6: Calculate the number of people who selected each option and their percentage, and generate a visual statistical result.

[0013] Further, step S2 includes:

[0014] S21: The image acquisition device is fixed in the classroom, and the focal length is adjusted to take multiple images covering the entire depth of the classroom at different focal lengths;

[0015] S22: The Sobel operator is used to calculate the sharpness of each image pixel by pixel. The gradient value is obtained by horizontal gradient convolution and vertical gradient convolution. The pixels with high sharpness in each image are extracted and stitched together according to the pixel coordinates to generate a clear original image with full depth of field.

[0016] Furthermore, the optimization method for the YOLO26 network in step S3 is to reduce the downsampling step size of the P2 branch structure, such that the feature map size and step size satisfy the following: ,

[0017] in, Input image size, S represents the output feature map size, and S represents the downsampling step size.

[0018] Increase the number of channels in the P2 branch to enhance the feature extraction capabilities for fine textures, edges, and minute contours.

[0019] Furthermore, the training method for the optimized YOLO26 network is as follows:

[0020] Classroom images under multiple states are collected. The images collected under each state are stitched together using panoramic depth to obtain a clear panoramic depth original image. Targets are labeled on the clear panoramic depth original image, with the label categories including students and seats, and front and back student areas are distinguished. Back students are used as key samples for small target detection. A training dataset is constructed.

[0021] Based on the basic YOLO26 network architecture, the P2 branch is optimized to obtain an optimized YOLO26 network;

[0022] The training dataset is input into the optimized YOLO26 network for supervised learning. The entire weight parameter of the YOLO network is used as the individual for optimization, and Epoch training is performed.

[0023] After each epoch of training is completed, the learnable weights are iterated and optimized twice using the DOA algorithm, and the optimized weight parameters are output as the initial weights for the next epoch of training.

[0024] Multiple rounds of Epoch training and secondary optimization using the DOA algorithm are performed until the detection accuracy is met.

[0025] Further, step S1 includes:

[0026] The system captures courseware images from the teaching equipment in real time and sends them to the VLM model for real-time inference.

[0027] The VLM model detects whether there is a classroom question-and-answer task in the courseware image that requires students to raise their hands to answer based on preset prompt words. If a classroom question-and-answer task is detected, the detection process is triggered to start step S2.

[0028] Furthermore, the classroom question-and-answer task contained in the courseware image includes questions, options, answer gestures, and statistical forms; the courseware image also contains instructions prompting students to start answering.

[0029] Further, step S4 includes:

[0030] S41: Input the VLM model with a tagged image containing location information and a clear original image of panoramic depth;

[0031] S42: The VLM model performs occlusion detection on the marked image based on preset occlusion rules. It compares the image content row by row and person by person with the location information to determine whether there are occluded body parts or answering areas of students at each position and outputs the occlusion status.

[0032] S43: If no student is obstructed, the detection is passed and proceed to step S5; if there are obstructed students, output the adjustment prompt with the row number of the seat corresponding to the obstructed student, and then repeat steps S2 and S3 until all students are unobstructed.

[0033] Further, step S5 includes:

[0034] After step S43 determines that the test has passed, the question-and-answer session ends.

[0035] Based on the current courseware images, extract the answer rules and generate dynamic prompts;

[0036] The VLM model localization information divides the input labeled image into several image blocks, extracts visual features block by block based on the student's serial number, and identifies the answer options corresponding to the gestures based on dynamic prompts.

[0037] After iterating through all students, if there are any unrecognizable gestures, output a message indicating incomplete statistics and output the structured answer results.

[0038] Furthermore, in the step of extracting answer rules based on the current courseware image and generating dynamic prompts, the answer rules include the correspondence between gestures and options, question types, and statistical output requirements;

[0039] In step S6, visual statistical results are generated based on the dynamic prompt words.

[0040] A classroom question-and-answer hand-raising detection system is used to perform the above-mentioned classroom question-and-answer hand-raising detection method;

[0041] The system includes:

[0042] The image acquisition module is used to acquire images inside the classroom;

[0043] The courseware capture module is used to capture courseware images from the teaching computer in real time;

[0044] The data processing module, equipped with an optimized YOLO26 network and VLM model, is used to perform panoramic depth stitching on the acquired images to obtain a clear panoramic depth original image, acquire marked images with positioning information, perform occlusion detection, identify the answer options corresponding to each student's gesture, and generate visual statistical results.

[0045] The interactive prompt module is used to output prompts for occlusion adjustment or to visualize statistical results.

[0046] The technical solution provided by this invention may include the following beneficial effects:

[0047] The system enhances the teaching experience and effectiveness by employing an anonymous gesture-based answering mode, eliminating students' psychological awkwardness when answering questions in front of others and enabling full participation and interaction from the entire class. Real-time visualization of statistical results allows teachers to quickly grasp the overall learning situation and identify knowledge gaps, while students can reinforce their understanding of key concepts through feedback, significantly improving classroom teaching effectiveness. From triggering Q&A sessions on courseware, automatic occlusion verification, automatic gesture recognition to automatic result generation, the entire process requires no manual operation from the teacher, ensuring uninterrupted teaching and significantly saving classroom interaction time. Furthermore, the system can directly reuse existing classroom cameras and teaching computers, achieving all functions with only the accompanying software. Its low deployment threshold and maintenance costs make it suitable for large-scale deployment in ordinary classrooms.

[0048] The method of this invention is specifically adapted to complex classroom scenarios. It solves the problem of blurred imaging in the back row by using panoramic depth stitching from a single camera. It improves YOLO26 from two dimensions: P2 branch structure optimization and DOA weight secondary optimization to enhance the detection accuracy of small targets in the back row. It adopts an architecture that combines YOLO26 coarse localization with VLM fine recognition to balance detection efficiency and versatility. It also has the ability to automatically recognize VLM courseware and generate dynamic prompt words, and supports teachers to customize answer rules and display formats. Attached Figure Description

[0049] Figure 1 This is a flowchart of a classroom question-and-answer hand-raising detection method according to an embodiment of the present invention. Detailed Implementation

[0050] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that the specific embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit the present invention.

[0051] Reference Figure 1 An embodiment of the present invention provides a method for detecting hand-raising in classroom question-and-answer sessions, comprising the following steps:

[0052] S1: Real-time capture of courseware images. The VLM model identifies the content of the current courseware image based on preset prompt words. When a classroom Q&A task is detected, the detection process is triggered.

[0053] S2: The image acquisition device acquires images in the classroom, and performs panoramic depth stitching on the acquired images to obtain a clear panoramic depth original image;

[0054] S3: Based on the optimized YOLO26 network, locate the student area and seat area in the clear panoramic depth original image, and output a marked image with location information;

[0055] S4: The VLM model performs occlusion detection on the marked image based on the preset occlusion rules. If a student is detected to be occluded, the model outputs an adjustment prompt for the student. Then, steps S2 and S3 are repeated until all students are unoccluded.

[0056] S5: After the question and answer session, the VLM model identifies the answer options corresponding to gestures in the student regions of the latest labeled image one by one, based on the student regions and dynamic prompts. The dynamic prompts include the gesture option correspondence rules updated according to the courseware images.

[0057] S6: Calculate the number of people who selected each option and their percentage, and generate a visual statistical result.

[0058] The method of this invention features improved teaching experience and effectiveness, full-process automation, enhanced classroom efficiency, and highly targeted algorithms adapted to complex classroom scenarios. Specifically, the panoramic depth stitching solution can solve the problems of large classroom depth and blurred images of the back rows using only a single camera, providing high-quality input for subsequent detection without the need for a multi-camera network.

[0059] This solution uses a VLM model to identify courseware content in real time. When a classroom question-and-answer task is detected, the detection process is automatically triggered without interrupting the teaching rhythm. The VLM model automatically recognizes courseware and completes gesture recognition based on generated dynamic prompts, allowing teachers to customize answer gestures, question types, and presentation formats without pre-fixing program rules. It adopts a two-level architecture of YOLO26 coarse localization and VLM fine recognition. YOLO26 ensures detection speed, while VLM enables flexible adaptation to customized gestures. Precise coordinate input also effectively reduces VLM inference errors, balancing efficiency and versatility.

[0060] For example, a classroom has 10 rows of seats, with 15 seats in each row. A general-purpose camera is installed at the front of the classroom and connected to a teaching computer. During class, the teacher plays a PPT presentation on the teaching computer. When the teacher turns to the multiple-choice question, the camera automatically recognizes and starts the detection, sequentially completing image acquisition, area positioning, occlusion verification, and gesture recognition. Finally, a statistical chart pops up next to the presentation, without the teacher needing to operate the mouse at all.

[0061] In one embodiment of the present invention, step S2 includes:

[0062] S21: The image acquisition device is fixed in the classroom, and the focal length is adjusted to take multiple images covering the entire depth of the classroom at different focal lengths;

[0063] S22: The Sobel operator is used to calculate the sharpness of each image pixel by pixel. The gradient value is obtained by horizontal gradient convolution and vertical gradient convolution. The pixels with high sharpness in each image are extracted and stitched together according to the pixel coordinates to generate a clear original image with full depth of field.

[0064] The preferred image acquisition device is a camera. However, automatically acquired data may suffer from pixel blurring due to the large distance between students in the classroom and those outside the depth of field. This solution uses a single camera to capture at least three images from near, middle, and far distances, maintaining the same composition. Sharpness is calculated pixel-by-pixel using the Sobel operator, and the images are then processed through horizontal and vertical gradient convolutions to obtain G. x and G y The gradient value, and the formula for calculating sharpness is: , The pixel sharpness at coordinates (x, y);

[0065] During stitching, the sharpness values ​​of the three images are compared coordinate by coordinate, and the pixel with the highest sharpness at the same position is retained. Finally, a full-view depth image with clear front and back rows is generated, which is used for model training and inference.

[0066] 3. The method according to claim 1, characterized in that the optimization method of the optimized YOLO26 network in step S3 is to reduce the downsampling step size of the P2 branch structure, wherein the feature map size and step size satisfy: ,

[0067] in, Input image size, S represents the output feature map size, and S represents the downsampling step size.

[0068] Increase the number of channels in the P2 branch to enhance the feature extraction capabilities for fine textures, edges, and minute contours.

[0069] Specifically, addressing the issue of small targets and weak features for students in the back rows, the YOLO26 network's high-resolution P2 branch for small targets is optimized: the downsampling stride is reduced, for example, from 8 to 4. With the input size unchanged, the information density of the output feature map doubles, preserving more details of small targets. Simultaneously, the number of convolutional channels in the P2 branch is increased, for example, the branch channel configuration is adjusted from [-1,2,C3k2,[128,True]] to [-1,2,C3k2,[256,True]], enhancing the network's ability to extract features from fine textures, edges, and small contours, adapting to the detection needs of students at a distance.

[0070] P2 is a shallow high-resolution feature branch of the backbone network with a 4-fold downsampling (Stride=4). When the input is a default 640×640 image, the P2 feature map size is 160×160 and the number of channels is 128 / 256. It belongs to the native scalable fourth-scale detection branch of YOLO26 (the native YOLO26 only uses three layers, P3 / P4 / P5, by default, and P2 belongs to the small target enhancement extension branch).

[0071] To enhance the optimized YOLO26 network's ability to represent features of small targets and low-quality blurred targets in the back row, in one embodiment of the present invention, the training method of the optimized YOLO26 network is as follows:

[0072] Classroom images under multiple states are collected. The images collected under each state are stitched together using panoramic depth to obtain a clear panoramic depth original image. Targets are labeled on the clear panoramic depth original image, with the label categories including students and seats, and front and back student areas are distinguished. Back students are used as key samples for small target detection. A training dataset is constructed.

[0073] Based on the basic YOLO26 network architecture, the P2 branch is optimized to obtain an optimized YOLO26 network;

[0074] The training dataset is input into the optimized YOLO26 network for supervised learning. The entire weight parameter of the YOLO network is used as the individual for optimization, and Epoch training is performed.

[0075] After each epoch of training is completed, the learnable weights are iterated and optimized twice using the DOA algorithm, and the optimized weight parameters are output as the initial weights for the next epoch of training.

[0076] Multiple rounds of Epoch training and secondary optimization using the DOA algorithm are performed until the detection accuracy is met.

[0077] This solution improves YOLO26 from two dimensions: P2 branch structure optimization and DOA weight secondary optimization. It specifically addresses the industry challenges of low detection accuracy for small targets in the back row and the suppression of gradients for small targets by large target loss, resulting in strong consistency in detection performance across the entire classroom. This training method addresses the problem of large target loss dominating and diluting the gradient of small targets in traditional training at the training strategy level. It balances the parameter update amplitudes of large and small targets, alleviates the suppression problem by large target loss, and strengthens the model's ability to represent features of small targets and low-quality blurred targets in the back row. Without modifying the network structure or loss function, only secondary optimization of weights is added after each training round, achieving low cost, high adaptability, and compatibility with the small target optimization needs of various YOLO series models.

[0078] Specifically, the training steps for the optimized YOLO26 network are as follows:

[0079] T1: Collect classroom scenes with different lighting conditions and shooting angles. All images are first processed by panoramic depth-of-view enhancement and stitching to obtain clear and consistent training original images. Targets are labeled in the images, with the label categories including students and desks. At the same time, the front and back student areas are distinguished, and the back students are used as the key samples for small target detection.

[0080] T2: Data Augmentation and Input Standardization: Perform operations such as random horizontal flipping, brightness / contrast perturbation, random cropping, and mosaic enhancement on the training set images to simulate complex classroom lighting and perspective changes; scale all images to the model's specified input size and normalize pixels to form standardized training input.

[0081] T3: Divide the labeled dataset into training, validation and test sets according to a preset ratio to ensure that each subset covers classroom scenarios with different depths of field, different lighting and different number of students, and that the proportion of small target samples is balanced to avoid distribution shift.

[0082] T4: Based on the basic YOLO26 network architecture, the structure was modified to meet the needs of small target detection in the back row of the classroom, and the model initialization was completed.

[0083] Adjust the downsampling step size of the P2 branch: Modify the downsampling step size of the high-resolution P2 branch for small targets in the Neck layer; while keeping the input image size unchanged, adjust the step size from 8 to 4 to improve the resolution of the P2 feature map and retain more detailed information of small targets in the back row;

[0084] Increase the number of convolution channels in the P2 branch, for example, adjust the branch channel configuration from [-1,2,C3k2,[128,True]] to [-1,2,C3k2,[256,True]], to enhance the network's ability to extract features from fine textures, edges, and small contours, and adapt to the detection needs of students with weak features in the back row;

[0085] Detection head adaptation and adjustment: Modify the number of output channels of the detection head to match the classification and regression output requirements of students, tables, and two types of targets, and complete the initialization of the overall network structure.

[0086] T4: Regular Supervised Training Phase (Single Epoch Main Training Process) Each training epoch first performs regular supervised learning to complete the basic weight update:

[0087] Forward feature propagation: Batch training images are input into the model, multi-scale features are extracted by the backbone network, and multi-scale feature fusion is completed by the Neck layer. Finally, the target bounding box coordinates, class confidence and classification probability are output by the detection head.

[0088] Multi-task loss calculation: Calculate the total network loss, which includes three parts: bounding box regression loss (using CIoU loss), target confidence loss, and classification loss. In this stage, the default global weight balancing is used, and the loss contribution of students with large targets in the front row is higher.

[0089] Backpropagation and weight update: A gradient descent optimizer (such as AdamW) is used to perform backpropagation based on the total loss to update all learnable weights of the network; during training, a cosine annealing strategy is used to dynamically adjust the learning rate to improve convergence stability;

[0090] Phase output: After completing the regular training of a single epoch, the basic weight parameters of the current epoch are obtained; however, the weights suffer from the problem of large target loss dominating and the gradient of small target in the later stages being diluted, so a secondary optimization phase is needed for targeted optimization.

[0091] T5: After each epoch of regular training, while keeping the mapping between network input features and backbone features unchanged, the DOA algorithm is introduced to perform a second iteration to optimize the network's learnable weights, compensating for gradient defects in small objectives.

[0092] The initial optimization population is all the learnable weights obtained from the training at the current epoch. The optimization targets include the convolutional layer weights, BN layer weights, and bias parameters of the backbone network and the detection head.

[0093] Leveraging the forgetting filter mechanism of DOA, weight dimensions associated with small targets, weak edges, and ambiguous features are identified and prioritized for optimization. Weight dimensions dominated by large targets retain their original training values ​​to avoid disrupting the learned representation of large target features. Combining DOA's memory retention strategy with the dream information sharing mechanism, multiple rounds of iterative optimization are performed. The weight update formula is as follows:

[0094] ,

[0095] In the formula, Let be the update value of the i-th weighted individual in the j-th dimension at the (t+1)-th iteration. The historical best weight value for the j-th dimension. , Rand represents the upper and lower bounds of the j-th dimension parameter. A random number within a given interval, where t is the current iteration number. This represents the maximum number of iterations in a single round of quadratic optimization. By iteratively updating the weight dimensions corresponding to smaller objectives, the parameter update magnitudes of both large and small objectives are balanced, mitigating the problem of loss suppression for larger objectives.

[0096] Secondary weight output: After reaching the maximum number of iterations or when the verification accuracy tends to stabilize, output the weight parameters after secondary optimization, which will be used as the initial weights for the next round of regular training.

[0097] T6: A fixed number of epochs are used at each interval to evaluate the model's accuracy using a validation set. The core evaluation metrics include overall detection mAP, individual AP of students with small target scores in the back row, and single-graph inference speed.

[0098] If the verification accuracy reaches the preset threshold and does not improve for several consecutive rounds, the model is considered to have converged, training is terminated, and the final YOLO26 detection model is output; if it does not meet the threshold, the process returns to S3 to continue the next round of training iterations until the accuracy requirement is met.

[0099] In one embodiment of the present invention, step S1 includes:

[0100] The system captures courseware images from the teaching equipment in real time and sends them to the VLM model for real-time inference.

[0101] The VLM model detects whether there is a classroom question-and-answer task in the courseware image that requires students to raise their hands to answer based on preset prompt words. If a classroom question-and-answer task is detected, the detection process is triggered to start step S2.

[0102] Preferably, the classroom question-and-answer task contained in the courseware image includes questions, options, answer gestures, and statistical forms; the courseware image also includes instructions prompting students to start answering.

[0103] Specifically, the program captures the screen of the teaching computer's courseware at a fixed frequency (e.g., 500ms / time or 2s / time), decodes it, and continuously inputs it into the VLM model, operating in a silent background monitoring state. Preset prompts clearly define the VLM's recognition target, determining whether the screen contains classroom questions and answers or gesture-based responses. If so, it extracts three core types of information: question type, gesture rules, and statistical format. The VLM simultaneously analyzes the textual semantics and visual elements (option numbers, gesture diagrams, and answer instructions) in the courseware, comprehensively determining whether to trigger the detection process, thus avoiding accidental triggering of ordinary explanation pages.

[0104] As an example, the pre-set prompt is "Identify whether the current PPT contains multiple-choice or true / false questions requiring hand gestures; if so, extract the question type, gesture correspondence rules, and result chart type"; during the lesson, when the PPT is turned to a knowledge point explanation page, VLM outputs "No question-and-answer task" and continues to monitor; when the PPT is turned to a multiple-choice or true / false question page containing the instruction "Please make a gesture to answer", VLM determines that a question-and-answer task has been detected and automatically starts the subsequent image acquisition process.

[0105] For example, a multiple-choice question courseware page includes: the question stem "Which of the following belong to deep learning frameworks?", four options A and D, instructions to answer "Extend the fingers corresponding to the order of the options, extend them simultaneously if multiple selections are made", result display requirements "A bar chart showing the number of people", and the instruction text "Please start answering"; VLM can completely extract all of the above information and use it to generate dynamic prompts later. Setting dynamic prompts improves teaching flexibility.

[0106] To improve the efficiency of occlusion detection, in one embodiment of the present invention, step S4 includes:

[0107] S41: Input the VLM model with a tagged image containing location information and a clear original image of panoramic depth;

[0108] S42: The VLM model performs occlusion detection on the marked image based on preset occlusion rules. It compares the image content row by row and person by person with the location information to determine whether there are occluded body parts or answering areas of students at each position and outputs the occlusion status.

[0109] S43: If no student is obstructed, the detection is passed and proceed to step S5; if there are obstructed students, output the adjustment prompt with the row number of the seat corresponding to the obstructed student, and then repeat steps S2 and S3 until all students are unobstructed.

[0110] Preferably, the preset occlusion rules also include the number of rows of classroom seats, the number of seats per row, and the total number of students. This data is also used to generate statistical results. Occlusion detection uses specific prompts to clearly define the criteria: if a student's hands and upper limbs in the marked image are obscured by objects such as students in front, desks, or chairs, making it impossible to fully recognize their gestures, it is considered occlusion. The output must also include the global occlusion status and the corresponding row number. The prompts are simultaneously output via voice broadcast or text displayed on the teaching computer, accurately indicating the location of obscured students without requiring teachers to check each student individually.

[0111] Further, step S5 includes:

[0112] After step S43 determines that the test has passed, the question-and-answer session ends.

[0113] Based on the current courseware images, extract the answer rules and generate dynamic prompts;

[0114] The VLM model localization information divides the input labeled image into several image blocks, extracts visual features block by block based on the student's serial number, and identifies the answer options corresponding to the gestures based on dynamic prompts.

[0115] After iterating through all students, if there are any unrecognizable gestures, output a message indicating incomplete statistics and output the structured answer results.

[0116] Preferably, in the step of extracting answer rules based on the current courseware image and generating dynamic prompts, the answer rules include the correspondence between gestures and options, question types, and statistical output requirements;

[0117] In step S6, visual statistical results are generated based on the chart requirements in the dynamic prompts.

[0118] Specifically, the VLM visual encoder divides the student regions in the labeled image into image blocks, extracts visual features such as edges, contours, colors, and structures block by block, and outputs a high-dimensional feature vector with gesture information. It then quickly locates the corresponding region using the student region coordinates, inferring from student number i to the total number of students s. If a student's gesture is not standard or cannot be recognized, the number i is not updated. After traversal, if there are unrecognized samples, the program displays a "Statistics incomplete" prompt, which can either return to re-capture the image for recognition or determine whether to re-capture the image based on a threshold. For example, with a threshold of 90%, if the number of recognized gestures is greater than 95% of the total number of students, a "Statistics incomplete" prompt will appear and the statistical results will be output; if the number of recognized gestures is less than or equal to 95% of the total number of students, a "Retake Image for Recognition" prompt will appear, and the image will be re-captured. Finally, a structured text result is output, for example, the multiple-choice answer for the i-th student is [B, C, D].

[0119] The visual statistical results generated in step S6 can be in the form of pie charts, bar charts, column charts, or custom charts, indicated by the dynamic prompts. Generating different chart styles increases the interest and intuitiveness for students viewing the results, while also providing the percentage of students who selected the first option, facilitating data analysis and guidance for teachers on incorrect answers.

[0120] Accordingly, the present invention also provides a classroom question-and-answer hand-raising detection system, and the above-mentioned classroom question-and-answer hand-raising detection method;

[0121] The system includes:

[0122] The image acquisition module is used to acquire images inside the classroom;

[0123] The courseware capture module is used to capture courseware images from the teaching computer in real time;

[0124] The data processing module, equipped with an optimized YOLO26 network and VLM model, is used to perform panoramic depth stitching on the acquired images to obtain a clear panoramic depth original image, acquire marked images with positioning information, perform occlusion detection, identify the answer options corresponding to each student's gesture, and generate visual statistical results.

[0125] The interactive prompt module is used to output prompts for occlusion adjustment or to visualize statistical results.

[0126] The system of this invention can directly reuse existing classroom cameras and teaching computers, and can achieve all functions with only supporting software. There is no need to purchase special answer devices, interactive terminals and other hardware. The deployment threshold and maintenance cost are low, and it is suitable for large-scale popularization in ordinary classrooms.

[0127] Specifically, the image acquisition module can directly reuse existing classroom surveillance cameras or be paired with a dedicated detection device. The dedicated device includes a magnetic adsorption module, a bracket module, a pan-tilt adjustment module, and a camera module, supporting vertical and horizontal rotation and focus adjustment. It connects to the teaching computer via a USB port. All modules are connected using screws and sliding bearings, making installation convenient. The courseware capture module and data processing module can be deployed locally on the teaching computer, eliminating the need for cloud transmission and ensuring classroom data privacy. For example, the interactive prompt module includes a voice broadcast unit and a projection display unit. The voice unit is used to issue start and end commands for answering questions and to provide occlusion adjustment prompts, while the projection unit is used to display visualized statistical results. It should be noted that in other embodiments, occlusion adjustment prompts and visualized statistical results are directly displayed on the teaching computer.

[0128] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0129] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory / random access memory, magnetic disk, optical disk) and includes several instructions to cause a multimedia terminal device (which may be a mobile phone, computer, television receiver, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0131] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for detecting hand-raising in classroom question-and-answer sessions, characterized in that, Includes the following steps: S1: Real-time capture of courseware images. The VLM model identifies the content of the current courseware image based on preset prompt words. When a classroom Q&A task is detected, the detection process is triggered. S2: The image acquisition device acquires images in the classroom, and performs panoramic depth stitching on the acquired images to obtain a clear panoramic depth original image; S3: Based on the optimized YOLO26 network, locate the student area and seat area in the clear panoramic depth original image, and output a marked image with location information; S4: The VLM model performs occlusion detection on the marked image based on the preset occlusion rules. If a student is detected to be occluded, the model outputs an adjustment prompt for the student. Then, steps S2 and S3 are repeated until all students are unoccluded. S5: After the question and answer session, the VLM model identifies the answer options corresponding to gestures in the student regions of the latest labeled image one by one, based on the student regions and dynamic prompts. The dynamic prompts include the gesture option correspondence rules updated according to the courseware images. S6: Calculate the number of people who selected each option and their percentage, and generate a visual statistical result.

2. The method according to claim 1, characterized in that, Step S2 includes: S21: The image acquisition device is fixed in the classroom, and the focal length is adjusted to take multiple images covering the entire depth of the classroom at different focal lengths; S22: The Sobel operator is used to calculate the sharpness of each image pixel by pixel. The gradient value is obtained by horizontal gradient convolution and vertical gradient convolution. The pixels with high sharpness in each image are extracted and stitched together according to the pixel coordinates to generate a clear original image with full depth of field.

3. The method according to claim 1, characterized in that, The optimization method for the YOLO26 network in step S3 is to reduce the downsampling step size of the P2 branch structure, such that the feature map size and step size satisfy the following: , in, Input image size, S represents the output feature map size, and S represents the downsampling step size. Increase the number of channels in the P2 branch to enhance the feature extraction capabilities for fine textures, edges, and minute contours.

4. The method according to claim 3, characterized in that, The training method for the optimized YOLO26 network is as follows: Classroom images under multiple states are collected. The images collected under each state are stitched together using panoramic depth to obtain a clear panoramic depth original image. Targets are labeled on the clear panoramic depth original image, with the label categories including students and seats, and front and back student areas are distinguished. Back students are used as key samples for small target detection. A training dataset is constructed. Based on the basic YOLO26 network architecture, the P2 branch is optimized to obtain an optimized YOLO26 network; The training dataset is input into the optimized YOLO26 network for supervised learning. The entire weight parameter of the YOLO network is used as the individual for optimization, and Epoch training is performed. After each epoch of training is completed, the learnable weights are iterated and optimized twice using the DOA algorithm, and the optimized weight parameters are output as the initial weights for the next epoch of training. Multiple rounds of Epoch training and secondary optimization using the DOA algorithm are performed until the detection accuracy is met.

5. The method according to claim 1, characterized in that, Step S1 includes: The system captures courseware images from the teaching equipment in real time and sends them to the VLM model for real-time inference. The VLM model detects whether there is a classroom question-and-answer task in the courseware image that requires students to raise their hands to answer based on preset prompt words. If a classroom question-and-answer task is detected, the detection process is triggered to start step S2.

6. The method according to claim 1, characterized in that, The courseware images contain classroom question-and-answer tasks including questions, options, answer gestures, and statistical formats; the courseware images also contain instructions prompting students to start answering.

7. The method according to claim 1, characterized in that, Step S4 includes: S41: Input the VLM model with a tagged image containing location information and a clear original image of panoramic depth; S42: The VLM model performs occlusion detection on the marked image based on preset occlusion rules. It compares the image content row by row and person by person with the location information to determine whether there are occluded body parts or answering areas of students at each position and outputs the occlusion status. S43: If no student is obstructed, the detection is passed and proceed to step S5; if there are obstructed students, output the adjustment prompt with the row number of the seat corresponding to the obstructed student, and then repeat steps S2 and S3 until all students are unobstructed.

8. The method according to claim 7, characterized in that, Step S5 includes: After step S43 determines that the test has passed, the question-and-answer session ends. Based on the current courseware images, extract the answer rules and generate dynamic prompts; The VLM model localization information divides the input labeled image into several image blocks, extracts visual features block by block based on the student's serial number, and identifies the answer options corresponding to the gestures based on dynamic prompts. After iterating through all students, if there are any unrecognizable gestures, output a message indicating incomplete statistics and output the structured answer results.

9. The method according to claim 8, characterized in that, In the step of extracting answer rules based on the current courseware image and generating dynamic prompts, the answer rules include the correspondence between gestures and options, question types, and statistical output requirements. In step S6, visual statistical results are generated based on the dynamic prompt words.

10. A classroom question-and-answer hand-raising detection system, characterized in that, Used to perform the classroom question-and-answer hand-raising detection method according to any one of claims 1-9; The system includes: The image acquisition module is used to acquire images inside the classroom; The courseware capture module is used to capture courseware images from the teaching computer in real time; The data processing module, equipped with an optimized YOLO26 network and VLM model, is used to perform panoramic depth stitching on the acquired images to obtain a clear panoramic depth original image, acquire marked images with positioning information, perform occlusion detection, identify the answer options corresponding to each student's gesture, and generate visual statistical results. The interactive prompt module is used to output prompts for occlusion adjustment or to visualize statistical results.

Images