Examination room behavior analysis method based on deep learning, storage medium and equipment
By constructing an examination behavior detection model based on spatial flow and temporal flow, the shortcomings of traditional manual invigilation and existing deep learning methods in detecting examination behavior are addressed, achieving high-precision and continuous effective detection of examination behavior and reducing invigilation costs.
Patent Information
- Application Number
- CN202511926126.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-03-20
AI Technical Summary
Traditional human proctoring suffers from attentional bias, blind spots, and subjective judgment bias, which leads to the failure to detect subtle movements in the examination room in a timely manner. Furthermore, existing deep learning-based examination behavior analysis methods are unable to simultaneously detect instantaneous, continuous, and interactive behaviors, resulting in decreased detection performance.
A test-taking behavior detection model based on spatial flow and temporal flow is constructed. Intra-frame features are extracted from the video stream through spatial flow structure, and inter-frame features are extracted through temporal flow structure. These features are then fused through a feature concatenation module. The model training is optimized using a multi-objective loss function to achieve continuous and effective detection of test-taking behavior.
It improves the accuracy and sustained effectiveness of exam behavior detection, reduces the probability of false alarms and omissions, and saves the workload of invigilators.
Smart Images
Figure CN121708548A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of examination behavior analysis technology, specifically to a deep learning-based examination behavior analysis method, storage medium, and device. Background Technology
[0002] Traditional manual invigilation suffers from limitations such as invigilators' attention spans, blind spots, and subjective judgment, leading to the failure to detect subtle movements during exams. To address this, many important exams typically employ at least two invigilators, resulting in high labor costs. Furthermore, prolonged periods of intense invigilation can cause significant fatigue for invigilators. Therefore, there is an urgent need for an objective and comprehensive automated technology for analyzing exam behavior.
[0003] Deep learning-based exam behavior analysis has become a research hotspot in the intersection of computer vision and educational technology in recent years. With breakthroughs in computer vision technology, current deep learning-based visual processing methods for exam behavior analysis meet the needs of the times, enabling automatic detection of exam behavior information and reducing costs. However, the exam environment is complex, with interference from lighting changes, partial occlusion, and multiple people interacting. Current deep learning-based visual processing methods are prone to performance degradation due to missing information or noise. Furthermore, they struggle to simultaneously detect various exam behaviors, including instantaneous, continuous, and interactive behaviors, leading to some scenarios failing to be detected and impacting the sustained effectiveness of exam behavior detection throughout the entire invigilation period. Summary of the Invention
[0004] To address the problems existing in the prior art, this invention provides a deep learning-based method, storage medium, and device for analyzing exam behavior. By fusing spatial and temporal flow features, it greatly improves the accuracy of exam behavior detection and remains effective throughout the entire invigilation process.
[0005] To achieve the above technical objectives, the present invention adopts the following technical solution: A deep learning-based method for analyzing exam behavior includes the following steps: Step S1: Collect the video stream from the examination room and manually mark the areas and types of abnormal behavior; Step S2: Construct an examination behavior detection model based on spatial flow and temporal flow; Step S3: Use the collected exam room video stream as input to the exam room behavior detection model, and use manually labeled abnormal behavior areas and abnormal behavior types as labels to train the exam room behavior detection model; Step S4: Repeat step S3 until the multi-objective loss function converges, completing the training of the examination behavior detection model; Step S5: Input the real-time video stream of the examination room into the trained examination room behavior detection model to predict the areas and types of abnormal behavior in real time.
[0006] Furthermore, the examination room behavior detection model includes a backbone network, a neck network, and a head network. The backbone network consists of a spatial flow structure, a temporal flow structure, and a feature concatenation module. The spatial flow structure is used to extract intra-frame features from the examination room video stream, the temporal flow structure is used to extract inter-frame features from the examination room video stream, and the feature concatenation module is used to concatenate and fuse intra-frame and inter-frame features. The neck network is used to perform cross-scale fusion of the concatenated features to obtain multi-scale feature maps. The head network predicts the abnormal behavior region and type based on the multi-scale feature maps.
[0007] Furthermore, the spatial flow structure includes a first AKconv module, a second AKconv module, a first C3K2 module, a third AKConv module, a second C3K2 module, a fourth AKConv module, a third C3K2 module, a fifth AKConv module, a fourth C3K2 module, and an SPPF module, which are sequentially connected in series. The first AKconv module, the second AKconv module, the third AKConv module, the fourth AKConv module, and the fifth AKConv module extract intra-frame local features that enhance semantic information. The first C3K2 module, the second C3K2 module, the third C3K2 module, and the fourth C3K2 module perform dimensionality reduction and fusion of the intra-frame local features to obtain intra-frame features at different scales. The SPPF module is used to perform multi-scale pooling on the intra-frame features at different scales, fusing the intra-frame features at different scales into a fixed-length vector to obtain the final intra-frame features.
[0008] Furthermore, the temporal stream structure includes a first TAKConv module, a second TAKConv module, a first TC3K2 module, a third TAKConv module, a second TC3K2 module, a fourth TAKConv module, a third TC3K2 module, a fifth TAKConv module, a fourth TC3K2 module, and a TCN module, which are connected in series. The first TAKConv module, the second TAKConv module, the third TAKConv module, the fourth TAKConv module, and the fifth TAKConv module capture the temporal continuity of the inter-frame features of the video stream to obtain temporal stacking features. The first TC3K2 module, the second TC3K2 module, the third TC3K2 module, and the fourth TC3K2 module fuse the temporal stacking features into 3D temporal features. The TCN module performs long-term temporal dependency analysis on the 3D temporal features to obtain the inter-frame features of the video stream.
[0009] Furthermore, the feature splicing module splices intra-frame features and inter-frame features along the channel dimension to obtain spliced and fused features.
[0010] Furthermore, the neck network includes a first Upsample module, a first Contat module, a fifth C3K2 module, a second Upsample module, a second Contat module, a sixth C3K2 module, a sixth AKconv module, a third Contat module, a seventh C3K2 module, a seventh AKconv module, a fourth Contat module, and an eighth C3K2 module connected in sequence. The output of the fifth C3K2 module is also connected to the input of the third Contat module. The features obtained by splicing and fusing intra-frame features and inter-frame features are used as the inputs of the first Upsample module, the first Contat module, the second Contat module, and the fourth Contat module, respectively, to obtain multi-scale feature maps.
[0011] Furthermore, the head network includes a first head module, a second head module, and a third head module connected in parallel, used to capture multi-scale information of multi-scale feature maps and predict abnormal behavior regions and types; wherein, the input of the first head module is connected to the output of the sixth C3K2 module, the input of the second head module is connected to the output of the third head module, and the input of the third head module is connected to the output of the eighth C3K2 module.
[0012] Furthermore, the calculation process of the multi-objective loss function is as follows:
[0013] in, Represents a multi-objective loss function. Represents the cross-entropy loss function. express The weighting coefficients, This represents the localization loss function. express The weighting coefficients, Represents the confidence loss function. express The weighting coefficients, This represents the action classification loss function. express The weighting coefficients.
[0014] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program that causes a computer to execute the deep learning-based examination behavior analysis method described above.
[0015] Furthermore, the present invention also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the deep learning-based examination behavior analysis method.
[0016] Compared with existing technologies, the present invention has the following beneficial effects: The examination behavior analysis method based on deep learning of the present invention constructs an examination behavior detection model based on spatial flow and temporal flow. It uses the spatial flow structure to extract intra-frame features of the video stream. The spatial flow structure is robust to changes in illumination and local occlusion, and can stably extract features of the visible area. By analyzing the video stream frame by frame, it extracts static features within the examination room, and is more sensitive to subtle behavioral features, avoiding misjudgments caused by temporal ambiguity. The temporal flow structure extracts inter-frame features of the video stream. By compensating for the loss of single-frame information caused by occlusion or sudden changes in illumination through multi-frame information, it can analyze motion patterns between consecutive frames, capture the temporal dependence of behavior, and is more effective for behavioral features that need to be observed continuously, avoiding missed detections due to insufficient information in a single frame. By fusing intra-frame and inter-frame features, the examination behavior analysis is realized. The examination behavior detection model can adapt to complex scenes, reduce the probability of false alarms and missed alarms, and greatly improve the accuracy of examination behavior detection. Attached Figure Description
[0017] Figure 1 This is a flowchart of the deep learning-based exam behavior analysis method of the present invention; Figure 2 This is a schematic diagram of the examination room behavior detection model in this invention. Detailed Implementation
[0018] The technical solution of the present invention will be further explained and described below with reference to the accompanying drawings.
[0019] like Figure 1 This is a flowchart of the deep learning-based exam behavior analysis method of the present invention, which includes the following steps: Step S1: Collect the video stream from the examination room and manually mark the areas and types of abnormal behavior. The types of abnormal behavior include: the candidate moving to the right, the candidate moving to the left, the candidate moving backward, the candidate standing, the candidate moving forward, the candidate looking around, the candidate asking questions, the candidate bowing, the candidate's hand gestures, the candidate looking down, the candidate cheating, and the candidate peeking. Step S2: Construct an examination behavior detection model based on spatial flow and temporal flow. The spatial flow structure is responsible for extracting the candidate's action features within a single frame, while the temporal flow structure is responsible for extracting the stacked action features of candidates across three to five frames. Step S3: Use the collected exam room video stream as input to the exam room behavior detection model, and use manually labeled abnormal behavior areas and abnormal behavior types as labels to train the exam room behavior detection model; Step S4: Repeat step S3 until the multi-objective loss function converges, completing the training of the examination behavior detection model; Step S5: Input the real-time video stream of the examination room into the trained examination room behavior detection model to predict the areas and types of abnormal behavior in real time.
[0020] This invention, based on deep learning, provides a method for analyzing exam behavior that can meticulously identify all behaviors of each examinee in the exam room and provide real-time feedback on the exam situation, greatly reducing the workload of invigilators.
[0021] In one technical solution of the present invention, such as Figure 2 The examination room behavior detection model in this invention includes a backbone network, a neck network, and a head network. The backbone network consists of a spatial flow structure, a temporal flow structure, and a feature concatenation module. The spatial flow structure is used to extract intra-frame features from the examination room video stream, the temporal flow structure is used to extract inter-frame features from the examination room video stream, and the feature concatenation module is used to concatenate and fuse intra-frame and inter-frame features. The neck network is used to perform cross-scale fusion of the concatenated features to obtain multi-scale feature maps. The head network predicts the abnormal behavior region and type based on the multi-scale feature maps.
[0022] The spatial flow structure comprises a first AKconv module, a second AKconv module, a first C3K2 module, a third AKConv module, a second C3K2 module, a fourth AKConv module, a third C3K2 module, a fifth AKConv module, a fourth C3K2 module, and an SPPF module, all connected in series. The first, second, third, fourth, and fifth AKConv modules utilize a deformable convolutional AKConv module. This module generates new coordinates to define the initial position of the convolutional kernel and dynamically adjusts the kernel sampling position by learning offsets and modulation scalars, thereby expanding the sensing... The model extracts more semantically meaningful intra-frame local features to adapt to action occlusion scenarios at different angles in the examination room, enhancing its ability to capture local deformations of human posture. Through stacked convolutional layers using the first, second, third, and fourth C3K2 modules, the computational load is reduced while preserving feature dimensionality. This allows for dimensionality reduction and fusion of intra-frame local features, resulting in intra-frame features at different scales. The SPPF module performs multi-scale pooling on these intra-frame features, fusing them into a fixed-length vector to obtain the final intra-frame features. This enhances the adaptability of the examination room behavior detection model to targets of different sizes, ensuring accurate classification of actions within a single frame.
[0023] The time-series stream structure includes a first TAKConv module, a second TAKConv module, a first TC3K2 module, a third TAKConv module, a second TC3K2 module, a fourth TAKConv module, a third TC3K2 module, a fifth TAKConv module, a fourth TC3K2 module, and a TCN module, all connected in series. The first, second, third, fourth, and fifth TAKConv modules introduce a time dimension, allowing comparison of the current frame with three to five previous frames to capture inter-frame characteristics of the video stream. The system assesses the temporal continuity of the features, adjusts the weights of the current frame to enhance temporal modeling capabilities, and obtains temporal stacked features. It constructs the temporal dimension using 3D convolutional kernels through the first, second, third, and fourth TC3K2 modules, and performs temporal stacking on the captured features, fusing the stacked features into 3D temporal features. By modeling the motion trajectories in consecutive frames, it extracts the temporal sequence features of the actions. The TCN module performs long-term temporal dependency analysis on the 3D temporal features, filters short-term noise, ensures stable recognition of continuous actions, obtains inter-frame features of the video stream, and reduces false detections.
[0024] Feature splicing module through The convolutional layers reduce the number of channels for intra-frame and inter-frame features, balancing feature dimensionality and computational efficiency, and enabling the concatenation of intra-frame and inter-frame features along the channel dimension to obtain concatenated and fused features.
[0025] The neck network comprises a first Upsample module, a first Contat module, a fifth C3K2 module, a second Upsample module, a second Contat module, a sixth C3K2 module, a sixth AKconv module, a third Contat module, a seventh C3K2 module, a seventh AKconv module, a fourth Contat module, and an eighth C3K2 module connected sequentially. The output of the fifth C3K2 module is also connected to the input of the third Contat module. The features obtained by concatenating and fusing intra-frame and inter-frame features are used as inputs to the first Upsample module, the first Contat module, the second Contat module, and the fourth Contat module, respectively, to obtain multi-scale feature maps. The neck network performs cross-scale fusion on the concatenated and fused features, enhancing the detection capability for moving targets of different sizes through bidirectional transmission between upper and lower layer features, and outputting multi-scale feature maps corresponding to action detection requirements at different resolutions.
[0026] The head network includes a first head module, a second head module, and a third head module connected in parallel, used to capture multi-scale information from multi-scale feature maps and predict abnormal behavior regions and types; wherein, the input of the first head module is connected to the output of the sixth C3K2 module, the input of the second head module is connected to the output of the third head module, and the input of the third head module is connected to the output of the eighth C3K2 module.
[0027] In one technical solution of this invention, the multi-objective loss function includes: a cross-entropy loss function, a localization loss function, a confidence loss function, and an action classification loss function. The classification loss function is used to optimize the accuracy of the examination behavior detection model. The localization loss function, also known as the bounding box regression loss function, is used to optimize the difference between the predicted bounding box and the actual bounding box. The confidence loss function is used to optimize the problem of imbalance in abnormal behavior categories. The action classification loss function adopts the cross-entropy loss function of a multi-classification task. Specifically, the calculation process of the multi-objective loss function is as follows:
[0028] in, Represents a multi-objective loss function. Represents the cross-entropy loss function. , Indicates the number of training samples. express index, For the first The true labels of each training sample Indicates the first The predicted output of each training sample. express Weighting coefficients; This represents the localization loss function. , This indicates the degree of overlap between the predicted bounding box and the ground truth bounding box. This represents the straight-line distance between the center point of the predicted bounding box and the center point of the actual bounding box. and These are the center points of the predicted bounding box and the ground truth bounding box, respectively. The diagonal distance of the smallest closure region containing the two boxes. It's the similarity in aspect ratio. express Weighting coefficients; Represents the confidence loss function. , This parameter represents the difficulty of classifying the samples. Indicates the regulating factor. Indicates the focus parameter, express The weighting coefficients, This represents the action classification loss function. , The number of candidates in the examination room. express index, Indicates the first In the training samples, the th The real label of each candidate Indicates the first In the training samples, the th The predicted output for each candidate. express The weighting coefficients.
[0029] This invention presents a deep learning-based examination behavior analysis method. It constructs an examination behavior detection model based on spatial and temporal flow. The spatial flow structure extracts intra-frame features from the video stream, demonstrating strong robustness to illumination changes and local occlusion, and stably extracts features from visible areas. Frame-by-frame analysis of the video stream extracts static features within the examination room, making it more sensitive to subtle behavioral characteristics and avoiding misjudgments due to temporal ambiguity. The temporal flow structure extracts inter-frame features from the video stream, compensating for single-frame information loss due to occlusion or sudden illumination changes through multi-frame information. It analyzes motion patterns between consecutive frames, capturing the temporal dependence of behavior, and is more effective for behavioral features requiring continuous observation, avoiding missed detections due to insufficient single-frame information. By fusing intra-frame and inter-frame features, examination behavior analysis is achieved. The examination behavior detection model can adapt to complex scenarios, reducing the probability of false positives and false negatives, and significantly improving the accuracy of examination behavior detection.
[0030] In one technical solution of the present invention, a computer-readable storage medium is also provided, storing a computer program that enables a computer to execute the deep learning-based examination behavior analysis method of the present invention.
[0031] In one technical solution of the present invention, an electronic device is also provided, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the deep learning-based examination behavior analysis method of the present invention.
[0032] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0033] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0034] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.
Claims
1. A method for analyzing exam behavior based on deep learning, characterized in that, Includes the following steps: Step S1: Collect the video stream from the examination room and manually mark the areas and types of abnormal behavior; Step S2: Construct an examination behavior detection model based on spatial flow and temporal flow; Step S3: Use the collected exam room video stream as input to the exam room behavior detection model, and use manually labeled abnormal behavior areas and abnormal behavior types as labels to train the exam room behavior detection model; Step S4: Repeat step S3 until the multi-objective loss function converges, completing the training of the examination behavior detection model; Step S5: Input the real-time video stream of the examination room into the trained examination room behavior detection model to predict the areas and types of abnormal behavior in real time.
2. The method for analyzing exam behavior based on deep learning according to claim 1, characterized in that, The examination room behavior detection model includes a backbone network, a neck network, and a head network. The backbone network consists of a spatial flow structure, a temporal flow structure, and a feature concatenation module. The spatial flow structure is used to extract intra-frame features from the examination room video stream, the temporal flow structure is used to extract inter-frame features from the examination room video stream, and the feature concatenation module is used to concatenate and fuse intra-frame and inter-frame features. The neck network is used to perform cross-scale fusion of the concatenated features to obtain multi-scale feature maps. The head network predicts the regions and types of abnormal behaviors based on the multi-scale feature maps.
3. The method for analyzing exam behavior based on deep learning according to claim 2, characterized in that, The spatial flow structure includes a first AKconv module, a second AKconv module, a first C3K2 module, a third AKConv module, a second C3K2 module, a fourth AKConv module, a third C3K2 module, a fifth AKConv module, a fourth C3K2 module, and an SPPF module, which are connected in series. Intra-frame local features for enhancing semantic information are extracted through the first AKconv module, the second AKconv module, the third AKConv module, the fourth AKConv module, and the fifth AKConv module. Intra-frame local features are reduced in dimensionality and fused using the first, second, third, and fourth C3K2 modules to obtain intra-frame features at different scales. The SPPF module is used to perform multi-scale pooling on intra-frame features of different scales, and to fuse intra-frame features of different scales into a vector of fixed length to obtain the final intra-frame features.
4. The method for analyzing exam behavior based on deep learning according to claim 2, characterized in that, The temporal stream structure includes a first TAKConv module, a second TAKConv module, a first TC3K2 module, a third TAKConv module, a second TC3K2 module, a fourth TAKConv module, a third TC3K2 module, a fifth TAKConv module, a fourth TC3K2 module, and a TCN module, which are connected in series. The first TAKConv module, the second TAKConv module, the third TAKConv module, the fourth TAKConv module, and the fifth TAKConv module capture the temporal continuity of the inter-frame features of the video stream to obtain temporal stacking features. The first TC3K2 module, the second TC3K2 module, the third TC3K2 module, and the fourth TC3K2 module fuse the temporal stacking features into 3D temporal features. The TCN module performs long-term temporal dependency analysis on the 3D temporal features to obtain the inter-frame features of the video stream.
5. The method for analyzing exam behavior based on deep learning according to claim 2, characterized in that, The feature stitching module stitches intra-frame features and inter-frame features along the channel dimension to obtain stitched and fused features.
6. The method for analyzing exam behavior based on deep learning according to claim 2, characterized in that, The neck network comprises a first Upsample module, a first Contat module, a fifth C3K2 module, a second Upsample module, a second Contat module, a sixth C3K2 module, a sixth AKconv module, a third Contat module, a seventh C3K2 module, a seventh AKconv module, a fourth Contat module, and an eighth C3K2 module connected in sequence. The output of the fifth C3K2 module is also connected to the input of the third Contat module. The features obtained by splicing and fusing intra-frame features and inter-frame features are used as the inputs of the first Upsample module, the first Contat module, the second Contat module, and the fourth Contat module, respectively, to obtain multi-scale feature maps.
7. The method for analyzing exam behavior based on deep learning according to claim 6, characterized in that, The head network includes a first head module, a second head module, and a third head module connected in parallel, used to capture multi-scale information of multi-scale feature maps and predict abnormal behavior regions and types; wherein, the input of the first head module is connected to the output of the sixth C3K2 module, the input of the second head module is connected to the output of the third head module, and the input of the third head module is connected to the output of the eighth C3K2 module.
8. The method for analyzing exam behavior based on deep learning according to claim 1, characterized in that, The calculation process of the multi-objective loss function is as follows: in, Represents a multi-objective loss function. Represents the cross-entropy loss function. express The weighting coefficients, This represents the localization loss function. express The weighting coefficients, Represents the confidence loss function. express The weighting coefficients, This represents the action classification loss function. express The weighting coefficients.
9. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to execute the deep learning-based exam behavior analysis method as described in any one of claims 1-8.
10. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the deep learning-based examination behavior analysis method as described in any one of claims 1-8.