Abnormal behavior recognition method and system based on teaching process image acquisition analysis

Abstract

The application discloses an abnormal behavior recognition method and system based on teaching process image collection and analysis, relates to the technical field of image processing, and realizes the real-time collection of an image matrix Ip through a high-definition camera, obtains a target position T and a target behavior feature F through a target detection algorithm, evaluates the attention score A of students, generates a comprehensive analysis index B according to the data, and realizes the accurate recognition of the dispersion or concentration of the attention of students. More importantly, by comparing with a preset feedback trigger threshold Bdis, the system can provide timely classroom feedback suggestions for teachers, help teachers master the real-time state of students, and timely adjust the teaching strategy. In a fixed period after the implementation of the feedback suggestions, the system will automatically perform secondary comparison and effectiveness verification, and adaptively adjust according to the change of the attention score A of students, so as to realize continuous optimization and accurate feedback, and ensure that the teaching activities are more efficient and targeted.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for identifying abnormal behavior based on image acquisition and analysis during the teaching process. Background Technology

[0002] With the rapid development of modern educational technology, students' learning status and emotional changes are crucial to teaching effectiveness. As educational philosophies shift, educators are increasingly emphasizing student-centered teaching methods. This approach prioritizes individual student differences and emotional needs, aiming to improve learning outcomes and classroom atmosphere. However, in actual teaching, teachers often find it difficult to monitor each student's learning status and emotional changes in real time. This presents challenges for teachers in adjusting teaching strategies and managing the classroom.

[0003] In current classroom management, especially in large classes, teachers often find it difficult to fully grasp the attention span of every student. Although teachers can try to detect whether students are focused through traditional observation methods, such as walking around and asking questions, these methods have obvious limitations: First, teachers have limited vision and cannot pay attention to every student's details at all times; second, students may be unwilling to express their true learning status to the teacher due to emotional reasons such as shyness and anxiety.

[0004] Subsequently, researchers found that most existing technologies are extensive studies targeting specific individuals with attention deficit disorder; for example, research applying attention mechanisms to computer vision, such as the significant achievements in image segmentation tasks using the Transformer architecture and its attention mechanism. Specifically, using Mask2Former with an occlusion attention mechanism, focusing on predicting local features within a mask region, it performs well in tasks such as panoptic segmentation, instance segmentation, and semantic segmentation. This technology can be applied to various image segmentation scenarios, including natural scene images based on gaze transfer segmentation. However, especially in the modern technological environment, students' dependence on mobile phones is increasing, and it has become common for students to use their phones, browse social media, or engage in other irrelevant activities in class. Such behavior usually does not directly affect students' outward performance, making it difficult for teachers to detect in a timely manner, thus posing a challenge to classroom management. Existing image detection methods, based solely on head gaze transfer analysis, cannot obtain comprehensive and accurate monitoring results, leading to unsatisfactory results in abnormal behavior recognition. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for identifying abnormal behavior based on image acquisition and analysis during the teaching process, thus solving the problems mentioned in the background section.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an abnormal behavior recognition method based on image acquisition and analysis during the teaching process, comprising the following steps:

[0007] By using high-definition cameras to set up classrooms, video streams in the classroom are captured in real time, and the video streams are preprocessed, including noise reduction, contrast enhancement, and image cropping, to output an image matrix IP.

[0008] An object detection algorithm is used on the image matrix Ip to extract the position and behavioral features of students in the image, which are labeled as target position T and target behavioral feature F. Based on the obtained target position T and target behavioral feature F, the attention score of the target student is evaluated to obtain the attention score A of the target student. Based on the obtained attention score A, it is combined with the target behavioral feature F, and through comprehensive analysis, abnormal behaviors of the target student are obtained to generate a comprehensive analysis index B.

[0009] Based on the comparison between the generated comprehensive analysis index B and the preset feedback trigger threshold Bdis, classroom feedback suggestion information is generated;

[0010] After a fixed period of time following the generation of classroom feedback suggestions, a second comparison is made with the comprehensive analysis index B of the target students to verify the effectiveness of the classroom feedback suggestions, and adaptive adjustments are made based on the attention score A.

[0011] Preferably, the method of using a high-definition camera to set up the classroom to collect video streams in real time, and preprocessing the video streams, including noise reduction, contrast enhancement and image cropping, and outputting an image matrix IP, specifically includes:

[0012] By deploying multiple high-definition cameras at different angles within the classroom, the system captures students' facial expressions, body movements, and the classroom environment in real time during the teaching process. Each high-definition camera is connected to a server via a network, transmitting video stream data in real time and marking it with timestamp information. Furthermore, the image data captured by each camera flows into the preprocessing step in the form of frames.

[0013] Each frame of the image after denoising preprocessing is subjected to image cropping preprocessing. Specifically, a portion of the image is cropped according to the classroom layout, retaining the teaching-related areas, which specifically include the student area and the podium area. The video stream data after denoising preprocessing and image cropping preprocessing is then output.

[0014] A contrast enhancement algorithm is used to perform contrast enhancement preprocessing on the video stream data after denoising preprocessing and image cropping preprocessing. The contrast of each frame image is enhanced, the discernibility of details in each frame image is adjusted, and the image matrix Ip is output.

[0015] The preprocessing steps include: performing denoising preprocessing on each received frame of image using image processing algorithms to remove image interference caused by camera noise or environmental factors; the image processing algorithms include Gaussian filtering and median filtering algorithms.

[0016] Preferably, the step of using a target detection algorithm on the image matrix Ip to extract the position and behavioral features of the students in the image and marking them as target position T and target behavioral features F includes S21 and S22;

[0017] S21. Use a target detection algorithm to detect all students from the image matrix Ip, identify the positions of the students' bodies, heads and hands, and integrate the position information of all students to form a target position T. The target position T is specifically obtained through steps S211 and S212.

[0018] S211. By performing bounding box prediction on the image matrix Ip, the coordinate information of the detected target student is extracted, and the center coordinates (x, y) and bounding box size (w, h) of the detected student are obtained.

[0019] S212. Integrate the center coordinates (x, y) and bounding box size (w, h) of all detected students to form the target location T, specifically in the form T = {(x(i), y(i), w(i), h(i)) | i∈1, 2, ..., N}, where N represents the total number of detected students.

[0020] Preferably, in step S22, within each target location T, key points are detected on the student's face, and the student's gaze direction, mouth opening and closing degree, eyebrow change information and hand behavior recognition are extracted and labeled as gaze direction vector De and hand-phone overlap rate Pp. Then, the pose estimation algorithm is used to calculate the pose of the detected student and obtain the pose key point set Fp.

[0021] Then, the attitude key point set Fp is calculated to obtain the attitude stability score Sfp;

[0022] By integrating the gaze direction vector De, the hand-phone overlap rate Pp, and the posture stability score Sfp, the target behavior feature F is obtained;

[0023] The specific form of the target behavior feature F is F={(De(i), Pp(i), Sfp(i))|i∈1,2,...,N}.

[0024] Preferably, the step of evaluating the attention score of the target student based on the acquired target location T and target behavioral characteristics F to obtain the target student's attention score A includes:

[0025] Based on the obtained target location T and target behavior characteristics F, the target student's attention score is evaluated, and the target student's attention score A is obtained. This score is then compared with the preset upper limit threshold Amax and lower limit threshold Amin for attention evaluation. The target student's status is then marked as Status, which reflects the current attention status of the target student.

[0026] Preferably, the attention score A is obtained by the following calculation formula:

[0027]

[0028] In the formula, De(i) represents the gaze direction vector of the i-th student, Dmax represents the preset maximum tolerance angle. When the gaze direction vector De(i) of the i-th student exceeds this maximum tolerance angle Dmax, it is considered that the student is not paying attention to the podium at all. Pp(i) represents the overlap rate between the i-th student's hand and the mobile phone, Sfp(i) represents the posture stability score of the i-th student, and a1, a2 and a3 represent the preset weight values ​​of the gaze direction vector De(i), the overlap rate between the i-th student's hand and the mobile phone Pp(i), and the posture stability score Sfp(i) of the i-th student, respectively, and a1+a2+a3=1;

[0029] The target student's status (Status) is obtained through the following comparison method:

[0030] When the upper limit threshold Amax < attention score A, the student's inattention assessment result is obtained, and the target student status Status = 9;

[0031] When the lower threshold Amin ≤ attention score A ≤ upper threshold Amax, the student's attention is observed to be scattered, and the target student's status is Status = 3.

[0032] When the attention score A is less than the lower threshold Amin, the student's attention concentration assessment result is obtained, and the target student status is Status=1.

[0033] Preferably, the step of combining the acquired attention score A with the target behavioral feature F to obtain abnormal behaviors of the target student through comprehensive analysis and generating a comprehensive analysis index B includes:

[0034] Based on the attention score A obtained, it is combined with the target behavioral characteristics F, and through comprehensive analysis, abnormal behaviors of the target student are obtained, generating a comprehensive analysis index B;

[0035] The comprehensive analysis index B is obtained through the following calculation formula:

[0036]

[0037] In the formula, B(i) represents the comprehensive analysis index of the i-th student, α, β and γ are constants, which are used to adjust the influence of the i-th student's gaze direction vector De(i), the i-th student's hand-phone overlap rate Pp(i), and the i-th student's posture stability score Sfp(i) on the comprehensive analysis index, and θ represents the deviation threshold.

[0038] Preferably, the step of comparing the generated comprehensive analysis index B with a preset feedback trigger threshold Bdis to generate classroom feedback suggestion information includes:

[0039] The comprehensive analysis index B(i) of the i-th student is compared with the preset feedback trigger threshold Bdis to generate classroom feedback suggestion information. The feedback trigger threshold Bdis is set by the teacher and is specifically used as the standard for the teacher to judge whether the student has abnormal behavior.

[0040] The classroom feedback suggestions were obtained through the following comparison methods:

[0041] When the comprehensive analysis index B(i) of the i-th student is greater than or equal to the feedback trigger threshold Bdis, a classroom feedback suggestion is generated. The classroom feedback suggestion includes prompting the students that the behavior of the i-th student is abnormal, and highlighting the i-th student on the visualization interface.

[0042] When the comprehensive analysis index B(i) of the i-th student is less than the feedback trigger threshold Bdis, no classroom feedback suggestion is generated, and the classroom feedback suggestion and the highlighting of the visualization interface for the i-th student are initialized.

[0043] Preferably, the step of conducting a second comparison of the comprehensive analysis index B of the target student after a fixed period of time following the generation of classroom feedback suggestions to verify the effectiveness of the classroom feedback suggestions, and making adaptive adjustments based on the attention score A, includes:

[0044] After a fixed period of generating classroom feedback suggestions, a second comparison is made on the comprehensive analysis index B of the target student. The second comparison is made by comparing the comprehensive analysis index B(i) of the i-th student with the feedback trigger threshold Bdis in the fixed period, and verifying the comparison result of the i-th student. The verification comparison result is made by extracting the classroom feedback suggestion content. When the classroom feedback suggestion content is extracted and generated, it means that the classroom feedback suggestion of the i-th student is invalid. The target student status Status is obtained based on the attention score A, and the feedback trigger threshold Bdis is adjusted adaptively according to the target student status Status.

[0045] The proportional adjustment feedback trigger threshold Bdis is adjusted in the following way:

[0046] When the target student's status is 9, the adjustment of the Bdis execution ratio for the feedback trigger threshold is reduced.

[0047] When the target student's status is 3, the feedback trigger threshold Bdis is not adjusted.

[0048] When the target student's status is Status=1, the adjustment of the feedback trigger threshold Bdis execution ratio is increased.

[0049] An abnormal behavior recognition system based on image acquisition and analysis of the teaching process includes an image acquisition module, an image detection module, an image analysis module, a comprehensive analysis module, a suggestion generation module, and a feedback optimization module.

[0050] The image acquisition module uses a high-definition camera to set up a classroom to capture video streams in real time, and preprocesses the video streams, including noise reduction, contrast enhancement, and image cropping, and outputs an image matrix Ip.

[0051] The image detection module uses a target detection algorithm on the image matrix Ip to extract the position and behavioral features of the students in the image, and marks them as the target position T and target behavioral features F;

[0052] The image analysis module evaluates the attention score of the target student based on the acquired target location T and target behavioral features F, and obtains the target student's attention score A.

[0053] The comprehensive analysis module combines the acquired attention score A with the target behavioral characteristics F to identify abnormal behaviors of the target student and generate a comprehensive analysis index B.

[0054] The suggestion generation module compares the generated comprehensive analysis index B with the preset feedback trigger threshold Bdis to generate classroom feedback suggestion information.

[0055] The feedback optimization module performs a second comparison of the comprehensive analysis index B of the target student after a fixed period of time following the generation of classroom feedback suggestions, verifies the effectiveness of the classroom feedback suggestions, and makes adaptive adjustments based on the attention score A.

[0056] This invention provides a method and system for identifying abnormal behavior based on image acquisition and analysis during the teaching process, which has the following beneficial effects:

[0057] (1) Classroom video streams are captured in real time using high-definition cameras and preprocessed, such as denoising, contrast enhancement, and image cropping, to obtain a clear image matrix Ip, providing a reliable data source for subsequent analysis. Then, using a target detection algorithm, the student's position and behavioral characteristics are extracted, yielding target position T and target behavioral characteristics F. This data lays the foundation for subsequent attention score evaluation and abnormal behavior analysis. Based on the target position T and target behavioral characteristics F, the system evaluates the student's attention score A and generates a comprehensive analysis index B, thereby achieving accurate identification of whether the student's attention is scattered or focused. More importantly, by comparing with a preset feedback trigger threshold Bdis, the system can provide teachers with timely classroom feedback suggestions, helping them grasp the students' real-time status and adjust teaching strategies accordingly. At fixed intervals after the feedback suggestions are implemented, the system automatically performs secondary comparisons and validity verification, adaptively adjusting based on changes in the student's attention score A, thus achieving continuous optimization and accurate feedback. The above method solves the problem of the lack of real-time monitoring and dynamic adjustment mechanisms in traditional classroom management, ensuring that teaching activities are more efficient and targeted.

[0058] (2) By assessing attention and conducting comprehensive behavioral analysis based on the target student's target position T and target behavioral characteristics F, this method can accurately characterize the student's attention state and provide dynamic and personalized classroom feedback. The system first calculates an attention score A, then combines this with the gaze direction vector De, hand-phone overlap rate Pp, and posture stability score Sfp to quantitatively assess the student's attention. By comparing this score with preset upper and lower thresholds Amax and Amin, a target student status Status is generated, intuitively reflecting the current attention level. This mechanism can not only accurately identify whether a student is completely inattentive, distracted, or highly focused, but also further combine it with a comprehensive analysis index B to assess abnormal behaviors such as frequent head-down movements, prolonged eye wandering, or phone use. This adapts to the needs of different teaching scenarios. Compared to traditional teacher observation or single-behavior detection methods, this method can improve the accuracy of classroom behavior recognition through multi-dimensional data analysis and provide real-time, data-driven feedback, helping teachers accurately adjust teaching strategies and ensuring continuous optimization of student participation and learning outcomes.

[0059] (3) By comparing the comprehensive analysis index B(i) with the preset feedback trigger threshold Bdis, the system can provide teachers with real-time and accurate classroom feedback suggestions based on students' behavioral patterns. The system continuously tracks each student's comprehensive analysis index. When the index exceeds the set feedback trigger threshold Bdis, it generates a warning of abnormal behavior, helping teachers quickly identify potential problem students and take appropriate intervention measures. Conversely, when the index is below the threshold, the system will not generate feedback, ensuring that teachers are not distracted by irrelevant information. Furthermore, the feedback trigger threshold Bdis is adaptively adjusted based on the student's attention status (Status), optimizing the threshold's sensitivity and preventing missed intervention opportunities due to prolonged poor student performance. In this way, the system can flexibly adapt to the dynamic changes in classroom teaching, making classroom management more precise and effective, and improving teaching quality and student learning experience. Attached Figure Description

[0060] Figure 1 This is a schematic diagram simulating the student's gaze angle at the reference pupil position in the abnormal behavior recognition method based on image acquisition and analysis of the teaching process of the present invention;

[0061] Figure 2 This is a schematic diagram simulating the angle of a student's gaze after the pupil position moves downward in the abnormal behavior recognition method based on image acquisition and analysis of the teaching process in this invention.

[0062] Figure 3 This is a schematic diagram of the abnormal behavior recognition method based on image acquisition and analysis of the teaching process according to the present invention;

[0063] Figure 4 This is a set of experimental effect comparison images of the abnormal behavior recognition method based on image acquisition and analysis of the teaching process according to the present invention;

[0064] Figure 5 This is a set of experimental effect comparison images of another scenario of the abnormal behavior recognition method based on image acquisition and analysis of the teaching process according to the present invention;

[0065] Figure 6 This is a comparative illustration of the line of sight from a normal perspective, a top-down perspective, and a bottom-up perspective in existing technologies.

[0066] Figure 7 This is a schematic diagram of the abnormal behavior recognition system based on image acquisition and analysis of the teaching process according to the present invention. Detailed Implementation

[0067] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0068] Example 1

[0069] This invention provides a method for identifying abnormal behavior based on image acquisition and analysis during the teaching process. Please refer to [link / reference]. Figure 3 This includes the following steps:

[0070] S1. By using a high-definition camera to set up the classroom, the video stream in the classroom is collected in real time, and the video stream is preprocessed to output an image matrix Ip; the preprocessing includes noise reduction, contrast enhancement and image cropping.

[0071] S2. Apply a target detection algorithm to the image matrix Ip to extract the student's position and behavioral features in the image. Mark the student's position in the image as the target position T, and mark the student's behavioral features in the image as the target behavioral features F.

[0072] S3. Evaluate the attention score of the target student based on the obtained target location T and target behavioral characteristics F, and obtain the target student's attention score A.

[0073] S4. Based on the obtained attention score A, it is combined with the target behavioral characteristics F to obtain the abnormal behaviors of the target student through comprehensive analysis and generate a comprehensive analysis index B.

[0074] S5. Based on the comparison between the generated comprehensive analysis index B and the preset feedback trigger threshold Bdis, classroom feedback suggestion information is generated;

[0075] S6. After a fixed period of time following the generation of classroom feedback suggestions, a second comparison is made with the comprehensive analysis index B of the target students to verify the effectiveness of the classroom feedback suggestions, and adaptive adjustments are made based on the attention score A.

[0076] In this embodiment, a high-definition camera captures real-time classroom video streams, which are then preprocessed, such as denoising, contrast enhancement, and image cropping, to obtain a clear image matrix Ip, providing a reliable data source for subsequent analysis. Next, a target detection algorithm extracts the students' location and behavioral characteristics, yielding target location T and target behavioral characteristics F. This data lays the foundation for subsequent attention score evaluation and abnormal behavior analysis. Based on the target location T and target behavioral characteristics F, the system evaluates the student's attention score A and generates a comprehensive analysis index B, thereby achieving accurate identification of whether a student's attention is scattered or focused. More importantly, by comparing the score with a preset feedback trigger threshold Bdis, the system can provide teachers with timely classroom feedback suggestions, helping them understand students' real-time status and adjust teaching strategies accordingly. At fixed intervals after the feedback suggestions are implemented, the system automatically performs a secondary comparison and validity verification, adaptively adjusting based on changes in the student's attention score A, thus achieving continuous optimization and accurate feedback. This method solves the problem of the lack of real-time monitoring and dynamic adjustment mechanisms in traditional classroom management, ensuring more efficient and targeted teaching activities.

[0077] Example 2

[0078] This embodiment is an explanation based on Embodiment 1. Please refer to it. Figure 3 Specifically: S1 includes S11 and S12;

[0079] S11. By deploying multiple high-definition cameras at different angles in the classroom, the high-definition camera layout in the classroom is carried out in real time to capture students' faces, body movements and classroom environment during the teaching process. Each high-definition camera is connected to the server through the network, transmits video stream data in real time and marks timestamp information. The image data collected by each camera flows into the preprocessing step in the form of frames. The preprocessing step includes S111 and S112.

[0080] S111. Each received frame of image is pre-processed for denoising using an image processing algorithm to obtain a denoised image (removing image interference caused by camera noise or environmental factors); the image processing algorithm includes a Gaussian filtering algorithm and a median filtering algorithm.

[0081] S112. Perform image cropping preprocessing on the denoised preprocessed image (specifically, crop a portion of the image according to the classroom layout, retaining the teaching-related area, which specifically includes the student area and the podium area), and output the video stream data after denoising preprocessing and image cropping preprocessing.

[0082] S12. Use a contrast enhancement algorithm to perform contrast enhancement preprocessing on the video stream data after denoising preprocessing and image cropping preprocessing (enhance the contrast of each frame image and adjust the discernibility of details in each frame image), and output the image matrix Ip.

[0083] S2 includes S21 and S22;

[0084] S21. Use a target detection algorithm to detect all students from the image matrix Ip, identify the positions of the students' bodies, heads and hands, and integrate the position information of all students to form a target position T. The target position T is specifically obtained through steps S211 and S212.

[0085] S211. By performing bounding box prediction on the image matrix Ip, the coordinate information of the detected target student is extracted, and the center coordinates (x, y) and bounding box size (w, h) of the detected student are obtained.

[0086] S212. Integrate the center coordinates (x, y) and bounding box dimensions (w, h) of all detected students to form the target location T. The specific form of the target location T is T = {(x(i), y(i), w(i), h(i)) | i ∈ 1, 2, ..., N}, where N represents the total number of detected students.

[0087] S22. Within each target location T, key points are detected on the student's face to obtain facial key points (including eye key points, mouth key points, and eyebrow key points); based on the facial key points, the student's gaze direction, mouth opening and closing degree, and eyebrow change information are extracted; the gaze direction vector De is obtained by analyzing the facial key points; and within each target location T, key points of the student's hands and key points of the mobile phone image are identified; based on the key points of the student's hands and the key points of the mobile phone image, the overlap rate Pp of the hands and the mobile phone is obtained through spatial position comparison analysis; based on the facial key points, the key points of the student's hands, and the key points of the mobile phone image, a pose estimation algorithm is used to calculate the pose of the detected student to obtain multiple pose key points Fp; based on the pose key points Fp, a pose key point set Q is constructed.

[0088] Then, the attitude key point set Q is calculated to obtain the attitude stability score Sfp, which reflects the student's attitude status.

[0089] The attitude stability score Sfp is obtained using the following formula:

[0090]

[0091] In the formula, Sfp(i) represents the posture stability score of the i-th student, (x(q), y(q)) represents the coordinates of the q-th posture key point, (x(r), y(r)) represents the reference coordinates of the q-th posture key point (obtained through empirical preset), and Q represents the set of posture key points.

[0092] By integrating the gaze direction vector De, the hand-phone overlap rate Pp, and the posture stability score Sfp, the target behavior feature F is obtained;

[0093] The target behavior feature F is specifically in the form of F={(De(i), Pp(i), Sfp(i))|i∈1,2,...,N};

[0094] The line-of-sight vector De is obtained by the following calculation formula:

[0095]

[0096] In the formula, De(i) represents the gaze direction vector of the i-th student, arctan represents the arctangent function, {x(right, i), y(right, i)} represents the center coordinates of the right eye of the i-th student, specifically the X-axis and Y-axis coordinates of the right eye of the i-th student, and {x(left, i), y(left, i)} represents the center coordinates of the left eye of the i-th student, specifically the X-axis and Y-axis coordinates of the left eye of the i-th student;

[0097] The overlap rate Pp between the hand and the mobile phone is obtained by the following formula.

[0098] Pp(i)=IoU(Fhand(i),Fobject(i));

[0099] In the formula, Pp(i) represents the overlap rate between the hand and the phone of the i-th student, IoU represents the intersection-union function, which is used to measure the degree of overlap between the hand region and the phone region, Fhand(i) represents the set of key points of the student's hand (the set of key points of the student's hand composed of all key points of the i-th student's hand, or the hand detection region), and Fobject(i) represents the set of key points of the phone image of the i-th student (the set of key points of the phone image composed of all key points of the phone image of the i-th student, or the phone target region).

[0100] In this embodiment, real-time video stream data is used to acquire students' attention performance and abnormal behavior, enabling accurate identification and assessment of students' classroom behavior. Multiple high-definition cameras are positioned from different angles in the classroom to capture students' faces, body movements, and the classroom environment in real time, ensuring that every dynamic detail of the students is captured. The video data transmitted in real time from each camera is sent to the server along with a timestamp. After preprocessing steps such as noise reduction, image cropping, and contrast enhancement, clearer and more accurate video data is obtained. Using this data, the system uses a target detection algorithm to extract students' position and behavioral features, thereby generating target position T and target behavioral features F. This provides crucial information for subsequent attention score assessment and abnormal behavior identification. Bounding box prediction is used to extract the position of each student, further identifying their gaze direction, mouth opening / closing, eyebrow changes, and hand behavior features. These features are quantified as a gaze direction vector De, hand-phone overlap rate Pp, and posture stability score Sfp, which together constitute the target behavioral feature F. Combining the target position T and target behavioral feature F, the system assesses students' attention scores and generates a comprehensive analysis index B through comprehensive analysis to determine whether students exhibit inattentive behavior.

[0101] Specifically, in step S22, the gaze direction vector De is obtained by analyzing the facial key points, including the following steps:

[0102] Step S221: Obtain an eye image based on the eye key points among the facial key points;

[0103] It should be noted that the above embodiment of this application obtains the eye image by connecting all the pixels in a local area image of the contour line formed by connecting the key points of the eye.

[0104] Step S222: The eye image is processed by pupil feature point detection using a pre-established and trained pupil detection model to obtain pupil feature points;

[0105] It should be noted that the above-described embodiment of this application uses a pre-established and trained pupil detection model. The eye image is input into the pupil detection model, and the output is the left pupil feature point of the current student's left eye and the right pupil feature point of the right eye.

[0106] Step S223: Analyze and calculate the gaze direction vector based on the displacement of the pupil feature points in the eye image within the time period;

[0107] It should be noted that in the above embodiments of this application, normal teaching activities, such as teachers demonstrating courseware, knocking on doors, or walking during lectures, will cause students to have normal visual deviations. At this time, it is obviously incorrect to still include the student's visual direction vector in the student's target behavior feature F. Furthermore, frequent teaching activities will lead to frequent student visual deviations. At this time, performing a large number of visual direction vector calculations will waste too much computing power, thereby affecting the efficiency of subsequent analysis.

[0108] Specifically, in step S223, the gaze direction vector is calculated by analyzing the displacement of the pupil feature points in the eye image within the time period, based on the pupil feature points. This includes the following steps:

[0109] Step S2231: Obtain the reference pupil position of the current student;

[0110] It should be noted that the aforementioned baseline pupil position was obtained by recording the current student's pupil position across multiple consecutive frames under stable lighting and environmental conditions, and then averaging the results. This pupil position is based on relative coordinates within the eye image. Figure 1 As shown, the student's line of sight is determined based on the position of the pupil. When the pupil is located in the center of the eye (i.e., the reference pupil position), the student's line of sight is 120 degrees directly in front.

[0111] Step S2232: Based on the reference pupil position, the displacement fluctuation rate is obtained through multi-frame continuous recognition and detection;

[0112] It should be noted that in the above embodiments of this application, the pupil position of the student is identified in multiple consecutive frames based on the reference pupil position, and the displacement fluctuation rate of the pupil position relative to the reference pupil position is detected. If the displacement fluctuation rate is low, it indicates that the current data is relatively stable in consecutive frames. If the fluctuation rate is high, it suggests that there is sudden interference (such as drastic changes in facial expression, making small movements, or sudden changes in ambient light).

[0113] Step S2233: Calculate the difference in symmetry between the left and right eyes based on the reference pupil position;

[0114] It should be noted that the difference in symmetry between the left and right eyes in the above embodiments of this application is a value that measures the consistency of the detection results of the left and right eyes of the same student in terms of the angle of deviation from the reference. When the offset values ​​obtained by the two eyes are relatively close, it indicates that the detection is relatively accurate. If the difference is too large, there may be detection abnormalities (for example, one eye may have an error due to temporary occlusion, strabismus or other reasons). It is calculated by measuring the absolute difference between the left and right eyes in the horizontal direction from the reference by measuring the real-time displacement fluctuation rate of the left and right eyes.

[0115] Step S2234: Determine whether the real-time change rate of the left-right eye deviation symmetry difference is greater than or equal to a preset left-right eye deviation difference threshold. If yes, mark the gaze direction vector as 0 (marked as 0 because the gaze direction vector of the student is due to gaze deviation caused by normal teaching behavior, therefore, its gaze deviation is not included in the calculation of the student's target behavior feature F); if no, further determine whether the displacement fluctuation rate is less than or equal to a preset displacement fluctuation rate threshold. If yes, mark the gaze direction vector as 0 (marked as 0 because the gaze direction vector of the student is due to gaze deviation caused by normal teaching behavior, therefore, its gaze deviation is not included in the calculation of the student's target behavior feature F). If not, then further random sampling is used to collect the displacement fluctuation rate of multiple students; it is determined whether the standard deviation of the student displacement fluctuation rate is greater than the displacement fluctuation rate threshold (at this time, it may be due to the teacher's walking or other collective reasons that the students' eyes change at the same time, so the current student's gaze direction vector is not further processed); if yes, then the gaze direction vector is marked as 0 (at this time, it is marked as 0 because the gaze direction vector of the above-mentioned student is due to the gaze deviation caused by normal teaching behavior, so its gaze deviation is not included in the calculation of the student's target behavior feature F); if not, then the gaze direction vector De is calculated based on the left eye center coordinate and the right eye center coordinate.

[0116] It should be noted that the above-described embodiments of this application use the displacement fluctuation rate of the reference pupil position and the difference in symmetry between the left and right eye deviations, along with the displacement fluctuation rate of the student's peers, to filter out normal student gaze deviations during normal teaching activities. Therefore, the calculation of the gaze direction vector is only performed on students with abnormal gaze deviations, thereby reducing computational burden and improving the efficiency of subsequent analysis. Figure 2 As shown, when the pupil position deviates from the aforementioned center position of the eye, the student's gaze angle will shift from the scene focus of the 120-degree field of vision directly in front. Taking the vertical deviation of the pupil position as an example, when the pupil deviates from the center upwards or downwards, the angle of gaze will change according to the distance of the deviation. Usually, each deviation is about 10-15 degrees, and sometimes even more. Based on this, the student's gaze direction vector De is calculated, thereby accurately identifying the student's attention status and then identifying the student's target behavioral characteristics F.

[0117] Example 3

[0118] This embodiment is an explanation based on Embodiment 2. Please refer to it. Figure 3 Specifically: S3 includes S31;

[0119] S31. Based on the obtained target location T and target behavior characteristics F, evaluate the target student's attention score to obtain the target student's attention score A. Compare the target student's attention score A with the preset upper limit threshold Amax and lower limit threshold Amin for attention evaluation to obtain the comparison result. Based on the comparison result, mark the target student's status (the target student's status reflects the current attention status of the target student).

[0120] The attention score A is obtained using the following formula:

[0121]

[0122] In the formula, De(i) represents the gaze direction vector of the i-th student, Dmax represents the preset maximum tolerance angle. When the gaze direction vector De(i) of the i-th student exceeds this maximum tolerance angle Dmax, it is considered that the student is not paying attention to the podium at all. Pp(i) represents the overlap rate between the i-th student's hand and the mobile phone, Sfp(i) represents the posture stability score of the i-th student, and a1, a2 and a3 represent the preset weight values ​​of the gaze direction vector De(i), the overlap rate between the i-th student's hand and the mobile phone Pp(i), and the posture stability score Sfp(i) of the i-th student, respectively, and a1+a2+a3=1;

[0123] The target student's status (Status) is obtained through the following comparison method:

[0124] When the upper limit threshold Amax < attention score A, the student's inattention assessment result is obtained, and the target student status Status = 9;

[0125] When the lower threshold Amin ≤ attention score A ≤ upper threshold Amax, the student's attention is observed to be scattered, and the target student's status is Status = 3.

[0126] When the attention score A is less than the lower threshold Amin, the student's attention concentration assessment result is obtained, and the target student status is Status=1.

[0127] S4 includes S41;

[0128] S41. Based on the obtained attention score A, it is combined with the target behavioral characteristics F to obtain the abnormal behaviors of the target student through comprehensive analysis and generate a comprehensive analysis index B.

[0129] The comprehensive analysis index B is obtained through the following calculation formula:

[0130]

[0131] In the formula, B(i) represents the comprehensive analysis index of the i-th student, α, β and γ are constants. Specifically, α is used to adjust the influence of the i-th student's gaze direction vector De(i) on the comprehensive analysis index, β is used to adjust the influence of the i-th student's hand-phone overlap rate Pp(i) on the comprehensive analysis index, γ is used to adjust the influence of the i-th student's posture stability score Sfp(i) on the comprehensive analysis index, and θ represents the deviation threshold, specifically the expected value of the normal hand-phone overlap rate. The greater the deviation from this value, the more likely the student is to be distracted.

[0132] In this embodiment, by assessing attention and conducting comprehensive behavioral analysis based on the target student's target location T and target behavioral characteristics F, this method can accurately characterize the student's attention state and provide dynamic and personalized classroom feedback. The system first calculates an attention score A, combining it with the gaze direction vector De, the hand-phone overlap rate Pp, and the posture stability score Sfp to quantitatively assess the student's attention. By comparing this score with preset upper and lower thresholds Amax and Amin, a target student status (Status) is generated, intuitively reflecting the current attention level.

[0133] This mechanism can accurately identify whether students are completely unfocused, distracted, or highly focused. Furthermore, by combining this with a comprehensive analysis index (B), it can assess abnormal student behaviors, such as frequent head-down movements, prolonged eye-wandering, or mobile phone use. This allows it to adapt to the needs of different teaching scenarios. Compared to traditional teacher observation or single-behavior detection methods, this method improves the accuracy of classroom behavior identification through multi-dimensional data analysis and provides real-time, data-driven feedback. This helps teachers precisely adjust their teaching strategies, ensuring continuous optimization of student participation and learning outcomes.

[0134] Example 4

[0135] This embodiment is an explanation based on Embodiment 3. Please refer to it. Figure 3 Specifically: S5 includes S51;

[0136] S51. Compare the comprehensive analysis index B(i) of the i-th student with the preset feedback trigger threshold Bdis to generate classroom feedback suggestions (Explanation: The role of classroom feedback suggestions is to provide prompts and suggestions to teachers in the classroom. The feedback trigger threshold Bdis is set by the teacher and is specifically used as the standard for teachers to judge whether students have abnormal behavior).

[0137] The classroom feedback suggestions were obtained through the following comparison methods:

[0138] When the comprehensive analysis index B(i) of the i-th student is greater than or equal to the feedback trigger threshold Bdis, a classroom feedback suggestion is generated. The classroom feedback suggestion includes prompting the students that the behavior of the i-th student is abnormal, and highlighting the i-th student on the visualization interface.

[0139] When the comprehensive analysis index B(i) of the i-th student is less than the feedback trigger threshold Bdis, no classroom feedback suggestion is generated, and the classroom feedback suggestion and the highlighting of the visualization interface for the i-th student are initialized.

[0140] S6 includes S61;

[0141] S61. After a fixed period of generating classroom feedback suggestions, a second comparison is made on the comprehensive analysis index B of the target student. The second comparison is made by comparing the comprehensive analysis index B(i) of the i-th student with the feedback trigger threshold Bdis in the fixed period, and verifying the comparison result of the i-th student. The verification comparison result is made by extracting the classroom feedback suggestion content. When the classroom feedback suggestion content is extracted and generated, it means that the classroom feedback suggestion of the i-th student is invalid. The target student status Status of the target student is obtained based on the attention score A, and the feedback trigger threshold Bdis is adjusted adaptively according to the target student status Status.

[0142] The proportional adjustment feedback trigger threshold Bdis is adjusted in the following way:

[0143] When the target student's status is 9, the adjustment of the Bdis execution ratio for the feedback trigger threshold is reduced.

[0144] When the target student's status is 3, the feedback trigger threshold Bdis is not adjusted.

[0145] When the target student's status is Status=1, the adjustment of the feedback trigger threshold Bdis execution ratio is increased.

[0146] In this embodiment, by comparing the comprehensive analysis index B(i) of the i-th student with a preset feedback trigger threshold Bdis, the system can provide teachers with real-time and accurate classroom feedback suggestions based on students' behavioral patterns. The system continuously tracks each student's comprehensive analysis index. When the index exceeds the set feedback trigger threshold Bdis, it generates a warning of abnormal behavior, helping teachers quickly identify potential problem students and take appropriate intervention measures. Conversely, when the index is below the threshold, the system does not generate feedback, ensuring that teachers are not distracted by irrelevant information. Furthermore, the feedback trigger threshold Bdis is adaptively adjusted based on the student's attention status (Status), optimizing the threshold's sensitivity and preventing missed intervention opportunities due to prolonged poor student performance. In this way, the system can flexibly adapt to the dynamic changes in classroom teaching, making classroom management more precise and effective, and improving teaching quality and student learning experience.

[0147] Researchers further examined the technical solution presented in Example 4, identifying it as relating to student attention assessment and abnormal behavior analysis. It mentions using steps S31 and S41 to calculate an attention score A and a comprehensive analysis index B, and using these scores to label the student's state. However, further experimental research revealed a problem: this solution may suffer from misjudgment of false abnormal behavior; that is, the system may incorrectly classify normal behavior as abnormal, such as a student bending down to pick something up being mistakenly identified as using a mobile phone (see Example 4). Figure 6 In the existing technology, when the student looks down to pick up something, the direction of the gaze can be identified, but the action of looking down also needs to be identified at the same time. Figure 6 The following demonstrations illustrate the line-of-sight directions from three perspectives: normal, top-down, and bottom-up (which are difficult to implement with existing technologies). This recognition system further adjusts the above processing steps to address this issue. The reason for this is that the technical solution presented in Example 4 lacks sufficient decoupling of behavioral features.

[0148] Example 5

[0149] This invention provides an abnormal behavior recognition system based on image acquisition and analysis during the teaching process, which performs behavior feature decoupling and optimization processing.

[0150] Regarding "behavioral feature decoupling optimization," in the analysis of students' classroom behavior, different behaviors may exhibit similar surface features (such as hand movements, gaze direction, etc.), but their actual intentions are completely different. The core goal of "decoupling optimization" is to separate the originally coupled (mutually interfering) behavioral features by introducing new feature dimensions or dynamic parameters, thereby more accurately distinguishing the essential differences between different behaviors. For example, a high overlap rate between the hand and the phone (Pp(i)↑) could be "using the phone" (abnormal behavior) or "quickly flipping through a book" (normal behavior). Unstable posture (Sfp(i)↓) could be "distracted swaying" (abnormal) or "actively raising the hand to answer a question" (normal).

[0151] The original embodiment 3's technical solution failed to distinguish these scenarios, leading to false anomaly detections. Decoupling optimization separates the correspondence between features and behaviors by introducing dynamic inhibition factors and intent recognition.

[0152] To address the issue of hand misjudgment, the calculation of Pp(i) is improved:

[0153]

[0154] Here, HandVelocity(i) is the inhibition factor; Hand movement speed HandVelocity(i) is introduced as an inhibition factor (λ = 0.2); fast hand movements (such as turning pages) will be automatically downweighted.

[0155] To improve attitude stability, the calculation of Sfp(i) is improved:

[0156]

[0157] Add PostureIntent(i) detection (determining active / passive movement through joint acceleration); μ = 0.3 for active posture adjustment (reducing penalty), μ = 1.0 for passive swaying;

[0158] The specific plan is as follows:

[0159] For example, a high overlap between the hand and the phone (Pp(i)↑) could indicate "using the phone" (abnormal behavior) or "quickly flipping through a book" (normal behavior). An unstable posture (Sfp(i)↓) could indicate "distracted shaking" (abnormal) or "actively raising a hand to answer a question" (normal). The original solution failed to distinguish these scenarios, leading to false anomalies. The technical solution adopted in this embodiment 4, through decoupling optimization, separates the correspondence between features and behaviors by introducing a dynamic inhibition factor and intent recognition.

[0160] Optimization of hand movements: Differentiating between "using a phone" and "flipping through a book" 1. Deficiencies of the original technical solution: It only calculates the overlap rate between the hand and the phone (Pp(i)), and cannot distinguish between static holding (playing with the phone) and dynamic movement (flipping through a book). False alarm scenario: When students flip through books quickly, their hands may briefly overlap with the phone area (Pp(i) increases briefly), which may be mistakenly identified as using the phone.

[0161] S310': Evaluate the attention score of the target student based on the obtained target location T and target behavior feature F, obtain the target student's attention score A, and compare the target student's attention score A with the preset upper limit threshold Amax and lower limit threshold Amin of attention evaluation to obtain the comparison result. Based on the comparison result, mark the target student's status (the target student's status reflects the current attention status of the target student).

[0162] The attention score A is obtained using the following formula:

[0163]

[0164] In the formula, De(i) represents the gaze direction vector of the i-th student, Dmax represents the preset maximum tolerance angle. When the gaze direction vector De(i) of the i-th student exceeds this maximum tolerance angle Dmax, it is considered that the student is not paying attention to the podium at all. Pp(i) represents the overlap rate between the i-th student's hand and the mobile phone, and Pp'(i) is the overlap rate between the i-th student's hand and the mobile phone after decoupling. The decoupled Sfp'(i) represents the posture stability score of the i-th student. a1, a2 and a3 represent the gaze direction vector De(i) of the i-th student, the overlap rate between the i-th student's hand and the mobile phone after decoupling, and the preset weight values ​​of the posture stability score Sfp'(i) of the i-th student after decoupling, respectively, and a1+a2+a3=1.

[0165] A new first decoupling optimization process is added using the decoupling optimization method. The specific formula is as follows:

[0166]

[0167] New parameter: Hand movement speed (HandVelocity(i)): Instantaneous speed (unit: pixels / second) calculated from the displacement time series of hand key points; λ is a suppression factor, its empirical value is set to 0.2 (adjustable), used to control the penalty weight of speed on hand overlap rate. Pp(i) is the hand-phone overlap rate of the i-th student before decoupling;

[0168] See technical effect verification Figure 4When the hand moves quickly (e.g., turning pages): HandVelocity(i) is higher → the denominator increases → Pp'(i) decreases significantly. When the hand is stationary or moves slowly (e.g., playing on a mobile phone): HandVelocity(i) is close to 0 → Pp'(i) ≈ Pp(i).

[0169] The target student's status (Status) is obtained through the following comparison method:

[0170] When the upper limit threshold Amax < attention score A, the student's inattention assessment result is obtained, and the target student status Status = 9;

[0171] When the lower threshold Amin ≤ attention score A ≤ upper threshold Amax, the student's attention is observed to be scattered, and the target student's status is Status = 3.

[0172] When the attention score A is less than the lower threshold Amin, the student's attention concentration assessment result is obtained, and the target student status is Status=1.

[0173] S4 includes S410;

[0174] S410. Based on the obtained attention score A, it is combined with the target behavioral characteristics F to obtain the abnormal behaviors of the target student through comprehensive analysis and generate a comprehensive analysis index B.

[0175] The comprehensive analysis index B is obtained through the following calculation formula:

[0176]

[0177] In the formula, B(i) represents the comprehensive analysis index of the i-th student, α, β and γ are constants. Specifically, α is used to adjust the influence of the i-th student's gaze direction vector De(i) on the comprehensive analysis index, β is used to adjust the influence of the i-th student's hand-phone overlap rate Pp(i) on the comprehensive analysis index, γ is used to adjust the influence of the i-th student's posture stability score Sfp(i) on the comprehensive analysis index, and θ represents the deviation threshold, specifically the expected value of the normal hand-phone overlap rate. The greater the deviation from this value, the more likely the student is to be distracted.

[0178] Optimization for posture stability: The original technical solution differentiates between "active adjustments" and "passive distractions." Its posture stability score, Sfp(i), is calculated solely based on the variance of joint displacements, failing to distinguish between active posture adjustments (such as raising a hand) and passive distraction-induced swaying. When a student actively raises their hand, the joint displacement variance increases, leading to a decrease in Sfp(i) and misjudgment as posture instability.

[0179] A second decoupling optimization process is performed to add an attitude stability parameter using a decoupling optimization method. The specific formula is as follows:

[0180] The above formula has a new parameter:

[0181] This includes implementing PostureIntent(i) detection: determining whether a movement is active based on the suddenness of joint acceleration. Calculation method: if the shoulder joint acceleration exceeds a threshold (e.g., 200 pixels / s) within 0.5 seconds... 2 If the expression is active, then PostureIntent(i) = 1 (active); otherwise, it is 0 (passive).

[0182] This includes implementing a dynamic penalty coefficient μ: when actively adjusting (PostureIntent(i) = 1): μ = 0.3 (minor penalty); when passively shaking (PostureIntent(i) = 0): μ = 1.0 (normal penalty);

[0183] The specific physical meaning of the processing is as follows: Actively adjusting posture (e.g., raising a hand): PostureIntent(i) = 1 → Denominator = 1 + 0.3 = 1.3 → Sfp'(i) ≈ 0.77 * Sfp(i) (only slightly reduces the score). Passively shaking (e.g., idly tapping one's leg): PostureIntent(i) = 0 → Denominator = 1 + 0 = 1 → Sfp'(i) = Sfp(i) (completely retains the original score).

[0184] See technical effect verification Figure 5 For the scenario of actively raising one's hand, the original Sfp(i) value is 0.6, and its PostureIntent(i) value is 1. The improved Sfp'(i) value is 0.46 (0.6 / 1.3≈0.46). The determination of whether it is a misjudgment is: the determination result is normal. For the scenario of passively shaking one's leg, the original Sfp(i) value is 0.4, and its PostureIntent(i) value is 0. The improved Sfp'(i) value is 0.4 (0.4 / 1=0.4). The determination of whether it is a misjudgment is: the determination result is abnormal.

[0185] The original values ​​are replaced with the decoupled Pp′(i) and Sfp'(i) to reduce the impact of misjudgments caused by hand dynamics and active posture adjustments. Step S410 (Comprehensive Analysis Index Calculation)

[0186] Modify the formula:

[0187]

[0188] Key changes: Introduce decoupled Pp′'(i) and Sfp'(i) to enhance adaptability to dynamic behaviors and intentions.

[0189] Real-world application example, scenario 1: Students quickly flipping through books (normal behavior); Original system:

[0190] Pp(i) = 0.8 (overlap between hand and phone area) → triggers anomaly (Status = 9). Improved system: HandVelocity(i) = 50 → Pp′(i) = 0.07 → reduced contribution of hand item in attention score A(i) → Status remains at 1 (normal).

[0191] Scenario 2: Students voluntarily raise their hands to answer questions (normal behavior); Original system: Sfp(i) = 0.6 (large joint displacement variance) → posture score decreases → may trigger Status = 3. Improved system: PostureIntent(i) = 1 → Sfp'(i) = 0.46 → overall score A(i) is still higher than Amin →

[0192] Status = 1 (Normal).

[0193] Embodiment 5 of this invention achieves accurate discrimination of "same features, different behaviors," significantly reducing the false positive rate. Accurate discrimination of "same features, different behaviors" refers to behaviors in classroom behavior monitoring where the same surface features (such as hand position, gaze shift, and posture changes) may correspond to different actual intentions (e.g., "flipping through a book" versus "playing on a phone," "raising a hand" versus "shaking a leg"). The original embodiment 4's technical solution led to misjudgments due to feature coupling; embodiment 5 achieves accurate differentiation through multi-dimensional decoupling optimization.

[0194] Example 6

[0195] This invention provides an abnormal behavior recognition system based on image acquisition and analysis during the teaching process. Please refer to [the relevant documentation]. Figure 7 Specifically, it includes an image acquisition module, an image detection module, an image analysis module, a comprehensive analysis module, a suggestion generation module, and a feedback optimization module;

[0196] The image acquisition module uses a high-definition camera to set up a classroom to capture video streams in real time, and preprocesses the video streams, including noise reduction, contrast enhancement, and image cropping, and outputs an image matrix Ip.

[0197] The image detection module uses a target detection algorithm on the image matrix Ip to extract the position and behavioral features of the students in the image, and marks them as the target position T and target behavioral features F;

[0198] The image analysis module evaluates the attention score of the target student based on the acquired target location T and target behavioral features F, and obtains the target student's attention score A.

[0199] The comprehensive analysis module combines the acquired attention score A with the target behavioral characteristics F to identify abnormal behaviors of the target student and generate a comprehensive analysis index B.

[0200] The suggestion generation module compares the generated comprehensive analysis index B with the preset feedback trigger threshold Bdis to generate classroom feedback suggestion information.

[0201] The feedback optimization module performs a second comparison of the comprehensive analysis index B of the target student after a fixed period of time following the generation of classroom feedback suggestions, verifies the effectiveness of the classroom feedback suggestions, and makes adaptive adjustments based on the attention score A.

[0202] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An abnormal behavior recognition method based on image acquisition and analysis during the teaching process, characterized in that: Includes the following steps: By using high-definition cameras to set up classrooms, video streams in the classroom are captured in real time, and the video streams are preprocessed, including noise reduction, contrast enhancement, and image cropping, to output an image matrix IP. An object detection algorithm is used on the image matrix Ip to extract the position and behavioral features of students in the image, which are labeled as target position T and target behavioral feature F. Based on the obtained target position T and target behavioral feature F, the attention score of the target student is evaluated to obtain the attention score A of the target student. Based on the obtained attention score A, it is combined with the target behavioral feature F, and through comprehensive analysis, abnormal behaviors of the target student are obtained to generate a comprehensive analysis index B. The step of evaluating the attention score of the target student based on the acquired target location T and target behavioral characteristics F, and obtaining the target student's attention score A, includes: Based on the obtained target location T and target behavior characteristics F, the target student's attention score is evaluated, the target student's attention score A is obtained, and compared with the preset upper limit threshold Amax and lower limit threshold Amin of attention evaluation, and the target student's status is marked as Status. The target student's status status reflects the current attention status of the target student. The attention score A is obtained using the following formula: ； In the formula, De(i) represents the gaze direction vector of the i-th student, Dmax represents the preset maximum tolerance angle. When the gaze direction vector De(i) of the i-th student exceeds this maximum tolerance angle Dmax, it is considered that the student is not paying attention to the podium at all. Pp(i) represents the overlap rate between the i-th student's hand and the mobile phone, Sfp(i) represents the posture stability score of the i-th student, and a1, a2 and a3 represent the preset weight values ​​of the gaze direction vector De(i), the overlap rate between the i-th student's hand and the mobile phone Pp(i), and the posture stability score Sfp(i) of the i-th student, respectively, and a1+a2+a3=1; The target student's status (Status) is obtained through the following comparison method: When the upper limit threshold Amax < attention score A, obtain the student's attention concentration assessment result and the target student status Status=1; When the lower threshold Amin ≤ attention score A ≤ upper threshold Amax, the assessment result of student attention being scattered is obtained, and the target student status Status=3; When the attention score A is less than the lower threshold Amin, the student's inattention assessment result is obtained, and the target student status is Status=9. Based on the comparison between the generated comprehensive analysis index B and the preset feedback trigger threshold Bdis, classroom feedback suggestion information is generated; After a fixed period of time following the generation of classroom feedback suggestions, a second comparison is made with the comprehensive analysis index B of the target students to verify the effectiveness of the classroom feedback suggestions, and adaptive adjustments are made based on the attention score A. The process of conducting a second comparison of the comprehensive analysis index B of the target students after a fixed period of time following the generation of classroom feedback suggestions verifies the effectiveness of the classroom feedback suggestions and makes adaptive adjustments based on the attention score A, including: After a fixed period of generating classroom feedback suggestions, a second comparison is made on the comprehensive analysis index B of the target student. The second comparison is made by comparing the comprehensive analysis index B(i) of the i-th student with the feedback trigger threshold Bdis in the fixed period, and verifying the comparison result of the i-th student. The verification result is made by extracting the classroom feedback suggestion content. When the classroom feedback suggestion content is extracted and generated, it means that the classroom feedback suggestion of the i-th student is invalid. The target student status Status is obtained based on the attention score A, and the feedback trigger threshold Bdis is adjusted adaptively according to the target student status Status. The proportional adjustment feedback trigger threshold Bdis is adjusted in the following way: When the target student's status is 9, the adjustment of the Bdis execution ratio for the feedback trigger threshold is reduced. When the target student's status is 3, the feedback trigger threshold Bdis is not adjusted. When the target student's status is Status=1, the adjustment of the feedback trigger threshold Bdis execution ratio is increased.

2. The abnormal behavior recognition method based on image acquisition and analysis of the teaching process according to claim 1, characterized in that: The method involves using high-definition cameras to set up classrooms, capturing video streams in real time, and preprocessing the video streams, including noise reduction, contrast enhancement, and image cropping, to output an image matrix IP. Specifically, this includes: By deploying multiple high-definition cameras at different angles within the classroom, the system captures students' facial expressions, body movements, and the classroom environment in real time during the teaching process. Each high-definition camera is connected to a server via a network, transmitting video stream data in real time and marking it with timestamp information. Furthermore, the image data captured by each camera flows into the preprocessing step in the form of frames. Each frame of the image after denoising preprocessing is subjected to image cropping preprocessing. Specifically, a portion of the image is cropped according to the classroom layout, retaining the teaching-related areas, which specifically include the student area and the podium area. The video stream data after denoising preprocessing and image cropping preprocessing is then output. A contrast enhancement algorithm is used to perform contrast enhancement preprocessing on the video stream data after denoising preprocessing and image cropping preprocessing. The contrast of each frame image is enhanced, the discernibility of details in each frame image is adjusted, and the image matrix Ip is output. The preprocessing steps include: performing denoising preprocessing on each received frame of image using image processing algorithms to remove image interference caused by camera noise or environmental factors; the image processing algorithms include Gaussian filtering and median filtering algorithms.

3. The abnormal behavior recognition method based on image acquisition and analysis of the teaching process according to claim 1, characterized in that: The image matrix Ip is processed using a target detection algorithm to extract the position and behavioral features of the students in the image, which are marked as target position T and target behavioral features F, including S21 and S22; S21. Use a target detection algorithm to detect all students from the image matrix Ip, identify the positions of the students' bodies, heads and hands, and integrate the position information of all students to form a target position T. The target position T is specifically obtained through steps S211 and S212. S211. By performing bounding box prediction on the image matrix Ip, the coordinate information of the detected target student is extracted, and the center coordinates (x, y) and bounding box size (w, h) of the detected student are obtained. S212. Integrate the center coordinates (x, y) and bounding box size (w, h) of all detected students to form the target location T, specifically in the form T={(x(i), y(i), w(i), h(i))|i∈1,2,...,N}, where N represents the total number of detected students.

4. The abnormal behavior recognition method based on image acquisition and analysis of the teaching process according to claim 3, characterized in that: S22. Within each target location T, key points are detected on the student's face. The student's gaze direction, mouth opening and closing, eyebrow change information and hand behavior recognition are extracted and labeled as gaze direction vector De and hand-phone overlap rate Pp. Then, the pose estimation algorithm is used to calculate the pose of the detected student and obtain the pose key point set Fp. Then, the attitude key point set Fp is calculated to obtain the attitude stability score Sfp; By integrating the gaze direction vector De, the hand-phone overlap rate Pp, and the posture stability score Sfp, the target behavior feature F is obtained; The target behavior feature F is specifically in the form of F={(De(i),Pp(i),Sfp(i))|i∈1,2,...,N}.

5. The abnormal behavior recognition method based on image acquisition and analysis of the teaching process according to claim 1, characterized in that: The attention score A obtained is then combined with the target behavioral characteristics F to obtain abnormal behaviors of the target student through comprehensive analysis, and a comprehensive analysis index B is generated, including: Based on the attention score A obtained, it is combined with the target behavioral characteristics F, and through comprehensive analysis, abnormal behaviors of the target student are obtained, generating a comprehensive analysis index B; The comprehensive analysis index B is obtained through the following calculation formula: ； In the formula, B(i) represents the comprehensive analysis index of the i-th student, and α, β, and γ represent constants, specifically used to adjust the influence of the i-th student's gaze direction vector De(i), the i-th student's hand-phone overlap rate Pp(i), and the i-th student's posture stability score Sfp(i) on the comprehensive analysis index. This indicates a deviation from the threshold.

6. The abnormal behavior recognition method based on image acquisition and analysis of the teaching process according to claim 5, characterized in that: The generated comprehensive analysis index B is compared with the preset feedback trigger threshold Bdis to generate classroom feedback suggestion information, including: The comprehensive analysis index B(i) of the i-th student is compared with the preset feedback trigger threshold Bdis to generate classroom feedback suggestion information. The feedback trigger threshold Bdis is set by the teacher and is specifically used as the standard for the teacher to judge whether the student has abnormal behavior. The classroom feedback suggestions were obtained through the following comparison methods: When the comprehensive analysis index B(i) of the i-th student is greater than or equal to the feedback trigger threshold Bdis, a classroom feedback suggestion is generated. The classroom feedback suggestion includes prompting the students that the i-th student's behavior is abnormal, and highlighting the i-th student on the visualization interface. When the comprehensive analysis index B(i) of the i-th student is less than the feedback trigger threshold Bdis, no classroom feedback suggestion is generated, and the classroom feedback suggestion and the highlighting of the visualization interface for the i-th student are initialized.

Images