Method and system for analyzing student attentiveness in a classroom based on machine vision

Abstract

The application relates to the technical field of classroom attention analysis, and relates to a classroom student attention analysis method and system based on machine vision, which comprises the following steps: constructing an attention visual analysis model, collecting vision by using the attention visual analysis model and a classroom visual detection device, obtaining a current behavior label group and an eye aspect ratio group, performing a facial expression attention assessment according to the eye aspect ratio group to obtain a facial expression concentration degree, performing a behavior attention assessment according to the current behavior label group to obtain a behavior concentration degree, calculating classroom attention based on the behavior concentration degree and the facial expression concentration degree, matching the classroom attention with a student ID, and obtaining classroom attention analysis data. The application can improve the accuracy and real-time performance of student attention analysis in an online classroom and reduce the risk of misjudgment caused by single features and individual differences.

Description

Technical Field

[0001] This invention relates to the field of classroom attention analysis technology, and in particular to a method and system for analyzing classroom student attention based on machine vision. Background Technology

[0002] With the rapid development of online education, real-time and objective assessment of students' attention in the classroom has become crucial for improving teaching quality and personalized tutoring. Machine vision-based student attention analysis methods can quantitatively analyze learning behavior by collecting visual data non-contactly, providing teachers with timely feedback and thus optimizing teaching interaction and classroom management, which has significant educational application value.

[0003] Traditional techniques typically rely on a single visual feature or human active observation to determine attention. This approach is susceptible to environmental or individual differences, leading to a high misjudgment rate. Furthermore, it lacks continuous data collection and is highly subjective, resulting in delayed and inaccurate evaluation results. Summary of the Invention

[0004] This invention provides a machine vision-based method for analyzing student attention in the classroom and a computer-readable storage medium. Its main purpose is to improve the accuracy and real-time performance of student attention analysis in online classrooms and reduce the risk of misjudgment due to single features and individual differences.

[0005] To achieve the above objectives, this invention provides a machine vision-based method for analyzing classroom student attention, comprising: Receive online classroom analysis instructions, determine the client set to be analyzed based on the online classroom analysis instructions, wherein the client set to be analyzed includes multiple clients to be analyzed, and each client to be analyzed corresponds to a student ID; Construct an attention visual analysis model, which includes an eye recognition model and a head pose recognition model; The clients to be analyzed are extracted sequentially from the client set to be analyzed, and the classroom visual detection devices of the extracted clients to be analyzed are identified. Visual data was collected using an attention visual analysis model and classroom visual detection equipment to obtain current behavior label groups and eye aspect ratio groups. Facial attention is assessed based on the eye aspect ratio group to obtain facial focus level, and behavioral attention is assessed based on the current behavior label group to obtain behavioral focus level. Classroom attention is calculated based on behavioral focus and facial focus. Classroom attention is then matched with the student IDs corresponding to the extracted client data to be analyzed, thus obtaining classroom attention analysis data. The classroom attention analysis data corresponding to each client to be analyzed is aggregated to obtain the classroom attention analysis dataset, thus completing the classroom student attention analysis based on machine vision.

[0006] Optionally, the step of using an attention visual analysis model and classroom visual detection equipment to collect visual data and obtain current behavior label groups and eye aspect ratio groups includes: Receive attention collection instructions and determine the attention evaluation cycle based on the attention collection instructions; Based on the attention collection instructions and classroom visual detection equipment, online classroom images are collected to obtain facial visual images; An eye recognition model is used to identify the aspect ratio of facial visual images, thus obtaining the aspect ratio of the eyes; A head pose recognition model is used to perform behavior recognition on facial visual images to obtain the current behavior label; The current evaluation period is obtained based on the attention collection command; If the current evaluation period is less than the attention evaluation period, return to the step of collecting online classroom images according to the attention collection instruction and the classroom visual detection device until the current evaluation period is not less than the attention evaluation period. If the current assessment period is not less than the attention assessment period, then the current behavior labels and eye aspect ratios are summarized separately to obtain the current behavior label group and the eye aspect ratio group.

[0007] Optionally, the step of using an eye recognition model to identify the aspect ratio of a facial visual image to obtain the eye aspect ratio includes: The face detection layer and eye feature extraction layer in the eye recognition model were identified, wherein the face detection layer contains the face detection algorithm; A face detection layer is used to perform face contour recognition on a face visual image to obtain the face region. The facial region is transferred to the eye feature extraction layer to obtain the eye key pixel set, which includes 6 eye key pixels, namely: outer corner of the eye, center of the upper eyelid, inner corner of the eye, center of the lower eyelid, upper eyelid point, and lower eyelid point. The aspect ratio of the eye is calculated based on the set of key pixels in the eye.

[0008] Optionally, the step of using a head pose recognition model to perform behavior recognition on a facial visual image to obtain a current behavior label includes: The facial visual image is input into the head pose recognition model to obtain the current pose rotation matrix, where the current pose rotation matrix is... matrix; Perform Euler angle analysis on the current attitude rotation matrix to obtain the current pitch angle and current yaw angle; Obtain a head posture attention scale, which includes: multiple head posture assessment criteria, wherein the head posture assessment criteria include non-face-viewing behavior labels or face-viewing behavior labels, and each head posture assessment criterion includes: standard pitch angle range and standard yaw angle range. Attention is assessed based on the current pitch and yaw angles using a head posture attention scale to obtain the current behavior label.

[0009] Optionally, before performing behavior recognition on the facial visual image using a head pose recognition model to obtain the current behavior label, the method further includes: Obtain a set of face-labeled images, wherein the set of face-labeled images includes multiple face-labeled images, and each face-labeled image contains an Euler angle label; The training set is constructed based on the face-labeled image set, resulting in multiple face training image groups. Each face training image group includes: anchor point labeled image, same pose labeled image, and contrast labeled image. The anchor point labeled image and the same pose labeled image have the same behavior label, while the anchor point labeled image and the contrast labeled image have different behavior labels. Construct the original pose recognition model, which includes a ResNet-50 network and fully connected layers; The original pose recognition model was trained using multiple sets of face training images to obtain a head pose recognition model.

[0010] Optionally, the step of training the original pose recognition model using multiple sets of face training images to obtain a head pose recognition model includes: Based on the original pose recognition model, the following operations are performed on each face training image group in multiple face training image groups: The face training image set is input into the ResNet-50 network in the original pose recognition model to obtain the head pose feature vector set, which includes multiple head pose feature vectors. The total loss value is constructed based on the head pose feature vector group; Summarize the total loss values ​​for each group of face training images to obtain the total loss value group; The original pose recognition model is back-optimized based on the total loss value set and the pre-built optimizer to obtain the head pose recognition model.

[0011] Optionally, the construction of the total loss value based on the head pose feature vector group includes: The anchor point posture feature vector, the same posture feature vector, and the comparative posture feature vector in the head posture feature vector group were identified. Calculate the same attitude prediction distance based on the anchor point attitude feature vector and the same attitude feature vector, and calculate the comparison attitude prediction distance based on the anchor point attitude feature vector and the comparison attitude feature vector. Construct triplet loss values ​​based on the predicted distances in the same pose and the predicted distances in contrasting poses; The anchor point pose feature vector is passed to the fully connected layer to obtain the predicted anchor point rotation matrix; Construct a real anchor rotation matrix based on the Euler angle labels corresponding to the anchor mark images, and calculate the pose loss value based on the real anchor rotation matrix and the predicted anchor rotation matrix; The total loss value is obtained by summing the triplet loss value and the attitude loss value.

[0012] Optionally, the assessment of facial attention based on eye aspect ratio to obtain facial focus includes: Extract the eye aspect ratios sequentially from the eye aspect ratio group, and record the extracted eye aspect ratios as the aspect ratios to be identified; Compare the preset normal aspect ratio, the preset closed aspect ratio, and the aspect ratio to be identified; If the aspect ratio to be identified is less than the closed aspect ratio, then the aspect ratio to be identified is recorded as the non-focused aspect ratio. If the aspect ratio to be identified is not less than the closed aspect ratio, and the aspect ratio to be identified is less than the normal aspect ratio, then the aspect ratio to be identified is recorded as the low focus aspect ratio. If the aspect ratio to be identified is not less than the normal aspect ratio, then the aspect ratio to be identified is recorded as the high-focus aspect ratio. The non-focused aspect ratio, low-focused aspect ratio, and high-focused aspect ratio are summarized separately to obtain the non-focused aspect ratio group, low-focused aspect ratio group, and high-focused aspect ratio group. The proportions of the non-focused ratio group, the low-focused ratio group, and the high-focused ratio group were statistically analyzed to obtain the proportions of non-focused, low-focused, and high-focused groups. The degree of focus in facial expressions is calculated based on the percentage of non-focused, low-focused, and high-focused individuals.

[0013] Optionally, the step of evaluating behavioral attention based on the current behavioral label group to obtain behavioral focus includes: Multiple standard behavioral labels are set based on the head posture attention scale; Perform the following operation for each of the multiple standard behavior labels: The standard behavior labels are assigned a focus score to obtain the standard behavior focus score. Identify multiple similar behavior tags that correspond to the standard behavior tag in the current behavior tag group, and count the number of similar tags among the multiple similar behavior tags; Summarize the standard behavior focus level and the number of similar tags for each standard behavior tag to obtain multiple standard behavior focus levels and multiple similar tags. By normalizing the weights of multiple similar tags, multiple focus weights are obtained. The behavioral focus score is obtained by weighting and summing multiple standard behavioral focus scores using multiple focus scores weights.

[0014] To achieve the above objectives, the present invention also provides a classroom student attention analysis system based on machine vision, comprising: The analysis instruction receiving module is used to receive online classroom analysis instructions and determine the set of clients to be analyzed based on the online classroom analysis instructions. The set of clients to be analyzed includes multiple clients to be analyzed, and each client to be analyzed corresponds to a student ID. The visual model building module is used to build attention visual analysis models, which include: eye recognition model and head pose recognition model; The focus calculation module is used to sequentially extract the clients to be analyzed from the client set, identify the classroom visual detection device of the extracted clients, use the attention visual analysis model and the classroom visual detection device to collect visual data, obtain the current behavior label group and the eye aspect ratio group, evaluate facial attention based on the eye aspect ratio group to obtain facial focus, and evaluate behavioral attention based on the current behavior label group to obtain behavioral focus. The student data matching module is used to calculate classroom attention based on behavioral focus and facial focus. It matches classroom attention with the student IDs corresponding to the extracted clients to be analyzed to obtain classroom attention analysis data. The classroom attention analysis data corresponding to each client to be analyzed is summarized to obtain a classroom attention analysis dataset.

[0015] To address the above problems, the present invention also provides an electronic device, the electronic device comprising: Memory, storing at least one instruction; The processor executes the instructions stored in the memory to implement the machine vision-based classroom student attention analysis method described above.

[0016] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one instruction, which is executed by a processor in an electronic device to implement the aforementioned machine vision-based classroom student attention analysis method.

[0017] To address the problems described in the background art, this invention first constructs an attention visual analysis model. This model integrates an eye recognition model and a head pose recognition model. Compared to existing technologies that often rely on only a single feature, such as head pose or eye state, for attention analysis, this step improves the comprehensiveness and accuracy of attention analysis through multimodal feature fusion, reducing misjudgments caused by the limitations of single features. Furthermore, visual data is acquired using the attention visual analysis model and classroom visual detection equipment to obtain the current behavior label group and eye aspect ratio group. This step involves continuous image acquisition by setting an attention evaluation cycle, which is significantly more efficient than existing technologies. This invention, which relies on discrete sampling or manual observation, achieves automated and continuous data collection, ensuring the real-time nature and objectivity of attention assessment and avoiding biases caused by human intervention. Finally, it assesses facial attention based on the eye aspect ratio group to obtain facial focus, and assesses behavioral attention based on the current behavior label group to obtain behavioral focus. This dual assessment mechanism comprehensively reflects the student's attention state, improves the accuracy of the assessment results, and reduces the impact of environmental and individual differences on the analysis results. Therefore, this invention can improve the accuracy and real-time nature of student attention analysis in online classrooms and reduce the risk of misjudgment caused by single features and individual differences. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a machine vision-based classroom student attention analysis method according to an embodiment of the present invention; Figure 2 A functional block diagram of a machine vision-based classroom student attention analysis system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device that implements the machine vision-based classroom student attention analysis method according to an embodiment of the present invention.

[0019] Explanation of reference numerals in the attached figures: 10. Electronic device; 11. Processor; 12. Memory; 13. Bus.

[0020] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0022] This application provides a machine vision-based method for analyzing student attention in a classroom. The execution entity of this machine vision-based method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the machine vision-based method for analyzing student attention in a classroom can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0023] Reference Figure 1 The diagram shown is a flowchart illustrating a machine vision-based classroom student attention analysis method according to an embodiment of the present invention. In this embodiment, the machine vision-based classroom student attention analysis method includes: S1. Receive online classroom analysis instructions, and determine the client set to be analyzed based on the online classroom analysis instructions. The client set to be analyzed includes multiple clients to be analyzed, and each client to be analyzed corresponds to a student ID.

[0024] Understandably, the online classroom analysis command refers to a manually initiated command to perform attention analysis on each online student in an online classroom. The set of clients to be analyzed refers to the client devices of multiple online students, and each client to be analyzed corresponds to a student ID, which is a unique identifier for the online student. For example, in a classroom using an online teaching platform, after the teacher initiates the online classroom analysis command, the teacher first determines the set of clients to be analyzed (such as students' computers or mobile devices) of all students participating in the class. Each client to be analyzed is associated with a student ID or username, thus enabling subsequent attention analysis.

[0025] S2. Construct an attention visual analysis model, which includes an eye recognition model and a head pose recognition model.

[0026] It should be explained that the attention visual analysis model refers to an integrated model capable of analyzing the attention of online students. This model integrates an eye recognition model and a head posture recognition model. The eye recognition model identifies the opening and closing of the eyes of online students, while the head posture recognition model identifies the orientation of their heads. These two models will be described in detail in subsequent embodiments. It should be noted that the aforementioned attention visual analysis model is deployed on the server side. After acquiring the visual data of students collected by each client (such as subsequent facial visual images), the student visual data is first sent to the server where the attention visual analysis model is deployed. The model then parses the student visual data to obtain a classroom attention analysis dataset. Finally, this dataset is sent to the teacher's terminal, thus reflecting the attention status of each online student in the current online classroom to the instructor.

[0027] S3. Extract the clients to be analyzed sequentially from the client set to be analyzed, and identify the classroom visual inspection devices of the extracted clients to be analyzed.

[0028] It is clear that the classroom visual detection device refers to a device that collects facial images of online students corresponding to the client to be analyzed, such as: the built-in camera of a laptop, an external USB camera, or the front camera of a mobile device.

[0029] S4. Visual data is collected using an attention visual analysis model and classroom visual detection equipment to obtain the current behavior label group and eye aspect ratio group.

[0030] It should be explained that the "current behavior label group" refers to a set of multiple current behavior labels, where each current behavior label represents the current head posture of the online student. The "eye aspect ratio group" refers to a set of multiple eye aspect ratios, where each eye aspect ratio is an indicator used to quantify the degree to which the online student's eyes are open or closed. A larger eye aspect ratio indicates a greater degree of eye openness. The purpose of collecting this eye aspect ratio is to objectively and continuously assess the eye state of online students in the online classroom. Since different students have different eye sizes, a normal aspect ratio and a closed aspect ratio were subsequently set for each student.

[0031] In detail, the process of using an attention visual analysis model and classroom visual detection equipment to collect visual data and obtain current behavior label groups and eye aspect ratio groups includes: Receive attention collection instructions and determine the attention evaluation cycle based on the attention collection instructions; Based on the attention collection instructions and classroom visual detection equipment, online classroom images are collected to obtain facial visual images; An eye recognition model is used to identify the aspect ratio of facial visual images, thus obtaining the aspect ratio of the eyes; A head pose recognition model is used to perform behavior recognition on facial visual images to obtain the current behavior label; The current evaluation period is obtained based on the attention collection command; If the current evaluation period is less than the attention evaluation period, return to the step of collecting online classroom images according to the attention collection instruction and the classroom visual detection device until the current evaluation period is not less than the attention evaluation period. If the current assessment period is not less than the attention assessment period, then the current behavior labels and eye aspect ratios are summarized separately to obtain the current behavior label group and the eye aspect ratio group.

[0032] It should be explained that the attention acquisition instruction refers to the instruction initiated by the server to the classroom visual detection device. Upon receiving this instruction, the classroom visual detection device needs to perform image acquisition on the online students corresponding to that device for a certain period (i.e., the attention assessment period). For example, when the server receives an online classroom analysis instruction initiated by the teacher, it needs to send attention acquisition instructions to each classroom visual detection device, thereby commanding each device to perform image acquisition. The attention assessment period refers to the total time length set manually for acquiring student visual data. This period is set by the teacher when initiating the online classroom analysis instruction, pre-setting a time value, for example, 5 minutes. The classroom visual detection device will then continuously acquire images of the online students within these 5 minutes. The current assessment period refers to the time elapsed from receiving the attention acquisition instruction to the current moment. If the current assessment period is less than the attention assessment period, image acquisition needs to continue.

[0033] In detail, the step of using an eye recognition model to identify the aspect ratio of a facial visual image to obtain the eye aspect ratio includes: The face detection layer and eye feature extraction layer in the eye recognition model were identified, wherein the face detection layer contains the face detection algorithm; A face detection layer is used to perform face contour recognition on a face visual image to obtain the face region. The facial region is transferred to the eye feature extraction layer to obtain the eye key pixel set, which includes 6 eye key pixels, namely: outer corner of the eye, center of the upper eyelid, inner corner of the eye, center of the lower eyelid, upper eyelid point, and lower eyelid point. The aspect ratio of the eye is calculated based on the set of key pixels in the eye.

[0034] It should be explained that the facial region refers to the image region containing complete facial features located by a face detection algorithm in a facial visual image. The above-mentioned use of the face detection layer in the eye recognition model to perform facial contour recognition on a facial visual image refers to: using the face detection algorithm contained in the face detection layer to perform face recognition on the facial visual image. Here, the face detection algorithm refers to a cascaded classifier based on Haar features. This face recognition method is existing technology, and the specific steps are as follows: First, the facial visual image is converted to grayscale. Then, a series of pre-trained Haar-like template features reflecting facial features such as the eyes, nose, and mouth are used to perform a sliding window scan at different positions in the grayscale facial visual image. The cascaded classifier quickly eliminates non-face regions during the sliding window scan process, ultimately locating the position and range of the face (i.e., the facial region). The eye feature extraction layer refers to the component in the eye recognition model responsible for further refining the eye location and extracting key eye points from the already located facial region. This eye feature extraction layer includes feature point detection algorithms, such as the Histogram of Oriented Gradients (HOG) combined with a linear classifier from the Dlib library. The method by which this eye feature extraction layer extracts the set of key eye pixels is an existing technology, and the specific steps are as follows: First, based on prior knowledge or rules (such as facial geometry), a rough area where the eyes are located is estimated within the facial region. Then, the HOG features of this rough area are extracted to describe the local gradient direction information of the rough area. Finally, these HOG features are input into a pre-trained shape predictor, which directly outputs the precise pixel coordinates of 6 predefined key points around the eyes, thus forming the set of key eye pixels.

[0035] Furthermore, the aforementioned key pixel set for the eyes refers to a set of six key pixels for the eyes. The outer corner of the eye is the outermost point of the eye socket in the eye region of the face; the center point of the upper eyelid is the highest point of the upper eyelid contour in the eye region of the face; the inner corner of the eye is the innermost point of the eye socket in the eye region of the face; the center point of the lower eyelid is the lowest point of the lower eyelid contour in the eye region of the face; the upper eyelid point is the center point located between the center point of the upper eyelid and the inner corner of the eye; and the lower eyelid point is the center point located between the center point of the upper eyelid and the inner corner of the eye. These six key pixels together outline the contour of the eye. The calculation of the eye aspect ratio based on the key pixel set refers to: calculating the aspect ratio of each individual eye (left and right) and calculating the average of the aspect ratios of each individual eye (left and right). This average value is the eye aspect ratio. The formula for calculating the aspect ratio of each individual eye is: in, Indicates the aspect ratio of the eyes. Indicates the upper eyelid point. Indicates the lower eyelid point. This indicates taking the absolute value. Indicates the center point of the upper eyelid. Indicates the center point of the lower eyelid. Indicates the inner corner of the eye. Indicates the outer corner of the eye.

[0036] In detail, the step of using a head pose recognition model to perform behavior recognition on a facial visual image to obtain a current behavior label includes: The facial visual image is input into the head pose recognition model to obtain the current pose rotation matrix, where the current pose rotation matrix is... matrix; Perform Euler angle analysis on the current attitude rotation matrix to obtain the current pitch angle and current yaw angle; Obtain a head posture attention scale, which includes: multiple head posture assessment criteria, wherein the head posture assessment criteria include non-face-viewing behavior labels or face-viewing behavior labels, and each head posture assessment criterion includes: standard pitch angle range and standard yaw angle range. Attention is assessed based on the current pitch and yaw angles using a head posture attention scale to obtain the current behavior label.

[0037] It should be explained that the current pose rotation matrix refers to the matrix output by the head pose recognition model, which describes the rotation state of the online student's head relative to the world coordinate system. The mathematical matrix, the current attitude rotation matrix, represents the rotational transformation from the world coordinate system to the head coordinate system, and can completely characterize the three-dimensional orientation of the online student's head. The current pitch angle refers to the angle at which the head is raised or lowered, i.e., the amplitude of the online student's nodding motion, and the current yaw angle refers to the angle at which the head is turned left or right, i.e., the amplitude of the online student's shaking motion. Thus, the current pitch angle and the current yaw angle can quantify the online student's head orientation, thereby determining whether the online student is looking down, up, or left and right. The larger the current pitch angle, the more the head is tilted up or down. The larger the current yaw angle, the more the head is turned to the left or right. The steps for performing Euler angle analysis on the current attitude rotation matrix are as follows: based on the mathematical transformation relationship between the rotation matrix (such as the current attitude rotation matrix mentioned above) and Euler angles, the yaw angle and pitch angle are calculated and decomposed using trigonometric functions. The yaw angle and pitch angle here are the current yaw angle and the current pitch angle. The calculation process is existing technology and will not be described in detail here.

[0038] It is clear that the head posture attention scale refers to a set of multiple head posture evaluation criteria constructed by humans. The non-face-viewing behavior label refers to the label indicating that online students exhibit head-down, head-up, and left-right looking behaviors, while the face-viewing behavior label refers to the label when the above-mentioned head-down, head-up, and left-right looking behaviors do not occur. The standard pitch angle range and standard yaw angle range refer to the pitch angle range and yaw angle range when the corresponding head posture assessment standard appears. These standard pitch angle range and standard yaw angle range can be manually set by observing the actual behavior of students. For example, when the pitch angle range is less than -15 degrees, it is considered that the online student has shown head-down behavior; when the pitch angle range is greater than 15 degrees, it is considered that the online student has shown head-up behavior; when the yaw angle range is greater than 20 degrees or less than -20 degrees, it is considered that the online student has shown left and right looking behavior. The above pitch angle range and yaw angle range are then combined, and the resulting pitch angle range {less than -15 degrees or greater than 15 degrees} and yaw angle range {greater than 20 degrees or less than -20 degrees} are the standard pitch angle range and standard yaw angle range corresponding to the non-frontal gaze behavior label. Conversely, the standard pitch angle range and standard yaw angle range corresponding to the frontal gaze behavior label are {-15 degrees, 15 degrees} and {-20 degrees, 20 degrees}, respectively. The above-mentioned attention assessment of the current pitch and yaw angles based on the head posture attention table refers to comparing the current pitch and yaw angles one by one with the standard pitch and yaw angle ranges corresponding to each head posture assessment standard in the head posture attention table. If any angle of the current pitch and yaw angles matches the standard pitch and yaw angle ranges corresponding to the non-eye-facing behavior label, then the non-eye-facing behavior label is recorded as the current behavior label; otherwise, the eye-facing behavior label is recorded as the current behavior label.

[0039] Specifically, before using a head pose recognition model to perform behavior recognition on a facial visual image to obtain the current behavior label, the method further includes: Obtain a set of face-labeled images, wherein the set of face-labeled images includes multiple face-labeled images, and each face-labeled image contains an Euler angle label; The training set is constructed based on the face-labeled image set, resulting in multiple face training image groups. Each face training image group includes: anchor point labeled image, same pose labeled image, and contrast labeled image. The anchor point labeled image and the same pose labeled image have the same behavior label, while the anchor point labeled image and the contrast labeled image have different behavior labels. Construct the original pose recognition model, which includes a ResNet-50 network and fully connected layers; The original pose recognition model was trained using multiple sets of face training images to obtain a head pose recognition model.

[0040] It should be explained that the face-marked image set refers to a collection of multiple face-marked images. The face-marked image refers to image data containing a clear face and whose head Euler angles (yaw angle, pitch angle) have been labeled manually or by professional equipment. The Euler angle label refers to the pre-labeled head posture angle values ​​of the face-marked image, including yaw angle and pitch angle. The Euler angle label is obtained by professional motion capture equipment or manual labeling tools. For example, when acquiring a face-marked image, a sensor is used to record the actual yaw angle and actual pitch angle of the head. The actual yaw angle and actual pitch angle are the Euler angle labels of the face-marked image. The anchor point labeled image refers to the face labeled image used as the reference when constructing the face training image set. The same pose labeled image refers to the face labeled image with the same behavioral label as the anchor point labeled image. The comparison labeled image refers to the face labeled image with different behavioral labels than the anchor point labeled image. The behavioral label refers to the label defined by Euler angle labels (in the same way as the current behavioral label obtained from the head pose attention table mentioned above), including non-frontal behavioral labels and frontal behavioral labels.

[0041] Furthermore, the construction of the training set based on the face-marked image set refers to: firstly, randomly selecting a face-marked image from the face-marked image set as an anchor point marker image, and determining the anchor point behavior label based on the Euler angle label of the anchor point marker image. The anchor point behavior label is either a non-frontal behavior label or a frontal behavior label. Then, randomly selecting a face-marked image from the face-marked image set that has the same behavior label as the anchor point behavior label as a pose-matching marker image, and randomly selecting a face-marked image that has a different behavior label than the anchor point behavior label as a contrast marker image. The triplet composed of the anchor point marker image, the pose-matching marker image, and the contrast marker image is the face training image set. This scheme uses triples for subsequent training, enabling the original pose recognition model to learn to distinguish features of different behavior labels. The anchor point image serves as the baseline, with behavior labels consistent between the pose label image and the anchor point image, used to narrow the feature distance between similar head poses. Conversely, the behavior labels of the contrast label image and the anchor point image are inconsistent, used to calculate the feature distance between different head poses. This approach enhances the accuracy of the original pose recognition model in recognizing changes in head pose and reduces interference from factors such as identity or lighting.

[0042] In detail, the step of training the original pose recognition model using multiple sets of face training images to obtain a head pose recognition model includes: Based on the original pose recognition model, the following operations are performed on each face training image group in multiple face training image groups: The face training image set is input into the ResNet-50 network in the original pose recognition model to obtain the head pose feature vector set, which includes multiple head pose feature vectors. The total loss value is constructed based on the head pose feature vector group; Summarize the total loss values ​​for each group of face training images to obtain the total loss value group; The original pose recognition model is back-optimized based on the total loss value set and the pre-built optimizer to obtain the head pose recognition model.

[0043] It should be explained that the ResNet-50 network refers to a deep convolutional neural network containing 50 residual blocks, capable of automatically extracting features from face training images. The head pose feature vector output by the ResNet-50 network is a 1024-dimensional numerical vector that encodes key information about the head pose in the face training images, such as contour and orientation features. The head pose feature vector set refers to a collection of multiple head pose feature vectors, where each head pose feature vector is the feature vector output by the ResNet-50 network after inputting a face training image. The total loss value refers to a scalar value used to measure the difference between the prediction result and the true value of the original pose recognition model on the current training sample (i.e., a set of face training images). The optimizer refers to an algorithm, such as the Adam optimizer, used to update and adjust the trainable parameters of the original pose recognition model (i.e., the parameters in the ResNet-50 network and fully connected layers) based on the total loss value. The aforementioned backpropagation optimization of the original pose recognition model based on the total loss set and the pre-built optimizer refers to: calculating the average loss value of the total loss set; using the backpropagation algorithm, calculating the gradient of this average loss value with respect to each trainable parameter of the original pose recognition model; and finally, updating each trainable parameter according to these gradients and its built-in rules (such as combining momentum and adaptive learning rate), thereby enabling the original pose recognition model to output more accurate predictions and reduce the loss value in subsequent training.

[0044] Specifically, the construction of the total loss value based on the head pose feature vector group includes: The anchor point posture feature vector, the same posture feature vector, and the comparative posture feature vector in the head posture feature vector group were identified. Calculate the same attitude prediction distance based on the anchor point attitude feature vector and the same attitude feature vector, and calculate the comparison attitude prediction distance based on the anchor point attitude feature vector and the comparison attitude feature vector. Construct triplet loss values ​​based on the predicted distances in the same pose and the predicted distances in contrasting poses; The anchor point pose feature vector is passed to the fully connected layer to obtain the predicted anchor point rotation matrix; Construct a real anchor rotation matrix based on the Euler angle labels corresponding to the anchor mark images, and calculate the pose loss value based on the real anchor rotation matrix and the predicted anchor rotation matrix; The total loss value is obtained by summing the triplet loss value and the attitude loss value.

[0045] It should be explained that the anchor pose feature vector refers to the head pose feature vector corresponding to the anchor point marker image. The same pose feature vector refers to the head pose feature vector corresponding to the same pose marker image. The contrasting pose feature vector refers to the head pose feature vector corresponding to the contrast marker image. The same pose prediction distance refers to the Euclidean distance between the anchor pose feature vector and the same pose feature vector. The contrasting pose prediction distance refers to the Euclidean distance between the anchor pose feature vector and the contrasting pose feature vector. The triplet loss value refers to a loss function used to quantify learning, and the triplet loss value is calculated as follows: ,in, This represents the triplet loss value. Represents the maximum value function. Indicates the predicted distance at the same pose. This represents the distance predicted by the comparison pose. Calculating the above triplet loss value ensures that the original pose recognition model makes features of similar poses closer together and features of different poses further apart, thereby improving the pose recognition discrimination ability of the original pose recognition model. The larger the triplet loss value, the worse the original pose recognition model's ability to distinguish pose features (e.g., the greater the distance between anchor pose feature vectors and feature vectors of the same pose).

[0046] Furthermore, the fully connected layer refers to a neural network layer that is connected to the output layer of the ResNet-50 network. This fully connected layer transforms the input 1024-dimensional anchor pose feature vector into a... The predicted anchor rotation matrix has the same composition as the true anchor rotation matrix. The parameters (i.e., trainable parameters) in the weight matrix of the fully connected layer need to be optimized through the training process described above. The true anchor rotation matrix refers to the matrix representing the true head orientation, obtained through mathematical transformation based on the Euler angle labels (i.e., the true head pose angles) corresponding to the anchor point marker images. The rotation matrix, where constructing the true anchor point rotation matrix based on the Euler angle labels corresponding to the anchor point marker images, refers to calculating a rotation matrix representing the true head orientation using the standard conversion formula from Euler angle labels (yaw, pitch, roll) to a rotation matrix (e.g., by rotating around the Z, Y, and X axes by specific angles). The attitude loss value represents the geometric difference between the predicted and true rotation matrices. A larger attitude loss value indicates a greater deviation between the predicted and true attitudes, resulting in poorer performance of the original attitude recognition model and a greater need to adjust the parameters in the weight matrix of the fully connected layers. The attitude loss value is calculated as follows: in, This represents the attitude loss value. This represents the inverse cosine function, whose output value indicates the shortest path between the true anchor rotation matrix and the predicted anchor rotation matrix. Represents the trace of a matrix. This represents the predicted anchor point rotation matrix. Represents the actual anchor point rotation matrix. This represents the transpose of the true anchor point rotation matrix. The above calculation of summing the triplet loss value and the attitude loss value refers to adding the triplet loss value to the attitude loss value; the sum is the total loss value.

[0047] S5. Evaluate facial attention based on the eye aspect ratio group to obtain facial focus level, and evaluate behavioral attention based on the current behavior label group to obtain behavioral focus level.

[0048] It should be explained that the "attentional focus" refers to a numerical value quantifying the online student's attentional focus in terms of facial expressions. The "behavioral focus" refers to a numerical value quantifying the online student's attentional focus in terms of behavioral expressions.

[0049] Specifically, the assessment of facial attention based on eye aspect ratio to obtain facial focus includes: Extract the eye aspect ratios sequentially from the eye aspect ratio group, and record the extracted eye aspect ratios as the aspect ratios to be identified; Compare the preset normal aspect ratio, the preset closed aspect ratio, and the aspect ratio to be identified; If the aspect ratio to be identified is less than the closed aspect ratio, then the aspect ratio to be identified is recorded as the non-focused aspect ratio. If the aspect ratio to be identified is not less than the closed aspect ratio, and the aspect ratio to be identified is less than the normal aspect ratio, then the aspect ratio to be identified is recorded as the low focus aspect ratio. If the aspect ratio to be identified is not less than the normal aspect ratio, then the aspect ratio to be identified is recorded as the high-focus aspect ratio. The non-focused aspect ratio, low-focused aspect ratio, and high-focused aspect ratio are summarized separately to obtain the non-focused aspect ratio group, low-focused aspect ratio group, and high-focused aspect ratio group. The proportions of the non-focused ratio group, the low-focused ratio group, and the high-focused ratio group were statistically analyzed to obtain the proportions of non-focused, low-focused, and high-focused groups. The degree of focus in facial expressions is calculated based on the percentage of non-focused, low-focused, and high-focused individuals.

[0050] It should be explained that the "normal aspect ratio" refers to the aspect ratio of a student's eyes under normal listening conditions. This normal aspect ratio is obtained by collecting multiple eye aspect ratios of a student under normal listening conditions (i.e., without looking left or right, looking down, looking up, or being fatigued), and recording the average of these multiple eye aspect ratios as the student's normal aspect ratio. The "closed aspect ratio" refers to the aspect ratio of a student's eyes when closed. This closed aspect ratio is obtained by collecting multiple eye aspect ratios of a student with their eyes closed, and recording the average of these multiple eye aspect ratios as the student's closed aspect ratio. In the actual comparison process, it is necessary to query the normal aspect ratio and closed aspect ratio of the student being compared based on the student's ID. If the aspect ratio to be identified is less than the closed aspect ratio, it indicates that the online student's eyes are closed and their classroom attention is low. If the aspect ratio to be identified is not less than the closed aspect ratio, and the aspect ratio to be identified is less than the normal aspect ratio, it indicates that the online student is listening, but their level of concentration is low. If the aspect ratio to be identified is not less than the normal aspect ratio, it indicates that the online student's attention is highly focused at this time.

[0051] It is clear that the non-focused percentage refers to the ratio of the number of non-focused aspect ratios in the non-focused aspect ratio group to the total number of aspect ratios, where the total number of aspect ratios refers to the number of eye aspect ratios in the eye aspect ratio group. The low focus percentage refers to the ratio of the number of low focus aspect ratios in the low focus aspect ratio group to the total number of aspect ratios. The high focus percentage refers to the ratio of the number of high focus aspect ratios in the high focus aspect ratio group to the total number of aspect ratios. The above calculation of facial focus based on the non-focused percentage, low focus percentage, and high focus percentage refers to setting non-focused level, low focus level, and high focus level, where the non-focused level, low focus level, and high focus level represent the relative concentration of students' classroom attention in non-focused, low focus, and high focus states, respectively. Optionally, the non-focused level, low focus level, and high focus level are set to 1, 2, and 3, respectively, then the formula for calculating facial focus is: ,in, Indicates the level of concentration. , and These represent the percentages of non-focused, low-focused, and high-focused individuals, respectively. , and These represent the levels of inattentiveness, low focus, and high focus, respectively.

[0052] Specifically, the behavioral attention assessment based on the current behavioral label group to obtain behavioral focus includes: Multiple standard behavioral labels are set based on the head posture attention scale; Perform the following operation for each of the multiple standard behavior labels: The standard behavior labels are assigned a focus score to obtain the standard behavior focus score. Identify multiple similar behavior tags that correspond to the standard behavior tag in the current behavior tag group, and count the number of similar tags among the multiple similar behavior tags; Summarize the standard behavior focus level and the number of similar tags for each standard behavior tag to obtain multiple standard behavior focus levels and multiple similar tags. By normalizing the weights of multiple similar tags, multiple focus weights are obtained. The behavioral focus score is obtained by weighting and summing multiple standard behavioral focus scores using multiple focus scores weights.

[0053] It should be explained that the standard behavior label refers to the behavior label in the head posture attention table, including non-face-to-face behavior labels and face-to-face behavior labels. The standard behavior focus refers to a numerical value that quantifies the degree of attention concentration of the behavior corresponding to the standard behavior label relative to (the behavior of other standard behavior labels). The higher the standard behavior focus, the higher the degree of attention concentration of the behavior corresponding to the standard behavior label. For example, the face-to-face behavior label represents the behavior of an online student looking directly at the screen, while the non-face-to-face behavior label represents the behavior of an online student not looking directly at the screen. Therefore, the standard behavior focus of the face-to-face behavior label needs to be greater than that of the non-face-to-face behavior label. Optionally, the standard behavior focus of the face-to-face behavior label and the non-face-to-face behavior label can be set to values ​​of 2 and 1, respectively. The same type of behavior label refers to the current behavior label that has the same behavior label as the standard behavior label. The standard behavior label refers to the number of similar behavior labels among multiple similar behavior labels. The focus weights refer to the weights obtained after weight normalization. Weight normalization of the number of similar tags involves calculating the sum of the numbers of similar tags and dividing the number of each similar tag by this sum. The resulting numerical values ​​are the focus weights. The formula for calculating the weighted sum of multiple standard behavioral focus scores using these focus weights is as follows: ,in, and These represent the focus weights corresponding to non-facing behavior labels and facing behavior labels, respectively. and These represent the standard behavioral focus levels corresponding to non-confrontational behavior labels and frontal behavior labels, respectively.

[0054] S6. Calculate classroom attention based on behavioral focus and facial focus, and match classroom attention with the student IDs corresponding to the extracted clients to be analyzed to obtain classroom attention analysis data.

[0055] Clearly, classroom attention refers to a numerical value quantifying the level of concentration of a particular online student in class. A higher classroom attention value indicates a more focused online student. Optionally, classroom attention is the sum of behavioral focus and facial focus. The classroom attention analysis data refers to a set of data containing classroom attention and student IDs. Matching classroom attention with the extracted student IDs corresponding to the clients being analyzed means placing classroom attention and student IDs into the same data set; this combined data set constitutes the classroom attention analysis data.

[0056] S7. Summarize the classroom attention analysis data corresponding to each client to be analyzed to obtain the classroom attention analysis dataset, and complete the classroom student attention analysis based on machine vision.

[0057] It should be explained that after obtaining the classroom attention analysis dataset, it needs to be sent to the teacher's end, and the teacher's end will display the classroom attention corresponding to each student ID.

[0058] To address the problems described in the background art, this invention first constructs an attention visual analysis model. This model integrates an eye recognition model and a head pose recognition model. Compared to existing technologies that often rely on only a single feature, such as head pose or eye state, for attention analysis, this step improves the comprehensiveness and accuracy of attention analysis through multimodal feature fusion, reducing misjudgments caused by the limitations of single features. Furthermore, visual data is acquired using the attention visual analysis model and classroom visual detection equipment to obtain the current behavior label group and eye aspect ratio group. This step involves continuous image acquisition by setting an attention evaluation cycle, which is significantly more efficient than existing technologies. This invention, which relies on discrete sampling or manual observation, achieves automated and continuous data collection, ensuring the real-time nature and objectivity of attention assessment and avoiding biases caused by human intervention. Finally, it assesses facial attention based on the eye aspect ratio group to obtain facial focus, and assesses behavioral attention based on the current behavior label group to obtain behavioral focus. This dual assessment mechanism comprehensively reflects the student's attention state, improves the accuracy of the assessment results, and reduces the impact of environmental and individual differences on the analysis results. Therefore, this invention can improve the accuracy and real-time nature of student attention analysis in online classrooms and reduce the risk of misjudgment caused by single features and individual differences.

[0059] like Figure 2 The diagram shown is a functional block diagram of a classroom student attention analysis system based on machine vision provided in an embodiment of the present invention.

[0060] The machine vision-based classroom student attention analysis system 100 of this invention can be installed in an electronic device. Depending on the functions implemented, the machine vision-based classroom student attention analysis system 100 may include an analysis instruction receiving module 101, a visual model construction module 102, an attention calculation module 103, and a student data matching module 104. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device. The analysis instruction receiving module 101 is used to receive online classroom analysis instructions and determine the client set to be analyzed based on the online classroom analysis instructions. The client set to be analyzed includes multiple clients to be analyzed, and each client to be analyzed corresponds to a student ID. The visual model construction module 102 is used to construct an attention visual analysis model, wherein the attention visual analysis model includes: an eye recognition model and a head posture recognition model; The attention calculation module 103 is used to sequentially extract clients to be analyzed from the client set, identify the classroom visual detection device of the extracted client, use the attention visual analysis model and the classroom visual detection device to collect visual data, obtain the current behavior label group and the eye aspect ratio group, evaluate facial attention based on the eye aspect ratio group to obtain facial attention, and evaluate behavioral attention based on the current behavior label group to obtain behavioral attention. The student data matching module 104 is used to calculate classroom attention based on behavioral focus and facial focus, match classroom attention with the student IDs corresponding to the extracted clients to be analyzed, obtain classroom attention analysis data, and summarize the classroom attention analysis data corresponding to each client to be analyzed to obtain a classroom attention analysis dataset.

[0061] In detail, the modules in the machine vision-based classroom student attention analysis system 100 described in this embodiment of the invention employ the same methods as described above. Figure 1 The method used is the same as the machine vision-based classroom student attention analysis method described above, and it can produce the same technical effect, so it will not be repeated here.

[0062] like Figure 3 The diagram shown is a structural schematic of an electronic device that implements a machine vision-based classroom student attention analysis method according to an embodiment of the present invention.

[0063] The electronic device 1 may include a processor 10, a memory 11 and a bus 12, and may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a machine vision-based classroom student attention analysis method program.

[0064] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 includes both internal storage units and external storage devices of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a machine vision-based classroom student attention analysis method program, but also to temporarily store data that has been output or will be output.

[0065] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., a machine vision-based classroom student attention analysis method program) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0066] The bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus 12 can be divided into an address bus, a data bus, a control bus, etc. The bus 12 is configured to realize the connection and communication between the memory 11 and at least one processor 10, etc.

[0067] Figure 3 Only electronic devices with components are shown; it will be understood by those skilled in the art that... Figure 3The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0068] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power the various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0069] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0070] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), and optionally, a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0071] The computer vision-based classroom student attention analysis method program stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When run in the processor 10, it can achieve the following: Receive online classroom analysis instructions, determine the client set to be analyzed based on the online classroom analysis instructions, wherein the client set to be analyzed includes multiple clients to be analyzed, and each client to be analyzed corresponds to a student ID; Construct an attention visual analysis model, which includes an eye recognition model and a head pose recognition model; The clients to be analyzed are extracted sequentially from the client set to be analyzed, and the classroom visual detection devices of the extracted clients to be analyzed are identified. Visual data was collected using an attention visual analysis model and classroom visual detection equipment to obtain current behavior label groups and eye aspect ratio groups. Facial attention is assessed based on the eye aspect ratio group to obtain facial focus level, and behavioral attention is assessed based on the current behavior label group to obtain behavioral focus level. Classroom attention is calculated based on behavioral focus and facial focus. Classroom attention is then matched with the student IDs corresponding to the extracted client data to be analyzed, thus obtaining classroom attention analysis data. The classroom attention analysis data corresponding to each client to be analyzed is aggregated to obtain the classroom attention analysis dataset, thus completing the classroom student attention analysis based on machine vision.

[0072] Specifically, the processor 10's implementation method for the above instructions can be found in [reference needed]. Figures 1 to 3 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0073] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0074] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following: Receive online classroom analysis instructions, determine the client set to be analyzed based on the online classroom analysis instructions, wherein the client set to be analyzed includes multiple clients to be analyzed, and each client to be analyzed corresponds to a student ID; Construct an attention visual analysis model, which includes an eye recognition model and a head pose recognition model; The clients to be analyzed are extracted sequentially from the client set to be analyzed, and the classroom visual detection devices of the extracted clients to be analyzed are identified. Visual data was collected using an attention visual analysis model and classroom visual detection equipment to obtain current behavior label groups and eye aspect ratio groups. Facial attention is assessed based on the eye aspect ratio group to obtain facial focus level, and behavioral attention is assessed based on the current behavior label group to obtain behavioral focus level. Classroom attention is calculated based on behavioral focus and facial focus. Classroom attention is then matched with the student IDs corresponding to the extracted client data to be analyzed, thus obtaining classroom attention analysis data. The classroom attention analysis data corresponding to each client to be analyzed is aggregated to obtain the classroom attention analysis dataset, thus completing the classroom student attention analysis based on machine vision.

[0075] In the embodiments provided by this invention, it should be understood that the disclosed devices, systems, and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and actual implementations may have other classification methods.

[0076] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0077] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0078] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A machine vision-based method for analyzing student attention in the classroom, characterized in that, The method includes: Receive online classroom analysis instructions, determine the client set to be analyzed based on the online classroom analysis instructions, wherein the client set to be analyzed includes multiple clients to be analyzed, and each client to be analyzed corresponds to a student ID; Construct an attention visual analysis model, which includes: an eye recognition model and a head pose recognition model; The clients to be analyzed are extracted sequentially from the client set to be analyzed, and the classroom visual detection devices of the extracted clients to be analyzed are identified. Visual data was collected using an attention visual analysis model and classroom visual detection equipment to obtain current behavior label groups and eye aspect ratio groups. Facial attention is assessed based on the eye aspect ratio group to obtain facial focus level, and behavioral attention is assessed based on the current behavior label group to obtain behavioral focus level. Classroom attention is calculated based on behavioral focus and facial focus. Classroom attention is then matched with the student IDs corresponding to the extracted client data to be analyzed, thus obtaining classroom attention analysis data. The classroom attention analysis data corresponding to each client to be analyzed is aggregated to obtain the classroom attention analysis dataset, thus completing the classroom student attention analysis based on machine vision.

2. The classroom student attention analysis method based on machine vision as described in claim 1, characterized in that, The process of using an attention visual analysis model and classroom visual detection equipment to collect visual data yields current behavior label groups and eye aspect ratio groups, including: Receive attention collection instructions and determine the attention evaluation cycle based on the attention collection instructions; Based on the attention collection instructions and classroom visual detection equipment, online classroom images are collected to obtain facial visual images; An eye recognition model is used to identify the aspect ratio of facial visual images, thus obtaining the aspect ratio of the eyes; A head pose recognition model is used to perform behavior recognition on facial visual images to obtain the current behavior label; The current evaluation period is obtained based on the attention collection command; If the current evaluation period is less than the attention evaluation period, return to the step of collecting online classroom images according to the attention collection instruction and the classroom visual detection device until the current evaluation period is not less than the attention evaluation period. If the current assessment period is not less than the attention assessment period, then the current behavior labels and eye aspect ratios are summarized separately to obtain the current behavior label group and the eye aspect ratio group.

3. The classroom student attention analysis method based on machine vision as described in claim 2, characterized in that, The method of using an eye recognition model to identify the aspect ratio of a facial visual image to obtain the eye aspect ratio includes: The face detection layer and eye feature extraction layer in the eye recognition model were identified, wherein the face detection layer contains the face detection algorithm; A face detection layer is used to perform face contour recognition on a face visual image to obtain the face region. The facial region is transferred to the eye feature extraction layer to obtain the eye key pixel set, which includes 6 eye key pixels, namely: outer corner of the eye, center of the upper eyelid, inner corner of the eye, center of the lower eyelid, upper eyelid point, and lower eyelid point. The aspect ratio of the eye is calculated based on the set of key pixels in the eye.

4. The classroom student attention analysis method based on machine vision as described in claim 3, characterized in that, The method of using a head pose recognition model to perform behavior recognition on facial visual images to obtain current behavior labels includes: The facial visual image is input into the head pose recognition model to obtain the current pose rotation matrix, where the current pose rotation matrix is... matrix; Perform Euler angle analysis on the current attitude rotation matrix to obtain the current pitch angle and current yaw angle; Obtain a head posture attention scale, which includes: multiple head posture assessment criteria, wherein the head posture assessment criteria include non-face-viewing behavior labels or face-viewing behavior labels, and each head posture assessment criterion includes: standard pitch angle range and standard yaw angle range. Attention is assessed based on the current pitch and yaw angles using a head posture attention scale to obtain the current behavior label.

5. The classroom student attention analysis method based on machine vision as described in claim 4, characterized in that, Before using a head pose recognition model to perform behavior recognition on a facial visual image and obtain the current behavior label, the method further includes: Obtain a set of face-labeled images, wherein the set of face-labeled images includes multiple face-labeled images, and each face-labeled image contains an Euler angle label; The training set is constructed based on the face-labeled image set, resulting in multiple face training image groups. Each face training image group includes: anchor point labeled image, same pose labeled image, and contrast labeled image. The anchor point labeled image and the same pose labeled image have the same behavior label, while the anchor point labeled image and the contrast labeled image have different behavior labels. Construct the original pose recognition model, which includes a ResNet-50 network and fully connected layers; The original pose recognition model was trained using multiple sets of face training images to obtain a head pose recognition model.

6. The classroom student attention analysis method based on machine vision as described in claim 5, characterized in that, The step of training the original pose recognition model using multiple sets of face training images to obtain a head pose recognition model includes: Based on the original pose recognition model, the following operations are performed on each face training image group in multiple face training image groups: The face training image set is input into the ResNet-50 network in the original pose recognition model to obtain the head pose feature vector set, which includes multiple head pose feature vectors. The total loss value is constructed based on the head pose feature vector group; Summarize the total loss values ​​for each group of face training images to obtain the total loss value group; The original pose recognition model is back-optimized based on the total loss value set and the pre-built optimizer to obtain the head pose recognition model.

7. The classroom student attention analysis method based on machine vision as described in claim 6, characterized in that, The total loss value constructed based on the head pose feature vector group includes: The anchor point posture feature vector, the same posture feature vector, and the comparative posture feature vector in the head posture feature vector group were identified. Calculate the same attitude prediction distance based on the anchor point attitude feature vector and the same attitude feature vector, and calculate the comparison attitude prediction distance based on the anchor point attitude feature vector and the comparison attitude feature vector. Construct triplet loss values ​​based on the predicted distances in the same pose and the predicted distances in contrasting poses; The anchor point pose feature vector is passed to the fully connected layer to obtain the predicted anchor point rotation matrix; Construct a real anchor rotation matrix based on the Euler angle labels corresponding to the anchor mark images, and calculate the pose loss value based on the real anchor rotation matrix and the predicted anchor rotation matrix; The total loss value is obtained by summing the triplet loss value and the attitude loss value.

8. The classroom student attention analysis method based on machine vision as described in claim 7, characterized in that, The assessment of facial attention based on the aspect ratio of the eyes yields facial focus, including: Extract the eye aspect ratios sequentially from the eye aspect ratio group, and record the extracted eye aspect ratios as the aspect ratios to be identified; Compare the preset normal aspect ratio, the preset closed aspect ratio, and the aspect ratio to be identified; If the aspect ratio to be identified is less than the closed aspect ratio, then the aspect ratio to be identified is recorded as the non-focused aspect ratio. If the aspect ratio to be identified is not less than the closed aspect ratio, and the aspect ratio to be identified is less than the normal aspect ratio, then the aspect ratio to be identified is recorded as the low focus aspect ratio. If the aspect ratio to be identified is not less than the normal aspect ratio, then the aspect ratio to be identified is recorded as the high-focus aspect ratio. The non-focused aspect ratio, low-focused aspect ratio, and high-focused aspect ratio are summarized separately to obtain the non-focused aspect ratio group, low-focused aspect ratio group, and high-focused aspect ratio group. The proportions of the non-focused ratio group, the low-focused ratio group, and the high-focused ratio group were statistically analyzed to obtain the proportions of non-focused, low-focused, and high-focused groups. The degree of focus in facial expressions is calculated based on the percentage of non-focused, low-focused, and high-focused individuals.

9. The classroom student attention analysis method based on machine vision as described in claim 8, characterized in that, The step of evaluating behavioral attention based on the current behavioral label group to obtain behavioral focus includes: Multiple standard behavioral labels are set based on the head posture attention scale; Perform the following operation for each of the multiple standard behavior labels: The standard behavior labels are assigned a focus score to obtain the standard behavior focus score. Identify multiple similar behavior tags that correspond to the standard behavior tag in the current behavior tag group, and count the number of similar tags among the multiple similar behavior tags; Summarize the standard behavior focus level and the number of similar tags for each standard behavior tag to obtain multiple standard behavior focus levels and multiple similar tags. By normalizing the weights of multiple similar tags, multiple focus weights are obtained. The behavioral focus score is obtained by weighting and summing multiple standard behavioral focus scores using multiple focus scores weights.

10. A classroom student attention analysis system based on machine vision, characterized in that, The system includes: The analysis instruction receiving module is used to receive online classroom analysis instructions and determine the set of clients to be analyzed based on the online classroom analysis instructions. The set of clients to be analyzed includes multiple clients to be analyzed, and each client to be analyzed corresponds to a student ID. The visual model building module is used to build attention visual analysis models, which include: eye recognition model and head pose recognition model; The focus calculation module is used to sequentially extract the clients to be analyzed from the client set, identify the classroom visual detection device of the extracted clients, use the attention visual analysis model and the classroom visual detection device to collect visual data, obtain the current behavior label group and the eye aspect ratio group, evaluate facial attention based on the eye aspect ratio group to obtain facial focus, and evaluate behavioral attention based on the current behavior label group to obtain behavioral focus. The student data matching module is used to calculate classroom attention based on behavioral focus and facial focus. It matches classroom attention with the student IDs corresponding to the extracted clients to be analyzed to obtain classroom attention analysis data. The classroom attention analysis data corresponding to each client to be analyzed is summarized to obtain a classroom attention analysis dataset.

Images