Learning state monitoring method and device, storage medium and computer equipment

Abstract

The invention discloses a learning state monitoring method and device, a storage medium and computer equipment. The method comprises the following steps: acquiring to-be-tested images of a student in a learning process at different sampling moments; character feature expressions of the student in the to-be-detected image at different sampling moments are recognized, wherein the character feature expressions comprise face feature information and / or target behaviors generated by the student in the learning process; and determining the learning state of the student based on the character feature performance. According to the embodiment of the invention, a series of actual behavior expressions of the students in the whole learning process are integrated, and different learning states of the students can be accurately found, so that corresponding measures can be timely and specifically taken to intervene or encourage and help the students to better learn, the self-study concentration of the students is improved, and the burden of teachers is relieved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a learning state monitoring method, device, storage medium and computer equipment. Background Art

[0002] At present, in the application scenarios of online study rooms, students are often distracted during self-study due to poor self-control, resulting in poor self-study results. Monitoring of students' learning status usually relies on regular checks by teachers. However, these methods often have problems such as high labor costs and difficulty in real-time monitoring, resulting in poor monitoring results and the inability to automatically and accurately judge students' learning status and feedback. Summary of the invention

[0003] In view of this, the present application provides a method, apparatus, storage medium and computer equipment for monitoring the learning status, which can accurately judge the learning status of students through the facial features of students and the target behaviors generated by students during the learning process.

[0004] According to one aspect of the present application, a method for monitoring a learning state is provided, comprising:

[0005] Obtain the images to be tested of students in the learning process at different sampling moments;

[0006] Identify the character features of students in the images to be tested at different sampling times, where the character features include facial feature information and / or target behaviors generated by students during the learning process;

[0007] Determine the learning status of students based on their character traits.

[0008] Optionally, the character feature expression includes facial feature information, and identifying the character feature expression of the student in the image to be tested at different sampling times includes:

[0009] Identify a human body detection frame in the image to be tested;

[0010] Perform face detection on the human body detection frame to obtain the face feature information.

[0011] Optionally, determining the learning status of the student based on facial feature information includes:

[0012] Calculate the cosine similarity between the student's preset facial feature information and the facial feature information obtained at the target sampling time;

[0013] If the cosine similarity at the target sampling moment is less than the first similarity threshold, it is determined that the student is in the leave state.

[0014] Optionally, the image to be tested includes a human body region, the human characteristic performance includes a target behavior generated by the student during the learning process, and the identifying the human characteristic performance of the student in the image to be tested at different sampling times includes:

[0015] Identify a human body detection frame in the human body area;

[0016] Performing posture detection on the human body detection frame to determine the human body posture in the human body detection frame;

[0017] Comparing the human body postures obtained at different sampling times, and determining the amount of posture change between the different sampling times;

[0018] The target behavior is determined based on the posture change amount.

[0019] Optionally, performing posture detection on the human body detection frame to determine the human body posture in the human body detection frame includes:

[0020] Perform human skeleton feature point detection on the human body detection frame to determine the skeleton feature points;

[0021] Calculate the positional relationship between different bone feature points based on the coordinates of the bone feature points;

[0022] Determine human posture based on positional relationships;

[0023] Among them, the bone feature points include at least one of the following: left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, nose tip, mouth center, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left ankle, right ankle.

[0024] Optionally, the learning status monitoring method further includes:

[0025] If the human body posture at the current sampling moment is an abnormal posture, determine a first warning period based on the current sampling moment and the human body posture, wherein the first warning period includes at least one sampling moment before the current sampling moment;

[0026] Compare the human body posture at the current sampling moment with the human body posture identified in the first warning cycle;

[0027] If the human body posture at the current sampling moment is the same as the human body posture identified in the first warning period, abnormal posture prompt information is output based on the human body posture.

[0028] Optionally, determining a target behavior based on the amount of posture change includes:

[0029] Matching the posture change amount with the posture change range of the preset behavior, and determining the preset behavior corresponding to the posture change range to which the posture change amount belongs as the target behavior;

[0030] The target behavior includes at least one of the following: dazed behavior, writing behavior, page turning behavior, reading aloud behavior, watching behavior, and leaving behavior.

[0031] Optionally, the image to be tested includes a learning tool area, the character feature performance includes a target behavior generated by the student during the learning process, and identifying the character feature performance of the student in the image to be tested at different sampling times includes:

[0032] Identify the text line detection box in the learning tool area;

[0033] Perform text detection on the text line detection frame to determine the text in the text line detection frame;

[0034] Compare the text in the text line detection box obtained at different sampling times to determine the text similarity between the different sampling times;

[0035] Identify target behaviors based on text similarity.

[0036] Optionally, comparing the text in the text line detection box obtained at different sampling times to determine the text similarity between the different sampling times includes:

[0037] Compare the text similarity between the handwritten text in the first text line detection box and the handwritten text in the second text line detection box, the first text line detection box is a file line detection box obtained based on the image to be tested at the current sampling moment, and the second text line detection box is a file line detection box obtained based on the image to be tested at the previous sampling moment, and the first text line detection box and the second text line detection box have the same positioning identifier.

[0038] Optionally, the learning status monitoring method further includes:

[0039] Extract the outline information of the text in the text line detection box;

[0040] Input the contour information into the classification model to determine the classification result of the text, the classification result includes handwriting class and printing class, and the classification model is trained based on the printing sample text and the handwriting sample text;

[0041] Based on the position information of the text line detection box corresponding to the printed text in the learning tool area and / or the printed text, a positioning mark of the text line detection box is generated, and the positioning mark is used to indicate the row number, column number or page number of the text line detection box.

[0042] Optionally, target behavior is determined based on text similarity, including:

[0043] If the text similarity is greater than the second similarity threshold, it is determined that the student is in a daze behavior;

[0044] If the text similarity is less than or equal to the second similarity threshold, it is determined that the student has writing behavior.

[0045] Optionally, the learning status monitoring method further includes:

[0046] If multiple human detection frames are detected, the cosine similarities between the facial feature information corresponding to the multiple human detection frames and the preset facial feature information of the student are calculated respectively, and the human detection frames corresponding to the facial feature information whose cosine similarity is less than the first similarity threshold are deleted.

[0047] Optionally, the learning status monitoring method further includes:

[0048] If multiple human detection frames are detected, the center distances between the center point of the image to be detected and the center point of each human detection frame are calculated respectively, and other human detection frames except the target human detection frame are deleted from the multiple human detection frames. The target human detection frame is the human detection frame with the largest center distance.

[0049] Optionally, the learning status monitoring method further includes:

[0050] If no key bone feature points are detected based on the image to be tested, or the coordinates of the bone feature points are not within the specified area of ​​the bone feature points, or the coordinate difference between adjacent boundary points of the learning tool is greater than the preset difference, an adjustment prompt message is output.

[0051] Optionally, the learning status monitoring method further includes:

[0052] If the student's learning state at the current sampling moment is an abnormal learning state, a second warning period is determined based on the current sampling moment and the learning state, and the second warning period includes at least one sampling moment before the current sampling moment;

[0053] Compare the learning state at the current sampling moment with the learning state identified in the second warning period;

[0054] If the learning state at the current sampling moment is the same as the learning state identified in the second warning cycle, an abnormal state prompt message is output.

[0055] According to another aspect of the present application, a learning status monitoring device is provided, comprising:

[0056] An acquisition module is used to acquire images to be tested of students during their learning process at different sampling moments;

[0057] An image processing module is used to identify the character features of students in the images to be tested at different sampling times, where the character features include facial feature information and / or target behaviors generated by students during the learning process;

[0058] The state recognition module is used to determine the learning state of students based on character characteristics.

[0059] Optionally, the character feature expression includes facial feature information, and the image processing module includes:

[0060] The face detection module is used to identify the human body detection frame in the image to be detected; perform face detection on the human body detection frame to obtain the face feature information.

[0061] Optionally, the state recognition module is specifically used to calculate the cosine similarity between the student's preset facial feature information and the facial feature information obtained at the target sampling moment; if the cosine similarity at the target sampling moment is less than a first similarity threshold, it is determined that the student is in the leaving state.

[0062] Optionally, the image to be measured includes a human body area, and the image processing module includes:

[0063] A posture detection module is used to identify a human body detection frame in a human body area; and perform posture detection on the human body detection frame to determine the human body posture in the human body detection frame; compare the human body postures obtained at different sampling times to determine the amount of posture change between different sampling times;

[0064] The behavior recognition module is used to determine the target behavior based on the posture change.

[0065] Optionally, the posture detection module is specifically used to detect human skeleton feature points in a human body detection frame to determine the skeleton feature points; calculate the positional relationship between different skeleton feature points based on the coordinates of the skeleton feature points; and determine the human body posture based on the positional relationship; wherein the skeleton feature points include at least one of the following: left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, nose tip, center of mouth, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left ankle, and right ankle.

[0066] Optionally, the behavior recognition module is specifically used to match the posture change amount with the posture change range of the preset behavior, and determine the preset behavior corresponding to the posture change range to which the posture change amount belongs as the target behavior; wherein the target behavior includes at least one of the following: dazed behavior, writing behavior, page turning behavior, reading aloud behavior, watching behavior, and leaving behavior.

[0067] Optionally, the learning status monitoring device further includes:

[0068] The posture abnormality detection module is used to determine a first warning period based on the current sampling time and the human body posture if the human body posture at the current sampling time is an abnormal posture, and the first warning period includes at least one sampling time before the current sampling time; and compare the human body posture at the current sampling time with the human body posture identified in the first warning period;

[0069] The first prompt module is used to output abnormal posture prompt information based on the human body posture if the human body posture at the current sampling moment is the same as the human body posture recognized in the first warning period.

[0070] Optionally, the image to be tested includes a learning tool area, and the image processing module includes:

[0071] A text detection module is used to identify a text line detection frame in the learning tool area; and perform text detection on the text line detection frame to determine the text in the text line detection frame; and compare the text in the text line detection frame obtained at different sampling times to determine the text similarity between different sampling times;

[0072] The behavior recognition module is used to determine the target behavior based on text similarity.

[0073] Optionally, the text detection module is specifically used to compare the text similarity between the handwritten text in the first text line detection box and the handwritten text in the second text line detection box, the first text line detection box is a file line detection box obtained based on the image to be tested at the current sampling moment, and the second text line detection box is a file line detection box obtained based on the image to be tested at the previous sampling moment, and the first text line detection box and the second text line detection box have the same positioning identifier.

[0074] Optionally, the text detection module is further used to extract contour information of the text in the text line detection box;

[0075] The learning status monitoring device also includes:

[0076] The classification module is used to input the contour information into the classification model to determine the classification results of the text. The classification results include handwritten and printed categories. The classification model is trained based on the printed sample text and the handwritten sample text; based on the position information of the text line detection box corresponding to the printed text in the learning tool area and / or the printed text, a positioning mark of the text line detection box is generated. The positioning mark is used to indicate the line number, column number or page number of the text line detection box.

[0077] Optionally, the behavior recognition module is specifically used to determine that the student has a dazed behavior if the text similarity is greater than a second similarity threshold; if the text similarity is less than or equal to the second similarity threshold, determine that the student has a writing behavior.

[0078] Optionally, the image processing module is also used to calculate the cosine similarity between the facial feature information corresponding to the multiple human detection frames and the preset facial feature information of the student if multiple human detection frames are detected, and delete the human detection frames corresponding to the facial feature information whose cosine similarity is less than the first similarity threshold.

[0079] Optionally, the image processing module is also used to calculate the center distance between the center point of the image to be detected and the center point of each human detection frame if multiple human detection frames are detected, and delete other human detection frames among the multiple human detection frames except the target human detection frame, and the target human detection frame is the human detection frame with the largest center distance.

[0080] Optionally, the learning status monitoring device further includes:

[0081] The second prompt module is used to output adjustment prompt information if no key bone feature points are detected based on the image to be tested, or the coordinates of the bone feature points are not within the specified area of ​​the bone feature points, or the coordinate difference between adjacent boundary points of the learning tool is greater than a preset difference.

[0082] Optionally, the learning status monitoring device further includes:

[0083] The state abnormality detection module is used to determine a second warning period based on the current sampling time and the learning state if the student's learning state at the current sampling time is an abnormal learning state, and the second warning period includes at least one sampling time before the current sampling time; and compare the learning state at the current sampling time with the learning state identified in the second warning period;

[0084] The third prompt module is used to output abnormal state prompt information if the learning state at the current sampling moment is the same as the learning state identified in the second warning period.

[0085] According to another aspect of the present application, a readable storage medium is provided, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the above-mentioned learning state monitoring method are implemented.

[0086] According to another aspect of the present application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, and the steps of the above-mentioned learning status monitoring method are implemented when the processor executes the program.

[0087] By means of the above technical scheme, the images to be tested of the students in the learning process at different sampling moments are processed respectively, and the character features of the students in each image to be tested are identified. The facial feature information in the character features obtained at different sampling moments is used to locate whether the student is on camera, and then determine whether the student has left his seat. Or the changes in actions between different sampling moments can be used to determine whether the student is staying focused or gradually slacking off, distracted, etc., and then analyze a series of actual behavioral performances of the students in the entire learning process, and determine whether the students have taken learning-related behaviors. In this way, the learning status of the students can be matched by the determined facial feature information and the target behavior, making the evaluation of the learning status more objective and scientific. In this way, the different learning states of the students can be accurately discovered, so that timely and targeted measures can be taken to intervene or encourage, help students learn better, improve students' self-study concentration, and reduce the burden on teachers.

[0088] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0090] Figure 1 A schematic diagram showing an application environment of a learning status monitoring method provided in an embodiment of the present application is shown;

[0091] Figure 2 One of the flow charts of the method for monitoring the learning status provided in the embodiment of the present application is shown;

[0092] Figure 3 The second flowchart of the method for monitoring the learning status provided in the embodiment of the present application is shown;

[0093] Figure 4 A structural block diagram of a learning status monitoring device provided in an embodiment of the present application is shown;

[0094] Figure 5 A schematic diagram of a human body detection frame and skeleton feature points in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0095] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0096] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as limiting the present application.

[0097] Those skilled in the art will appreciate that, unless expressly stated, the singular forms "a", "an", "said" and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, or there may be intermediate elements. In addition, the "connection" or "connection" used herein may include wireless connection or wireless fusion. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0098] Now, exemplary embodiments according to the present application will be described in more detail with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in a variety of different forms and should not be interpreted as being limited to the embodiments set forth herein. It should be understood that these embodiments are provided to make the disclosure of the present application thorough and complete, and to fully convey the concepts of these exemplary embodiments to those of ordinary skill in the art.

[0099] The learning status monitoring method provided in the embodiment of the present application can be applied in the following aspects: Figure 1 In an application environment, the client communicates with the server through a network. The server can obtain the test images of students in the learning process through the video or image captured by the camera component on the client, and obtain the test images of students in the learning process at different sampling times; identify the facial feature information in the test images at different sampling times and the target behavior generated by the students in the learning process; determine the learning status of the students based on the facial feature information and the target behavior. Finally, when the learning status is abnormal, feedback prompt information is returned to the client to prompt the students to improve their concentration. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablets, learning machines and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0100] In this embodiment, a method for monitoring a learning state is provided, such as Figure 2 As shown, the method includes:

[0101] Step 210, obtaining images to be tested of students during their learning process at different sampling moments.

[0102] The image to be tested includes a human body area and a learning tool area. The learning tools include books, homework, notes, stationery and other learning-related items.

[0103] It is understandable that the learning status monitoring method of the present application can be applied to learning scenarios such as live classes and self-study. The sampling time can be reasonably set according to different scenarios or network environment parameters in different scenarios. For example, when the network is busy or in a scenario where the teacher can observe the student status, the interval between different sampling times can be appropriately extended to save system resources; when in an unaccompanied self-study scenario, the interval between different sampling times can be shortened to better capture student behavior changes, which helps to increase the accuracy of learning status monitoring.

[0104] Step 220, identifying the character features of the students in the image to be tested at different sampling times.

[0105] The character feature representation includes facial feature information and / or target behaviors generated by students during the learning process.

[0106] In actual application scenarios, when the character feature expression includes facial feature information, the character feature expression of students in the image to be tested at different sampling times is identified in step 220, specifically including: identifying a human body detection frame in the image to be tested; performing face detection on the human body detection frame to obtain facial feature information.

[0107] Among them, face detection can be performed using a face detection algorithm based on deep learning, such as some models under the Convolutional Neural Networks (CNN) architecture.

[0108] like Figure 2 As shown, when the character feature performance includes the target behavior generated by the students in the learning process, in the actual application scenario, the character feature performance of the students in the image to be tested at different sampling moments is identified in step 220, which specifically includes the following steps:

[0109] Step 221, identifying the text line detection frame in the learning tool area and the human body detection frame in the human body area.

[0110] It is understandable that the identification of the text line detection frame in the learning tool area in step 221 may be performed in the following manner:

[0111] Method 1: Since the edges of text lines are usually obvious, edge detection operators (such as Sobel operator, Canny operator, etc.) can be used to calculate the gradient of pixels in the image to determine the edges. Based on edge detection, morphological operations are performed to make the edges of text lines more closely connected. The contours of all texts are obtained through contour extraction algorithms (such as the findContours function in the OpenCV library), and the contours that meet the characteristics of text lines are screened out as detection boxes by setting appropriate thresholds.

[0112] Method 2: Use a large number of labeled learning tool area images to train the object detection model, such as Faster-RCNN, YOLO (You Only Look Once) series, SSD (Single Shot MultiBox Detector), etc. The sub-image of the learning tool area segmented from the image to be tested is input into the trained model to determine the text line detection box in the learning tool area for selecting text.

[0113] Step 222: Perform posture detection on the human body detection frame to determine the human body posture in the human body detection frame.

[0114] In an actual application scenario, step 222, that is, performing posture detection on the human body detection frame to determine the human body posture in the human body detection frame, specifically includes the following steps:

[0115] Step 222-1, perform human skeleton feature point detection on the human body detection frame to determine the skeleton feature points.

[0116] Among them, Figure 5 As shown, the skeleton feature points are located in the human body detection frame, and the skeleton feature points include at least one of the following: left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, nose tip, mouth center, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left ankle, and right ankle.

[0117] Specifically, openpose, densepose, etc. can be used to detect multiple skeletal feature points of students and identify the nodes of the human head, torso, and limbs.

[0118] Furthermore, in one embodiment, after step 222-1, the learning status monitoring method also includes: if no key bone feature points are detected based on the image to be tested, or the coordinates of the bone feature points are not within the specified area of ​​the bone feature points, or the coordinate difference between adjacent boundary points of the learning tool is greater than a preset difference, outputting adjustment prompt information.

[0119] In this embodiment, after the system acquires the image to be tested, the human body detection frame in the image performs human skeleton feature point detection to determine the human body detection frame and skeleton feature points. If the key skeleton feature points cannot be detected or the detected skeleton feature points are deviated due to the partial obstruction of the human body, the user can be reminded to adjust the posture or change the shooting angle by adjusting the prompt information. Similarly, after the system acquires the image to be tested, it detects the boundary points of the learning tool and calculates the distance (coordinate difference) between adjacent boundary points. The distance is used to determine whether the image contains a complete learning tool boundary. If the coordinate difference between the adjacent boundary points of the learning tool is greater than the preset difference, that is, the learning tool partially exceeds the camera field of view, the user can be reminded to adjust the position of the learning tool or change the shooting angle by adjusting the prompt information. This ensures that the user's body and learning tool can be collected relatively completely by the camera, providing reliable data support for subsequent learning status monitoring.

[0120] It is worth mentioning that the key skeletal feature points can be different for different scenarios. For example, in the study room scene, considering that the lower body of the student is blocked by the desktop most of the time, the key points of the student's face and upper body play a more important role in subsequent judgments, then the facial (left eye center, right eye center, nose tip, mouth center) skeletal feature points and upper body (left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist) skeletal feature points can be used as key skeletal feature points. The designated area of ​​skeletal feature points can also be reasonably set according to the user's preset body shape and contour, so as to better detect whether the user is in the camera's field of view. Similarly, the preset difference can also be reasonably set according to the type of learning tool.

[0121] For example, the corner points of the image Z0 to be tested acquired by the camera on the client are monitored to obtain the coordinates C of the four corner points of the operation. (m,0) , where m represents the mth corner point and 0 represents the initial moment. (m,0) The specific expression is C (m,0) =(x,y), 1≤m≤4, x, y represent the horizontal and vertical coordinates. (m,0) Are they all within the specified operation area? If not, prompt the user to make adjustments. Figure 4 As shown, the human skeleton key points can also be detected on the image Z0 to be tested by the camera to obtain the key point coordinates P (i,j,0) and human detection frame B (i,0) , where i represents the i-th person, j represents the j-th key point, and 0 represents the initial moment. The human skeleton key point detection can obtain the horizontal and vertical coordinate values ​​of each key point on the image and the classification label of whether it is visible, that is, P (i,j,0)=(x,y,visibility),1≤j≤10,visibility∈{visible,invisible}. The human body detection frame format is the minimum and maximum values ​​of the horizontal coordinate of the frame, and the minimum and maximum values ​​of the vertical coordinate, that is, B (i,0) =(x min ,x max ,y min ,y max ). When there is no result of human body detection, the user is prompted to place the learning machine correctly to ensure that the user is in the frame. When the result of human body detection is only one human body detection frame B0 * . Determine whether the key points of the user's left eye center, right eye center, nose tip and mouth center are all visible (visibility=visible), and the horizontal and vertical coordinates are all within the specified area of ​​the head; and determine whether the key points of the user's left shoulder, right shoulder, left elbow, right elbow, left wrist and right wrist are all visible (visibility=visible), and the horizontal and vertical coordinates are all within the specified area of ​​the upper limbs; when any of the above two conditions of the human body detection frame are not met, prompt the user to adjust.

[0122] Step 222 - 2 , calculating the positional relationship between different bone feature points based on the coordinates of the bone feature points.

[0123] Step 222-3, determining the human body posture based on the position relationship.

[0124] In this embodiment, by determining the positional relationship between the skeleton feature points to detect the student's action details, it is possible to accurately identify whether the student is standing, sitting, reading (the head and eyes are in a relatively stable position, with both hands holding a book), or supporting the chin (the arm supports the head). This enables the accurate determination of the rotation angle, bending degree, and running trajectory of various parts of the human body, helping teachers to correct students' postures in a timely manner, and further detecting students' learning status through posture changes, which helps to improve the quality of students' learning.

[0125] It is understandable that the system has pre-stored judgment rules for different postures, and the human posture appearing in the current image to be tested can be determined by matching the positional relationship between the skeleton feature points with the judgment rules for different postures. (i,j,t) Position relationship, judge the student's learning status:

[0126] (1) When the feature points corresponding to both eyes are detected, but the feature points corresponding to the nose are not detected, and the longitudinal positions of the feature points of both eyes are within a certain range, it is recorded that the student has a posture of lying on the table in the current frame;

[0127] (2) When it is detected that the absolute value of the longitudinal difference between the feature points of the left shoulder and the right shoulder is greater than a certain threshold and both are visible, it is recorded that the student has a slanted shoulder posture in the current frame;

[0128] (3) When it is detected that only the feature points of the left shoulder or the right shoulder are visible, it is recorded that the student has a sideways posture in the current frame;

[0129] (4) When it is detected that all feature points are within a certain range and are visible, the student’s sitting posture deviation in the current frame is recorded;

[0130] (5) When it is detected that the relative horizontal inclination angle formed by the midpoint of the line connecting the two eyes and the nose is less than a certain threshold and both are visible, it is recorded that the student is learning a head-tilted posture in the current frame;

[0131] (6) When it is detected that the wrist feature point is higher than the table, the student’s hand-raising posture is recorded in the current frame;

[0132] (7) When it is detected that the feature points of the left shoulder, right shoulder, left elbow and right elbow are all visible, but only the feature point of the left wrist or the right wrist is visible, the student's posture of learning to write homework with one hand is recorded in the current frame;

[0133] (8) When it is detected that the fingertip feature point is located at the center of the palm, the student's pen-holding posture in the current frame is recorded.

[0134] Furthermore, in one embodiment, after step 224, the learning status monitoring method also includes: if the human body posture at the current sampling moment is an abnormal posture, determining a first warning period based on the current sampling moment and the human body posture; comparing the human body posture at the current sampling moment with the human body posture identified in the first warning period; if the human body posture at the current sampling moment is the same as the human body posture identified in the first warning period, outputting abnormal posture prompt information based on the human body posture.

[0135] The preset first warning period includes at least one sampling time before the current sampling time. The setting of the first warning period is usually determined based on the consideration of the duration of the abnormal situation in the specific application scenario and the expectation of giving appropriate observation time.

[0136] Specifically, abnormal postures include bad sitting postures such as leaning over the table, slouching shoulders, leaning sideways, sitting with an offset posture or tilting the head.

[0137] In this embodiment, when it is detected that the human body posture is abnormal and the abnormal posture is maintained for a long time, the abnormal posture is prompted to inform the user that the current posture is not standard and may affect physical health. On the one hand, by comparing the current sampling moment with the posture in the first warning period, dynamic monitoring of the human body posture can be achieved to ensure that abnormal situations can be discovered in time at any time, and the correction function of the user's posture is realized, which is convenient for the user to adjust the posture in time. On the other hand, by dynamically configuring the first warning period based on the current sampling moment and the human body posture, the adverse effects of different postures on the body are fully considered, the tolerance rate of posture correction is improved, and excessive reminders are effectively prevented from causing negative emotions to users.

[0138] For example, if the current frame detects that the student has a sitting posture of lying on the table, leaning shoulders, leaning sideways, sitting offset, tilting the head, or writing homework with one hand. If there are abnormal sitting posture records of the previous two frames, i.e. 20 seconds ago and 40 seconds ago, read the two recording results; if the results of these two frames have one or more abnormal behaviors that are the same as the current frame, output that the student has this or these abnormal sitting postures; if these two frames do not have one or more abnormal sitting postures that are the same as the current frame or there are no recording results of the previous two frames, no prompt will be given and the next frame of image detection will be carried out.

[0139] Step 223, performing text detection on the text line detection box to determine the text in the text line detection box.

[0140] Among them, algorithms such as optical character recognition (OCR) can be used for text detection.

[0141] Step 224, compare the text in the text line detection box obtained at different sampling times to determine the text similarity between the different sampling times; and / or compare the human body postures obtained at different sampling times to determine the posture changes between the different sampling times.

[0142] In this embodiment, by comparing the similarity of handwritten texts in the text line detection box at different sampling moments, the writing progress of students during the sampling interval is analyzed. Thus, by trying to identify the actions generated by students from the perspective of text changes, it is possible to timely discover the changes in the status of students in the learning process, which is convenient for teachers and other relevant personnel to quickly understand the situation and take corresponding measures. Similarly, different postures correspond to different positional relationships of different parts. The degree of change of human body movements can be determined by the changes in the positional relationship of the same part obtained at different times, that is, the amount of posture change. Then, the behavior generated by students over a period of time can be reflected by the amount of posture change. In this way, the actions of students can be captured more efficiently and flexibly, and the accuracy of behavioral action detection can be improved.

[0143] It is understandable that for different behaviors, the sampling intervals required to calculate the amount of posture change may be the same or different. After determining the preset behavior to be matched, the human posture data identified by the image to be detected at different sampling moments are extracted according to the corresponding sampling interval. For example, if the arm slides from left to right in a shorter sampling interval, it can be determined that the page turning behavior has occurred; if the student maintains a sideways sitting posture in a longer sampling interval, it can be determined that the dazed behavior has occurred.

[0144] It is understandable that for different behaviors, the sampling intervals required to calculate the amount of posture change may be the same or different. After determining the preset behavior to be matched, the human posture data identified by the image to be detected at different sampling moments are extracted according to the corresponding sampling interval. For example, if the arm slides from left to right in a shorter sampling interval, it can be determined that the page turning behavior has occurred; if the student maintains a sideways sitting posture in a longer sampling interval, it can be determined that the dazed behavior has occurred.

[0145] In actual application scenarios, step 224 compares the text in the text line detection box obtained at different sampling times to determine the text similarity between the different sampling times, specifically including: comparing the text similarity between the handwritten text in the first text line detection box and the handwritten text in the second text line detection box.

[0146] Among them, the first text line detection frame is a file line detection frame obtained based on the image to be tested at the current sampling moment, and the second text line detection frame is a file line detection frame obtained based on the image to be tested at the previous sampling moment. The first text line detection frame and the second text line detection frame have the same positioning identifier, and the positioning identifier is used to indicate the row number, column number or page number of the text line detection frame to ensure the subsequent accurate comparison and analysis of the corresponding text line detection frames at different sampling moments.

[0147] Furthermore, in one embodiment, the learning status monitoring method also includes: extracting contour information of the text in the text line detection box; inputting the contour information into the classification model to determine the classification result of the text, the classification result includes handwriting class and printing class; based on the position information of the text line detection box corresponding to the printed text in the learning tool area and / or the printed text, generating a positioning mark of the text line detection box.

[0148] The classification model is trained based on printed sample text and handwritten sample text. The classification model can learn the different patterns of the two types of text in terms of contour features, and can accurately and quickly determine whether the text is handwritten or printed based on the learned rules.

[0149] In this embodiment, by extracting the contour information of the text in the text line detection box, the morphological characteristics of the text can be obtained from the visual feature level. The contours of handwritten text are often relatively irregular, and the thickness changes of the strokes and the shapes of the starting and ending points are relatively natural and casual; while the contours of printed text are relatively regular and the lines are smooth. By extracting the contour information, these differences can be captured, thereby providing a classification basis for the text content identified in the figure. In order to identify the writing progress of students through handwritten text, and to determine the row number, column number or page number of the text line detection box through the position information of the printed text and / or the text line detection box in the learning tool area, it is convenient to compare the content of the handwritten text at a certain characteristic position.

[0150] Step 225 , determining the target behavior based on the text similarity and / or the gesture change amount.

[0151] In actual application scenarios, the target behavior is determined based on text similarity in step 225, specifically including: if the text similarity is greater than the second similarity threshold, it is determined that the student has a daydreaming behavior; if the text similarity is less than or equal to the second similarity threshold, it is determined that the student has a writing behavior.

[0152] In this embodiment, by comparing the similarity of handwritten texts in the text line detection box at different sampling moments, the student's writing progress during the sampling interval is analyzed. If the text similarity is greater than the second similarity threshold, that is, the student has written less text, then it can be determined that the student is in a daze. On the contrary, it means that the text content in the two images to be tested has been updated and changed, which is consistent with the characteristics of normal writing in which text is constantly generated and modified. The student has written more text, then it can be determined that the student has writing behavior. Thus, by trying to identify the actions generated by the students from the perspective of text changes, it is possible to timely discover the changes in the students' status during the learning process, making it easier for teachers and other relevant personnel to quickly understand the situation and take corresponding measures.

[0153] Among them, the second similarity threshold is used to determine whether the student has made a certain writing progress, that is, whether there is a continuous writing action. The second similarity threshold is associated with the duration between the current sampling moment and the previous sampling moment. Therefore, the second similarity threshold can be reasonably set according to the sampling interval and the student's writing habits, thereby avoiding misjudgment caused by different writing speeds of students. Taking writing speed as an example, the student's writing speed can be obtained based on the historical monitoring data generated during the learning process. For the same sampling interval, the second similarity threshold of students with fast writing speed can be set larger than that of students with slow writing speed.

[0154] In actual application scenarios, determining the target behavior based on the posture change amount in step 225 specifically includes: matching the posture change amount with the posture change range of the preset behavior, and determining the preset behavior corresponding to the posture change range to which the posture change amount belongs as the target behavior.

[0155] The target behavior includes but is not limited to at least one of the following: dazed behavior, writing behavior, page turning behavior, reading aloud behavior, watching behavior, and leaving behavior.

[0156] In this embodiment, different postures correspond to different positional relationships of different parts. The degree of change of human body movements can be determined by the changes in the positional relationship of the same part obtained at different times, that is, the posture change amount. Then, the posture change amount is matched with the posture change range corresponding to the preset behavior to obtain the target behavior that matches the posture change amount. This allows more efficient and flexible capture of students' movements and improves the accuracy of behavior and action detection.

[0157] For example, during the reading process, the head usually nods up and down regularly (neck-head joint angle changes), and the range may be around ±10°; the shoulders will sway slightly, and the swing angle of the left and right shoulders may vary within ±5°; the mouth will have obvious opening and closing movements, and the mandibular-maxillary joint angle may vary widely, perhaps around ±30°. As for limb position, the arms may hold books or papers, and the angle between the arms and the body remains relatively stable, with a range of about ±10°. The arm length remains basically unchanged, and the change is within 2 cm. However, the calculated posture change meets the above conditions, and it can be determined that the student is reading aloud.

[0158] It should be noted that for the same preset behavior, multiple groups of different posture change ranges can be set. For example, the dazed behavior can correspond to a posture change of 0 for the chin-supporting posture with both hands, and no change for 10 minutes, or a head tilt angle change within the range of ±20°.

[0159] It is worth mentioning that, in one embodiment, when multiple human body detection frames are detected, in order to accurately identify the learning status of the student himself, the human body detection frames can be filtered in the following manner:

[0160] Method 1: calculate the cosine similarity between the facial feature information corresponding to multiple human detection frames and the preset facial feature information of the student, and delete the human detection frames corresponding to the facial feature information whose cosine similarity is less than the first similarity threshold.

[0161] In this embodiment, the extracted facial feature information is converted into a vector form, and the preset facial feature information of the student associated with the account is also represented by a vector. The cosine value of the angle between the two vectors is calculated to reflect the similarity between the face in the image and the student himself. If the cosine similarity is less than the first similarity threshold, it means that the face appearing in the human body detection frame is not the student to be monitored, and the human body detection frame can be deleted. This helps to ensure the strictness of learning status monitoring, prevent the occurrence of violations such as substitution, and only leave the student's effective detection information for further behavior observation and analysis, thereby improving the accuracy and pertinence of data processing.

[0162] It is understandable that if all human body detection frames in the image to be tested are deleted, it means that the student himself is not in the image. The subsequent posture and writing progress analysis steps can be omitted, and the student can be directly judged to be in the leaving state.

[0163] Method 2: If multiple human detection frames are detected, the center distance between the center point of the image to be tested and the center point of each human detection frame is calculated respectively, and other human detection frames except the target human detection frame are deleted from the multiple human detection frames. The target human detection frame is the human detection frame with the largest center distance.

[0164] In this embodiment, the center distance between the center point of each human body detection frame in the image to be tested and the center point of the image to be tested is calculated respectively. The human body detection frame closest to the center point of the image to be tested is selected as the human body detection frame of the student himself, and other human body detection frames other than the student himself are ignored. In this way, the human body detection frame will not participate in the subsequent skeletal feature point detection, so that subsequent analysis, monitoring and other operations are carried out around the target student, reducing unnecessary data interference, avoiding misjudgment caused by multiple people entering the camera at the same time, and improving the accuracy and pertinence of data processing.

[0165] Step 230, determining the student's learning status based on the character characteristics.

[0166] The monitoring method of learning status proposed in the embodiment of the present application processes the images to be tested of the students in the learning process at different sampling moments respectively, and identifies the character feature performance of the students in each image to be tested. The facial feature information in the character feature performance obtained at different sampling moments is used to locate whether the student is on camera, and then judge whether the student has left the seat. Or from the changes in actions occurring between different sampling moments, it can be judged whether the student remains focused or gradually becomes lazy, distracted, etc., and then analyze a series of actual behavioral performances of the students in the entire learning process, and judge whether the students have taken learning-related behaviors. In this way, the learning status of the students can be matched by the determined facial feature information and the target behavior, making the evaluation of the learning status more objective and scientific. Thereby, the different learning states of the students can be accurately discovered, so that timely and targeted measures can be taken to intervene or encourage, help students learn better, improve students' self-study concentration, and reduce the burden on teachers.

[0167] In actual application scenarios, when the character feature expression includes facial feature information, step 230, that is, determining the student's learning status based on the character feature expression, specifically includes: calculating the cosine similarity between the student's preset facial feature information and the facial feature information obtained at the target sampling moment; if the cosine similarity at the target sampling moment is less than the first similarity threshold, it is determined that the student is in the leaving state.

[0168] Among them, the target sampling time is the sampling time that meets the preset sampling interval. The preset sampling interval can be reasonably set according to the monitoring requirements of the departure state, for example, 30 minutes, 1 hour, etc.

[0169] In this embodiment, the cosine value of the vector angle between the preset facial feature information and the facial feature information detected in the image is calculated to reflect the similarity between the face in the image and the student himself. If the cosine similarity calculated based on any frame image is less than the first similarity threshold, it means that the face appearing in the human body detection frame is not the student to be monitored, that is, the student himself is not in the frame, and it can be directly determined that the student is in the away state. In this way, the faces in a single frame can be quickly screened, and the judgment result on whether the student is in the away state can be given in a relatively short time, meeting the needs of real-time monitoring.

[0170] In actual application scenarios, when the character feature expression includes the target behavior, step 230, that is, determining the student's learning status based on the character feature expression, specifically includes: matching the target behavior with the preset behavior corresponding to the preset learning state, and determining the preset learning state corresponding to the preset behavior identical to the target behavior as the student's learning state.

[0171] In this embodiment, the details of the learning state are quantified through the behaviors that may exist during the learning period. The learning state is determined by matching the target behaviors detected from the images at different times with the preset behaviors, which can accurately determine the learning state of the student and ensure the reliability of the student state monitoring.

[0172] For example, the system presets a correspondence between learning status and behavior, and the learning status of the student can be matched according to the target behavior through the correspondence. For example, if the student currently has a leaving behavior, it can be determined that the student is in the leaving state. If the student has a writing behavior or a page turning behavior, it can be determined that the student is in a focused state.

[0173] Furthermore, in one embodiment, after step 230, the learning status monitoring method also includes: if the student's learning status at the current sampling moment is an abnormal learning state, determining the second warning cycle based on the current sampling moment and the learning status; comparing the learning status at the current sampling moment with the learning status identified in the second warning cycle; if the learning status at the current sampling moment is the same as the learning status identified in the second warning cycle, outputting abnormal status prompt information.

[0174] The preset second warning period includes at least one sampling time before the current sampling time. The setting of the second warning period is usually determined based on the consideration of the duration of the abnormal situation in the specific application scenario and the expectation of giving appropriate observation time.

[0175] Specifically, abnormal postures include walking away, being in a daze, and other bad states.

[0176] In this embodiment, the same principle as sitting posture correction is adopted. When an abnormal learning state occurs, the learning state at the current sampling moment is compared with the learning state identified in the second warning cycle corresponding to the abnormal learning state; if they are the same, the abnormal state prompt information is output based on the current abnormal learning state. This can timely remind users to enhance their concentration, help students learn better, improve students' self-study concentration, and reduce the burden on teachers.

[0177] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0178] Furthermore, if Figure 4 As shown, as a specific implementation of the above-mentioned learning status monitoring method, an embodiment of the present application provides a learning status monitoring device 300, and the learning status monitoring device 300 includes: an acquisition module 301, an image processing module 302, and a status recognition module 303.

[0179] Wherein, the acquisition module 301 is used to acquire the images to be tested of the students in the learning process at different sampling moments;

[0180] The image processing module 302 is used to identify the character feature performance of the students in the images to be tested at different sampling moments, where the character feature performance includes facial feature information and / or target behaviors generated by the students during the learning process;

[0181] The state recognition module 303 is used to determine the learning state of the student based on the character characteristics.

[0182] Furthermore, the character feature expression includes facial feature information, and the image processing module 302 includes: a face detection module (not shown in the figure).

[0183] The face detection module is used to identify the human body detection frame in the image to be detected; perform face detection on the human body detection frame to obtain the face feature information.

[0184] Furthermore, the state recognition module 303 is specifically used to calculate the cosine similarity between the student's preset facial feature information and the facial feature information obtained at the target sampling moment; if the cosine similarity at the target sampling moment is less than the first similarity threshold, it is determined that the student is in the leaving state.

[0185] Further, the image to be measured includes a human body area, and the image processing module 302 includes: a posture detection module (not shown in the figure) and a behavior recognition module (not shown in the figure);

[0186] A posture detection module is used to identify a human body detection frame in a human body area; and perform posture detection on the human body detection frame to determine the human body posture in the human body detection frame; compare the human body postures obtained at different sampling times to determine the amount of posture change between different sampling times;

[0187] The behavior recognition module is used to determine the target behavior based on the posture change.

[0188] Furthermore, the posture detection module is specifically used to detect human skeleton feature points in a human body detection frame to determine the skeleton feature points; calculate the positional relationship between different skeleton feature points based on the coordinates of the skeleton feature points; and determine the human body posture based on the positional relationship; wherein the skeleton feature points include at least one of the following: left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, nose tip, mouth center, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left ankle, and right ankle.

[0189] Furthermore, the behavior recognition module is specifically used to match the posture change amount with the posture change range of the preset behavior, and determine the preset behavior corresponding to the posture change range to which the posture change amount belongs as the target behavior; wherein the target behavior includes at least one of the following: dazed behavior, writing behavior, page turning behavior, reading aloud behavior, watching behavior, and leaving behavior.

[0190] Furthermore, the learning state monitoring device 300 further includes: a posture abnormality detection module (not shown in the figure) and a first prompt module;

[0191] The posture abnormality detection module is used to determine a first warning period based on the current sampling time and the human body posture if the human body posture at the current sampling time is an abnormal posture, and the first warning period includes at least one sampling time before the current sampling time; and compare the human body posture at the current sampling time with the human body posture identified in the first warning period;

[0192] The first prompt module is used to output abnormal posture prompt information based on the human body posture if the human body posture at the current sampling moment is the same as the human body posture recognized in the first warning period.

[0193] Furthermore, the image to be tested includes a learning tool area, and the image processing module 302 includes: a text detection module (not shown in the figure) and a behavior recognition module (not shown in the figure).

[0194] A text detection module is used to identify a text line detection frame in the learning tool area; and perform text detection on the text line detection frame to determine the text in the text line detection frame; and compare the text in the text line detection frame obtained at different sampling times to determine the text similarity between different sampling times;

[0195] The behavior recognition module is used to determine the target behavior based on text similarity.

[0196] Furthermore, the text detection module is specifically used to compare the text similarity between the handwritten text in the first text line detection box and the handwritten text in the second text line detection box. The first text line detection box is a file line detection box obtained based on the image to be tested at the current sampling moment, and the second text line detection box is a file line detection box obtained based on the image to be tested at the previous sampling moment. The first text line detection box and the second text line detection box have the same positioning identifier.

[0197] Furthermore, the learning status monitoring device further includes: a classification module (not shown in the figure);

[0198] The text detection module is also used to extract the outline information of the text in the text line detection box;

[0199] The classification module is used to input the contour information into the classification model to determine the classification results of the text. The classification results include handwritten and printed categories. The classification model is trained based on the printed sample text and the handwritten sample text; based on the position information of the text line detection box corresponding to the printed text in the learning tool area and / or the printed text, a positioning mark of the text line detection box is generated. The positioning mark is used to indicate the line number, column number or page number of the text line detection box.

[0200] Furthermore, the behavior recognition module is specifically used to determine that the student has a dazed behavior if the text similarity is greater than a second similarity threshold; if the text similarity is less than or equal to the second similarity threshold, determine that the student has a writing behavior.

[0201] Furthermore, the learning status monitoring device 300 further includes: a second prompting module (not shown in the figure);

[0202] The second prompt module is used to output adjustment prompt information if no key bone feature points are detected based on the image to be tested, or the coordinates of the bone feature points are not within the specified area of ​​the bone feature points, or the coordinate difference between adjacent boundary points of the learning tool is greater than a preset difference.

[0203] Furthermore, the image processing module 302 is also used to calculate the cosine similarity between the facial feature information corresponding to the multiple human detection frames and the preset facial feature information of the student if multiple human detection frames are detected, and delete the human detection frames corresponding to the facial feature information whose cosine similarity is less than the first similarity threshold.

[0204] Furthermore, the image processing module 302 is also used to calculate the center distance between the center point of the image to be detected and the center point of each human detection frame if multiple human detection frames are detected, and delete other human detection frames except the target human detection frame among the multiple human detection frames. The target human detection frame is the human detection frame with the largest center distance.

[0205] Furthermore, the learning state monitoring device 300 further includes: a state abnormality detection module (not shown in the figure) and a third prompt module (not shown in the figure);

[0206] The state abnormality detection module is used to determine a second warning period based on the current sampling time and the learning state if the student's learning state at the current sampling time is an abnormal learning state, and the second warning period includes at least one sampling time before the current sampling time; and compare the learning state at the current sampling time with the learning state identified in the second warning period;

[0207] The third prompt module is used to output abnormal state prompt information if the learning state at the current sampling moment is the same as the learning state identified in the second warning period.

[0208] For the specific definition of the learning state monitoring device, please refer to the definition of the learning state monitoring method above, which will not be repeated here. Each module in the above-mentioned learning state monitoring device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0209] Based on the above Figure 2 and Figure 3 The method shown in the embodiment of the present application accordingly provides a readable storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned Figure 2 and Figure 3 The learning status monitoring method is shown.

[0210] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0211] Based on the above Figure 2 and Figure 3 The method shown, and Figure 4 In order to achieve the above-mentioned purpose, the embodiment of the present application further provides a computer device, which can be a personal computer, a server, a network device, etc. The computer device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 2 and Figure 3 The learning status monitoring method is shown.

[0212] Optionally, the computer device may further include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a Bluetooth interface, a WI-FI interface), etc.

[0213] Those skilled in the art will appreciate that the computer device structure provided in this embodiment does not limit the computer device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0214] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages and saves the hardware and software resources of the computer device, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to realize communication between the components inside the storage medium, and communication with other hardware and software in the physical device.

[0215] Through the description of the above implementation methods, the technicians in this field can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform, or by hardware to obtain the images to be tested of the students in the learning process at different sampling moments; identify the character feature performance of the students in the images to be tested at different sampling moments, and the character feature performance includes facial feature information and / or the target behavior generated by the students in the learning process; determine the learning status of the students based on the character feature performance. In the embodiment of the present application, the images to be tested of the students in the learning process at different sampling moments are processed respectively, and the character feature performance of the students in each image to be tested is identified. The facial feature information in the character feature performance obtained at different sampling moments is used to locate whether the student is on the camera, and then determine whether the student leaves the seat. Or the changes in actions occurring between different sampling moments can determine whether the student remains focused or gradually becomes lazy, distracted, etc., and then analyze a series of actual behavior performances of the students in the entire learning process, and determine whether the students have taken learning-related behaviors. In this way, the learning status of the students can be matched by the determined facial feature information and target behavior, making the evaluation of the learning status more objective and scientific. This enables accurate detection of students' different learning states, so that timely and targeted measures can be taken to intervene or encourage them, helping students learn better, improving their concentration during self-study, and reducing the burden on teachers.

[0216] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0217] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. A method for monitoring a learning state, characterized in that: The method comprises: Obtain the images to be tested of students in the learning process at different sampling moments; Identify the character feature performance of the student in the image to be tested at different sampling moments, wherein the character feature performance includes facial feature information and / or target behavior generated by the student during the learning process; The learning status of the student is determined based on the character characteristic performance.

2. The method for monitoring learning status according to claim 1, characterized in that: The character feature performance includes facial feature information, and the identifying the character feature performance of the student in the image to be tested at different sampling times includes: Identify a human body detection frame in the image to be tested; Performing face detection on the human body detection frame to obtain the face feature information; The determining the learning status of the student based on the facial feature information includes: Calculating the cosine similarity between the preset facial feature information of the student and the facial feature information obtained at the target sampling time; If the cosine similarity at the target sampling moment is less than a first similarity threshold, it is determined that the student is in the away state.

3. The method for monitoring learning status according to claim 1, characterized in that: The image to be tested includes a human body region, the character feature performance includes a target behavior generated by the student during the learning process, and the identifying the character feature performance of the student in the image to be tested at different sampling times includes: Identify a human body detection frame in the human body area; Performing posture detection on the human body detection frame to determine the human body posture in the human body detection frame; Comparing the human body postures obtained at different sampling times, and determining the amount of posture change between the different sampling times; The target behavior is determined based on the posture change amount.

4. The method for monitoring learning status according to claim 3, characterized in that: The performing posture detection on the human body detection frame to determine the human body posture in the human body detection frame includes: Performing human skeleton feature point detection on the human body detection frame to determine the skeleton feature points; Calculate the positional relationship between different bone feature points based on the coordinates of the bone feature points; Determining the human body posture based on the positional relationship; The bone feature points include at least one of the following: left eye, right eye, nose, left ear, right ear, left shoulder, right shoulder, nose tip, mouth center, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left ankle, right ankle.

5. The method for monitoring the learning status according to any one of claim 3, characterized in that: The method further comprises: If the human body posture at the current sampling moment is an abnormal posture, determine a first warning period based on the current sampling moment and the human body posture, wherein the first warning period includes at least one sampling moment before the current sampling moment; Comparing the human body posture at the current sampling moment with the human body posture identified in the first warning period; If the human body posture at the current sampling moment is the same as the human body posture identified in the first warning period, abnormal posture prompt information is output based on the human body posture.

6. The method for monitoring learning status according to claim 3, characterized in that: The determining the target behavior based on the posture change amount includes: Matching the posture change amount with the posture change range of a preset behavior, and determining the preset behavior corresponding to the posture change range to which the posture change amount belongs as the target behavior; The target behavior includes at least one of the following: dazed behavior, writing behavior, page turning behavior, reading aloud behavior, watching behavior, and leaving behavior.

7. The method for monitoring learning status according to claim 1, characterized in that: The image to be tested includes a learning tool area, the character feature performance includes a target behavior generated by the student during the learning process, and the identifying the character feature performance of the student in the image to be tested at different sampling times includes: Identify a text line detection frame in the learning tool area; Perform text detection on the text line detection frame to determine the text in the text line detection frame; Compare the text in the text line detection box obtained at different sampling times to determine the text similarity between the different sampling times; The target behavior is determined based on the text similarity.

8. The method for monitoring learning status according to claim 7, characterized in that: The comparing the text in the text line detection box obtained at different sampling times to determine the text similarity between the different sampling times includes: Compare the text similarity between the handwritten text in the first text line detection box and the handwritten text in the second text line detection box, the first text line detection box is a file line detection box obtained based on the image to be tested at the current sampling moment, and the second text line detection box is a file line detection box obtained based on the image to be tested at the previous sampling moment, and the first text line detection box and the second text line detection box have the same positioning identifier.

9. The method for monitoring learning status according to claim 7, characterized in that: The method further comprises: Extracting outline information of the text in the text line detection box; Inputting the contour information into a classification model to determine a classification result of the text, wherein the classification result includes a handwriting class and a print class, and the classification model is trained based on print sample text and handwriting sample text; Based on the position information of the text row detection box corresponding to the printed text in the learning tool area and / or the printed text, a positioning mark of the text row detection box is generated, and the positioning mark is used to indicate the row number, column number or page number of the text row detection box.

10. The method for monitoring learning status according to claim 7, characterized in that: The determining the target behavior based on the text similarity includes: If the text similarity is greater than a second similarity threshold, it is determined that the student is in a daze behavior; If the text similarity is less than or equal to a second similarity threshold, it is determined that the student has a writing behavior.

11. The method for monitoring the learning status according to any one of claims 2 to 10, characterized in that: The method further comprises: If multiple human detection frames are detected, respectively calculate the cosine similarities between the facial feature information corresponding to the multiple human detection frames and the preset facial feature information of the student, and delete the human detection frames corresponding to the facial feature information whose cosine similarity is less than a first similarity threshold; or, If multiple human detection frames are detected, the center distances between the center point of the image to be detected and the center point of each human detection frame are calculated respectively, and other human detection frames except the target human detection frame among the multiple human detection frames are deleted. The target human detection frame is the human detection frame with the largest center distance.

12. The method for monitoring the learning status according to any one of claims 1 to 10, characterized in that: The method further comprises: If no key bone feature point is detected based on the image to be tested, or the coordinates of the bone feature point are not within the specified area of ​​the bone feature point, or the coordinate difference between adjacent boundary points of the learning tool is greater than the preset difference, an adjustment prompt message is output.

13. The method for monitoring the learning status according to any one of claims 1 to 10, characterized in that: The method further comprises: If the learning state of the student at the current sampling moment is an abnormal learning state, determine a second warning period based on the current sampling moment and the learning state, wherein the second warning period includes at least one sampling moment before the current sampling moment; Comparing the learning state at the current sampling moment with the learning state identified in the second warning period; If the learning state at the current sampling moment is the same as the learning state identified in the second warning period, output abnormal state prompt information.

14. A learning status monitoring device, characterized in that: The device comprises: An acquisition module is used to acquire images to be tested of students during their learning process at different sampling moments; An image processing module, used to identify the character feature performance of the student in the image to be tested at different sampling moments, wherein the character feature performance includes facial feature information and / or target behavior generated by the student during the learning process; A state recognition module is used to determine the learning state of the student based on the character feature performance.

15. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps of the method for monitoring the learning state according to any one of claims 1 to 13 are implemented.

16. A computer device comprising a storage medium, a processor, and a computer program stored in the storage medium and executable on the processor, characterized in that: When the processor executes the program, the method for monitoring the learning state according to any one of claims 1 to 13 is implemented.

Images