A learning state detection method and device based on artificial intelligence

Abstract

This application belongs to the field of human body detection technology and discloses a learning state detection method and device based on artificial intelligence. The method includes: acquiring self-learning video data and extracting multiple image frames; inputting the image frames into a MediaPipe model to obtain the coordinates of key points of the human skeleton; sorting the coordinates of the key points of the human skeleton in each image frame in a temporal order to obtain a skeleton coordinate set; inputting the skeleton coordinate set into an LSTM model to obtain the current state; determining whether learning timing has been started; if so, incrementing the inattentive count by 1 when the current state is a preset non-focused state; stopping the learning timing when the current state is a standing state; otherwise, starting the learning timing when the current state is a learning state. This application can identify multiple states of students in self-learning scenarios and has high robustness to situations such as students being occluded.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human body detection, and in particular to a learning state detection method and device based on artificial intelligence. BACKGROUND

[0002] The autonomous learning scenario refers to a learning scenario in which a student autonomously selects and arranges learning content, methods and progress according to his / her own interests and needs in an informal education environment such as a family, a library, a self-study class, the Internet, etc. Compared with a classroom scenario, the autonomous learning scenario has the following characteristics: 1. The student's learning process behavior is more diversified and complex, and there may be phenomena such as looking around and not paying attention. 2. The student's learning environment is more variable and uncontrollable, such as changes in light, the movement of irrelevant personnel, occlusion, etc.

[0003] However, the current learning state recognition technology is more designed for the classroom scenario and cannot fully adapt to the changing characteristics of the autonomous learning scenario, resulting in low learning state recognition accuracy in the autonomous learning scenario. SUMMARY

[0004] The present application provides a learning state detection method and device based on artificial intelligence, which can recognize multiple states of students in an autonomous learning scenario and has high robustness to situations such as occlusion of students.

[0005] In a first aspect, the embodiments of the present application provide a learning state detection method based on artificial intelligence, comprising:

[0006] obtaining autonomous learning video data and extracting a plurality of image frames;

[0007] inputting the image frames into a MediaPipe model to obtain human skeleton key point coordinates;

[0008] sequentially sorting the human skeleton key point coordinates of each image frame to obtain a skeleton coordinate set;

[0009] inputting the skeleton coordinate set into an LSTM model to obtain a current state;

[0010] determining whether learning timing has been started;

[0011] if yes, increasing a number of inattentive times by 1 when the current state is a preset inattentive state, and stopping the learning timing when the current state is a standing state; and if no, starting the learning timing when the current state is a learning state.

[0012] Further, the method further comprises:

[0013] if the learning timing has been started, inputting the autonomous learning video data into a Yolo model to obtain a learning type;

[0014] determining whether the type label of the learning timing is a learning type;

[0015] If not, resetting the learning timing and generating a corresponding type label according to the learning type.

[0016] Further, the method further comprises:

[0017] After stopping the learning timing or resetting the learning timing, uploading the obtained learning duration, corresponding type label, number of inattentive times and collection timestamp of the autonomous learning video data to a cloud server.

[0018] Further, the method further comprises:

[0019] After obtaining the plurality of image frames, converting the image frames into grayscale images;

[0020] According to the preset pixel size, dividing the grayscale images into a plurality of sub-regions;

[0021] Calculating adaptive parameters of each sub-region; the adaptive parameters include mean and variance;

[0022] Determining an adaptive threshold value of the corresponding sub-region according to the adaptive parameters;

[0023] According to the adaptive threshold value, binarizing the sub-region to obtain a binary image;

[0024] Inputting the binary image into a MediaPipe model to obtain human skeleton key point coordinates.

[0025] Further, the above calculation of the adaptive parameters of each sub-region comprises:

[0026] Adding the grayscale values of each pixel in the sub-region to obtain a total grayscale sum;

[0027] Dividing the total grayscale sum by the number of pixels in the sub-region to obtain the mean of the sub-region;

[0028] Calculating the difference between each grayscale value and the mean in the sub-region;

[0029] After squaring and adding each difference value, a total difference sum is obtained;

[0030] Dividing the total difference sum by the number of pixels in the sub-region to obtain the variance of the sub-region.

[0031] Further, the above determination of the adaptive threshold value of the corresponding sub-region according to the adaptive parameters comprises:

[0032] Determining whether the interval between the image frame and the image frame of the last update of the light sensitivity is greater than or equal to a preset frame number;

[0033] If yes, then determine the light sensitivity according to the variance and mean value of the image frame;

[0034] Multiply the variance and the light sensitivity and add the mean value to obtain the adaptive threshold.

[0035] Further, the method further comprises:

[0036] Before inputting the MediaPipe model, the binary image is morphologically processed.

[0037] Further, the method further comprises:

[0038] According to the preset pixel size, the image frame is divided into a plurality of sub-regions;

[0039] Feature recognition is performed on the sub-regions to obtain facial feature of the sub-regions;

[0040] According to the preset key feature and the facial feature, each sub-region is classified to obtain important regions and ordinary regions;

[0041] Each ordinary region is down-sampled by bilinear interpolation or wavelet transform until a preset resolution; or, each ordinary region is sparsely represented to obtain a quantized image;

[0042] The quantized image is input into the MediaPipe model to obtain human skeleton key point coordinates.

[0043] Further, the above sparsely representing each ordinary region comprises:

[0044] A sparse sampling grid of a preset quantization size is generated for the ordinary region;

[0045] A pixel in the sparse sampling grid is randomly selected as a reserved pixel; pixel values of each pixel in the sparse sampling grid except the reserved pixel are filled with zero values or a mean value of the ordinary region.

[0046] Further, the method further comprises:

[0047] Audio data is obtained and converted into text data;

[0048] Feature extraction and recognition are performed on the text data to obtain learning correlation;

[0049] Through an eye tracker, a learner's gaze concentration area, blink frequency and pupil diameter change curve are obtained;

[0050] The proportion of blank areas in the gaze concentration area is calculated;

[0051] The distraction probability is obtained according to the learning correlation, the blank area proportion, the blink frequency and the pupil diameter change curve; it is judged whether the distraction probability is greater than a preset reminding threshold; if yes, an attention reminding voice signal is generated.

[0052] Further, the distraction probability obtained according to the learning correlation, the blank area proportion, the blink frequency and the pupil diameter change curve includes:

[0053] It is judged whether the blank area proportion or the learning correlation is greater than a corresponding preset threshold;

[0054] If yes, an attention reminding voice signal is directly generated; if no, the blink frequency and the pupil diameter change curve are input into a distraction prediction model to obtain a distraction probability.

[0055] Further, the method further includes:

[0056] A set of training images in which the learner exists occlusion is obtained;

[0057] The hyperparameters of the LSTM model are determined based on a Hyperband algorithm;

[0058] The LSTM model is trained by using the set of training images.

[0059] In a second aspect, an embodiment of the present application provides a learning state detection device based on artificial intelligence, including:

[0060] An acquisition module is configured to acquire autonomous learning video data and extract a plurality of image frames;

[0061] A coordinate extraction module is configured to input the image frames into a MediaPipe model to obtain human skeleton key point coordinates;

[0062] A timing module is configured to time sequence sort the human skeleton key point coordinates of each image frame to obtain a skeleton coordinate set;

[0063] A state recognition module is configured to input the skeleton coordinate set into an LSTM model to obtain a current state;

[0064] A timing module is configured to judge whether learning timing has been started; if yes, when the current state is a preset non-concentrated state, the number of non-concentrated times is increased by 1; when the current state is a standing state, the learning timing is stopped; if no, when the current state is a learning state, the learning timing is started.

[0065] Further, the timing module is further configured to input the autonomous learning video data into a Yolo model to obtain a learning type after judging that the learning timing has been started; judge whether a type label of the learning timing is the learning type; if no, reset the learning timing and generate a corresponding type label according to the learning type.

[0066] Further, the timing module is further configured to upload the obtained learning duration, corresponding type label, inattention times and collection timestamp of the autonomous learning video data to a cloud server after stopping the learning timing or resetting the learning timing.

[0067] Further, the device further comprises:

[0068] The first image processing module is configured to convert the image frames into grayscale images after obtaining the plurality of image frames; divide the grayscale images into a plurality of sub-regions according to a preset pixel size; calculate adaptive parameters of each sub-region; the adaptive parameters include mean and variance; determine an adaptive threshold of the corresponding sub-region according to the adaptive parameters; binarize the sub-region according to the adaptive threshold to obtain a binary image; and input the binary image into a MediaPipe model to obtain human skeleton key point coordinates.

[0069] Further, the device further comprises:

[0070] The second image processing module is configured to divide the image frames into a plurality of sub-regions according to a preset pixel size; perform feature recognition on the sub-regions to obtain facial features of the sub-regions; classify each sub-region according to a preset key feature and the facial features to obtain important regions and ordinary regions; down-sample each ordinary region by bilinear interpolation or wavelet transform until a preset resolution; or perform sparse representation on each ordinary region to obtain a quantized image; and input the quantized image into a MediaPipe model to obtain human skeleton key point coordinates.

[0071] Further, the device further comprises:

[0072] The distraction judgment module is configured to obtain audio data and convert the audio data into text data; perform feature extraction and recognition on the text data to obtain a learning relevance; obtain a learner's gaze concentration area, blink frequency and pupil diameter change curve by an eye tracker; calculate a blank area proportion in the gaze concentration area; obtain a distraction probability according to the learning relevance, the blank area proportion, the blink frequency and the pupil diameter change curve; judge whether the distraction probability is greater than a preset reminding threshold; and if so, generate an attention reminding voice signal.

[0073] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to perform the steps of the learning state detection method based on artificial intelligence according to any of the above embodiments.

[0074] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps of the learning state detection method based on artificial intelligence according to any of the above embodiments.

[0075] Compared with the prior art, the technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0076] The learning state detection method based on artificial intelligence provided by the embodiments of the present application can obtain the current state of a student standing, learning or not paying attention by recognizing the human skeleton key point coordinate set output by the MediaPipe model through the long short-term memory neural network (LSTM model). Compared with the CNN (convolutional neural network), the LSTM model can learn the long-term dependence in the action skeleton key point sequence, extract the time sequence features of the skeleton key point coordinates, and effectively recognize different behavior states. Moreover, the long short-term memory neural network is robust to possible key point occlusion. By memorizing the positions of the key points in the previous image frames that are not occluded, the time dependence of the key points is captured, so that even if some skeleton key points are occluded, the possible posture classification can be inferred through the previous and existing key points, thereby improving the robustness of the incomplete key point detection. BRIEF DESCRIPTION OF DRAWINGS

[0077] Figure 1 A flowchart of a learning state detection method based on artificial intelligence is provided for an exemplary embodiment of the present application.

[0078] Figure 2 An architecture diagram of an LSTM model is provided for an exemplary embodiment of the present application.

[0079] Figure 3 A flowchart of an image frame preprocessing step is provided for an exemplary embodiment of the present application.

[0080] Figure 4 A flowchart of an image frame preprocessing step is provided for another exemplary embodiment of the present application.

[0081] Figure 5 A structure diagram of a learning state detection device based on artificial intelligence is provided for an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0082] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0083] All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative labor fall within the scope of the present application.

[0084] Please refer to Figure 1 The embodiments of the present application provide a learning state detection method based on artificial intelligence, comprising:

[0085] Step S11, acquiring autonomous learning video data and extracting a plurality of image frames.

[0086] Among them, the plurality of image frames are continuous, pre-set number of image frames in the autonomous learning video data.

[0087] Specifically, the autonomous learning video data can be acquired by a video acquisition module, such as a camera.

[0088] Step S12, inputting the image frame into the MediaPipe model to obtain the human skeleton key point coordinates.

[0089] Further, the personnel in the autonomous learning scene are usually uncontrollable, when there are multiple people in the video frame, the skeleton key points of irrelevant personnel may be extracted and recognized by mistake, resulting in the failure of the skeleton key point recognition function. Therefore, a human body detection preprocessing algorithm can be added before step S12 to first detect the bounding box of each human body, then independently recognize the posture in each box and track, so that the detection of multiple learning or the exclusion of irrelevant personnel interference can be realized.

[0090] Step S13, time sequence sorting the human skeleton key point coordinates of each image frame to obtain a skeleton coordinate set.

[0091] It should be noted that the time sequence mentioned here is the time sequence of each image frame in the video data.

[0092] Step S14, inputting the skeleton coordinate set into the LSTM model to obtain the current state.

[0093] The LSTM model, i.e. long short-term memory neural network model, is obtained based on training samples, and the training samples include video streams marked with learning state categories. The LSTM neural network model is as shown in Figure 2 It includes 3 layers of LSTM (long short-term memory neural network) and 3 layers of fully connected layer, and uses memory unit and gate control algorithm to solve the long time dependence problem.

[0094] In a possible implementation, the long short-term memory neural network model can be trained according to the following steps: obtaining different autonomous learning state video samples, and performing category labeling on the video samples; inputting the video samples into a MediaPipe human body posture estimation model, and converting the video samples into normalized human body key point coordinate sequence samples; and training the long short-term memory neural network according to the human body key point coordinate sequence samples, so as to identify input data and output three current states of standing, learning and looking around.

[0095] Compared with a CNN (convolutional neural network) used in a traditional technology, the LSTM can learn long-term dependencies in a motion skeleton key point sequence, extract time sequence features of the skeleton key point data, and thus effectively identify different learning states.

[0096] In the implementation process, a total of 335 skeleton key point coordinate sequence samples are used for training, an MX250 GPU is used for training, the training is performed for 30 rounds, the total training time is about 3 minutes, and the recognition accuracy of the student autonomous learning state in a real scene can reach 77.50%. After testing, if a small sample is trained for a certain student, that is, the training set is a real scene, the recognition accuracy can reach 90%. In the case of small sample training (30 key point sequence samples for each of the three categories), by setting a high Dropout probability of the second layer and the third layer LSTM, about 0.23 and 0.84, respectively, the overfitting problem that often occurs in small sample training can be avoided. It can be seen that for the LSTM, the training cost and the number of training times are in a low range.

[0097] Step S15, determining whether the learning timing has been started.

[0098] Step S16, if yes, when the current state is a preset non-concentrated state, increasing the number of non-concentrated times by 1; when the current state is a standing state, stopping the learning timing; and if no, when the current state is a learning state, starting the learning timing.

[0099] The preset non-concentrated state is a looking around state.

[0100] Specifically, if the learning timing has been started, it indicates that the current state of the user detected last time is a learning state; if the preset non-concentrated state is detected this time, it is considered that the user is in a non-concentrated situation in the learning process, and the number of non-concentrated times is increased by 1; and if the standing state is detected this time, it indicates that the user wants to end the learning state, and thus the learning timing is stopped.

[0101] After the number of non-concentrated times is increased by 1, the learning timing is stopped, the learning timing is started, or the current state detected in the state of starting the learning timing is still the learning state, returning to step S11 for next detection.

[0102] The learning state detection method based on artificial intelligence provided by the above embodiment can obtain the current state of the student standing, learning or not paying attention by recognizing the set of human skeleton key point coordinates output by the MediaPipe model through the long short-term memory neural network (LSTM model).

[0103] In some embodiments, the method further comprises:

[0104] In step S21, if the learning timing has been started, the autonomous learning video data is input into the Yolo model to obtain the learning type.

[0105] In step S22, it is judged whether the type label of the learning timing is the learning type.

[0106] In step S23, if not, the learning timing is reset, and the corresponding type label is generated according to the learning type.

[0107] The learning type can include reading and writing.

[0108] Specifically, the learning timing is reset, that is, the learning timing of the original type label is stopped, and the learning timing of the current learning type is started. For example, if the learning type is detected as "writing", the writing timing is started, and if the last detected learning type is "reading", that is, the type label is "reading", the reading timing is stopped. If the learning type is detected as "reading", the reading timing is started, and if the last detected learning type is "writing", the writing timing is stopped.

[0109] The timing of different type labels can be inherited, and all will be cleared unless the standing state is detected.

[0110] In some embodiments, the method further comprises:

[0111] After stopping the learning timing or resetting the learning timing, the obtained learning duration, corresponding type label, number of times of not paying attention and collection time stamp of the autonomous learning video data are uploaded to the cloud server.

[0112] Specifically, the cloud server will evaluate the learning quality of the student according to the learning duration, and the information of different types of duration, the number of inattentive times and the time period of self-study of the student, and the time period of efficient learning, or generate learning type distribution suggestion feedback to the self-study student, so that the self-study student can adjust the learning habit and improve the learning efficiency.

[0113] See Figure 3 In some embodiments, the method further comprises:

[0114] Step S31, after obtaining the plurality of image frames, converting the image frames into gray-scale images.

[0115] Specifically, the gray-scale information calculation formula of each pixel point is:

[0116] Gray = 0.299 * R + 0.587 * G + 0.114 * B

[0117] Wherein, R, G and B are the pixel values of the pixel point in three channels.

[0118] Step S32, dividing the gray-scale images into a plurality of sub-regions according to a preset pixel size.

[0119] Wherein, the preset pixel size can be 16*16 or 32*32 pixels, etc.

[0120] Step S33, calculating the adaptive parameters of each sub-region; the adaptive parameters include mean and variance.

[0121] Specifically, the above calculation of the adaptive parameters of each sub-region includes:

[0122] Step S331, adding the gray-scale values of each pixel in the sub-region to obtain the total gray-scale sum.

[0123] Step S332, dividing the total gray-scale sum by the number of pixels in the sub-region to obtain the mean of the sub-region.

[0124] Specifically, the mean of the i-th sub-region is μ i The calculation process is:

[0125]

[0126] Wherein, j is the j-th pixel point in the i-th sub-region, and n is the number of pixels in the i-th sub-region.

[0127] The mean of the sub-region represents the average level of the image gray-scale in the sub-region.

[0128] Step S333, calculating the difference between each gray-scale value and the mean in the sub-region.

[0129] Step S334, square each difference value and add them together to obtain a total difference value.

[0130] Step S335, divide the total difference value by the number of pixels in the sub-region to obtain the variance of the sub-region.

[0131] Specifically, the variance of the sub-region i is The calculation formula is:

[0132]

[0133] The variance reflects the dispersion of the pixel gray values in the sub-region relative to the mean value. The greater the variance, the more dispersed the distribution of the gray values in the sub-region, and the greater the possible change in light.

[0134] Step S34, determine the adaptive threshold value of the corresponding sub-region according to the adaptive parameter.

[0135] Specifically, the above-mentioned determination of the adaptive threshold value of the corresponding sub-region according to the adaptive parameter comprises:

[0136] Step S341, determine whether the interval between the image frame and the image frame in which the light sensitivity was last updated is greater than or equal to a preset frame number.

[0137] In the autonomous learning scenario, in addition to the turning on and off of the electronic light source, the light intensity does not change in most cases. Therefore, the application updates the light sensitivity every preset frame number to avoid an increase in computing power consumption.

[0138] Step S342, if yes, determine the light sensitivity according to the variance and the mean value of the image frame.

[0139] Specifically, the variance and the mean value are evaluated according to the segmentation rule, and are divided into low, medium, and high degrees, respectively.

[0140] After determining the corresponding range of the variance and the mean value, the light sensitivity is determined according to the following table:

[0141] Mean (μ) range Variance (σ 2 ) range Light sensitivity range Lighting conditions Low Medium, high 0.8-1.2 Uniform low light (library) Medium, high Low, medium, high 0.3-0.7 Desk lamp direct (desk) Medium High 0.5-1.0 Natural light shade

[0142] That is, in a weak light environment (low mean value and high variance), a larger light sensitivity is needed to avoid shadow noise; in a strong light environment (high mean value and low variance), a smaller light sensitivity is needed to avoid loss of details due to a too high adaptive threshold value.

[0143] After determining the value range of k, a value is randomly selected from the range as the light sensitivity.

[0144] Step S343, multiply the variance and the light sensitivity and add the mean value to obtain the adaptive threshold value.

[0145] Specifically, the calculation method of the adaptive threshold T is:

[0146]

[0147] Step S35, binarizing the sub-regions according to the adaptive threshold to obtain a binary image.

[0148] Specifically, according to the calculated adaptive threshold, the pixels in each sub-region are binarized. The pixels with a gray value greater than the adaptive threshold are set to 255 (white), representing the foreground; the pixels with a gray value less than the adaptive threshold are set to 0 (black), representing the background, to achieve binarization.

[0149] Step S36, inputting the binary image into the MediaPipe model to obtain the human skeleton key point coordinates.

[0150] It can be understood that compared with the classroom learning scene, the light in the autonomous learning scene may be unevenly distributed, such as brighter light near the window and darker light near the corner. Different light intensity and angle will cause shadows or highlights on the student's face, interfere with the extraction of facial features by the algorithm, and lead to inaccurate recognition of key point coordinates. In addition, the long autonomous learning process may also bring dynamic changes in light, such as light flickering or someone entering and exiting causing light obstruction, which also increases the difficulty of recognition. Through the above steps of histogram equalization and adaptive threshold adjustment, the influence of light changes on the quality of the student's face image can be effectively reduced, providing a better image basis for subsequent recognition of the human skeleton key point coordinate set.

[0151] In some embodiments, the method further comprises:

[0152] Before inputting the MediaPipe model, the binary image is subjected to morphological processing.

[0153] Specifically, the binarized image is post-processed to remove possible noise points or small isolated regions. Common post-processing methods include morphological operations such as erosion and dilation. The erosion operation can remove small noise points in the image, and the dilation operation can fill in some small holes or connect disconnected regions.

[0154] Please refer to Figure 4 In another possible embodiment, the method further comprises:

[0155] Step S41, dividing the image frame into multiple sub-regions according to a preset pixel size.

[0156] The preset pixel size can be 16x16 or 32x32 pixels, etc.

[0157] Step S42, feature recognition is performed on the sub-regions to obtain facial features of the sub-regions.

[0158] Specifically, the feature recognition is to recognize whether there is a part of a human face in the sub-region, and the facial features include arms, fingers, shoulders, eyes, noses, mouths, cheeks, ears, foreheads, chins, etc.

[0159] Step S43, each sub-region is classified according to the preset key features and the facial features to obtain important regions and ordinary regions. The preset key features can be eyes, mouths, and cheeks, that is, the sub-regions including the eyes, mouths, and cheek regions are marked as important regions, and the other sub-regions are marked as ordinary regions.

[0160] Step S44, each ordinary region is down-sampled by using a bilinear interpolation or a wavelet transform until a preset resolution, or each ordinary region is sparsely represented to obtain a quantized image.

[0161] It can be understood that the MediaPipe model in the prior art generally uses a target detection algorithm based on a convolutional neural network (CNN), which has relatively high requirements for CPU and memory. For example, for a simple application scenario, detection and key point positioning on a low-resolution video or image requires at least an Intel Core i5 or AMD Ryzen 5 series CPU. When processing high-resolution videos or real-time processing of multiple targets, a more powerful CPU, such as a multi-core server-level CPU, is required to provide sufficient computing resources to run the model. For simple applications, 8 GB of system memory can be sufficient, but when processing high-resolution videos or simultaneously processing multiple tasks, 16 GB or more of memory is required.

[0162] Therefore, the present application proposes that the ordinary regions lacking key information can be processed to reduce the amount of data input to the model.

[0163] Specifically, the important regions can maintain the original resolution, and the ordinary regions are down-sampled to a low resolution (such as 56x56); the use of a bilinear interpolation or a wavelet transform algorithm can reduce information loss during the down-sampling process.

[0164] The reduction in resolution can reduce the pixel data amount by 50%-70% while retaining the details of the key regions.

[0165] The above sparse representation of each ordinary region can include:

[0166] Step S441, a sparse sampling grid of a preset quantization size is generated for the ordinary region.

[0167] Step S442, randomly select a pixel in the sparse sampling grid as a reserved pixel; fill the pixel values of each pixel in the sparse sampling grid except the reserved pixel with zero values or the mean value of the normal region.

[0168] For example, the preset quantization size is 2x2, and the sparse representation is to reserve 1 out of every 4 pixels in the normal region, and the remaining positions are filled with zero values or the mean value. At the same time, the normal region is stored in a sparse matrix format (such as CSR) to reduce memory occupation.

[0169] After sparse representation, the data amount of the normal region can be reduced by 80%, and the overall input data amount can be reduced by 40%-60%.

[0170] Step S45, input the quantized image into the MediaPipe model to obtain the human skeleton key point coordinates.

[0171] The above embodiments can significantly reduce the data amount and calculation overhead of inputting the MediaPipe model and reduce memory occupation while maintaining the detection accuracy of the core region (face, facial features).

[0172] In some embodiments, the method further comprises:

[0173] Step S51, acquire audio data and convert the audio data into text data.

[0174] Step S52, perform feature extraction and recognition on the text data to obtain a learning relevance.

[0175] Step S53, acquire the learner's gaze concentration area, blinking frequency, and pupil diameter change curve through an eye tracker.

[0176] Step S54, calculate the blank area proportion in the gaze concentration area.

[0177] The "blank area" refers to the part of the area that has no text or pictures.

[0178] Step S55, obtain a distraction probability according to the learning relevance, the blank area proportion, the blinking frequency, and the pupil diameter change curve; determine whether the distraction probability is greater than a preset reminding threshold; if yes, generate an attention reminding voice signal.

[0179] Specifically, first determine whether the blank area proportion or the learning relevance is greater than the corresponding preset threshold; if yes, directly generate an attention reminding voice signal; that is, when it is detected that the learner's gaze focus point does not exist learning content, or it is detected that the learner is chatting about irrelevant content to the learning content, directly remind as a possible distraction. If not, input the blinking frequency and the pupil diameter change curve into a distraction prediction model to obtain a distraction probability.

[0180] It is worth noting that since distraction refers to the process of diverting an individual's attention from the educational task or content, thereby hindering the reception of the required information, it is a long process and does not occur instantaneously, so the above-mentioned embodiments are not real-time detection, but audio data, eye tracker analysis results in a period of time are collected for judgment.

[0181] Further, the above-mentioned embodiments are only executed in the learning timing process, if the learning timing stops, the distraction reminding is not performed.

[0182] Further, the attention reminding voice signal can also be uploaded to the cloud server with learning time and other information for learning quality or learning efficiency analysis, so as to optimize the learning situation for the learner or the guardian of the learner.

[0183] In some embodiments, the method further comprises:

[0184] Step S61, obtaining a set of training images in which the learner is blocked.

[0185] Step S62, determining the hyperparameters of the LSTM model based on the Hyperband algorithm.

[0186] Step S63, training the LSTM model using the set of training images.

[0187] Although the long short-term memory neural network (LSTM) adopted in the present application is robust to possible key point occlusion, the algorithm can remember the positions of key points that are not occluded in previous video frames, capture the time dependence of key points, and even if there is partial skeleton key point occlusion, it can also infer the possible pose classification through previous and existing key points, but it is difficult to cope with long-term fixed object occlusion (such as desk ornaments, pen containers, table lamps, etc.), therefore, adding occluded frame image samples in the training image set can improve the robustness of the LSTM model to incomplete key point detection.

[0188] At the same time, when the samples of the training image set change, the optimal hyperparameters of the LSTM model training may change, therefore, the present application further combines the Hyperband algorithm for optimizing hyperparameters, and obtains the optimal hyperparameters adapted to the training set through multiple training, so as to ensure the recognition accuracy of the model.

[0189] Please refer to Figure 5 Another embodiment of the present application provides a learning state detection device based on artificial intelligence, comprising:

[0190] The acquisition module 101 is configured to acquire autonomous learning video data and extract a plurality of image frames.

[0191] The coordinate extraction module 102 is configured to input the image frame into the MediaPipe model to obtain human skeleton key point coordinates.

[0192] The timing module 103 is configured to time sequence sort the human skeleton key point coordinates of each image frame to obtain a skeleton coordinate set.

[0193] The state recognition module 104 is configured to input the skeleton coordinate set into the LSTM model to obtain a current state.

[0194] The timing module 105 is configured to determine whether the learning timing has been started; if yes, when the current state is a preset non-concentrated state, increase the number of non-concentrated times by 1; when the current state is a standing state, stop the learning timing; if no, when the current state is a learning state, start the learning timing.

[0195] Further, the timing module 105 is further configured to, after determining that the learning timing has been started, input the autonomous learning video data into the Yolo model to obtain a learning type; determine whether the type label of the learning timing is the learning type; if no, reset the learning timing, and generate a corresponding type label according to the learning type.

[0196] Further, the timing module 105 is further configured to, after stopping the learning timing or resetting the learning timing, upload the obtained learning duration, the corresponding type label, the number of non-concentrated times and the collection time stamp of the autonomous learning video data to a cloud server.

[0197] Further, the device further comprises:

[0198] The first image processing module is configured to, after obtaining the plurality of image frames, convert the image frame into a grayscale image; divide the grayscale image into a plurality of sub-regions according to a preset pixel size; calculate adaptive parameters of each sub-region; the adaptive parameters include a mean value and a variance; determine an adaptive threshold of the corresponding sub-region according to the adaptive parameters; binarize the sub-region according to the adaptive threshold to obtain a binary image; and input the binary image into the MediaPipe model to obtain human skeleton key point coordinates.

[0199] Further, the device further comprises a second image processing module configured to divide the image frame into a plurality of sub-regions according to a preset pixel size; perform feature recognition on the sub-region to obtain facial feature of the sub-region; classify each sub-region according to a preset key feature and the facial feature to obtain important regions and ordinary regions; down-sample each ordinary region by bilinear interpolation or wavelet transform until a preset resolution; or, perform sparse representation on each ordinary region to obtain a quantized image; and input the quantized image into the MediaPipe model to obtain human skeleton key point coordinates.

[0200] Further, the device further comprises a distraction judgment module, configured to acquire audio data and convert the audio data into text data; perform feature extraction and recognition on the text data to obtain a learning correlation degree; acquire a learner's gaze concentration area, blink frequency and pupil diameter change curve through an eye tracker; calculate a blank area proportion in the gaze concentration area; obtain a distraction probability according to the learning correlation degree, the blank area proportion, the blink frequency and the pupil diameter change curve; judge whether the distraction probability is greater than a preset reminding threshold; and if yes, generate an attention reminding voice signal.

[0201] The specific limitation of the learning state detection device based on artificial intelligence provided in the embodiment can be referred to the embodiment of the learning state detection method based on artificial intelligence, which will not be repeated here. Each module in the learning state detection device based on artificial intelligence can be realized by software, hardware and combinations thereof in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0202] The computer device provided in the embodiment of the present application can comprise a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is configured to communicate with an external terminal through a network connection. The computer program is executed by the processor, so that the processor performs the steps of the learning state detection method based on artificial intelligence of any of the above embodiments.

[0203] The working process, working details and technical effects of the computer device provided in the embodiment can be referred to the embodiment of the learning state detection method based on artificial intelligence, which will not be repeated here.

[0204] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the steps of the learning state detection method based on artificial intelligence according to any one of the above embodiments. The computer readable storage medium refers to a carrier for storing data, which can include, but is not limited to, floppy disks, optical disks, hard disks, flash memories, USB flash disks and / or memory sticks, etc. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The working process, working details and technical effects of the computer readable storage medium provided by the embodiment can be referred to the above embodiments of the learning state detection method based on artificial intelligence, and will not be described here.

[0205] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiments can be included. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM) and memory bus dynamic RAM (RDRAM).

[0206] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, it should be considered that they are within the scope of the present application.

[0207] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific and detailed manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these are all within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A learning state detection method based on artificial intelligence, characterized by, The method comprises the following steps: acquiring autonomous learning video data and extracting a plurality of image frames; after obtaining the plurality of image frames, converting the image frames into grayscale images; dividing the grayscale images into a plurality of sub-regions according to a preset pixel size; calculating adaptive parameters of each sub-region; the adaptive parameters include mean and variance; specifically, adding the grayscale values of each pixel in the sub-region to obtain a total grayscale sum; dividing the total grayscale sum by the number of pixels in the sub-region to obtain the mean of the sub-region; calculating the difference between each grayscale value in the sub-region and the mean; after squaring each difference and adding them together, a total difference sum is obtained; dividing the total difference sum by the number of pixels in the sub-region to obtain the variance of the sub-region; determining an adaptive threshold corresponding to the sub-region according to the adaptive parameters; binarizing the sub-region according to the adaptive threshold to obtain a binary image; inputting the binary image into a MediaPipe model to obtain human skeleton key point coordinates; sequentially sorting the human skeleton key point coordinates of each image frame to obtain a skeleton coordinate set; inputting the skeleton coordinate set into an LSTM model to obtain a current state; determining whether the learning timing has been started; if yes, when the current state is a preset non-concentrated state, increasing the number of non-concentrated times by 1; when the current state is a standing state, stopping the learning timing; if no, when the current state is a learning state, starting the learning timing. 2.The artificial intelligence-based learning state detection method of claim 1, wherein, Further comprising: if the learning timing has been started, inputting the autonomous learning video data into a Yolo model to obtain a learning type; determining whether the type label of the learning timing is the learning type; if no, resetting the learning timing and generating a corresponding type label according to the learning type. 3.The learning state detection method based on artificial intelligence according to claim 2, characterized in that, Further comprising: after stopping the learning timing or resetting the learning timing, uploading the obtained learning duration, the corresponding type label, the number of non-concentrated times, and the collection timestamp of the autonomous learning video data to a cloud server. 4.The artificial intelligence-based learning state detection method of claim 1, wherein, The adaptive threshold corresponding to the sub-region is determined according to the adaptive parameters, which comprises: determining whether the interval between the image frame and the image frame of the last update of light sensitivity is greater than or equal to a preset frame number; if yes, determining the light sensitivity according to the variance and the mean of the image frame; multiplying the variance and the light sensitivity and adding the mean to obtain the adaptive threshold. 5.The artificial intelligence-based learning state detection method of claim 1, wherein, Further comprising: dividing the image frame into a plurality of sub-regions according to a preset pixel size; performing feature recognition on the sub-regions to obtain facial feature of the sub-regions; classifying each sub-region according to a preset key feature and the facial feature to obtain important regions and ordinary regions; performing downsampling on each ordinary region by bilinear interpolation or wavelet transform until a preset resolution is reached; or, performing sparse representation on each ordinary region to obtain a quantized image; inputting the quantized image into the MediaPipe model to obtain the human skeleton key point coordinates. 6.The learning state detection method based on artificial intelligence according to claim 5, characterized in that, The sparse representation of each ordinary region comprises: generating a sparse sampling grid of a preset quantization size for the common area; randomly selecting a pixel in the sparse sampling grid as a reserved pixel; filling the pixel values of each pixel in the sparse sampling grid except the reserved pixel as a zero value or the average value of the common area. 7.The artificial intelligence-based learning state detection method of claim 1, wherein, Further comprising: obtaining audio data and converting the audio data into text data; performing feature extraction and recognition on the text data to obtain a learning relevance; obtaining the learner's visual concentration area, blink frequency and pupil diameter change curve through an eye tracker; calculating the blank area proportion in the visual concentration area; obtaining a distraction probability according to the learning relevance, the blank area proportion, the blink frequency and the pupil diameter change curve; judging whether the distraction probability is greater than a preset reminding threshold; if yes, generating an attention reminding voice signal.

8. An artificial intelligence-based learning state detection device characterized by comprising: Comprise: an acquisition module for acquiring autonomous learning video data and extracting a plurality of image frames; a first image processing module for converting the image frames into grayscale images after obtaining a plurality of image frames; dividing the grayscale images into a plurality of sub-regions according to a preset pixel size; calculating adaptive parameters of each sub-region; the adaptive parameters include mean and variance; Specifically, the sum of the gray values of each pixel in the sub-region is obtained to obtain the total gray value; the total gray value is divided by the number of pixels in the sub-region to obtain the mean value of the sub-region; the difference between each gray value and the mean value in the sub-region is calculated; after each difference is squared and added, the total difference is obtained; the total difference is divided by the number of pixels in the sub-region to obtain the variance of the sub-region; the adaptive threshold corresponding to the sub-region is determined according to the adaptive parameters; the sub-region is binarized according to the adaptive threshold to obtain a binary image; the binary image is input into a MediaPipe model to obtain human skeleton key point coordinates; a timing module for sequentially sorting the human skeleton key point coordinates of each image frame to obtain a skeleton coordinate set; a state recognition module for inputting the skeleton coordinate set into an LSTM model to obtain a current state; a timing module for judging whether the learning timing has been started; if yes, when the current state is a preset non-concentration state, the number of non-concentration times is increased by 1; when the current state is a standing state, the learning timing is stopped; if not, when the current state is a learning state, the learning timing is started.

Images