Intelligent video teaching method and system for realizing action recognition

Abstract

The application provides an intelligent video teaching method and system for action recognition, comprising: obtaining historical learning information of a learner; obtaining and playing a learning video based on the historical learning information of the learner; wherein, in the process of playing the learning video, the position of a body joint of the learner is obtained in real time, the body action of the learner is judged, the playing progress of the learning video is controlled based on the body action of the learner; meanwhile, a face color image of the learner is obtained, the face key points in the face color image are identified based on the face color image of the learner, the mean value of left and right eye closure and the mouth opening angle are calculated based on the face key points, the head rotation angle is calculated based on the face color image, and the video playing content is updated based on the mean value of left and right eye closure, the mouth opening angle and the head rotation angle. The body control of the playing progress and the autonomous update of the playing content in online education are realized.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent video teaching, and particularly relates to an intelligent video teaching method and system for realizing action recognition. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] In the era of knowledge economy, the demand for people to quickly learn and master new knowledge is increasing day by day. How to use advanced information technology to build a learning system suitable for the needs of different groups of people and form a learning society in which everyone can learn, everywhere can learn and at any time can learn is a major issue currently facing.

[0004] The learning form based on computer technology is one of the main forms currently recommended for use. The combination of serious learning activities and computer interactive learning makes the learning process no longer boring by using the idea of learning through play. The traditional learning method mainly focuses on watching learning content, and there is a problem that long-term sitting or long-time staring at the screen is not conducive to eyesight. Therefore, it is necessary to explore new learning modes, which not only focus on the learning effect of learners, but also pay more attention to the physical and mental health in the learning process. Researches in pedagogy and psychology show that even the minimum physical activity can support the learning process, because the tasks actively completed are more likely to be remembered than the information passively accepted, and the body movement, walking and posture interaction are also the natural interaction in real life. Therefore, the use of walking and body movement in the learning process not only enables natural interaction with virtual objects, but also helps to improve the learning efficiency.

[0005] However, due to the individual differences of different learners, the learning time differences and the limitations, the existing method is difficult to provide personalized and adaptive learning content, and also cannot realize the autonomous update of video playing content and relieve student fatigue. SUMMARY

[0006] In order to solve the technical problems existing in the background art, the present application provides an intelligent video teaching method and system for realizing action recognition, which not only realizes the autonomous update of video playing content in online education and relieves student fatigue, but also realizes the interaction with learning content by using common interaction modes such as walking and action interaction.

[0007] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0008] The first aspect of the present application provides an intelligent video teaching method for realizing action recognition, which comprises:

[0009] obtaining historical learning information of the learner;

[0010] Based on the historical learning information of the learner, a learning video is acquired and played;

[0011] In the process of playing the learning video, the position of the body joint of the learner is acquired in real time, the body action of the learner is judged, the playing progress of the learning video is controlled based on the body action of the learner; meanwhile, the color image of the face of the learner is acquired, the key points of the face in the color image of the face are identified based on the color image of the face of the learner, the mean value of the closing degree of the left and right eyes and the opening angle of the mouth are calculated based on the key points of the face, the turning angle of the head is calculated based on the color image of the face, and the video playing content is updated based on the mean value of the closing degree of the left and right eyes, the opening angle of the mouth and the turning angle of the head.

[0012] Further, the specific method for acquiring the historical learning information of the learner is as follows:

[0013] The color image of the face of the learner is acquired, and face searching is performed in the face library; if the consistent color image of the face is searched, the learner number is acquired, and the database is traversed based on the learner number to obtain the historical learning information of the learner; otherwise, the color image of the face of the learner is added in the face library, and the historical learning information of the learner in the database is initialized.

[0014] Further, the specific method for acquiring the color image of the face of the learner is as follows:

[0015] The color image and the depth image of the learner are acquired;

[0016] Based on the color image and the depth image of the learner, the three-dimensional positions of the nose joint and the neck joint of the learner are obtained;

[0017] The three-dimensional positions of the nose joint and the neck joint are mapped to the color image to obtain the two-dimensional positions of the nose joint and the neck joint in the color image;

[0018] Based on the two-dimensional positions of the nose joint and the neck joint in the color image, the side length of the face cropping square and the cropping origin point are calculated;

[0019] Based on the side length and the cropping origin point, the color image of the face is cropped from the color image.

[0020] Further, if the historical learning information of the learner is not empty, before the learning video is acquired and played, the historical learning information of the learner is used to acquire a plurality of exercise questions and sequentially push them.

[0021] Further, it further includes: after the learning video is acquired and played, based on the content of the learning video, a plurality of exercise questions are acquired and sequentially pushed.

[0022] Further, after each exercise question is pushed, the actual position of the learner in the three-dimensional space is obtained and converted into the position of the virtual image in the display device, and the answer of the learner is obtained according to the position of the virtual image in the display device.

[0023] Further, the method further comprises:

[0024] Setting an activity area parameter;

[0025] Based on the activity area parameter, the actual position of the learner in the three-dimensional space is converted into the relative position of the two-dimensional activity space, and the relative position of the two-dimensional activity space is mapped into the position of the virtual cartoon image in the display device.

[0026] The second aspect of the application provides an intelligent video teaching system for action recognition, comprising:

[0027] A historical learning information acquisition module configured to acquire historical learning information of the learner;

[0028] A learning video playing module configured to acquire and play a learning video based on the historical learning information of the learner;

[0029] During the playing of the learning video, the position of the body joint of the learner is acquired in real time, the body action of the learner is judged, the playing progress of the learning video is controlled based on the body action of the learner, the color image of the face of the learner is acquired, the key points of the face in the color image of the face are recognized based on the color image of the face of the learner, the average of the closing degree of the left and right eyes and the opening angle of the mouth are calculated based on the key points of the face, the turning angle of the head is calculated based on the color image of the face, and the playing content of the video is updated based on the average of the closing degree of the left and right eyes, the opening angle of the mouth and the turning angle of the head.

[0030] The third aspect of the application provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the intelligent video teaching method for action recognition.

[0031] The fourth aspect of the application provides a computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the intelligent video teaching method for action recognition when executing the program.

[0032] Compared with the prior art, the application has the following beneficial effects:

[0033] The application provides an intelligent video teaching method for action recognition, which can monitor the action of the learner, autonomously update the playing content and effectively relieve the fatigue of the student.

[0034] This invention provides an intelligent video teaching method that realizes action recognition. It uses common interaction methods such as walking and action interaction to interact with learning content. The interaction is simple and natural, and learners do not need additional training. It conforms to users' psychology and habits. It has a wide audience and is aimed at the general public, which can ensure learners' enthusiasm for active learning and their exercise.

[0035] This invention provides an intelligent video teaching method that realizes action recognition. It is not restricted by time and place, focuses on the individual characteristics and long-term learning of learners, and constructs a learning system suitable for the needs of different groups of people, realizing a new model where everyone can learn, learn anywhere, and learn anytime. Attached Figure Description

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0037] Figure 1 This is a flowchart of an intelligent video teaching method for implementing action recognition according to Embodiment 1 of the present invention;

[0038] Figure 2 This is a schematic diagram of the learner's nose and neck joints in Embodiment 1 of the present invention;

[0039] Figure 3 This is a flowchart of a method for interacting with content using movement and body movements according to Embodiment 1 of the present invention;

[0040] Figure 4 This is a schematic diagram of facial key points according to Embodiment 1 of the present invention;

[0041] Figure 5 This is a schematic diagram of the database structure according to Embodiment 1 of the present invention. Detailed Implementation

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0043] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0044] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0045] Example 1

[0046] This embodiment provides an intelligent video teaching method that implements action recognition, such as... Figure 1 As shown, it includes the following steps:

[0047] Step 1: Obtain the learner's color facial image. Use a facial recognition algorithm to search for the learner in the face database. If the learner is found in the face database, obtain the learner's user ID, retrieve the learner's historical learning information (including name, gender, age, learning time, learning points, learning progress, learning efficiency, etc.) from the system database, and upload it to the display interface. At this time, the display interface displays the learner's color facial image and displays the learner's historical learning information, including basic information such as name, age, and last learning time. The display method can be a computer monitor, a large screen, a projector display, etc. Otherwise, add the learner's facial image to the face database, add the learner to the database, and initialize the learner's historical learning information.

[0048] The study utilizes an RGB-D camera to capture color and depth images of learners and calculates the facial image region (color image of the face). The RGB-D camera employs a Microsoft Kinect sensor.

[0049] like Figure 5 As shown, the database mainly contains the user table, class table, classhistory table, question table, and misans table.

[0050] The `user` table stores learner information such as ID, name, information, age, last login time, number of study days, exercise volume, and points. The `class` table stores course ID, course name, and course video address. The `classhistory` table stores learner's course learning history, course exercise scores, and study time. The `question` table stores exercise questions, the course to which the question belongs, the correct answer, and the options for each question. The `misans` table stores information about learners' incorrect answers, mainly including the incorrect question ID, the learner's options, and the time the question was answered.

[0051] After face recognition, the learner ID is obtained from the face library, and the name, information, age, last login time, learning days, exercise amount, and points of the learner are retrieved from the user table in the system database according to the learner ID, the learning history of the learner is retrieved from the classhistory table according to the learner ID, and the historical wrong questions of the learner are retrieved from the misans table according to the learner ID.

[0052] In the method, a face color image of a learner is obtained, and the specific steps for searching the learner in the face library by using a face recognition algorithm are as follows:

[0053] Step 101, using an RGB-D camera, the position coordinates of the nose joint and the neck joint of the learner are obtained based on the color image and the depth image of the learner, as shown in the following formula: Figure 2 The nose joint PNOSE_3d(nose_x_3d, nose_y_3d, nose_z_3d) and the neck joint PNECK_3d(neck_x_3d, neck_y_3d, neck_z_3d) of the learner in the three-dimensional space are located by using a Kinect sensor;

[0054] Step 102, the three-dimensional positions of the nose joint and the neck joint are mapped to the color image to obtain the two-dimensional positions of the nose joint and the neck joint in the color image, specifically, the key point PNOSE(nose_x_3d, nose_y_3d, nose_z_3d) and the key point PNECK(neck_x_3d, neck_y_3d, neck_z_3d) are mapped from the three-dimensional space to the two-dimensional space by using the function KinectBase.calibration.TransformTo2D() to obtain the point PNOSE_2d(nose_x_2d, nose_y_2d) and the point PNECK_2d(neck_x_2d, neck_y_2d);

[0055] Step 103, the side length d of the face cropping square is calculated, and the cropping origin P(px, py) is calculated;

[0056]

[0057] px = nose_x_2d - d

[0058] py = nose_y_2d - d

[0059] Step 104, based on the side length and the cropping origin, the face color image is cropped from the color image. That is, the face image of the learner is obtained by cropping a square from the color image captured by the RGB-D camera with the point P as the origin and d as the side length;

[0060] Step 105, the cropped face image is encoded, specifically, the cropped face image is encoded into a base64 string using the Convert.ToBase64String() function;

[0061] Step 106, the face image obtained above is encoded in the form of a string as a parameter, and a face search is performed in the face library using a face recognition algorithm; specifically, the application uses the Search function provided by the Baidu face application to perform a face search in the face library.

[0062] Step 107, according to the face recognition result, corresponding operation is performed, if the learner is searched in the face library, the learner's ID is returned, the learner's historical learning information is retrieved in the database through the learner ID, including name, age, information, last login time, learning days, exercise amount, points, learning history, etc.; if the learner is not searched in the face library, the face image of the learner is added to the face library, and the color image of the face of the learner is added to the face library, and the historical learning information of the learner in the database is initialized, at this time, the historical learning information of the learner is empty.

[0063] Step 2, based on the learning information of the learner obtained in step 1, it is judged whether the learner is learning for the first time, that is, whether the historical learning information of the learner is empty; if it is the first time to learn, the learner is provided with default learning content (the learner can review the courses less than 60 points before, and then recommend courses to the learner according to the learner's review of the questions, if the learner still has less than 60 points in the review of the courses less than 60 points before, the course will be recommended to the learner to continue review), and jump to step 4; if it is not the first time to learn, that is, the historical learning information of the learner is not empty, the learning content is formulated for the learner according to the historical learning information of the learner (the first three courses can be recommended to the learner according to the order of the courses), and jump to step 3. In this example, at most two review courses are recommended for each learning, if the learner has less than 60 points in the review of more than two courses, the two courses with the lowest scores are selected to be recommended to the learner for review.

[0064] Step 3, review stage, before learning the course, the learner reviews the previously learned content through the way of motion and body interaction (the question types mainly include judgment questions, single choice questions, multiple choice questions, etc.), and examines the student's mastery of the learned content. That is, according to the historical learning information of the learner, a plurality of practice questions are obtained and pushed in sequence, for each practice question, the actual position of the learner is obtained, according to the actual position of the learner, it is judged whether the answer given by the learner is correct, when the learner answers all the practice questions, jump to step 4.

[0065] Among them, for example,Figure 3 As shown, the actual position of the learner is acquired, and according to the actual position of the learner, the specific method for judging whether the answer given by the learner is correct is:

[0066] (1) In the initialization stage, the capture area of the RGB-D camera, i.e. the effective activity area of the learner, is marked, and the present application takes the Microsoft Kinect sensor as an example for illustration. The activity area parameters (area_height, area_width, d1, d2) are set, specifically, the learner activity area with a length of area_height and a width of area_width is marked in front of the Kinect sensor, the distance d1 of the activity area from the Kinect sensor is determined, the horizontal offset d2 of the center of the activity area from the Kinect is determined, in addition, the resolution of the display device is screen_height*screen_width.

[0067] (2) The actual position PREAL(x_real, y_real, z_real) of the learner in the three-dimensional space is acquired by using the Kinect, and is converted into the relative position PRELATIVE(x_relative, y_relative) in the two-dimensional activity space:

[0068] x_relative=d2+area_width / 2-x_real

[0069] y_relative=z_real-d1

[0070] (3) The relative position PRELATIVE(x_relative, y_relative) in the two-dimensional activity space is mapped into the position PSCREEN(x_screen, y_screen) of the virtual image in the display device, wherein:

[0071] x_screen=x_relatie / area_width*screen_width

[0072] y_screen=y_relative / area_height*screen_height

[0073] (4) The virtual cartoon image is used to represent the learner, and the coordinates of the virtual cartoon image in the display device are set as PSCREEN(x_screen, y_screen), so that the motion position of the learner is converted into the motion position of the virtual image in the learning content in real time, and the interaction between the learner and the learning content is realized.

[0074] (5) judging whether the learner's answer is correct: comparing the position of the PSCREEN with the answer information in the database, specifically, obtaining the learner's answer according to the position of the virtual image in the display device, comparing the learner's answer with the standard answer stored in the database, if the answer is correct, giving sound and image prompts, and increasing the learner's score; otherwise, giving the wrong answer sound and image prompts, and giving the correct answer, and recording the wrong answer information to the wrong answer book of the learner database, to ensure subsequent review and consolidation.

[0075] Step 4, learning stage, the learner mainly learns the learning content through video, animation and other forms. That is, based on the learning information of the learner, the learning video is pushed.

[0076] Step 5, judging whether the learner starts learning, if starting learning, playing the learning video, the learner watches the learning video; otherwise, not playing the learning video. During the course learning process, the learner can control the play and pause of the teaching video, control the volume of the teaching video, control the play progress of the teaching video, repeatedly learn a certain knowledge point to strengthen understanding. In the process of playing the learning video, the position of the body joint is obtained in real time, the learner's body action is judged, and the play progress of the learning video is controlled based on the learner's body action. Specifically, the interaction of the learner's body mainly includes that the learner can control the video play and pause during the video learning process, that is, when the learner raises the right hand, the video is played, when the left hand is raised, the video is paused, when the learner's right hand is stretched horizontally, the video is fast forwarded, and when the learner's left hand is stretched horizontally, the video is rewound. That is, the position of the learner's body joint is obtained by using Kinect, and the definition of the body action is completed by judging the relative position of the joint:

[0077] Right hand raising action: if the y coordinate of the right hand joint HAND_RIGHT is less than the y coordinate of the right shoulder coordinate SHOULDER_RIGHT, at this time the learner's right hand is in the state of raising;

[0078] Left hand raising action: if the y coordinate of the left hand joint HAND_LEFT is less than the y coordinate of the left shoulder coordinate SHOULDER_LEFT, at this time the learner's left hand is in the state of raising;

[0079] Right hand stretching action: if the x coordinate of the right hand joint HAND_RIGHT is less than the x coordinate of the right shoulder SHOULDER_RIGHT and the y coordinate of the right hand joint HAND_RIGHT is between the y coordinate of the right shoulder SHOULDER_RIGHT and the y coordinate of the spine center SPINE_NAVAL, at this time the learner's right hand is in the state of stretching horizontally;

[0080] left hand stretch action: if the x coordinate of the left hand joint HAND_LEFT is greater than the x coordinate of the left shoulder SHOULDER_LEFT and the y coordinate of the left hand joint HAND_LEFT is between the y coordinate of the left shoulder SHOULDER_LEFT and the y coordinate of the spine center SPINE_NAVAL, at this time, the left hand of the learner is in a stretch state;

[0081] Step 6, obtaining a color image of the face of the learner, identifying a face key point in the color image of the face based on the color image of the face of the learner, calculating a left and right eye closure degree mean and a mouth opening angle based on the face key point, calculating a head rotation angle based on the color image of the face, and updating the video playing content based on the left and right eye closure degree mean, the mouth opening angle and the head rotation angle.

[0082] For example, when the left and right eye closure degree mean EAR of the learner is less than a threshold value, or the mouth opening angle is greater than a threshold value, or the head rotation angle is greater than a threshold value for the first time, the video playing content is updated, for example, the learner is video relaxed, the learner is relaxed through beautiful natural scenery and soothing music, and after the relaxation is finished, the course learning is continued; during the continuous learning process, if the left and right eye closure degree mean EAR of the learner is less than a threshold value, or the mouth opening angle is greater than a threshold value, or the head rotation angle is greater than a threshold value again, the video playing content can be updated again, for example, an exercise is played or a gopher icon appears randomly at 4 positions on the screen, the learner drives the virtual image on the screen to hit the gopher by moving himself, and if the gopher is hit, the score is increased by 10 points, and the gopher that is not hit disappears after 5 seconds, and when the score of the learner reaches 100 points, the relaxation is finished, and the learner continues the course learning.

[0083] The specific method for identifying the face key point in the color image of the face, calculating the left and right eye closure degree mean and the mouth opening angle based on the face key point, and calculating the head rotation angle based on the color image of the face is as follows:

[0084] Step 601, using a Kinect to obtain a continuous color image and a depth image sequence, and cutting a color image of the face of the learner in the color image. The color image of the face of the learner is obtained in the same way as steps 101-104.

[0085] Step 602, using a face recognition algorithm to locate a face key point, and obtaining the face key point, as shown in Figure 4 The application uses a Baidu face API.

[0086] Step 603, acquire the upper and lower eyelid key points of the right eye, eye_right_upper(x32, y32) and eye_right_lower(x36, y36), the left and right corners of the right eye, eye_right_corner_left(x30, y30) and eye_right_corner_right(x34, y34); acquire the upper and lower eyelid key points of the left eye, eye_left_upper(x15, y15) and eye_left_lower(x19, y19), the left and right corners of the left eye, eye_left_corner_left(x13, y13) and eye_left_corner_right(x17, y17).

[0087] Step 604, calculate the left and right eye closure degrees EAR_LEFT and EAR_RIGHT, wherein:

[0088]

[0089]

[0090] Step 605, calculate the average of the left and right eye closure degrees EAR:

[0091] EAR=(EAR_LEFT+EAR_RIGHT) / 2

[0092] When the EAR value is less than the threshold value, the eye is in a closed state, and then the learner may be in a fatigue state; according to the empirical value, the threshold value in the application is set to 0.2.

[0093] Step 606, acquire the left corner key point of the mouse, mouse_corner_left_outter(x58, y58), the upper corner key point of the mouse, mouse_lip_upper_outer(x60, y60), and the lower corner key point of the mouse, mouse_lip_lower_outer(x64, y64).

[0094] Step 607, calculate the vectors v1 and v2, wherein:

[0095] v1=(x60-x58, y60-y58)

[0096] v2=(x64-x58, y64-y58)

[0097] Step 608, calculate the included angle θ of the vectors v1 and v2, that is, the mouth opening angle:

[0098] θ=arccos(v1·v2 / (||v1||×||v2||))

[0099] If θ is greater than the threshold value, it indicates that the learner may be yawning, i.e. the learner is in a state of fatigue. According to the empirical value, the threshold value in the present application is set to 60°.

[0100] Step 609, the acquired face color image is analyzed by using a face recognition algorithm (such as Baidu face API) to obtain the rotation angles pitch, yaw and roll of the head in x, y and z directions. The present application mainly utilizes the change of the yaw angle, i.e. if the yaw angle is greater than 60° or less than -60°, it indicates that the learner may be in a state of distraction at this time.

[0101] Step 610, the present application takes 20s as a state judgment period, and collects color images and depth images.

[0102] As an embodiment, if the number of frames in which the learner is in a state of distraction exceeds 50% in a period, the learning state of the learner is poor at this time, and the learner needs to be adjusted.

[0103] Step 7, practice stage, after the learning video is acquired and played, a plurality of practice questions are acquired based on the content of the learning video and are pushed in sequence, i.e. after the learner learns a theme content, the practice questions are trained in sequence. The learner answers the practice questions by moving and interacting with the body, and the specific mode is the same as that in step 3.

[0104] Specifically, the corresponding practice questions are pushed to the learner for practice in order to consolidate the learned courses. After the learner reads the questions, the learner drives the virtual image on the screen by moving in the physical space to select the answer; if the correct answer is selected, it is prompted that the answer is correct, and the score is added by 1; if the wrong answer is selected, the learner is prompted that the answer is wrong and the correct answer is prompted; when the learner moves, the distance between the current position of the learner and the position of the learner at the last time is constantly calculated, and the amount of movement of the learner is counted.

[0105] Step 8, it is judged whether the learning content set by the system is learned or not, if not, it is jumped to step 5; otherwise, a learning evaluation report of the learner at this time and a mistake notebook are generated.

[0106] The evaluation report shows the basic information of the learner, the learning points, the exercise amount in the current learning process, the total exercise amount, the concentration degree, the courses that have been learned and the mastering degree. The concentration degree in the application refers to the proportion of the learner's concentration time in the total learning time, and the mastering degree of the learning courses is analyzed according to the answer of each exercise of the learner. In the evaluation report interface, the learner can drive the virtual image on the screen to click the exit button to return to the start interface and continue learning. The learner can also use the body movement to open the wrong question book to strengthen the learning of the wrong exercises in the current learning process. The wrong question book shows the original question and options, the correct answer and the choice of the learner at that time, for the learner to analyze the wrong questions. The learner selects the previous question and the next question button to switch the questions. Clicking the return report button can return to the evaluation report page. Clicking the exit button can jump to the start interface to continue learning.

[0107] Step 9, the learner can realize the viewing of the evaluation report and the wrong question book through the motion and body interaction.

[0108] Step 10, judging whether the viewing is ended, if yes, the current learning is ended, otherwise, jumping to step 9.

[0109] The application formulates the corresponding content of review, learning and practice for the learner according to the historical learning information of the current learner. The learner learns in different stages according to the formulated learning path. In the review and practice stage, the color image and the depth image of the learner are acquired to realize the interaction with the content through position tracking and action recognition; in the learning stage, the color image of the learner is acquired to detect the learning state of the learner, detect whether the learner is in the state of poor learning such as looking around, dozing off, yawning for a long time, and adjust the current content of the learner in real time according to the detection result, including the adjustment of different difficulty learning content, the introduction of the form of arousing the learning enthusiasm of the learner or relaxing content, etc. In the evaluation stage, the learner learns the recommended learning content, and gives the corresponding learning report of the learner according to the learning state in the learning process, the interaction rate and the answer accuracy rate in the review and practice stage.

[0110] The present application considers individual differences of different learners, learning time differences and limitations, focuses on individual characteristics and long-term learning of learners, and provides personalized and adaptive learning content, and builds a learning method suitable for different population needs. The present application provides a self-learning mode of review, learning, practice and evaluation, obtains the learned knowledge of the learner through face recognition of the learner, and formulates personalized learning content for the learner according to the learned knowledge and learning effect. At the same time, the learning state of the learner (such as looking around, dozing off, yawning, etc.) is always paid attention to, and corresponding content adjustment is performed according to the different learning states, so as to ensure the learning efficiency of the learner; the learner can realize interaction with the content through walking and gesture interaction, and the active learning enthusiasm of the learner is ensured.

[0111] Embodiment two

[0112] The embodiment provides an intelligent video teaching system for realizing action recognition, which specifically comprises the following modules:

[0113] A historical learning information acquisition module configured to acquire historical learning information of the learner;

[0114] A learning video playing module configured to acquire and play a learning video based on the historical learning information of the learner;

[0115] In the process of playing the learning video, the face color image of the learner is acquired based on a state judgment period, the face key points in the face color image are recognized, the left and right eye closure degree averages and the mouth opening angle are calculated based on the face key points, and whether the learner is in a fatigue state is judged based on the left and right eye closure degree averages and the mouth opening angle; at the same time, the head key points of the learner are captured, the head rotation angle is calculated to judge whether the learner is in a distraction state.

[0116] It should be noted that each module in the embodiment corresponds to each step in the embodiment one, and the specific implementation process is the same, which will not be repeated here.

[0117] Embodiment three

[0118] The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to realize the steps of the intelligent video teaching method for realizing action recognition in the above embodiment one.

[0119] Embodiment four

[0120] The embodiment provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the program to realize the steps of the intelligent video teaching method for realizing action recognition in the above embodiment one.

[0121] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage and so forth) embodying computer-readable program code.

[0122] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for performing the function specified by the flowchart illustrations and / or block diagrams block or blocks.

[0123] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for performing the function specified by the flowchart illustrations and / or block diagrams block or blocks.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flowchart illustrations and / or block diagrams block or blocks. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams block or blocks. Figure 1 means for performing the function specified by the flowchart illustrations and / or block diagrams block or blocks.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the program can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like.

[0126] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A smart video teaching method for enabling action recognition, characterized by, include: Obtain learners' historical learning information; Based on learners' historical learning information, acquire and play learning videos; During the playback of the learning video, the system acquires the position of the learner's body joints in real time, judges the learner's limb movements, and controls the playback progress of the learning video based on the learner's limb movements. At the same time, it acquires a color image of the learner's face, identifies facial key points in the color image, calculates the average closure degree of the left and right eyes and the mouth opening angle based on the facial key points, calculates the head rotation angle based on the color image of the face, and updates the video playback content based on the average closure degree of the left and right eyes, the mouth opening angle, and the head rotation angle. Set the active area parameters area_height, area_width, d1, d2. Mark the learner's active area with length area_height and width area_width in front of the Kinect sensor. Determine the distance d1 between the active area and the Kinect sensor, and the horizontal offset d2 between the center of the active area and the Kinect sensor. The resolution of the display device is screen_height * screen_width. Use the Kinect to obtain the learner's actual position PREAL(x_real, y_real, z_real) in three-dimensional space, and convert it into a relative position PRELATIVE(x_relative, y_relative) in two-dimensional active space. Map the relative position PRELATIVE(x_relative, y_relative) in the two-dimensional activity space to the position PSCREEN(x_screen, y_screen) of the virtual avatar on the display device, where: Using virtual cartoon characters to represent learners, the coordinates of the virtual cartoon characters on the display device are set to PSCREEN(x_screen,y_screen), and the movement position of the learner is converted into the movement position of the virtual character in the learning content in real time, so as to realize the interaction between the learner and the learning content. Determining the correctness of learners' answers: The position of the PSCREEN is compared with the answer information in the database. Based on the position of the virtual image on the display device, the learner's answer is obtained. The learner's answer is compared with the standard answer stored in the database. If the answer is correct, sound and image prompts are given and the learner's points are increased; otherwise, sound and image prompts for incorrect answers are given, the correct answer is given, and the incorrect answer is recorded in the learner's error notebook in the database. Using Kinect, the positions of the learner's body joints are obtained, and the definition of limb movements is completed by judging the relative positions of the joints. Right hand raised action: If the y-coordinate of the right hand joint HAND_RIGHT is less than the y-coordinate of the right shoulder SHOULDER_RIGHT, the learner's right hand is raised. Left hand raised action: If the y-coordinate of the left hand joint HAND_LEFT is less than the y-coordinate of the left shoulder SHOULDER_LEFT, the learner's left hand is raised. Right hand extended: If the x-coordinate of the right hand joint HAND_RIGHT is less than the x-coordinate of the right shoulder SHOULDER_RIGHT and the y-coordinate of the right hand joint HAND_RIGHT is between the y-coordinates of the right shoulder SHOULDER_RIGHT and the center of the spine SPINE_NAVAL, then the learner's right hand is in an extended state. Left hand extended: If the x-coordinate of the left hand joint HAND_LEFT is greater than the x-coordinate of the left shoulder SHOULDER_LEFT and the y-coordinate of the left hand joint HAND_LEFT is between the y-coordinates of the left shoulder SHOULDER_LEFT and the center of the spine SPINE_NAVAL, then the learner's left hand is in an extended state. The Kinect was used to acquire continuous color and depth image sequences. The learner's face was then cropped from the color images. A face recognition algorithm was used to locate facial landmarks. The key points for the right eye were obtained: eye_right_upper(x32,y32) and eye_right_lower(x36,y36), and the key points for the left and right corners of the eyes were eye_right_corner_left(x30,y30) and eye_right_corner_right(x34,y34). The key points for the left eye were also obtained: eye_left_upper(x15,y15) and eye_left_lower(x19,y19), and the key points for the left and right corners of the eyes were eye_left_corner_left(x13,y13) and eye_left_corner_right(x17,y17). The eye closure degrees EAR_LEFT and EAR_RIGHT were calculated. Calculate the mean EAR of eye closure for both eyes: When the EAR value is less than the threshold, the eyes are closed, and the learner is in a state of fatigue. Obtain the key points for the left corner of the mouth (mouse_corner_left_outer(x58, y58)), the upper corner of the mouth (mouse_lip_upper_outer(x60, y60)), and the lower corner of the mouth (mouse_lip_lower_outer(x64, y64)); calculate the vectors v1 and v2, where: Calculate the angle θ between vectors v1 and v2, which is the mouth opening angle: If θ is greater than the threshold, it indicates that the learner is yawning and is in a state of fatigue. The acquired color facial images are analyzed using a facial recognition algorithm to obtain the head's pitch, yaw, and roll angles in the x, y, and z directions. The change in the yaw angle is used to determine whether the learner is distracted or not. If the yaw angle is greater than 60° or less than -60°, it indicates that the learner is not focused and is in a distracted state. A state judgment cycle of 20 seconds is used to collect color and depth images. If the learner is distracted for more than 50% of the frames in a cycle, the learner's learning state is poor and state adjustment is performed.

2. The intelligent video teaching method for recognizing actions as claimed in claim 1, wherein, The specific method for obtaining learners' historical learning information is as follows: Obtain the learner's color face image and search for the face in the face database. If a matching color face image is found, obtain the learner's ID and iterate through the database based on the learner's ID to obtain the learner's historical learning information; otherwise, add the learner's color face image to the face database and initialize the learner's historical learning information in the database.

3. The intelligent video teaching method for recognizing actions as claimed in claim 1, wherein, The specific method for obtaining the learner's color facial image is as follows: Acquire color and depth images of the learner; Based on the learner's color and depth images, the three-dimensional positions of the learner's nasal and neck joints are obtained. The three-dimensional positions of the nasal joint and the neck joint are mapped onto the color image to obtain the two-dimensional positions of the nasal joint and the neck joint in the color image. Based on the two-dimensional positions of the nose and neck joints in the color image, calculate the side length and origin of the face cropping square; Based on the side length and the cropping origin, the color image of the face is cropped from the color image.

4. The intelligent video teaching method for recognizing actions as claimed in claim 1, wherein, If the learner's historical learning information is not empty, before acquiring and playing the learning video, it is also necessary to acquire several practice questions based on the learner's historical learning information and push them out in sequence.

5. The intelligent video teaching method for recognizing actions as claimed in claim 1, wherein, Also includes: After acquiring and playing the learning video, several practice questions are acquired based on the content of the learning video and pushed out sequentially.

6. The intelligent video teaching method for action recognition according to any one of claims 4 or 5, wherein, After each exercise question is pushed, the learner's actual position in three-dimensional space is obtained and converted into the position of the virtual image on the display device. Based on the position of the virtual image on the display device, the learner's answer is obtained.

7. The intelligent video teaching method for achieving action recognition as described in claim 6, characterized in that, Also includes: The method for obtaining the position of the virtual image on the display device is as follows: Set the active area parameters; Based on the activity area parameters, the learner's actual position in three-dimensional space is transformed into a relative position in two-dimensional activity space, and the relative position in two-dimensional activity space is mapped to the position of the virtual cartoon character in the display device.

8. An intelligent video teaching system for implementing action recognition, employing an intelligent video teaching method for implementing action recognition as described in any one of claims 1-7, characterized in that, include: The historical learning information acquisition module is configured to acquire learners' historical learning information. The learning video playback module is configured to: acquire and play learning videos based on the learner's historical learning information; During the playback of the learning video, the system acquires the positions of the learner's body joints in real time, judges the learner's limb movements, and controls the playback progress of the learning video based on the learner's limb movements. Simultaneously, it acquires a color image of the learner's face, identifies facial key points in the color image, calculates the average closure degree of the left and right eyes and the mouth opening angle based on the facial key points, calculates the head rotation angle based on the color image of the face, and updates the video playback content based on the average closure degree of the left and right eyes, the mouth opening angle, and the head rotation angle.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the intelligent video teaching method for action recognition as described in any one of claims 1-7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the intelligent video teaching method for action recognition as described in any one of claims 1-7.

Images