A classroom behavior analysis method based on facial recognition

Abstract

The present invention relates to the field of image processing technology, and in particular to a classroom behavior analysis method based on face recognition, including: performing face recognition on the video images at each moment of the acquired classroom duration, obtaining the facial feature points of each image, obtaining the standard video image of each student according to the completeness of the facial feature points, matching the facial feature points of the video image at each moment, obtaining the head turning amplitude of each student each time based on the matching of the facial feature points in the image sequence of the head turning process of each student during the classroom duration, and obtaining the classroom content attention index of each student in combination with the head turning duration; obtaining the class concentration of each student according to the attention of the eyeball position corresponding to each eye feature point of each student at all times to the classroom content, and assisting in analyzing the classroom behavior of the students. The present invention analyzes the classroom behavior of students through the changes in the heads and faces of students in the classroom.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a classroom behavior analysis method based on face recognition. Background Art

[0002] With the development of science and technology, "Internet + Education" has gradually entered the campus. Facial recognition can be used to capture and analyze students' facial expressions in the classroom. Depth cameras can capture the depth information of objects in the scene and provide richer spatial perception. In the classroom, students' eye movements and postures can be tracked more accurately.

[0003] The existing facial recognition classroom behavior analysis only records students' abnormal behaviors in class in the background, but does not analyze and judge the abnormal behaviors. As a result, a large amount of complex and redundant record information is generated, which makes it necessary for teachers to analyze unnecessary record information when analyzing students' class status, and it is impossible to manage students' class status in a targeted manner. Summary of the invention

[0004] The invention provides a classroom behavior analysis method based on face recognition to solve the existing problems.

[0005] The present invention provides a classroom behavior analysis method based on facial recognition using the following technical solutions:

[0006] An embodiment of the present invention provides a classroom behavior analysis method based on face recognition, the method comprising the following steps:

[0007] Get video images at every moment during the class;

[0008] Perform face recognition on the video images at each moment to obtain the facial feature points of each image, and obtain the standard video image of each student based on the completeness of the facial feature points;

[0009] Match the facial feature points of the video image at each moment with the standard video image to obtain the image sequence and head turning duration of each student during the class;

[0010] Based on the matching of facial feature points in the image sequence of each student's head turning process during class time, the head turning amplitude of each student is obtained;

[0011] According to the head turning amplitude and head turning duration of each student during the class, the class content attention index of each student is obtained;

[0012] According to the eyeball position presented by the eye feature points in the facial feature points of each student at each moment, the eyeball position corresponding to each eye feature point of each student at each moment is obtained;

[0013] The concentration of each student in class is obtained based on the attention paid to the classroom content by the eye position corresponding to each eye feature point at all times; the concentration of all students in class is used to assist in analyzing students' classroom behavior.

[0014] Furthermore, the steps of obtaining facial feature points of each image include the following:

[0015] The facial recognition algorithm based on CNN neural network is used to perform face recognition on the video images to obtain all the facial feature points of each student in the video image at each moment.

[0016] Furthermore, the step of obtaining a standard video image of each student according to the completeness of facial feature points includes the following specific steps:

[0017] The first video image containing all facial feature points in the face recognition result of the i-th student is used as the standard video image of the i-th student.

[0018] Furthermore, the step of obtaining the head turning process image sequence and the head turning duration of each student during the class includes the following specific steps:

[0019] The sequence consisting of the matching degrees of the video images at all times between the start moment and the end moment of the i-th student's head turning is recorded as the image sequence of the i-th student's head turning process, and the time interval from the start moment and the end moment of the i-th student's head turning is recorded as the head turning duration of the i-th student's corresponding head turning.

[0020] Furthermore, the specific method of obtaining the matching degree is as follows:

[0021] SIFT feature matching is performed on the facial feature points in all video images at consecutive moments and the standard video image of the i-th student. The ratio of the number of facial feature points successfully matched in the video image of the i-th student at each moment to the number of facial feature points of the i-th student in the standard video image is recorded as the matching degree of the video image of the i-th student at each moment.

[0022] Furthermore, the specific method for obtaining the head turning start time and the head turning end time is as follows:

[0023] A matching threshold is preset. When the matching degree of the video image of the ith student at the jth moment is less than the matching threshold, the jth moment is recorded as the starting moment of the ith student's head turning. The matching degree of the video image at each moment after the starting moment of the head turning is further judged. If the matching degree of the video image of the ith student at each moment after the starting moment of the head turning is less than the matching threshold, the next moment is judged. This process is repeated until there is a constant c such that the matching degree of the video image of the ith student at the j+cth moment is greater than or equal to the matching threshold. In this case, the j+cth moment is recorded as the ending moment of the ith student's head turning.

[0024] Furthermore, the step of obtaining the head turning amplitude of each student each time of turning the head includes the following specific steps:

[0025] θ i,v =exp(-ω i,v,min )

[0026] In the formula, θ i,v represents the head turning amplitude of the i-th student’s v-th head turning; ω i,v,min It represents the minimum value of the matching degree of the i-th student in the image sequence of the head turning process of the v-th head turning.

[0027] Furthermore, the specific steps of obtaining the classroom content attention index of each student are as follows:

[0028]

[0029] In the formula, ε i represents the class content attention index of the i-th student, t i,v represents the duration of the vth head turn of the i-th student, T represents the class time, N i represents the number of image sequences of the head turning process of the i-th student, θ i,v It represents the head turning amplitude of the i-th student’s v-th head turning.

[0030] Furthermore, the step of obtaining the eyeball position corresponding to each eye feature point of each student at each moment includes the following specific steps:

[0031] Taking each eye feature point among all facial feature points of each student in the video image at each moment as the center, two eye areas of each student in the video image at each moment are selected by a preset radius, and the Otsu threshold algorithm is used to perform threshold segmentation on each eye area of ​​each student in each video image. The area composed of pixels with grayscale values ​​less than the threshold is recorded as the foreground area of ​​each eye area, and the centroid position in the foreground area of ​​each eye area is taken as the eyeball position corresponding to each eye feature point.

[0032] Furthermore, the specific steps of obtaining each student's concentration in class are as follows:

[0033]

[0034] In the formula, represents the concentration of the i-th student in class, ε i represents the attention index of the i-th student to the classroom content, T represents the class duration, ω i,t represents the degree of matching of the key points of the screen of the i-th student at the t-th time, Δd i,t represents the Euclidean distance between the eye feature point and the eyeball position corresponding to the eye feature point of the i-th student at time t, f(a,Δd i,t ) represents the head turning judgment function, a represents the head turning judgment parameter, when the video image has two facial feature points of the eyes, the head turning judgment parameter a=1; when the video image has one facial feature point of the eye, the head turning judgment parameter a=-1; when the input a=1, the output result is Δd i,t The normalized result of the positive proportion is Δd. When the input a=-1, the output result is i,t The inverse proportional normalization result.

[0035] The beneficial effects of the technical solution of the present invention are: performing face recognition on the video images at each moment within the acquired class duration to obtain the facial feature points of each image as the basis for subsequent matching of different images; obtaining the standard video image of each student according to the completeness of the facial feature points, matching the facial feature points of the video image at each moment, and obtaining the head turning amplitude of each student each time based on the matching of the facial feature points in the image sequence of each student's head turning process within the class duration, and obtaining the class content attention index of each student in combination with the head turning duration, which shows the specific situation of the student's head turning during class; the position of the eyeball in the eye also reflects the student's listening situation in class, and the concentration of each student in class is obtained according to the attention of each student to the class content at all times corresponding to each eye feature point, so as to assist in analyzing the student's classroom behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 The present invention is a flow chart of the steps of a classroom behavior analysis method based on facial recognition. DETAILED DESCRIPTION

[0038] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of a classroom behavior analysis method based on facial recognition proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0039] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0040] The following is a detailed description of a specific scheme of a classroom behavior analysis method based on facial recognition provided by the present invention in conjunction with the accompanying drawings.

[0041] See also Figure 1 , which shows a flow chart of a classroom behavior analysis method based on facial recognition provided by an embodiment of the present invention, the method comprising the following steps:

[0042] Step S001: Obtain video images at each moment during the class duration.

[0043] The purpose of this embodiment is to analyze students' classroom behavior through changes in their heads and faces in class; therefore, it is necessary to install a depth camera in the front of the classroom to collect video images at each moment during a class. In this embodiment, the collection duration is the time of a class, and the collection frequency is once per second. This is used as an example for description. Other embodiments may set other values, which are not specifically limited in this embodiment.

[0044] Step S002: Match the facial feature points of the video image at each moment with the standard video image to obtain the matching degree of each image; obtain the head turning amplitude of each student each time according to the matching of the facial feature points in the image sequence of each student's head turning process during the class; obtain the attention index of each student to the class content according to the amplitude and duration of each student's head turning.

[0045] It should be noted that during the class, the degree of head shaking of the students is reflected in the change of the matching degree of the key points of the students' faces between each video image at consecutive moments. When the key points of the students' faces on the video image match the key points of the students' faces on the standard video image used as a reference to a large extent, it means that the students are less likely to turn their heads at this moment; when only some of the key points of the students' faces on the video image can be successfully matched with the key points of the students' faces on the standard video image, it means that the students are more likely to turn their heads.

[0046] Specifically, face recognition is performed on video images using a facial recognition algorithm based on a CNN neural network, and key points of key facial parts such as the mouth, eyes, and nose in the video images after face recognition are located to obtain all facial feature points of each student in the video images at each moment. The facial feature points described in this embodiment include left and right corners of the mouth feature points, two eye feature points, and a nose feature point. The facial recognition algorithm based on a CNN neural network is an existing well-known technology and will not be described in detail in this embodiment.

[0047] For the ith student, the first video image containing all facial feature points in the face recognition result of the ith student is used as the standard video image of the ith student, and SIFT feature matching is performed on the facial feature points in all video images at consecutive moments and the standard video image of the ith student. The ratio of the number of facial feature points successfully matched in the video image of the ith student at each moment to the number of facial feature points of the ith student in the standard video image is recorded as the matching degree of the video image of the ith student at each moment. In this embodiment, SIFT feature matching is a well-known technology and will not be described in detail in this embodiment.

[0048] A matching threshold is preset. For the video image of the ith student at the jth moment, when the matching degree of the video image of the ith student at the jth moment is greater than or equal to the matching threshold, it indicates that the ith student's classroom performance is normal, and the jth moment is recorded as a normal moment of the ith student.

[0049] When the matching degree of the video image of the i-th student at the j-th moment is less than the matching threshold, it means that the i-th student has turned his head at the j-th moment, and the j-th moment is recorded as the starting moment of the i-th student's head turning; further judge the matching degree of the video image at each moment after the starting moment of the head turning, if the matching degree of the video image of the i-th student at each moment after the starting moment of the head turning is less than the matching threshold, then judge the next moment; and so on, until there is a constant c such that the matching degree of the video image of the i-th student at the j+c-th moment is greater than or equal to the matching threshold, then the j+c-th moment is recorded as the ending moment of the i-th student's head turning. The sequence of the matching degrees of the video images at all moments between the starting moment of the i-th student's head turning and the ending moment of the head turning is recorded as the image sequence of the i-th student's head turning process, and the time interval from the starting moment of the head turning to the ending moment of the head turning of the i-th student is recorded as the head turning duration of the i-th student's corresponding head turning, and obtain the image sequences of several head turning processes of the i-th student in the video images at all moments.

[0050] For the image sequence of the vth head turning process of the ith student, the head turning amplitude of the vth head turning is obtained according to the matching degree of the video images at all moments in the image sequence of the vth head turning process of the ith student. Then, the calculation method of the head turning amplitude of the ith student in the vth head turning is:

[0051] θ i,v =exp(-ω i,v,min )

[0052] In the formula, θ i,v represents the head turning amplitude of the i-th student’s v-th head turning; ω i,v,min It represents the minimum value of the matching degree of the i-th student in the image sequence of the head turning process of the v-th head turning.

[0053] The lower the matching degree between the video image at the tth moment and the standard video image, the lower the overlap between the facial feature points of the student and the facial feature points on the standard video image. The greater the amplitude of the head turning, the more likely the student's face is facing the surroundings.

[0054] When a student frequently turns his head during class, and each time the amplitude of the head turn is large or the duration is long, it means that the student's attention is scattered and the degree of attention to the class content is not high. According to the amplitude of the head turn, the number of head turns and the frequency of head turns shown by the facial feature points of the students in all continuous video images, the class content attention index of the i-th student is obtained:

[0055]

[0056] In the formula, ε i represents the class content attention index of the i-th student, ti,v represents the duration of the vth head turn of the i-th student, T represents the class time, N i represents the number of image sequences of the head turning process of the i-th student, θ i,v It represents the head turning amplitude of the i-th student’s v-th head turning.

[0057] It indicates the time spent turning the head for the vth time as a percentage of the entire class time. The larger the percentage, the longer the student turns his head, and the lower the class attention index. It means that the larger the amplitude of the i-th student's head turning each time and the longer the duration, the lower the student's attention index to the classroom content.

[0058] At this point, the attention index of each student to the classroom content is obtained.

[0059] Step S003: Obtain the concentration of each student in class based on the relationship between the changes in each student's eyes and the twisting of his head.

[0060] It should be noted that the expression of students' facial features also reflects the students' listening situation in class, among which changes in eye expression are an important influence on students' concentration on class content; when students turn their heads, if their eyes are still looking at the blackboard, it means that the students may turn their heads for other reasons, but their attention is still focused on the teaching content taught by the teacher; under normal circumstances, students' eyes will follow the twisting of the head and look in the direction of the head twisting, and the position of the eyeballs in the eyes remains basically unchanged. When the students' eyes are still looking at the blackboard, the eyeballs do not change with the twisting of the head, and the position of the eyeballs in the eyes changes; but when the students do not turn their heads, the position of the eyeballs in the eyes basically does not change.

[0061] Specifically, taking each eye feature point among all facial feature points of each student in the video image at each moment as the center, two eye areas of each student in the video image at each moment are selected by a preset radius, and the Otsu threshold algorithm is used to perform threshold segmentation on each eye area of ​​each student in each video image, and the area composed of pixels with a grayscale value less than the threshold is recorded as the foreground area of ​​each eye area, and the centroid position in the foreground area of ​​each eye area is used as the eyeball position corresponding to each eye feature point.

[0062] It should be further explained that when the student does not turn his head, the video image has facial feature points of two eyes, and the position of the eyeballs in the eyes remains basically unchanged; but when the student turns his head, due to the twisting of the student's head, the facial image of the face obtained by the camera is limited. When the student's head turns to the left, the feature points in the video image and the standard video image match the right face part. There is only the right eye in the video image, and it is necessary to distinguish the student's status by analyzing the position of the eyeball in the right eye. When the student's head turns to the right, there is only the left eye in the video image, and it is necessary to distinguish the student's status by analyzing the position of the eyeball in the left eye.

[0063] Furthermore, based on the position of the eyeball in the eye, the amplitude of the student's head twisting, and the student's attention to the class content, the calculation method for the i-th student's class concentration is:

[0064]

[0065] In the formula, represents the concentration of the i-th student in class, ε i represents the attention index of the i-th student to the classroom content, T represents the class duration, ω i,t represents the degree of matching of the key points of the screen of the i-th student at the t-th time, Δd i,t represents the Euclidean distance between the eye feature point and the eyeball position corresponding to the eye feature point of the i-th student at time t, f(a,Δd i,t ) represents the head turning judgment function, a represents the head turning judgment parameter, when the video image has two facial feature points of the eyes, the head turning judgment parameter a=1; when the video image has one facial feature point of the eye, the head turning judgment parameter a=-1; when the input a=1, the output result is Δd i,t The normalized result of the positive proportion is Δd. When the input a=-1, the output result is i,t It should be noted that the inverse proportional normalization result of the head turning determination function is not specifically limited to the direct proportional normalization and inverse proportional normalization models in this embodiment.

[0066] f(a,Δd i,t ) indicates that when a=1, the student did not turn his head, which means that the smaller the Euclidean distance between the eyeball and the center of the eye, the more focused the student is on looking at the blackboard, and the higher the concentration in class. When a=-1, the student turned his head, which means that the smaller the Euclidean distance between the eyeball and the center of the eye, the student did not look at the blackboard while turning his head, and the lower the concentration in class.

[0067] At this point, each student's concentration in class is obtained.

[0068] After each class, all students’ concentration levels will be summarized and sent to the teacher to help the teacher adjust students’ status in time.

[0069] So far, the present invention is completed.

[0070] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A classroom behavior analysis method based on facial recognition, characterized in that: The method comprises the following steps: Get video images at every moment during the class; Perform face recognition on the video images at each moment to obtain the facial feature points of each image, and obtain the standard video image of each student based on the completeness of the facial feature points; Match the facial feature points of the video image at each moment with the standard video image to obtain the image sequence and head turning duration of each student during the class; Based on the matching of facial feature points in the image sequence of each student's head turning process during class time, the head turning amplitude of each student is obtained; According to the head turning amplitude and head turning duration of each student during the class, the class content attention index of each student is obtained; According to the eyeball position presented by the eye feature points in the facial feature points of each student at each moment, the eyeball position corresponding to each eye feature point of each student at each moment is obtained; The concentration of each student in class is obtained based on the attention paid to the classroom content by the eye position corresponding to each eye feature point at all times; the concentration of all students in class is used to assist in analyzing students' classroom behavior; The specific steps of obtaining each student's classroom content attention index are as follows: In the formula, ε i represents the class content attention index of the i-th student, t i，v represents the duration of the vth head turn of the i-th student, T represents the class time, N i represents the number of image sequences of the head turning process of the i-th student, θ i，v represents the head turning amplitude of the i-th student’s v-th head turning; The specific steps of obtaining each student's concentration in class are as follows: In the formula, represents the concentration of the i-th student in class, ε i represents the attention index of the i-th student to the classroom content, T represents the class duration, ω i，t represents the degree of matching of the key points of the screen of the i-th student at the t-th time, Δd i，t represents the Euclidean distance between the eye feature point and the eyeball position corresponding to the eye feature point of the i-th student at time t, f(a, Δd i，t ) represents the head turning judgment function, a represents the head turning judgment parameter, when the video image has two facial feature points of the eyes, the head turning judgment parameter a=1; when the video image has one facial feature point of the eye, the head turning judgment parameter a=-1; when the input a=1, the output result is Δd i，t The normalized result of the positive proportion is Δd. When the input a=-1, the output result is i，t The inverse proportional normalization result.

2. A classroom behavior analysis method based on facial recognition according to claim 1, characterized in that: The specific steps of obtaining facial feature points of each image are as follows: The facial recognition algorithm based on CNN neural network is used to perform face recognition on the video images to obtain all the facial feature points of each student in the video image at each moment.

3. A classroom behavior analysis method based on facial recognition according to claim 1, characterized in that: The specific steps of obtaining the standard video image of each student according to the completeness of facial feature points are as follows: The first video image containing all facial feature points in the face recognition result of the i-th student is used as the standard video image of the i-th student.

4. A classroom behavior analysis method based on facial recognition according to claim 1, characterized in that: The specific steps of obtaining the head turning process image sequence and the head turning duration of each student during the class time are as follows: The sequence consisting of the matching degrees of the video images at all times between the start moment and the end moment of the i-th student's head turning is recorded as the image sequence of the i-th student's head turning process, and the time interval from the start moment and the end moment of the i-th student's head turning is recorded as the head turning duration of the i-th student's corresponding head turning.

5. A classroom behavior analysis method based on facial recognition according to claim 4, characterized in that: The specific method of obtaining the matching degree is as follows: SIFT feature matching is performed on the facial feature points in all video images at consecutive moments and the standard video image of the i-th student. The ratio of the number of facial feature points successfully matched in the video image of the i-th student at each moment to the number of facial feature points of the i-th student in the standard video image is recorded as the matching degree of the video image of the i-th student at each moment.

6. A classroom behavior analysis method based on facial recognition according to claim 5, characterized in that: The specific method for obtaining the head turning start time and the head turning end time is as follows: A matching threshold is preset. When the matching degree of the video image of the ith student at the jth moment is less than the matching threshold, the jth moment is recorded as the starting moment of the ith student's head turning. The matching degree of the video image at each moment after the starting moment of the head turning is further judged. If the matching degree of the video image of the ith student at each moment after the starting moment of the head turning is less than the matching threshold, the next moment is judged. This process is repeated until there is a constant c such that the matching degree of the video image of the ith student at the j+cth moment is greater than or equal to the matching threshold. In this case, the j+cth moment is recorded as the ending moment of the ith student's head turning.

7. The classroom behavior analysis method based on facial recognition according to claim 1, characterized in that: The specific steps of obtaining the head turning amplitude of each student each time of turning the head are as follows: i i，v =exp(-ω i，v，min ) In the formula, θ i，v represents the head turning amplitude of the i-th student’s v-th head turning; ω i，v，min It represents the minimum value of the matching degree of the i-th student in the image sequence of the head turning process of the v-th head turning.

8. The classroom behavior analysis method based on facial recognition according to claim 1, characterized in that: The specific steps of obtaining the eyeball position corresponding to each eye feature point of each student at each moment are as follows: Taking each eye feature point among all facial feature points of each student in the video image at each moment as the center, two eye areas of each student in the video image at each moment are selected by a preset radius, and the Otsu threshold algorithm is used to perform threshold segmentation on each eye area of ​​each student in each video image. The area composed of pixels with grayscale values ​​less than the threshold is recorded as the foreground area of ​​each eye area, and the centroid position in the foreground area of ​​each eye area is taken as the eyeball position corresponding to each eye feature point.

Images