Human-computer interaction method and system based on computer vision

Abstract

The invention discloses a man-machine interaction method and system based on computer vision. The method comprises the steps that student labels are pasted according to student scores and subject types, and score label information is obtained; the subject types are divided into general observation subjects and key observation subjects based on the learning direction of the to-be-detected class, whether the students have violation or not is detected when the to-be-detected class carries out course projection learning of the general observation subjects, if yes, marking is carried out, and when the to-be-detected class is detected to carry out course projection learning of the key observation subjects, marking is carried out. Determining class violation behaviors of key subjects according to the teaching content and the historical class conditions, detecting whether the students violate the class or not based on the class violation behaviors of the key subjects, and marking if the students violate the class; and detecting the learning input degree of the student with the label to obtain a target student learning input degree P, and if the target student learning input degree P does not exceed a preset student classroom input degree threshold, marking the student as a feature student. The method has the effect of improving the classroom learning man-machine interaction efficiency based on computer vision.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a human-computer interaction method and interaction system based on computer vision. Background Art

[0002] Computer vision is the study of how computers can simulate or implement human vision. Through techniques such as image processing and machine learning, computers are equipped to perceive, recognize, and understand images. Computer vision involves acquiring images through image sensors, processing and analyzing them using image processing techniques, and finally classifying, recognizing, and understanding them through techniques such as machine learning. Human-computer interaction refers to the exchange of information between humans and computer systems using a specific language and interactive methods to accomplish specific tasks.

[0003] The existing computer vision-based human-computer interaction method for classroom teaching is to use a camera to capture images of students and then judge the students' behavior in class based on the captured image information to determine whether they have violated the rules, such as leaning sideways to communicate or sleeping. However, the existing computer vision-based human-computer interaction method for classroom teaching does not take into account the different levels of seriousness in different subjects. When studying in a classroom with a lower level of seriousness, students leaning sideways to communicate and discuss cannot be judged as violations. The accuracy of judging students' violations is low, resulting in low efficiency of human-computer interaction in classroom learning based on computer vision, and there is room for improvement. Summary of the Invention

[0004] In order to improve the efficiency of human-computer interaction in classroom learning based on computer vision, the present application provides a human-computer interaction method and interaction system based on computer vision.

[0005] In the first aspect, the present application provides a human-computer interaction method based on computer vision, which adopts the following technical solutions:

[0006] A human-computer interaction method based on computer vision comprises the following steps:

[0007] Label students according to their grades and subject types to obtain grade label information;

[0008] Based on the learning direction of the tested class, the subject types are divided into general observation subjects and key observation subjects. When the tested class is conducting course projection learning of the general observation subjects, whether students have violated the rules is detected. If so, the rules are marked. When the tested class is conducting course projection learning of the key observation subjects, the violations in the key subject classes are determined based on the teaching content and historical class conditions. Based on the violations in the key subject classes, whether students have violated the rules is detected. If so, the rules are marked.

[0009] According to the grade label information, the learning engagement of the labeled students is detected to obtain the target student's learning engagement P. If the target student's learning engagement P does not exceed the preset student classroom engagement threshold, the student is marked as a feature student and sent to the background monitoring system.

[0010] Preferably, each student in the class to be tested is numbered to obtain student number information;

[0011] Obtain the real-time score information of each student in the class to be tested, store and update the real-time score information in real time, and match the real-time score of each student with the student number;

[0012] Calculate the average score of each subject in the class to be tested based on the real-time score information to obtain the average score;

[0013] Compare the scores of each student in each subject in the tested class with the average scores of each subject. If the student's score is lower than the average score, calculate the difference between the student's score and the average score to get the score difference.

[0014] Set the score difference threshold for each subject based on the full score of each subject's examination paper and the average score;

[0015] Comparing the grade difference with the grade difference threshold to obtain a grade difference comparison result, judging whether each student's grades in each subject are weak based on the grade difference comparison result, if the grade difference is greater than or equal to the grade difference threshold, it means that the student's subject grade is weak, judging the subject type of the student's weak subject grade based on the subject type to obtain the weak subject type;

[0016] Based on the student number information and the weak subject type, students with weak subject performance are labeled with weak subject type.

[0017] Preferably, based on the real-time score information, the real-time score of each subject type of each student in the tested class is compared with the subject score of the same subject type in the previous time to obtain a subject score comparison result;

[0018] Based on the subject score comparison results, if the real-time score is lower than the previous subject score of the same subject type, the difference between the real-time score and the previous subject score of the same subject type is calculated to obtain the theoretical score regression difference H1;

[0019] Detecting the difficulty of each subject type test corresponding to the real-time score to obtain a first test difficulty value K1, and detecting the difficulty of the test corresponding to the previous subject score of the same subject type to obtain a second test difficulty value K2;

[0020] According to the first test difficulty value K1 and the second test difficulty value K2, based on the regression threshold function The theoretical performance regression difference threshold H2 is calculated, where is the scale factor and is greater than 0;

[0021] Compare the theoretical score regression difference H1 with the theoretical score regression difference threshold H2. If the theoretical score regression difference H1 is greater than or equal to the theoretical score regression difference threshold H2, it means that the student's subject score has regressed. Based on the subject type, determine the subject type in which the student's subject score has regressed and obtain the regressed subject type.

[0022] Based on the student ID information and the subject type of decline, students whose subject performance declines are labeled with the subject type of decline;

[0023] The weak subject type label and the regressed subject type label are combined to form the performance label information.

[0024] Preferably, the subject types are divided into general observation subjects and key observation subjects based on the learning direction of the class to be tested;

[0025] When the class to be tested is studying a general observation subject, the camera is used to detect whether the student has left their seat. If so, it is determined whether the student left their seat to answer a question. If the student did not leave their seat to answer a question, the duration of the student's absence is detected to obtain the absence duration information.

[0026] Compare the time spent away from the seat with the preset time threshold. If the time spent away from the seat is greater than the preset time threshold, mark the student who left the seat as a general class violation.

[0027] The names of students who have violated the rules in general subject classes are displayed on a display screen, and the names of students who have violated the rules in general subject classes are sent to a background monitoring system based on a wireless communication module.

[0028] Preferably, when the class to be tested is studying a course of a key observation subject, the teaching content of the current course of the key observation subject is obtained to obtain the teaching content information of the key observation subject;

[0029] Based on the teaching content information of the key observed subject, determine the proportion of the content that needs to be discussed by students in the teaching content to obtain the teaching content discussion ratio, and obtain the teaching content influence coefficient AX based on the teaching content discussion ratio;

[0030] Obtaining the historical teaching situation of the teachers teaching the key observed subjects of the tested classes, wherein the historical teaching situation includes the duration of historical class discussions and the intensity of historical class discussions;

[0031] According to the length of history class discussions and the intensity of history class discussions, the seriousness of the history teaching of the teachers of the key observation subjects in the tested classes is judged.

[0032] According to the teaching content influence coefficient AX and the history teaching seriousness AY, the classroom seriousness W of the key observation subject of the class to be tested is calculated based on the classroom seriousness function W = a1 × AX + a2 × AY;

[0033] Based on the teaching content and classroom seriousness W, the classroom violations of the key observed subjects of the tested class are determined to obtain the key subject classroom violations;

[0034] Based on the camera, detect whether there are any key subject classroom violations in the key observation subject classroom of the tested class. If so, mark the students who have violated the key subject classroom violations as key subject classroom violations.

[0035] The names of students who have violated the rules in key subject classes are displayed on the display screen, and the names of students who have violated the rules in key subject classes are sent to the background monitoring system based on the wireless communication module;

[0036] The general subject classroom violations and the key subject classroom violations are combined to form classroom violations.

[0037] Preferably, based on the performance label information, students with weak subject type labels or regressing subject type labels are recorded as target students, and the relevant subject types in the performance label information of the target students are recorded as benchmark subjects;

[0038] Based on the camera detection, the target student answers questions when studying the benchmark subject, and the answer frequency impact coefficient BR is determined based on the target student's answer frequency.

[0039] Detect the teacher's response information after the target student answers the question, judge the correctness of the target student's answer based on the teacher's response information to obtain the correctness of the answer, and determine the correctness influence coefficient BH based on the correctness of the answer;

[0040] The number of times the target student makes eye contact with the teacher of the benchmark course when studying the benchmark subject is detected to obtain the teacher-student eye contact information, and the learning eye contact impact coefficient BD is determined based on the teacher-student eye contact information;

[0041] According to the influence coefficient of answer times BR, the influence coefficient of correct answer BH and the influence coefficient of learning eye contact times BD, the learning engagement function P = b1×BR+b2×BH+b3×BD is used to calculate the target student's learning engagement P;

[0042] Compare the target student's learning engagement P with the preset student classroom engagement threshold to obtain a learning engagement comparison result;

[0043] Based on the comparison results of learning engagement, if the target student's learning engagement P does not exceed the preset student classroom engagement threshold, the target student is marked as a characteristic student;

[0044] Based on the wireless communication module, the characteristic students are sent to the background monitoring system.

[0045] In a second aspect, the present application provides a human-computer interaction system based on computer vision, which adopts the following technical solutions:

[0046] A human-computer interaction system based on computer vision, comprising:

[0047] The grade labeling module is used to label students according to their grades and subject types to obtain grade label information;

[0048] The classroom violation detection and marking module is configured to divide the subject types into general observation subjects and key observation subjects based on the learning direction of the tested class. When the tested class is studying the general observation subject, whether the students have violated the rules is detected. If so, the violation is marked. When the tested class is studying the key observation subject, the violation of the key subject class is determined based on the teaching content and the historical class situation. Based on the violation of the key subject class, whether the students have violated the rules is detected. If so, the violation is marked.

[0049] The classroom learning engagement analysis module is used to detect the learning engagement of students with labels based on the grade label information to obtain the target student's learning engagement P. If the target student's learning engagement P does not exceed the preset student classroom engagement threshold, the student is marked as a feature student and sent to the background monitoring system.

[0050] In summary, this application includes at least one of the following beneficial technical effects:

[0051] 1. By comparing the difference between the student score of each student in the tested class and the average score in each subject with the score difference threshold, it is determined whether the score of each student in the tested class is weak. If weak, a weak subject type label is attached to the student, so as to provide data support for human-computer interaction used in classroom teaching in the future. With the help of the real-time score information of the tested class, the real-time score of each student in each subject type exam in the tested class is compared with the subject score of the same subject type in the previous exam, and then the theoretical score decline difference H1 is obtained. The theoretical score decline difference threshold H2 is calculated according to the difficulty value K1 of the first exam and the difficulty value K2 of the second exam. The theoretical score decline difference H1 is compared with the theoretical score decline difference threshold H2 to determine whether the students in the tested class have declined in their scores. If the theoretical score decline difference H1 is greater than the theoretical score decline difference threshold H2, a decline subject type label is attached to the student, so as to provide data support for human-computer interaction used in classroom teaching in the future.

[0052] 2. When the class to be tested is studying the key observed subject, the teaching content of the current course of the key observed subject is obtained to obtain the teaching content information of the key observed subject, the teaching content influence coefficient AX is obtained according to the teaching content, the historical teaching seriousness AY of the teacher of the key observed subject of the class to be tested is obtained according to the historical teaching situation of the teacher of the key observed subject of the class to be tested, the classroom seriousness W of the key observed subject of the class to be tested is obtained according to the teaching content influence coefficient AX and the historical teaching seriousness AY, and then the key subject classroom violations are determined, and the key subject classroom violation standards are formulated for different classroom seriousnesses, thereby improving the accuracy of human-computer interaction for classroom teaching, and the key subject classroom of the class to be tested is detected based on the camera. If there is a key subject classroom violation in the class of the key observed subject of the class to be tested, the student is marked as having committed a key subject classroom violation, and the name of the student who has committed a key subject classroom violation is displayed on the display screen, so that the teacher of the key observed subject of the class to be tested can remind the student, and the name of the student who has committed a key subject classroom violation is sent to the background monitoring system, so that the head teacher of the class to be tested can understand the key subject classroom situation and facilitate unified management of students;

[0053] 3. With the help of a camera, the target students are detected in the course of the benchmark subject, and the number of times they answer questions is obtained, and the answer number influence coefficient BR is obtained. The correctness of the target students' answers to the questions is analyzed to obtain the answer correctness influence coefficient BH. The number of times the target students look at the teacher of the benchmark course when studying the benchmark subject is detected, and the learning eye contact influence coefficient BD is obtained. According to the answer number influence coefficient BR, the answer correctness influence coefficient BH and the learning eye contact influence coefficient BD, the target students' learning engagement P is obtained, which improves the detection accuracy of the target students' learning engagement P. The target students' learning engagement P is compared with the preset student classroom engagement threshold. If the target students' learning engagement P does not exceed the preset student classroom engagement threshold, they are marked as feature students. The feature students are sent to the background monitoring system based on the wireless communication module, which improves the efficiency of human-computer interaction in classroom learning based on computer vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flowchart of a human-computer interaction method based on computer vision.

[0055] Figure 2 This is a module diagram of a human-computer interaction system based on computer vision, which is mainly embodied in this embodiment.

[0056] Attached figures: 1. Grade label setting module; 2. Classroom violation detection marking module; 3. Classroom learning input analysis module. DETAILED DESCRIPTION

[0057] The present application is further described in detail below with reference to the accompanying drawings.

[0058] The embodiment of the present application discloses a human-computer interaction method based on computer vision.

[0059] A human-computer interaction method based on computer vision comprises the following steps:

[0060] Reference Figure 1 In step S1, students are labeled according to their grades and subject types to obtain grade label information. Step S1 specifically includes the following sub-steps:

[0061] Step SA1: Number each student in the class to be tested to obtain student number information.

[0062] Step SA2: Obtain the scores of each student in the class to be tested in real time to obtain real-time score information, store and update the real-time score information in real time, and match the real-time score of each student with the student ID.

[0063] Step SA3: Calculate the average score of each subject in the class to be tested based on the real-time score information to obtain the average score.

[0064] Step SA4: Compare the scores of each subject of each student in the class to be tested with the average scores of each subject. If the student score is lower than the average score, calculate the difference between the student score and the average score to obtain the score difference.

[0065] Step SA5: Set the score difference threshold for each subject based on the full score and average score of each subject. The greater the full score of the test paper, the greater the score difference threshold, and the lower the average score, the greater the score difference threshold.

[0066] Step SA6, compare the grade difference with the grade difference threshold to obtain the grade difference comparison result, and judge whether the grades of each student in each subject are weak based on the grade difference comparison result. If the grade difference is greater than or equal to the grade difference threshold, it means that the student's subject grade is weak. Based on the subject type, judge the subject type in which the student's subject grade is weak to obtain the weak subject type.

[0067] Step SA7: assign weak subject type labels to students with weak subject performance based on student ID information and weak subject types.

[0068] In specific applications, by comparing the difference between the scores of each student in each subject in the tested class and the average score with the score difference threshold, it is determined whether the scores of each subject of each student in the tested class are weak. If weak, the student is labeled with a weak subject type, which is convenient for providing data support for human-computer interaction used in classroom teaching.

[0069] Step S1 also includes the following sub-steps:

[0070] Step SB1: Based on the real-time score information, the real-time score of each subject type of each student in the tested class is compared with the previous subject score of the same subject type to obtain a subject score comparison result.

[0071] Step SB2, based on the subject score comparison result, if the real-time score is less than the previous subject score of the same subject type, then calculate the score difference between the real-time score and the previous subject score of the same subject type to obtain the theoretical score regression difference H1.

[0072] Step SB3: Detect the difficulty of each subject type test corresponding to the real-time score to obtain a first test difficulty value K1, and detect the difficulty of the previous test corresponding to the subject score of the same subject type to obtain a second test difficulty value K2.

[0073] Step SB4: Based on the first test difficulty value K1 and the second test difficulty value K2, the regression threshold function The theoretical performance regression difference threshold H2 is calculated, where is the scaling factor and is greater than 0.

[0074] Step SB5, compare the theoretical score regression difference H1 with the theoretical score regression difference threshold H2. If the theoretical score regression difference H1 is greater than or equal to the theoretical score regression difference threshold H2, it means that the student's subject score has regressed. Based on the subject type, the subject type of the student's subject score regression is determined to obtain the regressed subject type.

[0075] Step SB6: Based on the student ID information and the subject type of decline, labels of the subject type of decline are attached to the students whose subject performance has declined.

[0076] Step SB7: The weak subject type label and the regressing subject type label are combined to form the performance label information.

[0077] In specific application, the real-time score information of the class to be tested is used to compare the real-time scores of each student in each subject type in the class to be tested with the subject scores of the same subject type in the previous time, and then the theoretical score regression difference H1 is obtained. The theoretical score regression difference threshold H2 is calculated according to the difficulty value K1 of the first test and the difficulty value K2 of the second test. The theoretical score regression difference H1 is compared with the theoretical score regression difference threshold H2 to determine whether there is any decline in the grades of each student in the class to be tested. If the theoretical score regression difference H1 is greater than the theoretical score regression difference threshold H2, the student is labeled with a regression subject type, which is convenient for providing data support for human-computer interaction used in classroom teaching.

[0078] Reference Figure 1 In step S2, based on the learning direction of the class to be tested, the subject types are divided into general observation subjects and key observation subjects. When the class to be tested is conducting course projection learning of the general observation subject, whether the students have violated the rules is detected. If so, the rules are marked. When the class to be tested is conducting course projection learning of the key observation subject, the key subject classroom violations are determined based on the teaching content and historical class conditions. Based on the key subject classroom violations, whether the students have violated the rules is detected. If so, the rules are marked. Step S2 specifically includes the following sub-steps:

[0079] Step SC1: Classify the subject types into general observation subjects and key observation subjects based on the learning direction of the class to be tested.

[0080] Step SC2: When the class to be tested is studying a general observation subject, the camera is used to detect whether the student has left his seat. If the student has left his seat, it is determined whether the student left his seat to answer a question. If the student did not leave his seat to answer a question, the duration of the student's absence is detected to obtain the duration of the absence.

[0081] Step SC3: compare the time spent away from the seat with the preset time threshold for leaving the seat. If the time spent away from the seat is greater than the preset time threshold for leaving the seat, the student who left the seat will be marked as a general subject classroom violation.

[0082] In step SC4, the names of students who have violated the general subject classroom rules are displayed on the display screen, and the names of students who have violated the general subject classroom rules are sent to the background monitoring system based on the wireless communication module.

[0083] Step S2 also includes the following sub-steps:

[0084] Step SD1: When the class to be tested is studying the course of the key observation subject, the teaching content of the current course of the key observation subject is obtained to obtain the teaching content information of the key observation subject.

[0085] Step SD2: Based on the key observed subject teaching content information, determine the proportion of content in the teaching content that requires student discussion, and obtain the teaching content discussion ratio. Based on the teaching content discussion ratio, obtain the teaching content influence coefficient AX. The greater the teaching content discussion ratio, the greater the teaching content influence coefficient AX.

[0086] Step SD3: Obtain the historical teaching situation of the teacher teaching the key observed subject of the class to be tested, where the historical teaching situation includes the duration of historical class discussions and the intensity of historical class discussions.

[0087] Step SD4: Based on the duration and intensity of history class discussions, determine the history teaching seriousness AY of the teacher teaching the key observed subject in the tested class. The longer the history class discussions, the lower the history teaching seriousness AY, and the more intense the history class discussions, the lower the history teaching seriousness AY.

[0088] Step SD5, according to the teaching content influence coefficient AX and the history teaching seriousness AY, based on the classroom seriousness function W=a1×AX+a2×AY, calculate the classroom seriousness W of the key observation subject of the class to be tested.

[0089] Step SD6: Based on the teaching content and the classroom seriousness W, the classroom violations of the key observed subjects of the class to be tested are determined to obtain the key subject classroom violations.

[0090] Step SD7: Detect whether there are any key subject classroom violations in the key observed subject classroom of the tested class based on the camera. If so, mark the students who have violated the key subject classroom rules as key subject classroom violations.

[0091] Step SD8: Display the names of students who have violated the rules in key subject classes on the display screen, and send the names of students who have violated the rules in key subject classes to the background monitoring system based on the wireless communication module.

[0092] Step SD9: General subject classroom violations and key subject classroom violations are combined to form classroom violations.

[0093] In a specific application, when the class to be tested is studying the key observed subject, the teaching content of the current course of the key observed subject is obtained to obtain the teaching content information of the key observed subject, the teaching content influence coefficient AX is obtained according to the teaching content, the historical teaching seriousness AY of the teacher of the key observed subject of the class to be tested is obtained according to the historical teaching situation of the teacher of the key observed subject of the class to be tested, the classroom seriousness W of the key observed subject of the class to be tested is obtained according to the teaching content influence coefficient AX and the historical teaching seriousness AY, and then the key subject classroom violations are determined, and key subject classroom violation standards are formulated for different classroom seriousnesses, thereby improving the accuracy of human-computer interaction for classroom teaching, and based on the camera, whether there are key subject classroom violations in the key observed subject classroom of the class to be tested, if so, the student is marked as having committed key subject classroom violations, and the name of the student with the key subject classroom violation is displayed on the display screen, so that the teacher of the key observed subject of the class to be tested can remind the student, and the name of the student with the key subject classroom violation is sent to the background monitoring system, so that the head teacher of the class to be tested can understand the key subject classroom situation and facilitate unified management of students.

[0094] Reference Figure 1 In step S3, based on the grade label information, the learning engagement of the labeled students is detected to obtain the target student's learning engagement P. If the target student's learning engagement P does not exceed the preset student class engagement threshold, the student is marked as a characteristic student and sent to the background monitoring system. Step S3 specifically includes the following sub-steps:

[0095] Step S31: Based on the performance label information, students with weak subject type labels or regressing subject type labels are recorded as target students, and the relevant subject types in the performance label information of the target students are recorded as benchmark subjects.

[0096] Step S32: Detecting the number of times the target student answers questions while studying the target subject based on camera, and determining the number of times the target student answers questions. The more times the target student answers questions, the greater the number of times the target student answers questions.

[0097] Step S33: Detect the teacher's response information after the target student answers the question. Based on the teacher's response information, determine the correctness of the target student's answer to obtain the degree of answer accuracy. Based on the degree of answer accuracy, determine the correct answer influence coefficient BH. The higher the target student's degree of answer accuracy, the greater the correct answer influence coefficient BH.

[0098] Step S34: Detect the number of times the target student makes eye contact with the teacher of the benchmark course while studying the benchmark subject, and obtain information on the number of times the target student makes eye contact with the teacher. Based on this information, a learning eye contact impact coefficient (BD) is determined. The greater the information on the number of times the target student makes eye contact with the teacher, the greater the learning eye contact impact coefficient (BD).

[0099] Step S35 , according to the influence coefficient of the number of answers BR, the influence coefficient of the correct answer BH and the influence coefficient of the number of learning interactions BD, the learning input function P=b1×BR+b2×BH+b3×BD is used to calculate the target student's learning input P.

[0100] Step S36: Compare the target student's learning engagement P with a preset student class engagement threshold to obtain a learning engagement comparison result.

[0101] Step S37: Based on the learning engagement comparison result, if the target student's learning engagement P does not exceed the preset student classroom engagement threshold, the target student is marked as a characteristic student.

[0102] Step S38: Send the characteristic students to the background monitoring system based on the wireless communication module.

[0103] In specific applications, the number of times the target student answers questions when studying the benchmark subject course is detected by camera, and the answer frequency influence coefficient BR is obtained. The correctness of the target student's answer to the question is analyzed to obtain the answer correctness influence coefficient BH. The number of times the target student looks at the teacher of the benchmark course when studying the benchmark subject is detected, and the learning eye contact influence coefficient BD is obtained. According to the answer frequency influence coefficient BR, the answer correctness influence coefficient BH and the learning eye contact influence coefficient BD, the target student's learning engagement P is obtained, which improves the detection accuracy of the target student's learning engagement P. The target student's learning engagement P is compared with a preset student classroom engagement threshold. If the target student's learning engagement P does not exceed the preset student classroom engagement threshold, it is marked as a feature student. The feature student is sent to the background monitoring system based on the wireless communication module, which improves the efficiency of human-computer interaction in classroom learning based on computer vision.

[0104] The embodiment of the present application also discloses a human-computer interaction system based on computer vision.

[0105] Reference Figure 2 , a human-computer interaction system based on computer vision includes:

[0106] The grade labeling module is used to label students according to their grades and subject types to obtain grade label information.

[0107] The classroom violation detection and marking module is configured to divide the subject types into general observation subjects and key observation subjects based on the learning direction of the class to be tested. When the class to be tested is studying the general observation subject course, whether the students have violated the rules, if so, they will be marked. When the class to be tested is studying the key observation subject course, the key subject classroom violations are determined according to the teaching content and historical class conditions. Based on the key subject classroom violations, it is detected whether the students have violated the rules, if so, they will be marked.

[0108] The classroom learning engagement analysis module is used to detect the learning engagement of students with labels based on the grade label information to obtain the target student's learning engagement P. If the target student's learning engagement P does not exceed the preset student classroom engagement threshold, the student is marked as a feature student and sent to the background monitoring system.

[0109] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A human-computer interaction method based on computer vision, characterized in that: The following steps are involved: Label students according to their grades and subject types to obtain grade label information; Based on the learning direction of the class to be tested, the subject types are divided into general observation subjects and key observation subjects. When the class to be tested is conducting course projection learning of the general observation subject, whether the students have any violations, if so, they are marked. When the class to be tested is conducting course projection learning of the key observation subject, the key subject classroom violations are determined according to the teaching content and historical class conditions. Based on the key subject classroom violations, whether the students have any violations is detected, if so, they are marked. According to the grade label information, the learning engagement of the labeled students is detected to obtain the target student's learning engagement P. If the target student's learning engagement P does not exceed the preset student classroom engagement threshold, the student is marked as a characteristic student and sent to the background monitoring system.

2. A human-computer interaction method based on computer vision according to claim 1, characterized in that ,The steps of labeling students according to their scores and subject types to obtain the ,score label information include: Number each student in the class to be tested to obtain student number information; Obtain the scores of each student in the class to be tested in real time to obtain real-time score information, store and update the real-time score information in real time, and match the real-time score of each student with the student number; Calculate the average score of each subject in the class to be tested based on the real-time score information to obtain the average score; Compare the scores of each student in each subject in the tested class with the average scores of each subject. If the student's score is lower than the average score, calculate the difference between the student's score and the average score to get the score difference. Set the score difference threshold for each subject based on the full score of each subject's examination paper and the average score; The score difference is compared with the score difference threshold to obtain a score difference comparison result, and based on the score difference comparison result, whether the scores of each student in each subject are weak is judged. If the score difference is greater than or equal to the score difference threshold, it means that the student's subject score is weak, and based on the subject type, the subject type of the student's weak subject score is judged to obtain the weak subject type; Based on the student number information and the weak subject type, students with weak subject performance are labeled with weak subject type.

3. The human-computer interaction method based on computer vision according to claim 2, characterized in that: The step of labeling students according to their scores and subject types to obtain score label information also includes: Based on the real-time score information, the real-time score of each subject type of each student in the tested class is compared with the subject score of the same subject type in the previous time to obtain the subject score comparison result; Based on the subject score comparison result, if the real-time score is less than the last subject score of the same subject type, the difference between the real-time score and the last subject score of the same subject type is calculated to obtain the theoretical score regression difference H1; Detect the test difficulty of each subject type test corresponding to the real-time score to obtain a first test difficulty value K1, and detect the test difficulty of the last test corresponding to the subject score of the same subject type to obtain a second test difficulty value K2; According to the first test difficulty value K1 and the second test difficulty value K2, based on the regression threshold function The theoretical performance regression difference threshold H2 is calculated, where: is the scale factor and is greater than 0; Compare the theoretical score regression difference H1 with the theoretical score regression difference threshold H2. If the theoretical score regression difference H1 is greater than or equal to the theoretical score regression difference threshold H2, it means that the student's subject score has regressed. Based on the subject type, the subject type of the student's subject score regression is determined to obtain the regressed subject type. Based on the student ID information and the subject type of decline, students whose subject performance declines are labeled with the subject type of decline; The weak subject type label and the regressive subject type label are combined to form the performance label information.

4. The human-computer interaction method based on computer vision according to claim 3, characterized in that ,Based on the learning direction of the class to be tested, the subject types are divided into general observation subjects and key observation subjects. When the class to be tested is conducting course projection learning of general observation subjects, whether students have any violations, if so, they are marked. When the class to be tested is conducting course projection learning of key observation subjects, the key subject classroom violations are determined according to the teaching content and historical class conditions. Based on the key subject classroom violations, whether students have any violations, if so, they are marked. The steps specifically include: Based on the learning orientation of the class to be tested, the subject types are divided into general observation subjects and key observation subjects; When the class to be tested is studying a general observation subject, the camera is used to detect whether the student has left his seat. If the student has left his seat, it is determined whether the student left his seat to answer a question. If the student did not leave his seat to answer a question, the length of time the student has been away from his seat is detected to obtain the length of time the student has been away from his seat. Compare the time information of leaving the seat with the preset time threshold of leaving the seat. If the time information of leaving the seat is greater than the preset time threshold of leaving the seat, mark the student who leaves the seat as a general subject class violation; The names of students who have violated the rules in general subject classes are displayed on a display screen, and the names of students who have violated the rules in general subject classes are sent to a background monitoring system based on a wireless communication module.

5. The human-computer interaction method based on computer vision according to claim 4, characterized in that ,Based on the learning direction of the class to be tested, the subject types are divided into general observation subjects and key observation subjects. When the class to be tested conducts the course projection learning of the general observation subject, whether the students have any violations, if so, they are marked. When the class to be tested conducts the course projection learning of the key observation subject, the key subject classroom violations are determined according to the teaching content and the historical class situation. Based on the key subject classroom violations, whether the students have any violations, if so, they are marked, and the steps also include: When the class to be tested is studying the course of the key observed subject, the teaching content of the current course of the key observed subject is obtained to obtain the teaching content information of the key observed subject; Based on the teaching content information of the key observed subject, determine the proportion of the content in the teaching content that needs to be discussed by students to the teaching content information of the key observed subject, obtain the teaching content discussion ratio, and obtain the teaching content influence coefficient AX based on the teaching content discussion ratio; Obtaining the historical teaching situation of the teachers of the key observed subjects of the tested class, wherein the historical teaching situation includes the duration of historical class discussions and the intensity of historical class discussions; According to the length of the history class discussion and the intensity of the history class discussion, judge the seriousness of the history teaching of the key observation subject teachers in the tested class AY; According to the teaching content influence coefficient AX and the history teaching seriousness AY, the classroom seriousness W of the key observed subject of the class to be tested is calculated based on the classroom seriousness function W = a1 × AX + a2 × AY; Based on the teaching content and classroom seriousness W, the classroom violations of the key observed subjects of the class to be tested are determined to obtain the classroom violations of the key subjects; Based on the camera, detect whether there are any key subject classroom violations in the key subject classroom of the tested class. If so, mark the key subject classroom violations for the students who have committed the key subject classroom violations. The names of students who have violated the rules in key subject classes are displayed on the display screen, and the names of students who have violated the rules in key subject classes are sent to the background monitoring system based on the wireless communication module; The general subject classroom violations and the key subject classroom violations are combined to form classroom violations.

6. The human-computer interaction method based on computer vision according to claim 5, characterized in that ,According to the grade label information, the learning engagement of the students with labels is detected to obtain the learning engagement P of the target students. If the learning engagement P of the target students does not exceed the preset student classroom engagement threshold, they are marked as characteristic students and the characteristic students are sent to the background monitoring system. The steps specifically include: According to the score label information, students with weak subject type labels or regressing subject type labels are recorded as target students, and the relevant subject types in the score label information of the target students are recorded as benchmark subjects; Based on the camera detection, the target students answer questions when they are studying the benchmark subject, and determine the answer frequency influence coefficient BR based on the target students’ answer times; Detect the teacher's response information after the target student answers the question, judge the correctness of the target student's answer to the question based on the teacher's response information, and obtain the correctness of the answer, and determine the correctness influence coefficient BH based on the correctness of the answer; The number of times the target students look at the teacher of the benchmark course when studying the benchmark subject is detected to obtain the information of the number of times the teacher and the student look at each other, and the learning eye contact influence coefficient BD is determined based on the information of the number of times the teacher and the student look at each other; According to the influence coefficient of answer times BR, the influence coefficient of correct answer BH and the influence coefficient of learning eye contact times BD, the learning engagement degree P of the target student is calculated based on the learning engagement function P = b1×BR+b2×BH+b3×BD; Compare the target student's learning engagement P with the preset student classroom engagement threshold to obtain a learning engagement comparison result; Based on the comparison results of learning engagement, if the target student’s learning engagement P does not exceed the preset student class engagement threshold, then the target student is marked as a characteristic student; Based on the wireless communication module, the characteristic students are sent to the background monitoring system.

7. A human-computer interaction system based on computer vision, characterized in that: The computer vision-based human-computer interaction system is used to implement the computer vision-based human-computer interaction method described in any one of claims 1 to 6, including: The grade labeling module is used to label students according to their grades and subject types to obtain grade label information; The classroom violation detection and marking module is configured to divide the subject types into general observation subjects and key observation subjects based on the learning direction of the class to be tested, detect whether the students have violated the rules when the class to be tested is studying the general observation subjects, and mark them if they do; when the class to be tested is studying the key observation subjects, determine the key subject classroom violations based on the teaching content and the historical class situation, and detect whether the students have violated the rules based on the key subject classroom violations, and mark them if they do; The classroom learning engagement analysis module is used to detect the learning engagement of students with labels based on the grade label information to obtain the target student's learning engagement P. If the target student's learning engagement P does not exceed the preset student classroom engagement threshold, the student is marked as a characteristic student and sent to the background monitoring system.

Images