Classroom behavior display method, device, storage medium and system

Abstract

The invention relates to a classroom behavior display method and device, a storage medium and a system, and the method comprises the steps: obtaining a classroom video, the position information of tables and chairs in a classroom, and face images of students, determining the corresponding relation between the face images and the position information of the tables and chairs, recognizing the classroom behavior information in the classroom video and the face images corresponding to the classroom behaviors, and displaying the classroom behaviors. The method comprises the steps of establishing a corresponding relation among classroom behavior information, face images and desk and chair position information, and obtaining and displaying a classroom behavior display graph of faces and classroom behaviors corresponding to desk and chair positions. The face images and the classroom behavior identifiers of the students are orderly distributed according to the desk and chair positions of the students, the displayed information does not interfere with each other, and the watching experience of the user can be improved.

Description

Technical Field

[0001] The present invention relates to the field of classroom behavior analysis, and in particular to a classroom behavior display method, device, storage medium and system. Background Art

[0002] Classroom behavior refers to the teacher's teaching behavior and the student's interactive behavior during class, such as students raising their hands or going to the podium to share. In the prior art, when identifying classroom behavior, artificial intelligence algorithms are usually used to analyze classroom videos, and the time period when the classroom behavior occurs is obtained to obtain video key frames in the classroom video. Based on the video key frames, classroom behavior pictures are obtained, and the classroom behavior pictures are displayed on the front end for users to view.

[0003] However, the classroom behavior pictures displayed in the above manner are prone to blocking each other, affecting the user's viewing experience. Summary of the invention

[0004] The embodiments of the present application provide a classroom behavior display method, device, storage medium and system, which can avoid the problem of mutual obstruction of classroom behavior information display and improve the classroom behavior display effect.

[0005] In a first aspect, an embodiment of the present application provides a classroom behavior display method, comprising the following steps:

[0006] Obtain classroom videos, classroom desk and chair location information, and students’ facial images;

[0007] Determine the correspondence between the face image and the table and chair position information; wherein the correspondence between the face image and the table and chair position information is used to display the face image at a first target position corresponding to the table and chair position information;

[0008] Identify classroom behavior information in classroom videos, establish the correspondence between classroom behavior information, face images, and table and chair position information, and obtain classroom behavior display diagrams;

[0009] Display the classroom behavior display diagram; wherein the classroom behavior display diagram includes a face image and a classroom behavior identifier; the classroom behavior identifier is generated according to the classroom behavior information, the face image is displayed at a first target position corresponding to the student's desk and chair position, and the classroom behavior identifier is displayed at a second target position corresponding to the student's desk and chair position.

[0010] In a second aspect, an embodiment of the present application provides a classroom behavior display device, the device comprising:

[0011] Classroom information acquisition module, used to obtain classroom videos, classroom desk and chair location information and students' face images;

[0012] A correspondence determination module, used to determine the correspondence between the face image and the table and chair position information; wherein the correspondence between the face image and the table and chair position information is used to display the face image at a first target position corresponding to the table and chair position information;

[0013] A display image acquisition module is used to identify classroom behavior information in classroom videos, establish a corresponding relationship between classroom behavior information, face images, and table and chair position information, and obtain a classroom behavior display image;

[0014] A display module is used to display the classroom behavior display diagram; wherein the classroom behavior display diagram includes a face image and a classroom behavior identifier; the classroom behavior identifier is used to determine whether the student has classroom behavior, the face image is displayed at a first target position corresponding to the student's desk and chair position, and the classroom behavior identifier is displayed at a second target position corresponding to the student's desk and chair position.

[0015] In a third aspect, an embodiment of the present application provides a classroom behavior display system, including a server, an educational system, a video acquisition device, and an electronic device; the server, the educational system, the video acquisition device, and the electronic device are communicatively connected;

[0016] The electronic device comprises a memory, a processor, and a computer program stored in the memory and executable by the processor;

[0017] When the processor executes the computer program, the steps of implementing any one of the above-mentioned classroom behavior display methods are implemented.

[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the classroom behavior display method as described in any one of the above.

[0019] In an embodiment of the present application, by obtaining classroom videos, classroom desk and chair position information and students' facial images, the correspondence between facial images and desk and chair position information is determined, classroom behavior information in classroom videos and facial images corresponding to the classroom behaviors are identified, a correspondence between classroom behavior information, facial images and desk and chair position information is established, and a classroom behavior display diagram of faces and classroom behaviors corresponding to desk and chair positions is obtained. Compared with the prior art, the classroom behavior display diagram generated by the present application displays students' facial images and classroom behaviors in an orderly manner according to the students' desk and chair positions, and the displayed information will not interfere with each other, thereby improving the user's viewing experience.

[0020] For better understanding and implementation, the present invention is described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of an application scenario of a classroom behavior display method in one embodiment of the present invention;

[0022] Figure 2 A flow chart of a classroom behavior display method in one embodiment of the present invention;

[0023] Figure 3 A schematic diagram of the distribution of face image quality and position in another embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a display interface in one embodiment of the present invention;

[0025] Figure 5 A schematic diagram of classroom behavior information in one embodiment of the present invention;

[0026] Figure 6 A schematic diagram of a classroom behavior display diagram in one embodiment of the present invention;

[0027] Figure 7 It is a structural schematic diagram of a classroom behavior display device in one embodiment of the present invention;

[0028] Figure 8 It is a structural diagram of a classroom behavior display system in one embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the objectives, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0030] It should be clear that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0031] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the embodiments of the present application. The singular forms of "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0032] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims. In the description of the present application, it should be understood that the terms "first", "second", "third", etc. are only used to distinguish similar objects, and do not have to be used to describe a specific order or sequence, nor can they be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0033] In addition, in the description of this application, unless otherwise specified, "several" means two or more. "And / or" describes the corresponding relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0034] The classroom behavior display method of this application can be applied in Figure 1 In the application scenario shown, Figure 1 As shown, the application scenario includes a server 110, an educational system 120, a video acquisition device 130, and a classroom behavior display device 140;

[0035] The server 110 includes an IoT service 111 , a real-time audio and video streaming service 112 , and a device management service 113 ;

[0036] The classroom behavior display device 140 is respectively communicated with the real-time audio and video streaming service 112, the equipment management service 113, and the teaching system 120; the equipment management service 113 is respectively communicated with the IoT service 111 and the teaching system 120, and the real-time audio and video streaming service 112 is communicated with the video acquisition device 130.

[0037] IoT service 111 is mainly responsible for connecting video acquisition devices (student cameras, microphones, teacher cameras and other smart terminal devices) to the Internet of Things platform, and is responsible for device access and signaling transmission, and management and control of devices.

[0038] The real-time audio and video streaming service 112 is used to receive the audio and video streams pushed by the video acquisition device and record and store them.

[0039] The device management service 113 is used to bind the device to the classroom space.

[0040] The educational affairs system 120 may store educational affairs information such as the school's classroom space, student information, teacher information, student portraits, and course schedules entered by teachers or administrators of the educational affairs office.

[0041] The video acquisition device 130 may include a student camera, a microphone, a teacher camera, and other devices for acquiring classroom videos.

[0042] The classroom behavior display device 140 can be used to execute the classroom behavior display method of the present application.

[0043] See also Figure 2 , the present application provides a classroom behavior display method, the method comprising the following steps:

[0044] S110: Acquire classroom videos, classroom desk and chair location information, and students’ face images;

[0045] The classroom video may be a teacher's teaching video, and the classroom video may include the desks and chairs in the classroom and the images of students. The classroom video may be acquired by a video acquisition device arranged in the classroom and used to acquire a panoramic image of the classroom, and the panoramic image of the classroom may include all the desks and chairs in the classroom and all the students.

[0046] The table and chair position information is used to determine the position of the tables and chairs in the classroom. Specifically, the table and chair position information can be input in advance by the user, or the existing table and chair detection algorithm can be used to detect the tables and chairs in the classroom video, and the visual depth estimation algorithm can be used to obtain the depth information of the tables and chairs, so as to determine the position of the tables and chairs in the image according to the depth information. The classroom video can be a classroom video collected when the teacher is teaching, or a classroom video collected when no one is present.

[0047] To obtain the student's face image, key frames can be obtained from the classroom video at a set interval (e.g., 5 seconds), and face recognition detection and depth estimation algorithm analysis can be performed on the key frames to obtain the student's face image and the student's position, such as Figure 3 As shown, when there is at least one face image at the same position, the face image with the highest image quality can be selected as the face image corresponding to the position.

[0048] The face images are face images of students in the classroom. When acquiring the face images, an existing face recognition algorithm can be used to perform face recognition from classroom videos to acquire the face images of the students.

[0049] S130: Determine the correspondence between the face image and the table and chair position information; wherein the correspondence between the face image and the table and chair position information is used to display the face image at a first target position corresponding to the table and chair position information;

[0050] The correspondence between the facial image and the table and chair position information is used to determine the position where the facial image is displayed. The correspondence between the facial image and the table and chair position information can be pre-input by users such as teachers and administrators; or, in another embodiment, the correspondence between the facial image and the table and chair position information can be determined based on the position of the facial image and the table and chair position information. Specifically, when the position of a facial image is the same or similar to the table and chair position information of a table and chair, the correspondence between the facial image and the table and chair position information can be established, so that the facial image is displayed at the corresponding position of the table and chair.

[0051] The first target position can be set according to user needs, such as Figure 4 As shown, the classroom video screen includes a table 110 and students' facial images 120 , wherein each student's facial image 120 is superimposed and displayed at the position of the corresponding table 110 , for example, the facial image 120 is displayed on the desktop of the table 110 .

[0052] S140: identifying classroom behavior information in the classroom video, establishing a correspondence between the classroom behavior information, face images, and table and chair position information, and obtaining a classroom behavior display diagram;

[0053] Classroom behavior information is used to determine students' classroom behavior, which can be teaching interactive behaviors such as students sharing on stage, raising hands, asking questions, answering questions, and discussing.

[0054] Specifically, a classroom behavior recognition model can be used to identify classroom behaviors in classroom videos. The classroom behavior recognition model is used to identify students' classroom behaviors in classroom videos. The classroom behavior recognition model can be a model constructed based on an existing behavior recognition algorithm. In this application, users can collect a large number of video clips of students performing various classroom behaviors through the Internet in advance as training samples, and use the training samples to pre-train the behavior recognition model to obtain a classroom behavior recognition model that can recognize student behaviors in classroom video clips.

[0055] Classroom behavior information may include information about students who perform classroom behaviors and classroom behavior location information. The student information may be information such as student facial information or student identity identification information that can be used to determine the identity of the student who performs the classroom behavior. The classroom behavior location information is used to determine the location where the classroom behavior occurs. When establishing a correspondence between classroom behavior information, facial images, and desk and chair location information, the student who performs the classroom behavior can be determined based on the student information, and the location where the classroom behavior occurs can be determined based on the classroom behavior location information, thereby corresponding the classroom behavior information to the student facial images and the desk and chair locations, and establishing a correspondence between the classroom behavior information, facial images, and desk and chair location information.

[0056] In this application, a student can correspond to a facial image, a desk and chair position, and multiple classroom behaviors. The correspondence between classroom behavior information, facial images, and desk and chair position information can be bound in the form of a matrix diagram to improve data accuracy.

[0057] The classroom behavior display diagram can be used to show the positions of students' desks and chairs, students' facial images, and students' classroom behavior during class.

[0058] S150: Displaying the classroom behavior display diagram; wherein the classroom behavior display diagram includes a face image and a classroom behavior identifier; the classroom behavior identifier is generated according to the classroom behavior information, the face image is displayed at a first target position corresponding to the desk and chair position of the student, and the classroom behavior identifier is displayed at a second target position corresponding to the desk and chair position of the student.

[0059] Classroom behavior display charts can be used for teachers to review their teaching behaviors, etc., and can also be used by teaching evaluators to observe and evaluate teachers' classroom teaching quality.

[0060] Classroom behavior markers can be used to identify classroom behaviors performed by students during class.

[0061] The first target position and the second target position may be two different positions near tables and chairs, and the first target position and the second target position may be specifically set according to the size and shape of the facial image and classroom behavior logo to be displayed or the viewing requirements of the user.

[0062] In the classroom behavior display diagram of the present application, the face image is displayed at the first target position corresponding to the position of the student's desk and chair, and the classroom behavior mark is displayed at the second target position corresponding to the position of the student's desk and chair. The face image and classroom behavior mark of each student are distributed in order according to the position of the student's desk and chair. The face image and classroom behavior mark of each student will not block each other, which is convenient for users to understand students' classroom behavior by viewing the classroom behavior display diagram, thereby improving user experience.

[0063] In an embodiment of the present application, by obtaining classroom videos, classroom desk and chair position information and students' facial images, the correspondence between facial images and desk and chair position information is determined, classroom behavior information in classroom videos and facial images corresponding to the classroom behaviors are identified, a correspondence between classroom behavior information, facial images and desk and chair position information is established, and a classroom behavior display diagram of faces and classroom behaviors corresponding to desk and chair positions is obtained and displayed. Compared with the prior art, when displaying students' facial images and classroom behaviors, the classroom behavior display diagram generated by the present application has students' facial images and classroom behaviors distributed in an orderly manner according to the students' desk and chair positions, and the displayed information will not interfere with each other, thereby improving the user's viewing experience.

[0064] In one embodiment, the following steps are also included:

[0065] In response to a triggering operation on the face image of the classroom behavior display diagram, the classroom behavior information of the student is displayed at a third target position corresponding to the position of the face image.

[0066] like Figure 5 As shown, classroom behavior information includes the number of times students go on stage, the number of times they raise their hands, and the number of times they ask and answer questions.

[0067] The third target position may be a position near the face image. In an embodiment of the present application, the third target position may specifically be on the right side of the face image. The student's classroom behavior information is displayed on the right side of the face image. In other embodiments, the student's classroom behavior information may also be displayed on the left side of the face image or at other positions.

[0068] In one embodiment, the step of displaying the classroom behavior display diagram specifically includes:

[0069] Determining whether the student has any classroom behavior according to the classroom behavior information; wherein the classroom behavior includes raising hands, going on stage, and asking questions;

[0070] For students who have classroom behavior, the student's face image is displayed at a first target position corresponding to the student's desk and chair position, and the classroom behavior mark is displayed at a second target position corresponding to the desk and chair position;

[0071] For students who do not have any classroom behavior, the student's facial image is displayed at the first target position corresponding to the student's desk and chair position, and the classroom behavior mark is not displayed at the second target position corresponding to the desk and chair position.

[0072] A student without classroom behavior may refer to a student whose classroom behaviors such as raising hands, going on stage, and asking questions are not identified in the classroom video. When a student exhibits any of the classroom behaviors such as raising hands, going on stage, and asking questions, it is determined that the student has classroom behavior.

[0073] In the present application, for students who have classroom behavior, the corresponding classroom behavior logo is displayed on the second target position corresponding to the desk and chair position. For students who do not have classroom behavior, the classroom behavior logo is not displayed on the second target position corresponding to the desk and chair position. Therefore, the user can quickly identify whether the student has performed classroom behavior during class by observing whether the classroom behavior logo is displayed at the second target position.

[0074] The shape, pattern, display position and display method of classroom behavior signs can be set according to user needs.

[0075] In a preferred embodiment, classroom behavior identifiers can also be set according to students' specific classroom behaviors. For example, for hand-raising behavior, a hand-raising classroom behavior identifier corresponding to the hand-raising behavior can be displayed at the second target position; for on-stage behavior, a on-stage classroom behavior identifier corresponding to the hand-raising behavior can be displayed at the second target position, etc., so that it is more convenient for users to know the classroom behaviors performed by students during class.

[0076] like Figure 6 As shown, it is a schematic diagram of a classroom behavior display diagram in an embodiment of the present application. In this embodiment, the classroom behavior display diagram includes students' facial images, tables and chairs, and classroom behavior labels. The students' facial images are displayed on the desktop, and the classroom behavior labels can be displayed in the lower right corner of the students' corresponding facial images. For students who do not have classroom behavior, the classroom behavior label is not displayed in the lower right corner of the students' corresponding facial images.

[0077] Alternatively, in another embodiment, two different classroom behavior identifiers may be set to

[0078] Specifically, the step of displaying the classroom behavior display diagram specifically includes:

[0079] Determining whether the student has any classroom behavior according to the classroom behavior information; wherein the classroom behavior includes raising hands, going on stage, and asking questions;

[0080] For students who have classroom behaviors, the student's face image is displayed at a first target position corresponding to the student's desk and chair position, and a first classroom behavior identifier is displayed at a second target position corresponding to the desk and chair position;

[0081] For students who do not have classroom behavior, the student's face image is displayed at the first target position corresponding to the student's desk and chair position, and the second classroom behavior mark is displayed at the second target position corresponding to the desk and chair position;

[0082] Among them, the first classroom behavior identifier is different from the second classroom behavior identifier, the first classroom behavior identifier and the second classroom behavior identifier can be different shapes, or the first classroom behavior identifier and the second classroom behavior identifier can be different patterns, or the first classroom behavior identifier and the second classroom behavior identifier can be displayed in different ways, etc., which are not specifically limited here.

[0083] In the present application, for students who have classroom behavior, the first classroom behavior identifier is displayed at the second target position corresponding to the desk and chair position; for students who have no classroom behavior, the second classroom behavior identifier is not displayed at the second target position corresponding to the desk and chair position. Thus, the user can quickly identify whether the student has performed classroom behavior during class by observing that the second target position is the first classroom behavior identifier or the second classroom behavior identifier.

[0084] In one embodiment, the step of acquiring the student image specifically includes:

[0085] Performing face recognition on the classroom video at set time intervals to obtain at least one face image of each student;

[0086] Specifically, existing face recognition algorithms can be used to perform face recognition on key frames of classroom videos.

[0087] Perform image quality analysis on the at least one facial image to obtain a facial image with the highest image quality, and use the facial image with the highest image quality as the facial image of the student.

[0088] Specifically, the image quality of the face image can be analyzed using an existing image quality analysis algorithm.

[0089] Preferably, when selecting the facial image with the highest image quality, facial images at the same position may be compared based on the position information of the facial image, and the facial image with the highest image quality at the same position may be selected as the facial image of the student.

[0090] In one embodiment, after obtaining the face image with the highest image quality, the following steps are also included:

[0091] If the image quality of the facial image with the highest image quality is still less than the image quality threshold, the facial image of the student is obtained from the educational system, and the facial image obtained from the educational system is used as the facial image of the student.

[0092] The image quality may be clarity, and the image quality threshold may be a clarity threshold.

[0093] The educational affairs system includes educational affairs information such as school classroom space, student information, teacher information, student images, and course schedules entered by school users.

[0094] When the image quality of the facial image with the highest image quality is still less than the image quality threshold, indicating that no clear facial image has been found in the entire class, the teaching affairs system will use the facial recognition similarity algorithm to obtain a clear facial image of the student from the student portrait information, thereby ensuring the image quality of the student images displayed in the classroom behavior display diagram and improving the user viewing experience.

[0095] In one embodiment, the following steps are also included:

[0096] Based on the classroom behavior display diagram, generating a classroom behavior observation report;

[0097] The classroom behavior observation report is displayed.

[0098] The classroom behavior observation report may include information such as classroom behavior display diagrams and classroom behavior statistics, where the classroom behavior statistics can be obtained based on the classroom behavior statistics of each student based on the classroom behavior information. The classroom behavior statistics may include classroom behavior, facial images, student positions and the correspondence between classroom behavior, facial images and student positions.

[0099] The classroom behavior observation report can be displayed on the classroom behavior display device, or, in another embodiment, the classroom behavior observation report can also be displayed on a user terminal that is communicatively connected to the classroom behavior display device. The user terminal can be a terminal where a teacher, teaching evaluator, or other user who needs to view the classroom behavior observation report is located.

[0100] The classroom behavior observation report can reflect the interactive behavior between students and teachers in the classroom. Teachers can review their teaching behaviors during class by watching the classroom behavior observation report. Teaching evaluators can evaluate the teaching quality of teachers by watching the classroom behavior observation report.

[0101] like Figure 7 As shown, the embodiment of the present application also provides a classroom behavior display device, the device comprising:

[0102] The classroom information acquisition module 310 is used to acquire classroom videos, classroom desk and chair location information, and students' face images;

[0103] The correspondence determination module 320 is used to determine the correspondence between the face image and the table and chair position information; wherein the correspondence between the face image and the table and chair position information is used to display the face image at a first target position corresponding to the table and chair position information;

[0104] The display image acquisition module 330 is used to identify classroom behavior information in the classroom video, establish a correspondence between the classroom behavior information, face images and table and chair position information, and obtain a classroom behavior display image;

[0105] The display module 340 is used to display the classroom behavior display diagram; wherein the classroom behavior display diagram includes a face image and a classroom behavior identifier; the classroom behavior identifier is used to determine whether the student has classroom behavior, the face image is displayed at a first target position corresponding to the student's desk and chair position, and the classroom behavior identifier is displayed at a second target position corresponding to the student's desk and chair position.

[0106] In one embodiment, the classroom behavior display device further includes:

[0107] The classroom behavior information display module is used to display the student's classroom behavior information at a third target position corresponding to the position of the face image in response to a trigger operation on the face image of the classroom behavior display diagram.

[0108] In one embodiment, the display module 340 includes:

[0109] A first classroom behavior determination unit is used to determine whether the student has classroom behavior according to the classroom behavior information; wherein the classroom behavior includes raising hands, going on stage and asking questions;

[0110] A first display unit is used to display a face image of a student who has classroom behavior at a first target position corresponding to the position of the student's desk and chair, and to display the classroom behavior mark at a second target position corresponding to the position of the desk and chair;

[0111] The second display unit is used to display the student's face image at a first target position corresponding to the student's desk and chair position for a student who has no classroom behavior, and not to display the classroom behavior mark at a second target position corresponding to the desk and chair position.

[0112] Alternatively, in another embodiment, the display module 340 includes:

[0113] A second classroom behavior determination unit is used to determine whether the student has classroom behavior according to the classroom behavior information; wherein the classroom behavior includes raising hands, going on stage and asking questions;

[0114] A third display unit is used to display, for a student who has classroom behavior, a face image of the student at a first target position corresponding to the position of the student's desk and chair, and to display a first classroom behavior identifier at a second target position corresponding to the position of the desk and chair;

[0115] The fourth display unit is used to display the student's facial image at a first target position corresponding to the student's desk and chair position, and to display a second classroom behavior identifier at a second target position corresponding to the desk and chair position, for a student who has no classroom behavior; wherein the first classroom behavior identifier is different from the second classroom behavior identifier.

[0116] In one embodiment, the classroom information acquisition module 310 includes:

[0117] A face image acquisition unit, used to perform face recognition on the classroom video at set time intervals to acquire at least one face image of each student;

[0118] The facial image quality analysis unit is used to perform image quality analysis on the at least one facial image, obtain the facial image with the highest image quality, and use the facial image with the highest image quality as the facial image of the student.

[0119] In one embodiment, the classroom information acquisition module 310 includes:

[0120] The facial image acquisition unit is used to acquire the facial image of the student from the educational system if the image quality of the facial image with the highest image quality is still less than the image quality threshold, and use the facial image acquired from the educational system as the facial image of the student.

[0121] In one embodiment, the classroom behavior display device further includes:

[0122] An observation report generating module, used to generate a classroom behavior observation report based on the classroom behavior display diagram;

[0123] The observation report display module is used to display the classroom behavior observation report.

[0124] It should be noted that when the classroom behavior display device provided in the above embodiment executes the classroom behavior display method, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the classroom behavior display device provided in the above embodiment and the classroom behavior display method in the above embodiment belong to the same concept. The implementation process is detailed in the method embodiment, which will not be repeated here.

[0125] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the classroom behavior display method as described in any one of the above.

[0126] The present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0127] like Figure 8 As shown, the present application also provides a classroom behavior display system, including a server 410, an educational system 420, a video acquisition device 430 and an electronic device 440 as described above;

[0128] The server 410 includes an IoT service 411 , a real-time audio and video streaming service 412 , and a device management service 413 ;

[0129] The classroom behavior display device 300 is respectively communicated with the real-time audio and video streaming service 412, the equipment management service 413, and the teaching system 420; the equipment management service 413 is respectively communicated with the IoT service 411 and the teaching system 420, and the real-time audio and video streaming service 412 is communicated with the video acquisition device 430.

[0130] IoT Service 411: Mainly responsible for connecting video acquisition devices (student cameras, microphones, teacher cameras and other smart terminal devices) to the Internet of Things platform, responsible for device access and signaling transmission, and management and control of devices.

[0131] The real-time audio and video streaming service 412 is used to receive the audio and video streams pushed by the video acquisition device and record and store them.

[0132] The device management service 413 is used to bind the device to the school classroom space.

[0133] The educational affairs system 420 may store educational affairs information such as the school's classroom space, student information, teacher information, student portraits, and course schedules entered by teachers or administrators of the educational affairs office.

[0134] The video acquisition device 430 includes a student camera, a microphone, a teacher camera, etc.

[0135] The electronic device 440 includes a memory, a processor, and a computer program stored in the memory and executable by the processor;

[0136] When the processor executes the computer program, the steps of the classroom behavior display method as described in any one of the above items are implemented.

[0137] The memory includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electronically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0138] The processor is the control core (Control Unit) of the electronic device, which uses various interfaces and lines to connect the various components of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing programs or modules stored in the memory, and calling data stored in the memory. For example, when the processor executes the computer program stored in the memory, it implements all or part of the steps of the classroom behavior display method described in the embodiment of the present application; or implements all or part of the functions of the classroom behavior display device. The processor can be composed of an integrated circuit, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same or different functions, including one or more central processing units (CPU), microprocessors, digital processing chips, graphics processors and various control chips. Combination of chips, etc.

[0139] This application uses visual desk and chair detection algorithms and depth estimation to construct a classroom distribution map. By obtaining key frames at specified intervals in the classroom video, face detection algorithms, depth estimation and face quality detection are performed on the key frames of the classroom video to obtain a distribution map corresponding to the positions of desks and chairs and the clearest student face portraits. The teaching behavior-face information-student position is identified through a behavior recognition algorithm, and behavior aggregation is performed to facilitate the monitoring of classroom behavior data.

[0140] The present invention is not limited to the above-mentioned embodiments. If various changes or modifications to the present invention do not depart from the spirit and scope of the present invention, and if these changes and modifications fall within the scope of the claims and equivalent technologies of the present invention, the present invention is also intended to include these changes and modifications.

Claims

1. A classroom behavior display method, characterized in that: The following steps are involved: Obtain classroom videos, classroom desk and chair location information, and students’ facial images; Determine the correspondence between the face image and the table and chair position information; wherein the correspondence between the face image and the table and chair position information is used to display the face image at a first target position corresponding to the table and chair position information; Identify classroom behavior information in classroom videos, establish the correspondence between classroom behavior information, face images, and table and chair position information, and obtain classroom behavior display diagrams; Display the classroom behavior display diagram; wherein the classroom behavior display diagram includes a face image and a classroom behavior identifier; the classroom behavior identifier is generated according to the classroom behavior information, the face image is displayed at a first target position corresponding to the student's desk and chair position, and the classroom behavior identifier is displayed at a second target position corresponding to the student's desk and chair position.

2. The classroom behavior display method according to claim 1, characterized in that: The following steps are also included: In response to a triggering operation on the face image of the classroom behavior display diagram, the classroom behavior information of the student is displayed at a third target position corresponding to the position of the face image.

3. The classroom behavior display method according to claim 1, characterized in that: The steps of displaying the classroom behavior display diagram specifically include: Determining whether the student has any classroom behavior according to the classroom behavior information; wherein the classroom behavior includes raising hands, going on stage, and asking questions; For students who have classroom behavior, the student's face image is displayed at a first target position corresponding to the student's desk and chair position, and the classroom behavior mark is displayed at a second target position corresponding to the desk and chair position; For students who do not have any classroom behavior, the student's facial image is displayed at the first target position corresponding to the student's desk and chair position, and the classroom behavior mark is not displayed at the second target position corresponding to the desk and chair position.

4. The classroom behavior display method according to claim 1, characterized in that: The steps of displaying the classroom behavior display diagram specifically include: Determining whether the student has any classroom behavior according to the classroom behavior information; wherein the classroom behavior includes raising hands, going on stage, and asking questions; For students who have classroom behaviors, the student's face image is displayed at a first target position corresponding to the student's desk and chair position, and a first classroom behavior mark is displayed at a second target position corresponding to the desk and chair position; For students who have no classroom behavior, the student's facial image is displayed at the first target position corresponding to the student's desk and chair position, and the second classroom behavior identifier is displayed at the second target position corresponding to the desk and chair position; wherein the first classroom behavior identifier is different from the second classroom behavior identifier.

5. The classroom behavior display method according to claim 1, characterized in that: The steps of obtaining the student's face image specifically include: Performing face recognition on the classroom video at set time intervals to obtain at least one face image of each student; Perform image quality analysis on the at least one facial image to obtain a facial image with the highest image quality, and use the facial image with the highest image quality as the facial image of the student.

6. The classroom behavior display method according to claim 5, characterized in that: After obtaining the face image with the highest image quality, the following steps are also included: If the image quality of the facial image with the highest image quality is still less than the image quality threshold, the facial image of the student is obtained from the educational system, and the facial image obtained from the educational system is used as the facial image of the student.

7. The classroom behavior display method according to claim 1, characterized in that: The following steps are also included: Based on the classroom behavior display diagram, generating a classroom behavior observation report; The classroom behavior observation report is displayed.

8. A classroom behavior display device, characterized in that: The device comprises: Classroom information acquisition module, used to obtain classroom videos, classroom desk and chair location information and students' face images; A correspondence determination module, used to determine the correspondence between the face image and the table and chair position information; wherein the correspondence between the face image and the table and chair position information is used to display the face image at a first target position corresponding to the table and chair position information; A display image acquisition module is used to identify classroom behavior information in classroom videos, establish a corresponding relationship between classroom behavior information, face images, and table and chair position information, and obtain a classroom behavior display image; A display module is used to display the classroom behavior display diagram; wherein the classroom behavior display diagram includes a face image and a classroom behavior identifier; the classroom behavior identifier is generated based on the classroom behavior information, the face image is displayed at a first target position corresponding to the student's desk and chair position, and the classroom behavior identifier is displayed at a second target position corresponding to the student's desk and chair position.

9. A classroom behavior display system, characterized in that: It includes a server, an educational system, a video acquisition device and an electronic device; the server, the educational system, the video acquisition device and the electronic device are communicatively connected; The electronic device comprises a memory, a processor, and a computer program stored in the memory and executable by the processor; When the processor executes the computer program, the steps of the classroom behavior display method as described in any one of claims 1-7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the classroom behavior display method as described in any one of claims 1 to 7 are implemented.

Images