Student classroom behavior expression recognition system based on image recognition

Abstract

The invention discloses a student classroom behavior expression recognition system based on image recognition, and the system comprises an image collection module, a data processing module, an expression recognition module, a behavior recognition module, a learning condition analysis module, and a real-time feedback module, can comprehensively collect student classroom image data, accurately captures the behaviors and expressions of students from different perspectives, and improves the recognition efficiency. The data preprocessing module guarantees the quality of a data set, improves the generalization ability of the recognition system, and improves the recognition accuracy of student behaviors and expressions through the improvement of Resnet18 and Mobilene-V2 networks, thereby providing real-time and accurate classroom feedback information for teachers, facilitating the optimization of teaching decisions, improving the teaching quality, and improving the teaching efficiency. The data security module ensures the security of student data and protects the privacy of students, the system optimization module continuously improves the system performance and adapts to different teaching requirements and environments, and the multi-modal data fusion module provides more comprehensive student learning state insight and provides a more scientific basis for teaching decision making.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a student classroom behavior expression recognition system based on image recognition. Background Art

[0002] At present, the traditional classroom model is gradually shifting towards intelligence and digitization, and smart classrooms have emerged. In smart classrooms, accurately identifying students' behaviors and expressions is of great significance for fully grasping the learning situation and optimizing teaching decisions. However, traditional classroom observation methods have limitations, and it is difficult to fully and accurately grasp the learning status of each student.

[0003] Therefore, a student classroom behavior expression recognition system based on image recognition is proposed. Summary of the invention

[0004] In response to the above problems, the present invention provides a student classroom behavior and expression recognition system based on image recognition, which uses image recognition technology to accurately capture and analyze students' behaviors and expressions in the classroom, and provide teachers with real-time and accurate learning feedback to optimize teaching decisions and personalized teaching, so as to solve the problems raised in the above background technology.

[0005] The technical solution of the present invention is:

[0006] A student classroom behavior and expression recognition system based on image recognition includes an image acquisition module, a data processing module, an expression recognition module, a behavior recognition module, a learning situation analysis module, a real-time feedback module, a data security module, a system optimization module and a multimodal data fusion module.

[0007] The working principle of the above technical solution is as follows:

[0008] The data processing module extracts video frames, removes video privacy information, deletes useless perspectives and information, and divides and labels the collected images. At the same time, it converts the video data into a format that can be used for image recognition and removes privacy information in the video. The processed image data is transmitted to the expression recognition module based on Resnet18 and the behavior recognition module based on Mobilenet-V2. The expression recognition module performs preliminary feature extraction on the input image, fine processing of residual blocks in multiple stages, and finally average pooling layer aggregation and fully connected layer output classification results to achieve accurate recognition of students' expressions. The behavior recognition module adopts an inverse residual structure, a linear bottleneck structure and Deep separable convolution achieves efficient recognition of student behavior. The expression recognition module and behavior recognition module transmit the recognition results to the learning situation analysis module. The learning situation analysis module analyzes students' learning status, emotional state, participation and other information, and establishes an individual learning feature model of students. The learning situation analysis results are fed back to teachers in real time through the real-time feedback module so that teachers can adjust their teaching strategies. The data security module adopts data encryption technology and security management measures to ensure data security. The system optimization module continuously optimizes and upgrades the system to improve recognition accuracy and speed. The multimodal data fusion module combines students' expression analysis with physiological signals collected by wearable devices to provide more comprehensive insights into students' learning status.

[0009] The image acquisition module can use a high-resolution camera to ensure that clear image data is collected.

[0010] The above technical solution solves the technical problem in the prior art that the traditional classroom observation method is difficult to fully and accurately grasp the learning status of each student.

[0011] In a further technical solution, the data processing module operates strictly in accordance with the established process to ensure the accuracy and completeness of the data.

[0012] The above technical solution solves the technical problems of inefficient and non-standard data processing in the prior art.

[0013] In a further technical solution, the expression recognition module improves the Resnet18 network, including disassembling the large convolution kernel and replacing it with multiple small convolution kernels, and improving the downsampling module.

[0014] The above technical solution solves the technical problem of low accuracy of expression recognition in the prior art.

[0015] In a further technical solution, the behavior recognition module improves the Mobilenet-V2 network, including increasing the diversity of convolution kernels, the diversity of residual connections, and improving the feature fusion strategy.

[0016] The above technical solution solves the technical problem of low accuracy of behavior recognition in the prior art.

[0017] In further technical solutions, the learning situation analysis module deeply mines various aspects of students' information and provides teachers with a data basis for personalized teaching.

[0018] The above technical solution solves the technical problem in the prior art that it is impossible to fully understand students' learning characteristics.

[0019] In further technical solutions, the real-time feedback module promptly feeds back the learning situation analysis results to teachers, making it easier for them to adjust their teaching strategies.

[0020] The above technical solution solves the technical problem in the prior art that teachers cannot grasp the learning situation in real time.

[0021] In further technical solutions, the data security module ensures the security of student data and avoids data leakage.

[0022] The above technical solution solves the technical problem of insufficient data security in the prior art.

[0023] In further technical solutions, the system optimization module continuously improves system performance to adapt to different teaching needs and environments.

[0024] The above technical solution solves the technical problem in the prior art that system performance cannot be continuously optimized.

[0025] In a further technical solution, the multimodal data fusion module provides more comprehensive insights into students' learning status.

[0026] The above technical solution solves the technical problem of incomplete insight into students' learning status in the prior art.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] It can comprehensively collect students' classroom image data and accurately capture students' behaviors and expressions from different perspectives. The data preprocessing module ensures the quality of the data set and improves the generalization ability of the recognition system. By improving the Resnet18 and Mobilenet-V2 networks, the recognition accuracy of students' behaviors and expressions is improved, thereby providing teachers with real-time and accurate classroom feedback information, which helps to optimize teaching decisions and improve teaching quality. The data security module ensures the security of student data and protects student privacy. The system optimization module continuously improves system performance to adapt to different teaching needs and environments. The multimodal data fusion module provides more comprehensive insights into students' learning status and provides a more scientific basis for teaching decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a flow chart of image acquisition and processing of the present invention;

[0030] Figure 2 It is a flow chart of facial expression recognition of the present invention;

[0031] Figure 3 is a flow chart of behavior identification of the present invention;

[0032] Figure 4 It is a flow chart of the learning situation analysis and feedback of the present invention. DETAILED DESCRIPTION

[0033] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0034] In the description of the present invention, it should be noted that the terms "upper", "lower", "inner", "outer", "front end", "rear end", "two ends", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0035] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0036] Example:

[0037] See also Figure 1 -4, a student classroom behavior and expression recognition system based on image recognition, including an image acquisition module, a data processing module, an expression recognition module, a behavior recognition module, a learning situation analysis module, a real-time feedback module, a data security module, a system optimization module and a multimodal data fusion module.

[0038] The working principle of the above technical solution is as follows:

[0039] The image acquisition module is installed at different locations in the classroom to collect students' behaviors and facial expressions in class and transmit the image data to the data processing module. After processing the image, the data processing module transmits the data to the expression recognition module and the behavior recognition module. These two modules recognize the students' expressions and behaviors and transmit the results to the learning situation analysis module. After analysis, the learning situation analysis module feeds back the results to the teacher through the real-time feedback module. At the same time, the data security module ensures data security, the system optimization module continuously optimizes the system, and the multimodal data fusion module can provide more comprehensive insights into students' learning status as needed.

[0040] See also Figure 1 - Figure 4 ,The image acquisition module can use a high-resolution camera to ensure that clear image data is collected.

[0041] See also Figure 1 - Figure 4 ,The data processing module operates strictly in accordance with the established ,process to ensure the accuracy and completeness of the data.

[0042] See also Figure 1 - Figure 4 ,The expression recognition module improves the Resnet18 network to improve the accuracy of expression recognition.

[0043] See also Figure 1 - Figure 4 ,The behavior recognition module improves the Mobilenet-V2 network to improve the accuracy of behavior recognition.

[0044] See also Figure 1 - Figure 4 The learning situation analysis module deeply mines students' multi-faceted information and provides teachers with a data basis for personalized teaching.

[0045] See also Figure 1 - Figure 4 The real-time feedback module promptly feeds back the learning situation analysis results to teachers, making it easier for them to adjust their teaching strategies.

[0046] See also Figure 1 - Figure 4 ,The data security module adopts data encryption technology and ,security management measures to ensure data security.

[0047] See also Figure 1 - Figure 4 ,The system optimization module continuously improves system performance to adapt to different teaching needs and environments.

[0048] See also Figure 1 - Figure 4 ,The multimodal data fusion module provides more comprehensive insights into students’ learning status.

[0049] When working, various modules work together to achieve accurate recognition of students' classroom behaviors and expressions and feedback on their learning situations, providing strong support for teachers to optimize their teaching decisions.

[0050] The above-mentioned embodiments only express the specific implementation of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A student classroom behavior expression recognition system based on image recognition, characterized in that: It includes an image acquisition module, which is used to collect image data of students in the classroom. The image acquisition module can be installed in different positions in the classroom to obtain image information from different perspectives, and is also used to capture students' behaviors and facial expressions in the classroom.

2. The student classroom behavior expression recognition system based on image recognition as claimed in claim 1, characterized in that: It also includes a data processing module, which is used to perform operations such as video frame extraction, video privacy information removal, useless perspective and information deletion, and data set division and labeling on the collected images. It is also used to convert the collected video data into a format that can be used for image recognition and remove privacy information in the video.

3. The student classroom behavior expression recognition system based on image recognition as claimed in claim 1, characterized in that: There is an expression recognition module based on Resnet18. This module can accurately recognize students' expressions by performing preliminary feature extraction on the input image, fine processing of residual blocks in multiple stages, and finally average pooling layer aggregation and fully connected layer output classification results. This module builds a behavior and expression recognition model based on a deep learning algorithm to recognize students' expressions and body movements.

4. The student classroom behavior expression recognition system based on image recognition as claimed in claim 3, characterized in that: It also includes a learning situation analysis module, which is used to analyze students' learning status, emotional state, participation and other information, and establish a student individual learning feature model.

5. The student classroom behavior expression recognition system based on image recognition as claimed in claim 4, characterized in that: It also includes a real-time feedback module, which is used to provide real-time feedback of learning situation analysis results to teachers so that teachers can adjust teaching strategies.

6. The student classroom behavior expression recognition system based on image recognition as claimed in claim 5, characterized in that: It also includes a data security module, which uses data encryption technology and security management measures to ensure data security.

7. The student classroom behavior expression recognition system based on image recognition as claimed in claim 6, characterized in that: It also includes a system optimization module, which is used to continuously optimize and upgrade the system to improve recognition accuracy and recognition speed.

8. The student classroom behavior expression recognition system based on image recognition as claimed in claim 7, characterized in that: It also includes a multimodal data fusion module, which is used to combine the student's expression analysis with the physiological signals collected by the wearable device to provide more comprehensive insights into the student's learning status.

Images