Student behavior analysis method and device based on AI smart campus, terminal and medium

Abstract

The invention discloses a student behavior analysis method and device based on an AI smart campus, a terminal and a medium, and the method comprises the steps: collecting the facial expression data, behavior action data and environmental factor data of a student, and carrying out the preprocessing; performing feature extraction on the facial expression data, the behavior action data and the environmental factor data; inputting the facial expression features and the environmental factor features into a first recognition model, and performing emotion recognition; inputting the emotion recognition result and the body movement features into a second recognition model, and analyzing the behavior pattern of the student; according to the method, the facial expressions, the behavior actions and the environmental factor data are fused, comprehensive analysis of the emotion and the behavior of the student is achieved, meanwhile, the corresponding recognition model is designed according to the analysis dimension, and the recognition accuracy is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of smart campuses, and in particular to a student behavior analysis method, device, terminal and medium based on AI smart campuses. Background Art

[0002] With the rapid development of information technology and artificial intelligence (AI), smart campus, as an important research direction in the field of educational technology, is gradually leading the transformation of the educational environment. In recent years, the application of intelligent technology in the campus environment has continued to expand, from traditional campus management systems to integrated intelligent education platforms. The core of these technologies is to optimize the educational process and environment in a data-driven way. In particular, the progress of emotion recognition, behavior analysis and environmental monitoring technology has made it possible to achieve personalized education and the all-round development of students.

[0003] Although existing technologies have made some progress in emotion and behavior analysis, there are still some significant shortcomings. Traditional methods mainly focus on the processing of a single data source, such as emotion recognition through facial expression data alone, or pattern analysis through behavioral action data. However, a single data source cannot fully consider the impact of the environment on the student's state, thus affecting the accuracy and reliability of the analysis results. Summary of the invention

[0004] In order to overcome the defects in the above-mentioned prior art, the present invention provides a student behavior analysis method based on AI smart campus, including: collecting students' facial expression data, behavioral action data and environmental factor data, and preprocessing them; extracting features of the facial expression data, behavioral action data and environmental factor data respectively; inputting the facial expression features and environmental factor features into a first recognition model to perform emotion recognition; inputting the emotion recognition results and body movement features into a second recognition model to analyze the students' behavior patterns.

[0005] As a preferred solution of the student behavior analysis method based on AI smart campus described in the present invention, wherein: the facial expression features include optical flow features and color features; feature extraction of facial expression data includes: extracting the positions of facial key points; calculating the displacement of facial key points between consecutive video frames of facial expressions, identifying the micro-movements of facial muscles, and analyzing the micro-movements of facial muscles through an action unit system to obtain micro-expression features; arranging the micro-expression features in chronological order to form a micro-expression sequence; calculating the motion flow graph and strain graph from the starting frame to the highest peak frame in each micro-expression sequence, and merging them into a three-channel image, performing feature extraction on the three-channel image to obtain optical flow features; converting the color information of the facial area into a color difference map, calculating the color gradient to capture the color changes caused by micro-expressions; fusing the optical flow features, color features and environmental factor features to generate a comprehensive feature representation.

[0006] As a preferred solution of the student behavior analysis method based on AI smart campus described in the present invention, wherein: feature extraction of environmental factor data includes: deploying environmental sensors to obtain light intensity, temperature and humidity data, performing feature extraction on the light intensity, temperature and humidity data to obtain environmental factor characteristics; inputting facial expression characteristics and environmental factor characteristics into a first recognition model to perform emotion recognition.

[0007] As a preferred solution of the student behavior analysis method based on AI smart campus described in the present invention, wherein: the first recognition model includes an input layer, a convolution layer, a spatial attention mechanism module, a channel attention mechanism module, a fully connected layer and an output layer; the convolution layer includes multiple convolution blocks, each convolution block is connected with a residual connection, and the convolution block includes 1 1*1 convolution kernel, 2 3*3 convolution kernels, a batch normalization layer, an activation layer and a pooling layer; the convolution layer is sequentially connected with a spatial attention mechanism and a channel attention mechanism; the features output by the channel attention module are subjected to emotion classification through a fully connected layer, and the emotion recognition result is output using a Softmax activation function in the output layer.

[0008] As a preferred solution of the student behavior analysis method based on AI smart campus described in the present invention, feature extraction of behavior action data includes: processing the real-time video stream through the OpenPose model to extract key points of the student's body, including the head, shoulders, elbows, wrists, hips, knees, and ankles; calculating the relative positions between the key points, the angles formed by the connecting lines, and the overall posture to obtain body movement characteristics.

[0009] As a preferred solution of the student behavior analysis method based on AI smart campus described in the present invention, it also includes: inputting the emotion recognition results and body movement features into the second recognition model to analyze the students' concentration, participation and health status, wherein the concentration is used to evaluate the students' concentration in class, and the participation is used to evaluate the students' interaction in class; the second recognition model includes an input layer, a feature extraction module, a backbone network and an output layer; the feature extraction module includes an emotion subnetwork and a body movement subnetwork, the backbone network includes a connection layer, a fully connected layer, a batch normalization layer and a Dropout layer, and the output layer includes a fully connected layer for outputting concentration prediction, participation prediction and health status classification results.

[0010] As a preferred solution of the student behavior analysis method based on AI smart campus described in the present invention, it also includes: selecting the cross entropy loss function as the target loss function of the second recognition model, performing back propagation training on the second recognition model, and continuously adjusting the network parameters and weight deviations until the target loss function value reaches the minimum.

[0011] The present invention also provides a student behavior analysis device based on AI smart campus, including: a method for implementing a student behavior analysis method based on AI smart campus as described in any of the above items, the student behavior analysis device based on AI smart campus includes: a data acquisition module, configured to collect students' facial expression data, behavioral action data and environmental factor data, and perform preprocessing; a feature extraction module, configured to perform feature extraction on facial expression data, behavioral action data and environmental factor data respectively; an emotion recognition module, configured to input facial expression features and environmental factor features into a first recognition model to perform emotion recognition; a behavior analysis module, configured to input emotion recognition results and body movement features into a second recognition model to analyze students' behavior patterns.

[0012] The present invention also provides a terminal device, comprising:

[0013] one or more processors;

[0014] A memory, coupled to the processor, for storing one or more programs;

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the student behavior analysis method based on AI smart campus as described in any of the above items.

[0016] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the student behavior analysis method based on AI smart campus as described in any of the above items.

[0017] Beneficial effects of the present invention: The present invention realizes a comprehensive analysis of students' emotions and behaviors by integrating facial expressions, behavioral actions and environmental factors data, and designs corresponding recognition models according to the analysis dimensions, which significantly improves the accuracy of recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative labor. Among them:

[0019] Figure 1 This is a flow chart of the student behavior analysis method based on AI smart campus described in the first embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0022] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0023] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0024] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0025] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0026] Example 1

[0027] Reference Figure 1, which is the first embodiment of the present invention, and provides a student behavior analysis method based on AI smart campus, including:

[0028] S1: Collect students’ facial expression data, behavioral action data, and environmental factor data, and perform preprocessing.

[0029] Facial expression data: Use high-resolution cameras to capture students’ facial expressions in a classroom or learning environment.

[0030] Behavior and movement data: Use sensors or cameras to collect students’ body movements;

[0031] Environmental factor data: Deploy environmental sensors to obtain light intensity, temperature, and humidity data.

[0032] Facial expression data, behavioral action data, and environmental factor data are streamed in real time, and streaming data processing platforms (such as Apache Kafka or AWS Kinesis) are used to achieve real-time integration and synchronization of data to ensure data consistency and timeliness.

[0033] S2: Extract features from facial expression data, behavioral action data, and environmental factor data respectively.

[0034] Facial expression features include optical flow features and color features.

[0035] Specifically, the steps for feature extraction of facial expression data are as follows:

[0036] (1) Extract the positions of facial key points. Facial key points include eyes, nose, mouth, eyebrows, and ears. These facial key points represent the different positions and states of facial muscles.

[0037] (2) Calculate the displacement of facial key points between consecutive video frames of facial expressions, identify the micro-movements of facial muscles, and analyze the micro-movements of facial muscles through the Facial Action Coding System (FACS) to obtain micro-expression features;

[0038] By calculating the displacement of facial feature points between consecutive frames, subtle changes in facial expressions can be identified. Each action unit represents a specific movement of a part of the face, and the combination of these action units forms a micro-expression.

[0039] By analyzing the position changes of feature points in each frame, micro-expressions (such as the expansion of a smile, the rise of an eyebrow) as well as their duration and amplitude over time can be identified.

[0040] (3) Arrange the micro-expression features in chronological order to form a micro-expression sequence;

[0041] Micro-expression sequences refer to small and short-term changes in facial expressions captured in a time series. Micro-expression sequences can help identify and understand short-term emotional changes.

[0042] (4) Calculate the motion flow graph and strain graph from the starting frame to the highest peak frame in each micro-expression sequence, merge them into a three-channel image, extract features from the three-channel image, and obtain optical flow features;

[0043] (5) The color information of the facial area is converted into a color difference map and the color gradient is calculated to capture the color changes caused by micro-expressions.

[0044] Furthermore, by considering the influence of environmental factor data, which may have an impact on emotional state, we can more accurately understand and explain the emotion recognition results. Specifically, ResNet-10 is used to extract features of light intensity, temperature and humidity data to obtain environmental factor features.

[0045] The optical flow features, color features and environmental factor features are integrated to generate a comprehensive feature representation.

[0046] Furthermore, the real-time video stream is processed through the OpenPose model. First, the OpenPose model is loaded. The model can be loaded and inferred using the Python API of OpenPose or through deep learning frameworks such as TensorFlow / PyTorch.

[0047] Each frame of the video stream is processed and OpenPose is used to detect the key points of the student's body, including the head, shoulders, elbows, wrists, hips, knees, and ankles;

[0048] The coordinates of each key point are extracted, and the relative positions between the key points (for example, calculating the distance between the hand and the shoulder), the angle formed by the connecting lines (for example, calculating the angle between the shoulder, elbow and wrist to evaluate the degree of bending of the limbs) and the overall posture (the overall posture is calculated by combining the position information of the key points. For example, the overall inclination or posture of the body is calculated using the key point data) are calculated to obtain the limb movement characteristics.

[0049] S3: Input the facial expression features and environmental factor features into the first recognition model to perform emotion recognition.

[0050] The facial expression features and environmental factor features are input into the first recognition model to perform emotion recognition.

[0051] The first recognition model includes an input layer, a convolutional layer, a spatial attention mechanism module, a channel attention mechanism module, a fully connected layer, and an output layer;

[0052] The convolution layer consists of multiple convolution blocks, each of which is connected with a residual connection (to help alleviate the gradient vanishing problem in deep network training). The convolution block includes a 1*1 convolution kernel, two 3*3 convolution kernels, a batch normalization layer, an activation layer, and a pooling layer.

[0053] The convolutional layer is followed by a spatial attention mechanism (focusing on important areas in the image to improve the model's sensitivity to specific facial areas (such as eyes and mouth)) and a channel attention mechanism (adaptively adjusting the weights of different feature channels to enhance the ability to extract key information) to strengthen the focus on important areas and features.

[0054] The features output by the channel attention module are passed through the fully connected layer for emotion classification, and the Softmax activation function is used in the output layer to output the emotion recognition results.

[0055] The facial expression features and environmental factor features are input into the first recognition model to perform emotion recognition, for example:

[0056] Input: Facial expression images showing students’ facial features, environmental data such as classroom temperature and lighting.

[0057] Output emotion classification probabilities: happy: 0.75, sad: 0.15, angry: 0.05, surprised: 0.05.

[0058] S4: Input the emotion recognition results and body movement features into the second recognition model to analyze the students' behavior patterns.

[0059] The emotion recognition results and body movement features are input into the second recognition model to analyze the students’ concentration, participation and health status. The concentration is used to evaluate the students’ concentration in class, and the participation is used to evaluate the students’ interaction in class.

[0060] The second recognition model includes an input layer, a feature extraction module, a backbone network and an output layer; the feature extraction module includes an emotion sub-network and a body movement sub-network, the backbone network includes a connection layer, a fully connected layer, a batch normalization layer and a Dropout layer, and the output layer includes a fully connected layer, which is used to output concentration prediction, engagement prediction and health status classification results.

[0061] Specifically, the emotion sub-network is responsible for processing the emotion recognition results and extracting emotion-related features from them. The emotion sub-network includes a first fully connected layer and a second fully connected layer. The first fully connected layer includes 128 neurons and uses a ReLU activation function; the second fully connected layer includes 64 neurons and uses a ReLU activation function.

[0062] The limb movement subnetwork is responsible for processing limb movement features and extracting movement-related features from them. The limb movement subnetwork includes the third fully connected layer and the fourth fully connected layer. The third fully connected layer includes 128 neurons and uses the ReLU activation function; the fourth fully connected layer includes 64 neurons and uses the ReLU activation function.

[0063] The connection layer of the backbone network is used to merge the outputs of the emotion sub-network and the body movement sub-network, and further process them through the fully connected layer to learn the relationship between features; the backbone network includes the fifth fully connected layer (including 256 neurons, using the ReLU activation function) and the sixth fully connected layer (including 128 neurons, using the ReLU activation function), the first batch normalization layer and the second batch normalization layer, the first Dropout layer (discard rate 0.5) and the second Dropout layer (discard rate 0.5). The connection layer, the fifth fully connected layer, the first batch normalization layer, the first Dropout layer, the sixth fully connected layer, the second batch normalization layer and the second Dropout layer are connected in sequence.

[0064] The output layer includes the seventh fully connected layer (including 1 neuron, using a linear activation function), the eighth fully connected layer (including 1 neuron, using a linear activation function) and the ninth fully connected layer (including k neurons, k is the number of health status categories, using a Softmax activation function). The seventh fully connected layer is used for concentration prediction (regression task), the eighth fully connected layer is used for engagement prediction (regression task), and the ninth fully connected layer is used for health status classification (classification task).

[0065] Furthermore, the cross entropy loss function is selected as the target loss function of the second recognition model, and the second recognition model is trained by back propagation, and the network parameters and weight deviations are continuously adjusted until the target loss function value reaches the minimum.

[0066] The following is a specific example of the results output by the second recognition model:

[0067] Assume that the input feature emotion recognition results are: happy: 0.7, sad: 0.1, other emotions: 0.2

[0068] The characteristics of body movements are: sitting stability: 0.8, hand activity frequency: 0.3, head movement amplitude: 0.4, body tilt angle: 0.2

[0069] Model output

[0070] 1. Concentration prediction (regression task):

[0071] Predicted value: 0.75 (indicating high concentration and strong focus in learning or activities)

[0072] 2. Engagement prediction (regression task):

[0073] Predicted value: 0.60 (indicating moderate participation)

[0074] 3. Health status classification (classification task, with 3 health status categories: good, average, and needs attention):

[0075] Prediction value: 0.70, good

[0076] By combining body movement features and emotion recognition results, this application realizes real-time analysis of student behavior patterns, which can help teachers better understand students' concentration and classroom participation, thereby optimizing teaching strategies and improving classroom interaction effects.

[0077] Example 2

[0078] The present invention provides a student behavior analysis device based on AI smart campus, which is used to implement the steps of the student behavior analysis method based on AI smart campus as described in any of the above embodiments. The student behavior analysis based on AI smart campus includes:

[0079] The data collection module is configured to collect students' facial expression data, behavioral action data, and environmental factor data, and perform preprocessing;

[0080] A feature extraction module is configured to perform feature extraction on the facial expression data, the behavior action data and the environmental factor data respectively;

[0081] An emotion recognition module is configured to input facial expression features and environmental factor features into a first recognition model to perform emotion recognition;

[0082] The behavior analysis module is configured to input the emotion recognition results and body movement features into the second recognition model to analyze the student's behavior pattern.

[0083] Example 3

[0084] This embodiment provides a terminal device, including:

[0085] one or more processors;

[0086] A memory, coupled to the processor, for storing one or more programs;

[0087] When the one or more programs are executed by the one or more processors, the one or more processors implement the student behavior analysis method based on AI smart campus as described above.

[0088] The processor is used to control the overall operation of the terminal device to complete all or part of the steps of the above-mentioned student behavior analysis method based on AI smart campus. The memory is used to store various types of data to support the operation of the terminal device, which data may include, for example, instructions for any application or method used to operate on the terminal device, and application-related data. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, referred to as EPROM), programmable read-only memory (Programmable Read-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic memory, flash memory, disk or optical disk.

[0089] The terminal device can be implemented by one or more application-specific integrated circuits (Application Specific1n Integrated Circuit, ASC), digital signal processors (Digital Signal Processor, DSP), digital signal processing devices (Digital Signal Processing Device, DSPD), programmable logic devices (Programmable Logic Device, PLD), field programmable gate arrays (Field Programmable Gate Array, FPGA), controllers, microcontrollers, microprocessors or other electronic components, and is used to execute the student behavior analysis method based on AI smart campus as described in any of the above embodiments, and achieve the same technical effect as the above method.

[0090] Example 4

[0091] This embodiment provides a computer-readable storage medium, and when the program instructions are executed by a processor, the steps of the student behavior analysis method based on AI smart campus as described in any of the above embodiments are implemented. For example, the computer-readable storage medium can be the above-mentioned memory including program instructions, and the above-mentioned program instructions can be executed by the processor of the terminal device to complete the student behavior analysis method based on AI smart campus as described in any of the above embodiments, and achieve the same technical effect as the above method.

[0092] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in an assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed ASIC for this purpose.

[0093] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0094] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques described in the present invention, the present invention also includes the computer itself. The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on a display.

[0095] As used in this application, the terms "component", "module", "system", etc. are intended to refer to a computer-related entity, which can be hardware, firmware, a combination of hardware and software, software, or software in operation. For example, a component can be, but is not limited to: a process running on a processor, a processor, an object, an executable file, a thread in execution, a program, and / or a computer. As an example, an application running on a computing device and the computing device can both be components. One or more components can exist in a process and / or thread in execution, and a component can be located in a computer and / or distributed between two or more computers. In addition, these components can be executed from various computer-readable media having various data structures thereon. These components can communicate in a local and / or remote process manner, such as according to a signal having one or more data packets (e.g., data from a component that interacts with another component in a local system, a distributed system, and / or interacts with other systems in a signal manner through a network such as the Internet).

[0096] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A student behavior analysis method based on AI smart campus, characterized in that: include: Collect students' facial expression data, behavioral action data, and environmental factor data, and perform preprocessing; Extract features from facial expression data, behavioral action data, and environmental factor data respectively; Inputting facial expression features and environmental factor features into a first recognition model to perform emotion recognition; The emotion recognition results and body movement features are input into the second recognition model to analyze the students' behavior patterns.

2. The student behavior analysis method based on AI smart campus according to claim 1, characterized in that: The facial expression features include optical flow features and color features; Feature extraction of facial expression data includes: Extract the location of facial key points; Calculate the displacement of facial key points between consecutive video frames of facial expressions, identify the micro-movements of facial muscles, and analyze the micro-movements of facial muscles through the action unit system to obtain micro-expression features; Arrange the micro-expression features in chronological order to form a micro-expression sequence; Calculate the motion flow graph and strain graph from the starting frame to the highest peak frame in each micro-expression sequence, merge them into a three-channel image, extract features from the three-channel image, and obtain optical flow features; The color information of the facial area is converted into a color difference map and the color gradient is calculated to capture the color changes caused by micro-expressions; The optical flow features, color features and environmental factor features are integrated to generate a comprehensive feature representation.

3. The student behavior analysis method based on AI smart campus according to claim 2, characterized in that: Feature extraction of environmental factor data includes: Deploy environmental sensors to obtain light intensity, temperature and humidity data, perform feature extraction on the light intensity, temperature and humidity data, and obtain environmental factor characteristics; The facial expression features and environmental factor features are input into the first recognition model to perform emotion recognition.

4. The student behavior analysis method based on AI smart campus according to claim 3, characterized in that: The first recognition model includes an input layer, a convolutional layer, a spatial attention mechanism module, a channel attention mechanism module, a fully connected layer and an output layer; The convolution layer includes a plurality of convolution blocks, each of which is connected with a residual connection, and the convolution block includes a 1*1 convolution kernel, two 3*3 convolution kernels, a batch normalization layer, an activation layer and a pooling layer; The convolutional layer is sequentially connected with a spatial attention mechanism and a channel attention mechanism; The features output by the channel attention module are passed through the fully connected layer for emotion classification, and the Softmax activation function is used in the output layer to output the emotion recognition results.

5. The student behavior analysis method based on AI smart campus according to claim 4, characterized in that: Feature extraction of behavioral action data includes: The real-time video stream is processed through the OpenPose model to extract the key points of the student's body, including the head, shoulders, elbows, wrists, hips, knees, and ankles; Calculate the relative positions between key points, the angles formed by the connecting lines, and the overall posture to obtain the body movement characteristics.

6. The student behavior analysis method based on AI smart campus according to claim 5, characterized in that: Also includes: Input the emotion recognition results and body movement features into the second recognition model to analyze the students' concentration, participation and health status, wherein the concentration is used to evaluate the students' concentration in class, and the participation is used to evaluate the students' interaction in class; The second recognition model includes an input layer, a feature extraction module, a backbone network and an output layer; the feature extraction module includes an emotion subnetwork and a body movement subnetwork, the backbone network includes a connection layer, a fully connected layer, a batch normalization layer and a Dropout layer, and the output layer includes a fully connected layer for outputting concentration prediction, engagement prediction and health status classification results.

7. The student behavior analysis method based on AI smart campus according to claim 6, characterized in that: Also includes: The cross entropy loss function is selected as the target loss function of the second recognition model. The second recognition model is trained by back propagation, and the network parameters and weight deviations are continuously adjusted until the target loss function value reaches the minimum.

8. A student behavior analysis device based on AI smart campus, used to implement the student behavior analysis method based on AI smart campus according to any one of claims 1 to 7, the student behavior analysis device based on AI smart campus comprising: The data collection module is configured to collect students' facial expression data, behavioral action data, and environmental factor data, and perform preprocessing; A feature extraction module is configured to perform feature extraction on the facial expression data, the behavior action data and the environmental factor data respectively; An emotion recognition module is configured to input facial expression features and environmental factor features into a first recognition model to perform emotion recognition; The behavior analysis module is configured to input the emotion recognition results and body movement features into the second recognition model to analyze the student's behavior pattern.

9. A terminal device, characterized in that: include: one or more processors; A memory, coupled to the processor, for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the student behavior analysis method based on AI smart campus as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the student behavior analysis method based on AI smart campus as described in any one of claims 1 to 7.

Images