Student activity management system and method based on multi-dimensional data

Abstract

The invention relates to the technical field of student activity management, and particularly discloses a student activity management system and method based on multi-dimensional data, and the system comprises a camera which is installed in a student activity area and is used for collecting a student image; the monitoring interface is connected to the camera and is used for receiving and extracting the student images; the participation condition detector is used for receiving the student image transmitted by the monitoring interface, identifying and marking the identity of the student based on the student image, and detecting and uploading the participation condition of the student; and the management database is used for filing and storing the participation conditions of the students. Therefore, more comprehensive feature information about the student object in the activity can be acquired in a multi-dimensional data acquisition mode, so that the complex behavior pattern of the student can be more effectively captured, and the activity participation condition of the student can be identified and evaluated.

Description

Technical Field

[0001] The present application relates to the technical field of student activity management, and more specifically, to a student activity management system and method based on multidimensional data. Background Art

[0002] The existing student activity management system works well in a single application scenario (such as classroom), but has obvious shortcomings in other non-classroom environments (such as self-study, club activities, etc.). The main problem is that the student behavior patterns in different scenarios vary greatly, and directly applying the evaluation model in a single scenario will lead to deviations in the evaluation results.

[0003] In response to the above problems, Chinese patent CN112598550A proposes a multi-dimensional management system and method for student activities based on behavioral analysis. After performing simple feature extraction on student images captured by surveillance cameras, the system compares and matches the features with statistical models, thereby obtaining the students' participation status in different types of activities through simple calculations, so as to comprehensively evaluate students' performance.

[0004] In the above-mentioned multi-dimensional management system of student activities based on behavioral analysis, although this method can solve the problem of student behavior analysis in multiple scenarios, when it conducts student image analysis to judge student participation, due to the diversity of student behavior, the behavioral characteristics in different scenarios may be different, and this simple feature extraction and comparison matching method may have limitations when dealing with complex behavior patterns, resulting in the system being insufficient to fully describe and accurately evaluate student participation in some cases. In addition, this method requires the system to adjust the behavior model according to different scenarios. Although the system can adapt to multiple scenarios by comparing statistical models, if the behavior pattern changes too much or new forms of activities emerge, the existing model may not be able to adapt in time, resulting in inaccurate evaluation results.

[0005] Therefore, an optimized student activity management system is desired. Summary of the invention

[0006] The present application provides a student activity management system and method based on multidimensional data, which can obtain more comprehensive feature information about student objects in activities through multidimensional data collection, so as to more effectively capture the complex behavior patterns of students and identify and evaluate the status of students' participation in activities. In addition, when responding to different scene changes, it can better adapt to different scene changes by jointly analyzing the facial features and behavioral features of student objects, thereby improving the flexibility and adaptability of the student activity management system.

[0007] In a first aspect, a student activity management system based on multidimensional data is provided, comprising:

[0008] A camera installed in the student activity area, the camera is used to collect images of students;

[0009] A monitoring interface connected to the camera, the monitoring interface being used to receive and extract the student image;

[0010] a participation status detector, for receiving the student image transmitted by the monitoring interface, identifying and marking the identity of the student based on the student image, and detecting and uploading the student's participation status;

[0011] A management database for archiving and storing the student's participation status;

[0012] Wherein, the participation status detector includes:

[0013] A student image time series data acquisition module is used to obtain the time sequence of student images;

[0014] A student face-behavior joint representation module is used to extract and jointly represent each student image in the time queue of the student images based on facial features and behavioral features to obtain a time queue of face-behavior joint semantic feature representation;

[0015] A student face-behavior temporal information aggregation representation module is used to perform temporal dynamic information propagation fusion on the time queue of the face-behavior joint semantic feature representation to obtain a multi-dimensional joint representation of the student object behavior semantics;

[0016] The student participation status detection module is used to determine the level label of the student's participation status based on the multi-dimensional joint representation of the student object behavior semantics.

[0017] In a possible implementation, the student face-behavior joint representation module includes:

[0018] A facial feature extraction unit, used for inputting each student image in the time queue of the student images into a facial feature extractor based on a dilated convolutional neural network model to obtain a time queue of facial semantic feature vectors of student objects;

[0019] A behavior feature extraction unit, used for inputting each student image in the time queue of the student images into a behavior feature extractor based on a deep separable convolutional neural network model to obtain a time queue of student object behavior semantic feature vectors;

[0020] The feature-to-feature significant interaction encoding processing unit is used to perform feature-to-feature significant interaction encoding on each group of corresponding student object behavior semantic feature vectors and student object facial semantic feature vectors in the time queue of the student object behavior semantic feature vectors and the time queue of the student object facial semantic feature vectors to obtain a time queue of facial-behavior joint semantic feature vectors as the time queue of the facial-behavior joint semantic feature representation.

[0021] In a possible implementation, the student face-behavior temporal information aggregation representation module includes:

[0022] A principal component extraction unit, configured to extract the time queue of the current face-behavior joint semantic feature representation and the time queue of the historical face-behavior joint semantic feature representation from the time queue of the face-behavior joint semantic feature representation, and then perform principal component extraction to obtain a sequence of the current face-behavior joint semantic feature principal component representation and the historical face-behavior joint semantic principal component representation;

[0023] A time-decayed dynamic aggregation processing unit is used to perform time-decayed dynamic aggregation on the sequence of the current face-behavior joint semantic feature principal component representation and the historical face-behavior joint semantic principal component representation to obtain the student object behavior semantic multi-dimensional joint representation.

[0024] In a possible implementation, the main component extraction unit is used to:

[0025] Extracting a current face-behavior joint semantic feature vector from the time queue of the face-behavior joint semantic feature vector, and performing eigenvalue-based principal component extraction on the current face-behavior joint semantic feature vector to obtain a current face-behavior joint semantic feature principal component representation vector as the current face-behavior joint semantic feature principal component representation;

[0026] The other facial-behavior joint semantic feature vectors in the time queue of the facial-behavior joint semantic feature vectors are defined as historical facial-behavior joint semantic feature vectors to obtain a sequence of historical facial-behavior joint semantic feature vectors, and principal component extraction based on eigenvalues ​​is performed on each historical facial-behavior joint semantic feature vector in the sequence of historical facial-behavior joint semantic feature vectors to obtain a sequence of historical facial-behavior joint semantic principal component representation vectors as the sequence of historical facial-behavior joint semantic principal component representations.

[0027] In a possible implementation, the time-series attenuation dynamic aggregation processing unit includes:

[0028] A temporal propagation attenuation entropy absolute factor calculation subunit, used to calculate the temporal propagation attenuation entropy absolute factor of each historical face-behavior joint semantic principal component representation vector in the sequence of the historical face-behavior joint semantic principal component representation vector relative to the current face-behavior joint semantic feature principal component representation vector to obtain a sequence of face-behavior joint temporal propagation attenuation entropy absolute factors;

[0029] A time dimension modulation processing subunit, for performing time dimension modulation on each face-behavior joint temporal propagation attenuation entropy absolute factor in the sequence of face-behavior joint temporal propagation attenuation entropy absolute factors based on the time span between each historical face-behavior joint semantic principal component representation vector in the sequence of historical face-behavior joint semantic principal component representation vectors and the current face-behavior joint semantic feature principal component representation vector to obtain a sequence of face-behavior joint temporal span modulation propagation attenuation entropy factors;

[0030] An information transfer screening processing subunit, used for inputting the sequence of face-behavior joint temporal span modulation propagation attenuation entropy factors into an information transfer screening module based on a gating function to obtain a sequence of face-behavior joint temporal span modulation propagation attenuation weights;

[0031] A vector sequence weighted sum processing subunit, for calculating the weighted sum of the sequence of historical face-behavior joint semantic principal component representation vectors based on the sequence of face-behavior joint temporal span modulation propagation attenuation weights to obtain a historical face-behavior joint semantic principal component significant transfer aggregation representation vector;

[0032] The student object behavior semantics multidimensional joint processing subunit is used to calculate the positional sum of the historical face-behavior joint semantics principal component significant transfer aggregation representation vector and the current face-behavior joint semantics feature principal component representation vector to obtain the student object behavior semantics multidimensional joint representation vector as the student object behavior semantics multidimensional joint representation.

[0033] In a possible implementation, the time series propagation attenuation entropy absolute factor calculation subunit is used to:

[0034] Calculating the positional division of the historical face-behavior joint semantic principal component representation vector and the current face-behavior joint semantic feature principal component representation vector at corresponding positions to obtain a face-behavior joint semantic feature principal component attenuation vector;

[0035] Calculating the base-two logarithmic function value of the absolute value of each eigenvalue of the face-behavior joint semantic feature principal component attenuation vector to obtain the face-behavior joint semantic feature principal component attenuation logarithmic vector;

[0036] After calculating the positional dot product of the historical face-behavior joint semantic principal component representation vector and the face-behavior joint semantic feature principal component attenuation logarithm vector, each position feature value in the obtained dot product vector is multiplied by position point to obtain the face-behavior joint temporal propagation attenuation value;

[0037] An exponential function of the face-behavior joint temporal propagation attenuation value with the natural constant e as the base is calculated to obtain the face-behavior joint temporal propagation attenuation entropy absolute factor.

[0038] In a possible implementation, the time dimension modulation processing subunit is used to:

[0039] Subtracting the timestamp of the current face-behavior joint semantic feature principal component representation vector from the timestamp of the historical face-behavior joint semantic principal component representation vector and rounding down to obtain a face-behavior joint time span value;

[0040] Calculating an exponential function with the natural constant e as the base and the face-behavior joint time span value as the exponent to obtain a face-behavior joint time span modulation value;

[0041] The face-behavior joint temporal propagation attenuation entropy absolute factor corresponding to the historical face-behavior joint semantic principal component representation vector is divided by the face-behavior joint time span modulation value to obtain the face-behavior joint temporal span modulation propagation attenuation entropy factor.

[0042] In a possible implementation, the information transmission screening processing subunit is used to:

[0043] Comparing each face-behavior joint temporal span modulation propagation attenuation entropy factor in the sequence of face-behavior joint temporal span modulation propagation attenuation entropy factors with a predetermined threshold to obtain a sequence of face-behavior joint temporal span modulation propagation attenuation weights;

[0044] Among them, in response to the face-behavior joint timing span modulation propagation attenuation entropy factor being greater than the predetermined threshold, the face-behavior joint timing span modulation propagation attenuation entropy factor greater than the predetermined threshold is input into the sigmoid function; in response to the face-behavior joint timing span modulation propagation attenuation entropy factor being less than or equal to the predetermined threshold, the face-behavior joint timing span modulation propagation attenuation entropy factor less than or equal to the predetermined threshold is set to zero.

[0045] In a possible implementation, the student participation status detection module is used to: input the student object behavior semantic multi-dimensional joint representation vector into a classifier-based participation status evaluator to obtain an evaluation result, and the evaluation result is used to represent the level label of the participation status.

[0046] In a second aspect, a student activity management method based on multidimensional data is provided, characterized in that it includes:

[0047] Collect student images through cameras;

[0048] Receiving and extracting the student image through a monitoring interface;

[0049] Receiving the student image transmitted by the monitoring interface, identifying and marking the identity of the student based on the student image, and detecting and uploading the student's participation status;

[0050] Archiving and storing the student's participation status;

[0051] The steps of receiving the student image transmitted by the monitoring interface, identifying and marking the identity of the student based on the student image, and detecting and uploading the student's participation status include:

[0052] Get the time queue of the student images;

[0053] Extracting and jointly representing each student image in the time queue of student images based on facial features and behavioral features respectively to obtain a time queue represented by facial-behavior joint semantic features;

[0054] Performing temporal dynamic information propagation fusion on the time queue of the face-behavior joint semantic feature representation to obtain a multi-dimensional joint representation of the student object behavior semantics;

[0055] Based on the multi-dimensional joint representation of the student object behavior semantics, a grade label of the student's participation status is determined.

[0056] The present application provides a student activity management system and method based on multidimensional data, which includes: a camera installed in the student activity area, the camera is used to collect student images; a monitoring interface connected to the camera, the monitoring interface is used to receive and extract the student image; a participation status detector is used to receive the student image transmitted by the monitoring interface, and identify and mark the identity of the student based on the student image, and detect and upload the student's participation status; a management database is used to archive and store the student's participation status. In this way, more comprehensive feature information about student objects in the activity can be obtained through multidimensional data collection, so as to more effectively capture the complex behavior patterns of students and identify and evaluate the status of students' participation in activities. In addition, when dealing with different scene changes, it is possible to better adapt to different scene changes by jointly analyzing the facial features and behavioral features of student objects, thereby improving the flexibility and adaptability of the student activity management system. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application are briefly introduced below. Obviously, the drawings described below only relate to some embodiments of the present application, and are not intended to limit the present application.

[0058] Figure 1 This is a schematic block diagram of a participation status detector in a student activity management system based on multidimensional data according to an embodiment of the present application.

[0059] Figure 2 This is a data flow diagram of a participation status detector in a student activity management system based on multidimensional data in an embodiment of the present application.

[0060] Figure 3 This is a schematic block diagram of a student face-behavior joint representation module in a student activity management system based on multidimensional data according to an embodiment of the present application.

[0061] Figure 4 This is a schematic block diagram of a student facial-behavior temporal information aggregation representation module in a student activity management system based on multidimensional data according to an embodiment of the present application.

[0062] Figure 5 A schematic block diagram of a time-degradation dynamic aggregation processing unit in a student activity management system based on multidimensional data according to an embodiment of the present application.

[0063] Figure 6 This is a schematic flowchart of a student activity management method based on multidimensional data according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work also fall within the scope of protection of the present application.

[0065] In the technical solution of the present application, a student activity management system based on multidimensional data is proposed, including: a camera installed in the student activity area, the camera is used to collect student images; a monitoring interface connected to the camera, the monitoring interface is used to receive and extract the student images; a participation status detector is used to receive the student images transmitted by the monitoring interface, and identify and mark the identity of the students based on the student images, and detect and upload the student participation status; a management database is used to archive and store the student participation status. Through this student activity management system based on multidimensional data, more comprehensive feature information about student objects in the activity can be obtained through multidimensional data collection, including facial features and behavioral features, so as to more effectively capture the complex behavior patterns of students and identify and evaluate the student participation status. In addition, when dealing with different scene changes, it is possible to better adapt to different scene changes by jointly analyzing the facial features and behavioral features of student objects, thereby improving the flexibility and adaptability of the student activity management system.

[0066] Specifically, in the above-mentioned student activity management system based on multidimensional data, first, according to the different scene requirements of student activities, a suitable location is selected in the student activity area to install a camera. The camera should be installed in a place that can cover as many student activity areas as possible, while ensuring clear image quality and avoiding dead angles. Since students will perform different activities in different occasions, it is necessary to deploy cameras in multiple student activity areas to monitor students' participation in different situations. For example, in the classroom, it may be necessary to observe students' performance in class, and in the library, it may be necessary to pay attention to their reading habits. The installed camera will capture student images in real time and transmit these image data to the back-end participation status detector through the line connected to the monitoring interface. Among them, the monitoring interface serves as a bridge connecting the camera and other system components (such as the participation status detector) to ensure the effective transmission of data. It enables the original image data to be smoothly transferred from the acquisition point to the data analysis layer, laying the foundation for subsequent processing. The monitoring interface not only receives images, but also undertakes the task of preliminary data extraction. For example, it can compress, convert the format or perform other preprocessing operations on the original image to make the image more suitable for the next step of analysis. Processing image data through a dedicated interface can help filter out useless information and ensure the quality of data transmitted to subsequent systems. Furthermore, the participation status detector is an important component of a student activity management system based on multidimensional data. Its role is to receive student images transmitted by the monitoring interface, identify and mark the identity of students based on these images, and detect and upload the student's participation status. The application of this technology is crucial to understanding and evaluating students' performance in various activities. Finally, as the system collects more and more student image data, a unified place is needed to store and manage this data for subsequent analysis and use. Storing students' participation status allows schools or administrators to check historical records at any time to understand students' performance in different time periods, providing a basis for formulating more effective teaching strategies. By tracking students' participation status for a long time, each student's interests and weaknesses can be discovered, thereby achieving personalized teaching and improving teaching quality. Regularly evaluating students' participation and storing the results can provide students with continuous feedback, help them recognize their own strengths and weaknesses, and promote self-development.

[0067] In particular, in the above-mentioned student activity management system based on multidimensional data, it is crucial to identify and evaluate students' participation status based on the analysis of student images. This not only requires effective and multi-faceted feature capture of student images to identify and mark the identity of students, but also requires attention to the temporal characteristics of the multi-dimensional joint semantics of student objects. This is because the movements and facial semantics of students when participating in activities are constantly changing. By paying attention to the temporal characteristics of this semantics, students' participation status can be identified and evaluated more accurately.

[0068] Specifically, the technical concept of the present application is to collect the time series data of student images, analyze the time series data of these student images, so as to capture and characterize the facial semantic features and behavioral semantic features of the student objects in the student images, and then use the time series characteristics of these two multi-dimensional semantic joint interaction features of the student objects to identify the student identity and understand the actions, so as to evaluate the student participation status. In this way, the flexibility and adaptability of the student activity management system can be improved, which helps to better manage the participation status of students in activities.

[0069] Figure 1 This is a schematic block diagram of a participation status detector in a student activity management system based on multidimensional data according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of a participation status detector in a student activity management system based on multidimensional data in an embodiment of the present application. Figure 1 and Figure 2 As shown, the participation status detector 100 includes: a student image time series data acquisition module 110, which is used to obtain a time queue of student images; a student face-behavior joint representation module 120, which is used to extract and jointly represent each student image in the time queue of the student images based on facial features and behavioral features to obtain a time queue of face-behavior joint semantic feature representation; a student face-behavior time series information aggregation representation module 130, which is used to perform time series dynamic information propagation fusion on the time queue of face-behavior joint semantic feature representation to obtain a multi-dimensional joint representation of student object behavior semantics; a student participation status detection module 140, which is used to determine the level label of the student's participation status based on the multi-dimensional joint representation of student object behavior semantics.

[0070] In the above-mentioned student activity management system based on multidimensional data, the student image time series data acquisition module 110 is used to obtain the time queue of student images. It should be understood that by obtaining the time queue of student images, the dynamic behavior patterns of students during the activity can be analyzed, such as changes in students' attention, fluctuations in participation, etc., so as to more accurately evaluate the students' participation in the activity. The images in the time queue allow the system to identify the same student at different time points, and even if the student's position or posture changes, it can continue to track their behavior performance to ensure the accuracy and continuity of recognition. Using multiple images in the time queue, facial features and behavioral features can be comprehensively extracted from multiple angles and time points, and then facial-behavior joint representation can be performed to improve the comprehensiveness and complexity of feature expression. Students' behavior and facial expressions will change over time, and the image data of the time queue can reflect this dynamic change, which helps the system better adapt to the evaluation needs in different scenarios and provide more accurate participation status evaluation.

[0071] In the above-mentioned student activity management system based on multidimensional data, the student face-behavior joint representation module 120 is used to extract and jointly represent each student image in the time queue of the student images based on facial features and behavioral features to obtain a time queue of face-behavior joint semantic feature representation. Optionally, in one embodiment of the present application, Figure 3 Schematic block diagram of a student face-behavior joint representation module in a student activity management system based on multidimensional data in an embodiment of the present application. Figure 3 As shown, the student face-behavior joint representation module 120 includes: a facial feature extraction unit 121, which is used to input each student image in the time queue of the student images into a facial feature extractor based on a void convolutional neural network model to obtain a time queue of student object facial semantic feature vectors; a behavior feature extraction unit 122, which is used to input each student image in the time queue of the student images into a behavior feature extractor based on a deep separable convolutional neural network model to obtain a time queue of student object behavior semantic feature vectors; and a feature significance interaction encoding processing unit 123, which is used to perform feature significance interaction encoding on each group of corresponding student object behavior semantic feature vectors and student object facial semantic feature vectors in the time queue of the student object behavior semantic feature vectors and the time queue of the student object facial semantic feature vectors to obtain a time queue of face-behavior joint semantic feature vectors as the time queue of the face-behavior joint semantic feature representation.

[0072] Optionally, in one embodiment of the present application, the facial feature extraction unit 121 is used to input each student image in the time queue of the student image into a facial feature extractor based on a dilated convolutional neural network model to obtain a time queue of facial semantic feature vectors of student objects. It should be understood that each student image in the time queue of the student image is input into a facial feature extractor based on a dilated convolutional neural network model for feature mining to extract implicit semantic feature information about the face of the student object in each student image, thereby obtaining a time queue of facial semantic feature vectors of the student object. Through the facial feature extractor based on the dilated convolutional neural network model, feature information such as facial implicit features, facial expressions, and head postures of the student object can be captured. Facial implicit features help to identify the identity of the student, while facial expressions and head posture semantics help to understand the student's participation in activities. For example, facial expressions can be used to determine whether a student is focused, tired, or active in participating in activities. In addition, a student who listens carefully usually has a fixed head position, while a student who participates in a discussion may nod or shake his head frequently.

[0073] Optionally, in one embodiment of the present application, the behavior feature extraction unit 122 is used to input each student image in the time queue of the student images into a behavior feature extractor based on a deep separable convolutional neural network model to obtain a time queue of student object behavior semantic feature vectors. It should be understood that each student image in the time queue of the student images is input into a behavior feature extractor based on a deep separable convolutional neural network model for feature extraction to characterize the action semantic feature information about the student object in each student image, thereby obtaining a time queue of student object behavior semantic feature vectors.

[0074] Optionally, in one embodiment of the present application, the inter-feature significant interaction coding processing unit 123 is used to perform inter-feature significant interaction coding on each group of corresponding student object behavior semantic feature vectors and student object facial semantic feature vectors in the time queue of the student object behavior semantic feature vectors and the time queue of the student object facial semantic feature vectors to obtain a time queue of facial-behavior joint semantic feature vectors as the time queue of the facial-behavior joint semantic feature representation. It should be understood that since the student object behavior semantic feature vectors and the student object facial semantic feature vectors in the time queue of the student object behavior semantic feature vectors and the student object facial semantic feature vectors respectively contain the student object behavior semantic features and facial semantic features at a specific time point, the behavior features include the semantic information of the student object's body movements, such as gestures or movement direction, and the facial features involve semantic information such as expression and sight direction. In order to interactively fuse the behavior semantic features and facial semantic features of the learning object at each specific time point to generate a more comprehensive joint semantic feature representation of the student object, in the technical solution of the present application, each group of corresponding student object behavior semantic feature vectors and student object facial semantic feature vectors in the time queue of the student object behavior semantic feature vectors and the time queue of the student object facial semantic feature vectors are further subjected to inter-feature saliency interaction encoding to obtain a time queue of facial-behavioral joint semantic feature vectors as the time queue of the facial-behavioral joint semantic feature representation. The inter-feature saliency interaction encoding method can fuse the joint saliency key features and potential connections between the facial semantics and behavior semantics of the student object, thereby better reflecting the essential relationship and potential complex association between the facial features and behavior features of the student. This strengthens the expression ability of the more important semantic association features that are more relevant to the subsequent participation status assessment task, and reduces the interference of irrelevant information, thereby providing a basis for the subsequent generation of participation status levels.

[0075] Optionally, in one embodiment of the present application, the inter-feature saliency interaction coding processing unit 123 is used to perform inter-feature saliency interaction coding on each group of corresponding student object behavior semantic feature vectors and student object facial semantic feature vectors in the time queue of the student object behavior semantic feature vectors and the time queue of the student object facial semantic feature vectors using the following semantic feature joint formula to obtain a time queue of facial-behavior joint semantic feature vectors as a time queue of facial-behavior joint semantic feature representation;

[0076] The semantic feature combination formula is:

[0077]

[0078] V c =sigmoid(V J )

[0079]

[0080]

[0081]

[0082]

[0083] V1'=V1w1

[0084] V2=V2w2

[0085]

[0086] Among them, V1 and V2 are the behavior semantic feature vector of the student object and the facial semantic feature vector of the student object respectively. To add by position, W VT and b VT are the weight matrix and bias vector respectively, tanh(·) is the tanh function, V J is the joint implicit feature vector of the student object’s face-behavior semantics, sigmoid(·) is the sigmoid function, V c represents the facial-behavior semantic conditional feature vector of the student object, is the feature value of each position in the student object behavior semantic feature vector, is the feature value of each position in the facial semantic feature vector of the student object, is the eigenvalue of each position in the face-behavior semantic conditional feature vector of the student object, L is the number of eigenvalues ​​in the student object behavior semantic feature vector and the student object facial semantic feature vector, log represents the logarithmic function value with base 2, exp(·) represents the exponential function with base e as the natural constant, CI1 is the student object behavior semantic contribution, CI2 is the student object facial semantic contribution, w1 and w2 are the normalized student object behavior semantic contribution and the normalized student object facial semantic contribution, V1' and V2' are the modulated student object behavior semantic feature vector and the modulated student object facial semantic feature vector, V2' T is the transposed vector of V2', d is the length of V2', represents vector multiplication, Softmax(·) is the Softmax function, V r are the individual face-behavior joint semantic feature vectors in the temporal queue of the face-behavior joint semantic feature vectors.

[0087] In the above-mentioned student activity management system based on multidimensional data, the student face-behavior temporal information aggregation representation module 130 is used to perform temporal dynamic information propagation fusion on the time queue of the face-behavior joint semantic feature representation to obtain a multi-dimensional joint representation of student object behavior semantics. It should be understood that since each face-behavior joint semantic feature vector in the time queue of the face-behavior joint semantic feature vector respectively contains the joint semantic interaction feature representation between the facial features and behavioral features of the student object at a specific time point, and the face-behavior joint semantic interaction feature representations at different time points have a temporal association relationship, and the face-behavior joint semantics of different time sequences have different importance and contribution to the subsequent participation status assessment task. Therefore, in order to extract meaningful interactive semantic feature information from the time queue of the face-behavior joint semantic feature vector and establish connections between multiple face-behavior joint semantic time points, so as to better understand the behavior of the student object and provide a basis for the subsequent participation status level determination, in the technical solution of the present application, the time queue of the face-behavior joint semantic feature representation is further subjected to temporal dynamic information propagation fusion to obtain a multi-dimensional joint representation of student object behavior semantics. In particular, the processing method of time-series dynamic information propagation fusion is to capture the trend of node features of student object face-behavior joint semantic interaction feature representation over time by combining principal component analysis and gradient decay mechanism, and use these trends for effective information propagation and aggregation. Specifically, it performs principal component analysis on the face-behavior joint semantic feature vector at each time node to extract the most representative principal component, then calculates the principal component change rate between different time nodes and its corresponding decay entropy factor, and further introduces time span modulation and gating function to filter out important student object face-behavior joint semantic information, and finally generates a student object behavior semantic multi-dimensional joint representation vector that can reflect the dynamic changes of each time node feature (i.e., student object face-behavior joint semantic interaction feature representation) in the time series as the student object behavior semantic multi-dimensional joint representation.

[0088] Optionally, in one embodiment of the present application, Figure 4 Schematic block diagram of a student facial-behavior temporal information aggregation representation module in a multi-dimensional data-based student activity management system according to an embodiment of the present application. Figure 4As shown, the student face-behavior temporal information aggregation representation module 130 includes: a principal component extraction unit 131, which is used to extract the time queue of the current face-behavior joint semantic feature representation and the time queue of the historical face-behavior joint semantic feature representation from the time queue of the face-behavior joint semantic feature representation, and then perform principal component extraction to obtain a sequence of the current face-behavior joint semantic feature principal component representation and the historical face-behavior joint semantic principal component representation; a temporal decay dynamic aggregation processing unit 132, which is used to perform temporal decay dynamic aggregation on the sequence of the current face-behavior joint semantic feature principal component representation and the historical face-behavior joint semantic principal component representation to obtain the student object behavior semantic multi-dimensional joint representation.

[0089] Optionally, in one embodiment of the present application, the principal component extraction unit is used to: extract the current face-behavior joint semantic feature vector from the time queue of the face-behavior joint semantic feature vectors, and perform eigenvalue-based principal component extraction on the current face-behavior joint semantic feature vector to obtain the current face-behavior joint semantic feature principal component representation vector as the current face-behavior joint semantic feature principal component representation; define other face-behavior joint semantic feature vectors in the time queue of the face-behavior joint semantic feature vectors as historical face-behavior joint semantic feature vectors to obtain a sequence of historical face-behavior joint semantic feature vectors, and perform eigenvalue-based principal component extraction on each historical face-behavior joint semantic feature vector in the sequence of historical face-behavior joint semantic feature vectors to obtain a sequence of historical face-behavior joint semantic principal component representation vectors as the sequence of historical face-behavior joint semantic principal component representations.

[0090] Optionally, in one embodiment of the present application, Figure 5 Schematic block diagram of a time-series decay dynamic aggregation processing unit in a student activity management system based on multidimensional data in an embodiment of the present application. Figure 5As shown, the temporal attenuation dynamic aggregation processing unit 132 includes: a temporal propagation attenuation entropy absolute factor calculation subunit 1321, which is used to calculate the temporal propagation attenuation entropy absolute factor of each historical face-behavior joint semantic principal component representation vector in the sequence of the historical face-behavior joint semantic principal component representation vector relative to the current face-behavior joint semantic feature principal component representation vector to obtain a sequence of face-behavior joint temporal propagation attenuation entropy absolute factors; a time dimension modulation processing subunit 1322, which is used to perform time dimension modulation on each face-behavior joint temporal propagation attenuation entropy absolute factor in the sequence of the face-behavior joint temporal propagation attenuation entropy absolute factors based on the time span between each historical face-behavior joint semantic principal component representation vector in the sequence of the historical face-behavior joint semantic principal component representation vector and the current face-behavior joint semantic feature principal component representation vector to obtain a face-behavior joint temporal span modulation subunit 1322. The invention relates to a method for controlling the sequence of propagation attenuation entropy factors modulated by the face-behavior joint temporal span modulation. The method comprises the following steps: a) inputting the sequence of propagation attenuation entropy factors modulated by the face-behavior joint temporal span into the information transfer screening module based on the gating function to obtain a sequence of propagation attenuation weights modulated by the face-behavior joint temporal span modulation; b) calculating the weighted sum of the sequence of historical face-behavior joint semantic principal component representation vectors based on the sequence of propagation attenuation weights modulated by the face-behavior joint temporal span modulation to obtain a historical face-behavior joint semantic principal component significant transfer aggregation representation vector; and b) calculating the positional sum of the historical face-behavior joint semantic principal component significant transfer aggregation representation vector and the current face-behavior joint semantic feature principal component representation vector to obtain a student object behavior semantic multidimensional joint representation vector as the student object behavior semantic multidimensional joint representation.

[0091] Optionally, in one embodiment of the present application, the temporal propagation attenuation entropy absolute factor calculation subunit is used to: calculate the positional division of the corresponding positions of the historical face-behavior joint semantic principal component representation vector and the current face-behavior joint semantic feature principal component representation vector to obtain a face-behavior joint semantic feature principal component attenuation vector; calculate the base-two logarithmic function value of the absolute value of each eigenvalue of the face-behavior joint semantic feature principal component attenuation vector to obtain a face-behavior joint semantic feature principal component attenuation logarithmic vector; calculate the positional dot product of the historical face-behavior joint semantic principal component representation vector and the face-behavior joint semantic feature principal component attenuation logarithmic vector, and then perform positional multiplication on each positional eigenvalue in the obtained dot product vector to obtain a face-behavior joint temporal propagation attenuation value; calculate an exponential function of the face-behavior joint temporal propagation attenuation value with the natural constant e as the base to obtain the face-behavior joint temporal propagation attenuation entropy absolute factor.

[0092] Optionally, in one embodiment of the present application, the time dimension modulation processing subunit is used to: subtract the timestamp of the current face-behavior joint semantic feature principal component representation vector from the timestamp of the historical face-behavior joint semantic principal component representation vector, and then round down to obtain a face-behavior joint time span value; calculate an exponential function with the natural constant e as the base and the face-behavior joint time span value as the exponent to obtain a face-behavior joint time span modulation value; divide the face-behavior joint temporal propagation attenuation entropy absolute factor corresponding to the historical face-behavior joint semantic principal component representation vector by the face-behavior joint time span modulation value to obtain the face-behavior joint temporal span modulation propagation attenuation entropy factor.

[0093] Optionally, in one embodiment of the present application, the information transmission screening processing sub-unit is used to: compare each face-behavior joint timing span modulation propagation attenuation entropy factor in the sequence of the face-behavior joint timing span modulation propagation attenuation entropy factor with a predetermined threshold to obtain a sequence of the face-behavior joint timing span modulation propagation attenuation weights; wherein, in response to the face-behavior joint timing span modulation propagation attenuation entropy factor being greater than the predetermined threshold, the face-behavior joint timing span modulation propagation attenuation entropy factor greater than the predetermined threshold is input into a sigmoid function; in response to the face-behavior joint timing span modulation propagation attenuation entropy factor being less than or equal to the predetermined threshold, the face-behavior joint timing span modulation propagation attenuation entropy factor less than or equal to the predetermined threshold is set to zero.

[0094] In summary, in the embodiment of the present application, the time queue of the face-behavior joint semantic feature representation is subjected to time-series dynamic information propagation fusion using the following time-series feature propagation fusion formula to obtain the multi-dimensional joint representation of the student object behavior semantics;

[0095] Among them, the time series feature propagation fusion formula is:

[0096] X={x1,x2,...,x i ,...,x p}

[0097] v i = PCA(x i )

[0098]

[0099] V = {v1, v2, ..., v i , ..., v p}

[0100]

[0101]

[0102]

[0103]

[0104] Where X is the time sequence of the face-behavior joint semantic feature vector, x1, x2, ..., x i , ..., x p are the 1st, 2nd, ..., ith, ..., pth face-behavior joint semantic feature vectors in the time queue of the face-behavior joint semantic feature vector, PCA(x i ) is x i Perform principal component extraction based on eigenvalues, v i For x i The corresponding face-behavior joint semantic principal component representation vector, x i T For x i The transposed vector of n is the number of eigenvalues ​​in the face-behavior joint semantic feature vector, C i is x i The corresponding face-behavior joint covariance matrix, U i is x i The corresponding face-behavior joint principal component orthogonal matrix, U i T For U i The transposed matrix, v i1, v i2 , ..., v im is each face-behavior joint principal component vector in the sequence of face-behavior joint principal component vectors, Λ i For x i The corresponding face-behavior joint diagonal matrix, diag(λ i1 ,λ i2 , ..., λ im ) is the matrix whose diagonal elements are λ i1 ,λ i2 , ..., λ im The face-action joint diagonal matrix, λ i1 ,λ i2 , ..., λ im are the weights of the joint principal component vectors of each face-behavior, arg max k (·) is the k value corresponding to the maximum value returned, j is the maximum approximate matching value, V is the sequence of face-behavior joint semantic principal component representation vectors, v1, v2, ..., v i , ..., v p are respectively the 1st, 2nd, ..., ith, ..., pth face-behavior joint semantic principal component representation vectors in the sequence of the face-behavior joint semantic principal component representation vectors, v i The eigenvalues ​​at each position in , v p The eigenvalues ​​of each position in, L is v i The number of eigenvalues ​​in the equation, log represents the logarithmic function value with base 2, exp(·) represents the exponential function with the natural constant e as the base, d i v i and v p The absolute factor of the decay entropy of the face-behavior joint temporal propagation, t p and t i denote the timestamps of the pth and ith face-behavior joint semantic principal component representation vectors, respectively. For floor rounding operation, e (i→p) Yes i and v p The face-behavior joint temporal span modulates the propagation attenuation entropy factor between them, mask(·) is the masking process, sigmoid(·) is the sigmoid function, θ is the predetermined threshold, w i is the joint temporal span modulation propagation attenuation weight of each face-behavior in the sequence of the face-behavior joint temporal span modulation propagation attenuation weight, p is the number of vectors in the sequence of the face-behavior joint semantic principal component representation vector, v p+1It is the device face-behavior joint semantic temporal principal component propagation aggregate representation vector.

[0105] In the above-mentioned student activity management system based on multidimensional data, the student participation status detection module 140 is used to determine the grade label of the student's participation status based on the multidimensional joint representation of the student object behavior semantics. Optionally, in one embodiment of the present application, the student participation status detection module is used to: input the student object behavior semantics multidimensional joint representation vector into a classifier-based participation status evaluator to obtain an evaluation result, and the evaluation result is used to represent the grade label of the participation status. In other words, by performing classification processing on the student object's behavior semantics multidimensional joint feature representation, the student participation status is evaluated to obtain the grade label of the student activity participation status. In this way, the flexibility and adaptability of the student activity management system can be improved, which helps to better understand and manage the participation status of student activities.

[0106] In summary, a student activity management system based on multidimensional data according to an embodiment of the present application is explained, which includes: a camera installed in the student activity area, the camera is used to collect student images; a monitoring interface connected to the camera, the monitoring interface is used to receive and extract the student image; a participation status detector is used to receive the student image transmitted by the monitoring interface, and identify and mark the identity of the student based on the student image, and detect and upload the student's participation status; a management database is used to archive and store the student's participation status. In this way, more comprehensive feature information about student objects in the activity can be obtained through multidimensional data collection, so as to more effectively capture the complex behavior patterns of students, and identify and evaluate the status of students' participation in activities. In addition, when dealing with different scene changes, it is possible to better adapt to different scene changes by jointly analyzing the facial features and behavioral features of student objects, thereby improving the flexibility and adaptability of the student activity management system.

[0107] Figure 6 FIG. 1 is a schematic flow chart of a method for managing student activities based on multidimensional data according to an embodiment of the present application. Figure 6As shown, the student activity management method based on multidimensional data includes: S1, collecting student images through a camera; S2, receiving and extracting the student images through a monitoring interface; S3, receiving the student images transmitted by the monitoring interface, and identifying and marking the identity of the students based on the student images, and detecting and uploading the student participation status; S4, archiving and storing the student participation status; wherein, receiving the student images transmitted by the monitoring interface, identifying and marking the identity of the students based on the student images, and detecting and uploading the student participation status, includes: S31, obtaining a time queue of student images; S32, extracting and jointly representing each student image in the time queue of the student images based on facial features and behavioral features to obtain a time queue represented by facial-behavioral joint semantic features; S33, performing temporal dynamic information propagation fusion on the time queue represented by facial-behavioral joint semantic features to obtain a multi-dimensional joint representation of student object behavioral semantics; S34, determining the level label of the student's participation status based on the multi-dimensional joint representation of student object behavioral semantics.

[0108] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned student activity management method based on multidimensional data have been referred to above. Figures 1 to 5 The description of a student activity management system based on multidimensional data has been introduced in detail, and therefore, its repeated description will be omitted.

[0109] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference to a figure in a claim should not be considered as limiting the claim to which it relates.

[0110] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units stated in the device claim can also be implemented by one unit through software or hardware.

[0111] Finally, it should be noted that the above description has been given for the purpose of illustration and description. In addition, the above embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A student activity management system based on multidimensional data, characterized in that: include: A camera installed in the student activity area, the camera is used to collect images of students; A monitoring interface connected to the camera, the monitoring interface being used to receive and extract the student image; a participation status detector, for receiving the student image transmitted by the monitoring interface, identifying and marking the identity of the student based on the student image, and detecting and uploading the student's participation status; A management database for archiving and storing the student's participation status; Wherein, the participation status detector includes: A student image time series data acquisition module is used to obtain the time sequence of student images; A student face-behavior joint representation module is used to extract and jointly represent each student image in the time queue of the student images based on facial features and behavioral features to obtain a time queue of face-behavior joint semantic feature representation; A student face-behavior temporal information aggregation representation module is used to perform temporal dynamic information propagation fusion on the time queue of the face-behavior joint semantic feature representation to obtain a multi-dimensional joint representation of the student object behavior semantics; The student participation status detection module is used to determine the level label of the student's participation status based on the multi-dimensional joint representation of the student object behavior semantics.

2. A student activity management system based on multidimensional data according to claim 1, characterized in that: The student face-behavior joint representation module includes: A facial feature extraction unit, used for inputting each student image in the time queue of the student images into a facial feature extractor based on a dilated convolutional neural network model to obtain a time queue of facial semantic feature vectors of student objects; A behavior feature extraction unit, used for inputting each student image in the time queue of the student images into a behavior feature extractor based on a deep separable convolutional neural network model to obtain a time queue of student object behavior semantic feature vectors; The feature-to-feature significant interaction encoding processing unit is used to perform feature-to-feature significant interaction encoding on each group of corresponding student object behavior semantic feature vectors and student object facial semantic feature vectors in the time queue of the student object behavior semantic feature vectors and the time queue of the student object facial semantic feature vectors to obtain a time queue of facial-behavior joint semantic feature vectors as the time queue of the facial-behavior joint semantic feature representation.

3. A student activity management system based on multidimensional data according to claim 2, characterized in that: The student face-behavior temporal information aggregation representation module includes: A principal component extraction unit, configured to extract the time queue of the current face-behavior joint semantic feature representation and the time queue of the historical face-behavior joint semantic feature representation from the time queue of the face-behavior joint semantic feature representation, and then perform principal component extraction to obtain a sequence of the current face-behavior joint semantic feature principal component representation and the historical face-behavior joint semantic principal component representation; A time-decayed dynamic aggregation processing unit is used to perform time-decayed dynamic aggregation on the sequence of the current face-behavior joint semantic feature principal component representation and the historical face-behavior joint semantic principal component representation to obtain the student object behavior semantic multi-dimensional joint representation.

4. A student activity management system based on multidimensional data according to claim 3, characterized in that: The main component extraction unit is used to: Extracting a current face-behavior joint semantic feature vector from the time queue of the face-behavior joint semantic feature vector, and performing eigenvalue-based principal component extraction on the current face-behavior joint semantic feature vector to obtain a current face-behavior joint semantic feature principal component representation vector as the current face-behavior joint semantic feature principal component representation; The other facial-behavior joint semantic feature vectors in the time queue of the facial-behavior joint semantic feature vectors are defined as historical facial-behavior joint semantic feature vectors to obtain a sequence of historical facial-behavior joint semantic feature vectors, and principal component extraction based on eigenvalues ​​is performed on each historical facial-behavior joint semantic feature vector in the sequence of historical facial-behavior joint semantic feature vectors to obtain a sequence of historical facial-behavior joint semantic principal component representation vectors as the sequence of historical facial-behavior joint semantic principal component representations.

5. A student activity management system based on multidimensional data according to claim 4, characterized in that: The timing attenuation dynamic aggregation processing unit comprises: A temporal propagation attenuation entropy absolute factor calculation subunit, used to calculate the temporal propagation attenuation entropy absolute factor of each historical face-behavior joint semantic principal component representation vector in the sequence of the historical face-behavior joint semantic principal component representation vector relative to the current face-behavior joint semantic feature principal component representation vector to obtain a sequence of face-behavior joint temporal propagation attenuation entropy absolute factors; A time dimension modulation processing subunit, for performing time dimension modulation on each face-behavior joint temporal propagation attenuation entropy absolute factor in the sequence of face-behavior joint temporal propagation attenuation entropy absolute factors based on the time span between each historical face-behavior joint semantic principal component representation vector in the sequence of historical face-behavior joint semantic principal component representation vectors and the current face-behavior joint semantic feature principal component representation vector to obtain a sequence of face-behavior joint temporal span modulation propagation attenuation entropy factors; An information transfer screening processing subunit, used for inputting the sequence of face-behavior joint temporal span modulation propagation attenuation entropy factors into an information transfer screening module based on a gating function to obtain a sequence of face-behavior joint temporal span modulation propagation attenuation weights; A vector sequence weighted sum processing subunit, for calculating the weighted sum of the sequence of historical face-behavior joint semantic principal component representation vectors based on the sequence of face-behavior joint temporal span modulation propagation attenuation weights to obtain a historical face-behavior joint semantic principal component significant transfer aggregation representation vector; The student object behavior semantics multidimensional joint processing subunit is used to calculate the positional sum of the historical face-behavior joint semantics principal component significant transfer aggregation representation vector and the current face-behavior joint semantics feature principal component representation vector to obtain the student object behavior semantics multidimensional joint representation vector as the student object behavior semantics multidimensional joint representation.

6. A student activity management system based on multidimensional data according to claim 5, characterized in that: The time series propagation attenuation entropy absolute factor calculation subunit is used to: Calculating the positional division of the historical face-behavior joint semantic principal component representation vector and the current face-behavior joint semantic feature principal component representation vector at corresponding positions to obtain a face-behavior joint semantic feature principal component attenuation vector; Calculating the base-two logarithmic function value of the absolute value of each eigenvalue of the face-behavior joint semantic feature principal component attenuation vector to obtain the face-behavior joint semantic feature principal component attenuation logarithmic vector; After calculating the positional dot product of the historical face-behavior joint semantic principal component representation vector and the face-behavior joint semantic feature principal component attenuation logarithm vector, each position feature value in the obtained dot product vector is multiplied by position point to obtain the face-behavior joint temporal propagation attenuation value; An exponential function of the face-behavior joint temporal propagation attenuation value with the natural constant e as the base is calculated to obtain the face-behavior joint temporal propagation attenuation entropy absolute factor.

7. A student activity management system based on multidimensional data according to claim 6, characterized in that: The time dimension modulation processing subunit is used for: Subtracting the timestamp of the current face-behavior joint semantic feature principal component representation vector from the timestamp of the historical face-behavior joint semantic principal component representation vector and rounding down to obtain a face-behavior joint time span value; Calculating an exponential function with the natural constant e as the base and the face-behavior joint time span value as the exponent to obtain a face-behavior joint time span modulation value; The face-behavior joint temporal propagation attenuation entropy absolute factor corresponding to the historical face-behavior joint semantic principal component representation vector is divided by the face-behavior joint time span modulation value to obtain the face-behavior joint temporal span modulation propagation attenuation entropy factor.

8. A student activity management system based on multidimensional data according to claim 7, characterized in that: The information transmission screening processing subunit is used for: Comparing each face-behavior joint temporal span modulation propagation attenuation entropy factor in the sequence of face-behavior joint temporal span modulation propagation attenuation entropy factors with a predetermined threshold to obtain a sequence of face-behavior joint temporal span modulation propagation attenuation weights; Among them, in response to the face-behavior joint timing span modulation propagation attenuation entropy factor being greater than the predetermined threshold, the face-behavior joint timing span modulation propagation attenuation entropy factor greater than the predetermined threshold is input into the sigmoid function; in response to the face-behavior joint timing span modulation propagation attenuation entropy factor being less than or equal to the predetermined threshold, the face-behavior joint timing span modulation propagation attenuation entropy factor less than or equal to the predetermined threshold is set to zero.

9. A student activity management system based on multidimensional data according to claim 8, characterized in that: The student participation status detection module is used to: input the student object behavior semantic multi-dimensional joint representation vector into a classifier-based participation status evaluator to obtain an evaluation result, and the evaluation result is used to represent the level label of the participation status.

10. A student activity management method based on multidimensional data, characterized in that: include: Collect student images through cameras; Receiving and extracting the student image through a monitoring interface; Receiving the student image transmitted by the monitoring interface, identifying and marking the identity of the student based on the student image, and detecting and uploading the student's participation status; Archiving and storing the student's participation status; The steps of receiving the student image transmitted by the monitoring interface, identifying and marking the identity of the student based on the student image, and detecting and uploading the student's participation status include: Get the time queue of the student images; Extracting and jointly representing each student image in the time queue of student images based on facial features and behavioral features respectively to obtain a time queue represented by facial-behavior joint semantic features; Performing temporal dynamic information propagation fusion on the time queue of the face-behavior joint semantic feature representation to obtain a multi-dimensional joint representation of the student object behavior semantics; Based on the multi-dimensional joint representation of the student object behavior semantics, a grade label of the student's participation status is determined.

Images