Intelligent management system and method based on campus digital twinning

Abstract

The invention relates to an intelligent management system and method based on campus digital twinning, and the system carries out the mining of potential data elements through the recognition and analysis of the behavior and emotional characteristics of a person in a campus, carries out the visual data display through the combination of a data report and a digital twinning mode, and achieves the intelligent management of the campus digital twinning. Behavior and emotion recognition and analysis results are closely combined with daily teaching; digital twins of the campus environment and personnel are established, and the campus environment and personnel states (positioning, behavior and emotional states) are visually displayed; reference can be provided for formulation and implementation of campus management policies and teaching schemes, and then the campus management efficiency and the classroom teaching effect are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of smart management technology, and in particular to a smart management system based on campus digital twins and a smart management method and electronic equipment based on campus digital twins. Background Art

[0002] A smart campus is an intelligent campus environment built based on technologies such as the Internet of Things, cloud computing, big data, and artificial intelligence. The introduction of new technologies can effectively improve teaching quality and campus management efficiency and reduce safety risks.

[0003] In recent years, the recognition and analysis of people's behaviors and emotions have become a new application hotspot in smart campuses. Through sensors such as cameras, students' concentration, hand-raising times and other behaviors are monitored in real time, and their emotional states, such as confusion and excitement, are identified through facial expression analysis. Students' classroom performance is then digitized and refined, helping teachers adjust teaching strategies and improve teaching effectiveness. Behavior recognition can also be used to monitor and identify abnormal behaviors on campus, such as fighting and falling, and issue warnings in a timely manner.

[0004] In addition, by making full use of various sensors on campus, the activity trajectories of people on campus can be effectively collected. Using this technology, students' activities on campus can be digitized more finely, and it can also be used to track intruders and ensure campus safety.

[0005] On the other hand, campus activity data also needs an intuitive display method. Digital twin is a technology that uses digital means to create a virtual model of a physical entity or system to simulate and monitor its behavior. It has a wide range of applications in industrial manufacturing, medical health, transportation and other fields. Digital twin can also be used to display campus data. By building a virtual campus model, it can reflect the status and changes of the real campus environment and the real-time trajectory of people on campus in real time.

[0006] However, the campus environment has complex building layouts, diverse activity scenarios, and specific security and management requirements. Some existing solutions only provide general designs, or apply management system designs from other scenarios such as factories and residential areas. In practice, these solutions cannot fully meet the needs of campus management.

[0007] Some solutions lack intuitiveness in information display, mainly displaying real-time data in the form of graphic reports, without targeted design of personnel trajectories, behaviors, emotional states, etc. Users cannot quickly retrieve data elements and understand the displayed content when viewing data.

[0008] On the other hand, digital twin technology has good display effects and is applied in various industries such as industrial simulation and transportation. However, existing digital twin solutions usually focus on simulation, and most models are real models of target entities, which is not conducive to privacy protection.

[0009] In addition, in existing solutions based on behavior and emotion analysis, most stop at the generation of data reports and lack linkage at the system level and subsequent teaching links. As an important part of teaching feedback, behavior analysis should be closely combined with subsequent teaching activities to achieve personalized teaching and precise intervention. There is still room for improvement in existing solutions in this direction. Summary of the Invention

[0010] On the one hand, the present application proposes an intelligent management system based on campus digital twin, including:

[0011] A data acquisition module for acquiring dynamic data of the campus environment, campus facilities and personnel on campus, including corresponding audio-visual camera data, sensing data, positioning and behavior data, and / or facial emotion images;

[0012] A personnel positioning and behavior recognition module for identifying the positioning information, behavior and / or emotion state information of personnel on campus based on the dynamic data;

[0013] A digital twin module for constructing digital twin models of the campus and personnel on campus based on digital twin technology according to the dynamic data of the campus environment, campus facilities and personnel on campus, and updating and managing the digital twin models of personnel on campus based on the real-time updated positioning information, behavior and / or emotion state information of personnel on campus;

[0014] A storage module for storing system data;

[0015] A client for logging in to the console and participating in campus management;

[0016] A console for controlling the above modules and teaching affairs management;

[0017] The data acquisition module, the personnel positioning and behavior recognition module, the digital twin module, the storage module and the client are respectively communicatively connected to the console.

[0018] As an optional implementation scheme of the present application, optionally, the data acquisition module includes at least one of the following devices:

[0019] Cameras, microphones, infrared light sensors, NFC or RFID card readers.

[0020] As an optional implementation scheme of the present application, optionally, the personnel positioning and behavior recognition module includes:

[0021] A regional positioning unit for identifying the campus area where a person on campus is located;

[0022] An in-region precise positioning unit for identifying the coordinate position of a person on campus in the campus area;

[0023] A behavior recognition unit for real-time recognizing behavior / emotion features in the behavior images and / or facial emotion images of a person on campus based on a preset CNN-dynamic recognition model, and outputting corresponding feature recognition results.

[0024] As an optional implementation of the present application, optionally, the calculation method of the coordinate position:

[0025] Let:

[0026]

[0027] Wherein:

[0028] (x i , y i ) is the precise coordinate of person Pi in the region,

[0029] S is the number of multi-source data types,

[0030] W s is the weight of data source S,

[0031] C s，m (P i ) is the coordinate of person Pi in the m-th detection of data source S,

[0032] Confidence(C s，m ) is the confidence of the detection result.

[0033] As an optional implementation of the present application, optionally, the generation method of the CNN-dynamic recognition model includes:

[0034] Preparing a number of groups of historical behavior images and historical facial emotion images of persons on campus;

[0035] Using a CNN network to extract behavior features in the historical behavior images and emotion features in the historical facial emotion images respectively;

[0036] Assigning corresponding behavior labels and emotion labels to the behavior features and the emotion features respectively, and the feature values of the corresponding features are recorded on the labels;

[0037] Counting each of the behavior features and the emotion features, and dividing them into corresponding training sets and validation sets according to a preset ratio;

[0038] Input the training set of the behavioral features and the training set of the emotional features into a preset dual-branch lightweight CNN model in sequence to perform multi-task joint training on the behavioral features and the emotional features, and construct the initial CNN-dynamic recognition model; wherein, the dual-branch lightweight CNN model includes a first convolutional base branch and a second convolutional base branch respectively used for processing the behavioral features and processing the emotional features, and each convolutional base branch includes a convolutional layer and a fully connected layer;

[0039] Use the validation set of the behavioral features and the validation set of the emotional features to verify the recognition performance of the CNN-dynamic recognition model in sequence:

[0040] If the recognition of both behavior and emotion passes the verification, then deploy and apply the CNN-dynamic recognition model;

[0041] Otherwise, repeat the above steps to reconstruct the CNN-dynamic recognition model.

[0042] As an optional implementation of this application, optionally, the console is further configured to:

[0043] According to the recognition result of the personnel positioning and behavior recognition module, monitor and judge whether the personnel on campus show behavioral features with behavior labels of positive evaluation:

[0044] If it appears, record the occurrence frequency and time period of the appearance, and extract the behavior frame images of the personnel on campus at the time point corresponding to the occurrence frequency from the behavior video, and write the behavior frame images into a preset first highlight archive and store them in the storage module;

[0045] If it does not appear, generate a first warning notice corresponding to the behavioral features and send it to the teaching system where the corresponding teaching staff is located and / or the client where the family members of the personnel on campus are located;

[0046] and / or

[0047] According to the recognition result of the personnel positioning and behavior recognition module, monitor and judge whether the personnel on campus show emotional features with emotion labels of positive evaluation:

[0048] If it appears, record the occurrence frequency and time period of the appearance, and extract the facial frame images of the personnel on campus at the time point corresponding to the occurrence frequency from the facial video, and write the facial frame images into a preset second highlight archive and store them in the storage module;

[0049] If it does not appear, generate a second warning notice corresponding to the emotional features and send it to the teaching system where the corresponding teaching staff is located and / or the client where the family members of the personnel on campus are located.

[0050] As an alternative implementation of this application, optionally, the console is further configured to:

[0051] Statistical performance data of personnel on campus this semester, including:

[0052] Read the course schedule, homework after class, and grades this semester of personnel on campus from the educational administration system,

[0053] Obtain the behavior characteristics and emotion characteristics of personnel on campus from the recognition results output by the personnel positioning and behavior recognition module;

[0054] Generate a portrait benchmark of personnel on campus this semester based on the performance data this semester;

[0055] Compare the portrait benchmark this semester with the historical portrait benchmark of the previous semester to determine whether the portrait benchmark this semester exceeds the allowable value of the historical portrait benchmark of the previous semester:

[0056] If it exceeds, generate a corresponding third warning notice and send it to the educational administration system where the corresponding educational administration personnel are located and / or the client where the families of the personnel on campus are located;

[0057] Otherwise, give up.

[0058] As an alternative implementation of this application, optionally, the console is further configured to:

[0059] Statistical emotion characteristics of personnel on campus at the time points of each occurrence frequency during a preset learning stage, and output the characteristic values marked in the emotion labels on each emotion characteristic;

[0060] Calculate the average value of the characteristic values marked in the emotion labels on each emotion characteristic to obtain an emotion state evaluation value of the personnel on campus during this learning stage;

[0061] Compare the emotion state evaluation value with a preset emotion baseline:

[0062] If the emotion state evaluation value is not lower than the emotion baseline, continue monitoring;

[0063] If the emotion state evaluation value is lower than the emotion baseline, generate a corresponding psychological warning notice and send it to the educational administration system where the corresponding educational administration personnel are located and / or the client where the families of the personnel on campus are located.

[0064] On the other hand, this application proposes an intelligent management method based on campus digital twins, which is implemented based on the intelligent management system based on campus digital twins described in the above item, and includes the following steps:

[0065] Collect dynamic data of the campus environment, campus facilities, and personnel on campus, including corresponding audio and video camera data, sensing data, positioning and behavior data, and / or facial emotion images;

[0066] Based on the dynamic data, identify the positioning information, behavior, and / or emotion state information of personnel on campus;

[0067] Based on the positioning information, behavior, and / or emotion state information of personnel on campus that is updated in real time, perform update management on the digital twin model of personnel on campus.

[0068] On the other hand, this application also proposes an electronic device, including:

[0069] A processor;

[0070] A memory for storing instructions executable by the processor;

[0071] Wherein, when the processor is configured to execute the executable instructions, it implements the described intelligent management method based on campus digital twin.

[0072] Technical effects of the present invention:

[0073] The system proposed by the present invention identifies and analyzes the behavior and emotion characteristics of personnel on campus, excavates potential data elements, and combines data reports and digital twins for intuitive data display, closely integrating the results of behavior and emotion recognition and analysis with daily teaching; establishes digital twins of the campus environment and personnel, intuitively displaying the campus environment and personnel status (positioning, behavior, and emotion state); can provide reference for the formulation and implementation of campus management policies and teaching plans, thereby improving campus management efficiency and classroom teaching effect.

[0074] According to the detailed description of the exemplary embodiments with reference to the following accompanying drawings, other features and aspects of the present disclosure will become clear. Brief Description of the Drawings

[0075] The accompanying drawings included in the specification and constituting a part of the specification show the exemplary embodiments, features, and aspects of the present disclosure together with the specification, and are used to explain the principles of the present disclosure.

[0076] Figure 1 Shown as a schematic structural diagram of the system composition of the present invention;

[0077] Figure 2 Shown as a schematic structural diagram of the dual-branch lightweight CNN model of the present invention;

[0078] Figure 3 Shown as a schematic flowchart of student behavior and emotion recognition of the present invention;

[0079] Figure 4 It shows a cartoon schematic diagram of the student status displayed on the APP side of the present invention;

[0080] Figure 5 It shows an auxiliary teaching schematic diagram of the system of the present invention;

[0081] Figure 6 It shows an application schematic diagram of the electronic device of the present invention. Detailed implementation manners

[0082] Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0083] The special term "exemplary" herein means "serving as an example, embodiment, or illustration". Any embodiment described as "exemplary" herein need not be construed as superior to or better than other embodiments.

[0084] In addition, for a better description of the present disclosure, numerous specific details are given in the following detailed implementation manners. Those skilled in the art should understand that the present disclosure can also be implemented without some specific details. In some instances, means, elements, and circuits well known to those skilled in the art are not described in detail so as to highlight the gist of the present disclosure.

[0085] Embodiment 1

[0086] As Figure 1 shown, a smart management system based on campus digital twin includes:

[0087] A data acquisition module for collecting dynamic data of the campus environment, campus facilities, and personnel within the campus, including corresponding audio-visual camera data, sensing data, positioning and behavior data, and / or facial emotion images;

[0088] A personnel positioning and behavior recognition module for identifying the positioning information, behavior, and / or emotion state information of personnel within the campus based on the dynamic data;

[0089] A digital twin module for constructing digital twin models of the campus and personnel within the campus based on digital twin technology according to the dynamic data of the campus environment, campus facilities, and personnel within the campus, and updating and managing the digital twin models of personnel within the campus based on the real-time updated positioning information, behavior, and / or emotion state information of personnel within the campus;

[0090] A storage module for storing system data;

[0091] A client for logging in to the console and participating in campus management;

[0092] A console for controlling the above-mentioned modules and educational administration management.

[0093] The data acquisition module, personnel positioning and behavior recognition module, digital twin module, storage module, and the client are respectively communicatively connected to the console.

[0094] This system mainly combines digital twin, behavior and emotion recognition technologies for the background visualization management of campus personnel. The personnel on campus are mainly students. The system includes six modules: data acquisition, digital twin, personnel positioning and behavior recognition, storage, console, and client. Among them, the data acquisition module collects campus environment, facilities, and personnel data, transmits it to the personnel positioning and behavior recognition module to extract the action trajectory, current behavior, and emotion characteristics of personnel, and then reports it to the console. The console further analyzes the extracted data and updates the virtual model in the digital twin module to reflect the state changes of the corresponding entity. The client can connect to the console to receive notifications or actively query data. During the operation of the system, all data is saved by the storage module.

[0095] The system can also be connected to other educational administration systems for data sharing, which can be specifically understood in combination with the subsequent description.

[0096] Digital twin technology can map the entities of the campus environment and campus personnel to the virtual space, generate corresponding virtual models (digital twin models), and update the digital twin models according to the real-time collected and updated data.

[0097] Next, the functions and interaction controls of each module of the system will be described separately.

[0098] As an optional implementation of this application, optionally, the data acquisition module includes at least one of the following devices:

[0099] Cameras, microphones, infrared light sensors, NFC or RFID card readers.

[0100] 1. Data Acquisition

[0101] The system data covers multimedia data such as videos, images, audios, and texts, and also covers card swiping records, network access records, etc. Correspondingly, the system acquisition devices include cameras, microphones, infrared light sensors, NFC, and RFID card readers, and the network access records can be obtained through the system background.

[0102] The system acquisition device has a unique ID in the system and can bind its installation location and functional attributes. When a device is newly added, removed, or the same device is migrated to a different installation location, it can be reconfigured through the device ID. According to business needs, the camera can be additionally calibrated so that the image coordinates can correspond to the real-world coordinates of the captured entity.

[0103] As long as it can record data within the campus and data on students' behaviors within the campus, it can be applied.

[0104] 2. Digital Twin

[0105] The environment and personnel within the campus are represented by digital twins in the system. The generation methods of digital twins include two types: manual modeling and 3D scanning modeling. The former obtains 2D or 3D maps, paintings, and cartoon images through manual drawing; the latter obtains the model of the original entity by collecting videos, pictures, or 3D point cloud data of the target real scene. When the state and position of the entity in reality change, its corresponding digital twin changes accordingly.

[0106] The form of the digital twin can be the same as the entity or an artistic image with similar characteristics. One entity can correspond to multiple digital twin forms and can be configured according to business needs.

[0107] At the same time, the digital twin can effectively isolate the original data from the external display interface. By converting specific data and presenting it in the form of a virtual image externally, it can effectively protect the original identity of personnel and is more conducive to the system for permission control, protecting the personal privacy and data security of teachers and students on campus.

[0108] The console synchronizes the student's trajectory, behavior, and emotional state to the digital twin module, updating the position and (behavior and / or emotional) state of the corresponding mark of the student. In the parent version of the digital twin, a corresponding cartoon animation will be additionally played.

[0109] Two API ports can be deployed on the console, which are respectively used for teachers and family members to log in through the corresponding clients (understood in combination with the subsequent application embodiments of the digital twin). A corresponding educational administration management system (for teachers to log in and participate in educational administration management) and a family member management system (for students' family members to log in and participate in management) can be deployed on the console. It can be understood and implemented in combination with the existing educational administration system, but for the specific management content of each management system, please understand and configure it in combination with the application services of teachers and family members in the present invention.

[0110] 3. Positioning and Behavior, Emotional State Recognition

[0111] As an optional implementation scheme of the present application, optionally, the personnel positioning and behavior recognition module includes:

[0112] The area positioning unit is used to identify the campus area where the person is located;

[0113] An intra-region precise positioning unit is used to identify the coordinate positions of people on campus in the campus area;

[0114] The behavior recognition unit is used to recognize the behavior / emotion features in the behavior images and / or facial emotion images of people on campus in real time based on the preset CNN-dynamic recognition model, and output the corresponding feature recognition results.

[0115] Personnel positioning includes two types: positioning in the area to which they belong and precise positioning within the area.

[0116] Regional positioning is used to identify the area where people are located, such as playgrounds, classrooms, offices, canteens, etc. Regional positioning relies on the video, images and card swiping records at the entrances and exits of the area. When a specific person is detected entering the entrance and exit, the person is assigned to the area. Regional positioning can be understood in combination with the positioning monitoring technology of the existing monitoring area.

[0117] Precise positioning in the area further describes the detailed coordinates of the personnel, such as the personnel's seat number, etc., based on the regional positioning. This function relies on the video, image and location annotation in the area. The present invention provides a method for calculating positioning coordinates:

[0118] As an optional implementation scheme of the present application, optionally, the coordinate position is calculated as follows:

[0119] make:

[0120]

[0121] in:

[0122] (x i ,y i ) is the precise coordinate of person Pi in the area,

[0123] S is the number of multi-source data types, for example, S=3 corresponds to video, image, and sensor.

[0124] W s is the weight of the data source S, such as video weight w1=0.6, image w2=0.3, sensor w3=0.1;

[0125] C s，m (P i ) is the coordinates of person Pi in the mth detection of the data source S, such as video frame coordinates and image annotation coordinates, which can be identified and calculated by the coordinate system of the monitoring system;

[0126] Confidence(C s，m) is the confidence level of the detection result, such as the confidence level of video detection and the accuracy of image annotation, which is set by default in the system or by the administrator.

[0127] The weighted average of the multiple detection results for each data source (video, image, sensor) is calculated, and the weights are jointly determined by the reliability of the data source itself (W s ) and the confidence level of a single detection. The final coordinates fuse the results of all data sources to ensure that high-confidence data dominates the positioning.

[0128] The behavior of a person is jointly determined by the video, image, regional attributes, and time period within the area. When the person's body posture, facial expression, and the objects they interact with are clearly visible in the video or image, the person's behavior and their emotional state can be directly recognized. For example, the person is currently standing up to speak and has a happy expression.

[0129] When the video or image is unavailable, the person's behavior is approximately determined by the area and time. For example, when the person is on the playground and it is currently physical education class time, their behavior is recorded as exercising. In this case, the emotional state of the person cannot be judged.

[0130] The recognition of behavior and state can be carried out through a CNN - Convolutional Neural Network. The dynamic recognition model constructed by the present invention using CNN can real - time recognize the behavior images and facial emotions of people on campus and output corresponding feature results. The structure and training application principles of the convolutional neural network are not elaborated here.

[0131] To take care of the high - efficiency processing performance of the model, the present invention considers lightweight CNN architectures, such as MobileNet or EfficientNet, so as to have a high processing efficiency in the multi - task processing of behavior and emotion recognition. By adopting a dual - branch lightweight CNN model, the emotional features in the behavior actions and facial expressions of students are trained in parallel. To reduce the complexity of model training, a dual - branch lightweight CNN model is used as the initial model for model training.

[0132] As Figure 2 shown, it includes a first convolutional base branch and a second convolutional base branch respectively for processing the behavior features and processing the emotional features. Each of the convolutional base branches includes a convolutional layer and a fully - connected layer.

[0133] First, the basic structure of a CNN includes convolutional layers, pooling layers, activation functions, etc. Therefore, the structure of the dual-branch lightweight CNN model designed in the present invention adopts a dual-branch structure, one for processing behavior and one for processing emotion, and then fuses the results. Or design a multi-task learning model that shares underlying features, and the upper branches process different tasks separately. For example, the input image passes through a shared convolutional base and then is divided into two branches, each with its own convolutional layer and fully connected layer. For behavior recognition, it may output the behavior category, and for emotion recognition, it outputs the emotion category. The loss function may need to combine the losses of the two tasks, such as weighted cross-entropy loss, and combine the original CNN design loss function, etc.

[0134] In terms of real-time performance, the model needs to be lightweight, using depthwise separable convolutions instead of standard convolutions to reduce the number of parameters.

[0135] Secondly, the convolutional layers of the first convolutional base branch and the second convolutional base branch, which are respectively used to process the behavior features and the emotion features, can be expressed as follows:

[0136] Let:

[0137]

[0138] Where:

[0139] F behavior is to extract the spatio-temporal dynamic features (such as movement trajectories, limb postures) in the human action R (facial image in the following branch);

[0140] X in ∈R H×W×C is the input image or feature map (height H, width W, number of channels C);

[0141] W behavior ∈R k×k×C×D is the behavior feature convolution kernel (k×k is the kernel size, D is the number of output channels);

[0142] M motion ∈R H×W×1 is the motion feature map (extracted by optical flow method or temporal difference);

[0143] α∈[0, 1] is the dynamic attention weight (calculated from the motion intensity);

[0144] is the feature map concatenation;

[0145] ReLU is the activation function used for detection, and ReLU is used as the activation function of Conv2D (a model function for creating a two-dimensional convolutional layer).

[0146] Through the motion feature map Mmotion Enhance spatio-temporal features and focus on significant motion regions.

[0147] The lightweight convolutional kernel (k = 3×3, D = 64) ensures real-time performance.

[0148] Let:

[0149] F emotion = GAP(DepthwiseConv2D(X in ,W emotion )⊙M facial ),

[0150] Where:

[0151] W emotion ∈R 3×3×C is a depthwise separable convolutional kernel (lightweight design);

[0152] GAP is global average pooling (Global Average Pooling), which outputs a feature vector;

[0153] F emotion extracts local detailed features of facial expressions (such as changes in eyebrows and corners of the mouth caused by emotional changes);

[0154] M facial ∈R H×W×1 is a facial key point mask (generated by a pre-trained model, annotating the eye, nose, and mouth regions, and can be trained and constructed based on the CNN principle);

[0155] ⊙ is: element-wise multiplication (Masking operation, suppressing noise in non-face regions).

[0156] Depthwise separable convolution (Depthwise) reduces the number of parameters (80% less than standard convolution). By M facial forcing the model to focus on key facial regions, improving the robustness of emotion classification.

[0157] The structural features of the above two branch convolutional layers are shown in Table 1 below:

[0158]

[0159]

[0160] Table 1

[0161] Based on the above model, the present invention will conduct model training.

[0162] As an optional implementation of this application, optionally, the method for generating the CNN-dynamic recognition model includes:

[0163] Prepare several groups of historical behavior images and historical facial emotion images of people on campus;

[0164] Use a CNN network to extract the behavior features from the historical behavior images and the emotion features from the historical facial emotion images respectively;

[0165] Assign corresponding behavior labels and emotion labels to the behavior features and the emotion features respectively, and the feature values of the corresponding features are recorded on the labels;

[0166] Count each of the behavior features and the emotion features, and divide them into corresponding training sets and validation sets according to a preset ratio;

[0167] Input the training set of the behavior features and the training set of the emotion features into a preset dual-branch lightweight CNN model in sequence to perform multi-task joint training on the behavior features and the emotion features, and construct the initial CNN-dynamic recognition model; wherein, the dual-branch lightweight CNN model includes a first convolutional base branch and a second convolutional base branch respectively used to process the behavior features and the emotion features, and each convolutional base branch includes a convolutional layer and a fully connected layer;

[0168] Use the validation set of the behavior features and the validation set of the emotion features (such as in a ratio of 7:3) to verify the recognition performance of the CNN-dynamic recognition model in sequence:

[0169] If the recognition of both behavior and emotion passes the verification, then deploy and apply the CNN-dynamic recognition model;

[0170] Otherwise, repeat the above steps to reconstruct the CNN-dynamic recognition model.

[0171] Collect the previous behavior images and facial images of several students, perform image feature recognition and extraction using a CNN, and extract the corresponding action features and emotion features reflected by facial expressions respectively. Extract and save the image features reflected by the corresponding actions and emotions on the images and assign labels. For example, the behaviors include 5 types of labels: raising the hand (front), standing up (front), reading and writing (front), lying on the table, listening (front) (the labels carry corresponding feature values, such as the raising the hand label represents the feature value "5", and so on), and the emotions include 6 types of labels: calm (front), happy (front), sad, surprised, afraid, angry (the labels carry corresponding feature values, such as the calm label represents the feature value "6", and so on; if it is found that the label feature value output is 3, it means being in a sad mood). Specifically, the administrator can perform labeling and annotation.

[0172] Regarding the training process of the CNN model, it will not be elaborated in this embodiment. One can understand and train by combining the CNN and the structure of the dual-branch lightweight CNN model of the present invention described above.

[0173] After the model is constructed, the performance of the model can be verified using the validation set. For example, verification metrics such as accuracy or AUC value can be used for verification. Specifically, it can be verified by the administrator inputting the validation set. If the model achieves the indicators in Table 2 below, it means the verification passes:

[0174]

[0175] Table 2

[0176] 4. Storage

[0177] The storage module is used to save various system data for other modules to call. The stored content includes:

[0178] Device attributes, such as device type, ID, installation location, and functions, etc.;

[0179] Data uploaded by data acquisition devices, including videos, audios, pictures, card swiping records, etc.;

[0180] Digital twin models, such as campus environment models, facility models, personnel models, etc.;

[0181] Educational administration information, such as class schedules, lists, etc.;

[0182] Campus personnel attributes, such as names, student IDs, or employee IDs, etc.;

[0183] Platform user attributes, such as usernames, passwords, IPs, permissions, etc.;

[0184] Operation caches, such as target tracking status, etc.;

[0185] Big data analysis results, such as classroom activity, etc.;

[0186] Alarm information, such as personnel intrusion records, etc.;

[0187] System deployment and software operation-related configurations, such as service interface IPs, UI configurations, etc.

[0188] 5. Console

[0189] The console is the core of the system, and its functions cover the following aspects:

[0190] Device management and configuration

[0191] The console provides comprehensive device management and functions. Administrators can access and configure infrastructure such as servers and networks, as well as various collection devices such as cameras and card readers through the console, and deploy and manage algorithms on relevant devices. They can also monitor the running status of hardware and algorithm tasks, perform fault diagnosis and maintenance to ensure the stable operation of the system.

[0192] Educational Affairs Access

[0193] The console supports the access of the educational affairs system and conducts data interaction such as course arrangement, basic student information, behavior and emotion statistics. The console queries course and student information from the educational affairs system to assist in the recognition of student behavior and emotions, and returns the recognition and analysis results to the educational affairs system to be associated with student classroom performance, grades, etc. Teachers can directly query the comprehensive statistical results from the educational affairs system to provide reference for the formulation of teaching plans.

[0194] Digital Twin Management

[0195] The console provides functions for creating, managing and monitoring digital twins. Administrators can import virtual models through the console and bind them to the entities of people, facilities and environment on campus. When new data is collected or entered, the console can update the status of the virtual model in real time to reflect the status changes of its corresponding entity.

[0196] Data Analysis

[0197] The console integrates data analysis tools and supports the statistics, analysis and processing of campus environmental changes, personnel behavior and emotion data. Users can collect, clean, analyze and visualize data through the console to mine the implicit elements of the data, thereby assisting in the formulation and implementation of campus management policies and teaching plans.

[0198] Security Alarm

[0199] The console provides comprehensive campus security management and alarm functions. Administrators can configure monitoring areas and abnormal behaviors through the console, such as fighting in the classroom. The console further issues monitoring instructions to downstream devices and systems according to the configuration. When abnormal behaviors are monitored, the console can send alarms through various methods such as emails, text messages, push notifications, etc., to ensure that relevant personnel can respond and handle security incidents in a timely manner and guarantee campus security.

[0200] Data Display

[0201] The console provides rich data display functions, supporting various data visualization methods, such as charts, dashboards, reports, etc. Users can intuitively view the running status and performance metrics of the system through the console, and can also query the data analysis results of the campus environment and personnel behavior and emotions. The console also supports custom data display, and administrators can formulate display methods and rules by themselves to meet the business requirements of different scenarios.

[0202] 6. Client

[0203] The client is used to view system information, and its implementation methods include web pages, computer programs, mobile APPs, and mini-programs based on third-party platforms, such as WeChat and Alipay mini-programs. The client connects to the console through the network, receives unified message pushes, and can also actively query specific information, such as student class schedules and the current classrooms. The client needs to log in using an account, and the account is allocated and controlled by the console.

[0204] The following is the application implementation description of this system.

[0205] As an optional implementation solution of this application, optionally, the console is further used for:

[0206] According to the recognition results of the personnel positioning and behavior recognition module, monitor and judge whether the personnel on campus show behavioral characteristics with behavior tags of positive evaluation:

[0207] If it appears, record the occurrence frequency and time period of the appearance, and extract the behavior frame images of the personnel on campus at the time points corresponding to the occurrence frequency from the behavior video, and write the behavior frame images into a preset first highlight collection file and store it in the storage module;

[0208] If it does not appear, generate a first warning notice corresponding to the behavioral characteristics and send it to the educational administration system where the corresponding educational administration personnel are located and / or the client where the families of the personnel on campus are located;

[0209] and / or

[0210] According to the recognition results of the personnel positioning and behavior recognition module, monitor and judge whether the personnel on campus show emotional characteristics with emotion tags of positive evaluation:

[0211] If it appears, record the occurrence frequency and time period of the appearance, and extract the facial frame images of the personnel on campus at the time points corresponding to the occurrence frequency from the facial video, and write the facial frame images into a preset second highlight collection file and store it in the storage module;

[0212] If it does not appear, generate a second warning notice corresponding to the emotional characteristics and send it to the educational administration system where the corresponding educational administration personnel are located and / or the client where the families of the personnel on campus are located.

[0213] The highlight archive is mainly a storage database of frame images, which can save the images of students' behaviors or positive emotion facial expressions with positive label annotations at different time points. For example, extract and save the action images of students raising their hands or happy facial expression images, and write them into the highlight archive. Several frame images in a certain period can be extracted and saved to form a continuous highlight archive.

[0214] The form of the alarm notification can be customized by the administrator.

[0215] For specific positive marked behaviors and emotions, the administrator can just mark them. For example:

[0216] Combined with Figure 3 As shown, the system uses a camera to collect videos during classroom teaching and identify the behaviors and emotional states of each student. When positive behaviors such as raising hands and standing up to speak are detected, or when the student is in a happy mood, the current video frame is saved separately as a highlight. After the entire class, the overall classroom performance of the students is statistically analyzed, and corresponding data analysis is carried out to generate a report.

[0217] The behaviors in the report include 5 types of labels: raising hands (positive), standing up (positive), reading and writing (positive), lying on the table, listening (positive) (each label has a corresponding characteristic value. For example, the raising hands label represents the characteristic value "5", and so on), and the emotions include 6 types of labels: calm (positive), happy (positive), sad, surprised, afraid, angry (each label has a corresponding characteristic value. For example, the calm label represents the characteristic value "6", and so on; if the label characteristic value output is 3, it means being in a sad mood). The report contains the occurrence frequencies and time periods of the above behaviors and emotions, gives alarm reminders for outliers in the data, and indicates individuals with relatively prominent data. Teachers can view the statistical report of the entire class through the console or the teaching affairs system connected to it after class. Parents can view the statistical data related to their children and the highlights of the class through the client.

[0218] The system proposed based on the present invention can identify students' campus activities and display them in real time. Through the system console, information about the campus environment and facilities can be imported in advance to establish a digital twin of the campus. The overall layout of the campus park, each teaching building, the distribution of classrooms in the building, the playground and fitness facilities all have corresponding virtual models in the digital twin, and there are also corresponding markers for the personnel inside the school. Teachers and students are represented by different markers, and students in different classes are also represented by different markers.

[0219] Such as Figure 4As shown in the figure, in order to distinguishably realize the visual display and viewing of students' status, the digital twin is divided into two versions: the teacher version and the parent version. The digital twin of the teacher version adopts a simulation model. The morphological features of the model are similar to those of the real entity, and the geometric distance between the models is proportional to the actual distance between their entities, so as to intuitively observe the campus environment and quickly locate the target area. Teachers can also directly access the cameras in the target area through the digital twin display interface to watch the real-time monitoring video.

[0220] The digital twin of the parent version uses cartoon images that have a certain similarity to the entities. The geometric distance between the models can be appropriately adjusted to adapt to the display effect of the client. Through permission control, the digital twin of the parent version only displays information related to the access account and cannot access school hardware facilities such as cameras.

[0221] The campus activity trajectories of students are recorded through a variety of sensors and positioning technologies. First, devices such as access control gates and entrance and exit cameras are used to determine whether a student enters an area. If there are no relevant devices in the area, the student is searched for through the cameras inside the area. If there are no information collection devices in the area, the student's current location is approximately determined based on the student's last known location and the day's class schedule. When the student enters an area equipped with data collection devices again, the trajectory is verified immediately.

[0222] When a student is in the teaching area, the cameras in the area are used to identify the student's behavior and emotions. The identification results are reported to the console in real time. The console synchronizes the student's trajectory, behavior, and emotional state to the digital twin module, updating the position and (behavior and / or emotion) state of the corresponding marker of the student. In the digital twin of the parent version, a corresponding cartoon animation will be played additionally.

[0223] As Figure 5 shown, the system proposed by the present invention can also be connected to the educational administration system or the medical system to provide data services in a more efficient form. The system proposed by the present invention analyzes the classroom behavior and emotions of students, compares the analysis data with the students' course progress, the quantity of homework after class, and their grades, and forms a portrait benchmark for each student. When the later data fluctuates relative to the baseline, a reminder is sent to the teacher in a timely manner.

[0224] As an optional implementation of this application, optionally, the console is further configured to:

[0225] Statistical data on the performance of personnel on campus this semester, including:

[0226] Read the course arrangements, homework after class, and grades of personnel on campus this semester from the educational administration system,

[0227] Obtain the behavioral characteristics and emotional characteristics of the personnel on campus from the recognition results output by the personnel positioning and behavior recognition module;

[0228] Generate a benchmark portrait of the personnel on campus in this semester based on the performance data of this semester;

[0229] Compare the benchmark portrait of the personnel in this semester with the historical portrait benchmark of the previous semester to determine whether the benchmark portrait of the personnel in this semester exceeds the allowable value of the historical portrait benchmark of the previous semester:

[0230] If it exceeds, generate a corresponding third warning notice and send it to the educational administration system where the corresponding educational administration personnel are located and / or the client where the families of the personnel on campus are located;

[0231] Otherwise, give up.

[0232] The behavioral and emotional performances of students in each semester can be recognized through the above-mentioned solution. By combining the learning situation of students and their behavioral and emotional characteristic performances, a portrait of the students is constructed (such as the comparison data between the grades of this semester and the characteristic values of the corresponding behaviors and emotions), and the performance benchmark (grade baseline) for the next semester is set. It is convenient to discover the changes in students' grades by combining the changes in students' behavioral and emotional characteristics, correct students' behaviors and emotions in a timely manner, and enable parents and teachers to compare and analyze the grade changes based on the behavioral and emotional characteristics, and adjust students' behaviors and emotions in a timely manner.

[0233] As Figure 5 shown, the system can also be connected to the medical system, especially medical services related to mental health. When the emotional state of a student shows a significant fluctuation relative to the benchmark value, prompt the teacher or a professional psychiatrist to communicate with him / her to prevent the student from having mental problems.

[0234] As an optional implementation solution of this application, optionally, the console is further configured to:

[0235] Statistically analyze the emotional characteristics of the personnel on campus at the time points of each occurrence frequency within a preset learning stage, and output the characteristic values marked in the emotional labels of each emotional characteristic;

[0236] Calculate the average value of the characteristic values marked in the emotional labels of each emotional characteristic to obtain the emotional state evaluation value of the personnel on campus in this learning stage;

[0237] Compare the emotional state evaluation value with a preset emotional baseline:

[0238] If the emotional state evaluation value is not lower than the emotional baseline, continue to monitor;

[0239] If the emotional state evaluation value is lower than the emotional baseline, a corresponding psychological warning notice is generated and sent to the educational administration system where the corresponding educational staff is located and / or the client where the family members of the campus personnel are located.

[0240] It is also possible to compare and analyze the emotional changes of students and timely notify teachers to introduce psychological correction strategies for students.

[0241] It is possible to count the emotional feature recognition results of students in this week every other week (at least once a day for identification and statistics), and according to the emotional labels on the emotional features output by each identification, read the feature values marked therein. Calculate the mean value of each feature value to obtain the emotional state evaluation value of the campus personnel at this learning stage. This reflects the average emotional state of students in school this week from the side. For example, if the mean value is lower than 4 points, it means that they are not very happy on campus. At this time, an alarm is issued to prompt teachers or professional psychologists to communicate with them to avoid students having psychological problems.

[0242] Obviously, those skilled in the art should understand that to implement all or part of the processes in the above embodiments, it can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above control embodiments. Those skilled in the art can understand that to implement all or part of the processes in the above embodiments, it can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above control embodiments. Among them, the storage medium can be a magnetic disk, an optical disc, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memories.

[0243] Embodiment 2

[0244] Based on the implementation principle of Embodiment 1, on the other hand, this application proposes an intelligent management method based on campus digital twins, which is implemented based on the intelligent management system based on campus digital twins described in the above item, and includes the following steps:

[0245] Collect dynamic data of the campus environment, campus facilities and campus personnel, including corresponding audio and video camera data, sensing data, positioning and behavior data, and / or facial emotion images;

[0246] Identify the location information, behavior, and / or emotional state information of the personnel on campus based on the dynamic data;

[0247] Based on the location information, behavior, and / or emotional state information of the personnel on campus that is updated in real time, update and manage the digital twin model of the personnel on campus.

[0248] For the above method steps, please understand and implement them in combination with the description in Embodiment 1. This embodiment will not be elaborated here.

[0249] Each module or step of the present invention described above can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module for implementation. In this way, the present invention is not limited to any specific combination of hardware and software.

[0250] Embodiment 3

[0251] As Figure 6 shown, further, on the other hand, the present application also proposes an electronic device, including:

[0252] A processor;

[0253] A memory for storing instructions executable by the processor;

[0254] Wherein, when the processor is configured to execute the executable instructions, it implements the method of a smart management system based on campus digital twins described above.

[0255] The electronic device in the embodiment of the present disclosure includes a processor and a memory for storing instructions executable by the processor. Wherein, when the processor is configured to execute the executable instructions, it implements the smart management method based on campus digital twins described in any of the foregoing.

[0256] Here, it should be noted that the number of processors can be one or more. At the same time, in the electronic device of the embodiment of the present disclosure, an input device and an output device can also be included. Among them, the processor, the memory, the input device, and the output device can be connected through a bus or in other ways, which will not be specifically limited here.

[0257] The memory, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and various modules, such as the programs or modules corresponding to a smart management method based on campus digital twins according to an embodiment of the present disclosure. The processor executes various functional applications and data processing of the electronic device by running the software programs or modules stored in the memory.

[0258] The input device can be used to receive input numbers or signals. Among them, the signal can be a key signal related to the user settings and function control of the device / terminal / server. The output device can include display devices such as a display screen.

[0259] The above has described the embodiments of the present disclosure. The above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations are obvious to those of ordinary skill in the art in the technical field without departing from the scope and spirit of the described embodiments. The selection of the terms used herein is intended to best explain the principles of the embodiments, the practical application, or the technical improvement of the technology in the market, or to enable other ordinary skill in the art in the technical field to understand the embodiments disclosed herein.

Claims

1. A smart management system based on campus digital twin, characterized in that, Including: A data acquisition module for acquiring dynamic data of the campus environment, campus facilities, and personnel on campus, including corresponding audio and video camera data, sensing data, positioning and behavior data, and / or facial emotion images; A personnel positioning and behavior recognition module for identifying the positioning information, behavior, and / or emotion state information of personnel on campus based on the dynamic data; A digital twin module for constructing digital twin models of the campus and personnel on campus based on digital twin technology according to the dynamic data of the campus environment, campus facilities, and personnel on campus, and updating and managing the digital twin models of personnel on campus based on the real-time updated positioning information, behavior, and / or emotion state information of personnel on campus; A storage module for storing system data; A client for logging in to the console and participating in campus management; A console for controlling the above modules and for educational administration management; The data acquisition module, the personnel positioning and behavior recognition module, the digital twin module, the storage module, and the client are respectively communicatively connected to the console.

2. The intelligent management system based on campus digital twin according to claim 1, characterized in that, The data acquisition module includes at least one of the following devices: A camera, a microphone, an infrared light sensor device, an NFC or RFID card reader.

3. A smart management system based on campus digital twin according to claim 1, characterized in that, The personnel positioning and behavior recognition module includes: A regional positioning unit for identifying the campus area where a person on campus is located; An in-region precise positioning unit for identifying the coordinate position of a person on campus in the campus area; A behavior recognition unit for real-time recognizing behavior / emotion features in the behavior images and / or facial emotion images of personnel on campus based on a preset CNN-dynamic recognition model and outputting corresponding feature recognition results.

4. The intelligent management system based on campus digital twin according to claim 3, characterized in that, The calculation method of the coordinate position: Let: Where: (x i , y i ) is the exact coordinate of person Pi within the area. S is the number of multi-source data types, W s is the weight of data source S C s，m (P i ) is the coordinate of person Pi in the m-th detection of data source S. Confidence(C s，m ) is the confidence level of the detection result.

5. The intelligent management system based on campus digital twin according to claim 3, characterized in that, The generation method of the CNN-dynamic recognition model includes: Preparing several groups of historical behavior images and historical facial emotion images of personnel on campus; Using a CNN network to extract behavior features from the historical behavior images and emotion features from the historical facial emotion images respectively; Assigning corresponding behavior labels and emotion labels to the behavior features and the emotion features respectively, and the labels record the feature values of the corresponding features; Counting each of the behavior features and the emotion features and dividing them into corresponding training sets and validation sets according to a preset ratio; Sequentially inputting the training set of the behavior features and the training set of the emotion features into a preset dual-branch lightweight CNN model for multi-task joint training of the behavior features and the emotion features to construct the initial CNN-dynamic recognition model; wherein, the dual-branch lightweight CNN model includes a first convolutional base branch and a second convolutional base branch respectively used for processing the behavior features and processing the emotion features, and each convolutional base branch includes a convolutional layer and a fully connected layer; Using the validation set of the behavior features and the validation set of the emotion features to sequentially verify the recognition performance of the CNN-dynamic recognition model: If the recognition of both behavior and emotion passes the verification, then deploy and apply the CNN-dynamic recognition model; Conversely, repeat the above steps to reconstruct the CNN-dynamic recognition model.

6. The intelligent management system based on campus digital twin according to claim 5, wherein, The console is further configured to: According to the recognition result of the personnel positioning and behavior recognition module, monitor and determine whether the behavior characteristics with positive evaluation behavior labels appear in the campus personnel: If it appears, record the occurrence frequency and time period, extract the behavior frame images of the campus personnel at the time point corresponding to the occurrence frequency from the behavior video, write the behavior frame images into a preset first highlight collection file and store them in the storage module; If it does not appear, generate a first warning notice corresponding to the behavior characteristics and send it to the educational administration system where the corresponding educational administration personnel are located and / or the client where the family members of the campus personnel are located; and / or According to the recognition result of the personnel positioning and behavior recognition module, monitor and determine whether the emotional characteristics with positive evaluation emotion labels appear in the campus personnel: If it appears, record the occurrence frequency and time period, extract the facial frame images of the campus personnel at the time point corresponding to the occurrence frequency from the facial video, write the facial frame images into a preset second highlight collection file and store them in the storage module; If it does not appear, generate a second warning notice corresponding to the emotional characteristics and send it to the educational administration system where the corresponding educational administration personnel are located and / or the client where the family members of the campus personnel are located.

7. A smart management system based on campus digital twin according to claim 5, characterized in that, The console is further configured to: Statistically analyze the performance data of the campus personnel in this semester, including: Read the course arrangements, after-class assignments and semester grades of the campus personnel from the educational administration system, Obtain the behavior characteristics and emotional characteristics of the campus personnel from the recognition results output by the personnel positioning and behavior recognition module; Based on the performance data in this semester, generate a portrait benchmark of the campus personnel in this semester; Compare the portrait benchmark in this semester with the historical portrait benchmark in the previous semester to determine whether the portrait benchmark in this semester exceeds the allowable value of the historical portrait benchmark in the previous semester: If it exceeds, generate a corresponding third warning notice and send it to the educational administration system where the corresponding educational administration personnel are located and / or the client where the family members of the campus personnel are located; Conversely, give up.

8. A smart management system based on campus digital twin according to claim 7, characterized in that, The console is further configured to: Statistically analyze the emotional characteristics of the campus personnel at the time points corresponding to each occurrence frequency within a preset learning stage, and output the characteristic values marked in the emotion labels on each emotional characteristic; Calculate the average value of the characteristic values marked in the emotion labels on each emotional characteristic to obtain the emotional state evaluation value of the campus personnel in this learning stage; Compare the emotional state evaluation value with a preset emotional baseline: If the emotional state evaluation value is not lower than the emotional baseline, continue to monitor; If the emotional state evaluation value is lower than the emotional baseline, generate a corresponding psychological warning notice and send it to the educational administration system where the corresponding educational administration personnel are located and / or the client where the family members of the campus personnel are located.

9. A smart management method based on campus digital twin, which is implemented based on the smart management system based on campus digital twin described in any one of claims 1-8, characterized in that, It includes the following steps: Collect the dynamic data of the campus environment, campus facilities and campus personnel, including the corresponding audio and video camera data, sensing data, positioning and behavior data, and / or facial emotion images; Identify the location information, behavior, and / or emotional state information of the personnel on campus based on the dynamic data; Based on the location information, behavior, and / or emotional state information of the personnel on campus that is updated in real time, update and manage the digital twin model of the personnel on campus.

10. An electronic device, characterized in that, It includes: A processor; A memory for storing the executable instructions that can be executed by the processor; Wherein, when the processor is configured to execute the executable instructions, it implements the intelligent management method based on campus digital twin described in claim 9.

Images