Educational place management method and system based on monitoring big data

Abstract

The embodiment of the present application provides an educational venue management method and system based on monitoring big data, which achieves in-depth analysis of the behavior patterns of the target educational venue by extracting vectors from basic monitoring video images and student behavior record documents, and performing multi-semantic level video and behavior feature domain transformation. In particular, between each two semantic levels that are related to each other, the feature interaction aggregation mechanism effectively integrates video monitoring and student behavior data, enhancing the accuracy and comprehensiveness of feature expression. Ultimately, the method generates summary screen data that covers both the educational venue environment and student behavior characteristics, providing intuitive and efficient data support for educational venue management, and significantly improving the accuracy and response speed of management decisions.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to an education venue management method and system based on monitoring big data. Background Art

[0002] In the daily management of educational places, how to efficiently and accurately grasp and analyze student behavior and the state of the place environment has always been the focus of managers. Traditional educational place management methods mostly rely on manual monitoring and recording, which is not only time-consuming and laborious, but also difficult to achieve real-time analysis and in-depth mining of a large amount of monitoring data. With the continuous development of monitoring technology and the widespread application of big data technology, educational place management methods based on monitoring big data are gradually emerging.

[0003] At present, there are mainly the following problems in the processing of big monitoring data in educational places: first, the amount of monitoring data is large and complex, making it difficult to use it directly for analysis and management; second, there is a lack of effective connection and integration between monitoring data and student behavior records, which leads to one-sided and inaccurate analysis results; third, when processing monitoring data, traditional methods often only focus on features at a single level, ignoring behavioral patterns and feature interactions at multiple semantic levels. Summary of the invention

[0004] In order to at least overcome the above-mentioned deficiencies in the prior art, the purpose of the embodiments of the present application is to provide an educational venue management method and system based on monitoring big data.

[0005] According to one aspect of the present application, a method for managing an educational venue based on monitoring big data is provided, the method comprising:

[0006] Obtain basic surveillance video footage and student behavior record documents of target educational venues from surveillance big data;

[0007] Extracting vectors from the basic surveillance video screen and the student behavior record document respectively to generate a surveillance video screen vector and a student behavior vector;

[0008] Performing a multi-semantic level video feature domain transformation on the surveillance video screen vector to generate a behavior pattern vector of the target educational venue in the basic surveillance video screen at multiple semantic levels;

[0009] The student behavior vectors are respectively subjected to multi-semantic level behavior feature domain transformation; wherein the semantic level of the multi-semantic level behavior feature domain transformation is the same as the semantic level of the multi-semantic level video feature domain transformation; between every two behavior feature domain transformations of the semantic levels associated with each other, the behavior feature domain transformation result generated by the preceding behavior feature domain transformation of the two semantic levels associated with each other and the behavior pattern vector of the corresponding semantic level are subjected to feature interaction aggregation; when the subsequent behavior feature domain transformation of the two semantic levels associated with each other is subjected to behavior feature domain transformation, the student behavior vector and the feature interaction aggregation result after the feature interaction aggregation are subjected to behavior feature domain transformation;

[0010] A summary is generated for the feature interaction aggregation vector generated after the multi-semantic hierarchical behavior feature domain transformation, and summary screen data is generated that simultaneously represents the education venue vector of the target education venue and the student behavior description vector of the student behavior record document, and the summary screen data is output as education venue management basis data.

[0011] According to one aspect of an embodiment of the present application, an education venue management system based on monitoring big data is provided, the education venue management system based on monitoring big data includes a processor and a machine-readable storage medium, the machine-readable storage medium stores machine executable instructions, the machine executable instructions are loaded and executed by the processor to implement the education venue management method based on monitoring big data in any one of the possible implementations described above.

[0012] In the technical solutions provided in some embodiments of the present application, the embodiments of the present application achieve in-depth analysis of the behavior patterns of the target educational venues by performing vector extraction on basic surveillance video images and student behavior record documents, and performing multi-semantic level video and behavior feature domain transformation. In particular, between each two semantic levels associated with each other, the feature interaction aggregation mechanism effectively integrates video surveillance and student behavior data, enhancing the accuracy and comprehensiveness of feature expression. Ultimately, the method generates summary screen data that covers both the educational venue environment and student behavior characteristics, providing intuitive and efficient data support for educational venue management, and significantly improving the accuracy and response speed of management decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be extracted in combination with these drawings without creative work.

[0014] Figure 1 A flowchart of an educational venue management method based on monitoring big data provided in an embodiment of the present application;

[0015] Figure 2 A schematic block diagram of the architecture of an education venue management system based on monitoring big data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following description is intended to enable one of ordinary skill in the art to implement and incorporate the present application, and the description is provided in the context of a specific application scenario and its requirements. It will be apparent to one of ordinary skill in the art that various changes may be made to the disclosed embodiments, and that the general principles defined in the present application may be applied to other embodiments and application scenarios without departing from the principles and scope of the present application.

[0017] Figure 1 This is a flow chart of an educational venue management method based on monitoring big data provided by an embodiment of the present application. The educational venue management method based on monitoring big data is introduced in detail below.

[0018] Step S110, obtaining basic surveillance video images and student behavior record documents of the target educational place from the surveillance big data.

[0019] In detail, the basic surveillance video footage refers to the unprocessed raw video data captured from the surveillance cameras installed in the target educational venues (such as classrooms in teaching buildings). These video data record various activities that occur in the classroom, including the behavior of students and teachers. The student behavior record document is an electronic record of student behavior in the teaching building stored in the school student management system. These records may include attendance, classroom performance, recess activities, etc., and are stored in a structured text form (such as XML or JSON) for easy parsing and processing.

[0020] For example, in this embodiment, the campus network center of a comprehensive school is responsible for managing and processing data related to the school's educational venues. In this school, there are multiple educational venues such as teaching buildings, libraries, and gymnasiums. The target educational venue of the server this time is set to a classroom area on a certain floor of the teaching building. This area contains multiple classrooms, each of which is equipped with a surveillance camera to ensure the safety of students and supervise the teaching order.

[0021] For student behavior record documents, the school has a dedicated student management system. The student management system records various student behavior information in the teaching building, such as attendance, classroom performance, recess activities, etc. These records are stored in the database in the form of electronic documents and are associated with each student's student number.

[0022] The monitoring system in the teaching building uses network cameras, which transmit the real-time video images captured to the specified storage area of the server through the campus network. For example, the server receives video data from the cameras in each classroom of the teaching building every day during the period from morning classes to afternoon school dismissal. These video data are the original unprocessed monitoring video images, which contain various activity images of students and teachers in the classroom, such as scenes of attending classes, group discussions, and teachers' lectures.

[0023] Each video file is named and classified for storage according to information such as date and classroom number for subsequent query and processing. For example, a video file named "20230915-Classroom 301.mp4" represents the monitoring video of Classroom 301 on September 15, 2023.

[0024] The server obtains the student behavior record document from the database of the school's student management system. Taking a certain class as an example, the head teacher of this class will regularly update the students' behavior records in the student management system. For example, for student Xiaoming, his behavior record document may contain information such as his attendance status (full attendance) in the recent week, the number of times he answered questions in class (3 times), and whether there was any behavior violating classroom discipline (none). These records exist in a structured text form, possibly stored in XML or JSON format for easy parsing and subsequent processing by the server.

[0025] Step S120, respectively perform vector extraction on the basic monitoring video image and the student behavior record document to generate a monitoring video image vector and a student behavior vector.

[0026] Specifically, the monitoring video image vector is a numerical vector generated from the basic monitoring video image through the vector extraction process. It represents the characteristics of the content in the video image, such as the outlines of people, the shapes of objects, and the color distribution. The student behavior vector is a numerical vector generated from the student behavior record document through the vector extraction process. It summarizes the behavior characteristics of students in the teaching building, such as attendance status and classroom performance.

[0027] For example, in this embodiment, the server is configured with a monitoring video vector generator module. When receiving the basic monitoring video image (such as the "20230915-Classroom 301.mp4" video mentioned above), this module will perform frame-by-frame analysis on the video.

[0028] For example, for each frame of the video, the monitoring video vector generator will first convert the pixel information of the frame into a digital matrix. Assuming the resolution of a frame is 640x480, then a 640x480 digital matrix will be obtained, where each element represents the color value of a pixel (possibly RGB value or grayscale value).

[0029] Then, this module will extract features from this digital matrix. It can calculate the outline features of the characters in the picture, the shape features of the objects, the color distribution features, etc. For example, the outline position of the characters is determined by the edge detection algorithm, and the distribution ratio of the main colors in the picture is counted by the color histogram.

[0030] Next, these extracted features are combined into a vector according to certain rules. Assuming that 100 feature values ​​are obtained after feature extraction, a 100-dimensional surveillance video screen vector will be generated. This vector can represent the content characteristics of this frame to a certain extent. For the entire video, the surveillance video vector generator may perform such processing on multiple frames (such as taking one frame every 10 frames), and then combine or average the vectors corresponding to these frames to finally obtain a surveillance video screen vector that can represent the entire basic surveillance video screen.

[0031] For example, in this embodiment, for a student behavior record document, the server calls a behavior vector generator. Taking the previously mentioned Xiao Ming's behavior record document as an example, if it is in JSON format, the behavior vector generator will parse the structure of this document.

[0032] In detail, different types of behavior information can be converted into corresponding numerical values ​​or vector representations. For example, attendance can be represented by 0 (absent) or 1 (present); the number of times a question is answered in class can be directly used as a numerical value; whether or not a class discipline is violated can be represented by -1 (violation), 0 (none), or 1 (good performance).

[0033] Then, these values ​​are combined into a vector in a certain order. Assuming that the document records 5 different types of behavior information, a 5-dimensional student behavior vector will be generated. This vector can summarize Xiao Ming's behavior characteristics in the teaching building. For the student behavior record documents of the entire class or the entire teaching building, the behavior vector generator will perform similar processing on each student's record, and then combine these student behavior vectors together to form a student behavior vector that can represent the entire student behavior record document.

[0034] Step S130, performing multi-semantic level video feature domain transformation on the surveillance video screen vector, and generating a behavior pattern vector of the target educational place in the basic surveillance video screen at multiple semantic levels.

[0035] In detail, multi-semantic level video feature domain transformation is the process of converting surveillance video image vectors into vectors representing behavior patterns at multiple semantic levels. Each semantic level provides a higher level of understanding of the video content, such as the basic action patterns of characters, the purpose of behavior, the interactive relationship between characters, etc.

[0036] The behavior pattern vector is a vector generated during the multi-semantic level video feature domain transformation process, and represents the behavior pattern of the target educational venue at a specific semantic level.

[0037] In this embodiment, the surveillance video coding network in the server is pre-trained with a large amount of surveillance video data, and includes multiple semantic coding nodes. Assume that the surveillance video coding network has 3 semantic coding nodes.

[0038] When a multi-semantic level video feature domain transformation is to be performed on a surveillance video picture vector, the first semantic level video feature domain transformation is performed first.

[0039] Using the first-level semantic coding node among multiple semantic coding nodes, this first-level semantic coding node can be regarded as a module for preliminary semantic understanding of the original surveillance video screen vector. For example, the basic action patterns of the characters in the screen can be identified, such as standing, sitting, walking, etc. If the surveillance video screen vector is 100-dimensional, after the transformation of the first-level semantic coding node, a 50-dimensional vector may be obtained. This vector represents the behavior pattern vector of the target educational venue (classroom) at the first semantic level, which preliminarily summarizes the basic behavior patterns of the characters in the video screen.

[0040] Then, the second semantic level video feature domain transformation is performed. The behavior pattern vector (50-dimensional vector) of the first semantic level is transformed using the second semantic coding node. This second semantic coding node may further analyze the purpose of the character's behavior, such as whether they are listening, discussing, or doing other things. After this node transformation, a 30-dimensional vector may be obtained, which is the behavior pattern vector of the target educational venue at the second semantic level.

[0041] Finally, the third semantic level video feature domain transformation is performed. The behavior pattern vector (30-dimensional vector) of the second semantic level is transformed using the third semantic encoding node. This node may analyze the interaction between characters, such as group interaction or individual independent learning. After the transformation, a 20-dimensional vector is obtained, which is the behavior pattern vector of the target educational venue at the third semantic level.

[0042] Step S140, performing multi-semantic level behavior feature domain transformation on the student behavior vector respectively. The semantic level of the multi-semantic level behavior feature domain transformation is the same as the semantic level of the multi-semantic level video feature domain transformation. Between every two behavior feature domain transformations of the previously associated semantic levels, feature interaction aggregation is performed on the behavior feature domain transformation result generated by the previous behavior feature domain transformation of the two previously associated semantic levels and the behavior pattern vector of the corresponding semantic level. When performing the subsequent behavior feature domain transformation of the two previously associated semantic levels, a behavior feature domain transformation is performed on the student behavior vector and the feature interaction aggregation result after the feature interaction aggregation.

[0043] In detail, multi-semantic level behavior feature domain transformation is the process of transforming student behavior vectors into vectors representing behavior features at multiple semantic levels. Similar to video feature domain transformation, each semantic level provides a higher level of understanding of student behavior records.

[0044] Feature interaction aggregation is the process of combining and interacting the results of behavioral feature domain transformation and the behavioral pattern vector of the corresponding semantic level between two related semantic levels. This aggregation helps to integrate the relevant information between video content and student behavior to provide a more comprehensive analysis.

[0045] For example, in this embodiment, the behavior coding network in the server is also pre-trained, and it includes multiple cascaded second feature conversion units, a behavior focusing unit is connected between each two adjacent second feature conversion units, and the number of second feature conversion units is the same as the number of first feature conversion units in the surveillance video coding network (here it is assumed to be 3).

[0046] Before the student behavior vector is transformed into the first semantic level behavior feature domain, a random representation vector of the behavior coding network is generated based on the preset random vector generation mechanism. For example, this random vector is a 10-dimensional random number vector. Then this random representation vector and the previously generated student behavior vector (assuming it is 5-dimensional) are loaded into the behavior coding network as input vectors.

[0047] The first-level second feature conversion unit among the multiple second feature conversion units is used to perform the first semantic-level behavior feature domain transformation on the student behavior vector and the random representation vector. For example, the first-level second feature conversion unit may perform an association analysis on the attendance in the student behavior vector and a dimension in the random vector, and perform a combined calculation on the number of questions answered in class and other dimensions of the random vector. After this transformation, the first semantic-level representation vector of the student behavior record document after the first semantic-level behavior feature domain transformation is generated, assuming that this vector is 8-dimensional.

[0048] After the student behavior vector is transformed into the second semantic level behavior feature domain, the second behavior focusing unit in the behavior encoding network is used to perform feature interactive aggregation on the second semantic level representation vector (assumed to be 6-dimensional) after the second semantic level behavior feature domain transformation and the behavior pattern vector (30-dimensional) generated by the second first feature conversion unit in the surveillance video encoding network.

[0049] First, the second semantic level representation vector and the behavior pattern vector generated by the second first feature conversion unit are merged in the vector component dimension to generate a 36-dimensional merged vector. Then, the merged vector is subjected to upper limit compression and mean compression respectively. For example, the upper limit compression operation will limit certain values ​​in the vector to a specific range, and the mean compression operation will calculate the average value of each component of the vector and adjust it. After the upper limit compression vector and the mean compression vector are generated, they are output as compressed vectors (assuming 20 dimensions). Then, the compressed vector is mapped to generate an association array between the second semantic level representation vector and the behavior pattern vector generated by the second first feature conversion unit. Based on this association array, the merged vector is subjected to feature interaction aggregation to generate the second feature interaction aggregation result (assuming 15 dimensions).

[0050] When the student behavior vector is transformed into the third semantic level behavior feature domain, the third second feature conversion unit among the multiple second feature conversion units is used to transform the student behavior vector and the second feature interaction aggregation result (15 dimensions) generated by the second behavior focus unit into the third semantic level behavior feature domain. This transformation may comprehensively consider the previous behavior information and interaction results to generate the third semantic level representation vector (assuming it is 10 dimensions) of the student behavior record document after the third semantic level behavior feature domain transformation. Finally, the representation vector (10 dimensions) generated after the last semantic level behavior feature domain transformation of the multi-semantic level behavior feature domain transformation is output as the feature interaction aggregation vector generated after the multi-semantic level behavior feature domain transformation.

[0051] Step S150, summarize the feature interaction aggregation vector generated after the multi-semantic hierarchical behavior feature domain transformation, generate summary screen data that simultaneously represents the education venue vector of the target education venue and the student behavior description vector of the student behavior record document, and output the summary screen data as education venue management basis data.

[0052] In this embodiment, the server is configured with a summary generation module. After receiving the feature interaction aggregation vector (10 dimensions) generated after the multi-semantic level behavior feature domain transformation, this module will perform a summary generation operation.

[0053] For example, each dimension of this 10-dimensional vector can be mapped to specific educational venue characteristics and student behavior descriptions according to predefined rules. Suppose the first dimension of the vector represents the overall activity of the classroom (a value between 0 and 1, where 0 means very inactive and 1 means very active), the second dimension represents the student's learning concentration (also a value between 0 and 1), and so on.

[0054] Through this mapping, summary screen data is generated that simultaneously represents the target educational venue (classroom) educational venue vector (such as a summary vector of classroom activity, order, etc.) and the student behavior description vector of the student behavior record document (such as a summary vector of student concentration, discipline compliance, etc.). This summary screen data may be a structured data block that contains key feature descriptions of the educational venue and student behavior.

[0055] Finally, the server outputs this summary screen data as the basis for educational venue management. School administrators can use this data to evaluate and manage the teaching situation in the classrooms in the teaching building and the learning status of students. For example, if the summary screen data shows that the classroom activity is low, administrators can consider adjusting teaching methods or organizing more interactive activities; if the student behavior description vector shows that the student's concentration is low, teachers can take measures to improve students' attention, etc.

[0056] Based on the above steps, the embodiment of the present application achieves in-depth analysis of the behavior patterns of the target educational venues by extracting vectors from basic surveillance video images and student behavior record documents, and performing multi-semantic level video and behavior feature domain transformation. In particular, between each two semantic levels that are related to each other, the feature interaction aggregation mechanism effectively integrates video surveillance and student behavior data, enhancing the accuracy and comprehensiveness of feature expression. Ultimately, the method generates summary screen data that covers both the educational venue environment and student behavior characteristics, providing intuitive and efficient data support for educational venue management, and significantly improving the accuracy and response speed of management decisions.

[0057] In a possible implementation, step S130 includes: performing a multi-semantic level video feature domain transformation on the surveillance video picture vector based on a plurality of semantic coding nodes in a surveillance video coding network that has previously performed parameter learning, and generating a behavior pattern vector of the target educational place in the basic surveillance video picture at multiple semantic levels.

[0058] Among them, when the monitoring video picture vector is transformed into the first semantic level video feature domain, the first-level semantic coding node among the multiple semantic coding nodes is used to perform the first semantic level video feature domain transformation on the monitoring video picture vector to generate the behavior pattern vector of the target education place in the basic monitoring video picture at the first semantic level.

[0059] When the surveillance video screen vector is transformed into the Xth semantic level video feature domain, the Xth semantic coding node among the multiple semantic coding nodes is used to perform the Xth semantic level video feature domain transformation on the behavior pattern vector of the X-1th semantic level to generate the behavior pattern vector of the target education place in the basic surveillance video screen at the Xth semantic level.

[0060] In this embodiment, the surveillance video coding network is previously trained with a large amount of annotated surveillance video data, and the surveillance video coding network is designed to understand and extract features from surveillance video images at multiple semantic levels. It contains multiple semantic coding nodes, which are the key parts of the network to achieve feature conversion at different semantic levels.

[0061] The server obtains the surveillance video screen vector from the previous step. Assuming that this surveillance video screen vector is generated from the surveillance video of a classroom in the teaching building, the vector has a high dimension and contains various feature information in the video screen, such as the position, posture, and distribution of objects of the characters.

[0062] The first-level semantic coding node begins to transform the surveillance video image vector into the first semantic-level video feature domain. This first-level semantic coding node is like a basic filter and feature extractor.

[0063] First, identify the basic elements in the picture, such as people and common objects (tables and chairs, etc.). For people, you can focus on the outline information of the person. Through a series of algorithms, the pixel-related features in the surveillance video screen vector are converted into a description of the person's outline. For example, if the original vector contains information such as the grayscale value change of the edge pixels of the person, the first-level semantic encoding node will convert it into the general shape features of the person, such as whether it is a standing human outline, a sitting outline, etc.

[0064] At the same time, for objects, the basic layout of tables and chairs can be identified. For example, based on the relative position and shape characteristics of the tables and chairs in the picture, it can be judged whether the tables and chairs are arranged neatly or in a disorderly manner.

[0065] After being processed by the first-level semantic coding nodes, a behavior pattern vector of the target educational venue in the basic surveillance video is generated at the first semantic level. This vector may focus on describing the basic status of people and objects in the picture, such as how many people are standing in the classroom, the general layout of tables and chairs, etc.

[0066] When performing the Xth semantic level video feature domain transformation, the server will input the behavior pattern vector of the X-1th semantic level into the Xth semantic encoding node. Assume that the behavior pattern vector of the X-1th semantic level has described the basic status of people and objects in the classroom. For example, this vector indicates that some people are walking around in the classroom, and the tables and chairs have partial changes.

[0067] The Xth semantic coding node will conduct a deeper analysis based on this. If X=2, this semantic coding node may focus on the purpose of the character's actions. For those who are walking, it can be analyzed whether they are walking around during normal class activities or have special behaviors (such as looking for objects, chasing and playing, etc.).

[0068] For changes in tables and chairs, it can be determined whether they are adjustments made during normal use (such as moving tables and chairs closer together during group discussions) or abnormal movements (such as accidentally knocking over tables and chairs).

[0069] This semantic encoding node generates a behavior pattern vector of the target educational venue at the Xth semantic level in the basic surveillance video screen by analyzing, combining and converting the information in the behavior pattern vector at the X-1th semantic level. This new vector will contain higher-level semantic information, such as the atmosphere of activities in the classroom (is it a normal and orderly activity or is there a sign of chaos).

[0070] Assume that there are three semantic coding nodes (i.e., three semantic levels) in the surveillance video coding network. After completing the transformation of the first semantic level, the transformation of the second and third semantic levels will be performed in sequence.

[0071] At the third semantic level, the semantic encoding node may analyze the interaction between characters and group behavior patterns. For example, is it the interaction between individual students or the collective activity of the entire class? And it will combine the information of the first two semantic levels, such as the basic status of the characters and the purpose of the actions, to fully describe the behavior patterns in the classroom, the target educational venue.

[0072] Finally, through the transformation of the video feature domain at these three semantic levels, the server obtains vectors that comprehensively describe the behavior patterns of the target educational venue (classroom) at multiple semantic levels. These vectors can provide a rich information basis for subsequent analysis (such as combined analysis with student behavior vectors).

[0073] In a possible implementation, the semantic encoding node is a first feature conversion unit, and the first feature conversion unit includes a video feature conversion space.

[0074] The method performs a multi-semantic level video feature domain transformation on the monitoring video picture vector based on multiple semantic coding nodes in the monitoring video coding network that has previously performed parameter learning, and generates a behavior pattern vector of the target educational place in the basic monitoring video picture at multiple semantic levels, including: performing a multi-semantic level video feature domain transformation on the monitoring video picture vector by calling the video feature conversion space in each first feature conversion unit in the multiple semantic coding nodes, and generating a behavior pattern vector of the target educational place in the basic monitoring video picture at multiple semantic levels.

[0075] In this embodiment, when the server processes the school teaching building surveillance video data, the semantic coding nodes involved in the surveillance video coding network are defined as first feature conversion units. Each first feature conversion unit contains a video feature conversion space, which is specially designed for feature conversion of surveillance video picture vectors.

[0076] When the server starts to perform the first semantic level video feature domain transformation on the surveillance video picture vector, the video feature transformation space in the first-layer first feature transformation unit in multiple semantic coding nodes (ie, the first feature transformation units) can be called.

[0077] This video feature conversion space is like a special processing workshop, which receives the previously generated surveillance video picture vector. This vector contains various original features extracted from the surveillance video of the classroom in the teaching building, such as the color distribution of the characters in the picture, the contour edge, the texture of the object and other information.

[0078] In this video feature conversion space, there are a series of predefined algorithms and parameters. First, the color distribution features of the characters can be converted. For example, if the original vector records the RGB value ratio of the color of the character's clothes in the picture, the algorithm in the video feature conversion space may convert it into a color category feature, such as the proportion of cold or warm colors.

[0079] For the contour edge features of the character, the original contour edge pixel information can be converted into more abstract contour shape features through optimized versions of some edge detection algorithms, such as whether it is a straight contour (which may indicate a standing posture) or a curved contour (which may indicate a sitting posture or bending posture).

[0080] The video feature conversion space converts the texture features of objects into more semantic object type features. For example, based on the texture features of tables and chairs, it can accurately determine the type of tables and chairs (single-person table or double-person table, wooden chair or plastic chair, etc.).

[0081] After processing this video feature conversion space, the first feature conversion unit of the first layer generates the behavior pattern vector of the target educational venue in the basic surveillance video screen at the first semantic level. This vector contains the features of people and objects after conversion, such as the general posture type of people in the classroom (based on color and outline), the type of tables and chairs, and other information.

[0082] When performing the video feature domain transformation at the Xth semantic level, the server calls the video feature transformation space in the Xth first feature transformation unit in the multiple semantic encoding nodes. This video feature transformation space receives the behavior pattern vector at the X-1th semantic level. Assume that the behavior pattern vector at the X-1th semantic level already contains information such as the basic posture types of the characters in the classroom and the types of tables and chairs.

[0083] In this video feature conversion space, if X=2, the character posture type in the behavior pattern vector at the X-1th semantic level can be further analyzed. For example, if it is previously determined that a character is sitting, it can be analyzed whether this sitting posture is focused on listening (features such as leaning forward and looking towards the blackboard) or relaxed (features such as leaning back and looking around).

[0084] For the type of tables and chairs, the use of the tables and chairs can be judged in combination with the posture of the characters. If it is a double table, it can be judged whether it is used by one person or two people, and whether the use method is in line with the normal learning scene (such as whether there are reasonable books placed on the table, etc.).

[0085] After processing this video feature conversion space, the Xth first feature conversion unit generates the behavior pattern vector of the target educational venue in the basic surveillance video at the Xth semantic level. This vector contains deeper semantic information, such as the learning status of people in the classroom (judged by sitting posture), the actual use efficiency of tables and chairs, etc.

[0086] Assume that there are three semantic coding nodes (i.e., three semantic levels) in the surveillance video coding network. After completing the transformation of the first semantic level, the server will sequentially call the video feature transformation space in the second and third first feature transformation units for processing.

[0087] In the video feature conversion space of the third semantic level, the information of the first two semantic levels can be combined to analyze the group behavior in the classroom. For example, based on the learning status of the characters and the efficiency of the use of tables and chairs, it can be judged whether the learning atmosphere of the entire classroom is positive (most people sit attentively, the tables and chairs are used reasonably and there is interaction) or negative and dull (many people sit in a relaxed position, the tables and chairs are used chaotically and there is no interaction).

[0088] Finally, by calling the video feature conversion space in each first feature conversion unit to perform multi-semantic level video feature domain transformation, the server obtains vectors that comprehensively describe the behavior patterns of the target educational venues (classrooms) at multiple semantic levels. These vectors can provide rich and hierarchical information for further educational venue management analysis.

[0089] In a possible implementation, step S140 includes:

[0090] According to the behavior coding network that has previously performed parameter learning, the student behavior vectors are respectively transformed into multi-semantic level behavior feature domains.

[0091] The behavior coding network includes a plurality of cascaded second feature conversion units, and a behavior focusing unit is connected between each two adjacent second feature conversion units. The number of the second feature conversion units is the same as the number of the first feature conversion units in the surveillance video coding network.

[0092] In a possible implementation, the behavior encoding network that has previously performed parameter learning performs multi-semantic level behavior feature domain transformation on the student behavior vectors, including:

[0093] Step S141, when performing the first semantic level behavior feature domain transformation on the student behavior vector, utilizing the first-layer second feature conversion unit among the multiple second feature conversion units, performs the first semantic level behavior feature domain transformation on the student behavior vector, and generates the first semantic level representation vector of the student behavior record document after the first semantic level behavior feature domain transformation.

[0094] Step S142, after the student behavior vector is transformed into the Yth semantic level behavior feature domain, the Yth behavior focusing unit in the behavior coding network is used to perform the feature interaction aggregation on the Yth semantic level representation vector after the Yth semantic level behavior feature domain transformation and the behavior pattern vector generated by the Yth first feature conversion unit in the surveillance video coding network to generate the Yth feature interaction aggregation result.

[0095] Step S143, when performing the Zth semantic level behavior feature domain transformation on the student behavior vector, using the Zth second feature conversion unit among the multiple second feature conversion units, performs the Zth semantic level behavior feature domain transformation on the student behavior vector and the Z-1th feature interaction aggregation result generated by the Z-1th behavior focusing unit, and generates the Zth semantic level representation vector of the student behavior record document after the Zth semantic level behavior feature domain transformation.

[0096] Step S144, outputting the representation vector generated after the last semantic level behavior feature domain transformation of the multi-semantic level behavior feature domain transformation as the feature interaction aggregation vector generated after the multi-semantic level behavior feature domain transformation.

[0097] In a possible implementation, the plurality of second feature conversion units include: one or more second feature conversion units for performing information dimension reduction on the student behavior vector, and one or more second feature conversion units for performing information dimension increase on the student behavior vector.

[0098] The number of the second feature conversion units for performing information dimension reduction on the student behavior vector is the same as the number of the second feature conversion units for performing information dimension up-scaling on the student behavior vector. Furthermore, the one or more second feature conversion units for performing information dimension up-scaling on the student behavior vector are connected after the one or more second feature conversion units for performing information dimension reduction on the student behavior vector.

[0099] In this embodiment, when processing student behavior data, it relies on a behavior encoding network that has been previously parameter-learned. This behavior encoding network is specially constructed for performing multi-semantic hierarchical feature conversion on student behavior vectors.

[0100] The behavior coding network includes multiple cascaded second feature conversion units, and a behavior focusing unit is connected between each two adjacent second feature conversion units. The number of these second feature conversion units is the same as the number of first feature conversion units in the surveillance video coding network. This design ensures that when processing student behavior vectors and surveillance video image vectors, the number of semantic levels can correspond, which is convenient for subsequent analysis and interactive operations.

[0101] First, the server inputs the previously generated student behavior vector into the behavior encoding network. This student behavior vector contains information extracted from the student behavior record document, such as the student's attendance, the number of questions answered in class, and whether he violated discipline.

[0102] First, the first-layer second feature conversion unit among the plurality of second feature conversion units is used to perform a first semantic level behavior feature domain transformation on the student behavior vector. The first-layer second feature conversion unit performs a preliminary semantic transformation on each element in the student behavior vector.

[0103] Taking attendance as an example, if the attendance in the original student behavior vector is represented by 0 or 1 (0 for absence and 1 for attendance), the first-layer second feature conversion unit may convert it into a more semantic representation, such as converting attendance into the stability feature of attendance. If the student has continuous full attendance, it may be converted to "high stability attendance"; if he is often absent, it may be converted to "low stability attendance".

[0104] For the number of questions answered in class, this value can be converted into a characteristic of student participation in class. For example, a large number of questions answered may be converted into "high participation", and a small number of questions answered may be converted into "low participation".

[0105] After being processed by the second feature conversion unit of the first layer, the first semantic level representation vector of the student behavior record document after transformation in the first semantic level behavior feature domain is generated. This vector contains the converted student behavior features, such as the stability of attendance, enthusiasm for class participation, and other information.

[0106] When performing the Yth semantic level behavior feature domain transformation, the server will use the Yth second feature transformation unit to transform the student behavior vector. Assume that the Yth second feature transformation unit receives the result of the previous Y-1th semantic level behavior feature domain transformation (here is a vector containing the student behavior features after the previous semantic level transformation).

[0107] If Y=2, this second feature conversion unit may further analyze the student's behavioral characteristics. For example, for the stability of attendance and the enthusiasm of class participation, these two factors can be considered comprehensively to judge the student's learning attitude. If the student's attendance is stable and the participation is high, it may be converted to "active learning attitude"; if the attendance is stable but the participation is low, it may be judged as "passive learning attitude".

[0108] After being processed by the Yth second feature conversion unit, the Yth semantic level representation vector after the Yth semantic level behavior feature domain transformation is generated. This vector contains deeper semantic information, such as the student's learning attitude.

[0109] Then, the Yth behavior focusing unit in the behavior encoding network is used to perform feature interactive aggregation on the Yth semantic level representation vector after the Yth semantic level behavior feature domain transformation and the behavior pattern vector generated by the Yth first feature conversion unit in the surveillance video encoding network.

[0110] For example, the student learning attitude feature in the Yth semantic level representation vector interacts with the learning atmosphere feature in the classroom behavior pattern vector generated by the corresponding semantic level (Yth) in the surveillance video encoding network. If the classroom learning atmosphere is positive and the student learning attitude is positive, the behavior focus unit may strengthen and aggregate these two features, indicating a positive match between the student and the classroom environment.

[0111] On the contrary, if the classroom learning atmosphere is negative and boring, and the students' learning attitude is also negative, the behavior focus unit will identify this negative match and mark it in the aggregation process. Through this feature interaction aggregation, the Yth feature interaction aggregation result is generated, which contains the interaction information between student behavior and classroom environment at the corresponding semantic level.

[0112] When performing the behavior feature domain transformation of the last semantic level (assuming the Zth semantic level) on the student behavior vector, the server uses the Zth second feature transformation unit among the multiple second feature transformation units. This second feature transformation unit performs the behavior feature domain transformation of the Zth semantic level on the student behavior vector and the Z-1th feature interaction aggregation result generated by the Z-1th behavior focusing unit.

[0113] For example, the Zth second feature conversion unit may comprehensively consider all previous semantic level information and the interaction results with the classroom environment to evaluate the overall learning effect of the student. If the previous interaction results show that the student always maintains a positive attitude in a positive classroom environment, and the previous learning attitude, participation enthusiasm, etc. are good, then the overall learning effect of the student may be evaluated as "excellent".

[0114] After this transformation, the Zth semantic level representation vector of the student behavior record document is generated after the transformation of the Zth semantic level behavior feature domain.

[0115] The representation vector generated by the last semantic level behavior feature domain transformation of the multi-semantic level behavior feature domain transformation is output as the feature interaction aggregation vector generated after the multi-semantic level behavior feature domain transformation. This vector contains comprehensive information about the student behavior after multi-semantic level transformation and interaction aggregation with the classroom environment, providing an important data basis for subsequent operations such as summary generation.

[0116] In a possible implementation, before performing the first semantic level behavior feature domain transformation on the student behavior vector, the method further includes:

[0117] Step A110: Generate a random representation vector of the behavior coding network based on a preset random vector generation mechanism.

[0118] Step A110, loading the random representation vector and the student behavior vector as input vectors into the behavior coding network.

[0119] Step S141 includes: using the first-layer second feature conversion unit to perform a first semantic-level behavior feature domain transformation on the student behavior vector and the random representation vector to generate the first semantic-level representation vector.

[0120] In this embodiment, the server is configured with a preset random vector generation mechanism, which generates random vectors based on specific algorithms and rules. For example, the random vector generation mechanism may be based on a pseudo-random number generation algorithm, which generates a series of seemingly random numbers based on an initial seed value (the seed value may be the system time or other fixed values).

[0121] This random vector generation mechanism is designed to generate a random representation vector for the behavior coding network. Assume that the dimension of this random representation vector is pre-set according to the structure of the behavior coding network, such as a 10-dimensional vector.

[0122] The server starts to generate random representation vectors according to this mechanism. For each dimension, the random vector generation mechanism generates a random number in a predefined range of values. For example, if the range is [-1,1], then 0.3 may be generated in the first dimension, -0.5 may be generated in the second dimension, and so on, until a complete 10-dimensional random representation vector is generated.

[0123] Among them, the server has obtained the student behavior vector, which contains the quantitative representation of various behavioral information of students in the educational venue, such as the attendance mentioned above, the number of times the class questions were answered, etc. At the same time, a random representation vector is also generated.

[0124] These two vectors will be loaded into the behavior coding network as input vectors. During the loading process, the server will format the two vectors according to the input interface specification of the behavior coding network. For example, it ensures that the dimension order and data type of the vectors meet the requirements of the network.

[0125] The behavior encoding network is designed to process this composite input vector, and its first-layer second feature conversion unit will perform the first semantic level behavior feature domain transformation based on these two vectors.

[0126] After receiving the student behavior vector and the random representation vector, the first-layer second feature conversion unit starts to perform the first semantic level behavior feature domain transformation.

[0127] For the attendance in the student behavior vector (assuming that 1 represents full attendance), the first-layer second feature conversion unit will convert it by combining the value of a dimension in the random representation vector (assuming that the first dimension value of the random representation vector is 0.3). It may use a predefined function, such as adding the attendance value to the value of the first dimension of the random representation vector and then normalizing it to obtain a new value. This value may represent a semantic feature of attendance affected by random factors, such as "highly stable attendance under the influence of random factors."

[0128] For the number of times a student answers questions in class in the student behavior vector (assuming it is 5 times), combined with the other dimension of the random representation vector (assuming the second dimension value is -0.5), the first-level second feature conversion unit may use a different calculation method, such as multiplying the number of times a student answers questions by the value of the second dimension of the random representation vector and then taking the absolute value, to obtain a new semantic feature representing class participation, such as "the degree of active class participation under random adjustment."

[0129] For other elements in the student behavior vector, such as whether or not the student violates discipline, they will also be combined with the corresponding dimensions of the random representation vector for calculation and conversion in a similar manner.

[0130] After such processing, the first-level second feature conversion unit generates the first semantic level representation vector of the student behavior record document after transformation in the first semantic level behavior feature domain. This vector contains the semantic features of the student behavior after interactive transformation with the random representation vector. It is different from the result obtained based solely on the student behavior vector conversion. It introduces random factors, increases the diversity and uncertainty of the features, and provides a richer starting point for subsequent deeper semantic level analysis.

[0131] In a possible implementation, the scale of the Yth semantic level representation vector is the same as the scale of the behavior pattern vector generated by the Yth first feature conversion unit.

[0132] Step S142 includes:

[0133] Step S1421, using the Yth behavior focusing unit in the behavior encoding network, merges the Yth semantic level representation vector and the behavior pattern vector generated by the Yth first feature conversion unit in the vector component dimension to generate a merged vector.

[0134] Step S1422, performing an upper limit compression operation and a mean compression operation on the merged vector respectively to generate an upper limit compression vector and a mean compression vector, and outputting the upper limit compression vector and the mean compression vector as a compression vector.

[0135] Step S1423: performing a mapping operation on the compressed vector to generate an association array between the Yth semantic level representation vector and the behavior pattern vector generated by the Yth first feature conversion unit.

[0136] Step S1424, performing feature interaction aggregation on the merged vector according to the association array to generate the Yth feature interaction aggregation result.

[0137] In this embodiment, when performing this stage of operation, the Yth semantic level representation vector and the behavior pattern vector generated by the Yth first feature conversion unit have been obtained. The two vectors have the same scale, for example, they are both 20-dimensional vectors. Each dimension represents specific semantic information, which is the result obtained after the previous semantic level conversion.

[0138] The Yth semantic level representation vector may contain the student's behavioral characteristics at the Yth semantic level, such as the semantic feature representation of the student's concentration in learning, the enthusiasm for interacting with classmates, etc. The behavior pattern vector generated by the Yth first feature conversion unit may describe the state of the target educational place (such as a classroom) at the Yth semantic level, such as the activity level and order in the classroom.

[0139] When operating with the Yth behavior focusing unit in the behavior encoding network, a merge operation is first performed on the vector component dimension. The server merges the Yth semantic level representation vector and the behavior pattern vector generated by the Yth first feature conversion unit in a one-to-one correspondence according to the dimension.

[0140] For example, the first dimension of the Yth semantic level representation vector (assuming it represents the student's concentration value) and the first dimension of the behavior pattern vector generated by the Yth first feature conversion unit (assuming it represents the classroom's quietness value) are merged into the first dimension of the new vector. Each dimension is merged in turn, and a 40-dimensional merged vector is finally generated. This merged vector contains comprehensive information about student behavior characteristics and classroom behavior pattern characteristics.

[0141] The server performs an upper limit compression operation on the merged vector. Assume that the value of each dimension in the merged vector has a certain range, such as [-10, 10]. The upper limit compression operation checks the value of each dimension. If the value of a dimension exceeds the set upper limit (such as 8), it is set to the upper limit value 8; if the value of a dimension is lower than the lower limit (such as -8), it is set to the lower limit value -8.

[0142] For example, the original value of a dimension of the merged vector is 12. After the upper limit compression operation, the value of this dimension becomes 8. In this way, the upper limit compression operation is performed on the entire 40-dimensional merged vector to generate an upper limit compression vector. This upper limit compression vector avoids the value of some dimensions being too large or too small, thereby preventing abnormal values ​​from appearing in subsequent calculations and having a large impact on the results.

[0143] Next, the mean compression operation is performed. The server calculates the average value of all dimensions of the merged vector. Assuming that the sum of all dimension values ​​of the merged vector is 200, the mean is 200 / 40=5.

[0144] Then, for each dimension in the merged vector, the mean compression operation compares the value of the dimension with the mean. If the difference between the value of a dimension and the mean exceeds a certain threshold (such as 3), the value of the dimension will be adjusted to move closer to the mean. For example, if the value of a dimension is 10, the mean is 5, and the difference is 5, which exceeds the threshold of 3, then the value of this dimension may be adjusted to 7 (the specific adjustment method can be based on a predefined algorithm, such as moving closer to the mean in a certain proportion).

[0145] After the mean compression operation, a mean compression vector is generated. Finally, the upper limit compression vector and the mean compression vector are output as a compression vector. This compression vector retains the information of the merged vector to a certain extent, while optimizing it and reducing data fluctuations and anomalies.

[0146] The server performs a mapping operation on the compressed vector. This mapping operation is based on a pre-set mapping function. Assume that this mapping function is a nonlinear function, which maps the value of each dimension of the compressed vector to a new space.

[0147] For example, for the value of the first dimension of the compressed vector (assuming it is 5), it may be mapped to a new value (such as 3.5) through the mapping function. This new value has a different semantic meaning in the new space, which may indicate the degree of association between the Yth semantic level representation vector and the behavior pattern vector generated by the Yth first feature conversion unit in this dimension.

[0148] By performing such a mapping operation on each dimension of the compressed vector, an association array between the Yth semantic level representation vector and the behavior pattern vector generated by the Yth first feature conversion unit is generated. This association array is a two-dimensional structure, whose rows and columns correspond to the dimensions of the two vectors, and each element represents the degree of association between the two vectors in the corresponding dimension.

[0149] The merged vector is interactively aggregated based on this association array. The server performs a weighted sum operation on each dimension in the merged vector based on the association degree value in the association array.

[0150] For example, if the first dimension of the Yth semantic level representation vector in the association array and the first dimension of the behavior pattern vector generated by the Yth first feature conversion unit have a high degree of correlation (such as a correlation value of 0.8), then when performing feature interaction aggregation on the first dimension of the merged vector, this dimension will be given a larger weight.

[0151] Assuming that the value of the first dimension of the merged vector is 10, when performing weighted summation, it may be calculated according to the association value of 0.8 to obtain a new value (such as 8). Each dimension of the merged vector is subjected to feature interaction aggregation operation according to the association array in this way, and finally the Yth feature interaction aggregation result is generated. This result combines the student behavior characteristics and classroom behavior pattern characteristics, and reasonably integrates them according to the degree of association between them, providing more valuable data for subsequent analysis and processing.

[0152] In a possible implementation, step S120 includes:

[0153] Step S121, calling a behavior vector generator to generate a behavior vector for the student behavior record document, thereby generating the student behavior vector.

[0154] Step S122, calling the monitoring video vector generator to generate a monitoring video vector for the basic monitoring video picture, thereby generating the monitoring video picture vector.

[0155] In this embodiment, the student behavior record document received by the server is a detailed record of student behavior in a target educational location (such as a classroom in a teaching building) in a school. This document has a specific structure, for example, it may be a table-like document, in which each row represents a student and each column represents a behavior attribute.

[0156] Take a class's student behavior record document as an example. The column titles may include "student number", "name", "attendance", "number of questions answered in class", "whether to violate discipline", "participation in group activities", etc. For each student, there are corresponding records under these columns. For example, student A's attendance is perfect, he answered questions in class 3 times, did not violate discipline, and actively participated in group activities.

[0157] The server calls the behavior vector generator to process the student behavior record document. The behavior vector generator first reads the structure and content of the document.

[0158] For the attribute "attendance", the behavior vector generator will convert it into a specific numerical representation. Full attendance may be converted to 1, and absence to 0. For "Number of questions answered in class", the original number is used directly. "Whether to violate discipline" can be converted to -1 (violation), 0 (no violation), 1 (excellent performance). "Participation in group activities" can be converted to a value between 0 and 1 according to the degree of participation, for example, very active is 1, and never participate is 0.

[0159] Then, the behavior vector generator will combine these converted values ​​into a vector in a certain order. Assuming that in the order of "attendance-number of questions answered in class-whether to violate discipline-participation in group activities", part of student A's behavior vector may be [1,3,0,0.8]. For the student behavior record document of the entire class, the behavior vector generator will perform such a transformation for each student and combine the vectors of all students (possibly by splicing or other means), and finally generate a student behavior vector that can represent the entire student behavior record document. This vector contains quantitative information on all students on various behavioral attributes.

[0160] The basic surveillance video images obtained by the server are videos taken by surveillance cameras in the target educational venue (classroom). This video contains various scenes in the classroom, such as teachers teaching in front of the blackboard, students listening in their seats, interactions between students, and activities during breaks.

[0161] A video is composed of a series of continuous frames, each of which is an image that contains rich information, such as the position, posture, and expression of a person, the shape, color, and position of an object, etc.

[0162] The server calls the surveillance video vector generator to process the basic surveillance video image. The surveillance video vector generator first analyzes the video frame by frame.

[0163] For each frame, it extracts multiple features. For example, in terms of the position of the character, it calculates the coordinate position of the character in the picture, and may divide the picture into a coordinate plane, and the coordinates of the center position of the character (x, y) will be recorded. For the posture of the character, some posture estimation algorithms are used to determine whether the character is standing, sitting, bending over, etc., and convert it into a numerical value or vector representation. For expressions, expression recognition algorithms can be used to determine whether it is happy, serious, or other expressions, and also converted into specific representations.

[0164] In terms of objects, for objects such as tables and chairs, the surveillance video vector generator will identify their shape, color and position. The shape can be extracted through the edge detection algorithm to extract the contour features, the color can be counted through the color histogram to calculate the main color components, and the position is also expressed in the form of coordinates.

[0165] Then, the surveillance video vector generator will combine all the features extracted from a frame into a vector according to certain rules. Assuming that a frame has 50 feature values ​​after feature extraction, a 50-dimensional vector is generated to represent the frame.

[0166] For the entire video, the surveillance video vector generator may not process every frame, but at a certain sampling frequency, for example, one frame is taken every 10 frames for processing. Then the vectors corresponding to these sampled frames are combined or averaged (depending on the specific algorithm) to finally generate a surveillance video screen vector that can represent the entire basic surveillance video screen. This vector contains many feature information about people and objects in the video screen.

[0167] In a possible implementation, the method is implemented based on a summary generation network model that has previously been subjected to parameter learning. The summary generation network model includes a first target network and a second target network. The first target network includes at least a behavior vector generator and a behavior encoding network, and the second target network includes a surveillance video vector generator and a surveillance video encoding network.

[0168] The training step of the summary generation network model includes:

[0169] Step B110, loading the template surveillance video picture in the training data into the second target network, using the surveillance video vector generator in the second target network to extract the surveillance video picture vector of the template surveillance video picture, and generating the template surveillance video picture vector.

[0170] Step B120, using the surveillance video encoding network in the second target network to perform multi-semantic level video feature domain transformation on the template surveillance video picture vector, to generate a template behavior pattern vector of the template target education place in the template surveillance video picture at multiple semantic levels.

[0171] Step B130, loading the template student behavior record document in the training data into the first target network, using the behavior vector generator in the first target network to extract text vectors from the template student behavior record document to generate a template student behavior vector.

[0172] Step B140, using the behavior encoding network in the first target network to perform multi-semantic level behavior feature domain transformation on the template student behavior vector, to generate a template feature interaction aggregation vector.

[0173] Step B150, calculating the training cost parameters of the summary generation network model according to the template behavior pattern vector and the template feature interaction aggregation vector, and generating the training cost parameters of the summary generation network model.

[0174] Step B160, optimizing the parameters of the summary generation network model according to the training cost parameters to generate an optimized summary generation network model.

[0175] In this embodiment, the summary generation network model includes a first target network and a second target network. The first target network at least has a behavior vector generator and a behavior encoding network, which is mainly used to process data related to student behavior; the second target network includes a monitoring video vector generator and a monitoring video encoding network, which is mainly responsible for processing data related to monitoring videos.

[0176] First, the server obtains template surveillance video images from the training data. These template surveillance video images are representative surveillance videos collected from the target educational places (such as classrooms) of the school.

[0177] The template surveillance video image is loaded into the second target network. The surveillance video vector generator in the second target network starts to process it. The surveillance video vector generator processes the video image in a similar way to the previous processing of the basic surveillance video image.

[0178] For example, for each frame of the template surveillance video, it extracts the position, posture, expression of the person, and the shape, color, position and other features of the object (such as tables and chairs). These features are combined into vectors according to specific rules to represent each frame. Assume that after processing, a frame of the picture has obtained a vector composed of 60 feature values. Then the entire video is processed according to a certain sampling strategy (such as taking one frame every 5 frames), and finally these frame vectors are combined or averaged to generate the template surveillance video picture vector.

[0179] Next, the surveillance video coding network in the second target network performs a multi-semantic level video feature domain transformation on the template surveillance video picture vector. The semantic coding node (first feature conversion unit) in this surveillance video coding network works as described above.

[0180] At the first semantic level, the first-layer semantic coding node converts the template surveillance video screen vector. For example, it converts the basic posture features of the characters in the screen (such as standing, sitting, etc.) into more abstract behavioral state features, such as whether they are in a static state or a simple moving state. After being processed by the first-layer semantic coding node, the template behavior pattern vector of the template target education venue in the template surveillance video screen at the first semantic level is obtained.

[0181] Assuming there are three semantic levels, at the second semantic level, the second semantic encoding node will further transform the template behavior pattern vector of the first semantic level. It may analyze the purpose of the character's action, such as whether the character walks to the podium (perhaps to ask questions or hand in homework) or walks to other students (perhaps to discuss issues). After being processed by the second semantic encoding node, the template behavior pattern vector at the second semantic level is obtained.

[0182] At the third semantic level, the third semantic encoding node will integrate the information of the first two levels and analyze more complex semantic information such as the interaction between characters, thereby generating a template behavior pattern vector at the third semantic level.

[0183] Then, the server takes out the template student behavior record documents from the training data. These documents are similar to the structure of the student behavior record documents mentioned above and contain various student behavior information, such as attendance, class performance, etc.

[0184] The template student behavior record document is loaded into the first target network, and the behavior vector generator in the first target network starts working. It converts the behavior information in the document into a vector representation. For example, information such as attendance (full attendance is 1, absence is 0), the number of questions answered in class (directly take the number), whether the discipline is violated (-1 means violation, 0 means no violation, 1 means excellent performance), etc. are combined into vectors in a specific order to generate a template student behavior vector.

[0185] Then, the behavior encoding network in the first target network performs multi-semantic level behavior feature domain transformation on the template student behavior vector. The second feature conversion unit and the behavior focusing unit in this behavior encoding network work together in the same way as before.

[0186] At the first semantic level, the first-level second feature conversion unit converts the basic behavior information in the template student behavior vector into a more semantic representation. For example, the attendance and the number of questions answered in class are combined to convert them into the student's learning participation stability feature. The first semantic level representation vector of the template student behavior record document after the first semantic level behavior feature domain transformation is generated.

[0187] Assuming there are three semantic levels, at the second semantic level, the second second feature conversion unit will conduct further in-depth analysis. For example, the learning participation stability feature of the first semantic level and other information in the document (such as whether to participate in group activities) are combined to convert it into the student's learning enthusiasm feature. After that, the second behavior focusing unit will perform feature interaction aggregation on the second semantic level representation vector and the template behavior pattern vector of the second semantic level (from the surveillance video encoding network) to generate the second feature interaction aggregation result.

[0188] At the third semantic level, the third second feature conversion unit will integrate the previous information and convert it into the overall learning effect features of the students. Finally, the template feature interaction aggregation vector is obtained.

[0189] The server calculates the training cost parameters of the summary generation network model based on the template behavior pattern vector and the template feature interaction aggregation vector.

[0190] First, the first training cost parameter is calculated based on the template feature interaction aggregation vector and the template surveillance video screen. For example, the template feature interaction aggregation vector is summarized. This summary generation process converts the information in the vector into a comprehensive description of the student's behavior and the state of the educational venue. Assume that the generated summary description is an overall evaluation of the student's learning status in the classroom. Then the mean square error value between this summary description and the template surveillance video screen is calculated. This mean square error value is the first training cost parameter.

[0191] Next, the second training cost parameter is calculated based on the template behavior pattern vector and the template feature interaction aggregation vector. The template behavior pattern vector is summarized to obtain a summary description of the classroom behavior pattern. The template feature interaction aggregation vector is also summarized to obtain the previously mentioned overall evaluation of the student's learning status in the classroom. Then, regional attention features are extracted for these two summary descriptions. For example, for the summary description of the classroom behavior pattern, attention features about different areas in the classroom (such as the podium area, student seat area, etc.) are extracted; for the overall evaluation of the student's learning status, attention features related to the classroom area are extracted. The cosine distance between the two regional attention features is calculated, and this cosine distance is the second training cost parameter.

[0192] Finally, the sum of the first training cost parameter and the second training cost parameter is used as the training cost parameter of the summary generation network model.

[0193] The summary generation network model is optimized based on the calculated training cost parameter. The server uses an optimization algorithm (such as a gradient descent algorithm) to adjust various parameters in the summary generation network model based on the value of the training cost parameter.

[0194] For example, if the training cost parameter is large, it means that the prediction result of the model deviates greatly from the actual situation. The server will adjust the parameters in the first target network and the second target network in the direction of gradient descent, such as the behavior vector generator, the behavior encoding network, the monitoring video vector generator, and the weights in the monitoring video encoding network. After multiple iterations of adjustment, until the training cost parameter reaches a smaller value or meets the stop condition, the optimized summary generation network model is generated. This optimized model can more accurately process the monitoring video screen and student behavior record documents, and generate more reasonable summary screen data for educational venue management data.

[0195] In a possible implementation, step B150 includes:

[0196] Step B151, calculating a first training cost parameter of the summary generation network model according to the template feature interaction aggregation vector and the template monitoring video screen, and generating a first training cost parameter.

[0197] Step B152: Calculate a second training cost parameter for the summary generation network model based on the template behavior pattern vector and the template feature interaction aggregation vector to generate a second training cost parameter.

[0198] Step B153: output the sum of the first training cost parameter and the second training cost parameter as the training cost parameter of the summary generation network model.

[0199] Step B151 includes:

[0200] Step B1511, generating a summary of the template feature interaction aggregation vector, generating first template summary screen data that simultaneously represents the education venue vector of the template target education venue and the student behavior description vector of the template student behavior record document.

[0201] Step B1512: outputting the mean square error value between the first template summary screen data and the template monitoring video screen as the first training cost parameter.

[0202] In this embodiment, the server obtains a template feature interaction aggregation vector, which is a result of performing multi-semantic level behavior feature domain transformation on the template student behavior vector and performing feature interaction aggregation with the template behavior pattern vector.

[0203] This template feature interaction aggregation vector is summarized. For example, this vector contains quantitative and semantic information about the various behaviors of students in the classroom (the template target educational venue), as well as the interaction information between these behaviors and the classroom environment (the relevant information is obtained from the template behavior pattern vector).

[0204] During the summary generation process, the server integrates the information in the vector according to predefined rules. If one dimension in the vector represents the student's concentration and the other dimension represents the classroom's activity, then the information of these two dimensions may be combined during summary generation to generate a description of the student's concentration under the current classroom activity. Through similar processing of the information of each dimension in the vector, the first template summary screen data is generated, which simultaneously represents the education venue vector of the template target education venue and the student behavior description vector of the template student behavior record document. This data may be a structured data block that contains a comprehensive description of the classroom environment and student behavior, such as the order of the classroom (such as quiet, orderly, etc.) and the learning status of students in this environment (such as concentration, active participation, etc.).

[0205] The server already has the first template summary screen data, and also has the template monitoring video screen. This template monitoring video screen contains rich visual information such as the actual scene in the classroom, such as the actual movements of the characters, the actual layout of the objects, etc.

[0206] In order to calculate the first training cost parameter, the server calculates the mean square error value between the first template summary screen data and the template monitoring video screen. First, the server needs to compare the description in the first template summary screen data with the actual situation in the template monitoring video screen.

[0207] For example, the first template summary screen data describes that the classroom is quiet and orderly, and the students are focused. The server will analyze the corresponding scenes in the template monitoring video screen, such as judging whether the classroom is really quiet and orderly by observing the movement amplitude of the characters in the screen, sound detection (if there is relevant data), etc., and judging whether the students are focused by observing the expressions and sight directions of the students.

[0208] Then, according to the mean square error calculation method, for each comparison dimension (such as classroom order, student concentration, etc.), the square error between the predicted value (the description in the first template summary screen data) and the actual value (the value obtained from the analysis of the template monitoring video screen) is calculated, and then these square errors are summed and averaged. This average value is the mean square error value between the first template summary screen data and the template monitoring video screen, and the server outputs this mean square error value as the first training cost parameter. This parameter reflects the degree of difference between the summary generation network model and the actual monitoring video screen when generating summary screen data based on the template feature interaction aggregation vector.

[0209] Next, the server obtains the template behavior pattern vector, which includes the behavior pattern information of the template target education place (classroom) in the template monitoring video screen at multiple semantic levels.

[0210] Generate a summary of this template behavior pattern vector. For example, if the template behavior pattern vector describes the flow of people in the classroom (such as frequent movement, seldom movement, etc.) and the interaction of people (such as group interaction, individual independent activities, etc.) at a certain semantic level, when generating the summary, the server will integrate this information into a description of the overall activity pattern of the classroom and generate the second template summary screen data. This data mainly focuses on the behavior pattern of the classroom environment itself, such as whether the classroom is an active environment full of interaction or a relatively quiet and independent environment.

[0211] As in the previous calculation of the first training cost parameter, the server again generates a summary of the template feature interaction aggregation vector to obtain the first template summary screen data that simultaneously represents the education venue vector of the template target education venue and the student behavior description vector of the template student behavior record document. This data includes a comprehensive description of the interaction between student behavior and the classroom environment, such as the active participation of students in an active classroom environment.

[0212] The server extracts regional attention features for the second template summary screen data and the first template summary screen data. For the second template summary screen data, the server may focus on the behavioral pattern features of different areas in the classroom, such as the podium area, the classroom corner, and the middle seat area. For example, whether there are frequent activities of teachers in the podium area, whether there are students alone in the corner of the classroom, etc. For the first template summary screen data, the server will extract student behavior features related to these areas, such as whether students near the podium area are more focused, and whether students in the corner of the classroom are less engaged.

[0213] Then, the cosine distance between the two regional attention features is calculated. The cosine distance can measure the degree of difference between the two feature vectors in direction. If the directions of the two feature vectors are closer, the cosine distance is smaller, which means that the template behavior pattern vector and the template feature interaction aggregation vector are more matched in terms of regional attention features; conversely, if the cosine distance is larger, it means that the difference between the two is greater. The server outputs this cosine distance as the second training cost parameter. This parameter reflects the degree of difference between the template behavior pattern vector and the template feature interaction aggregation vector in terms of regional attention features.

[0214] The server adds the first training cost parameter to the second training cost parameter, and the result is the training cost parameter of the summary generation network model. This training cost parameter comprehensively considers the difference between the prediction and actual surveillance video images of the summary generation network model (the first training cost parameter) and the difference between the template behavior pattern vector and the template feature interaction aggregation vector in terms of regional attention features (the second training cost parameter). It can be used to evaluate the performance of the summary generation network model and optimize the model based on this parameter.

[0215] In a possible implementation, step B152 includes:

[0216] Step B1521, performing summary generation on the template behavior pattern vector to generate second template summary screen data.

[0217] Step B1522, generating a summary of the template feature interaction aggregation vector, generating first template summary screen data that simultaneously represents the education venue vector of the template target education venue and the student behavior description vector of the template student behavior record document.

[0218] Step B1523, performing regional attention feature extraction on the second template summary screen data and the first template summary screen data respectively to generate a first regional attention feature and a second regional attention feature.

[0219] Step B1524, output the cosine distance between the first area attention feature and the second area attention feature as the second training cost parameter.

[0220] In this embodiment, the server obtains the template behavior pattern vector, which contains the behavior pattern information of the template target education place (such as classroom) in the template monitoring video screen at multiple semantic levels. For example, at a lower semantic level, the vector may contain the basic action status of people in the classroom (standing, sitting, walking, etc.) and the layout status of objects (tables and chairs, etc.); at a higher semantic level, it may contain the interaction mode between people (group discussion, individual learning, etc.) and the overall activity atmosphere (active, quiet, etc.).

[0221] The server generates a summary of the template behavior pattern vector according to predefined rules. For the basic action status of the personnel, if most people are sitting and relatively still, the server may describe it in the summary as "most people in the classroom are in a static sitting state." For the layout of tables and chairs, if they are neatly arranged, it will be described as "the tables and chairs are neatly arranged." For the interaction mode of personnel, if group discussion is the main mode, it will be recorded as "the interaction mode in the classroom is mainly group discussion." For the overall activity atmosphere, if the atmosphere is active, it will be summarized as "the overall activity atmosphere in the classroom is active."

[0222] By integrating the information of each level and each aspect in the vector, the second template summary screen data is generated. This data is a general description of the behavior mode of the template target education venue, focusing on describing the status and activity mode of the venue itself.

[0223] The template feature interaction aggregation vector is obtained by transforming the multi-semantic hierarchical behavior feature domain of the template student behavior vector and performing feature interaction aggregation with the template behavior pattern vector. This vector contains comprehensive information about the interaction between student behavior and the classroom environment. For example, some dimensions in the vector may represent the learning status of students in a specific classroom atmosphere (such as concentration, participation, etc.), as well as information such as the matching degree between student behavior and classroom layout and personnel interaction mode.

[0224] The server generates a summary of the template feature interaction aggregation vector according to a specific algorithm. If the vector indicates that students are more focused in an active atmosphere, the summary may be described as "students show higher concentration in an active classroom atmosphere." If students' participation matches the classroom's group discussion mode, it will be recorded as "students actively participate in activities that match the classroom's group discussion mode."

[0225] By integrating these complex information in the vector, the first template summary screen data is generated, which simultaneously represents the education venue vector of the template target education venue and the student behavior description vector of the template student behavior record document. This data mainly emphasizes the student's behavior performance in a specific education venue environment and its relationship with the venue environment.

[0226] The server extracts regional attention features from the second template summary screen data. Assume that the classroom is divided into a podium area, a student seat area (which can be further subdivided into front row, middle row, back row, etc.), a classroom corner area, etc.

[0227] For the podium area, the server extracts the behavior pattern features related to the podium from the second template summary screen data. For example, if the second template summary screen data mentions that the teacher's activities on the podium are frequent, then the feature of the podium area may be "podium area-teachers' activities are frequent".

[0228] For the student seating area, if it is mentioned that most students are having group discussions in their seats, then the feature of "front row seating area - actively participating in group discussions" may be extracted for the front row seating area, and the feature of "back row seating area - partially participating in group discussions" may be extracted for the back row seating area. For the classroom corner area, if it is mentioned that the corner is relatively quiet, the feature of "classroom corner area - quiet" will be extracted.

[0229] By extracting relevant features of each area of ​​the classroom, the first area attention feature is generated. This feature is a set of information related to behavioral patterns in different areas of the classroom.

[0230] Similarly, the server extracts regional attention features from the first template summary screen data. For the podium area, if the first template summary screen data mentions that students pay more attention to the teacher in the podium area, then the feature of "podium area - high student attention" is extracted.

[0231] For the student seating area, if it is mentioned that the front row students are more focused in an active atmosphere, the feature of "front row seating area - high concentration in an active atmosphere" will be extracted. For the classroom corner area, if it is mentioned that the students in the corner are less engaged, the feature of "classroom corner area - low engagement" will be extracted.

[0232] This generates the second regional attention feature, which is also a collection of information related to student behavior in different areas of the classroom.

[0233] The server calculates the cosine distance between the first area attention feature and the second area attention feature. The calculation of the cosine distance is based on the cosine value of the angle between two vectors in the vector space. Here, the first area attention feature and the second area attention feature can be regarded as two vectors, and the elements in each vector correspond to the feature descriptions of different areas of the classroom.

[0234] For example, the attention feature vector of the first region is [podium area - frequent teacher activities, front row seat area - active participation in group discussions, back row seat area - partial participation in group discussions, classroom corner area - quiet], and the attention feature vector of the second region is [podium area - high student attention, front row seat area - high concentration in an active atmosphere, back row seat area - low participation, classroom corner area - low participation].

[0235] The server calculates the inner product of the two vectors according to the cosine distance calculation formula, and then divides it by the product of the modulus lengths of the two vectors to obtain the value of the cosine distance. This value is between 0 and 1, where 0 means that the two vectors are exactly the same (that is, the attention features of the two regions match completely), and 1 means that the two vectors are completely opposite (that is, the attention features of the two regions differ the most).

[0236] The server outputs this cosine distance value as the second training cost parameter. This second training cost parameter reflects the degree of difference between the template behavior pattern vector and the template feature interaction aggregation vector in terms of regional attention features, and can be used to evaluate the performance of the summary generation network model in this regard.

[0237] Figure 2The education venue management system 100 based on monitoring big data shown includes: a processor 1001 and a memory 1003. The processor 1001 and the memory 1003 are connected, such as through a bus 1002. Optionally, the education venue management system 100 based on monitoring big data may also include a transceiver 1004, which may be used for data interaction between the server and other servers, such as data transmission and / or data reception. It should be noted that the transceiver 1004 is not limited to one in actual scheduling, and the structure of the education venue management system 100 based on monitoring big data does not constitute a limitation on the embodiments of the present application.

[0238] Processor 1001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 1001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0239] The bus 1002 may include a path to transmit information between the above components. The bus 1002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 1002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0240] The memory 1003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage medium, other magnetic storage devices, or any other medium that can be used to carry or store program code and can be read by a computer, without limitation herein.

[0241] The memory 1003 is used to store program codes for executing the embodiments of the present application, and the execution is controlled by the processor 1001. The processor 1001 is used to execute the program codes stored in the memory 1003 to implement the steps shown in the above method embodiments.

[0242] In addition, an embodiment of the present application also provides a readable storage medium, in which computer executable instructions are pre-installed. When the processor executes the computer executable instructions, the above-mentioned educational venue management method based on monitoring big data is implemented.

[0243] Similarly, it should be noted that in order to simplify the description disclosed in this application, thereby helping to understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present application, multiple features are sometimes combined into one embodiment, drawings, or descriptions thereof. Similarly, it should be noted that in order to simplify the description disclosed in this application, thereby helping to understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present application, multiple features are sometimes combined into one embodiment, drawings, or descriptions thereof.

Claims

1. A method for managing educational venues based on monitoring big data, characterized in that: The method comprises: Obtain basic surveillance video footage and student behavior record documents of target educational venues from surveillance big data; Extracting vectors from the basic surveillance video screen and the student behavior record document respectively to generate a surveillance video screen vector and a student behavior vector; Performing a multi-semantic level video feature domain transformation on the surveillance video screen vector to generate a behavior pattern vector of the target educational venue in the basic surveillance video screen at multiple semantic levels; The student behavior vectors are respectively subjected to multi-semantic level behavior feature domain transformation; wherein the semantic level of the multi-semantic level behavior feature domain transformation is the same as the semantic level of the multi-semantic level video feature domain transformation; between every two behavior feature domain transformations of the semantic levels associated with each other, the behavior feature domain transformation result generated by the preceding behavior feature domain transformation of the two semantic levels associated with each other and the behavior pattern vector of the corresponding semantic level are subjected to feature interaction aggregation; when the subsequent behavior feature domain transformation of the two semantic levels associated with each other is subjected to behavior feature domain transformation, the student behavior vector and the feature interaction aggregation result after the feature interaction aggregation are subjected to behavior feature domain transformation; Summarizing the feature interaction aggregation vector generated after the multi-semantic hierarchical behavior feature domain transformation, generating summary screen data that simultaneously represents the education venue vector of the target education venue and the student behavior description vector of the student behavior record document, and outputting the summary screen data as education venue management basis data; The performing multi-semantic level video feature domain transformation on the surveillance video screen vector to generate a behavior pattern vector of the target education place in the basic surveillance video screen at multiple semantic levels includes: According to a plurality of semantic coding nodes in a surveillance video coding network that has previously undergone parameter learning, a multi-semantic level video feature domain transformation is performed on the surveillance video screen vector to generate a behavior pattern vector of the target educational venue in the basic surveillance video screen at multiple semantic levels; Wherein, when the first semantic level video feature domain transformation is performed on the monitoring video screen vector, the first semantic level video feature domain transformation is performed on the monitoring video screen vector using the first level semantic coding node among the multiple semantic coding nodes to generate a behavior pattern vector of the target education place in the basic monitoring video screen at the first semantic level; When performing the video feature domain transformation of the surveillance video screen vector at the Xth semantic level, the video feature domain transformation of the behavior pattern vector at the X-1th semantic level is performed using the Xth semantic coding node among the multiple semantic coding nodes to generate the behavior pattern vector of the target education venue at the Xth semantic level in the basic surveillance video screen; The semantic coding node is a first feature conversion unit, and the first feature conversion unit includes a video feature conversion space; The method of performing multi-semantic level video feature domain transformation on the surveillance video screen vector based on multiple semantic coding nodes in the surveillance video coding network that has previously performed parameter learning, and generating a behavior pattern vector of the target educational venue in the basic surveillance video screen at multiple semantic levels, includes: By calling the video feature conversion space in each first feature conversion unit in the multiple semantic encoding nodes, the monitoring video picture vector is transformed into a multi-semantic level video feature domain to generate a behavior pattern vector of the target education place in the basic monitoring video picture at multiple semantic levels; The performing multi-semantic level behavior feature domain transformation on the student behavior vectors respectively includes: According to the behavior coding network that has previously performed parameter learning, the student behavior vectors are respectively transformed into multi-semantic level behavior feature domains; The behavior coding network includes a plurality of cascaded second feature conversion units, and a behavior focusing unit is connected between each two adjacent second feature conversion units; the number of the second feature conversion units is the same as the number of the first feature conversion units in the surveillance video coding network; The method of performing multi-semantic level behavior feature domain transformation on the student behavior vectors based on the behavior coding network that has previously performed parameter learning includes: When performing a first semantic level behavior feature domain transformation on the student behavior vector, using the first-layer second feature transformation unit among the plurality of second feature transformation units, the student behavior vector is transformed into a first semantic level behavior feature domain, and a first semantic level representation vector of the student behavior record document after the first semantic level behavior feature domain transformation is generated; After performing the Yth semantic level behavior feature domain transformation on the student behavior vector, using the Yth behavior focusing unit in the behavior coding network, the feature interaction aggregation is performed on the Yth semantic level representation vector after the Yth semantic level behavior feature domain transformation and the behavior pattern vector generated by the Yth first feature conversion unit in the monitoring video coding network to generate the Yth feature interaction aggregation result; When performing a Zth semantic level behavior feature domain transformation on the student behavior vector, using a Zth second feature transformation unit among the plurality of second feature transformation units, performing a Zth semantic level behavior feature domain transformation on the student behavior vector and the Z-1th feature interaction aggregation result generated by the Z-1th behavior focusing unit, and generating a Zth semantic level representation vector of the student behavior record document after the Zth semantic level behavior feature domain transformation; The representation vector generated after the last semantic-level behavior feature domain transformation of the multi-semantic-level behavior feature domain transformation is output as the feature interaction aggregation vector generated after the multi-semantic-level behavior feature domain transformation.

2. The educational venue management method based on monitoring big data according to claim 1 is characterized in that: The plurality of second feature conversion units include: one or more second feature conversion units for performing information dimension reduction on the student behavior vector, and one or more second feature conversion units for performing information dimension increase on the student behavior vector; Among them, the number of second feature conversion units used for performing information dimensionality reduction on the student behavior vector is the same as the number of second feature conversion units used for performing information dimensionality upgrade on the student behavior vector; and, the one or more second feature conversion units used for performing information dimensionality upgrade on the student behavior vector are connected after the one or more second feature conversion units used for performing information dimensionality reduction on the student behavior vector.

3. The educational venue management method based on monitoring big data according to claim 1 is characterized in that: Before performing a first semantic level behavior feature domain transformation on the student behavior vector, the method further includes: Based on a preset random vector generation mechanism, generating a random representation vector of the behavior coding network; Loading the random representation vector and the student behavior vector as input vectors into the behavior encoding network; The method of using the first-layer second feature conversion unit among the plurality of second feature conversion units to perform a first semantic level behavior feature domain transformation on the student behavior vector to generate a first semantic level representation vector of the student behavior record document after the first semantic level behavior feature domain transformation comprises: The first-layer second feature conversion unit is used to perform a first semantic-level behavior feature domain transformation on the student behavior vector and the random representation vector to generate the first semantic-level representation vector.

4. The educational venue management method based on monitoring big data according to claim 1 is characterized in that: The size of the Yth semantic level representation vector is the same as the size of the behavior pattern vector generated by the Yth first feature conversion unit; The Yth behavior focusing unit in the behavior coding network is used to perform the feature interaction aggregation on the Yth semantic level representation vector after the Yth semantic level behavior feature domain is transformed and the behavior pattern vector generated by the Yth first feature conversion unit in the monitoring video coding network to generate the Yth feature interaction aggregation result, including: Using the Yth behavior focusing unit in the behavior encoding network, the Yth semantic level representation vector and the behavior pattern vector generated by the Yth first feature conversion unit are merged in the vector component dimension to generate a merged vector; performing an upper limit compression operation and a mean compression operation on the merged vector respectively to generate an upper limit compression vector and a mean compression vector, and outputting the upper limit compression vector and the mean compression vector as a compression vector; Performing a mapping operation on the compressed vector to generate an association array between the Yth semantic level representation vector and the behavior pattern vector generated by the Yth first feature conversion unit; Perform feature interaction aggregation on the merged vector according to the association array to generate the Yth feature interaction aggregation result.

5. The educational venue management method based on monitoring big data according to claim 1 is characterized in that: The step of extracting vectors from the basic surveillance video screen and the student behavior record document to generate surveillance video screen vectors and student behavior vectors includes: Calling a behavior vector generator to generate a behavior vector for the student behavior record document to generate the student behavior vector; The monitoring video vector generator is called to generate the monitoring video vector for the basic monitoring video picture, so as to generate the monitoring video picture vector.

6. The educational venue management method based on monitoring big data according to any one of claims 1 to 5, characterized in that: The method is implemented based on a summary generation network model that has previously been subjected to parameter learning; the summary generation network model includes a first target network and a second target network; the first target network includes at least a behavior vector generator and a behavior encoding network, and the second target network includes a monitoring video vector generator and a monitoring video encoding network; The training step of the summary generation network model includes: Loading the template surveillance video picture in the training data into the second target network, extracting the surveillance video picture vector from the template surveillance video picture using the surveillance video vector generator in the second target network, and generating the template surveillance video picture vector; Using the surveillance video encoding network in the second target network to perform a multi-semantic level video feature domain transformation on the template surveillance video screen vector, generating a template behavior pattern vector of the template target education venue in the template surveillance video screen at multiple semantic levels; Loading the template student behavior record document in the training data into the first target network, extracting text vectors from the template student behavior record document using the behavior vector generator in the first target network, and generating a template student behavior vector; Using the behavior encoding network in the first target network to perform multi-semantic level behavior feature domain transformation on the template student behavior vectors, and generate a template feature interaction aggregation vector; Calculating training cost parameters of the summary generation network model according to the template behavior pattern vector and the template feature interaction aggregation vector to generate training cost parameters of the summary generation network model; Optimizing the parameters of the summary generation network model according to the training cost parameters to generate an optimized summary generation network model; The step of calculating the training cost parameters of the summary generation network model based on the template behavior pattern vector and the template feature interaction aggregation vector to generate the training cost parameters of the summary generation network model includes: Calculating a first training cost parameter for the summary generation network model according to the template feature interaction aggregation vector and the template monitoring video screen to generate a first training cost parameter; Calculating a second training cost parameter for the summary generation network model according to the template behavior pattern vector and the template feature interaction aggregation vector to generate a second training cost parameter; Outputting the sum of the first training cost parameter and the second training cost parameter as the training cost parameter of the summary generation network model; The step of calculating a first training cost parameter for the summary generation network model based on the template feature interaction aggregation vector and the template monitoring video screen to generate the first training cost parameter includes: Summarizing the template feature interaction aggregation vector to generate first template summary screen data that simultaneously represents an education venue vector of the template target education venue and a student behavior description vector of the template student behavior record document; Outputting the mean square error value between the first template summary screen data and the template monitoring video screen as the first training cost parameter; The step of calculating a second training cost parameter for the summary generation network model based on the template behavior pattern vector and the template feature interaction aggregation vector to generate the second training cost parameter includes: Summarizing the template behavior pattern vector to generate second template summary screen data; Summarizing the template feature interaction aggregation vector to generate first template summary screen data that simultaneously represents an education venue vector of the template target education venue and a student behavior description vector of the template student behavior record document; Extracting regional attention features from the second template summary screen data and the first template summary screen data respectively to generate a first regional attention feature and a second regional attention feature; The cosine distance between the first region attention feature and the second region attention feature is output as the second training cost parameter.

7. An educational venue management system based on monitoring big data, characterized in that: The education venue management system based on monitoring big data includes a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and the machine-executable instructions are loaded and executed by the processor to implement the education venue management method based on monitoring big data as described in any one of claims 1-6.

Images