Online intelligent supervision method based on multi-modal feature fusion
By building an online intelligent supervision system that integrates multimodal features, combining video spatiotemporal features, frequency domain features and personnel relationship features, and using models such as Transformer-XL and Action Transformer Network, the low recognition accuracy and poor practicality in the existing technology are solved. The problem realizes accurate perception and recognition of complex behaviors.
Patent Information
- Application Number
- CN202411561508.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-04
- Publication Date
- 2025-02-18
AI Technical Summary
The existing invigilator and supervisory systems rely on a single video space-time feature, making it difficult to accurately identify and understand complex behaviors, especially the behavioral differences between different individuals, the behavioral duration is uncertain, and the lack of frequency domain feature analysis, resulting in low recognition accuracy and poor practicality.
Using an online intelligent supervision method based on multimodal feature fusion, an online intelligent supervision system for multimodal feature fusion is constructed, combining video spatiotemporal features, frequency domain features and personnel relationship features, and advanced models such as Transformer-XL and Action Transformer Network are used. Conduct behavioral analysis and identification.
It realizes accurate perception and recognition of complex behaviors, improves the recognition accuracy of the model and the ability to understand complex scenarios, and can sensitively perceive the relationships between the inspection teachers and students and other behavioral dynamics.
Smart Images
Figure CN119478606A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of teaching systems, and in particular to an online intelligent supervision method based on multimodal feature fusion. Background Art
[0002] The general implementation method of existing invigilation and supervision systems is to identify behaviors by extracting spatiotemporal key points in the video. These key points are usually the most significant parts of the video, or the dense trajectories in the video are tracked by optical flow method to extract motion features. These visual-based features are used as input to use convolution and its variants or two-stream networks (Two-Stream Networks), 3D convolutional neural networks (3D CNNs), etc. or combine recurrent neural networks (RNNs) and long short-term memory networks (LSTM) or Transformer and multi-head attention mechanisms to capture dependencies in the time dimension.
[0003] The disadvantages of the prior art are:
[0004] The information channel is single and mainly relies on the spatiotemporal information obtained by continuous image processing, so the effect accuracy is limited.
[0005] The behaviors of different individuals vary greatly, the differences within the class are significant, the recognition difficulty is high and the accuracy is low.
[0006] The duration of a behavior ranges from a few seconds to several minutes, and it is difficult to define the start and end time nodes.
[0007] The lack of relationships between people and between people and objects in the picture leads to poor practicality in solving business problems.
[0008] It is basically based on the characteristics and analysis of space and time domains, lacks the combination with frequency domain characteristics, and is poor at solving business problems. Summary of the invention
[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and propose an online intelligent supervision method based on multimodal feature fusion, which can sensitively perceive, for example, the action characteristics of the inspector's influence on the cheater, the actions of multiple cheaters, the relationship between the inspectors and students, bullying, fighting, etc.
[0010] The present invention solves the technical problem by adopting the following technical solutions:
[0011] An online intelligent supervision method based on multimodal feature fusion includes building an online intelligent supervision system based on multimodal feature fusion, wherein the online intelligent supervision system based on multimodal feature fusion includes an external data loading module, a video parsing module, a face recognition module, a personnel ID coding module, an event frequency domain construction module, a personnel relationship construction module, a result / intermediate result storage and update module, a fusion coding module, a personnel position coding module, a video spatiotemporal feature extraction module, a multimodal fusion behavior analysis model, a behavior recognition, and a behavior prediction result output module, wherein the external data loading module is respectively connected to the personnel ID coding module and the event frequency domain construction module, the video parsing module, the face recognition module, and the personnel ID coding module are connected in sequence, the event frequency domain construction module, the personnel relationship construction module, the fusion coding module, and the multimodal fusion behavior analysis model are connected in sequence, the video parsing module is connected to the video spatiotemporal feature extraction module, the video spatiotemporal feature extraction module is respectively connected to the personnel relationship construction module and the multimodal fusion behavior analysis model, the personnel relationship construction module is connected to the fusion coding module, the multimodal fusion behavior analysis model, the behavior recognition, the behavior prediction result output module, the result / intermediate result storage, and the external data loading module are connected in sequence.
[0012] Moreover, the external data loading module loads external data through a unified interface for system use, wherein the external data includes video source, face library, teacher library, class library and event library, and the external data loading module outputs the recognition result through a unified interface.
[0013] Moreover, the video analysis module is used to extract original video frame image data from the video stream; the face recognition module is used to use the face recognition algorithm to identify the characteristics of people in the scene, so as to determine the identity of people in the scene; the person ID encoding module is used to retrieve the face library for face feature comparison based on the face recognition results, and obtain the person's business identity ID.
[0014] Moreover, the video spatiotemporal feature extraction module extracts the spatiotemporal features of each frame of video through Inflated 3D ConvNet; the event frequency domain construction module is used to extract the frequency domain features of the event using fast Fourier transform.
[0015] Moreover, the personnel relationship construction module constructs personnel relationship graph relation data in the video through logical coding; the personnel position coding module retains the positions of personnel in the coded video by using relative position coding.
[0016] Moreover, the fusion coding module uses the personnel relationship data and personnel position coding data constructed by personnel relationship to construct a feature vector of comprehensive information such as personnel relationship and personnel position through the GCN model.
[0017] Moreover, the multimodal fusion behavior analysis model takes the video spatial flow features and optical flow features extracted from the video spatiotemporal features, and the GCN fusion features generated by fusion coding as input, uses some characteristics of the Transformer-XL encoder to transform the Action Transformer Network model, generates a multimodal fusion behavior analysis model, and analyzes the behavior in the video; the behavior recognition and behavior prediction result output module uses the softmax algorithm to output the behavior recognition and prediction results.
[0018] Furthermore, the result / intermediate result saving and updating module is used to save and update the results or intermediate results.
[0019] The advantages and positive effects of the present invention are:
[0020] 1. The present invention constructs an online intelligent supervision method based on multimodal feature fusion through an external data loading module, a video parsing module, a face recognition module, a personnel ID encoding module, an event frequency domain construction module, a personnel relationship construction module, a result / intermediate result preservation, an update module, a fusion encoding module, a personnel position encoding module, a video spatiotemporal feature extraction module, a multimodal fusion behavior analysis model, a behavior recognition, and a behavior prediction result output module. The present invention uses multimodal feature fusion technology to comprehensively encode the spatial features, time features, and frequency domain features of related action events in video stream information, the relationship between participating characters, and the spatial position relationship and other multi-dimensional features, and uses the behavior recognition intelligent body trained by multi-dimensional features in intelligent supervision. The model can sensitively perceive, for example, the action characteristics of the patrol teacher's influence on the cheater, the actions of multiple cheaters, and the relationship between the supervisors, bullying, fighting, etc. can all be accurately perceived.
[0021] 2. The present invention combines spatiotemporal features with frequency domain features to more comprehensively capture the dynamic changes of actions and improve the recognition accuracy of the model.
[0022] 3. The present invention utilizes advanced models such as Two-Stream Networks and Transformer to efficiently extract and aggregate spatiotemporal features in videos.
[0023] 4. Through graph-related technologies such as GCN, the spatial relationship between multiple people can be effectively modeled, improving the ability to understand actions in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 is a structural diagram of the system of the present invention;
[0025] Figure 2 This is a structural diagram of the multimodal fusion behavior analysis model of the present invention. DETAILED DESCRIPTION
[0026] The present invention is further described in detail below with reference to the accompanying drawings.
[0027] The construction idea of the present invention is:
[0028] The present invention uses multimodal feature fusion technology to comprehensively encode the spatial features, temporal features, frequency domain features of related action events, relationships between participating characters, spatial position relationships and other multi-dimensional features in video stream information, and uses the behavior recognition agent trained with multi-dimensional features in intelligent supervision. The model can sensitively perceive, for example, the action characteristics of the inspector's influence on the cheater, the actions of multiple cheaters, and the relationship between the inspector and the students, bullying, fighting, etc. It can accurately perceive. The technical points are as follows:
[0029] 1. Face recognition and encoding
[0030] First, face recognition technology is used to identify each person in the video, and combined with coding technology, a unique code is generated for each person. This code contains not only personal identity information, but also their specific location information in the video. This ensures that everyone can be uniquely identified and tracked throughout the video analysis process.
[0031] 2. Constructing position relationship features
[0032] Using graph convolutional networks (GCN) or other graph-related technologies, the positional relationship features between people are constructed based on the features generated in the first step and other information (such as distance, relative position, etc.). Specifically, each person can be regarded as a node in the graph, and the edges between the nodes represent the spatial relationship between them. Through GCN, these relationship features can be effectively captured and modeled.
[0033] 3. Behavior Analysis Model
[0034] In the behavior analysis model, Two-Stream Networks can be used to process RGB frames and optical flow information separately to capture the spatiotemporal dynamic features in the video. Then, combined with the frequency domain information features of character relationships and actions, the Action Transformer Network is used to aggregate these features for higher-level action classification and prediction.
[0035] 4. Frequency domain feature extraction and fusion
[0036] Frequency domain features are extracted from the analyzed spatiotemporal features of the action. Specifically, the action sequence of each person can be Fourier transformed to extract frequency domain features. These frequency domain features can reflect the periodicity and frequency information of the action. Then, the spatiotemporal features and frequency domain features are fused to form comprehensive features for the final action analysis and judgment.
[0037] 5. Output classification or prediction results
[0038] Based on the above features, the final action classification result or prediction result is output through a classifier or prediction model. The classifier can be a traditional machine learning model (such as SVM, random forest) or a deep learning model (such as a fully connected neural network). The prediction model can predict actions based on a time series model (such as LSTM, GRU).
[0039] Online intelligent supervision methods based on multimodal feature fusion, such as Figure 1 As shown, it includes an external data loading module, a video parsing module, a face recognition module, a personnel ID coding module, an event frequency domain construction module, a personnel relationship construction module, a result / intermediate result storage, an update module, a fusion coding module, a personnel position coding module, a video spatiotemporal feature extraction module, a multimodal fusion behavior analysis model, a behavior recognition, and a behavior prediction result output module, wherein the external data loading module is respectively connected to the personnel ID coding module and the event frequency domain construction module, the video parsing module, the face recognition module, and the personnel ID coding module are connected in sequence, the event frequency domain construction module, the personnel relationship construction module, the fusion coding module, and the multimodal fusion behavior analysis model are connected in sequence, the video parsing module is connected to the video spatiotemporal feature extraction module, the video spatiotemporal feature extraction module is respectively connected to the personnel relationship construction module and the multimodal fusion behavior analysis model, the personnel relationship construction module is connected to the fusion coding module, the multimodal fusion behavior analysis model, the behavior recognition, the behavior prediction result output module, the result / intermediate result storage, and the external data loading module are connected in sequence.
[0040] The external data loading module uses the interface layer to read external data, and can read external business data, personnel information, teacher information, class information and other additional information. The external data includes video source, face library, teacher library, class library and event library. At the same time, the external data loading module outputs the recognition results through a unified interface.
[0041] The video parsing module is used to pull the video and decapsulate and decode the video to extract the video frame
[0042] The face recognition module is used to use the face recognition algorithm to identify the characteristics of people in the scene, so as to determine the identity of people in the scene.
[0043] The personnel ID encoding module is used to retrieve the face database for facial feature comparison based on the results of face recognition and obtain the personnel business identity ID.
[0044] The video spatiotemporal feature extraction module extracts the spatiotemporal features of each frame of video through Inflated 3D ConvNet; Inflated 3D ConvNet (I3D) is used to extract the spatiotemporal features of each frame of video. The I3D model can process the spatial and temporal information of the video at the same time by expanding the 2D convolution to 3D convolution. The optical flow between adjacent frames is calculated to generate an optical flow map to capture motion information. By using pre-trained 2D network weights, I3D can converge faster and perform better on small data sets.
[0045] The event frequency domain construction module is used to extract the frequency domain features of events using fast Fourier transform. The event library information is read regularly using the unified interface provided by the external data loading module, and the events are classified according to the structure of <person, action, person>, and then the frequency domain feature information of the related events is extracted using fast Fourier transform. The first person is the initiator of the action, and the second person is the receiver of the action. This information can be used to determine the characteristics of the frequency domain information of events that occur when someone performs a certain type of action on someone else.
[0046] The personnel relationship construction module constructs the personnel relationship graph relation data in the video through logical coding.
[0047] The personnel position encoding module uses relative position encoding to retain the position of personnel in the encoded video. By analyzing the spatial information extracted from the spatiotemporal features of the video and the personnel ID information, the position information of the personnel in space is determined. The position feature information is retained by using relative position encoding. The position encoding is generated by combining the position of the face in the video frame (such as the coordinates of the bounding box). The position encoding includes the coordinates of the center point of the face in the image (x, y) and the width and height (w, h) of the bounding box. Steps: Detect facial feature points: Use a facial feature point detection algorithm (such as Dlib) to detect facial feature points. Calculate relative position encoding: Calculate relative position encoding based on the position of the facial feature points.
[0048] Generate a unique code: Combine the face recognition ID and relative position code to generate a unique code.
[0049] The fusion coding module uses the personnel relationship data and personnel location coding data constructed by personnel relationships to construct feature vectors of comprehensive information such as personnel relationships and personnel locations through the GCN model. A graph structure is constructed in which nodes represent each person and edges represent the relationship between people (such as distance). GCN is used to extract features in the graph structure and capture the positional relationship between people.
[0050] The multimodal fusion behavior analysis model takes the video spatial flow features and optical flow features extracted from the video spatiotemporal features, as well as the GCN fusion features generated by fusion coding as input, uses some characteristics of the Transformer-XL encoder to transform the Action Transformer Network model, generates a multimodal fusion behavior analysis model, and analyzes the behavior in the video; the behavior recognition and behavior prediction result output module uses the softmax algorithm to output the behavior recognition and prediction results.
[0051] Renovation steps:
[0052] 1. Introducing relative position encoding
[0053] Transformer-XL uses relative position encoding to replace the traditional absolute position encoding to better capture long-term dependencies. Relative position encoding can be implemented by the following formula:
[0054]
[0055] Among them, R is the relative position encoding matrix.
[0056] 2. Memory mechanism
[0057] Transformer-XL introduces a memory mechanism that can pass the hidden state of the previous period as memory to the current time step, thereby capturing longer-term dependencies. The specific implementation steps are as follows:
[0058] Initialize memory: In each Transformer layer, a memory matrix is initialized to store the hidden state of the previous period.
[0059] Update memory: At each time step, add the hidden state of the current time step to the memory matrix and remove the oldest hidden state, keeping the length of the memory matrix fixed.
[0060] 3. Model architecture adjustment
[0061] Adjust the architecture of Action Transformer Network to that of Transformer-XL. The specific steps are as follows:
[0062] Encoder layer: Use the encoder layer of Transformer-XL, which includes relative position encoding and memory mechanism.
[0063] The result / intermediate result saving and updating module is used to save and update the results or intermediate results.
[0064] like Figure 2As shown in the figure, the model includes input layer, feature embedding layer, relative position encoding layer, memory mechanism, ActionTransformer encoder layer, feature fusion layer, CNN convolution layer and classification layer.
[0065] 1. Input layer
[0066] Function: Receives raw video data and fused coded feature vectors. Input: Video frame sequence and fused coded feature vectors.
[0067] 2. Feature Embedding Layer
[0068] Purpose: Convert the fused encoded features into an embedding vector that matches the dimension of the video data. Implementation: Use a linear transformation or embedding layer.
[0069] Purpose: The main purpose of the feature embedding layer is to convert the fused coded features into an embedding vector of the same dimension as the video features. This step is to ensure that different types of features can be processed and compared in the same feature space. Through feature embedding, different types of features can be unified into the same dimension, which facilitates subsequent feature fusion and processing.
[0070] 3. Relative Position Encoding Layer
[0071] Purpose: Introduces relative position encoding to enhance the model's ability to perceive long-term dependencies. Implementation: Uses the relative position encoding module.
[0072] 4. Memory mechanism
[0073] Purpose: Save the hidden state of the previous period of time to capture longer-term dependencies. Implementation: Update the memory matrix at each time step.
[0074] 5. Action Transformer Encoder Layer
[0075] Purpose: Processes comprehensive feature vectors and captures spatiotemporal dependencies. Implementation: Uses multi-head self-attention mechanism and feedforward neural network.
[0076] 6. Feature Fusion Layer
[0077] Function: Combine video features and fusion coding features to form a comprehensive feature vector. Implementation: Use feature concatenation or feature fusion technology. Purpose: To utilize information from multiple features to improve the performance of the model. The main purpose of the feature fusion layer is to combine other embedded features with video features to form a comprehensive feature vector. Through feature fusion, the information of video features and other features can be comprehensively utilized to capture richer feature representations, thereby improving the recognition ability of the model.
[0078] 7.CNN Convolutional Layer
[0079] Purpose: Further extract and process the fused features to capture local spatial features and patterns. Implementation: Use convolutional layers and pooling layers.
[0080] 8. Classification Layer
[0081] Function: Classify the comprehensive feature vector and output the action category. Implementation: Use the fully connected layer.
[0082] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.
Claims
1. An online intelligent supervision method based on multimodal feature fusion, characterized by: The invention comprises constructing an online intelligent supervision system of multimodal feature fusion, wherein the online intelligent supervision of multimodal feature fusion comprises an external data loading module, a video analysis module, a face recognition module, a personnel ID coding module, an event frequency domain construction module, a personnel relationship construction module, a result / intermediate result preservation and updating module, a fusion coding module, a personnel position coding module, a video spatiotemporal feature extraction module, a multimodal fusion behavior analysis model, a behavior recognition, and a behavior prediction result output module, wherein the external data loading module is respectively connected to the personnel ID coding module and the event frequency domain construction module, the video analysis module, the face recognition module, and the personnel ID coding module are connected in sequence, the event frequency domain construction module, the personnel relationship construction module, the fusion coding module, and the multimodal fusion behavior analysis model are connected in sequence, the video analysis module is connected to the video spatiotemporal feature extraction module, the video spatiotemporal feature extraction module is respectively connected to the personnel relationship construction module and the multimodal fusion behavior analysis model, the personnel relationship construction module is connected to the fusion coding module, the multimodal fusion behavior analysis model, the behavior recognition, the behavior prediction result output module, the result / intermediate result preservation, and the external data loading module are connected in sequence.
2. The online intelligent supervision method based on multimodal feature fusion according to claim 1, characterized in that: The external data loading module loads external data through a unified interface for system use, wherein the external data includes video source, face library, teacher library, class library and event library, and the external data loading module outputs recognition results through a unified interface.
3. The online intelligent supervision method based on multimodal feature fusion according to claim 1, characterized in that: The video analysis module is used to extract original video frame image data from the video stream; the face recognition module is used to use the face recognition algorithm to identify the characteristics of people in the scene, so as to determine the identity of people in the scene; the person ID encoding module is used to retrieve the face library for face feature comparison based on the face recognition results, and obtain the person's business identity ID.
4. The online intelligent supervision method based on multimodal feature fusion according to claim 1, characterized in that: The video spatiotemporal feature extraction module extracts the spatiotemporal features of each frame of video through Inflated 3D ConvNet; The event frequency domain building block is used to extract the frequency domain features of events using fast Fourier transform.
5. The online intelligent supervision method based on multimodal feature fusion according to claim 1, characterized in that: The personnel relationship construction module constructs personnel relationship graph relation data in the video through logical coding; the personnel position coding module retains the positions of personnel in the coded video by using relative position coding.
6. The online intelligent supervision method based on multimodal feature fusion according to claim 1, characterized in that: The fusion coding module uses the personnel relationship data and personnel position coding data constructed by personnel relationships to construct a feature vector of comprehensive information such as personnel relationships and personnel positions through a GCN model.
7. The online intelligent supervision method based on multimodal feature fusion according to claim 1, characterized in that: The multimodal fusion behavior analysis model takes the video spatial flow features and optical flow features extracted from the video spatiotemporal features, and the GCN fusion features generated by fusion coding as input, uses some characteristics of the Transformer-XL encoder to transform the ActionTransformer Network model, generates a multimodal fusion behavior analysis model, and analyzes the behavior in the video; the behavior recognition and behavior prediction result output module uses the softmax algorithm to output the behavior recognition and prediction results.
8. The online intelligent supervision method based on multimodal feature fusion according to claim 1, characterized in that: The result / intermediate result saving and updating module is used to save and update the result or intermediate result.