Offline examination room abnormal behavior monitoring and detecting system based on multi-modal fusion

Abstract

The invention discloses an offline examination room abnormal behavior monitoring and detecting system based on multi-modal fusion, and relates to an examination room abnormal behavior detecting system, a data processing layer of the system runs on edge computing equipment, and pre-processing and feature extraction are carried out on multi-modal data. And the analysis decision-making layer adopts a graph neural network (GNN) to carry out dynamic weighted fusion on the multi-modal features, carries out time sequence analysis on fused feature vectors in combination with an LSTM model, and identifies abnormal behaviors (such as using electronic equipment and crossing over). And the system integration layer realizes data storage and user interaction through the cloud (Ali cloud OSS), so that invigilators can check abnormal behavior reports in real time. Through the multi-modal data fusion and deep learning technology, the problems that an existing examination room monitoring system depends on single-modal data and is high in misjudgment rate and insufficient in real-time performance are solved, and the examination room monitoring system has the advantages of being efficient, accurate and real-time and can be widely applied to various examination room monitoring scenes to ensure fairness and safety of examinations.

Description

Technical Field

[0001] The present invention relates to a monitoring and detection system for abnormal behaviors in an examination room, and particularly to an offline examination room abnormal behavior monitoring and detection system based on multi-modal fusion. Background Art

[0002] Examination room monitoring technology is an important means to ensure the fairness and security of examinations. With the continuous expansion of the scale of educational examinations, traditional monitoring methods have been difficult to meet the needs of modern examination management. Existing examination room monitoring systems mainly rely on single-modal data, such as video monitoring or audio monitoring, and have obvious limitations. Video monitoring captures candidates' behaviors through cameras, but it relies on manual observation, is prone to missing abnormal behaviors, and is affected by factors such as light and occlusion, with a relatively high misjudgment rate. Audio monitoring collects ambient sounds through microphones and can detect abnormal behaviors that are difficult to capture by visual monitoring, but it is also affected by environmental noise interference and has a relatively high misjudgment rate.

[0003] Existing technologies generally rely on single-modal data, making it difficult to comprehensively and accurately identify abnormal behaviors, and they have poor real-time performance and adaptability. For example, video monitoring cannot detect hidden electronic devices, while audio monitoring cannot identify silent abnormal behaviors. In addition, existing systems usually rely on manual observation or post-event analysis and cannot detect and handle abnormal behaviors in real time. In an examination room environment with dim light or high noise, the performance of existing systems will further decline. With the continuous expansion of the scale of educational examinations, the market's demand for an efficient, accurate, and real-time examination room monitoring system is increasing day by day. Summary of the Invention

[0004] The purpose of the present invention is to propose an offline examination room abnormal behavior monitoring and detection system based on multi-modal fusion. By combining multiple data sources such as vision, sound, infrared, and electromagnetic sensors, and using deep learning technology and edge computing technology, the system can achieve efficient, accurate, and real-time detection of abnormal behaviors, and can relatively accurately identify abnormal behaviors in the examination room (such as whispering to each other, looking around, using electronic devices, etc.), so it has high practical value and broad prospects for promotion.

[0005] The technical solution of the present invention is as follows: An offline examination room abnormal behavior monitoring and detection system based on multi-modal fusion, the system includes the following modules: A data acquisition module for collecting multi-modal data in the examination room, including: A camera module for capturing candidates' behaviors (such as head movements, hand movements, body postures); A microphone module for collecting ambient sounds in the examination room and identifying abnormal sounds (such as the sound of whispering to each other); An infrared sensor module for detecting candidates' body temperature changes or abnormal heat sources (such as hidden electronic devices); A signal analyzer module for monitoring changes in the examination room environment (such as electromagnetic interference); A data processing module for preprocessing and feature extraction of multimodal data, including: A visual data processing module for object detection and behavior analysis using YOLOv8; A sound data processing module for audio noise reduction and feature extraction using PyAudio; An infrared data processing module for heat source detection and temperature analysis using OpenCV; An electromagnetic data processing module for data standardization and abnormal feature extraction using NumPy in Python; A multimodal fusion module for fusing multimodal data and performing cross-modal feature fusion using a graph neural network (GNN); An anomaly detection module for identifying abnormal behaviors in the examination room and performing temporal analysis on the fused multimodal data using an LSTM model; A system integration module for deploying the algorithm model to the cloud (Alibaba Cloud OSS) and providing a user interface for invigilators to view abnormal behavior reports.

[0006] For the described offline examination room abnormal behavior monitoring and detection system based on multimodal fusion, the cameras, microphones, infrared sensors, and environmental sensors in the data acquisition module perform data preprocessing and preliminary analysis through the edge computing device NVIDIA Jetson Xavier NX.

[0007] For the described offline examination room abnormal behavior monitoring and detection system based on multimodal fusion, the multimodal fusion module uses a graph neural network (GNN) to perform dynamic weighted fusion on visual, sound, infrared, and electromagnetic data.

[0008] For the described offline examination room abnormal behavior monitoring and detection system based on multimodal fusion, the anomaly detection module performs temporal analysis on the fused multimodal data through an LSTM model to identify abnormal behaviors (such as whispering to each other and using electronic devices).

[0009] For the described offline examination room abnormal behavior monitoring and detection system based on multimodal fusion, the system integration module deploys the algorithm model to the cloud (Alibaba Cloud OSS) and provides real-time abnormal behavior reports through a Web interface or a mobile application.

[0010] For the described offline examination room abnormal behavior monitoring and detection system based on multimodal fusion, the detection process of the system includes the following steps: Collect multimodal data in the examination room through cameras, microphones, infrared sensors, and signal analyzers; Preprocess and extract features from multimodal data; Use a graph neural network (GNN) to fuse multimodal data; Perform time series analysis on the fused multimodal data through an LSTM model to identify abnormal behaviors; Deploy the algorithm model to the cloud (Alibaba Cloud OSS) and provide a user interface for invigilators to view abnormal behavior reports. Description of the Drawings

[0011] Figure 1 It is the overall system architecture diagram of the present invention; Figure 2 It is the flowchart of the multimodal fusion algorithm of the present invention; Figure 3 It is the example diagram of abnormal behavior detection of the present invention. Detailed Implementation Manner

[0012] The detailed implementation manner of the present invention realizes real-time detection and early warning of abnormal behaviors in the examination room through a multi-level system architecture. The system forms a complete closed loop from data collection, processing, analysis to final user interaction. Each link is closely connected to ensure the efficient flow and processing of data.

[0013] A system for monitoring and detecting abnormal behaviors in an offline examination room based on multimodal fusion according to the present invention. In terms of data collection, it captures the behaviors of candidates through cameras (Hikvision DS-2CD2143G0-I), such as head movements, hand movements, and body postures, and collects the sounds in the examination room environment through microphones (Shure MXA910) to identify abnormal sounds, such as whispering and the sounds of using electronic devices. It detects the body temperature changes or abnormal heat sources (such as hidden electronic devices) of candidates through infrared sensors (Hikvision DS-2TP31B-3AUF), and monitors the changes in the examination room environment (such as electromagnetic interference) through a signal analyzer (RF Explorer).

[0014] In terms of data processing and fusion, preprocess and extract features from multimodal data: use YOLOv8 for object detection and behavior analysis in visual data processing; use PyAudio for audio noise reduction and feature extraction in sound data processing; use OpenCV for heat source detection and temperature analysis in infrared data processing; use Python's NumPy for data standardization and abnormal feature extraction in environmental data. And finally use a graph neural network (GNN) to perform dynamic weighted fusion on multimodal data. Abnormal behavior detection: perform time series analysis on the fused multimodal data through an LSTM model to identify abnormal behaviors in the examination room (such as whispering to each other, using electronic devices, etc.).

[0015] In terms of system integration, the algorithm model is deployed to the cloud (Alibaba Cloud OSS), and real-time abnormal behavior reports are provided through a web interface or a mobile application.

[0016] Cameras (Hikvision DS-2CD2143G0-I) are deployed at the four corners of the examination room to ensure full coverage of the entire examination room area. These cameras transmit video streams in real-time through the RTSP protocol, capturing details of the examinees' behaviors, such as head movements, hand movements, and body postures. Microphones (Shure MXA910) are deployed in the center of the examination room and use high-sensitivity array technology to collect ambient sounds, capable of capturing abnormal sounds such as whispered conversations. Infrared sensors (Hikvision DS-2TP31B-3AUF) are deployed at the entrance and in the center of the examination room, detecting changes in the body temperature of examinees or abnormal heat sources (such as hidden electronic devices) through thermal imaging technology. A signal analyzer (RF Explorer) is deployed in the center of the examination room to monitor changes in electromagnetic signals in real-time and identify electromagnetic interference that may be caused by electronic devices. The above audio-visual and sensor devices are connected to the edge computing device (NVIDIA Jetson Xavier NX) through USB interfaces to ensure stable data transmission.

[0017] In the data processing stage, the edge computing device (NVIDIA Jetson Xavier NX) undertakes the core computing tasks. Visual data is processed in real-time through the YOLOv8 model to extract key features of the examinees' behaviors, such as head postures, hand movements, etc. The YOLOv8 model has been pre-trained and fine-tuned to efficiently identify abnormal behaviors in the examination room, such as looking around, poking the head forward and backward, etc. Sound data is denoised and feature-extracted through the PyAudio library, and combined with a pre-trained speech recognition model of deep learning to identify abnormal sounds, such as the whispered conversations and chatting of examinees. Infrared data is processed through the OpenCV library to detect abnormal heat sources (such as mobile phones, electronic devices) in thermal imaging images and extract the location and temperature data of the heat sources. Electromagnetic signal data is normalized through the NumPy library of Python to extract abnormal features of electromagnetic signals (such as electromagnetic interference at specific frequencies). All processed data is accelerated by the GPU of the edge computing device to ensure real-time performance and efficiency.

[0018] In the analysis and decision-making stage, the system uses multi-modal fusion technology and deep learning models to analyze the processed data. The multi-modal fusion module is based on the graph neural network (GNN). It takes the features of visual, sound, infrared, and electromagnetic signal data as the nodes of the graph, and through the attention mechanism, dynamically weights and fuses them to generate a fused multi-modal feature vector. This fusion method can make full use of the complementarity of different modal data and improve the accuracy of abnormal behavior detection. The anomaly detection module uses the LSTM (Long Short-Term Memory) model to perform temporal analysis on the fused multi-modal feature vector to identify abnormal behaviors (such as inattention). The LSTM model can capture the temporal patterns of the examinee's behavior and further improve the detection accuracy. The detection results are transmitted to the cloud (Alibaba Cloud OSS) through the network interface of the edge computing device to ensure the secure storage and efficient access of data.

[0019] In the system integration stage, the cloud (Alibaba Cloud OSS) undertakes the tasks of data storage and user interaction. The detection results and the original data are uploaded to Alibaba Cloud OSS through the edge computing device to ensure the secure storage and efficient access of data. The user interface is developed using the Flask framework, providing a Web interface or a mobile application for the invigilators to view the abnormal behavior reports in real time. The user interface uses ECharts to visualize the abnormal behavior data (such as heat maps and time series graphs), and realizes the real-time update and interaction of data through the API interface of Alibaba Cloud. The invigilators can view the specific details of the abnormal behaviors (such as the time, location, and type of the abnormal behavior) through the user interface and take corresponding handling measures.

[0020] The operation of the entire system depends on the collaborative work of edge computing and cloud computing. The data processing layer and the analysis and decision-making layer run on the edge computing device, and the processing speed is improved through GPU accelerated computing. The system integration layer is connected to the edge computing device through the API interface of Alibaba Cloud to realize the cloud storage of data and user interaction. This hierarchical architecture not only ensures the real-time performance and efficiency of the system, but also provides a convenient data management and interaction method through cloud storage and the user interface.

[0021] Example Scenarios Scenario 1: Candidate A looks down at the mobile phone: Visual data: YOLOv8 detects that Candidate A is looking down and locates the position of their hand.

[0022] Infrared data: OpenCV detects an abnormal heat source at the position of Candidate A's hand, further confirming the presence of the mobile phone.

[0023] System judgment: There may be an abnormal behavior for Candidate A, and the system issues a warning to the invigilator through the user interface.

[0024] Scenario 2: Candidate B frequently looks around and whispers to others: Visual data: YOLOv8 detected that candidate B's head turned frequently, and its behavior pattern was analyzed by combining time series data.

[0025] Audio data: PyAudio detected that someone was speaking, and combined with visual data to confirm that candidate A might have abnormal behavior.

[0026] System judgment: Candidate B might have abnormal behavior, and the system sent a reminder to the invigilator through the user interface.

Claims

1. An offline examination room abnormal behavior monitoring and detection system based on multimodal fusion, characterized in that, The system includes the following modules: A data acquisition module for collecting multimodal data in the examination room, including: A camera module for capturing the behaviors of examinees (such as head movements, hand movements, body postures); A microphone module for collecting the sounds in the examination room environment and identifying abnormal sounds (such as whispering sounds); An infrared sensor module for detecting the temperature changes or abnormal heat sources of examinees (such as hidden electronic devices); A signal analyzer module for monitoring the changes in the examination room environment (such as electromagnetic interference); A data processing module for preprocessing and feature extraction of multimodal data, including: A visual data processing module for object detection and behavior analysis using YOLOv8; An audio data processing module for audio noise reduction and feature extraction using PyAudio; An infrared data processing module for heat source detection and temperature analysis using OpenCV; An electromagnetic data processing module for data standardization and abnormal feature extraction using NumPy in Python; A multimodal fusion module for fusing multimodal data and performing cross-modal feature fusion using a graph neural network (GNN); An anomaly detection module for identifying abnormal behaviors in the examination room and performing temporal analysis on the fused multimodal data using an LSTM model; A system integration module for deploying the algorithm model to the cloud (Alibaba Cloud OSS) and providing a user interface for invigilators to view the abnormal behavior reports; 2. The offline examination room abnormal behavior monitoring and detection system based on multi-modal fusion according to claim 1, characterized in that, The cameras, microphones, infrared sensors, and environmental sensors in the data acquisition module perform data preprocessing and preliminary analysis through an edge computing device NVIDIA Jetson Xavier NX; 3. The offline examination room abnormal behavior monitoring and detection system based on multimodal fusion according to claim 1, characterized in that, The multimodal fusion module uses a graph neural network (GNN) to perform dynamic weighted fusion on visual, audio, infrared, and electromagnetic data; 4. The offline examination room abnormal behavior monitoring and detection system based on multi-modal fusion according to claim 1, characterized in that The anomaly detection module performs temporal analysis on the fused multimodal data through an LSTM model to identify abnormal behaviors (such as whispering and using electronic devices); 5. The offline examination room abnormal behavior monitoring and detection system based on multimodal fusion according to claim 1, characterized in that, The system integration module deploys the algorithm model to the cloud (Alibaba Cloud OSS) and provides real-time abnormal behavior reports through a web interface or a mobile application; 6. The offline examination room abnormal behavior monitoring and detection system based on multimodal fusion according to claim 1, characterized in that, The detection process of the system includes the following steps: Collect multimodal data in the examination room through cameras, microphones, infrared sensors, and signal analyzers; Preprocess and extract features from the multimodal data; Fuse the multimodal data using a graph neural network (GNN); Perform temporal analysis on the fused multimodal data through an LSTM model to identify abnormal behaviors; Deploy the algorithm model to the cloud (Alibaba Cloud OSS) and provide a user interface for invigilators to view the abnormal behavior reports.

Images