College student behavior anomaly early warning method and system based on multi-source data fusion
By using a multi-source data fusion method, real-time collection and analysis of college student behavior data has solved the robustness and accuracy problems of traditional surveillance video in anomaly identification, and achieved efficient identification and timely early warning of abnormal behavior.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG GUGEL INTELLIGENT TECHNOLOGY CO LTD
- Filing Date
- 2026-06-04
- Publication Date
- 2026-07-03
AI Technical Summary
Traditional surveillance videos are easily affected by external environmental factors in abnormal behavior identification, the model recognition is not robust, and image, voice and semantic features are difficult to extract stably, resulting in low accuracy and efficiency in personnel identification.
A multi-source data fusion-based approach is adopted to collect video streams in the target area in real time, extract visual tensors, audio feature sequences and semantic vectors, calculate anomaly coefficients for anomaly warning through multimodal feature alignment and fusion, and combine multi-source data such as access control, classroom attendance, campus consumption and library borrowing data for anomaly determination.
It improves the robustness of recognition and the real-time performance of early warning in complex monitoring scenarios, reduces detection latency and computational redundancy, avoids the limitations of audio noise interference and single visual modality, and improves the detection accuracy and response speed of abnormal behavior.
Smart Images
Figure CN122332841A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and more specifically to a method and system for early warning of abnormal behavior among college students based on multi-source data fusion. Background Technology
[0002] With the continuous advancement of digital and smart campus construction in universities, campus management and student safety are gradually transforming towards informatization and intelligence. The learning status, daily routines, consumption habits, activity patterns, and online behavior of university students during their time on campus can objectively reflect their physical and mental health and safety. However, the traditional management model, which relies on manual patrols, counselor visits, and post-event reporting, struggles to comprehensively, in real-time, and continuously grasp the true dynamics of students, resulting in a significant lag in the perception of potential risks.
[0003] Publication No. CN121259925A discloses a classroom student behavior monitoring system; an identity generation module; an image acquisition module; a data storage module; a behavior analysis module; a multimodal data integration module; and an interactive feedback module. This classroom student behavior monitoring system, through the collaborative work of the identity generation module, image acquisition module, data storage module, behavior analysis module, multimodal data integration module, and interactive feedback module, further optimizes the algorithm design of the head posture analysis unit, hand movement recognition unit, electronic device usage detection unit, and sitting posture assessment unit. With the help of the interactive feedback module, it provides student behavior trajectory maps and abnormal situation reports as dynamic monitoring results to external users, solving the shortcomings of existing technologies in terms of recognition accuracy, real-time performance, multi-dimensional data analysis, and privacy protection.
[0004] Traditional surveillance videos are easily affected by external environmental factors in abnormal behavior identification. The model has poor robustness in practical applications. Image, voice and semantic features are difficult to extract stably and are coarse in the fusion process, resulting in poor accuracy and low efficiency in personnel identification. Summary of the Invention
[0005] The purpose of this invention is to address the problems mentioned in the background art, such as the susceptibility of traditional surveillance video to interference from external environmental factors in abnormal behavior identification, the poor robustness of the model in practical applications, the difficulty in stably extracting image, voice and semantic features and the coarseness in the fusion process, which leads to poor accuracy and low efficiency in personnel identification. Therefore, this invention proposes a method and system for early warning of abnormal behavior of college students based on multi-source data fusion.
[0006] A first aspect of this invention provides a method for early warning of abnormal behavior among college students based on multi-source data fusion, the method comprising: The system collects video streams of people within a target area in real time, and extracts classification features from the video streams to obtain visual tensors, audio feature sequences, and semantic vectors; the target area includes high-risk areas and sub-regions. The visual tensor, the audio feature sequence, and the semantic vector are aligned frame by frame according to a preset frame rate to obtain behavioral features. Events occurring in each sub-region of the behavioral features are obtained, the temporal overlap of each event is calculated to obtain associated events, and the associated events are sorted in chronological order to obtain a behavioral chain. A determined list is obtained based on the behavioral chain. Multi-source data of individuals in the determined list is collected, and an anomaly coefficient is calculated based on the multi-source data. If the anomaly coefficient is greater than a threshold, the target individual is determined to be an individual with abnormal behavior, and an anomaly warning is issued based on the behavioral characteristics corresponding to the individual with abnormal behavior. The multi-source data includes: access control data, classroom attendance data, campus consumption data, library borrowing data, and campus network usage data. The target individual is any one of the individuals in the determined list.
[0007] Optionally, performing classification feature extraction on the video stream to obtain a visual tensor, audio feature sequence, and semantic vector includes: The video stream is segmented to obtain fixed-duration segments. The fixed-duration segments are sampled at a preset frame rate to obtain an image sequence. Each image is scaled and combined to obtain a standard image sequence. Each standard image in the standard image sequence is pixel normalized and then arranged in chronological order to obtain a visual tensor. The original audio waveform signal is extracted from the fixed duration segment. The original audio waveform signal is segmented according to the preset frame rate to obtain short-time audio sub-segments. A short-time Fourier transform is performed on each short-time audio sub-segment to obtain multiple spectra. Each spectrum is input into the Mel filter bank to obtain the frequency band energy. The frequency band energy is subjected to discrete cosine transform to obtain the feature vector. The feature vectors are concatenated in order to obtain the audio feature sequence. Each standard image in the standard image sequence is input into the BLIP visual encoder to obtain multiple visual features. The visual features are then input into the BLIP text decoder to obtain natural language description statements. The natural language description statements are concatenated to obtain scene text. The scene text is then input into the BLIP text encoder to obtain a semantic vector.
[0008] Optionally, aligning the visual tensor, the audio feature sequence, and the semantic vector frame-by-frame according to a preset frame rate to obtain behavioral features includes: The visual tensor is input into the visual encoder to obtain depth visual features. The depth visual features are concatenated in chronological order to obtain a temporal visual feature sequence. The temporal visual feature sequence is input into the linear projection layer to obtain a visual encoding sequence. The audio feature sequence is input into an audio encoder to obtain deep audio features, the deep audio features are input into a VGGish structure to obtain normalized audio, and the normalized audio is input into a linear projection layer to obtain an audio encoding sequence. The semantic vector is input into the semantic projection module to obtain a temporal semantic coding sequence. The visual coding sequence, the audio coding sequence, and the temporal semantic coding sequence are concatenated to obtain a fusion sequence. The concatenated fusion sequence is used as a behavioral feature.
[0009] Optionally, concatenating the visual coding sequence, the audio coding sequence, and the temporal semantic coding sequence to obtain a fused sequence includes: The visual coding sequence, the audio coding sequence, and the temporal semantic coding sequence are concatenated to obtain a concatenated feature. The concatenated feature is then input into a fully connected projection layer to obtain a linear transformation feature. Finally, ReLU nonlinear activation is applied to the linear transformation feature to obtain a fusion feature. The feature dimension and sequence length of the fused feature are determined, a position encoding matrix is generated based on the sequence length and the feature dimension, and the fused feature is added element by element to the position encoding matrix to obtain the temporal fused feature; The temporal fusion features are input into a Transformer encoder to obtain multimodal features, and the multimodal features are input into a fully connected layer to obtain a fusion sequence.
[0010] Optionally, issuing anomaly warnings based on the behavioral characteristics of the individuals exhibiting abnormal behavior includes: Obtain a preset feature table, calculate the similarity between the behavioral feature and each standard feature in the preset feature table, determine the standard feature with the highest similarity value as the target standard feature, and issue an anomaly warning based on the target standard feature; the preset feature table contains multiple standard features.
[0011] A second aspect of this invention provides a college student behavior anomaly early warning system based on multi-source data fusion, the system comprising: The feature acquisition module is used to acquire video streams of people within a target area in real time, and to extract visual tensors, audio feature sequences, and semantic vectors from the video streams by performing classification feature extraction; the target area includes high-risk areas and sub-regions; The feature fusion module is used to align the visual tensor, the audio feature sequence, and the semantic vector frame by frame according to a preset frame rate to obtain behavioral features, obtain the events that occur in the behavioral features in each sub-region, calculate the temporal overlap of each event to obtain related events, sort the related events in chronological order to obtain a behavioral chain, and obtain a determined list based on the behavioral chain. The personnel identification module is used to collect multi-source data of personnel in the identified list, calculate an anomaly coefficient based on the multi-source data, and determine the target personnel as abnormal behavior personnel if the anomaly coefficient is greater than a threshold. An anomaly warning is issued based on the behavioral characteristics of the abnormal behavior personnel. The multi-source data includes: access control data, classroom attendance data, campus consumption data, library borrowing data, and campus network usage data. The target personnel is any one of the personnel in the identified list.
[0012] Optionally, the video acquisition module includes: The image preprocessing module is used to segment the video stream to obtain fixed-duration segments, perform frame sampling on the fixed-duration segments according to a preset frame rate to obtain an image sequence, perform size scaling and combination on each image to obtain a standard image sequence, and perform pixel normalization on each standard image in the standard image sequence and arrange them in chronological order to obtain a visual tensor. The audio preprocessing module is used to extract the original audio waveform signal from the fixed-duration segments, divide the original audio waveform signal into segments according to a preset frame rate to obtain short-time audio sub-segments, perform a short-time Fourier transform on each short-time audio sub-segment to obtain multiple spectra, input each spectrum into the Mel filter bank to obtain the frequency band energy, perform a discrete cosine transform on the frequency band energy to obtain the feature vector, and concatenate each feature vector in order to obtain the audio feature sequence. The language preprocessing module is used to input each standard image in the standard image sequence into the BLIP visual encoder to obtain multiple visual features, input the visual features into the BLIP text decoder to obtain natural language description statements, concatenate the natural language description statements to obtain scene text, and input the scene text into the BLIP text encoder to obtain semantic vectors.
[0013] Optionally, the anomaly warning module includes: The visual encoding module is used to input the visual tensor into the visual encoder to obtain depth visual features, concatenate the depth visual features in chronological order to obtain a temporal visual feature sequence, and input the temporal visual feature sequence into the linear projection layer to obtain a visual encoding sequence. An audio encoding module is used to input the audio feature sequence into an audio encoder to obtain deep audio features, input the deep audio features into a VGGish structure to obtain normalized audio, and input the normalized audio into a linear projection layer to obtain an audio encoding sequence. The semantic encoding module is used to input the semantic vector into the semantic projection module to obtain a temporal semantic encoding sequence, concatenate the visual encoding sequence, the audio encoding sequence and the temporal semantic encoding sequence to obtain a fusion sequence, and use the concatenated fusion sequence as a behavioral feature.
[0014] Optionally, the semantic encoding module includes: The feature transformation module is used to concatenate the visual coding sequence, the audio coding sequence, and the temporal semantic coding sequence to obtain concatenated features, input the concatenated features into a fully connected projection layer to obtain linear transformation features, and perform ReLU nonlinear activation on the linear transformation features to obtain fused features; An element addition module is used to determine the feature dimension and sequence length of the fused feature, generate a position encoding matrix based on the sequence length and the feature dimension, and add the fused feature element by element to the position encoding matrix to obtain the temporal fused feature; The feature encoding module is used to input the temporal fusion features into the Transformer encoder to obtain multimodal features, and input the multimodal features into the fully connected layer to obtain the fusion sequence.
[0015] Optionally, the personnel identification module is further configured to obtain a preset feature table, calculate the similarity between the behavioral features and each standard feature in the preset feature table, determine the standard feature with the highest similarity value as the target standard feature, and issue an anomaly warning based on the target standard feature; the preset feature table contains multiple standard features.
[0016] The beneficial effects of this invention are: This invention proposes a method for early warning of abnormal behavior among college students based on multi-source data fusion. By calculating the anomaly coefficient through source data fusion, it achieves preliminary screening of hidden risks, reducing the detection delay and computational redundancy caused by traditional full-process monitoring analysis. Then, for high-risk individuals, it initiates real-time acquisition of video streams and cloud-based behavioral feature extraction, avoiding the limitations of audio noise interference and single visual modality, and improving the robustness of recognition and the real-time performance of early warning in complex monitoring scenarios. Attached Figure Description
[0017] Figure 1 The flowchart illustrates a method for early warning of abnormal behavior among college students based on multi-source data fusion, as provided in an embodiment of the present invention. Detailed Implementation
[0018] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.
[0019] This invention provides a method for early warning of abnormal behavior among college students based on multi-source data fusion. See also... Figure 1 , Figure 1 A flowchart illustrating a method for early warning of abnormal behavior among university students based on multi-source data fusion, provided in an embodiment of the present invention. The method includes the following steps: S101: Real-time acquisition of video streams of people within the target area; classification feature extraction of the video stream to obtain visual tensors, audio feature sequences, and semantic vectors. S102, the visual tensor, audio feature sequence and semantic vector are aligned frame by frame according to the preset frame rate to obtain behavioral features, the events that occur in each sub-region of behavioral features are obtained, the temporal overlap of each event is calculated to obtain related events, the related events are sorted according to the time order to obtain behavioral chains, and a determined list is obtained based on the behavioral chains. S103: Collect multi-source data of the personnel in the confirmed list, calculate the abnormality coefficient based on the multi-source data, and if the abnormality coefficient is greater than the threshold, determine that the target personnel are abnormal behavior personnel and issue an abnormal warning based on the behavioral characteristics of the abnormal behavior personnel. The target area includes high-risk areas and sub-areas; the multi-source data includes: access control data, classroom attendance data, campus consumption data, library borrowing data, and campus network usage data; the target personnel are any one of the identified individuals.
[0020] Based on the method for early warning of abnormal behavior of college students based on multi-source data fusion provided in this invention, the method realizes the preliminary screening of hidden risks by calculating the abnormal coefficient through multi-source data fusion, reducing the detection delay and computational redundancy brought about by traditional full-process monitoring analysis. Then, the method initiates real-time acquisition of video streams and extraction of behavioral features in the cloud for high-risk personnel, avoiding the limitations of audio noise interference and single visual modality, and improving the recognition robustness and real-time warning in complex monitoring scenarios.
[0021] In one implementation, the target area includes a high-risk area and sub-areas. The target area is, for example, a teaching building. The high-risk area is, for example, a dangerous area with few people, such as a pool or a rooftop. The sub-areas are, for example, areas where the identity of people can be verified, such as the access control at the entrance of the teaching building, the library's reading area, and the card reader in the cafeteria.
[0022] In one implementation, when someone appears in a high-risk area, the system collects video streams from surveillance cameras deployed in the high-risk area. Based on the video streams, it determines the person's behavioral characteristics. To identify the person, it collects video streams from sub-areas to obtain corresponding events (the behavioral characteristics of people in the events are highly similar, as they are likely to have been done by the same person). It calculates the overlap of each event, connects them to obtain a behavioral chain, and traces the behavior chain. It identifies the person by means of access control card attendance records, etc., and obtains the person's multi-dimensional data to determine whether the person is a person with abnormal behavior. If the person is a person with abnormal behavior, an alert is issued.
[0023] In one implementation, events involving behavioral characteristics are acquired, the temporal overlap of these events is calculated to obtain related events, and the related events are sorted chronologically to obtain a behavioral chain. A definitive list is then obtained based on the behavioral chain. For example, in a target area (a public area such as a cafeteria, library, or teaching building), video streams of internal personnel are collected to extract behavioral characteristics. Since behavioral characteristics alone are insufficient to determine a person's identity, video streams of the target area prior to the current moment are acquired, events involving persons with similar behavioral characteristics are obtained, the temporal overlap of these events is calculated to obtain related events, and the related events are sorted chronologically to obtain a behavioral chain. A definitive list is then obtained based on the behavioral chain; this list represents the access control data for the person entering the target area, used to determine the person's identity. Formula for calculating time overlap: ; Among them, S t For time overlap, T c t represents the total duration of the time overlap between the two events. s1 The start time of event 1, t e1 The end time of event 1, t s2 The start time of event 2, t e2 The end time of event 2; S t Time overlap, with a value range of [0,1]. The closer to 1, the higher the time overlap between the two events; equal to 0, the two events have no time intersection and do not overlap at all; only events that are highly overlapping in time or sequentially connected have the basic conditions to be included in the same behavior chain; events with a time overlap of 0 and that are completely separate are likely to be unrelated independent events and should not be included in the same behavior chain.
[0024] In one implementation, the data comes from multiple sources: access control data, classroom attendance data, campus consumption data, library borrowing data, and campus network usage data. Access control data: Dormitory entry and exit time dispersion. Dormitory entry and exit time dispersion refers to the degree of deviation and fluctuation of the time when a student enters and leaves the dormitory each time within a continuous period of time compared to their regular daily routine. By statistically analyzing the entry and exit times in the daily access control check-in records, the regularity and stability of students' daily routines can be characterized. The larger the value of this indicator, the more disordered and fluctuating the students' daily return and exit times are, indicating a lack of a fixed daily routine and reflecting a disordered lifestyle and unbalanced behavioral patterns. The smaller the value, the more fixed the students' dormitory entry and exit times are, and the more stable and normal their daily routines are. This can objectively quantify the degree of students' disrupted routines and abnormal behavioral tendencies.
[0025] Class attendance data: Absence rate = Absence hours ÷ Total expected attendance hours; Campus consumption data: Rate of compliance with regular mealtime consumption targets; Library borrowing data: Zero days of entry into the library. Zero days of entry into the library refers to the total number of days within a set statistical period in which students have no record of entering the library. The statistics are based on the campus library access card swipe and entry sign-in data. Campus network usage data: Abnormal internet access time ÷ total daily internet access time; Abnormal internet access time specifically refers to the duration of campus network internet access that deviates from the normal rest time of college students, usually defined as internet access behavior during the late night and early morning rest periods, such as 23:00 to 06:00 the next day as abnormal internet access time.
[0026] One implementation method integrates multi-source behavioral data, including access control, classroom attendance, campus spending, library borrowing, and campus network usage, to construct a preliminary assessment mechanism for students' hidden risks. Based on anomaly coefficient calculation methods (weighted summation) of daily behavioral trajectories, this approach can continuously capture deviations from students' daily routines and social habits without increasing their perceptual burden, enabling early screening for potential psychological or safety hazards. Compared to methods relying on a single data source or post-event investigations, this solution improves the coverage and proactiveness of risk identification.
[0027] In one implementation, after initially identifying individuals with potential hidden risks, the system further triggers targeted real-time video stream acquisition and cloud-based analysis. Through efficient extraction of behavioral characteristics in the cloud, abnormal actions such as prolonged stillness, climbing, running, and falling can be accurately identified, and the resulting spatiotemporal context can be used to determine whether a real threat is posed. This tiered response mechanism avoids continuously performing costly video analysis on all individuals, instead initiating deep visual monitoring only after risk clues are clear. This reduces the system's computational and bandwidth burden while ensuring timely response capabilities in critical scenarios.
[0028] In one implementation, behavioral characteristics include: spending long periods alone, squatting or sitting still for long periods, running quickly, loitering, lingering, walking back and forth frequently, gathering in groups, engaging in physical conflict-like actions, staying in secluded corners for long periods, wandering alone late at night, sitting on a table for long periods, looking down for long periods, and spending long periods alone without socializing.
[0029] In one embodiment, extracting classification features from a video stream to obtain a visual tensor, an audio feature sequence, and a semantic vector includes: The video stream is segmented to obtain fixed-duration segments. The fixed-duration segments are sampled at a preset frame rate to obtain an image sequence. Each image is scaled and combined to obtain a standard image sequence. Each standard image in the standard image sequence is normalized to pixels and then arranged in chronological order to obtain a visual tensor. The original audio waveform signal is extracted from fixed-duration segments. The original audio waveform signal is segmented according to a preset frame rate to obtain short-time audio segments. A short-time Fourier transform is performed on each short-time audio segment to obtain multiple spectra. Each spectrum is input into the Mel filter bank to obtain the frequency band energy. The discrete cosine transform is performed on the frequency band energy to obtain the feature vector. The feature vectors are concatenated in order to obtain the audio feature sequence. Each standard image in the standard image sequence is input into the BLIP visual encoder to obtain multiple visual features. The visual features are then input into the BLIP text decoder to obtain natural language descriptions. The natural language descriptions are then concatenated to obtain scene text. Finally, the scene text is input into the BLIP text encoder to obtain semantic vectors.
[0030] In one implementation, the accuracy and robustness of student abnormal behavior detection are improved through the collaborative processing of multimodal data streams. The visual stream is processed through frame sampling, scaling, and normalization to form a standardized visual tensor; the audio stream undergoes short-time Fourier transform, Mel filtering, and discrete cosine transform to extract feature sequences; and the semantic stream uses a BLIP model to generate scene text from the image sequence and encodes it into a fixed-dimensional semantic vector. The three feature streams are precisely aligned frame-by-frame in time, ensuring the consistency of cross-modal information at every moment and providing high-quality structured input for subsequent fusion and spatiotemporal modeling. BLIP text encoder: The basic skeleton is a BERT-Base pre-trained model; configuration: 12-layer Transformer structure, 768 hidden dimensions, 12-head self-attention, and a bidirectional attention mechanism; The BLIP text decoder is a causal decoder (BertLMHeadModel) based on BERT-Base. It retains the basic BERT network structure and replaces bidirectional self-attention with causal self-attention to adapt to text sequence generation tasks.
[0031] In one implementation, the complementarity of vision and audio can handle complex environments such as lighting changes and occlusion, while the introduction of semantic information further enhances the model's ability to discriminate ambiguous behaviors (such as distinguishing between playful fighting and real violence). On the other hand, the end-to-end preprocessing pipeline design supports real-time or near-real-time inference, and combined with subsequent Transformer temporal coding and attention pooling mechanisms, accurate detection and start-time localization of abnormal behaviors can be achieved under low-latency conditions. Overall, this method is suitable for practical deployment scenarios such as campus surveillance, maintaining high detection accuracy while possessing good environmental adaptability and engineering feasibility.
[0032] In one embodiment, aligning the visual tensor, audio feature sequence, and semantic vector frame-by-frame according to a preset frame rate to obtain behavioral features includes: The visual tensor is input into the visual encoder to obtain depth visual features. The depth visual features are concatenated in time order to obtain a temporal visual feature sequence. The temporal visual feature sequence is input into the linear projection layer to obtain the visual encoding sequence. The audio feature sequence is input into the audio encoder to obtain deep audio features, the deep audio features are input into the VGGish structure to obtain normalized audio, and the normalized audio is input into the linear projection layer to obtain the audio coding sequence. The semantic vector is input into the semantic projection module to obtain the temporal semantic coding sequence. The visual coding sequence, audio coding sequence and temporal semantic coding sequence are concatenated to obtain the fusion sequence. The concatenated fusion sequence is used as the behavioral feature.
[0033] In one implementation, the visual encoder adopts the ViT-Base (VisionTransformer-Base) model; the input image size is fixed at 224×224, divided into 16×16 image blocks, with a total of 196 patches, a hidden dimension of 768, and includes 12 Transformer encoding layers and 12 attention heads.
[0034] In one implementation, the audio encoder employs a combination structure of a VGGish network and 1D convolutional layers. MFCC features, which are Mel-frequency cepstral coefficients, are used as input. Deep temporal features of the audio are extracted based on VGGish pre-trained weights, and local temporal feature modeling is completed by combining one-dimensional convolution.
[0035] In one implementation, spatial features are extracted using ResNet50 for each frame of the image to obtain frame-level depth visual features. The frame-level features are then concatenated in chronological order to form a temporal visual feature sequence. The feature dimensions are then uniformly mapped to a shared dimension d through a linear projection layer to output a visual encoding sequence.
[0036] In one implementation, the VGGish structure consists of 8 convolutional layers, 5 pooling layers, and 3 fully connected layers. The audio feature sequence is extracted using multi-layer one-dimensional convolution to extract audio spectrum and rhythm features. The VGGish structure is used to further generate standardized audio embeddings. The feature dimensions are mapped to the shared dimension d through a linear projection layer, and the final audio encoded sequence is output.
[0037] In one implementation, the semantic vector is input into the semantic projection module to obtain a temporal semantic coding sequence: the semantic vector is linearly transformed to map the dimension to a shared dimension d, and the semantic features are repeated T times in the time dimension to achieve complete temporal alignment with the video frame, and the temporal semantic coding sequence is output.
[0038] In one embodiment, concatenating the visual coding sequence, the audio coding sequence, and the temporal semantic coding sequence to obtain the fused sequence includes: The visual coding sequence, audio coding sequence, and temporal semantic coding sequence are concatenated to obtain concatenated features. The concatenated features are then input into a fully connected projection layer to obtain linear transformation features. ReLU nonlinear activation is applied to the linear transformation features to obtain fused features. Determine the feature dimension and sequence length of the fusion feature, generate a position encoding matrix based on the sequence length and feature dimension, and add the fusion feature element by element to the position encoding matrix to obtain the temporal fusion feature; The temporal fusion features are input into the Transformer encoder to obtain multimodal features, and the multimodal features are input into the fully connected layer to obtain the fusion sequence.
[0039] In one implementation, the visual coding sequence, audio coding sequence, and semantic coding sequence are concatenated at the same time step t to obtain concatenated features. The concatenated features are then input into a biased fully connected projection layer to obtain linear transformation features. The linear transformation features are then subjected to ReLU nonlinear activation to obtain the single-frame fusion features at that time step. All t=1~T are traversed in chronological order to obtain the complete fusion features.
[0040] In one implementation, the temporal fusion features are input into a Transformer encoder to obtain a multimodal feature sequence: the temporal fusion features are input into the Transformer encoder, and in the multi-head self-attention, the Query, Key, and Value projection matrices are calculated for the temporal fusion features respectively, the multi-head attention weights are calculated, the long temporal dependencies between frames are captured, the attention output is subjected to residual connection + layer normalization, and then fed into the feedforward network FFN to extract higher-order temporal modes. The residual connection + layer normalization is performed again to output the temporally enhanced multimodal feature sequence.
[0041] In one implementation, multimodal features are input into a fully connected layer to obtain the final structural features: each frame of multimodal features is input into a learnable attention fully connected layer, and after tanh activation, the raw attention score is obtained. The scores at all time steps are normalized by softmax to obtain the attention weights. The sequence is then weighted and summed using the weights to output the final structural features.
[0042] In one embodiment, abnormal warning based on the behavioral characteristics of individuals exhibiting abnormal behavior includes: Obtain a preset feature table, calculate the similarity between the behavioral features and each standard feature in the preset feature table, determine the standard feature with the highest similarity value as the target standard feature, and issue anomaly warnings based on the target standard feature; the preset feature table contains multiple standard features.
[0043] In one implementation, the preset feature table is a table determined by the experimenters. This table consists of multiple standard vectors, each of which corresponds to a classification of abnormal behavior. The standard vectors are sorted according to the abnormal behavior to obtain the preset feature table. The preset feature table can be divided into three intervals (first interval, second interval, and third interval). Each interval consists of standard vectors. The warning level of each interval is: first interval < second interval < third interval. The alerts include: 1. First zone: normal behavior, no alerts triggered; 2. Second zone: abnormal behavior, slight attention required; 3. Third zone: high-risk behavior, alert information automatically pushed to management personnel.
[0044] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should still fall within the scope of the claims of the present invention.
Claims
1. A university student behavior abnormality early warning method based on multi-source data fusion, characterized in that, The method includes: The system collects video streams of people within a target area in real time, and extracts classification features from the video streams to obtain visual tensors, audio feature sequences, and semantic vectors; the target area includes high-risk areas and sub-regions. The visual tensor, the audio feature sequence, and the semantic vector are aligned frame by frame according to a preset frame rate to obtain behavioral features. Events occurring in each sub-region of the behavioral features are obtained, the temporal overlap of each event is calculated to obtain associated events, and the associated events are sorted in chronological order to obtain a behavioral chain. A determined list is obtained based on the behavioral chain. Multi-source data of individuals in the determined list is collected, and an anomaly coefficient is calculated based on the multi-source data. If the anomaly coefficient is greater than a threshold, the target individual is determined to be an individual with abnormal behavior, and an anomaly warning is issued based on the behavioral characteristics corresponding to the individual with abnormal behavior. The multi-source data includes: access control data, classroom attendance data, campus consumption data, library borrowing data, and campus network usage data. The target individual is any one of the individuals in the determined list. 2.The university student behavior anomaly early warning method based on multi-source data fusion according to claim 1, characterized in that, The classification feature extraction of the video stream yields a visual tensor, an audio feature sequence, and a semantic vector, including: The video stream is segmented to obtain fixed-duration segments. The fixed-duration segments are sampled at a preset frame rate to obtain an image sequence. Each image is scaled and combined to obtain a standard image sequence. Each standard image in the standard image sequence is pixel normalized and then arranged in chronological order to obtain a visual tensor. The original audio waveform signal is extracted from the fixed duration segment. The original audio waveform signal is segmented according to the preset frame rate to obtain short-time audio sub-segments. A short-time Fourier transform is performed on each short-time audio sub-segment to obtain multiple spectra. Each spectrum is input into the Mel filter bank to obtain the frequency band energy. The frequency band energy is subjected to discrete cosine transform to obtain the feature vector. The feature vectors are concatenated in order to obtain the audio feature sequence. Each standard image in the standard image sequence is input into the BLIP visual encoder to obtain multiple visual features. The visual features are then input into the BLIP text decoder to obtain natural language description statements. The natural language description statements are concatenated to obtain scene text. The scene text is then input into the BLIP text encoder to obtain a semantic vector. 3.The university student behavior anomaly early warning method based on multi-source data fusion according to claim 2, characterized in that, The behavioral features are obtained by aligning the visual tensor, the audio feature sequence, and the semantic vector frame by frame according to a preset frame rate, including: The visual tensor is input into the visual encoder to obtain depth visual features. The depth visual features are concatenated in chronological order to obtain a temporal visual feature sequence. The temporal visual feature sequence is input into the linear projection layer to obtain a visual encoding sequence. The audio feature sequence is input into an audio encoder to obtain deep audio features, the deep audio features are input into a VGGish structure to obtain normalized audio, and the normalized audio is input into a linear projection layer to obtain an audio encoding sequence. The semantic vector is input into the semantic projection module to obtain a temporal semantic coding sequence. The visual coding sequence, the audio coding sequence, and the temporal semantic coding sequence are concatenated to obtain a fusion sequence. The concatenated fusion sequence is used as a behavioral feature.
4. The university student behavior anomaly early warning method based on multi-source data fusion according to claim 3, characterized in that, The fusion sequence obtained by concatenating the visual coding sequence, the audio coding sequence, and the temporal semantic coding sequence includes: The visual coding sequence, the audio coding sequence, and the temporal semantic coding sequence are concatenated to obtain a concatenated feature. The concatenated feature is then input into a fully connected projection layer to obtain a linear transformation feature. Finally, ReLU nonlinear activation is applied to the linear transformation feature to obtain a fusion feature. The feature dimension and sequence length of the fused feature are determined, a position encoding matrix is generated based on the sequence length and the feature dimension, and the fused feature is added element by element to the position encoding matrix to obtain the temporal fused feature; The temporal fusion features are input into a Transformer encoder to obtain multimodal features, and the multimodal features are input into a fully connected layer to obtain a fusion sequence.
5. The method for early warning of abnormal behavior of college students based on multi-source data fusion according to claim 1, characterized in that, Anomaly warnings based on the behavioral characteristics of the individuals exhibiting abnormal behavior include: Obtain a preset feature table, calculate the similarity between the behavioral feature and each standard feature in the preset feature table, determine the standard feature with the highest similarity value as the target standard feature, and issue an anomaly warning based on the target standard feature; the preset feature table contains multiple standard features.
6. A college student behavior anomaly early warning system based on multi-source data fusion, characterized in that, The system includes: The feature acquisition module is used to acquire video streams of people within a target area in real time, and to extract visual tensors, audio feature sequences, and semantic vectors from the video streams by performing classification feature extraction; the target area includes high-risk areas and sub-regions; The feature fusion module is used to align the visual tensor, the audio feature sequence, and the semantic vector frame by frame according to a preset frame rate to obtain behavioral features, obtain the events that occur in the behavioral features in each sub-region, calculate the temporal overlap of each event to obtain related events, sort the related events in chronological order to obtain a behavioral chain, and obtain a determined list based on the behavioral chain. The personnel identification module is used to collect multi-source data of personnel in the identified list, calculate an anomaly coefficient based on the multi-source data, and determine the target personnel as abnormal behavior personnel if the anomaly coefficient is greater than a threshold. An anomaly warning is issued based on the behavioral characteristics of the abnormal behavior personnel. The multi-source data includes: access control data, classroom attendance data, campus consumption data, library borrowing data, and campus network usage data. The target personnel is any one of the personnel in the identified list.
7. The college student behavior anomaly early warning system based on multi-source data fusion according to claim 6, characterized in that, The video acquisition module includes: The image preprocessing module is used to segment the video stream to obtain fixed-duration segments, perform frame sampling on the fixed-duration segments according to a preset frame rate to obtain an image sequence, perform size scaling and combination on each image to obtain a standard image sequence, and perform pixel normalization on each standard image in the standard image sequence and arrange them in chronological order to obtain a visual tensor. The audio preprocessing module is used to extract the original audio waveform signal from the fixed-duration segments, divide the original audio waveform signal into segments according to a preset frame rate to obtain short-time audio sub-segments, perform a short-time Fourier transform on each short-time audio sub-segment to obtain multiple spectra, input each spectrum into the Mel filter bank to obtain the frequency band energy, perform a discrete cosine transform on the frequency band energy to obtain the feature vector, and concatenate each feature vector in order to obtain the audio feature sequence. The language preprocessing module is used to input each standard image in the standard image sequence into the BLIP visual encoder to obtain multiple visual features, input the visual features into the BLIP text decoder to obtain natural language description statements, concatenate the natural language description statements to obtain scene text, and input the scene text into the BLIP text encoder to obtain semantic vectors.
8. The college student behavior anomaly early warning system based on multi-source data fusion according to claim 6, characterized in that, The anomaly warning module includes: The visual encoding module is used to input the visual tensor into the visual encoder to obtain depth visual features, concatenate the depth visual features in chronological order to obtain a temporal visual feature sequence, and input the temporal visual feature sequence into the linear projection layer to obtain a visual encoding sequence. An audio encoding module is used to input the audio feature sequence into an audio encoder to obtain deep audio features, input the deep audio features into a VGGish structure to obtain normalized audio, and input the normalized audio into a linear projection layer to obtain an audio encoding sequence. The semantic encoding module is used to input the semantic vector into the semantic projection module to obtain a temporal semantic encoding sequence, concatenate the visual encoding sequence, the audio encoding sequence and the temporal semantic encoding sequence to obtain a fusion sequence, and use the concatenated fusion sequence as a behavioral feature.
9. The college student behavior anomaly early warning system based on multi-source data fusion according to claim 8, characterized in that, The semantic encoding module includes: The feature transformation module is used to concatenate the visual coding sequence, the audio coding sequence, and the temporal semantic coding sequence to obtain concatenated features, input the concatenated features into a fully connected projection layer to obtain linear transformation features, and perform ReLU nonlinear activation on the linear transformation features to obtain fused features; An element addition module is used to determine the feature dimension and sequence length of the fused feature, generate a position encoding matrix based on the sequence length and the feature dimension, and add the fused feature element by element to the position encoding matrix to obtain the temporal fused feature; The feature encoding module is used to input the temporal fusion features into the Transformer encoder to obtain multimodal features, and input the multimodal features into the fully connected layer to obtain the fusion sequence.
10. The college student behavior anomaly early warning system based on multi-source data fusion according to claim 6, characterized in that, The personnel identification module is also used to obtain a preset feature table, calculate the similarity between the behavioral features and each standard feature in the preset feature table, determine the standard feature with the highest similarity value as the target standard feature, and issue an anomaly warning based on the target standard feature; the preset feature table contains multiple standard features.