Campus bullying intelligent identification system and method based on multi-modal data fusion and real-time early warning

Abstract

The invention discloses a campus bullying intelligent identification system and method based on multi-modal data fusion and real-time early warning, and relates to the technical field of campus safety monitoring. According to the system, a low-power-consumption edge AI box is additionally arranged on the basis of existing monitoring, 4K videos, 48kHz audios and campus social media texts are synchronously collected and sent to a double-layer model after passing through a unified Kafka message queue, a visual layer is used for improving YOLOv8 to detect limb conflicts, and a semantic layer is used for finely adjusting BERT to recognize abuse, threat and other speech bulles; and the comprehensive decision engine performs weighted fusion on the visual confidence coefficient and the semantic confidence coefficient, triggers short message and telephone dual-channel early warning which is less than or equal to 1.2 s when the comprehensive probability is greater than or equal to 0.75, and synchronously pushes structured event information to the counselor, the security office and the student management office. According to the method, the hidden bullying event can be found and intervened in real time with low cost and high precision, and the campus safety management capability is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of campus safety monitoring, in particular to a campus bullying intelligent identification system and method based on multi-modal data fusion and real-time early warning. BACKGROUND

[0002] Traditional campus bullying prevention measures mainly rely on post-reporting and manual intervention, which has obvious hysteresis and is difficult to prevent bullying behavior in a timely manner. Campus bullying incidents are sudden, hidden and highly dangerous. The existing technology mainly has the following defects:

[0003] 1. Single data mode: only relying on surveillance cameras or manual reporting, it is difficult to find "no picture" or "no sound" bullying;

[0004] 2. Recognition lag: manual review of video after the event, unable to stop the harm in the first time;

[0005] 3. High false alarm rate: traditional YOLO target detection model only focuses on physical conflict, ignoring verbal bullying;

[0006] 4. Lack of linkage: early warning information cannot be pushed to teachers, parents and security departments simultaneously.

[0007] Therefore, there is an urgent need for a campus bullying intelligent identification system and method that can real-time fuse video, audio, and text multi-modal data, and use artificial intelligence models to achieve high accuracy, low latency, and multi-channel early warning. SUMMARY

[0008] The present application aims to overcome the above-mentioned defects and provide an end-to-end campus bullying intelligent identification system and method. The core technical concept includes:

[0009] 1. Multi-modal data pipeline: video, audio, and text (including social media text) are uniformly connected to the Kafka message queue after preprocessing;

[0010] 2. Double-layer model fusion:

[0011] a) Visual layer: improved YOLOv8 is used to detect abnormal behaviors such as physical conflict, pushing, and blocking;

[0012] b) Semantic layer: NLP model based on bullying-related word similarity calculation is used to identify verbal bullying such as insults, threats, and exclusion;

[0013] 3. Comprehensive decision engine: the visual confidence and semantic confidence are weighted and fused to output a Boolean result of whether bullying has occurred;

[0014] 4. Real-time early warning closed loop: if bullying is determined, structured data is sent to the campus bullying information platform immediately, and SMS and phone dual-channel early warning are triggered;

[0015] The present application realizes the following beneficial effects through the above technical means:

[0016] Improved recognition accuracy: the comprehensive F1 value is greatly improved on the self-built campus scene dataset;

[0017] Reduced response time: the end-to-end average time delay is less than or equal to 1.2 seconds, meeting the real-time intervention requirement;

[0018] Low deployment cost: reuse existing monitoring and broadcasting hardware, only need to add edge AI box;

[0019] Multi-role linkage: counselors, security departments and student management departments receive early warning synchronously. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 : System overall flowchart.

[0021] Figure 2 : System detailed flowchart.

[0022] Figure 3 : Kafka topic partition and data flow topology diagram. DETAILED DESCRIPTION

[0023] Embodiment 1 System architecture

[0024] 1. Data acquisition layer

[0025] Video: 4K network camera, RTSP streaming;

[0026] Audio: 48kHz pickup, AAC encoding;

[0027] Text: authorized API pull from campus forums, WeChat groups, etc.

[0028] 2. Preprocessing module

[0029] Video: frame extraction, face blurring, resolution uniformity 640x640;

[0030] Audio: after noise reduction, convert to text, then word segmentation, stop word removal, pinyin error correction;

[0031] Text: word segmentation, stop word removal, pinyin error correction.

[0032] 3. Multi-modal fusion inference

[0033] Vision reasoning: YOLOv8 + Bullying behavior detection head (3 new anchors, Focus layer replaced by SPD-Conv);

[0034] Semantic reasoning: BERT-Base-Chinese fine-tuning, using contrastive learning loss, corpus contains 12,000 bullying-related texts. See Table 1 for semantic similarity model training corpus examples. Label "1" indicates semantic equivalence or high similarity, "0" indicates non-equivalence or low similarity.

[0035] Table 1 Semantic Similarity Model Training Corpus

[0036] Serial number Sentence 1 Sentence 2 Label 1 Don't walk away after school, let's meet in the grove After class, go to the back hill, I'm waiting for you 1 2 They poured glue on my hair Someone glued super glue into my hair 1 3 Don't tell the teacher, or you'll be sorry Dare to report, you will not get off scot-free 1 4 I just accidentally knocked his book off I accidentally knocked his book off the table 0 5 Play ball after school? Don't walk away after school, go to the grove to get beaten up 0

[0037] Decision fusion: Weighted formula P = α·Pvision + β·Ptext, α = 0.65, β = 0.35, threshold θ = 0.75.

[0038] 4. Early warning and storage

[0039] Early warning: HTTP / 2 interface → Campus bullying information platform → Tencent cloud SMS + voice phone API;

[0040] Storage: Event ID, timestamp, location, de-identified feature vector, model confidence written to PostgreSQL; original file AES-256 encrypted local disk, retained for 30 days automatically rotated.

[0041] Example 2 Edge AI Box Hardware

[0042] SoC: 8GB;

[0043] Interface: HDMI x 1, USB3.0 x 4, RJ45 x 2, 4G / 5G module;

[0044] Power consumption: typical 15W, supports PoE power supply;

[0045] Operating system: Ubuntu 22.04, containerized deployment (Docker + k3s).

[0046] Example 3 Model Training and Updating

[0047] Training data: Cooperate with schools to collect 10,000 hours of de-identified data;

[0048] Incremental learning: Automatically pull new event data every month, use replay buffer + distillation strategy to avoid catastrophic forgetting.

Claims

1. A smart identification system for school bullying based on multimodal data fusion and real-time early warning, characterized in that, include: The data acquisition module uses a 4K network camera to collect video data, a 48 kHz microphone to collect audio data, and authorized APIs to retrieve text data from campus forums, WeChat groups, etc. The preprocessing module, connected to the data acquisition module, performs unified formatting and de-identification processing on the multimodal data, and then pushes the processed data to a Kafka message queue. This module converts audio data into text data; The visual reasoning module uses an improved YOLOv8 model to detect abnormal physical conflict behaviors in videos and outputs the first confidence score. The semantic reasoning module uses an NLP model based on the similarity of bullying-related words to detect verbal bullying in audio-transcribed text and standalone text and outputs a second confidence score. The comprehensive decision-making module is used to weight and fuse the first confidence level and the second confidence level to obtain the final determination result of whether a bullying incident has occurred; The early warning push module is used to send structured data to the campus bullying information platform and simultaneously trigger SMS and telephone warnings when the final judgment result is "yes". The storage module is used to store the structured data into a database and to encrypt and store the original multimodal data locally.

2. The system according to claim 1, wherein the improved YOLOv8 model includes a newly added bullying behavior detection head, the detection head containing 3 anchors, and the focus layer is replaced with SPD-Conv to improve the detection accuracy of small targets.

3. The system according to claim 1 or 2, wherein the NLP model is based on BERT-Base-Chinese fine-tuning and employs a contrastive learning loss function, and the training corpus includes at least 12,000 bullying-related texts.

4. The system according to any one of claims 1-3, wherein the weighted fusion formula is P = α·Pvision + β·Ptext, where α = 0.65, β = 0.35, and an early warning is triggered when P ≥ 0.

75.

5. The system according to any one of claims 1-4, wherein the early warning push module communicates with the campus bullying information platform via the HTTP / 2 protocol and calls the Tencent Cloud SMS and voice call API to achieve dual-channel early warning.

6. The system according to any one of claims 1-5, wherein the storage module uses the AES-256 encryption algorithm to locally encrypt the original multimodal data and sets a 30-day automatic rotation strategy.

7. A method for intelligent identification of school bullying based on the system described in any one of claims 1-6, characterized in that, Includes the following steps: S1: After preprocessing video, audio, and text data, push the data to Kafka in real time; S2: Preprocessing operations on the data, including face blurring, noise reduction, audio data to text conversion, word segmentation, and stop word removal; S3: Call the visual reasoning module and the semantic reasoning module in parallel, and output the first confidence score and the second confidence score respectively; S4: The first confidence level and the second confidence level are weighted and fused to generate the final judgment result; S5: If the determination result is bullying, immediately send structured data to the campus bullying information platform and trigger SMS and telephone warnings; S6: Store structured data in a database and encrypt the raw data for local storage.

8. The method of claim 7, wherein the visual inference in step S3 is performed on an edge device with an inference latency of ≤300 ms.

9. The method according to claim 7 or 8, wherein step S5 further comprises: The event ID, timestamp, model version, and dataset hash value of early warning events are recorded by blockchain to enable compliance auditing.

10. The method according to any one of claims 7-9, wherein the visual reasoning module and the semantic reasoning module are updated monthly using an incremental learning strategy, and the update data is derived from newly added campus bullying incident data in the past 30 days.

Images