Device cloud collaboration and AI keyword-based bully identification notification method

Abstract

The invention provides a bullying recognition notification method based on equipment cloud collaboration and AI keywords, and relates to the field of campus security and artificial intelligence, and the method comprises the steps: an equipment end collects environment audio in real time, detects bullying keywords in real time through a local AI model, and triggers data uploading; the cloud receives the audio and associates the audio with the scene video, performs secondary identification by using a multi-modal fusion model, and generates a behavior emergency index UI and a scene risk coefficient SRF; calculating a comprehensive risk value based on the UI and the SRF, and judging a bullying event in combination with a dynamic threshold value; and sending alarm information to a guardian, a teacher or a security platform according to the comprehensive risk value grade. According to the method, the recognition accuracy is remarkably improved through equipment cloud cooperative processing and multi-modal fusion analysis; a dynamically quantized UI / SRF evaluation system is introduced to realize scientific risk grading; an anti-disassembly mechanism, a local cache, breakpoint resume and an edge computing disaster recovery strategy are built in to guarantee the robustness of the system; the problems of inaccurate identification, risk quantification deficiency, response stiffness and easy equipment failure are solved.

Description

Technical Field

[0001] This invention relates to the fields of campus safety and artificial intelligence technology, and in particular to a method for identifying and reporting bullying based on device cloud collaboration and AI keywords. Background Technology

[0002] Current bullying detection technologies rely excessively on a single modality, resulting in insufficient accuracy and robustness. Mainstream school bullying detection systems primarily operate on two paths: one relies on audio keyword triggering mechanisms from wearable devices (such as smart bracelets and badges); the other relies on video behavior analysis from fixed surveillance cameras. The former is limited by the computing power and storage space of the devices, typically deploying only simple keyword matching algorithms. In noisy school environments (such as playgrounds and cafeterias), it is highly susceptible to high false alarms (e.g., misinterpreting playful, heated words as violent threats) or false alarms (e.g., failing to identify disguised insults) due to background noise interference. While the latter can capture visual information, it is sensitive to changes in lighting and obstructions (such as crowds), making it difficult to accurately identify low-intensity but high-frequency verbal bullying or highly concealed social exclusion behaviors. More importantly, neither audio nor video monomodal analysis can effectively integrate contextual information (such as the correlation between voice tone and body language) and scene information (such as the location of the incident and the density of people in the surrounding area), making it difficult to distinguish between normal conflicts and deliberate bullying. The overall recognition accuracy of the system is insufficient to meet the needs of actual intervention, and false alarms and missed alarms are prominent problems.

[0003] There is a lack of comprehensive quantitative assessment and dynamic response mechanisms for bullying incidents. Existing technologies generally lack the ability to conduct in-depth risk assessments of identified "suspected bullying" incidents. Most systems can only provide simple "yes / no" alarm judgments or provide coarse classifications based on a single factor (such as the frequency of keyword occurrences), failing to dynamically quantify the urgency of the violence (such as escalating threats, distress signals) and the risk coefficients inherent in the environment (such as remote corners, the presence of high-risk individuals, or low population density). This directly leads to rigid response strategies: either sending the same level of alarm for all triggering events, causing information overload and desensitizing teachers or security personnel to frequent "crying wolf" alarms; or responding slowly or insufficiently due to the inability to accurately assess the true urgency of the incident, missing the optimal intervention opportunity. Furthermore, the system's judgment thresholds are usually statically set and cannot be dynamically adjusted based on historical data and real-time environmental changes, further reducing the accuracy and timeliness of alarms.

[0004] The physical protection of wearable devices is weak, and the system's robustness is poor, making it difficult to cope with emergencies. As the front-end node directly contacting potential perpetrators, the physical security of wearable devices is crucial. However, existing devices generally lack effective anti-tampering mechanisms. Once intentionally damaged or forcibly removed, the system immediately fails, unable to provide continuous location tracking and environmental information collection when victims need help most (e.g., after the device is stolen). Furthermore, the system is highly dependent on the network environment. When communication between the device and the cloud is interrupted (e.g., in campus network blind spots or signal interference) or when cloud services malfunction, the entire identification and alarm process will come to a standstill. The lack of effective local caching, breakpoint resumption mechanisms, and rapid cloud fault switching solutions (e.g., edge computing) makes it impossible to guarantee the system's availability and service continuity at critical moments. These deficiencies in physical protection and system fault tolerance significantly limit the reliable deployment and application value of the technical solution in the complex real-world campus environment. Summary of the Invention

[0005] To address the technical problems in existing technologies, such as low accuracy of single audio recognition on the device side in complex environments, lack of scene fusion in single-modal analysis leading to false positives and false negatives, lack of a quantitative assessment mechanism for bullying incident risks, rigid response strategies that cannot provide accurate and tiered reporting, weak physical protection of devices that are prone to failure, and poor robustness of the system to network and cloud failures, this invention provides a bullying recognition and reporting method based on device-cloud collaboration and AI keywords.

[0006] The technical solution provided by this invention is as follows:

[0007] This invention provides a bullying identification and reporting method based on device cloud collaboration and AI keywords, comprising:

[0008] S1. The device collects audio data of the user's environment in real time, detects preset bullying keywords through a local AI model, and triggers data upload when a keyword is detected.

[0009] S2. The cloud receives the uploaded audio data, combines it with the scene video data, and performs secondary recognition through a multimodal fusion model to generate the behavioral urgency index (UI) and scene risk coefficient (SRF).

[0010] S3. Calculate the overall risk value based on UI and SRF. If the overall risk value exceeds the dynamic threshold, it is determined to be a bullying event.

[0011] S4. Based on the severity of the bullying incident, send tiered alarm information to preset guardian terminals, teacher terminals, or security platforms.

[0012] Preferably, the data collected by the device in S1 includes audio, motion sensor data, and location information;

[0013] The local AI model uses a lightweight convolutional neural network (CNN) to detect keywords in real time. The keyword database includes violent and threatening words, insulting words, and distress phrases.

[0014] Preferably, the construction of the multimodal fusion model in S2 includes:

[0015] S201. Extract Mel-frequency cepstral coefficients (MFCC) features from the audio data;

[0016] S202. Extract spatiotemporal features from video data and process continuous frame sequences using a 3D convolutional network;

[0017] S203. Input the audio features and video features into the attention mechanism layer to generate a fused feature vector.

[0018] Preferably, the attention mechanism layer of S203 calculates the feature weights using the following formula:

[0019]

[0020] Where, α i Let i be the weight of the i-th feature. v and b are trainable parameters, W is the weight matrix, and h i This is the input feature vector.

[0021] Preferably, the secondary recognition in S2 adopts the support vector machine (SVM) algorithm, which takes the fused feature vector as input and outputs the initial values ​​of UI and SRF; UI reflects the violence level of the behavior and SRF reflects the degree of isolation in the environment.

[0022] Preferably, the behavioral urgency index UI in S3 is dynamically calculated using the following formula:

[0023]

[0024] Where: f k w represents the frequency of occurrence of the k-th keyword within the time window T; k The preset weight for this type of keyword; A v The peak value of the audio signal in decibels; A th ΔT is the environmental noise threshold; ΔT is the time decay factor, which increases with the duration of the event.

[0025] Preferably, the scenario risk factor SRF in S3 is calculated using the following formula:

[0026]

[0027] Where: D p The hazard level of the equipment location is determined based on the hazard zone level marked on the pre-stored campus map; λ is the environmental conditioning coefficient; Phigh The number of high-risk individuals identified in the video; P total C represents the total number of people in the scene; density C represents population density. max This is the preset upper limit threshold for density.

[0028] Preferably, the dynamic threshold adjustment method of S3 specifically includes:

[0029] Based on historical bullying incident data, the threshold is updated using an exponentially weighted moving average method; a high-risk alarm is triggered when UI > 0.7 or SRF > 0.8.

[0030] Preferably, the graded alarm strategy of S4 specifically includes:

[0031] S401. If the comprehensive risk value ∈ [0.4, 0.6), send a low-risk warning to the teacher's terminal;

[0032] S402. If the overall risk value ∈ [0.6, 0.8), notify the teacher and guardian's terminals simultaneously.

[0033] S403. If the overall risk value is ≥0.8, the campus security platform will be linked to initiate real-time location tracking.

[0034] Preferably, the device has a built-in anti-tampering sensor. If unauthorized disassembly is detected, the following operations are immediately triggered:

[0035] S1001, Send a disassembly alarm and final location coordinates to the cloud;

[0036] S1002, Cloud-based backup recording module continuously uploads ambient audio;

[0037] S1003. The alarm level of SRF is automatically upgraded according to the disassembly location.

[0038] The beneficial effects of the technical solution provided by this invention include at least the following:

[0039] (1) In this invention, the accuracy and environmental adaptability of bullying identification are significantly improved through device-cloud collaboration and multimodal fusion analysis. The device uses a lightweight CNN model to perform real-time initial screening of audio keywords, effectively filtering noise interference and triggering the upload of key data; the cloud combines synchronous video streams and uses an attention mechanism model that integrates audio MFCC features and video spatiotemporal features for in-depth analysis. This technique overcomes the limitations of single-modal analysis and can more accurately capture the correlation between speech content and body movements and environmental scenes, thereby significantly reducing the false alarm rate and false negative rate in complex campus environments and solving the problems of low accuracy and susceptibility to interference in single-modal recognition in the background technology.

[0040] (2) This invention innovatively introduces a dynamically quantified behavioral urgency index (UI) and scenario risk coefficient (SRF) to achieve scientific assessment and graded response to bullying risks. Utilizing multi-dimensional data such as preset keyword weights, audio characteristics, time factors, geographical location risk levels, and personnel composition and density, the UI and SRF are calculated in real-time using dynamic formulas, resulting in a comprehensive risk value. This technology constructs a refined risk assessment system that objectively reflects the urgency of the event and the level of environmental risk. The graded alarm strategy implemented based on this (from low-risk early warning to high-risk coordinated security) ensures the accuracy and timeliness of the response, effectively solving the problems of lack of risk quantification and rigid response strategies in the background technology, and avoiding the drawbacks of a "one-size-fits-all" approach or delayed response.

[0041] (3) In this invention, the physical protection of the equipment and the disaster recovery capability of the system are strengthened to ensure continuous and reliable operation under critical situations. The equipment has a built-in strain gauge sensor to detect illegal disassembly, triggering a disassembly alarm, activating the backup bone conduction recording module, and escalating the alarm. At the same time, data is cached locally and supports breakpoint resume when the network is interrupted. When the cloud fails, it automatically switches to the edge computing node and enables the degradation processing strategy. This technical means significantly enhances the equipment's resistance to damage and the overall robustness of the system. Even in extreme cases such as forced removal of the equipment, network instability, or cloud service interruption, the system can still provide location tracking, environmental information collection, and emergency response to the greatest extent, solving the key pain points of easy equipment failure, strong system dependence, and poor robustness in the background technology. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating a bullying identification and notification method based on device cloud collaboration and AI keywords, provided in an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram illustrating the construction process of a multimodal fusion model in a bullying identification and reporting method based on device cloud collaboration and AI keywords, provided in an embodiment of the present invention. Detailed Implementation

[0045] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0046] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0047] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, their intended meanings are consistent.

[0048] In this embodiment of the invention, sometimes a subscript such as W1 may be mistakenly written as a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0049] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0050] Reference manual attached Figure 1 The diagram illustrates a flowchart of a bullying identification and reporting method based on device cloud collaboration and AI keywords provided by an embodiment of the present invention.

[0051] This invention provides a method for identifying and reporting bullying based on device-cloud collaboration and AI keywords. The processing flow may include the following steps:

[0052] S1. The device collects audio data from the user's environment in real time and detects preset bullying keywords through a local AI model. When a keyword is detected, data upload is triggered.

[0053] In one possible implementation, the data collected by the device in S1 includes audio, motion sensor data, and location information. The local AI model uses a lightweight convolutional neural network (CNN) to detect keywords in real time; the keyword database includes violent threats, insults, and distress phrases.

[0054] It should be noted that the device uses a miniature microphone array to collect ambient audio data in real time, with a sampling rate set to 16kHz to cover the human voice frequency band. The local AI model is deployed on an embedded processor (such as the ARM Cortex-M7 architecture) and adopts a lightweight Convolutional Neural Network (CNN) architecture, containing 3 convolutional layers and 1 fully connected layer. The model parameter size is controlled within 50KB to ensure real-time performance. The keyword library is stored in the device's flash memory and is divided into three categories: violent threatening words (such as "I'll kill you"), insulting words (such as "useless"), and SOS phrases (such as "help me"). When a keyword match is detected, the device simultaneously collects triaxial accelerometer data (for detecting violent movement) and GPS positioning information, and triggers the upload of encrypted data packets via Wi-Fi or 4G network.

[0055] Furthermore, lightweight CNN training was performed using the TensorFlow Lite framework. The dataset contained 100,000 speech segments in noisy environments, and robustness was improved through data augmentation (adding -10dB to 5dB white noise). The SVM classifier employed grid search to optimize parameters: penalty coefficient C = 10, kernel function γ = 0.01.

[0056] S2: The cloud receives the uploaded audio data, combines it with the scene video data, and performs secondary recognition through a multimodal fusion model to generate the behavioral urgency index (UI) and the scene risk coefficient (SRF).

[0057] In one possible implementation, such as Figure 2 As shown, the construction of the multimodal fusion model in S2 includes:

[0058] S201. Extract Mel-frequency cepstral coefficients (MFCC) features from the audio data.

[0059] S202. Extract spatiotemporal features from video data and use a 3D convolutional network to process continuous frame sequences.

[0060] S203. Input the audio features and video features into the attention mechanism layer to generate a fused feature vector.

[0061] In one possible implementation, the attention mechanism layer of S203 calculates the feature weights using the following formula:

[0062]

[0063] Where, α i Let i be the weight of the i-th feature. v and b are trainable parameters, W is the weight matrix, and h i This is the input feature vector.

[0064] In one possible implementation, the secondary recognition in S2 employs a Support Vector Machine (SVM) algorithm, taking the fused feature vector as input and outputting initial values ​​for UI and SRF. UI reflects the violence level of the behavior, while SRF reflects the degree of isolation from the environment.

[0065] It should be noted that the end server adopts a distributed architecture to receive uploaded data and automatically associates it with video streams from scene surveillance cameras within the same time period. Audio feature extraction uses the LibROSA library to calculate Mel-Frequency Cepstral Coefficients (MFCC), taking the first 13 dimensions as the feature vector. Video processing uses a pre-trained I3D (Inflated 3D ConvNet) model to extract spatiotemporal features at a rate of 5 frames per second. The core of the multimodal fusion model is the attention mechanism layer, whose trainable parameters are initialized using the Xavier method, and the fusion feature vector dimension is set to 256. The Support Vector Machine (SVM) classifier uses a radial basis function kernel function, and during the training phase, it uses 100,000 labeled data points (including bullying and non-bullying scenarios) for supervised learning. The initial values ​​of the output UI are in the range [0,1], and the initial values ​​of the SRF are in the range [0,1].

[0066] Furthermore, the annotation process implements a three-level quality control system:

[0067] Basic annotations: Independently annotated by 5 certified safety education experts, using double-blind cross-validation.

[0068] Disagreement handling: Develop an automatic arbitration system for disagreement samples. When the difference in annotations among experts is greater than 30%, introduce the Transformer model for semantic depth analysis.

[0069] Dynamic correction: Deploy an online feedback mechanism to automatically add false alarm / missed alarm data of actual alarm events to the training set and update model parameters monthly.

[0070] S3. Calculate the overall risk value based on UI and SRF. If the overall risk value exceeds the dynamic threshold, it is determined to be a bullying event.

[0071] In one possible implementation, the behavioral urgency index UI in S3 is dynamically calculated using the following formula:

[0072]

[0073] Where f k Let w be the frequency of occurrence of the k-th keyword within the time window T. k As a preset weight for this type of keyword, A v A represents the peak value of the audio signal in decibels. thΔT is the environmental noise threshold, and ΔT is the time decay factor, which increases with the duration of the event.

[0074] Furthermore, the peak decibel value A in the UI formula v Dynamic range compression is used to avoid interference from sudden noise.

[0075] In one possible implementation, the scenario risk factor SRF in S3 is calculated using the following formula:

[0076]

[0077] Where D p The hazard level of the equipment location is determined based on the hazard zone level marked on the pre-stored campus map, where λ is the environmental adjustment coefficient and P is the hazard level. high P is the number of high-risk individuals identified in the video. total C represents the total number of people in the scene. density For personnel density, C max This is the preset upper limit threshold for density.

[0078] Furthermore, the input value of the tanh function in the SRF formula is limited to C. density / C max ≤3, to prevent numerical saturation caused by extremely crowded scenarios.

[0079] It should be noted that the comprehensive risk value is calculated using the following formula:

[0080] Overall risk value = 0.6 × UI + 0.4 × SRF

[0081] The weighting coefficients were trained based on 100,000 historical bullying incident data points, and their effectiveness was verified using a logistic regression model (AUC = 0.92). When UI > 0.7 or SRF > 0.8, the overall risk value was forcibly set to above 0.8 to ensure priority response to high-risk events.

[0082] In one possible implementation, the dynamic threshold adjustment method of S3 specifically includes:

[0083] Based on historical bullying incident data, the threshold is updated using an exponentially weighted moving average method. A high-risk alarm is triggered when UI > 0.7 or SRF > 0.8.

[0084] It should be noted that the time window T in UI calculations is 30 seconds by default, and the keyword weight w... k Based on expert evaluation, the weightings were set as follows: violent words 1.0, insulting words 0.7, and distress phrases 0.9. Environmental noise threshold A. thDynamic calibration is performed using the RMS value of the first 10 seconds of audio. The time decay factor ΔT is calculated using the formula ΔT = 0.05 × t / 60 (where t is the duration of the event in seconds), reflecting the cumulative effect of risk over time. The location hazard level D in the SRF calculation is also considered. p Mapped to numerical values: High-risk areas (such as abandoned warehouses) D p =1.0, Medium-risk area (such as a corner of the playground) D p =0.5, safe zone D p =0.1. The environmental adjustment coefficient λ has a default value of 2.0 and can be adjusted according to seasonal factors (e.g., λ increases to 2.5 during school dismissal time in winter). Population density C density The density upper limit C is calculated by video analysis to determine the number of people per unit area. max The threshold is set at 2 people per square meter. The dynamic threshold uses an exponentially weighted moving average method.

[0085] Threshold new =0.9 × Threshold old +0.1×R current

[0086] Where R current This represents the median of the overall risk value for that day.

[0087] Furthermore, the time window T employs a sliding window mechanism, updating every second, with a window length of 30 seconds. Specifically, this is implemented by maintaining a circular buffer to store keyword detection results within the most recent 30 seconds. Each time the UI is calculated, the oldest second's data is removed and the latest second's data is added, ensuring a balance between real-time update frequency and computational efficiency. The buffer data structure is implemented using a circular queue with a capacity of 30 (one record unit per second). Each record unit stores the keyword category and count detected within that second.

[0088] Furthermore, the identification of high-risk individuals is based on two dimensions: First, real-time matching is performed using the facial information of students with bullying records pre-stored in the campus safety database. The FaceNet model is used to extract facial feature vectors from video frames and compare them with the features in the database using cosine similarity (threshold 0.85). Second, individuals with aggressive postures are detected through a video behavior analysis model. This model uses OpenPose skeleton keypoint detection technology. When a punch is detected within 3 consecutive frames (elbow angle <30° and movement speed >1.5m / s) or a pushing action is detected (hands extended forward distance >0.6m and center of gravity shifted forward), the individual is marked as a high-risk individual.

[0089] S4. Based on the severity of the bullying incident, send tiered alarm information to preset guardian terminals, teacher terminals, or security platforms.

[0090] In one possible implementation, the graded alarm strategy of S4 specifically includes:

[0091] S401. If the comprehensive risk value ∈ [0.4, 0.6), send a low-risk warning to the teacher's terminal.

[0092] S402. If the comprehensive risk value is in the range of [0.6, 0.8), notify the teacher and guardian's terminals simultaneously.

[0093] S403. If the overall risk value is ≥0.8, the campus security platform will be linked to initiate real-time location tracking.

[0094] It should be noted that when the tiered alarm strategy is implemented, low-risk warnings (comprehensive risk value 0.4-0.6) only send text notifications to teachers' terminals; medium-risk alarms (0.6-0.8) simultaneously push a message containing a 10-second audio clip to the mobile apps of teachers and guardians; high-risk alarms (≥0.8) trigger a three-way linkage: sending real-time location coordinates (updated at a frequency of 1Hz) to the security platform, activating the campus broadcast system to play an alarm sound, and automatically unlocking the nearest emergency exit access control.

[0095] In one possible implementation, the device has a built-in anti-tamper sensor that, if unauthorized disassembly is detected, immediately triggers the following operation:

[0096] S1001, Send disassembly alarm and final location coordinates to the cloud.

[0097] S1002, The cloud-based backup recording module continuously uploads ambient audio.

[0098] S1003. The alarm level of SRF is automatically upgraded according to the disassembly location.

[0099] Furthermore, the SRF calculation for the dismantling location uses the most recent valid video analysis results (maximum 5 minutes back). If no video data is available, the location hazard level D is used only. p The calculation formula simplifies to:

[0100]

[0101] Where λ takes the default value of 2.0, D p The location is obtained by matching the GPS coordinates last reported by the device with the campus electronic map. At the same time, the emergency positioning mode is triggered, which scans the surrounding iBeacon beacons (deployed at a density of 1 beacon per 50 square meters) through the device's built-in Bluetooth 5.0 module, and uses the received signal strength indicator RSSI value to perform triangulation positioning, improving the positioning accuracy to ±2 meters.

[0102] It should be noted that the anti-disassembly sensor uses strain gauges mounted on the inner wall of the equipment casing. When abnormal deformation (>0.5mm displacement) is detected, it is determined to be unauthorized disassembly. After the backup recording module is activated, it uses a bone conduction microphone to continuously collect vibration signals and uploads compressed audio at a rate of 10Kbps via LoRa wireless technology. The alarm level escalation rule is: if the SRF at the disassembly location is >0.6, the high-risk alarm procedure is directly triggered.

[0103] In one possible implementation, when the network is interrupted, the device locally caches 15 minutes of data and resumes transmission once the connection is restored. When the cloud service fails, processing automatically switches to edge computing nodes, ensuring a response latency of <200ms.

[0104] Furthermore, the edge computing nodes are deployed on a local server cluster on campus, using a distributed architecture:

[0105] Hardware configuration: ARMv8 architecture processor (4 cores @ 2.0GHz) + NPU acceleration unit (computing power 4TOPS).

[0106] Preloaded model: Lightweight multimodal fusion model (parameter count compressed to 30% of the original model).

[0107] Data synchronization mechanism: Model parameters are synchronized with the cloud every 5 seconds via the MQTT protocol, and differential data is automatically incrementally synchronized after the network is restored.

[0108] Circuit breaker strategy: When the response delay is greater than 200ms for 3 consecutive times, automatically switch to pure audio analysis mode (disable video processing).

[0109] In one possible implementation, audio data is encrypted using AES-256 before uploading, and faces in video data are blurred in real time. All stored data is automatically destroyed after 72 hours, and guardians can review event records using an authorized key.

[0110] In one possible implementation, a pre-trained multimodal large model can be integrated into the cloud as a high-level analysis unit. This model employs a dual-stream Transformer architecture, where the text stream processes audio-to-text transcription, and the video stream processes keyframe sequences (1 frame per second). Cross-modal feature interaction is achieved through a co-attention mechanism. When the comprehensive risk value output by the existing multimodal fusion model is in the range of [0.6, 0.8] and the event duration exceeds a threshold T (default 10 seconds), the system automatically inputs the audio-text and video features into the large model to generate a bullying behavior confidence score C. The training data of the large model covers 100,000 labeled samples, including three scenarios: overt physical conflict, covert social exclusion, and normal interaction. A contrastive learning loss function is used to enhance the model's ability to identify subtle bullying behaviors, such as distinguishing between "joking" and "malicious mockery" through semantic context. In actual operation, the confidence score C output by the large model is fused with the UI and SRF values ​​of the existing multimodal model through a dynamic weighted formula:

[0111] The overall risk value is calculated as α × UI + β × SRF + γ × C, where the weighting coefficients α = 0.5, β = 0.3, and γ = 0.2 are dynamically adjusted based on historical alarm accuracy. To balance real-time performance, the large model inference process employs quantization-aware training to compress the number of parameters to 35% of the original model, and achieves 4-card parallel acceleration on a GPU cluster through tensor parallelism, ensuring that the latency of a single inference is controlled within 1.2 seconds. If the large model analysis times out due to network latency (exceeding 2 seconds), the system will automatically revert to the evaluation results of the existing multimodal model, and record the timeout event for subsequent model optimization.

[0112] Furthermore, the large-scale model is updated monthly via an online learning mechanism, with input consisting of false alarm / missed alarm samples manually reviewed from actual alarm events. Elastic Weight Consolidation technology is employed during updates to prevent catastrophic forgetting and ensure the model adapts to the dynamic changes in the campus environment. This large-scale model complements existing multimodal fusion models: the former excels at capturing long-term context and implicit social cues, while the latter ensures low-latency real-time responses, together constructing a comprehensive recognition system covering both overt and covert bullying behaviors.

[0113] It should be noted that the training method for large models achieves efficient learning through multi-stage data engineering and adaptive optimization strategies:

[0114] During the data construction phase, a hierarchical sampling strategy was adopted to ensure that the training set covered all bullying scenarios. The basic dataset consisted of three parts: First, 100,000 labeled samples from historical alarm events that were manually reviewed, including three scenarios: overt physical conflict, covert social exclusion, and normal interaction, with samples of each category divided in a 6:2:2 ratio; Second, synthetic data generated through a semi-automatic annotation process, using a rule engine to simulate high-frequency bullying phrases (such as "If you do that again, I'll..." + threatening suffix) and corresponding physical actions (pushing, blocking), and enhancing data diversity through Generative Adversarial Networks (GANs); Third, continuously collected unlabeled data from real-time campus scenes, with pseudo-labels generated through a teacher-student framework, where the teacher model is a converged full-parameter model and the student model is a lightweight model to be trained. During data cleaning, multimodal consistency verification was used: if threatening keywords were detected in the audio but there were no corresponding physical actions in the video, or if an isolated individual was identified in the video but there was no relevant context in the audio, the sample was transferred to the manual review queue to ensure data quality.

[0115] Furthermore, the training process employs a two-stage joint optimization strategy to balance model accuracy and efficiency. The first stage involves full parameter pre-training, using an improved LoRA (Low-Rank Adaptation) method to freeze the backbone network parameters, training only the common attention layer and classification head with low-rank matrix factorization to reduce memory usage while maintaining model expressive power. The optimization objective is a multi-task loss function, including classification loss (Cross-Entropy Loss) and contrastive learning loss (Triplet Loss). Contrastive learning enhances the model's ability to distinguish subtle bullying behaviors by constructing positive and negative sample pairs: positive sample pairs consist of different modalities of the same event (audio + video), while negative sample pairs consist of data from different events or normal interaction scenarios. The second stage involves scene adaptation fine-tuning. Data subsets are divided according to campus area characteristics (e.g., classrooms, playground, cafeteria). Hyperparameters (learning rate, batch size) are adjusted for each subset using grid search, and gradient accumulation technology is introduced to simulate the effects of large-batch training. During fine-tuning, a dynamic early stopping mechanism is used: when the validation set loss does not decrease for three consecutive epochs, training is automatically terminated and the optimal model is saved.

[0116] Furthermore, to enhance the model's ability to identify long-tail bullying scenarios, curriculum learning and hard example mining strategies are introduced during training. Curriculum learning is trained in stages according to data complexity: initially, only high-confidence samples (such as explicit physical conflicts) are used, gradually increasing the number of low-confidence samples (such as ambiguous social exclusion), so that the model's learning path conforms to human cognitive patterns. Hard example mining is implemented online: after each round of training, the top-20% of difficult samples are selected based on classification loss, and their sampling weights are increased in the next round of training, forcing the model to focus on boundary cases. For example, for semantically ambiguous scenarios such as "joking" and "malicious mockery," the model's ability to capture subtle differences in context and tone is strengthened by increasing the number of iterations for such samples. In addition, a multimodal consistency regularization term is integrated into the training process: if the cosine similarity between audio and video features is lower than a threshold (default 0.6), the model is penalized to prevent single-modal overfitting from causing cross-modal inference failure.

[0117] It should be noted that model compression and acceleration are achieved in the later stages of training through a combination of Quantization-Aware Training (QAT) and structured pruning. Quantization-Aware Training simulates an 8-bit integer (INT8) inference environment, backpropagating quantization errors to the feedforward network during backpropagation, allowing model parameters to adapt to a lower-precision representation. Structured pruning is based on the contribution of each channel to the classification loss: the gradient × weight product of each convolutional channel is calculated, and channels with a contribution below 30% of the global mean are removed. Gradual pruning is used in five rounds to gradually remove channels, preventing a sudden drop in model performance. After pruning, the model recovers accuracy through knowledge distillation: the unpruned model acts as the teacher, the pruned model as the student, and a soft target with a temperature parameter τ = 2 guides the student model to learn the decision boundary of the teacher model. Before actual deployment, the model optimizes the computation graph based on the computing power characteristics of the target edge device (such as NVIDIA Jetson AGX Orin) through hardware-aware training, and reduces memory access overhead through operator fusion to ensure that the inference latency meets real-time requirements.

[0118] It's important to note that the large-scale model's cross-modal feature interaction achieves deep semantic alignment between text and video through a dynamic co-attention mechanism. The text stream uses a pre-trained language model to encode audio transcribed into text, extracting multi-dimensional features containing semantics, sentiment, and contextual relationships. The video stream, on the other hand, captures dynamic changes in actions, postures, and scenes through a spatiotemporal convolutional network. The co-attention layer, through adaptive weight allocation, enables the model to focus on the correlation between implicit threats in the text (such as irony and metaphors) and nonverbal cues in the video (such as group alienation and deliberate avoidance), thereby improving its ability to identify covert bullying behaviors such as passive-aggressive behavior and social exclusion. This mechanism, through multiple rounds of feature interaction iterations, gradually strengthens the consistency constraints between modalities, significantly improving the model's feature fusion performance in complex scenarios.

[0119] It's important to note that, to adapt to the dynamic changes in the campus environment, the large model employs a layered online learning strategy for continuous optimization. The foundational layer periodically absorbs pattern changes from new data (such as seasonal behavioral differences and emerging bullying tactics) through incremental learning, preventing performance degradation due to data distribution shifts. The adaptation layer dynamically adjusts the model's attention weights based on real-time environmental factors (such as time of day and population density in different areas), making the decision-making logic more aligned with the characteristics of the current scenario. For example, during high-activity periods between classes, the model prioritizes the risk of physical conflict; while in low-density areas after school, it focuses on detecting abnormal behavior in isolated individuals. During online learning, the model transfers the decision-making capabilities of the full-parameter model to a lightweight student model through knowledge distillation, ensuring efficient inference even on resource-constrained edge devices.

[0120] It's worth noting that, to meet the deployment requirements of edge computing scenarios, the large model achieves lightweight design and efficient inference through multi-dimensional optimization. In terms of quantization, a hybrid precision quantization strategy is employed to dynamically compress the model weights and activation values, significantly reducing computational overhead while maintaining the expressive power of key features. Regarding sparsity, redundant channels and neurons are removed through structured pruning, making the model structure more compact. The compressed model exhibits significantly reduced inference latency on edge devices, while dynamic batch processing technology adjusts the parallel processing scale based on real-time request volume, further improving throughput. Furthermore, the model supports deploying different layers to different computing nodes using model partitioning techniques (e.g., processing shallow features on the device and deep semantics on the edge server), thereby balancing computational load and communication overhead.

[0121] It's important to note that the security and privacy of the large model are achieved through end-to-end encryption and anonymization technologies. During data acquisition, audio and video are transmitted with end-to-end encryption, and the keys are dynamically generated by the device and rotated periodically. In cloud storage, data is sharded and stored on distributed nodes, with each shard containing only partial feature information to prevent single-point leaks from causing global risks. During model training, differential privacy technology is used to perturb gradients with noise, ensuring that the impact of a single sample on model updates is limited to a controllable range. During inference, secure multi-party computation is used to decouple features, preventing the exposure of raw data on a single node. Furthermore, the system supports collaborative model training across multiple campuses using a federated learning framework, where participants only share model update parameters, not the raw data, thereby improving model generalization capabilities while protecting privacy.

[0122] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following:

[0123] (1) In this invention, the accuracy and environmental adaptability of bullying identification are significantly improved through device-cloud collaboration and multimodal fusion analysis. The device uses a lightweight CNN model to perform real-time initial screening of audio keywords, effectively filtering noise interference and triggering the upload of key data; the cloud combines synchronous video streams and uses an attention mechanism model that integrates audio MFCC features and video spatiotemporal features for in-depth analysis. This technique overcomes the limitations of single-modal analysis and can more accurately capture the correlation between speech content and body movements and environmental scenes, thereby significantly reducing the false alarm rate and false negative rate in complex campus environments and solving the problems of low accuracy and susceptibility to interference in single-modal recognition in the background technology.

[0124] (2) This invention innovatively introduces a dynamically quantified behavioral urgency index (UI) and scenario risk coefficient (SRF) to achieve scientific assessment and graded response to bullying risks. Utilizing multi-dimensional data such as preset keyword weights, audio characteristics, time factors, geographical location risk levels, and personnel composition and density, the UI and SRF are calculated in real-time using dynamic formulas, resulting in a comprehensive risk value. This technology constructs a refined risk assessment system that objectively reflects the urgency of the event and the level of environmental risk. The graded alarm strategy implemented based on this (from low-risk early warning to high-risk coordinated security) ensures the accuracy and timeliness of the response, effectively solving the problems of lack of risk quantification and rigid response strategies in the background technology, and avoiding the drawbacks of a "one-size-fits-all" approach or delayed response.

[0125] (3) In this invention, the physical protection of the equipment and the disaster recovery capability of the system are strengthened to ensure continuous and reliable operation under critical situations. The equipment has a built-in strain gauge sensor to detect illegal disassembly, triggering a disassembly alarm, activating the backup bone conduction recording module, and escalating the alarm. At the same time, data is cached locally and supports breakpoint resume when the network is interrupted. When the cloud fails, it automatically switches to the edge computing node and enables the degradation processing strategy. This technical means significantly enhances the equipment's resistance to damage and the overall robustness of the system. Even in extreme cases such as forced removal of the equipment, network instability, or cloud service interruption, the system can still provide location tracking, environmental information collection, and emergency response to the greatest extent, solving the key pain points of easy equipment failure, strong system dependence, and poor robustness in the background technology.

[0126] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0127] The following points need to be explained:

[0128] (1) The accompanying drawings of the embodiments of the present invention only involve the structures involved in the embodiments of the present invention. Other structures can refer to the general design.

[0129] (2) For clarity, the thickness of layers or regions is enlarged or reduced in the drawings used to describe embodiments of the present invention; that is, these drawings are not drawn to actual scale. It is understood that when an element such as a layer, film, region, or substrate is referred to as being “above” or “below” another element, the element may be “directly” located “above” or “below” the other element, or there may be intermediate elements.

[0130] (3) Where there is no conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other to obtain new embodiments.

[0131] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. The scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for identifying and reporting bullying based on device-cloud collaboration and AI keywords, characterized in that, include: S1. The device collects audio data of the user's environment in real time, detects preset bullying keywords through a local AI model, and triggers data upload when a keyword is detected. S2. The cloud receives the uploaded audio data, combines it with the scene video data, and performs secondary recognition through a multimodal fusion model to generate the behavioral urgency index (UI) and scene risk coefficient (SRF). S3. Calculate the overall risk value based on UI and SRF. If the overall risk value exceeds the dynamic threshold, it is determined to be a bullying event. S4. Based on the severity of the bullying incident, send tiered alarm information to preset guardian terminals, teacher terminals, or security platforms.

2. The bullying identification and reporting method based on device cloud collaboration and AI keywords according to claim 1, characterized in that, include: The data collected by the device in S1 includes audio, motion sensor data, and location information; The local AI model uses a lightweight convolutional neural network (CNN) to detect keywords in real time. The keyword database includes violent and threatening words, insulting words, and distress phrases.

3. The bullying identification and reporting method based on device cloud collaboration and AI keywords according to claim 2, characterized in that, The construction of the multimodal fusion model in S2 includes: S201. Extract Mel-frequency cepstral coefficients (MFCC) features from the audio data; S202. Extract spatiotemporal features from video data and process continuous frame sequences using a 3D convolutional network; S203. Input the audio features and video features into the attention mechanism layer to generate a fused feature vector.

4. The bullying identification and reporting method based on device cloud collaboration and AI keywords according to claim 3, characterized in that, include: The attention mechanism layer of S203 calculates feature weights using the following formula: Where, α i Let i be the weight of the i-th feature. v and b are trainable parameters, W is the weight matrix, and h i This is the input feature vector.

5. The bullying identification and reporting method based on device cloud collaboration and AI keywords according to claim 1, characterized in that, include: The secondary recognition in S2 uses the Support Vector Machine (SVM) algorithm, which takes the fused feature vector as input and outputs the initial values ​​of UI and SRF. UI reflects the level of violence in a behavior, while SRF reflects the degree of isolation in the environment.

6. The bullying identification and reporting method based on device cloud collaboration and AI keywords according to claim 1, characterized in that, include: The behavioral urgency index UI in S3 is dynamically calculated using the following formula: Where: f k w represents the frequency of occurrence of the k-th keyword within the time window T; k The preset weight for this type of keyword; A v The peak value of the audio signal in decibels; A th ΔT is the environmental noise threshold; ΔT is the time decay factor, which increases with the duration of the event.

7. The bullying identification and reporting method based on device cloud collaboration and AI keywords according to claim 1, characterized in that, include: The scenario risk factor SRF in S3 is calculated using the following formula: Where: D p The hazard level of the equipment location is determined based on the hazard zone level marked on the pre-stored campus map; λ is the environmental conditioning coefficient; P high The number of high-risk individuals identified in the video; P total C represents the total number of people in the scene; density C represents population density. max This is the preset upper limit threshold for density.

8. The bullying identification and reporting method based on device cloud collaboration and AI keywords according to claim 1, characterized in that, The dynamic threshold adjustment method of S3 specifically includes: Based on historical bullying incident data, the threshold is updated using an exponentially weighted moving average method; a high-risk alarm is triggered when UI > 0.7 or SRF > 0.

8.

9. The bullying identification and reporting method based on device cloud collaboration and AI keywords according to claim 1, characterized in that, The graded alarm strategy of S4 specifically includes: S401. If the comprehensive risk value ∈ [0.4, 0.6), send a low-risk warning to the teacher's terminal; S402. If the overall risk value ∈ [0.6, 0.8), notify the teacher and guardian's terminals simultaneously. S403. If the overall risk value is ≥0.8, the campus security platform will be linked to initiate real-time location tracking.

10. The bullying identification and reporting method based on device cloud collaboration and AI keywords according to claim 1, characterized in that, include: The device has a built-in anti-tamper sensor. If unauthorized disassembly is detected, the following actions will be triggered immediately: S1001, Send a disassembly alarm and final location coordinates to the cloud; S1002, Cloud-based backup recording module continuously uploads ambient audio; S1003. The alarm level of SRF is automatically upgraded according to the disassembly location.

Images