A method and device for detecting and judging bullying behavior based on audio and behavioral feature recognition

Abstract

This invention provides a method and apparatus for detecting and judging bullying behavior based on audio and behavioral feature recognition. The method includes: using an audio recognition module to extract and analyze emotion, keywords, and abnormal sound features to determine in real time whether bullying-related language or audio behavior exists; using a behavioral recognition module to process behavioral data and extract temporal features, and combining this with a GRU network for behavior classification to complete the detection and recognition of bullying-related behaviors; combining the recognition results from the audio and behavioral recognition modules to determine whether bullying behavior has occurred in the current scene; if the determination result is yes, a warning is issued and relevant personnel are notified to go to the current scene; otherwise, the recognition result is recorded in the cloud. The beneficial effects of this invention are: through multi-dimensional technical means and process design, it can efficiently, accurately, and sustainably monitor, warn, and record bullying behavior, providing strong technical support for creating a safe and friendly environment.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of behavior detection, and particularly relates to a bullying behavior detection and judgment method and device based on audio and behavior feature recognition. BACKGROUND

[0002] In some schools, training institutions or care centers, students or more vulnerable groups may occasionally experience some verbal, physical or emotional distress. Such situations can take many forms, including unkind words, occasional threats, accidental pushes, or in some cases, being ignored or isolated.

[0003] Behavior exerts often choose to perform these behaviors in blind areas of monitoring or relatively hidden places, and behavior bearers may feel difficult to seek help actively due to various reasons, which makes these situations often not be discovered and intervened in time.

[0004] Traditional video monitoring has limited recognition range. Video can only perceive the visible range and cannot perceive sound content and emotional fluctuations. Behavior recognition is also easily affected by shielding and complex changes in posture, and the recognition accuracy is not high. In addition, places such as toilets cannot be installed with video monitoring due to high privacy, which may result in monitoring dead angles. SUMMARY

[0005] Therefore, the present application aims to provide a bullying behavior detection and judgment method and device based on audio and behavior feature recognition, so as to solve at least one of the above technical problems.

[0006] To achieve the above-mentioned purpose, the technical scheme of the present application is as follows:

[0007] The present application provides a bullying behavior detection and judgment method based on audio and behavior feature recognition, comprising:

[0008] using an audio recognition module to extract and analyze emotional, keyword and abnormal sound features, and judging in real time whether there is bullying-related language or audio behavior;

[0009] using a behavior recognition module to perform behavior data processing and time sequence feature extraction, combining a GRU network to perform behavior classification, and completing detection and recognition of bullying-related behavior;

[0010] combining the recognition results of the audio recognition module and the behavior recognition module to judge whether bullying behavior occurs in the current scene, if the judgment result is yes, issuing a warning and notifying the corresponding personnel to go to the current scene, otherwise, recording the recognition result to the cloud.

[0011] Further, the working process of the audio recognition module comprises:

[0012] Real-time collection of environmental sound through microphones, and denoising and silence detection;

[0013] Extracting frequency domain and time-frequency features, feeding into emotion recognition, keyword recognition, and audio event detection models to identify abnormal emotions, abusive keywords, and abnormal sound events, respectively;

[0014] Comprehensive analysis of multiple recognition results to determine whether there is potential bullying behavior within a continuous time window, real-time recording, and triggering an alarm based on the judgment result.

[0015] Further, the process of extracting frequency domain and time-frequency features includes:

[0016] Denoising, silence detection, and normalization preprocessing of the original audio;

[0017] Calculating the zero-crossing rate and short-time energy in the time domain to assess sound activity and intensity, and extracting MFCC features in the frequency domain to convert frequency distribution into numerical representation;

[0018] Building an audio image through time-frequency analysis to capture spatial and temporal features, and calling the YAMNet pre-trained model to extract high-dimensional embeddings.

[0019] Further, the emotion recognition model extracts audio features of speech and analyzes their temporal changes to determine whether the speaker is in a high-risk emotional state;

[0020] The keyword recognition model detects whether sensitive words are included in the audio in real time to identify potential language violence;

[0021] The audio event detection model analyzes non-verbal sound features in the audio to identify aggressive behavior.

[0022] Further, the working process of the behavior recognition module includes:

[0023] Collecting human motion echo signals through millimeter wave radar, performing FFT processing on radar echo data in distance and velocity dimensions, and using the CFAR algorithm to detect potential target points;

[0024] Continuously tracking potential target points through Kalman filtering combined with the Hungarian algorithm to obtain time sequence trajectory data for each target;

[0025] For each target, extract statistical features and time sequence features in its motion process in a sliding window manner to form a feature sequence;

[0026] The sequence is fed into the GRU network for time sequence modeling and behavior classification to identify whether there is bullying-related behavior for each target.

[0027] Further, the process of performing FFT processing on the radar echo data in the distance and velocity dimensions and detecting potential target points using a CFAR algorithm comprises:

[0028] Performing distance dimension processing on the received time domain signal to extract distance information of the target;

[0029] Based on the results of the distance dimension processing, performing velocity dimension processing on the fixed distance unit to generate a distance-velocity spectrum diagram;

[0030] Using a CFAR algorithm to perform threshold decision on each cell in the distance-velocity spectrum diagram, dynamically extracting target points, and mapping the target points to physical space coordinates according to angle information to generate a target measurement vector containing position and velocity.

[0031] Further, the process of continuously tracking the potential target points by Kalman filtering combined with the Hungarian algorithm to obtain time sequence trajectory data of each target comprises:

[0032] Predicting the position of each target in the current frame by Kalman filtering, and calculating the matching cost combined with the detected target points;

[0033] Using the Hungarian algorithm to complete the optimal matching of the target and the detection point, thereby realizing consistent tracking of the target identity and obtaining a human motion trajectory with a unique ID;

[0034] Wherein, the matched trajectory is updated by Kalman filtering using the detection point to improve the positioning accuracy; the unmatched trajectory enters the loss management and is deleted if it exceeds the threshold; the newly appeared detection point creates a new trajectory after meeting the conditions.

[0035] The second aspect of the application provides an electronic device, comprising a processor and a memory connected with the processor and used for storing executable instructions of the processor, and the processor is used for executing the method of the first aspect.

[0036] The third aspect of the application provides a server, comprising at least one processor and a memory connected with the processor, the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to make the at least one processor execute the method of the first aspect.

[0037] The fourth aspect of the application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the method of the first aspect.

[0038] Compared with the prior art, the bullying behavior detection and judgment method and device based on audio and behavior feature recognition have the following beneficial effects:

[0039] Through real-time analysis of the audio and behavior dual modules, potential bullying speech or actions can be captured in the first time. Once an anomaly is identified, a warning is issued and relevant personnel are notified, minimizing escalation and psychological or physical harm to the victim.

[0040] By combining sound emotion, keywords, and abnormal acoustic features with behavior timing features, the GDPU (GRU) network can consider multiple signals and reduce the probability of misjudgment by a single mode, improving the robustness and accuracy of overall detection.

[0041] Fully automated emotion analysis, keyword extraction, and behavior recognition replace a large amount of manual monitoring and review work, saving manpower, reducing management burden, and allowing security personnel to focus more on disposal and intervention.

[0042] The system not only gives a judgment of "whether bullying occurs", but also provides high-risk keywords, abnormal voiceprint segments, and behavior classification results, which can be used as evidence for subsequent investigation and preservation.

[0043] When bullying is not detected, the system automatically records all audio and behavior features as normal monitoring data, which can be used for future review and help analyze the overall safety trend of the environment, achieving a "prevention-management-control" closed loop. BRIEF DESCRIPTION OF DRAWINGS

[0044] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of the present application illustrated in the drawings and their descriptions are used to explain the present application and are not intended to limit the present application. In the drawings:

[0045] Figure 1 The bullying behavior detection and judgment method flowchart described in the embodiments of the present application. DETAILED DESCRIPTION

[0046] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0047] In the description of the present application, it needs to be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like is based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second" and the like are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined with "first", "second" and the like can be explicitly or implicitly included one or more. In the description of the present application, unless otherwise stated, the meaning of "a plurality of" is two or more.

[0048] In the description of the present application, it needs to be understood that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connection" should be understood in a broad sense, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood through specific circumstances.

[0049] The present application will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0050] The bullying behavior detection and judgment method based on audio and behavior feature recognition includes:

[0051] S1, using an audio recognition module to extract and analyze emotion, keyword and abnormal sound features, and judging whether there is bullying related language or audio behavior in real time;

[0052] S2, using a behavior recognition module to process behavior data and extract time sequence features, combining a GRU network to classify behaviors, and completing detection and recognition of bullying related behaviors;

[0053] S3, combining the recognition results of the audio recognition module and the behavior recognition module, judging whether bullying behavior occurs in the current scene;

[0054] If the judgment result is yes, a warning is issued and the corresponding personnel is notified to go to the current scene; otherwise, the recognition result is recorded to the cloud.

[0055] The working process of the audio recognition module includes:

[0056] Real-time collection of environmental sound through a microphone, and noise removal and silence detection;

[0057] Extracting frequency domain and time-frequency features, sending into emotion recognition, keyword recognition, audio event detection model, respectively identifying abnormal emotions, abusive keywords, and abnormal sound events;

[0058] Comprehensive analysis of multiple recognition results, judge whether there is potential bullying behavior in the continuous time window, record in real time and trigger alarm according to the judgment result.

[0059] The process of extracting frequency domain and time-frequency features includes:

[0060] Denoising, silence detection and normalization preprocessing are performed on the original audio;

[0061] The zero-crossing rate and short-time energy are calculated in the time domain to evaluate the sound activity and intensity, and the MFCC features are extracted in the frequency domain to convert the frequency distribution into numerical representation;

[0062] Audio images are constructed by time-frequency analysis method to capture spatial and temporal features, and YAMNet pre-trained model is called to extract high-dimensional embedding.

[0063] In some embodiments, the working process of the above-mentioned audio recognition module is specifically:

[0064] Audio signal is a continuous time series, which needs to be deformed into machine usable and informative numerical features.

[0065] I. Extracting time domain features from the time angle:

[0066] 1. The number of times the signal crosses zero in a unit of time (from positive to negative or vice versa), which measures the "frequency activity" of the sound. The higher the ZCR, the higher the frequency of sharp sound or crying:

[0067] ;

[0068] Where ZCR represents the number of zero crossings of the signal in a unit of time; N is the total number of signal sampling points, representing the length of the signal; is the absolute value of the difference between the signs of adjacent sampling points. If the signs of two adjacent points are different (i.e. the signal changes from positive to negative or vice versa), the item is 1; if the signs are the same (i.e. the change of the signal is not from positive to negative), the item is 0.

[0069] 2. The sum of the energy in the audio frame, which reflects the sound intensity and is used to identify high-energy events such as screaming, fighting and slapping:

[0070] ;

[0071] Wherein, STE represents the total energy of the signal in a frame, which reflects the strength of the signal; N is the number of sampling points in the audio frame, indicating the total number of samples in the frame; x[n] is the value of the signal at the nth time, indicating the sampling value of the audio signal.

[0072] II. Extracting frequency domain features from the frequency angle:

[0073] 1. Apply a high-pass filter to the audio signal:

[0074] , wherein a∈[0.95, 0.97]; is the sampling value of the input signal at the nth time, and is the original data of the audio signal; is the sampling value of the output signal at the nth time, and is the signal after high-pass filtering; is the coefficient of the filter, which determines the behavior of the filter; is the sampling value of the signal at the nth time, indicating the value of the input signal at the previous time.

[0075] 2. Cut the continuous speech signal into short time periods, considering each frame as a stationary signal in a short time, with a frame length of 25ms (≈400 samples @ 16kHz) and a frame shift of 10ms (≈160 samples).

[0076] 3. Use a window function to reduce spectral leakage caused by frame boundary discontinuity:

[0077] ;

[0078] , wherein, w[n] is the value of the window function, indicating the weighting factor of each sampling point; is the length of the window, which is also usually the length of the signal frame; is the index in the window function, indicating each sampling point in the window; is a cosine function part, which defines the shape of the window function. Through the change of the cosine function, the value of the window function is maximum at the center of the window and gradually decreases to the two ends of the window. The cosine function helps to produce a smooth transition of the window.

[0079] 4. Convert each frame of signal from time domain to frequency domain, output the corresponding complex spectrum of each frame and take the modulus square to get the power spectrum:

[0080] ;

[0081] X[k] : is the complex spectrum of the signal in the frequency domain, indicating the complex value of the signal at the nth frequency point; Re (X[k]) ​​ is the real part of the spectrum X[k] at this frequency point, representing the real part of the signal at this frequency point; Im (X[k]) is the imaginary part of the spectrum X[k] at this frequency point, representing the imaginary part of the signal at this frequency point.

[0082] 5. Map the spectrum to the Mel frequency scale, consistent with human ear auditory perception, i.e., perform Mel-scale computation (from Hz to Mel):

[0083] ; where f is the input physical frequency and m is the corresponding Mel frequency.

[0084] Construct a set of triangular filters (usually 20-40) distributed on the Mel frequency; each filter performs weighted summation on the FFT output to obtain the Mel filter energy.

[0085] 6. Take the logarithm of the Mel filter energy (to prevent the energy range from spanning too large), to simulate the logarithmic response of the human ear to sound intensity: .

[0086] 7. Map the Mel energy to the cepstrum domain to achieve decorrelation and dimension compression:

[0087] ;

[0088] is the calculated cepstral coefficient; is the energy value of the th Mel filter; is the discrete cosine transform of the Mel energy value; is the index of the Mel filter, representing the th energy value of the filter; is the index of the cepstral coefficient; is the number of Mel filter sets, representing the dimension of the energy vector.

[0089] Output the first C DCT coefficients, called MFCC cepstral coefficients (usually take the first 12-13).

[0090] III. Construct time-frequency features jointly:

[0091] Perform short-time Fourier transform (STFT) on each windowed frame to obtain the amplitude spectrum .

[0092] Map the linear frequency axis to the Mel frequency axis, usually take M=128 Mel filters.

[0093] Compute the Mel band energies and take the log:

[0094] where, denotes the energy of the th Mel band at time t; is the Fourier transform coefficient of the corresponding signal at the th frequency, which is usually the spectral representation of the audio signal at a certain time. It is the representation of the signal in the frequency domain; is the response of the th filter of the Mel filter bank, the value at frequency . The Mel filter bank is constructed according to the Mel scale, which is used to map the spectrum onto the Mel frequency scale.

[0095] The result is a MxT matrix, T is the number of frames.

[0096] For each frame t, the three parts of the feature are spliced by column to get the feature vector :

[0097] ; where, is the energy of the th Mel band at time t; is the th Mel frequency cepstral coefficient at time t.

[0098] For each dimension of the feature, do zero mean unit variance normalization to [0, 1] on the whole corpus or batch;

[0099] The resulting feature matrix Tx(M+2+C); where, is the number of frames, is the number of Mel bands, is the number of MFCC coefficients selected.

[0100] This matrix not only retains rich time-frequency spectrum information, but also combines the refined expression of time energy / zero-crossing and frequency cepstrum.

[0101] The emotion recognition model judges whether the speaker is in a high-risk emotional state by extracting the audio features of the voice and analyzing the time sequence changes;

[0102] The keyword recognition model detects whether sensitive words are contained in the audio in real time to identify potential language violence;

[0103] ​The audio event detection model analyzes non-verbal sound features in the audio to identify aggressive behavior.

[0104] In some embodiments, the working processes of the above three models are as follows:

[0105] The three modules all receive a feature matrix X e R T×D , D = M + 2 + C, and perform the following operations respectively:

[0106] I. Emotion recognition model:

[0107] 1. Input the entire audio as a frame sequence into a bidirectional LSTM network:

[0108] where, represents the input feature at time ; BiLSTM is a bidirectional long short-term memory network; represents the LSTM hidden state vector at time ; : represents the hidden state vector at each time ; The dimension of is , that is, it is a -dimensional real number vector.

[0109] The output is the hidden state sequence at all time steps:

[0110] where, represents the total number of time steps, that is, the number of frames of the audio signal (i.e., the number of processed audio frames); represents the dimension of the hidden state vector, which is the dimension of the output vector generated by the LSTM network at each time step; is the final obtained matrix is a dimensional matrix containing context information from front to back and back to front.

[0111] 2. Attention weighting (Attention) extracts overall representation:

[0112] ;

[0113] where, is the attention weight at time step ; is the calculation of the weighted input, is the weight parameter related to the input , is the representation of the time in the input sequence; is the weighted input;

[0114] ;

[0115] is the weighted representation; is the input representation for all time steps is the weighted sum of all time steps with weights ; is the final weighted representation, which belongs to a dimensional vector space, is the dimension of the representation.

[0116] 3. Emotion classifier (fully connected + Softmax):

[0117] where, is the input is linearly transformed by a fully connected layer and bias ; is the weight matrix of the fully connected layer, with dimension , where is the number of hidden layer nodes; is the bias term, with dimension ; is the activated vector is input to another fully connected layer, which is linearly transformed by weight matrix and bias ; is the weight matrix, with dimension , where is the number of classes for emotion classification; is the bias term, with dimension ; is the probability of each class.

[0118] The output probability distribution includes 5 classes:

[0119] Angry = 0.72; Fear = 0.10; Sad = 0.08; Neutral = 0.06; Happy = 0.04.

[0120] Emotion recognition is a single-label classification task, using cross-entropy loss:

[0121] ;

[0122] is the number of classes; is the actual label, indicating the input sample is in the The true value on the class; is the predicted value of the model, representing the model's prediction of the class of the The predicted probability of the class, is the cross-entropy loss.

[0123] II. Keyword recognition model:

[0124] 1. CNN feature extraction (for local phoneme features):

[0125] Treat X as a "speech image" and use one-dimensional convolution:

[0126] ;

[0127] F2 = ReLU (F1);

[0128] F3 = MaxPool (F2).

[0129] 2. Global average pooling:

[0130] ;

[0131] is the number of time steps or the number of features in the pooling region; F 3 [t] is the feature map at time step (or other dimension) ; is the dimension space of the final output, representing the pooled feature vector; is the output vector after global average pooling; F is the feature dimension of the convolutional layer output, which is equal to the number of convolution kernels; Conv1D represents one-dimensional convolution operation; ReLU is a nonlinear activation function; MaxPool is the maximum pooling operation.

[0132] 3. Classifier (Softmax):

[0133] ; is a probability vector representing the probability distribution of the input audio signal (or other data) belonging to each keyword class.

[0134] 4. Output keywords, if the highest probability keyword > 0.8, i.e. determine to trigger the keyword, keyword recognition also uses cross-entropy loss.

[0135] III. Audio event recognition model:

[0136] 1. CNN + BiGRU dual modeling:

[0137] ;

[0138] .

[0139] 2. Classification (multi-label) for each frame:

[0140] For each event j ∈ {1, …, K}, we have: ;

[0141] We have: ;

[0142] Average or maximize over time dimension: ;

[0143] where, is the predicted value of event at time step , indicating the classification probability of event in the current frame; is the sigmoid activation function; is the weight matrix of event , which is multiplied by the feature vector of each time step; is the feature representation at time step ; is the bias term of event , used to adjust the prediction result of each event. The multi-label prediction matrix of all time steps and all events, with dimensions , represents the predicted value for each time step and each event , is the total number of time steps (e.g., the length of the sequence), is the total number of events.

[0144] 3. If the output exceeds a certain threshold (e.g., 0.7), it triggers the event recognition.

[0145] Audio events are multi-label binary classification problems, using BinaryCrossEntropy (BCE):

[0146] .

[0147] Four, alarm strategy:

[0148] Assume the model outputs three prediction probabilities per second:

[0149] pe: whether the emotion is intense (e.g., anger / fear, etc.);

[0150] pk: whether the keyword is hit;

[0151] PA: whether a violent event (such as screaming + hitting) is detected;

[0152] The fusion score is calculated as follows:

[0153]

[0154] where we=0.3, wk=0.3, wa=0.4;

[0155] If RiskScore>0.75, it is determined as a "high-risk bullying event".

[0156] The working process of the behavior recognition module includes:

[0157] The human motion echo signal is collected by the millimeter wave radar, the radar echo data is subjected to FFT processing in the distance and speed dimensions, and the CFAR algorithm is used to detect potential target points;

[0158] The Kalman filtering is combined with the Hungarian algorithm to continuously track the potential target points, and the time sequence trajectory data of each target is obtained;

[0159] For each target, statistical features and time sequence features in the motion process are extracted in a sliding window manner to form a feature sequence;

[0160] The sequence is sent to the GRU network for time sequence modeling and behavior classification, and whether there is bullying-related behavior for each target is identified.

[0161] The process of performing FFT processing in the distance and speed dimensions on the radar echo data and using the CFAR algorithm to detect potential target points includes:

[0162] The received echo signal is subjected to distance dimension processing to extract the distance information of the target;

[0163] Based on the result of the distance dimension processing, the speed dimension processing is performed on the fixed distance unit to generate a distance-speed spectrum diagram;

[0164] The CFAR algorithm is used to make threshold decisions on each unit in the distance-speed spectrum diagram, dynamically extract target points, and map the target points to physical space coordinates according to the angle information to generate a target measurement vector containing position and speed.

[0165] In some embodiments, the above distance dimension and speed dimension processing processes are as follows:

[0166] 1. Distance dimension processing:

[0167] The received radar echo signal is subjected to distance dimension processing to extract the energy distribution of the echo on different distance units.

[0168] ​The received signal is the echo data matrix received by the multi-pulse radar:

[0169] ;

[0170] Wherein, n is the fast time sampling point (for the distance dimension); m is the slow time pulse index (for the speed dimension); N is the number of sampling points in each pulse (depending on the sampling rate and pulse width); M is the number of pulses (the number of samples under PRF);

[0171] By matching filtering (equivalent to FFT) the received signal and the transmitted signal:

[0172] ;

[0173] Wherein, S r (k,m) is the distance frequency domain signal corresponding to the mth pulse; K is the distance unit index, indicating the position of the target distance; is the time domain sampling value of the received signal, indicating the signal amplitude of the th pulse at the th time point; is a complex exponential factor, responsible for converting the signal to the frequency domain, is the index of the time domain sampling point, is the index in the frequency domain (corresponding to the discrete frequency of the distance frequency domain), is the total number of samples of the signal.

[0174] The distance resolution is , C is the speed of light (about 3\times10^8m / s), and B is the signal bandwidth.

[0175] 2、Speed dimension processing:

[0176] Doppler FFT processing is performed on each distance unit to extract target speed information.

[0177] On the fixed distance unit k, the speed dimension FFT is performed on M pulses:

[0178] ;

[0179] Wherein, S rd (k,l) is the complex value of the kth distance unit and the lth speed unit in the distance Doppler spectrum, and l is the speed unit index;

[0180] The speed resolution is , wherein, is the wavelength of the radar, and T=1 / f PRF is the pulse repetition interval, and M is the number of pulses.

[0181] 3、CFAR detection:

[0182] Threshold detection on the amplitude square (power) in the range Doppler map to identify possible target points.

[0183] Power spectrum map:

[0184] ;

[0185] CFAR threshold setting (take CACFAR as an example):

[0186] Select a reference cell window, and set the threshold to:

[0187] ;

[0188] Where R(k, l) is a set of reference cells around the target cell (k, l), N ref is the number of reference cells, and a is a proportional coefficient preset according to the false alarm rate.

[0189] 4. Target positioning and measurement vector generation:

[0190] Project the detected target points into the physical space through the direction angle information, and output the target position and speed.

[0191] Distance calculation: ; Wherein, is the final calculation result; is a constant coefficient, is an incremental value representing the distance increment between the reference point and the target point.

[0192] Speed calculation: ; Wherein, is the final calculation result; is a constant factor; is a reference quantity; is another reference quantity; is the speed increment.

[0193] Angle estimation (beamforming or angle spectrum algorithm, such as MUSIC):

[0194] Take MIMO array as an example, the angle θ can be obtained from the phase difference of the receiving array:

[0195] Wherein, is the measured phase difference, and d is the distance between array elements.

[0196] Spatial coordinate transformation (take two-dimensional plane as an example):

[0197] ; is the calculation result of distance calculation;

[0198] The target position and velocity are combined into a final measurement vector: Z = [x, y, v].

[0199] The process of continuously tracking potential target points by Kalman filtering combined with the Hungarian algorithm to obtain time sequence trajectory data of each target includes:

[0200] The position of each target in the current frame is predicted by Kalman filtering, and the matching cost is calculated combined with the detected target points;

[0201] The Hungarian algorithm is used to complete the optimal matching of the target and the detection points, so as to realize consistent tracking of the target identity and obtain human motion trajectories with unique ID;

[0202] Among them, the matched trajectory is updated by Kalman using the detection point to improve the positioning accuracy; the unmatched trajectory enters the loss management and is deleted if it exceeds the threshold; the newly appeared detection point creates a new trajectory after meeting the conditions.

[0203] In some embodiments, the process of obtaining time sequence trajectory data of each target is as follows:

[0204] v is split into v x and v y ;

[0205] 1. Kalman filter prediction:

[0206] For each trajectory updated last time, the state transition model is used to predict the state of the current frame.

[0207] State and covariance:

[0208] Last frame filtered state: represents the estimated state after filtering processing at time , usually including position and velocity, ;

[0209] Last frame covariance: ;

[0210] State transition model: ;

[0211] F: n x n state transition matrix:

[0212] Under the uniform speed model, F = , and Δt is the inter-frame time interval;

[0213] Covariance prediction: ; Q is the process noise covariance, reflecting the model uncertainty (such as random changes in acceleration).

[0214] 2. Cost computation

[0215] For each predicted trajectory And each detected measurement vector z j Compute cost matrix C;

[0216] Observation model: ;

[0217] H is the observation matrix, mapping states to measurement space. If measurements only contain position, then

[0218] H= ;

[0219] Innovation covariance: ; R is the measurement noise covariance, reflecting detection accuracy.

[0220] The cost is computed as:

[0221] ;

[0222] z j is the jth detected measurement vector (position and velocity), C ij : the cost of matching trajectory i with detection j.

[0223] 3. Optimal matching:

[0224] For the cost matrix (Nt is the number of trajectories, Nd is the number of detections) use the Hungarian algorithm to find the matching pairs that minimize the total cost.

[0225] ;

[0226] The matching result includes: successful matching pairs M, unmatched trajectories T 失配 , and unmatched detections D 新出现 .

[0227] 4. Kalman filter update:

[0228] For each matched pair (i,j) ∈ M:

[0229] Compute Kalman gain: ;

[0230] State update: ;

[0231] Covariance update: ;

[0232] Updated covariance Improves positioning accuracy.

[0233] 5. Trajectory management:

[0234] Unmatched trajectory processing:

[0235] For each i∈T 失配 , maintain a loss counter , delete the trajectory; otherwise, keep the prediction state , L max is the mismatch maximum frame number threshold.

[0236] New detection point creates a new trajectory:

[0237] For each j∈D 新出现 , and meets the initial condition (such as detection confidence > T init ), initialize the new trajectory state:

[0238] ;

[0239] Where P0 is the initial covariance, reflecting a large uncertainty on the speed, loss counter , T init is the creation condition threshold (such as detection reliability).

[0240] 6、Output the human motion trajectory with a unique ID:

[0241] Each surviving trajectory retains , forming a continuous inter-frame motion trajectory.

[0242] An electronic device, comprising a processor and a memory connected in communication with the processor and configured to store instructions executable by the processor, the processor configured to execute the above bullying behavior detection and determination method.

[0243] A server, comprising at least one processor and a memory connected in communication with the processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the processor to cause the at least one processor to execute the bullying behavior detection and determination method.

[0244] A computer-readable storage medium storing a computer program, the computer program being executed by a processor to implement the bullying behavior detection and determination method.

[0245] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the present application, and they should be covered in the scope of the claims and the description of the present application.

[0246] The above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting and determining bullying behavior based on audio and behavioral feature recognition, characterized in that, Comprise: S1, using an audio recognition module to extract and analyze emotions, keywords and abnormal sound features, and judge in real time whether there is bullying related language or audio behavior; S2, using a behavior recognition module to process behavior data and extract time sequence features, combining a GRU network to classify behaviors, complete the detection and identification of bullying related behaviors; S3, combining the recognition results of the audio recognition module and the behavior recognition module, judge whether bullying behavior occurs in the current scene, if the judgment result is yes, issue a warning and notify the corresponding personnel to the current scene; otherwise, record the recognition result to the cloud; The audio recognition module in step S1 collects environmental sound in real time through a microphone, extracts time-frequency features in the environmental sound, and sends the time-frequency features to an audio event detection model, which analyzes non-verbal sound features in the audio to identify abnormal sound events; The time-frequency features are jointly constructed by frequency domain features and time domain features, and the feature vector expression of the time-frequency features is: ; is the energy of the Mth Mel frequency band at time t; are C Mel frequency cepstral coefficients at time t as frequency domain features; and are time domain features; Mapping the spectrum to the Mel frequency scale gives Mapping the Mel energies to the cepstral domain gives ; The non-verbal sound features are obtained by CNN and BiGRU double modeling analysis: ； ； For each event j e {1,..., K}, , ; The time dimension is averaged or maximized to obtain ; where H is the dimension of the hidden state vector; F is the feature dimension of the convolutional layer output; is the predicted value of event j at time step t; is the classification probability of event j at the current frame; is the sigmoid activation function; is the weight matrix of event ; is the feature representation at time step t; is the bias term of event j; is the multi-label prediction matrix containing all time steps and all events; is the total number of time steps; K is the total number of events; In step S2, the behavior recognition module collects human motion echo signals through a millimeter wave radar, detects potential target points of the echo signals using a CFAR algorithm, and continuously tracks the potential target points using Kalman filtering combined with the Hungarian algorithm to obtain time sequence trajectory data of each target; Wherein, Kalman filtering predicts the position of each target point in the current frame, and calculates the matching cost combined with the detected target points; the Hungarian algorithm completes the optimal matching of the target and the detected point, thereby realizing consistent tracking of the target identity, and obtaining the human motion trajectory with a unique ID; The trajectory management of the time sequence trajectory data of each target includes: The trajectory of the matched target and the detected point is updated by Kalman filtering; Target and detection point mismatched trajectories, maintain loss counter If delete the trajectory, otherwise keep the prediction state, is the maximum number of frame mismatch threshold For each new detected point, and if the initial condition is met, initialize the new trajectory state. 2.The audio, behavioral feature recognition based bullying detection determination method of claim 1, wherein, The working process of the audio recognition module further includes: Denoising and silence detection on the collected environmental sound; Send the time-frequency features to the emotion recognition and keyword recognition model to identify abnormal emotions and abusive keywords respectively; Comprehensively analyze the multiple recognition results to determine whether there is a potential bullying behavior in the continuous time window, and record in real time and trigger an alarm according to the judgment result. 3.The audio, behavioral feature recognition based bullying detection determination method of claim 2, wherein, The process of extracting frequency domain and time-frequency features includes: Denoising, silence detection and normalization preprocessing on the original audio; Calculate the zero-crossing rate and short-time energy in the time domain to evaluate the sound activity and intensity, and extract the MFCC feature in the frequency domain to convert the frequency distribution into a numerical representation; Build an audio image through time-frequency analysis method to capture the space-time features, and call YAMNet pre-training model to extract high-dimensional embedding.

4. The bullying behavior detection and judgment method based on audio and behavior feature recognition according to claim 2, characterized in that: The emotion recognition model judges whether the speaker is in a high-risk emotional state by extracting the audio features of the speech and analyzing the time sequence changes thereof; The keyword recognition model detects whether there are sensitive words in the audio in real time to identify potential language violence. 5.The audio, behavioral feature recognition based bullying detection determination method of claim 1, wherein, The working process of the behavior recognition module further includes: Performing FFT processing on radar echo data in the range and velocity dimensions; For each target, extract its statistical and timing features in the process of movement in a sliding window manner to form a feature sequence; The sequence is sent to the GRU network for timing modeling and behavior classification to identify whether each target has bullying-related behavior. 6.The audio, behavioral feature recognition based bullying detection determination method of claim 5, wherein, The process of performing FFT processing on radar echo data in the range and velocity dimensions and detecting potential target points using the CFAR algorithm includes: Performing range dimension processing on the received time domain signal to extract the range information of the target; Based on the results of the range dimension processing, performing velocity dimension processing on the fixed distance unit to generate a range-velocity spectrum diagram; Using the CFAR algorithm to perform threshold judgment on each cell in the range-velocity spectrum diagram, dynamically extracting target points, and mapping the target points to physical space coordinates according to the angle information to generate a target measurement vector containing position and velocity.

7. An electronic device, comprising a processor and a memory connected to the processor in communication, and configured to store executable instructions of the processor, characterized in that: The processor is configured to execute the method of any one of claims 1-6.

8. A server, characterized by: The computer program is executed by the processor to implement the method of any one of claims 1-6.

9. A computer readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to implement the method of any one of claims 1-6.

Images