A fighting behavior recognition method and device based on multi-modal feature fusion
By using a multimodal feature fusion method that combines video visual, audio, and temporal features, the accuracy and classification issues of fighting identification were resolved, achieving efficient and real-time fighting behavior identification and classification, and improving the intelligence level of the public safety monitoring system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- E SURFING VISION TECHNOLOGY CO LTD
- Filing Date
- 2026-03-03
- Publication Date
- 2026-06-05
AI Technical Summary
Existing technologies for identifying fights and brawls suffer from limited identification dimensions, low accuracy, high false detection rate, inability to differentiate the severity of the behavior, and an imbalance between reasoning efficiency and accuracy, failing to meet the needs of real-time alerts.
A multimodal feature fusion method is adopted, which combines video visual, audio and time-series features. By extracting human key points and audio feature parameters from video frames, convolutional neural networks and Transformer networks are used for feature extraction and fusion. A hierarchical inference model is then used to identify and classify fighting behavior.
It improved the accuracy of fight detection, reduced the false detection rate, achieved hierarchical identification of fight behavior, met the real-time alarm requirements, and improved the efficiency of handling public safety incidents.
Smart Images
Figure CN122153584A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to a method and apparatus for identifying fighting behavior based on multimodal feature fusion. Background Technology
[0002] With the acceleration of urbanization, public safety risks in densely populated places such as shopping malls, train stations, campuses, and bars have increased significantly. Fights, due to their suddenness, violence, and cascading consequences, have become a significant threat to public order. Traditional methods for identifying fights mainly rely on manual monitoring or simple algorithms based on single visual features, which have the following problems:
[0003] Single recognition dimension: Existing technologies mostly rely on the human body outline or movement amplitude in video images to make judgments, ignoring audio features (such as shouts and impact sounds accompanying a fight) and temporal features (such as the continuity and repetition of actions). This results in low recognition accuracy in complex scenes (such as dim lighting and crowd obstruction), with a false detection rate of over 30%.
[0004] The definition of behavior is vague: there is no clear distinction between fighting and normal physical contact (such as couples playing around or friends pushing each other), nor is the intensity of fighting behavior quantified, which leads to the monitoring system frequently triggering invalid alarms or missing serious violent incidents.
[0005] Imbalance between inference efficiency and accuracy: Processing every frame of the video can ensure the integrity of the action, but it will consume a lot of computing resources, resulting in an inference delay of more than 10 seconds, which cannot meet the real-time alarm requirements; if the number of frames processed is reduced, key action information will be lost, leading to an increase in the false negative rate.
[0006] Lack of a tiered response mechanism: The existing system can only determine "whether it is a fight" but cannot distinguish the level of harm caused by the behavior, which makes it impossible for managers to quickly allocate resources.
[0007] In recent years, breakthroughs in multimodal feature fusion and deep learning technologies have offered a possibility for solving the above problems. By fusing visual, audio, and temporal features, a more comprehensive representation of fighting behavior can be constructed; with the help of hierarchical classification models, both "whether a fight has occurred" and "the level of the fight" can be determined. Therefore, there is an urgent need for a fighting identification method that balances accuracy, real-time performance, and hierarchical classification capabilities to improve the intelligence level of public safety monitoring systems. Summary of the Invention
[0008] This application provides a method and apparatus for identifying fighting behavior based on multimodal feature fusion, which improves the technical problems of low recognition accuracy and inability to distinguish the degree of harm of fighting behavior in the existing technology that uses a single vision for fighting behavior recognition.
[0009] In view of this, the first aspect of this application provides a method for identifying fighting behavior based on multimodal feature fusion, including:
[0010] Human key points are extracted from video frames in the video data, and target feature parameters are extracted from audio data acquired synchronously with the video data; the target feature parameters include Mel frequency cepstral coefficients, audio energy, and time domain zero-crossing rate;
[0011] The video frames and corresponding human key points are input into a visual feature extraction network to extract visual features; the target feature parameters are input into an audio feature extraction network to extract audio features; the differences in visual features between adjacent frames are concatenated with the rate of change of audio energy between adjacent frames to obtain temporal features.
[0012] The visual features, audio features, and temporal features are fused to obtain multimodal fused features;
[0013] The multimodal fusion features are input into the behavior classification head of the hierarchical reasoning model to identify fighting behavior. If the identification result is fighting behavior, the similarity between the multimodal fusion features and the fighting behavior template features of each level is calculated through the hierarchical classification head of the hierarchical reasoning model to obtain the hierarchical result.
[0014] Optionally, the extraction of human key points from video frames in the video data includes:
[0015] The video data in the video data is subjected to frame extraction processing to obtain several video frames;
[0016] Motion blur detection is performed on the extracted video frames, and video frames with a resolution lower than the resolution threshold are replaced with adjacent video frames.
[0017] Human body detection and cropping are performed on video frames, and human body key points are extracted from the cropped video frames using a human body key point model.
[0018] Optionally, the visual feature extraction network includes a first convolutional neural network, a Transformer network, and a feature fusion module;
[0019] The step of inputting the video frame and corresponding human key points into a visual feature extraction network to extract visual features includes:
[0020] The video frame is input into a first convolutional neural network to extract global visual features. The human key points corresponding to the video frame are input into a Transformer network to capture the temporal correlation of the key points. The output of the first convolutional neural network and the output of the Transformer network are fused by a feature fusion module to obtain the visual features of the video frame.
[0021] Optionally, the audio feature extraction network includes a second convolutional neural network and a bidirectional long short-term memory network;
[0022] The step of inputting the target feature parameters into the audio feature extraction network to extract audio features includes:
[0023] The target feature parameters are input into the audio feature extraction network, and the local features of the target feature parameters are extracted by the second convolutional neural network. The local features are then input into the bidirectional long short-term memory network to capture the time dependence of the audio data, thereby obtaining the audio features.
[0024] Optionally, the feature fusion of the visual features, the audio features, and the temporal features to obtain multimodal fusion features includes:
[0025] The visual features, audio features, and temporal features are mapped to the same dimension through a fully connected layer;
[0026] The similarity between the mapped visual features, audio features, temporal features and fighting behavior template features is calculated respectively, and the weight of each modality feature is determined based on the similarity.
[0027] The multimodal fusion features are obtained by weighting and summing the mapped modal features according to their respective weights.
[0028] Optionally, the training process of the hierarchical inference model includes:
[0029] Fighting behavior is quantified and classified into levels based on three dimensions: action type, injury risk, and weapon use.
[0030] The collected audio and video data are labeled to obtain a dataset; the labels include fighting behavior, non-fighting behavior, and the level of fighting behavior.
[0031] Multimodal features are extracted from the dataset, including visual features, audio features, and temporal features;
[0032] The multimodal features are fused and then input into a dual-task model built on a Transformer network for training. The dual-task model is updated using a joint loss function based on cross-entropy loss and contrastive loss. The trained dual-task model is then used as a hierarchical inference model.
[0033] Optionally, the method further includes:
[0034] If fighting is detected in two out of N consecutive video frames, an alarm is triggered.
[0035] If the time interval between video frames detecting fighting behavior is less than or equal to the preset time interval and the movement trajectory of key human body points is continuous, the alarm priority is increased.
[0036] If the time interval between video frames detecting fighting behavior is greater than the preset time interval or the movement trajectory of key human points is broken, the alarm will be paused.
[0037] Optionally, the method further includes:
[0038] The fighting behavior level is cross-validated using multimodal features. Alarms or classification results of severe or extremely severe fighting behavior that show conflicting cross-validation are pushed to the monitoring center, where the accuracy of the classification results is determined manually.
[0039] Optionally, the method further includes:
[0040] The labeled data is obtained based on the results of manual review, and the hierarchical reasoning model is incrementally trained using the labeled data.
[0041] A second aspect of this application provides a fighting behavior recognition device based on multimodal feature fusion, comprising:
[0042] The preprocessing unit is used to extract human key points from video frames in video data and extract target feature parameters from audio data acquired synchronously with the video data; the target feature parameters include Mel frequency cepstral coefficients, audio energy, and time domain zero-crossing rate.
[0043] The multimodal feature extraction unit is used to input the video frames and corresponding human key points into the visual feature extraction network to extract visual features; input the target feature parameters into the audio feature extraction network to extract audio features; and concatenate the differences in visual features between adjacent frames with the rate of change of audio energy between adjacent frames to obtain temporal features.
[0044] The feature fusion unit is used to fuse the visual features, the audio features, and the temporal features to obtain multimodal fused features;
[0045] The classification and grading unit is used to input the multimodal fusion features into the behavior classification head in the grading inference model to identify fighting behavior. If the identification result is fighting behavior, the similarity between the multimodal fusion features and the fighting behavior template features of each level is calculated through the grading classification head in the grading inference model to obtain the grading result.
[0046] As can be seen from the above technical solutions, this application has the following advantages:
[0047] The fighting behavior recognition method based on multimodal feature fusion provided in this application integrates video visual, environmental audio and time sequence features to solve the problem of low recognition accuracy in complex scenes (occlusion, low light, noisy environment);
[0048] Furthermore, this application establishes a four-level classification system from P1 to P4 to accurately distinguish between normal physical contact and fighting behavior, and to quantify the intensity of fighting behavior, so as to avoid invalid alarms and omission of serious incidents.
[0049] Furthermore, an optimized frame extraction strategy and a lightweight model architecture were designed to control the inference time of a single video segment to within 5 seconds while ensuring the integrity of the actions, thus meeting the requirements for real-time monitoring and alarms.
[0050] Furthermore, based on the hierarchical results, differentiated alarm information is output (such as P1 level local alerts and P4 level coordinated police forces) to provide managers with precise decision support and improve the efficiency of handling public safety incidents. Attached Figure Description
[0051] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0052] Figure 1 A flowchart illustrating a fighting behavior recognition method based on multimodal feature fusion provided in this application embodiment;
[0053] Figure 2 The scope of fighting and brawling and the P1-P4 classification standard diagram provided for the embodiments of this application;
[0054] Figure 3 A comparison diagram showing the results of the proposed method in this application and a traditional single visual recognition method provided for embodiments of this application;
[0055] Figure 4 This is a schematic diagram of a fighting behavior recognition device based on multimodal feature fusion, provided in an embodiment of this application. Detailed Implementation
[0056] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.
[0057] For easier understanding, please refer to Figure 1 This application provides a method for identifying fighting behavior based on multimodal feature fusion, including:
[0058] Step 110: Extract human body key points from video frames in the video data, and extract target feature parameters from audio data collected synchronously with the video data;
[0059] After simultaneously acquiring video and audio data, human key points are extracted from the video frames. In one embodiment, human key points can be extracted from all video frames. Specifically, image normalization is performed on each video frame to unify the frame size, which can be standardized to 640×480 pixels, with pixel values normalized to [0,1]. Then, human detection and cropping are performed on the normalized video frames. Specifically, the YOLOv8 model can be used to detect human regions in the video frames, and background areas without human figures are cropped to reduce redundant information. Finally, human key points are extracted from each cropped video frame. Specifically, 17 human key points (such as head, shoulders, wrists, ankles, etc.) can be extracted using the OpenPose model for subsequent motion feature analysis.
[0060] In another embodiment, considering that the video consists of continuous frames at 25 frames per second, processing each frame would lead to data redundancy, while excessively large intervals would result in the loss of motion information. Through experimental verification (motion integrity testing on 1000 fight video clips), a strategy of extracting 1 frame every 8 frames was determined:
[0061] Extraction frequency: 25 frames / second ÷ 8 frames / time ≈ 3.125 times / second. 16 frames can be extracted from a 5-second video, which can not only fully preserve key actions (such as the start-process-end frames of punching or slamming), but also reduce the amount of data by 72% and improve the inference efficiency by more than 3 times.
[0062] Frame filtering optimization: Motion blur detection is performed on the extracted video frames. If the clarity of the extracted video frame is lower than the clarity threshold (e.g., grayscale variance < 50), the video frame is automatically replaced with an adjacent frame to avoid blurry frames affecting feature extraction. This application embodiment preferably uses frame extraction processing of video data to control the inference time of a single video segment to within 5 seconds while ensuring the integrity of the action, thus meeting the requirements of real-time monitoring and alarm.
[0063] The selected video frames are then subjected to normalization, human detection and cropping, and human key point extraction for subsequent motion feature analysis.
[0064] The audio data accompanying the video data is extracted at a sampling rate of 16kHz in mono format. Each sampled audio segment is aligned with the timestamp of the corresponding video frame (e.g., the first frame corresponds to the audio from 0-0.32 seconds, and the second frame corresponds to the audio from 0.32-0.64 seconds). The target feature parameters such as Mel frequency cepstral coefficients, audio energy, and time domain zero-crossing rate of each sampled audio segment are extracted. The focus is on capturing the roars (frequency 200-500Hz) and impact sounds (energy peak >0.8) that accompany the fight.
[0065] Record the timestamp of each frame and the magnitude of motion changes in adjacent frames (the magnitude of motion changes in adjacent frames can be obtained by calculating the Euclidean distance between human key points in adjacent frames) to construct the temporal correlation features of the motion.
[0066] Step 120: Input the video frames and corresponding human key points into the visual feature extraction network to extract visual features; input the target feature parameters into the audio feature extraction network to extract audio features; concatenate the differences in visual features between adjacent frames with the rate of change of audio energy between adjacent frames to obtain temporal features;
[0067] This application employs a visual feature extraction network to extract visual features. This network includes a first convolutional neural network (such as a ResNet50 residual network), a Transformer network, and a feature fusion module. Specifically, video frames are input into the first convolutional neural network to extract global visual features (such as human motion contours and scene information). The corresponding human keypoints in the video frames are input into the Transformer network to capture the temporal correlation of these keypoints (such as the movement trajectory of the wrist and shoulder when throwing a punch). The feature fusion module fuses the outputs of the first convolutional neural network and the Transformer network to obtain the visual features of the video frames. These visual features include motion morphology and temporal change information.
[0068] This application employs an audio feature extraction network to extract audio features. The audio feature extraction network in this application includes a second convolutional neural network and a bidirectional long short-term memory network. The second convolutional neural network can be a 3-layer 1D convolutional neural network. The second convolutional neural network extracts local feature information of the target feature parameters (such as the frequency peak of the impact sound). The local features are input into the bidirectional long short-term memory network to capture the time dependence of the audio data (such as the duration of continuous roaring). Finally, the audio features are output, which include sound type and emotional tendency (such as anger, pain).
[0069] The differences in visual features between adjacent frames can be obtained by calculating the cosine similarity of their visual features. Then, the differences in visual features between adjacent frames are concatenated with the rate of change of audio energy in adjacent frames to obtain temporal features, which can be used to quantify the continuity and intensity of actions.
[0070] Step 130: Perform feature fusion on visual features, audio features, and temporal features to obtain multimodal fusion features;
[0071] To address the issue of multimodal feature heterogeneity, an attention-weighted fusion mechanism is adopted to dynamically allocate the weights of each modality feature.
[0072] First, feature alignment is performed on each modality: visual features, audio features, and temporal features are mapped to the same dimension through a fully connected layer. For example, the extracted 2048-dimensional visual feature vector, 1024-dimensional audio feature vector, and 512-dimensional temporal feature vector are mapped to 1024 dimensions through a fully connected layer.
[0073] Then, the attention of each modality feature is calculated: the similarity between the mapped visual features, audio features, temporal features and the fighting behavior template features is calculated respectively, and the weight of each modality feature is determined based on the similarity. The higher the similarity, the greater the weight (e.g., in a weapon-wielding fighting scenario, the weight of the audio feature (the sound of weapons colliding) is increased to 0.4, the weight of the visual feature is 0.3, and the weight of the temporal feature is 0.3).
[0074] Finally, the mapped modal features are weighted and summed using their respective weights to obtain the multimodal fusion features. These multimodal fusion features simultaneously incorporate action patterns, sound information, and temporal correlations, providing a comprehensive basis for subsequent reasoning.
[0075] Step 140: Input the multimodal fusion features into the behavior classification head in the hierarchical reasoning model to identify fighting behavior. If the identification result is fighting behavior, calculate the similarity between the multimodal fusion features and the fighting behavior template features of each level in the hierarchical classification head of the hierarchical reasoning model to obtain the hierarchical result.
[0076] The training process of the hierarchical reasoning model in this application includes:
[0077] Fighting behavior is quantified and classified into levels based on three dimensions: action type, injury risk, and weapon use.
[0078] The collected audio and video data are labeled to obtain a dataset; the labels include fighting behavior, non-fighting behavior, and the level of fighting behavior.
[0079] Extract multimodal features from the dataset, including visual features, audio features, and temporal features;
[0080] After feature fusion of multimodal features, the data is input into a dual-task model built on a Transformer network for training. The dual-task model is updated using a joint loss function based on cross-entropy loss and contrastive loss. The trained dual-task model is then used as a hierarchical inference model.
[0081] To address the issue of ambiguous behavioral definitions, the scope of fighting and brawling must first be clearly defined, along with the P1-P4 grading standards. The behavioral characteristics, scenario characteristics, and risks associated with each level are as follows: Figure 2 As shown. This application quantifies the risk of injury through three dimensions: action type (contact / attack / armed), risk of injury (none / minor / moderate / major), and use of weapons (none / unregulated / regulated), to ensure that there is no overlap between the levels and to cover all fighting-related behaviors.
[0082] To improve the model's generalization ability, the collected historical audio and video data were augmented, and a labeled dataset was constructed:
[0083] Visual enhancement: Historical video data is enhanced using operations such as random flipping, brightness adjustment (±20%), and Gaussian blur (σ<0.5) to simulate lighting changes and slightly occluded scenes;
[0084] Audio enhancement: Add background noise (such as crowd noise, music) and volume scaling (±15%) to historical audio data to simulate audio interference in different scenarios;
[0085] Dataset Construction: Collect 100,000 public safety surveillance videos (including 50,000 videos of fighting and 50,000 videos of normal behavior), and label them according to P1-P4. Among the fighting videos, P1 accounts for 20%, P2 accounts for 30%, P3 accounts for 35%, and P4 accounts for 15%, to ensure that the data distribution matches the frequency of fighting behavior in actual scenarios.
[0086] Visual, audio, and temporal features are extracted from the labeled dataset to obtain multimodal features. Using the fused multimodal features as input and labels as output, a dual-task model based on a Transformer network is trained. This trained dual-task model is then used as the hierarchical inference model. During training, a joint loss function combining cross-entropy loss (for classification tasks) and contrastive loss (for hierarchical tasks) is used to calculate the loss value. The loss value is used to update the dual-task model in reverse. The contrastive loss is used to increase the distance between features of different levels, improving hierarchical accuracy. The AdamW optimizer is used during training with an initial learning rate of 1e-4, and cosine annealing is used for learning rate scheduling. The dataset is divided into training, validation, and test sets in a 7:2:1 ratio. Training stops when the accuracy for recognizing fighting behavior on the validation set reaches 96.2% and the hierarchical accuracy reaches 93.5%. The trained dual-task model is then used as the hierarchical inference model. During training, model pruning (removing redundant convolutional kernels) and quantization (converting 32-bit floating-point numbers to 16-bit) can reduce the model size by 60% and increase the inference speed to 4.2 seconds per video segment, meeting the needs of real-time alarms.
[0087] The multimodal fusion features extracted in the preceding steps are input into the behavior classification head of the hierarchical inference model for fighting behavior identification. The probabilities of fighting and non-fighting behaviors are output. If the probability of fighting behavior is greater than or equal to 70%, it is judged as suspected fighting behavior, triggering an initial alarm. The 70% threshold can be dynamically adjusted. In densely populated scenes (such as train stations), the threshold can be lowered to 65% (reducing the false negative rate), and in quiet scenes (such as libraries), it can be raised to 75% (reducing the false positive rate). For data judged as suspected fighting behavior, its multimodal fusion features are input into the hierarchical classification head, and the similarity with each level from P1 to P4 is calculated (e.g., 85% similarity with P2, 10% similarity with P3). The level corresponding to the highest similarity is taken as the final classification result. If the highest similarity is greater than or equal to the hierarchical similarity threshold (60%), the corresponding level is directly output; if the highest similarity is less than the hierarchical similarity threshold (e.g., 55% similarity with P1, 50% similarity with P2), manual review is triggered to avoid misclassification due to ambiguous scenes.
[0088] Furthermore, embodiments of this application also include:
[0089] Step 150: Perform time-series correlation verification on the video data.
[0090] Considering the possibility of misjudgment in a single frame (such as accidental physical contact between pedestrians), the fighting behavior needs to be confirmed by the correlation of continuous actions. Specifically, if two out of N consecutive video frames detect fighting behavior, an alarm is triggered; if the time interval between the video frames detecting fighting behavior is less than or equal to a preset time interval and the movement trajectory of the human body's key points is continuous, the alarm priority is increased; if the time interval between the video frames detecting fighting behavior is greater than the preset time interval or the movement trajectory of the human body's key points is broken, the alarm is suspended.
[0091] In this embodiment, if the model inference results are ≥70% similarity to fighting behavior (i.e., suspected fighting behavior) more than twice within 5 seconds (corresponding to 16 extracted frames), a formal alarm is triggered; if the time interval between the two suspected fighting video frames is ≤2 seconds (corresponding to 5 extracted frames), and the movement trajectory of the human body key points is continuous (e.g., the previous frame is the start of a punch and the next frame is the end of a punch), it is determined to be a valid association, and the alarm priority is increased; if the time interval between the two suspected fighting video frames is >2 seconds or the movement trajectory of the human body key points is broken, it is determined to be an invalid association, no alarm is triggered, and the next frame is observed.
[0092] Furthermore, embodiments of this application also include:
[0093] Step 160: Cross-validate the fighting behavior level using multimodal features. For alarms or classification results of severe or extremely severe fighting behavior that show contradictions in the cross-validation, push them to the monitoring center for manual determination of the accuracy of the classification results.
[0094] Visual-audio cross-validation: If the visual feature is determined to be P3 level (e.g., holding a weapon), but the audio feature does not detect the impact sound of the device (energy peak <0.5), the level is downgraded to P2 and marked for review;
[0095] Temporal-visual cross-validation: If the visual features determine it to be level P4 (falling to the ground), but the temporal features show that the duration of the action is less than 1 second (no continuous beating), then it is re-determined to be level P2;
[0096] Manual review mechanism: Alarms for classification results of P3-P4 or cross-validation contradictions are automatically pushed to the monitoring center for manual verification of classification accuracy. At the same time, the manual annotation results are fed back to the model for incremental training (for every 1,000 additional labeled data, the model classification accuracy improves by 0.8-1.2%).
[0097] This application designs different response procedures based on hazard levels P1-P4 to improve the efficiency of handling public safety incidents:
[0098] Level P1: Only a notification pops up on the local monitoring terminal (green icon), without the need to link external resources, and the on-duty personnel observe subsequent behavior;
[0099] Level P2: Local alert (yellow icon) + push alarm information to the area security terminal, security personnel must arrive at the scene within 5 minutes;
[0100] Level P3: Local audio and visual alarm (red icon) + linkage with local police station (pushing on-site video and location information), police officers must arrive within 10 minutes;
[0101] Level P4: Highest level alarm + linkage with 110 command center and 120 emergency medical services (simultaneous push of video, location and damage assessment), activation of emergency response plan.
[0102] The embodiments of this application can also continuously optimize model performance through alarm data feedback from real-world scenarios:
[0103] Data accumulation: Record video clips, classification results, manual review conclusions, and handling feedback for each alarm to construct a closed-loop dataset of "alarm-handling-annotation";
[0104] Model updates: The model is fine-tuned every quarter based on 5,000 newly added labeled data, with a focus on optimizing the recognition capabilities in complex scenes (such as low light at night and dense crowds).
[0105] Dynamic threshold adjustment: Based on alarm feedback from different scenarios (such as a high false detection rate of P1 level in a bar scenario), the alarm threshold for that scenario is automatically adjusted (75% for a bar scenario) to reduce invalid alarms.
[0106] To verify the superiority of the proposed method, actual surveillance data from three typical scenarios (bar, train station, and campus) were selected for testing and compared with traditional single-vision recognition methods. The results are as follows: Figure 3 As shown in the figure. The verification results show that the method proposed in this application is significantly superior to traditional technologies in terms of identification accuracy, reasoning efficiency, and classification capability, and can effectively meet the actual needs of public safety monitoring.
[0107] The fighting behavior recognition method based on multimodal feature fusion provided in this application has the following advantages:
[0108] Significantly improved recognition accuracy: The adoption of multimodal feature fusion solves the limitations of single visual recognition in complex scenes (occlusion, low light, noise), improving the recognition accuracy to 96.2% and reducing the false detection rate to 4.8%, thus avoiding interference from invalid alarms to management personnel.
[0109] More precise tiered response: The P1-P4 tiered system can differentiate the intensity of fighting behavior, enabling managers to allocate resources as needed (e.g., P1 level only requires local alerts, while P4 level requires coordinated emergency medical services), improving the efficiency of handling public safety incidents by more than 50%.
[0110] Inference efficiency meets real-time requirements: 8-frame interval extraction and lightweight model optimization reduce the inference latency of 5-second video to 4.2 seconds, enabling real-time response from event occurrence to alarm trigger, buying time to stop serious violent incidents.
[0111] High scene adaptability: Through dynamic threshold adjustment and incremental training, it can be adapted to different scenes such as bars, stations, and campuses, and the model's generalization ability continues to improve with data accumulation, without the need for repeated development for a single scene.
[0112] Cost and benefit balance: No new dedicated hardware is required, and it can be deployed based on existing monitoring equipment. Virtual point building (model simulation testing) avoids the trial and error costs of traditional manual point deployment, while reducing the installation of invalid cameras and reducing construction and maintenance costs by more than 30%.
[0113] Please refer to Figure 4 This application also provides a fighting behavior recognition device based on multimodal feature fusion, comprising:
[0114] The preprocessing unit 410 is used to extract human key points from video frames in video data and extract target feature parameters from audio data acquired synchronously with the video data; the target feature parameters include Mel frequency cepstral coefficients, audio energy, and time domain zero-crossing rate.
[0115] The multimodal feature extraction unit 420 is used to input video frames and corresponding human key points into the visual feature extraction network to extract visual features; input target feature parameters into the audio feature extraction network to extract audio features; and concatenate the differences in visual features between adjacent frames with the rate of change of audio energy between adjacent frames to obtain temporal features.
[0116] The feature fusion unit 430 is used to fuse visual features, audio features and temporal features to obtain multimodal fused features;
[0117] The classification and grading unit 440 is used to input the multimodal fusion features into the behavior classification head in the hierarchical reasoning model to identify fighting behavior. If the identification result is fighting behavior, the similarity between the multimodal fusion features and the fighting behavior template features at each level is calculated through the hierarchical classification head in the hierarchical reasoning model to obtain the grading result.
[0118] As a further improvement, the device also includes: a timing correlation verification unit, used for:
[0119] If fighting is detected in two out of N consecutive video frames, an alarm is triggered.
[0120] If the time interval between video frames detecting fighting behavior is less than or equal to the preset time interval and the movement trajectory of key human body points is continuous, the alarm priority is increased.
[0121] If the time interval between video frames detecting fighting behavior is greater than the preset time interval or the movement trajectory of key human points is broken, the alarm will be paused.
[0122] As a further improvement, the device also includes a graded cross-validation unit, which is used to cross-validate the level of fighting behavior through multimodal features. Alarms with contradictory cross-validation results or data with graded results of severe fighting behavior and extremely severe fighting behavior are pushed to the monitoring center, where the accuracy of the graded results is determined manually.
[0123] This application integrates video vision, environmental audio, and time sequence features to solve the problem of low recognition accuracy in complex scenes (occlusion, low light, noisy environment);
[0124] Furthermore, this application establishes a four-level classification system from P1 to P4 to accurately distinguish between normal physical contact and fighting behavior, and to quantify the intensity of fighting behavior, so as to avoid invalid alarms and omission of serious incidents.
[0125] Furthermore, an optimized frame extraction strategy and a lightweight model architecture were designed to control the inference time of a single video segment to within 5 seconds while ensuring the integrity of the actions, thus meeting the requirements for real-time monitoring and alarms.
[0126] Furthermore, based on the hierarchical results, differentiated alarm information is output (such as P1 level local alerts and P4 level coordinated police forces) to provide managers with precise decision support and improve the efficiency of handling public safety incidents.
[0127] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0128] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.
[0129] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0130] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0131] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0133] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for executing all or part of the steps of the methods described in the various embodiments of this application through a computer device (which may be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.
[0134] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A method for recognizing fighting behavior based on multimodal feature fusion, characterized in that, include: Extract human body key points from video frames in video data, and extract target feature parameters from audio data collected synchronously with the video data; The target feature parameters include Mel frequency cepstral coefficients, audio energy, and time-domain zero-crossing rate; The video frames and corresponding human key points are input into a visual feature extraction network to extract visual features; the target feature parameters are input into an audio feature extraction network to extract audio features; the differences in visual features between adjacent frames are concatenated with the rate of change of audio energy between adjacent frames to obtain temporal features. The visual features, audio features, and temporal features are fused to obtain multimodal fused features; The multimodal fusion features are input into the behavior classification head of the hierarchical reasoning model to identify fighting behavior. If the identification result is fighting behavior, the similarity between the multimodal fusion features and the fighting behavior template features of each level is calculated through the hierarchical classification head of the hierarchical reasoning model to obtain the hierarchical result.
2. The method for identifying fighting behavior based on multimodal feature fusion according to claim 1, characterized in that, The extraction of human key points from video frames in the video data includes: The video data in the video data is subjected to frame extraction processing to obtain several video frames; Motion blur detection is performed on the extracted video frames, and video frames with a resolution lower than the resolution threshold are replaced with adjacent video frames. Human body detection and cropping are performed on video frames, and human body key points are extracted from the cropped video frames using a human body key point model.
3. The method for identifying fighting behavior based on multimodal feature fusion according to claim 1, characterized in that, The visual feature extraction network includes a first convolutional neural network, a Transformer network, and a feature fusion module; The step of inputting the video frame and corresponding human key points into a visual feature extraction network to extract visual features includes: The video frame is input into a first convolutional neural network to extract global visual features. The human key points corresponding to the video frame are input into a Transformer network to capture the temporal correlation of the key points. The output of the first convolutional neural network and the output of the Transformer network are fused by a feature fusion module to obtain the visual features of the video frame.
4. The method for identifying fighting behavior based on multimodal feature fusion according to claim 1, characterized in that, The audio feature extraction network includes a second convolutional neural network and a bidirectional long short-term memory network; The step of inputting the target feature parameters into the audio feature extraction network to extract audio features includes: The target feature parameters are input into the audio feature extraction network, and the local features of the target feature parameters are extracted by the second convolutional neural network. The local features are then input into the bidirectional long short-term memory network to capture the time dependence of the audio data, thereby obtaining the audio features.
5. The method for identifying fighting behavior based on multimodal feature fusion according to claim 1, characterized in that, The feature fusion of the visual features, the audio features, and the temporal features to obtain multimodal fused features includes: The visual features, audio features, and temporal features are mapped to the same dimension through a fully connected layer; The similarity between the mapped visual features, audio features, temporal features and fighting behavior template features is calculated respectively, and the weight of each modality feature is determined based on the similarity. The multimodal fusion features are obtained by weighting and summing the mapped modal features according to their respective weights.
6. The method for identifying fighting behavior based on multimodal feature fusion according to claim 1, characterized in that, The training process of the hierarchical reasoning model includes: Fighting behavior is quantified and classified into levels based on three dimensions: action type, injury risk, and weapon use. The collected audio and video data are labeled to obtain a dataset; the labels include fighting behavior, non-fighting behavior, and the level of fighting behavior. Multimodal features are extracted from the dataset, including visual features, audio features, and temporal features; The multimodal features are fused and then input into a dual-task model built on a Transformer network for training. The dual-task model is updated using a joint loss function based on cross-entropy loss and contrastive loss. The trained dual-task model is then used as a hierarchical inference model.
7. The method for identifying fighting behavior based on multimodal feature fusion according to claim 1, characterized in that, The method further includes: If fighting is detected in two out of N consecutive video frames, an alarm is triggered. If the time interval between video frames detecting fighting behavior is less than or equal to the preset time interval and the movement trajectory of key human body points is continuous, the alarm priority is increased. If the time interval between video frames detecting fighting behavior is greater than the preset time interval or the movement trajectory of key human points is broken, the alarm will be paused.
8. The method for identifying fighting behavior based on multimodal feature fusion according to claim 7, characterized in that, The method further includes: The fighting behavior level is cross-validated using multimodal features. Alarms or classification results of severe or extremely severe fighting behavior that show conflicting cross-validation are pushed to the monitoring center, where the accuracy of the classification results is determined manually.
9. The method for identifying fighting behavior based on multimodal feature fusion according to claim 8, characterized in that, The method further includes: The labeled data is obtained based on the results of manual review, and the hierarchical reasoning model is incrementally trained using the labeled data.
10. A fighting behavior recognition device based on multimodal feature fusion, characterized in that, include: The preprocessing unit is used to extract human key points from video frames in video data and extract target feature parameters from audio data acquired synchronously with the video data. The target feature parameters include Mel frequency cepstral coefficients, audio energy, and time-domain zero-crossing rate; The multimodal feature extraction unit is used to input the video frames and corresponding human key points into the visual feature extraction network to extract visual features; input the target feature parameters into the audio feature extraction network to extract audio features; and concatenate the differences in visual features between adjacent frames with the rate of change of audio energy between adjacent frames to obtain temporal features. The feature fusion unit is used to fuse the visual features, the audio features, and the temporal features to obtain multimodal fused features; The classification and grading unit is used to input the multimodal fusion features into the behavior classification head in the grading inference model to identify fighting behavior. If the identification result is fighting behavior, the similarity between the multimodal fusion features and the fighting behavior template features of each level is calculated through the grading classification head in the grading inference model to obtain the grading result.