Real-time video behavior analysis system based on deep learning

Abstract

The invention relates to the technical field of computer vision, in particular to a real-time video behavior analysis system based on deep learning, and aims to improve the recognition capability and response speed of abnormal behaviors such as fighting in public places. According to the system, video streams are collected in real time through a camera, and key points of hands and arms of a human body are extracted through a key point detection algorithm. By analyzing the contact action, the system can judge whether a fighting behavior exists or not, so that the misjudgment probability under the condition of dense personnel is effectively reduced. The system is combined with a light sensor and a light supplementing lamp, so that the monitoring effect under different illumination conditions is ensured. Meanwhile, the system sets a behavior recognition threshold value, an alarm mechanism is automatically triggered, and related personnel are notified in various modes such as sound, short messages and mobile application pushing. The system has the advantages of strong real-time performance, high accuracy and high automation degree, and provides a more intelligent and effective technical solution for public safety management.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a real-time video behavior analysis system based on deep learning. Background Art

[0002] As social security issues become increasingly serious, the demand for monitoring in public places continues to increase. Traditional video surveillance systems mainly rely on manual monitoring and simple motion detection algorithms, which have shortcomings such as slow response, limited coverage, and high misjudgment rate. In a crowded environment, it is difficult for monitoring personnel to pay full attention to all activities, resulting in a significant reduction in the ability to identify abnormal behaviors (such as fighting) in a timely manner.

[0003] In this context, behavior recognition technology based on deep learning has gradually become a research hotspot. Deep learning algorithms can process large amounts of data, extract complex feature information, and improve the accuracy of dynamic behavior recognition. However, existing behavior recognition systems often fail to effectively solve the following problems:

[0004] Misjudgment in complex environments: In noisy and mixed crowds, factors such as motion blur and occlusion often make it difficult for the system to accurately determine whether there are abnormal behaviors such as fighting, resulting in misjudgment.

[0005] Real-time requirements: Many existing deep learning models have high requirements for computing resources when performing video behavior analysis and cannot meet the needs of real-time monitoring, affecting the timeliness of alarms and emergency responses.

[0006] Influence of lighting conditions: In an environment with insufficient lighting, the clarity of the monitoring image is reduced, further affecting the accuracy of behavior recognition. Traditional monitoring equipment cannot automatically adapt to different lighting conditions, resulting in ineffective monitoring in some cases.

[0007] Lack of key point analysis: Existing systems usually focus on the analysis of overall behavior, but pay insufficient attention to detailed movements such as hands and arms. This leads to a lack of necessary fine-grained analysis when judging behaviors such as fighting, and is unable to effectively reduce the probability of misjudgment.

[0008] Therefore, developing an intelligent behavior analysis system that can analyze video streams in real time, accurately identify abnormal behaviors, and automatically adjust monitoring conditions in combination with light sensing technology has become an urgent need to improve the efficiency of public safety management. In response to the above problems, the present invention proposes a real-time video behavior analysis system based on deep learning. By deeply analyzing the contact actions of hands and arms in the crowd, the probability of misjudgment in crowded places is effectively reduced, and the automatic alarm function is realized, thereby providing more reliable technical support for public safety. Summary of the invention

[0009] In view of this, the purpose of the present invention is to propose a real-time video behavior analysis system based on deep learning to solve the above-mentioned problems.

[0010] Based on the above objectives, the present invention provides a real-time video behavior analysis system based on deep learning.

[0011] The real-time video behavior analysis system based on deep learning includes a video stream processing module, a behavior recognition module, a light sensor control module and an alarm mechanism, among which:

[0012] The behavior recognition module extracts the key point coordinates of human hands and arms through the key point detection algorithm, and analyzes the hand movements based on the key point coordinates to determine whether there is fighting behavior, thereby reducing the probability of misjudgment in crowded places;

[0013] When the alarm mechanism identifies that the probability of fighting behavior exceeds the set threshold, it automatically sends an alarm signal and sends a notification to relevant personnel;

[0014] Video stream processing module: This module receives the video stream from the camera in real time and uses a frame extraction algorithm to extract 25 to 30 frames of images per second. Each frame of the image is then sent to the behavior recognition module for analysis;

[0015] Behavior recognition module: By using a deep learning model (CNN and LSTM combined), this module extracts image features and analyzes key points of the hands and arms to determine whether there is fighting behavior;

[0016] Alarm mechanism: This mechanism sets a threshold for behavior recognition. When the threshold is reached, the system automatically sends an alarm signal.

[0017] Furthermore, the behavior recognition module is obtained based on deep learning training. The specific training method and main axis are as follows:

[0018] S1. Data preparation:

[0019] 1.1. Key point detection: Use a human key point detection model: Use models such as OpenPose, PoseNet or MediaPipe to extract human key points in the video, especially the key points of the hands and arms;

[0020] 1.2. Data annotation: Labeling physical contact: Label samples containing contact actions and mark the contact between hands and arms to facilitate subsequent model training;

[0021] S2. Feature extraction

[0022] 2.1. Key point extraction: Extract key point coordinates: Extract the coordinate information of each key point from the video frame, especially the wrist, elbow and shoulder;

[0023] 2.2. Feature construction: Contact feature calculation: Calculate the distance and angle change between the hand and the arm to determine the contact action;

[0024] S3. Model training

[0025] 3.1. Designing a behavior recognition model: Focusing on hand and arm features: Add hand and arm feature inputs to the model so that it can recognize physical contact behaviors;

[0026] 3.2. Dataset construction: Generate training dataset: Integrate the labeled data containing hand and arm contacts into the training set to ensure data diversity;

[0027] S4. Identification and judgment

[0028] 4.1. Real-time analysis: Real-time processing of video streams: In real-time video analysis, key point detection models are used to extract hand and arm features; Contact action determination: Setting a contact threshold (distance less than a certain value) to determine whether physical contact occurs;

[0029] 4.2. Context judgment: Combine other context information to improve the accuracy of contact actions and reduce misjudgment.

[0030] Furthermore, the video frame extraction calculation method is as follows: the system extracts 25-30 frames of video per second for processing, and sets the frame rate to F (Hz) d

[0031]

[0032] Where N is the number of frames extracted, and T is the time (seconds);

[0033] The feature extraction calculation method is as follows: Use a convolutional neural network (CNN) to extract the features of each frame. Assuming that the CNN used has L layers, the output of the lth layer can be expressed as:

[0034] A l =f(W l A l-1 +b l ).

[0035] Furthermore, the behavior recognition calculation method is as follows:

[0036] Among them, A l-1 is the output of the previous layer, W l is the weight matrix of the lth layer, b l is the bias term, f is the activation function;

[0037] Contact action analysis: Focus on analyzing the contact between the hand and the arm, use the key point detection algorithm to extract the key point coordinates, and calculate the contact distance D:

[0038]

[0039] If D <T d (set contact threshold), it is considered that there is contact;

[0040] Behavior classification: Use softmax function for multi-class classification:

[0041]

[0042] Among them, z i is the unnormalized score of the i-th class in the output layer, and K is the total number of classes.

[0043] Furthermore, the light sensor control module monitors the ambient light intensity L in real time through the light sensor. <T l (set light threshold), the fill light is automatically turned on and the brightness B is adjusted. The calculation method is as follows:

[0044] B=k·(T l -L)

[0045] Wherein, k is the brightness adjustment coefficient.

[0046] Furthermore, the alarm mechanism is when the system detects the probability of fighting behavior P fight Exceeding the set threshold T p When , the alarm is triggered, and the calculation method is as follows:

[0047]

[0048] Furthermore, the system can collect user feedback and use new data to retrain and optimize behavior recognition models, thereby continuously improving recognition capabilities; the system automatically updates model parameters to enhance adaptability to new scenarios and new behaviors, thereby improving overall recognition results.

[0049] Beneficial effects of the present invention:

[0050] The system of the present invention can process video streams and perform behavior recognition in real time, ensuring timely discovery and response to abnormal behaviors in public places, improving the efficiency of security management, and through key point detection and contact action analysis, the system can accurately judge behaviors such as fighting, thereby significantly reducing the probability of misjudgment in crowded places and improving monitoring accuracy. Once fighting behavior is identified, the system can automatically send out an alarm signal to quickly notify relevant personnel, effectively improving the emergency response speed. Combined with light sensors and fill lights, the system can maintain good monitoring effects under various lighting conditions and ensure effective operation in different environments. The system continuously optimizes the behavior recognition model by collecting user feedback, improves its adaptability and recognition ability, thereby maintaining the advancement and practicality of the technology, and through automated monitoring and alarm mechanisms, reduces dependence on manual monitoring, reduces labor costs, and improves monitoring coverage and response efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 It is a schematic diagram of the structure of each component in an embodiment of the present invention;

[0053] Figure 2 is a schematic diagram of an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a key point detection operation flow of an embodiment of the present invention;

[0055] Figure 4 The figure is a schematic diagram of the operation flow of the analysis system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0057] See also Figure 1 ,Real-time video behavior analysis system based on deep learning,First of all, the system architecture is mainly composed of the following modules:

[0058] Manual alarm 100: used to allow the user to manually intervene and alarm when abnormal behavior is identified.

[0059] Camera 200: collects video streams from public places in real time, supporting multiple resolutions and frame rate settings.

[0060] Driving motor 300: used to control the rotation and movement of camera 200 to achieve a wider monitoring range.

[0061] Fill light 400: automatically turns on when lighting conditions are insufficient to ensure the clarity of the monitoring image.

[0062] Human body 500: refers to the human target identified by the system, and its features are mainly extracted through the key point detection algorithm.

[0063] See also Figures 1 to 4 In a specific implementation, the system mainly includes a video stream processing module, a behavior recognition module, a light sensor control module and an alarm mechanism, wherein:

[0064] The behavior recognition module extracts the key point coordinates of human hands and arms through a key point detection algorithm, and analyzes hand movements based on the key point coordinates to determine whether there is fighting behavior, thereby reducing the probability of misjudgment in crowded places;

[0065] When the alarm mechanism identifies that the probability of fighting behavior exceeds a set threshold, it automatically sends an alarm signal and sends a notification to relevant personnel;

[0066] The video stream processing module: This module receives the video stream from the camera in real time and uses a frame extraction algorithm to extract 25 to 30 frames of images per second. Each frame of the image is then sent to the behavior recognition module for analysis;

[0067] Behavior recognition module: By using a deep learning model (CNN and LSTM combined), this module extracts image features and analyzes key points of the hands and arms to determine whether there is fighting behavior;

[0068] Alarm mechanism: This mechanism sets a threshold for behavior recognition. When the threshold is reached, the system automatically sends an alarm signal.

[0069] Specifically, the behavior recognition module is obtained based on deep learning training. Its specific training method and main axis are as follows:

[0070] S1. Data preparation:

[0071] 1.1. Key point detection: Use a human key point detection model: Use models such as OpenPose, PoseNet or MediaPipe to extract human key points in the video, especially the key points of the hands and arms;

[0072] 1.2. Data annotation: Labeling physical contact: Label samples containing contact actions and mark the contact between hands and arms to facilitate subsequent model training;

[0073] S2. Feature extraction

[0074] 2.1. Key point extraction: Extract key point coordinates: Extract the coordinate information of each key point from the video frame, especially the wrist, elbow and shoulder;

[0075] 2.2. Feature construction: Contact feature calculation: Calculate the distance and angle change between the hand and the arm to determine the contact action;

[0076] S3. Model training

[0077] 3.1. Designing a behavior recognition model: Focusing on hand and arm features: Add hand and arm feature inputs to the model so that it can recognize physical contact behaviors;

[0078] 3.2. Dataset construction: Generate training dataset: Integrate the labeled data containing hand and arm contacts into the training set to ensure data diversity;

[0079] S4. Identification and judgment

[0080] 4.1. Real-time analysis: Real-time processing of video streams: In real-time video analysis, key point detection models are used to extract hand and arm features; Contact action determination: Setting a contact threshold (distance less than a certain value) to determine whether physical contact occurs;

[0081] 4.2. Context judgment: Combine other context information to improve the accuracy of contact actions and reduce misjudgment.

[0082] Specifically, the video frame extraction calculation method is as follows: the system extracts 25-30 frames of video per second for processing, and sets the frame rate to F (Hz) d

[0083]

[0084] Where N is the number of frames extracted, and T is the time (seconds);

[0085] The feature extraction calculation method is as follows: Use a convolutional neural network (CNN) to extract the features of each frame. Assuming that the CNN used has L layers, the output of the lth layer can be expressed as:

[0086] A l =f(W l A l-1 +b l ).

[0087] Specifically, the behavior recognition calculation method is as follows:

[0088] Among them, A l-1 is the output of the previous layer, W l is the weight matrix of the lth layer, b lis the bias term, f is the activation function;

[0089] Contact action analysis: Focus on analyzing the contact between the hand and the arm, use the key point detection algorithm to extract the key point coordinates, and calculate the contact distance D:

[0090]

[0091] If D <T d (set contact threshold), it is considered that there is contact;

[0092] Behavior classification: Use softmax function for multi-class classification:

[0093]

[0094] Among them, z i is the unnormalized score of the i-th class in the output layer, and K is the total number of classes.

[0095] Specifically, the light sensor control module monitors the ambient light intensity L in real time through the light sensor. <T l (set light threshold), the fill light is automatically turned on and the brightness B is adjusted. The calculation method is as follows:

[0096] B=k·(T l -L)

[0097] Wherein, k is the brightness adjustment coefficient.

[0098] Specifically, the alarm mechanism is when the system detects the probability of fighting behavior P fight Exceeding the set threshold T p When , the alarm is triggered, and the calculation method is as follows:

[0099]

[0100] Specifically, the system can collect user feedback and use new data to retrain and optimize behavior recognition models, thereby continuously improving recognition capabilities; the system automatically updates model parameters to enhance adaptability to new scenarios and new behaviors, thereby improving overall recognition effects.

[0101] The working principle of the above technical solution is as follows:

[0102] 1. Video acquisition: The system collects video streams of public places in real time through the camera 200, supporting 360-degree panoramic monitoring. The camera 200 is controlled by the drive motor 300 to ensure coverage of a wider monitoring area.

[0103] 2. Light adjustment: The light sensor continuously monitors the ambient light intensity. If the light intensity is lower than the set threshold T l, the fill light 400 is automatically turned on to ensure the clarity of the monitoring image in different environments.

[0104] 3. Key point detection: The behavior recognition module processes the video frames captured by the camera 200 and extracts the coordinates of key points (such as wrists, elbows, and shoulders) of the human body 500 through a deep learning model (such as OpenPose). These key point data are used to determine the movements of the hands and arms.

[0105] 4. Contact action analysis: The system calculates the contact distance D between the hand and the arm and determines whether D is less than the set contact threshold T d If contact occurs, the behavior is recorded and further analyzed.

[0106] 5. Behavior recognition: Combined with key point information, the behavior recognition module inputs the action features in the video frame into the deep learning model, classifies possible fighting behaviors, and outputs the fighting probability P fight .

[0107] 6. Alarm mechanism: If P fight >T p (Set threshold), the system triggers automatic alarm, and uses manual alarm 100 for supplementary alarm. The alarm information can be transmitted to relevant personnel through voice, SMS and mobile application push.

[0108] 7. User feedback and optimization: The system collects user feedback on recognition results for subsequent model retraining and optimization to improve overall recognition accuracy.

[0109] The present invention is intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention. The present invention covers any substitutions, modifications, equivalent methods and schemes made on the essence and scope of the present invention. In order to enable the public to have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, and those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, processes, components and circuits are not described in detail.

[0110] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A real-time video behavior analysis system based on deep learning, characterized in that: It includes a video stream processing module, a behavior recognition module, a light sensor control module and an alarm mechanism, among which: The behavior recognition module extracts the key point coordinates of human hands and arms through a key point detection algorithm, and analyzes hand movements based on the key point coordinates to determine whether there is fighting behavior, thereby reducing the probability of misjudgment in crowded places; When the alarm mechanism identifies that the probability of fighting behavior exceeds a set threshold, it automatically sends an alarm signal and sends a notification to relevant personnel; The video stream processing module: This module receives the video stream from the camera in real time and uses a frame extraction algorithm to extract 25 to 30 frames of images per second. Each frame of the image is then sent to the behavior recognition module for analysis; Behavior recognition module: By using a deep learning model (CNN and LSTM combined), this module extracts image features and analyzes key points of the hands and arms to determine whether there is fighting behavior; Alarm mechanism: This mechanism sets a threshold for behavior recognition. When the threshold is reached, the system automatically sends an alarm signal.

2. The real-time video behavior analysis system based on deep learning according to claim 1 is characterized in that the behavior recognition module is obtained based on deep learning training, and its specific training method and main axis are as follows: S1. Data preparation: 1.

1. Key point detection: Use a human key point detection model: Use models such as OpenPose, PoseNet or MediaPipe to extract human key points in the video, especially the key points of the hands and arms; 1.

2. Data annotation: Labeling physical contact: Label samples containing contact actions and mark the contact between hands and arms to facilitate subsequent model training; S2. Feature extraction 2.

1. Key point extraction: Extract key point coordinates: Extract the coordinate information of each key point from the video frame, especially the wrist, elbow and shoulder; 2.

2. Feature construction: Contact feature calculation: Calculate the distance and angle change between the hand and the arm to determine the contact action; S3. Model training 3.

1. Designing a behavior recognition model: Focusing on hand and arm features: Add hand and arm feature inputs to the model so that it can recognize physical contact behaviors; 3.

2. Dataset construction: Generate training dataset: Integrate the labeled data containing hand and arm contacts into the training set to ensure data diversity; S4. Identification and judgment 4.

1. Real-time analysis: Real-time processing of video streams: In real-time video analysis, key point detection models are used to extract hand and arm features; Contact action determination: Setting a contact threshold (distance less than a certain value) to determine whether physical contact occurs; 4.

2. Context judgment: Combine other context information to improve the accuracy of contact actions and reduce misjudgment.

3. The real-time video behavior analysis system based on deep learning according to claim 1, characterized in that: The video frame extraction calculation method is as follows: the system extracts 25-30 frames of video per second for processing, and sets the frame rate to F (Hz) d Where N is the number of frames extracted, and T is the time (seconds); The feature extraction calculation method is as follows: Use a convolutional neural network (CNN) to extract the features of each frame. Assuming that the CNN used has L layers, the output of the lth layer can be expressed as: A l =f(W l A l-1 +b l )。 4. The real-time video behavior analysis system based on deep learning according to claim 1, characterized in that: The behavior recognition calculation method is as follows: Among them, A l-1 is the output of the previous layer, W l is the weight matrix of the lth layer, b l is the bias term, f is the activation function; Contact action analysis: Focus on analyzing the contact between the hand and the arm, use the key point detection algorithm to extract the key point coordinates, and calculate the contact distance D: If D <T d (set contact threshold), then it is considered that there is contact; Behavior classification: Use softmax function for multi-class classification: Among them, z l is the unnormalized score of the i-th class in the output layer, and K is the total number of classes.

5. The real-time video behavior analysis system based on deep learning according to claim 1 is characterized in that: The light sensor control module monitors the ambient light intensity L in real time through the light sensor. If L<T l (set light threshold), the fill light is automatically turned on and the brightness B is adjusted. The calculation method is as follows: B=k·(T l -L) Wherein, k is the brightness adjustment coefficient.

6. The real-time video behavior analysis system based on deep learning according to claim 1, characterized in that: The alarm mechanism is that when the system detects a fighting behavior, the probability P fight Exceeding the set threshold T p When , the alarm is triggered, and the calculation method is as follows:

7. The real-time video behavior analysis system based on deep learning according to claim 1 is characterized in that: The system can collect user feedback and use new data to retrain and optimize behavior recognition models, thereby continuously improving recognition capabilities; the system automatically updates model parameters to enhance adaptability to new scenarios and new behaviors, thereby improving overall recognition effects.

Images