Human body behavior recognition method and system based on multiple video streams

Abstract

The invention relates to a human body behavior recognition method and system based on multiple video streams. The method comprises the following steps: acquiring video streams shot by a network camera to obtain image frames; detecting a human body in the image frame by using a yov7-pose algorithm, and performing pose estimation to obtain a human body detection frame and corresponding joint point coordinates; human body tracking is realized through joint point coordinates by using an SORT algorithm, and an identifier is distributed for each human body detection frame; distinguishing a human body detection frame according to the identifier, and realizing human body behavior recognition by using a space-time convolution graph network to obtain a behavior recognition result; and a behavior recognition result is obtained, real-time visual display is carried out, and an alarm is given out for dangerous behaviors. Compared with the prior art, the method has the advantages of high accuracy, high expandability and the like.

Description

Technical Field

[0001] The present invention relates to the field of image processing and video understanding, and in particular to a method and system for human behavior recognition based on multi-channel video streams. Background Art

[0002] Behavior recognition is an important basic technology for artificial intelligence to serve human production and life. Human behavior recognition technology based on computer vision has been widely used in security, medical, transportation and other fields. In recent years, algorithms based on deep learning have become the mainstream in the field of behavior recognition. Such data-driven algorithms have higher accuracy and wider adaptability. Human behavior generally lasts for a certain period of time. Accurately identifying behavior requires inputting multiple frames of continuous images or videos to synchronously extract and analyze their spatiotemporal features. For robustness considerations, existing behavior recognition systems often first obtain human skeleton information through personnel detection and human posture estimation as input to the core behavior recognition module. Skeleton information is an abstract representation of human motion information, including joint coordinates, joint index numbers and the connection relationship between joints. Compared with RGB image information, skeleton information is less affected by illumination, occlusion and complex background; skeleton information is natural topological map data, which is an intuitive modeling of human motion and is more suitable as input for behavior recognition. In this way, a large behavior recognition system is decomposed into several submodules. The modular decomposition process obviously makes the system development, debugging and maintenance easier, but too many modules may also lead to reduced system efficiency, requiring developers to find a balance between efficiency and robustness.

[0003] Deep learning-based algorithms also require higher computing power for devices. With the increase in the number of monitoring points and the improvement in the quality of monitoring images in actual scenarios, traditional single-threaded video transmission and processing processes or video offline processing methods can no longer meet actual needs. For multi-channel video input, simply copying the original single-threaded behavior recognition system or directly synthesizing multiple inputs into a large batch input obviously cannot fully utilize computing resources, limits the system's throughput performance, and makes it more difficult to cope with the expansion requirements of new input streams, algorithms, and models. Summary of the invention

[0004] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method and system for human behavior recognition based on multi-channel video streams, so as to improve the recognition accuracy and the scalability of the system.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A method for human behavior recognition based on multi-channel video streams comprises the following steps:

[0007] S1: Obtain the video stream captured by the network camera to obtain image frames;

[0008] S2: Use the yolov7-pose algorithm to detect the human body in the image frame and perform posture estimation to obtain the human body detection frame and the corresponding joint point coordinates;

[0009] S3: Use the SORT algorithm to track the human body through the coordinates of the joint points and assign a label to each human body detection frame;

[0010] S4: Distinguish the human body detection frame according to the mark, and then use the spatiotemporal convolutional graph network to realize human behavior recognition and obtain the behavior recognition result;

[0011] S5: Obtain the results of behavior recognition, perform real-time visualization and issue an alarm for dangerous behaviors.

[0012] Furthermore, the spatiotemporal convolutional graph model includes joint point flow and velocity flow. The joint point flow is the position information of the joint points in a single frame, and the velocity flow is the difference between two adjacent frames of the same joint point. The human body posture and motion trajectory are obtained based on the joint point flow and velocity flow, and then the human behavior recognition result is obtained.

[0013] Furthermore, the calculation expressions of joint point flow and velocity flow are:

[0014] p t,k =(x t,k ,y t,k ,s t,k )

[0015] v t,k =p t,k -p t-1,k

[0016] In the formula, p t,k is the joint point flow, v t,k is the velocity flow, x t,k For t,k For t,k is, t is the time, and k is the kth joint.

[0017] Further, the joint points include nose, left shoulder, right shoulder, left wrist, right wrist, left elbow, right elbow, left knee, right knee, right hip, left hip, right ankle, left ankle and the midpoint between the left shoulder and the right shoulder.

[0018] Furthermore, the spatiotemporal convolutional graph model sequentially includes an input layer, a spatiotemporal graph convolution layer, a batch normalization layer, a spatiotemporal graph pooling layer, a global pooling layer and an output layer. The input of the spatiotemporal convolutional graph model is the coordinates of the joint points, and the output is the probability value of the behavior category.

[0019] The second aspect of the present invention is a human behavior recognition system based on multiple video streams, including a client and a server. The client inputs multiple video stream data, sends image data and inference requests to the server, and the server receives the inference request and image data, performs inference on the image data and returns the inference result to the client. When the system is running, it executes any of the above human behavior recognition methods based on multiple video streams.

[0020] Furthermore, the client includes a camera, an image frame acquisition module, a human posture reasoning request module, a human body tracking module and a behavior recognition reasoning request module, and the server includes a human posture reasoning response request module and a behavior recognition reasoning response request module. The camera shoots the target to be identified and generates image frames through the image frame acquisition module. The human posture reasoning request module sends the posture reasoning request and the image frame to the human posture reasoning response request module, and returns after obtaining the posture reasoning result. The human body tracking module tracks according to the posture reasoning result to generate multi-stream data. The behavior recognition reasoning request module sends the multi-stream data to the behavior recognition reasoning response request module, and returns after obtaining the behavior recognition result.

[0021] Furthermore, the client also includes a dangerous behavior alarm module, and when the behavior recognition result is a dangerous behavior, the dangerous behavior alarm module issues an alarm.

[0022] Furthermore, the client's execution process is:

[0023] 1) Read the image data and convert it into Base64 encoding to obtain the encoded string;

[0024] 2) constructing a first dictionary, which includes a key-value pair, the key of the key-value pair is a preset client key value, and the value of the key-value pair is a Base64 encoding;

[0025] 3) Send the first dictionary to the server.

[0026] Furthermore, the execution process of the server is:

[0027] 1) Receive the first dictionary sent by the client;

[0028] 2) Decode the encoded string into an image array;

[0029] 3) Define the pre-processing operation of the image and perform the pre-processing operation to obtain the processed data;

[0030] 4) The processed data is saved in a second dictionary, and the second dictionary is sent back to the client.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] 1) The present invention uses the SORT algorithm to distinguish people by joint points to avoid recognition interference between people, and then uses the spatiotemporal convolution graph to obtain the motion trajectory of each joint according to the joint point coordinates by calculating the position information of the joint points in a single frame and the motion information of the joint points between consecutive frames. It can obtain a continuous motion process, thereby distinguishing the motion posture and accurately realizing the monitoring of the actions of people in the scene.

[0033] 2) The present invention can adopt the model service deployment method, placing the detection and recognition model on the server side, and sending inference requests as multiple clients after the input of multiple video streams, which can reduce the computing pressure of the client and has high scalability, and has more application scenarios in actual production. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the algorithm used in the present invention.

[0035] Figure 2 It is a schematic diagram of human joint points required for human behavior recognition in the present invention.

[0036] Figure 3 It is a system framework diagram of the present invention.

[0037] Explanation of the markings in the picture: 0. Nose, 1. Left shoulder, 2. Right shoulder, 3. Left elbow, 4. Right elbow, 5. Left wrist, 6. Right wrist, 7. Left hip, 8. Right hip, 9. Left knee, 10. Right knee, 11. Left ankle, 12. Right ankle, 13. The midpoint between the left and right shoulders. DETAILED DESCRIPTION

[0038] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0039] Example 1

[0040] The present invention is a method and system for human behavior recognition based on multi-channel video streams. Figure 1 As shown, the method comprises the following steps:

[0041] S1: Obtain the video stream captured by the network camera to obtain image frames;

[0042] S2: Use the yolov7-pose algorithm to detect the human body in the image frame and perform posture estimation to obtain the human body detection frame and the corresponding joint point coordinates;

[0043] The yolov7-pose algorithm network consists of a backbone and a head. The backbone is used to extract features, and the head is used to detect objects and predict key points. The backbone contains multiple convolutional layers, in which ReOrg and Concat operations are used to increase the depth and width of the network. In the head, the SPPCSPC operation is used to increase the receptive field and the number of channels, and the feature maps of different levels are combined through upsampling and routing operations for object detection and key point prediction.

[0044] To realize human body detection and posture estimation, the present invention mainly focuses on the detailed information of the limbs and does not need the detailed information of the head and facial features. Therefore, in the present invention, only the nose is used as the key point of the head. At the same time, the midpoint of the left shoulder and the right shoulder is taken as the position of the neck to eliminate redundant information, reduce the system burden, and improve the system operation speed.

[0045] like Figure 2 As shown, the selected joint points include nose 0, left shoulder 1, right shoulder 2, left wrist 3, right wrist 4, left elbow 5, right elbow 6, left knee 7, right knee 8, left hip 9, right hip 10, left ankle 11, right ankle 12 and the midpoint 13 between the left shoulder and the right shoulder.

[0046] S3: Use the SORT algorithm to track the human body through the coordinates of the joint points and assign a label to each human body detection frame;

[0047] S4: Distinguish the human body detection frame according to the mark, and then use the spatiotemporal convolutional graph network to realize human behavior recognition and obtain the behavior recognition result;

[0048] The Le2i Fall Dection video dataset was selected for the production of a dataset with human joint points, and the joint point data was manually extracted for training the graph convolutional neural network. The human posture estimation model was used to extract the human joint points from the dataset, and the behavior category of each frame was manually labeled.

[0049] The behavior categories include falling, lying down, walking, standing, sitting, standing up, and sitting down. The sequence number of the action category is encoded into a One-Hot vector.

[0050] The spatiotemporal convolutional graph model includes joint point flow and velocity flow. The joint point flow is the position information of the joint point in a single frame, and the velocity flow is the difference between the same joint point in two adjacent frames. They represent the position information of the joint point in a single frame and the motion information of the joint point between consecutive frames respectively. The velocity flow is obtained by taking the difference between two adjacent frames to represent the movement of the same joint point between adjacent frames, that is, for time t, the joint indexed as k:

[0051] p t,k =(x t,k ,yt,k ,s t,k )

[0052] v t,k =p t,k -p t-1,k

[0053] In the formula, p t,k is the joint point flow, v t,k is the velocity flow, x t,k For t,k For t,k , t is time, and k is the kth joint. The joint point stream and velocity stream have the same shape, both of which are vectors of (N*C*V*T), where N is the training batch size, C represents 3 channels, and V represents the number of joint points. The human posture and motion trajectory are obtained based on the joint point stream and velocity stream, and then the human behavior recognition result is obtained.

[0054] S5: Obtain the results of behavior recognition, perform real-time visualization and issue an alarm for dangerous behaviors.

[0055] The present invention also provides a human behavior recognition system based on multi-channel video streams, such as Figure 3 As shown, it includes a client and a server. The client inputs multiple video stream data, sends image data and inference requests to the server, and the server receives the inference request and image data, performs inference on the image data and returns the inference result to the client.

[0056] The client's execution process is:

[0057] 1) Read the image data and convert it into Base64 encoding to obtain the encoded string;

[0058] 2) constructing a first dictionary, which includes a key-value pair, the key of the key-value pair is a preset client key value, and the value of the key-value pair is a Base64 encoding;

[0059] 3) Send the first dictionary to the server.

[0060] The execution process of the server is:

[0061] 1) Receive the first dictionary sent by the client;

[0062] 2) Decode the encoded string into an image array;

[0063] 3) Define the pre-processing operation of the image and perform the pre-processing operation to obtain the processed data;

[0064] 4) The processed data is saved in a second dictionary, and the second dictionary is sent back to the client.

[0065] The server is accessed through Http. The basic principle and implementation process are as follows: first, the client needs to encapsulate the data in the request body of the Http request in the format agreed upon by Proto, and then the server deserializes the Proto format data from the JSON string for subsequent processing.

[0066] Using a unified service.proto structure has better versatility. Request is the input structure, key and value are paired string arrays. name and method correspond to URL: / / {ip}:{port} / {name} / {method}. logid and clientip facilitate users to connect service-level requests and custom policies in series. Response is the output structure, err_no and err_msg express the correctness and error information of the processing result, and key and value are the results.

[0067] Set the parameters in config.yml, where worker_num represents the framework main thread and gRPC thread represents the number of pool worker threads, which can be understood as the number of concurrent network synchronization threads. rpc_port and http_port are service ports, which can be opened at the same time and cannot be empty at the same time. The model parameter information is saved in the server_conf.prototxt file generated by the conversion.

[0068] The present invention supports simultaneous detection of video streams from multiple cameras. Multiple inference requests are sent through multiple clients. The server completes behavior recognition from multiple perspectives or multiple locations simultaneously through multi-threading. Due to the centralized processing of the server, the system has higher scalability and flexibility. This architecture not only reduces deployment costs, but also improves the system's adaptability in multiple scenarios, making it suitable for applications in large-scale production environments.

[0069] Compared with the prior art, the present invention has significant advantages in the following two aspects:

[0070] 1. Efficient posture estimation and behavior recognition:

[0071] The present invention introduces the yolov7-pose algorithm and the spatiotemporal convolutional graph network (ST-GCN) in human posture estimation and behavior recognition. The combination of the two significantly improves the accuracy and efficiency of the system.

[0072] Among them, the advantages of the yolov7-pose algorithm are: traditional human posture estimation methods often find it difficult to ensure high accuracy and real-time performance when facing complex scenes and multi-target environments. The yolov7-pose algorithm is based on the powerful detection capabilities of YOLOv7, and can achieve human body detection and posture estimation within a single framework, and performs particularly well when dealing with multiple human targets. The algorithm can not only quickly locate the human body, but also provide accurate joint point coordinates, so that subsequent behavior recognition can be based on more accurate data. Compared with other algorithms, yolov7-pose is more robust in complex backgrounds and can effectively cope with challenges such as changes in camera angles and lighting changes.

[0073] Application of Spatiotemporal Convolutional Graph Network (ST-GCN): The present invention adopts a skeleton-based behavior recognition method, which has the following advantages over traditional RGB information-based recognition: 1) It is more robust and can effectively ignore the background, lighting changes and differences in character appearance in the scene, reducing the impact of external interference on the recognition results. 2) Computational resource efficiency: Skeleton data requires less storage and computing resources, which is more efficient when processing multiple video streams in real time, reducing the computational burden. The ST-GCN used in the present invention can organically combine the spatial structure information of the human body with the time series information, and characterize the dynamic changes of human body movements by constructing a spatiotemporal graph.

[0074] Through the combination of yolov7-pose and ST-GCN, the present invention achieves efficient and accurate technological breakthroughs in the field of posture estimation and behavior recognition, meeting the needs of high-precision recognition in complex scenarios.

[0075] 2. Model service deployment:

[0076] By placing the detection and recognition models on the server side, the system can process multiple video streams as inference requests sent by multiple clients. This deployment method significantly reduces the computing pressure on the client, and due to the centralized processing on the server side, the system has higher scalability and flexibility. This architecture not only reduces the deployment cost, but also improves the system's adaptability in multiple scenarios, making it suitable for applications in large-scale production environments.

[0077] Other advantages include: 3. Real-time tracking and synchronous processing: The real-time nature of video data is guaranteed by the RTSP protocol, and the SORT algorithm ensures real-time tracking and correlation processing of targets in multiple video streams. This technical solution effectively solves the problem of target loss in monitoring scenarios. 4. Visualization and early warning function of the system: The present invention can intuitively visualize the behavior recognition results and issue real-time alarms for dangerous behaviors. This function has important practicality in security monitoring and production management, and can respond to potential safety risks in a timely manner.

[0078] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A human behavior recognition method based on multi-channel video streams, characterized in that: The following steps are involved: S1: Obtain the video stream captured by the network camera to obtain image frames; S2: Use the yolov7-pose algorithm to detect the human body in the image frame and perform posture estimation to obtain the human body detection frame and the corresponding joint point coordinates; S3: Use the SORT algorithm to track the human body through the coordinates of the joint points and assign a label to each human body detection frame; S4: Distinguish the human body detection frame according to the mark, and then use the spatiotemporal convolutional graph network to realize human behavior recognition and obtain the behavior recognition result; S5: Obtain the results of behavior recognition, perform real-time visualization and issue an alarm for dangerous behaviors.

2. A method for human behavior recognition based on multi-channel video streams according to claim 1, characterized in that: The spatiotemporal convolutional graph model includes a joint point flow and a velocity flow. The joint point flow is the position information of the joint point in a single frame, and the velocity flow is the difference between two adjacent frames of the same joint point. The human body posture and motion trajectory are obtained based on the joint point flow and the velocity flow, and then the human behavior recognition result is obtained.

3. A method for human behavior recognition based on multi-channel video streams according to claim 2, characterized in that: The calculation expressions of the joint point flow and velocity flow are respectively: p t,k =(x t,k ,y t,k ,s t,k ) v t,k =p t,k -p t-1,k In the formula, p t,k is the joint point flow, v t,k is the velocity flow, x t,k For t,k For t,k is, t is the time, and k is the kth joint.

4. A method for human behavior recognition based on multi-channel video streams according to claim 1, characterized in that: The joint points include nose, left shoulder, right shoulder, left wrist, right wrist, left elbow, right elbow, left knee, right knee, right hip, left hip, right ankle, left ankle and the midpoint between the left shoulder and the right shoulder.

5. A method for human behavior recognition based on multi-channel video streams according to claim 1, characterized in that: The spatiotemporal convolutional graph model sequentially comprises an input layer, a spatiotemporal graph convolutional layer, a batch normalization layer, a spatiotemporal graph pooling layer, a global pooling layer and an output layer. The input of the spatiotemporal convolutional graph model is the coordinates of the joint points, and the output is the probability value of the behavior category.

6. A human behavior recognition system based on multi-channel video streams, characterized in that: The system comprises a client and a server. When the system is running, a human behavior recognition method based on multi-channel video streams as described in any one of claims 1 to 5 is executed. The client executes steps S1, S3, and S5, and the server executes steps S2 and S4.

7. A human behavior recognition system based on multi-channel video streams according to claim 6, characterized in that: The client includes a camera, an image frame acquisition module, a human posture reasoning request module, a human tracking module and a behavior recognition reasoning request module, and the server includes a human posture reasoning response request module and a behavior recognition reasoning response request module. The camera shoots the target to be identified and generates image frames through the image frame acquisition module. The human posture reasoning request module sends the posture reasoning request and the image frame to the human posture reasoning response request module, and returns after obtaining the posture reasoning result. The human tracking module tracks according to the posture reasoning result to generate multi-stream data. The behavior recognition reasoning request module sends the multi-stream data to the behavior recognition reasoning response request module, and returns after obtaining the behavior recognition result.

8. A human behavior recognition system based on multi-channel video streams according to claim 7, characterized in that: The client further comprises a dangerous behavior alarm module, which issues an alarm when the behavior recognition result is a dangerous behavior.

9. The method for human behavior recognition based on multi-channel video streams according to claim 6, characterized in that: The execution process of the client is: 1) Read the image data and convert it into Base64 encoding to obtain the encoded string; 2) constructing a first dictionary, which includes a key-value pair, wherein the key of the key-value pair is a preset client key value, and the value of the key-value pair is a Base64 encoding; 3) Send the first dictionary to the server.

10. The method for human behavior recognition based on multi-channel video streams according to claim 9, characterized in that: The execution process of the server is as follows: 1) Receive the first dictionary sent by the client; 2) Decode the encoded string into an image array; 3) Define the pre-processing operation of the image and perform the pre-processing operation to obtain the processed data; 4) The processed data is saved in a second dictionary, and the second dictionary is sent back to the client.

Images