Behavior detection system for complex scenarios

Abstract

A behavior detection system for complex scenes, with the following structure: the network camera is connected to the hard disk recorder in the on-site control box through the switch in the box; the 5G wireless signal transceiver unit CPE is connected to the switch; the detection module with CPU and GPU as the core is connected to the switch; the detection module samples and processes the video stream collected by the network camera, and sends the processing results to the backend through CPE. In the detection module, an AI algorithm is used to process and identify the video stream. In the AI ​​algorithm, the video frame sequence obtained by the image acquisition device is firstly extracted to extract the skeleton information of the person in each frame; then the position features of the skeleton joints of these skeleton information in the corresponding frame are normalized; finally, the action skeleton recognition network is used for recognition and classification output.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition, and in particular to a behavior detection system applied to complex scenes. Technical Background

[0002] The purpose of behavior detection is to locate the specific location of the behavior of interest in time or even space from a long video. It is one of the most important video understanding tasks. In complex application scenarios, the target is usually affected by factors such as lighting changes, noise, and environment, and has a large appearance change, which leads to a decrease in algorithm performance. Accurately tracking pedestrians becomes a very challenging task. It is necessary to consider that complex scenes are changeable, and pedestrian tracking must adapt to the dynamic changes of the scene, especially the changes in lighting conditions, such as changes in light intensity caused by floating clouds and sudden changes in weather.

[0003] With the development of computer vision technology, many solutions have been proposed to the above problems. Usually, scene image sequences are acquired by a camera with a fixed position. Many scholars use background modeling to extract the moving area in the scene as the basis for detecting pedestrians. However, the method of using background subtraction to obtain the moving area in the scene and using a hierarchical framework to detect shadows only uses color features to detect shadows, which can only detect large shadow areas and has low robustness. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a behavior detection system for complex scenes, including an image acquisition device, an automatic recognition device and a communication device. The image acquisition device is a network camera, which is connected to the hard disk recorder in the on-site control box through the switch in the box; the communication device is a 5G wireless signal transceiver unit CPE; the CPE is connected to the switch; the automatic recognition device is a detection module with CPU and GPU as the core; the detection module is in the control box and connected to the switch; the detection module samples and processes the video stream collected by the network camera, and sends the processing results to the backend through the CPE;

[0005] In the detection module, AI algorithm is used to process and identify the video stream. In the AI ​​algorithm, the video frame sequence obtained by the image acquisition device is firstly extracted to extract the skeleton information of the person in each frame; then the position features of the skeleton joints of these skeleton information in the corresponding frame are normalized; finally, the action skeleton recognition network is used for recognition and classification output.

[0006] The image acquisition device can achieve high-precision acquisition in complex scenes such as large changes in illumination, and the automatic recognition device can realize data perception in complex scenes with ultra-dense conditions. This detection system can maintain strong robustness in complex scenes such as changes in illumination and scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 Schematic diagram of the structure of the detection system of this embodiment;

[0008] Figure 2 is a flow chart of the detection system algorithm of this embodiment;

[0009] Figure 3 4 is a diagram of the action skeleton recognition network structure of this embodiment. DETAILED DESCRIPTION

[0010] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments:

[0011] refer to Figure 1 and Figure 2 In this example, the system is used at traffic intersections to detect pedestrians. In this example, the image acquisition device is a network camera, which is connected to the hard disk recorder in the on-site control box through the switch in the box; there is also a 5G wireless signal transceiver unit CPE in the control box, which is connected to the switch; there is also a detection module with CPU and GPU as the core in the control box, which is connected to the switch, and samples and processes the video stream collected by the network camera, and sends the processing results to the background end through CPE;

[0012] In this system application, the detection module can be integrated on the basis of the existing monitoring equipment, and the use requirements of the back-end can be used to retrieve the original video or the output results of the detection module. In this system application, the operation part of the behavior detection is placed on the field end, which reduces the back-end calculation amount and data transmission amount.

[0013] In the detection module, an AI algorithm is used to process and identify the video stream. The AI ​​algorithm is:

[0014] The skeleton information of the person in each frame is extracted from the video frame sequence obtained by the image acquisition device; the position features of the skeleton joints of the skeleton information in the corresponding frame are normalized and then recognized by the action skeleton recognition network and classified and output.

[0015] refer to Figure 3 , in the action skeleton recognition network:

[0016] Step 1: Data preprocessing, using OpenPose to extract the skeleton information of each frame from the dataset video.

[0017] Step 2: Normalization. For the skeleton sequence extracted in step 1, the position features of the joints in different frames are normalized as the input of the network.

[0018] Step 3: Graph Convolutional Neural Network,The network consists of two parts, GCN and TCN. The graph convolution module uses GCN and TCN alternately to transform the time and space dimensions.

[0019] (a) For GCN, spatial shift graph operation is used to establish a lightweight spatial shift graph convolution model.

[0020] The spatial shift convolution consists of two operations: (1) Shift graph operation between nodes of the body skeleton graph: Given a spatial skeleton feature map F∈R N×C The shift distance of the i-th channel is i mod N, where N is the number of joint points and C is the input feature dimension. The shifted channel is used to fill the corresponding blank space. The above shift operation is performed on each node of F. After the shift, each node can obtain information from other nodes. (2) Point-by-point convolution is applied to exchange information between channels.

[0021] Next, we use adaptation to solve the problem of different bone importances. We calculate the element-wise product between the shifted features and the learnable mask:

[0022] This formula calculates a learnable mask Mask and shift feature The element product between them, where the learnable mask is calculated using tanh(M)+1, and M is the joint point that needs to be calculated.

[0023] (ii) For TCN, a temporal shift graph operation is used, and a lightweight adaptive temporal shift graph convolution is established to model the skeleton sequence.

[0024] The temporal shift graph convolution consists of two operations: (1) First, perform the temporal shift operation: given a skeleton sequence feature F∈R N×T×C , each channel has a learnable time shift parameter S i ,i=1,2,…,C. If the parameter is an integer, the gradient cannot be transferred, so the time shift parameter is relaxed from integer constraint to real constraint, and interpolation calculation is performed using linear interpolation; the time shift parameter is relaxed from integer constraint to real constraint. Non-integer shift can be calculated by linear interpolation: In the formula This operation is differentiable and can be trained via back-propagation. (2) Point-wise convolution.

[0025] In the formula, v is a node, and the residual produced by converting the integer into a real number is represented by λ. Since the anchor point falls on Therefore, interpolation is performed in this interval;

[0026] Step 4: Classify the features through average pooling and fully connected layers. The fully connected layer uses dropout technology to avoid overfitting, and randomly inactivates neurons with a probability of 0.5. The optimization algorithm uses stochastic gradient descent, with a learning rate of 0.01, which decays by 0.1 every 10 epochs.

[0027] The iterative process of the stochastic gradient descent algorithm is expressed as:

[0028] Among them, η is the learning rate, and i data samples are randomly selected. is the gradient of the addend function corresponding to the sample.

Claims

1. A behavior detection system for complex scenes, comprising an image acquisition device, an automatic recognition device and a communication device, characterized in that The image acquisition device is a network camera, which is connected to the hard disk recorder in the on-site control box through the switch in the box; the communication device is the 5G wireless signal transceiver unit CPE; the CPE is connected to the switch; the automatic identification device is a detection module with CPU and GPU as the core; the detection module is in the control box and connected to the switch; the detection module samples and processes the video stream collected by the network camera, and sends the processing results to the backend through the CPE; In the detection module, AI algorithms are used to process and identify video streams. In the AI ​​algorithms: Firstly, the skeleton information of the person in each frame is extracted from the video frame sequence obtained by the image acquisition device; Then the position features of the skeleton joints of these skeleton information in the corresponding frames are normalized; Finally, the action skeleton recognition network is used for recognition and classification output; Normalization processing is to normalize the position features of skeleton joints in the corresponding frames of the skeleton information sequence as the input of the action skeleton recognition network; in the action skeleton recognition network, the position features of the skeleton joints are transformed in the time and space dimensions; the features output by the action skeleton recognition network are classified and output through average pooling and fully connected layers; The action skeleton recognition network includes a graph convolutional network and a temporal convolutional network; the graph convolutional network and the temporal convolutional network are used alternately; The graph convolution network uses spatial shift graph operations to establish a lightweight spatial shift graph convolution model; the operations of the spatial shift graph convolution model include: a. First, perform the shift graph operation between the nodes of the body skeleton graph: Given a spatial skeleton feature map F∈R N×C The shift distance of the i-th channel is i mod N; the shifted channel is used to fill the corresponding blank space; the above shift operation is performed on each node of F, and after the shift, each node can obtain information from other nodes; where N is the number of joint points and C is the input feature dimension; b. Then apply point-by-point convolution to exchange information between channels; Adaptation is used to solve the problem of different importance of bones; the following formula is used to reflect the different importance of human bones: The above formula calculates a learnable mask Mask and shift feature The element product between , where the learnable mask is calculated using tanh(M)+1, where M is the joint point to be calculated; 3.2) The temporal convolutional network uses the time-shifted graph operation to establish a lightweight adaptive time-shifted graph convolution model. The operations of the time-shifted graph convolution include: a. Perform time shift operation first: Given a skeleton sequence feature F∈R N×T×C , each channel has a learnable time shift parameter S i , i = 1, 2, ..., C; if the parameter is an integer, the gradient cannot be transferred, so the time shift parameter is relaxed from the integer constraint to the real number constraint, and the interpolation calculation is performed using linear interpolation; the non-integer shift is calculated by linear interpolation as follows: Where v is a node, and the remainder after converting the integer to a real number is represented by λ. Since the anchor point falls on Therefore, interpolation is performed in this interval; this operation is differentiable and trained by back propagation; b. Point-wise convolution.

Images