A fence passing behavior recognition method based on skeleton extraction and graph convolution network

Abstract

The application provides a fence delivery behavior recognition method based on skeleton extraction and graph convolution network, and belongs to the field of video monitoring, and comprises the following steps: 1) a monitoring camera is arranged perpendicularly above a fence, so that the fence is distributed longitudinally in the field of view of the monitoring camera; a continuous video is collected in real time, and a video frame queue with a fixed length of k frames is maintained; 2) a human body joint extraction network is used to extract human body skeleton data in the video frame sequence, and two persons with potential interaction behaviors are screened out according to the skeleton size and the skeleton coordinates, and the interference of other persons in the background is removed; 3) the skeleton data of the video is sent into a graph convolution network as a set to perform behavior recognition classification, and whether the fence delivery behavior exists is judged; 4) if the fence delivery behavior exists, an alarm signal is output to a comprehensive monitoring system. The technical method can be used for station video monitoring, and automatic recognition and alarm of the fence delivery abnormal behavior are realized.

Description

Technical Field

[0001] This invention belongs to the field of video surveillance technology, and in particular relates to a method for recognizing object passing behavior across barriers based on skeleton extraction and graph convolutional networks. Background Technology

[0002] The main task of barrier-passing behavior recognition is to detect whether people are passing items between the two sides of the barrier. If so, it issues a real-time warning and notifies relevant personnel for handling. In scenarios such as subway stations, high-speed rail stations, or train stations, to prevent people from passing prohibited items to them from outside the barrier after passing them through security checks, thus avoiding potential safety hazards, an effective barrier-passing behavior recognition method is needed to promptly report the situation to management personnel.

[0003] Currently, there are several types of methods for recognizing objects passing through barriers: those based on traditional image processing, those based on target tracking, those based on key point detection, and those based on pose classification. The following section will analyze the problems existing in these methods.

[0004] In traditional image processing methods, Fa Qu Guanming et al. applied for an invention patent "A Video-Based Method for Detecting Object Delivery Behavior" (201711167860.4). This method uses frame difference and clumping detection to obtain the foreground of the suspected area and extract the contour. It judges whether there is object delivery behavior in the area of ​​interest based on the foreground pixels and gradient interpolation. This method is only suitable for situations where there are few people in the monitored field of view. In densely populated scenes such as subway stations, high-speed rail stations or train stations, this method is easily interfered with by other people who do not deliver objects, resulting in misidentification.

[0005] In the target tracking method, Zhang Enwei applied for an invention patent "A barrier-passing object detection algorithm" (201711372450.3). This method is based on deep learning, mean offset tracking algorithm and Bayesian network. It identifies and tracks objects and pedestrians, and determines whether there is barrier-passing behavior by predicting the movement trajectory of objects and pedestrians and the category to which the target belongs. However, the target tracking method requires a long computing time, resulting in poor real-time performance and difficulty in deployment in real-world scenarios.

[0006] In the key point detection-based method, Zhou Shengyao et al. applied for the invention patent "behavior recognition method, device, electronic device and readable storage medium" (202011121153.3). This method uses the railing as the target object and judges whether the line connecting the key points of a person's elbow and wrist intersects with the coordinates of the target object to recognize the behavior of passing objects across the railing. Since passing objects across the railing actually involves the transfer of items between two people, this method selects fewer key points, making it difficult to cover all situations and prone to misidentification.

[0007] In the pose-based classification method, Lu Fanbing et al. applied for the invention patent "Behavior detection method, device, equipment and readable storage medium" (202210262901.2). This method classifies the pose in each frame of the image. When the total number of target pose classification results in multiple frames of the image exceeds a threshold, it is determined that the target behavior is included within a preset time. This method does not consider the correlation and continuity of poses between multiple frames of the image.

[0008] In summary, current methods for identifying objects passing through barriers have the following problems: First, methods based on traditional image processing are only suitable for simple scenarios with few people in the monitored field of view, and are difficult to apply to complex scenarios with dense crowds, such as subway stations, high-speed rail stations, or train stations. Second, target tracking-based methods require a long computation time, resulting in poor real-time performance and making them difficult to deploy in real-world scenarios. Third, current key point detection-based methods select a limited number of key points, making it difficult to cover all situations and resulting in a high false recognition rate. Finally, current pose classification-based methods only classify single-frame images and do not consider the correlation and continuity of poses across multiple frames. Summary of the Invention

[0009] The purpose of this invention is to provide a method for recognizing object passing behavior across barriers based on skeleton extraction and graph convolutional networks, characterized by the following steps:

[0010] S1: Install surveillance cameras vertically above the fence, with the fence arranged longitudinally in the field of view of the surveillance cameras;

[0011] S2: Real-time acquisition of continuous video from surveillance cameras and maintenance of a video frame queue of length k frames;

[0012] S3: Extract human skeleton data from video frame sequences using a human joint extraction network, and filter out two people with potential interactive behaviors based on skeleton size and coordinates, while removing interference from other people in the background.

[0013] S4: Film videos of normal behavior and passing objects across the barrier, extract and filter the skeleton data from the video frame queues of the two videos, and use them as positive and negative sample sets, respectively, to train the graph convolutional neural network.

[0014] S5: Test the trained graph convolutional neural network by inputting a queue of video frames into the trained graph convolutional neural network for behavior classification to determine whether there is a continuous action of passing objects across a barrier.

[0015] S6: Based on the test steps in S5, input the skeleton data in the video frame queue of the monitoring video into the graph convolutional neural network after the test to determine whether there is a continuous action of passing objects across the barrier.

[0016] S7: If there is a case of passing items across a barrier, an alarm signal is received and an alarm signal is output to the integrated monitoring system.

[0017] Furthermore, in S2, specifically, a queue Q of length k is maintained, with a fixed length, to store consecutive video frames. Queue Q = {I1, I2, ..., I...} k}, I i Let i be the i-th frame image in the queue, where i = 1, 2, ..., k;

[0018] Whenever a new video frame is read in, the image at the head of the queue is popped and discarded, and the new video frame is added to the tail of the queue.

[0019] Furthermore, S3 specifically includes the following steps:

[0020] S31: Input each frame of the image in queue Q into the skeleton extraction network HigherHRNet frame by frame to obtain the skeleton sequence of each frame. Where, n i For image I i The number of human bodies in the middle;

[0021] S32: To improve recognition speed and reduce false recognition rate, the distance from the left shoulder node to the right hip node in each skeleton is calculated. right Sort the skeletons, set a threshold D1, and remove all skeletons smaller than the threshold D1 to obtain the processed skeleton sequence S′. i Complete the stock price sequence S i Filtering;

[0022] S33: Considering that the barrier-passing behavior recognition only involves two people, the skeleton sequence S′ i according to Calculate the average coordinates of the left and right shoulders and left and right hips for each skeleton in descending order of size. As the center coordinates of the skeleton, when there are skeletons whose center coordinates are less than a preset threshold D2, the skeletons whose center coordinates are less than the preset threshold D2 are retained. Remove the two largest skeletons; if no skeleton exists whose center coordinate distance is less than a preset threshold D2, then only retain the remaining skeletons. The two largest skeletons are used to obtain the processed skeleton sequence S″. i ;

[0023] S34: The skeleton sequence S″ of all image frames in queue Q. i Construct the skeleton sequence set S = {S″1, S″2, ..., S″} in chronological order. k}

[0024] Furthermore, S4 specifically involves: filming videos of normal behavior and passing objects across a barrier, cropping both videos into segments with a fixed number of frames k, extracting single-frame human skeleton information using HigherHRNet, using each video and its corresponding skeleton information as data, and using normal behavior and abnormal behavior of passing objects across a barrier as labels to construct a dataset, and using the constructed dataset to train the graph convolutional network 2s-AGCN.

[0025] Furthermore, S5 specifically involves inputting the queue Q and the corresponding skeleton sequence set S into the trained graph convolutional network 2s-AGCN for classification, resulting in a classification result set R = {R1, R2, ..., R...}. k}, R i Let the distribution of each human body in the i-th frame be categorized into class C = {normal behavior, abnormal behavior of passing objects across the barrier}. For R i Given the probability distribution of the j-th (0≤j≤2) individual on C, and setting a fixed threshold D3, when This indicates that an abnormal behavior of passing items across a barrier has occurred.

[0026] Compared with the prior art, the beneficial effects of the present invention are mainly reflected in:

[0027] 1. This invention uses a skeleton extraction network to extract the skeleton key points of the target pedestrian, and utilizes all the key points during classification, which is more stable than using only some key points to identify the behavior of passing objects across barriers.

[0028] 2. This invention establishes a series of methods for screening skeletons, which enables the invention to save computing resources and improve computing speed while ensuring accuracy.

[0029] 3. This invention maintains a video frame queue and classifies all skeleton information in a video frame as a whole, fully considering the continuity of the barrier-passing behavior and the correlation between barrier-passing behaviors between video frames, thereby improving the accuracy of classification. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the overall process of the method of the present invention.

[0031] Figure 2 This is a schematic diagram of the barrier-based delivery method of the present invention.

[0032] Figure 3 This is a schematic diagram illustrating the barrier-passing behavior and the skeleton extraction effect under normal conditions of the present invention. Detailed Implementation

[0033] The following will describe in more detail, with reference to the schematic diagram, a method for identifying barrier-passing behavior based on skeleton extraction and graph convolutional networks according to the present invention. The diagram illustrates a preferred embodiment of the present invention. It should be understood that those skilled in the art can modify the present invention described herein while still achieving the advantageous effects of the present invention. Therefore, the following description should be understood as being of general knowledge to those skilled in the art and is not intended to limit the present invention.

[0034] like Figure 1 As shown, the method for recognizing object passing behavior across barriers based on skeleton extraction and graph convolutional networks includes the following steps:

[0035] Step 1: Install surveillance cameras perpendicular to the top of the fence, so that the fence is vertically distributed in the field of view of the surveillance cameras, such as... Figure 2 As shown;

[0036] Step 2: Acquire continuous video in real time and maintain a video queue of fixed length k frames. The specific steps are as follows:

[0037] (1) Maintain a queue Q of length k, with a fixed length, used to store consecutive video frames. The queue is specifically defined as Q = {I1, I2, ..., I...} k}, where I i (i = 1, 2, ..., k) represents the i-th frame image in the queue;

[0038] (2) Each time a new video frame is read in, pop one frame from the head of the queue and discard it, and add the new video frame from the tail of the queue.

[0039] Step 3: Use a skeleton extraction network to extract skeleton data from the video frame sequence, and filter skeletons based on skeleton size and skeleton coordinates. The specific steps are as follows:

[0040] (1) Set queue Q = {I1, I2, ..., I...} k Each frame of the image in the dataset is fed into the skeleton extraction network HigherHRNet, frame by frame, to obtain the skeleton sequence of each frame. Where n i For image I i The number of human bodies in the middle;

[0041] (2) In order to improve the recognition speed and reduce the false recognition rate, the distance from the left shoulder node to the right hip node in each skeleton is calculated. right Sort the skeletons, set a fixed threshold D1, remove all skeletons smaller than this threshold, and obtain the processed skeleton sequence S′. i ;

[0042] (3) Considering that the recognition of the barrier-passing behavior only involves two people, the skeleton sequence S′ obtained in the previous step is processed...i according to Calculate the average coordinates of the left and right shoulders and left and right hips for each skeleton in descending order of size. As the center coordinates of the skeleton, the center coordinates are retained if they are less than a preset threshold D2. The two largest skeletons are removed, and the rest are discarded. If no skeleton exists whose center coordinate distance is less than a preset threshold D2, then only the two largest skeletons are retained. The two largest skeletons are used to obtain the processed skeleton sequence S″. i "The thresholds D1 and D2 above are determined by the size of the target person in the picture."

[0043] (4) The skeleton sequence S″ of all image frames in this queue. i "Construct a skeleton sequence set in chronological order."

[0044] """

[0045] S = {S1,S2,…,S} k}

[0046] Step 4: Use the skeleton data of this video segment as input to a graph convolutional network for behavior recognition and classification to determine whether there is any behavior of passing an object across a barrier. The specific steps are as follows:

[0047] (1) Take videos of normal behavior and videos of passing objects through a barrier, respectively. Cut the videos into segments with a fixed number of frames k. Extract the human skeleton information of a single frame using HigherHRNet. Use the video and the corresponding skeleton information as data, and use normal behavior and abnormal behavior of passing objects through a barrier as labels to construct a dataset. Use the constructed dataset to train the graph convolutional network 2s-AGCN. Figure 3 In the image, (a) represents the behavior of passing items across a barrier; and (b) represents the skeleton extraction effect under normal circumstances.

[0048] (2) Input the video queue Q and the corresponding skeleton sequence set S into the trained graph convolutional network 2s-AGCN for classification, and obtain the classification result set R = {R1, R2, ..., R}. k}, where R i Let C represent the distribution of each human body in the i-th frame on the category C = {normal, abnormal}. For R i The probability distribution of the j-th (0≤j≤2) individual on C is given by setting a fixed threshold D3. This indicates that an abnormal behavior of passing items across a barrier has occurred.

[0049] The threshold D3 is set to 0.75 by default. It can be increased or decreased appropriately according to actual use, but the increase or decrease shall not exceed 0.15.

[0050] Step 5: If there is any behavior of passing items across the barrier, an alarm signal is sent to the integrated monitoring system.

[0051] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.

Claims

1.A method for fence-passing behavior recognition based on skeleton extraction and graph convolution network, characterized in that, Comprise the following steps: S1: a monitoring camera is arranged vertically above the fence, which is longitudinally distributed in the monitoring camera field of view; S2: based on the monitoring camera, real-time continuous video is collected, and a video frame queue with a length of k frames is maintained; S3: human skeleton data in the video frame sequence is extracted by a human joint extraction network, and two persons with potential interaction behaviors are selected according to the skeleton size and skeleton coordinates, and the interference of other persons in the background is removed; S4: the video frames of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue of the two videos are extracted and screened, and the skeleton data in the video frame queue ​ ​ ​ In step S2, specifically, a queue Q of length k is maintained, with a fixed length, to store consecutive video frames. The queue Q = {I1, I2, ..., I...} k }, I i Let i be the i-th frame image in the queue, where i = 1, 2, ..., k; ​ ​ S31: input each frame of image in the queue Q into the skeleton extraction network HigherHRNet frame by frame to obtain a skeleton sequence of each frame of image wherein n i is the number of human bodies in the image I i ​ S32: To improve recognition speed and reduce false recognition rate, the distance from the left shoulder node to the right hip node in each skeleton is calculated. j = 1, 2, ..., n i ,right Sort the skeletons, set a threshold D1, and remove all skeletons smaller than the threshold D1 to obtain the processed skeleton sequence S. ' i Complete the skeleton sequence S i Filtering; S33: considering that the fence passing behavior recognition only involves two people, the skeleton sequence S ' i According to In descending order, the average value of the left and right shoulder and left and right hip coordinates of each skeleton is calculated As the center coordinates of the skeleton, when there is a skeleton with a center coordinate distance less than a preset threshold D2, the center coordinates less than the preset threshold D2 and The two largest skeletons are removed, and the remaining skeletons are removed; when there is no skeleton with a center coordinate distance less than a preset threshold D2, only the The two largest skeletons are removed, and the remaining skeletons are removed; when there is no skeleton with a center coordinate distance less than a preset threshold D2, only the i ; S34: skeleton sequence S" for all image frames in the queue Q i In the order of sequence, the skeleton sequence set S = {S"1, S"2, …, S"n} is constructed. k} 2. The fence-passing behavior recognition method based on skeleton extraction and graph convolution network according to claim 1, characterized in that, ​ 3. The fence-passing behavior recognition method based on skeleton extraction and graph convolution network according to claim 2, characterized in that, The S5 is specifically: inputting the queue Q and the corresponding skeleton sequence set S into the trained graph convolution network 2s-AGCN for classification, obtaining a classification result set R = {R1, R2, …, R k}。 i Ri is the distribution of each person in the i-th frame on the category C = {normal behavior, abnormal behavior of passing through the fence}, and let Ri,j is the probability distribution of the j-th (0≤j≤2) person in R i i on C, and a fixed threshold D3 is set, when it is indicated that the abnormal behavior of passing through the fence occurs.

Images