A method for identifying tailgating fare evasion behavior

Abstract

The present invention provides a method for identifying tailgating fare evaders. The method includes obtaining a surveillance video of the location of the gate ticket entrance as input; using a human posture estimation algorithm to obtain frame-by-frame human posture data in the original video; using a human posture tracking algorithm to match the human posture data detected in each frame with the previous and next frames to obtain a complete posture sequence of each pedestrian appearing in the original data over time; using a coverage time series algorithm to obtain the time interval between adjacent pedestrians leaving the gate leaving the gate ticket entrance; using a threshold rule, pedestrians with too short an interval from leaving the gate are judged as tailgating fare evaders. This invention can timely identify tailgating fare evaders at the gate ticket entrance through existing monitoring equipment, maintain the operating order and the legitimate interests of various parties such as subway companies, reduce the occurrence of such illegal phenomena, and protect the safety of passengers and equipment.

Description

Technical Field

[0001] The present invention relates to the field of urban rail transit, and in particular to a method for identifying tailgating and fare evasion behavior. Background Art

[0002] Ticket gates can be found everywhere in public places such as subways, scenic spots, and office buildings. Tailing and fare evasion or illegal entry by following others often occur. As a common abnormal behavior, fare evasion by following others refers to a pedestrian who has not purchased a ticket and follows a pedestrian who has passed through the gate normally, taking advantage of the time gap before the gate closes to pass through the gate, thereby evading the ticket. In this type of fare evasion, the two people in front and behind can know each other and cooperate, or the person being followed may not know about it.

[0003] Tailgating and fare evasion not only brings great trouble and losses to the operation and management team, but is also difficult to detect and deal with in a timely manner. Moreover, the tailgaters are often able to evade fare supervision and punishment, causing adverse effects on society. At the same time, during the process of tailgate fare evasion, if the gate closes too quickly, the tailgaters may be injured by the gates, and the gates may also be damaged. Therefore, tailgate fare evasion has certain potential risks and hazards.

[0004] Take the subway as an example. In recent years, with the improvement of subway networks in major cities, subways have gradually become the best choice for urban residents to travel. Subway stations are the only way for people to take the subway, and the population mobility is high. In addition to fare evasion, common abnormal behaviors include fighting, illegal intrusion, panic running, etc. Since urban rail transit is a huge social public property, it has invested a lot from the early construction to the later operation, and ticket revenue is the main reliance of urban rail transit in the later operation, so the identification of fare evasion is of top priority.

[0005] To ensure safety, cameras have been installed in subway stations. At the same time, a large amount of video data is generated. How to use this data to achieve real-time monitoring of abnormal behaviors such as fare evasion in stations has become an urgent problem to be solved.

[0006] In recent years, many scholars have tried to introduce various machine learning methods into video surveillance systems to achieve the goal of intelligent video surveillance, which needs to be able to automatically detect abnormal behavior and issue an alarm. However, factors such as heavy pedestrian occlusion and the uniqueness of abnormal behavior in crowded scenes such as subway stations pose challenges to the detection of abnormal behaviors such as fare evasion.

[0007] Tailing-style fare evasion involves the interaction between two pedestrians, which cannot be identified by a single frame image, nor can the identification be completed by analyzing only a single pedestrian. Currently, no effective identification method for tailgating-style fare evasion has been proposed. Summary of the invention

[0008] The present invention provides a method for identifying tailgating and fare evasion behaviors. Based on the surveillance video data of the gate ticket inspection entrance, an algorithm combining human posture estimation, human posture tracking and time series is adopted to identify tailgating and fare evasion behaviors, and tailgating and fare evasion behaviors are captured in time so as to persuade and stop them, thereby ensuring the safety and order of operations.

[0009] A method for identifying tailgating fare evasion behavior, characterized by comprising the following steps:

[0010] (1) Obtain the surveillance video of the gate entrance as input and convert it into image frames;

[0011] (2) using a human body posture estimation algorithm to obtain human body posture data in the image frame;

[0012] (3) Using the human posture tracking algorithm, the human posture data detected in each frame is matched to obtain the complete posture sequence of each pedestrian appearing in the surveillance video over time;

[0013] (4) Using the coverage time series algorithm, the time interval between adjacent pedestrians leaving the gate is obtained;

[0014] (5) A threshold is set, and pedestrians whose exit distance from the gate is lower than the threshold are judged as tailgating fare evaders.

[0015] Furthermore, the human posture estimation algorithm described in step (2) refers to the OpenPose algorithm (conventional algorithm), which uses the BODY_25 human skeleton model to detect the key points of the human body and connect the key points belonging to the same pedestrian to obtain the human skeleton data. The specific steps are as follows:

[0016] (2a): The first stage network generates a set of detection confidence maps S 1 =ρ 1 (F) and a set of joint affine field maps where ρ 1 and They are the CNN structures of the first stage. The input of each subsequent stage comes from the prediction results of the previous stage and the original image features F, producing more accurate prediction results.

[0017] (2b): According to the predicted confidence map, a discrete set of candidate joint points is obtained, and the formula is as follows:

[0018]

[0019] in,

[0020] represents the position of the mth joint point of the jth joint point, where j represents the joint point category;

[0021] (2c): Define the line set of all candidate joint points, the formula is as follows:

[0022]

[0023] in,

[0024] Candidate joint points Represents the position of the mth joint point of the j1th joint point;

[0025] Candidate joint points Represents the position of the nth joint point of the j2th joint point;

[0026] Represents candidate joint points and candidate joint points Is there a wired connection between them? Indicates that the two are connected by wire. Indicates that there is no connection between the two;

[0027] (2d): Consider all two candidate joint points j corresponding to limb c separately 1 and j 2 ,The optimal problem of graph matching with the highest total affinity value is solved by the Hungarian algorithm and the optimal matching ,is obtained. The optimization problem with the maximum total affinity is as follows:

[0028]

[0029]

[0030]

[0031] in,

[0032] E c represents affinity value, c represents each limb of human;

[0033] E mn Represents candidate joint points and candidate joint points Affinity between

[0034] (2e): When estimating the full body pose of multiple people, the K-point graph allocation problem is solved to assemble the links that share the same body parts into the full body pose of the human body. The formula is as follows:

[0035]

[0036] Among them, E represents the affinity value, c represents each human limb, and C represents the number of humans;

[0037] (2f): Set rules to clean the obtained human skeleton data. The formula is as follows:

[0038]

[0039] in,

[0040] Indicates the validity of the corresponding human skeleton data;

[0041] Indicates the number of joint points with a confidence greater than 0.2 in the human pose estimation result of the i-th pedestrian in the t-th frame image;

[0042] num J is the joint point number threshold.

[0043] Furthermore, the human posture tracking algorithm described in step (3) is a Pose Flow algorithm (conventional algorithm), which performs frame-by-frame matching on pedestrians in the picture. The specific steps are as follows:

[0044] (3a) For a frame to be matched, based on the extracted human posture information and the data of the historical frames, a pedestrian candidate set is established to compare the similarity of features with the pedestrian to be matched, and the pedestrian posture information between the previous and next frames is matched. The pedestrian candidate set records the following six attributes of each pedestrian: the detection frame position, the detection frame position confidence, the joint point position, the joint point position confidence, the pedestrian number, and the time when the pedestrian last appeared in the picture;

[0045] (3b) Based on human posture data, similarity features between pedestrians in the image are extracted, including four groups of features: intersection-over-union ratio of detection frames between pedestrians, intersection-over-union ratio of detection frames between pedestrian joints, intersection-over-union ratio of ORB feature points of detection frames between pedestrian joints, and intersection-over-union ratio of ORB feature points of detection frames between pedestrian joints. The intersection-over-union ratio is used to describe the overlap rate between detection frames of two pedestrians or between rectangular detection frames centered on joints. The larger the intersection-over-union ratio, the higher the degree of overlap between the two pedestrians. ORB feature points are special feature points in an image, most of which are contour points. The Hamming distance is used when matching ORB feature points of two images. The smaller the Hamming distance, the higher the degree of match between the feature points corresponding to the two descriptors. The intersection-over-union ratio of ORB feature points is the intersection-over-union ratio of the number of matching ORB feature points between pedestrian detection frames or joint point detection frames. All features will be weighted averaged to measure the similarity between the pedestrian to be matched and the pedestrians in the candidate set.

[0046] (3b1) Define the intersection-over-union (IOU) ratio between the detection boxes of pedestriansbox , the formula is as follows:

[0047]

[0048] in,

[0049] A and B are the areas where the two pedestrian detection frames are located;

[0050] (3b2) Define the intersection-over-union (IOU) ratio of the detection boxes between the pedestrian joint points pose_box The specific calculation process is as follows: With the joint point as the center, a square with a side length of l is constructed as the detection frame of the joint point; the intersection and union ratio of the joint point detection frames of the two pedestrians corresponding to the joint points is calculated. Where i = 0, 1, ..., 24; the 25 joint point detection frame intersection and union ratio calculation results are sorted from large to small according to the confidence of the joint point detection results, and the corresponding elements with a confidence of 0 are directly eliminated; the average of the first num joint point detection frame intersection and union ratios after sorting is taken as the joint point detection frame intersection and union ratio IOU between pedestrian C1 and pedestrian P1 pose_box .

[0051] (3b3) Define the ORB feature point intersection between the detection boxes of pedestrians box , the formula is as follows:

[0052]

[0053] in,

[0054] Kp A represents the set of ORB feature points in the detection box A, Kp B represents the set of ORB feature points in the detection box B, Kp A ∩Kp B Indicates the set of ORB feature points that are successfully matched in detection box A and detection box B, num(Kp A ∩Kp B ) represents the set Kp A ∩Kp B The number of feature points in ;

[0055] (3b4) Define the ORB feature point intersection and union ratio of the detection box between pedestrian joint points pose_box The calculation process is as follows: take the joint point as the center, construct a square with a side length of l as the detection frame of the joint point; calculate the intersection and union ratio of the feature points of the joint point detection frames of the two pedestrians corresponding to the joint points Where i = 0, 1, ..., 24; the calculated results of the feature point intersection and union ratio of the 25 joint point detection frames are sorted from large to small according to the confidence of the joint point detection results, and the corresponding elements with a confidence of 0 are directly eliminated; the average value of the feature point intersection and union ratio of the first num joint point detection frames after sorting is taken as the ORB feature point intersection and union ratio of the detection frame between the pedestrian joint points pose_box .

[0056] (3b5) Set the similarity S of pedestrian matching, and its formula is as follows:

[0057]

[0058] in,

[0059] w 1 、w 2 、w 3 、w 4 Feature IOU box , IOU pose_box 、ORB box 、ORB pose_box The weight of

[0060] (3c) According to the extracted pedestrian features, the pedestrians to be matched in the frame are matched with the pedestrians in the candidate set. For example, the Hungarian algorithm is used. Each pedestrian can be matched successfully or not. The Hungarian algorithm is used to match the pedestrians detected in the current frame with the pedestrians in the pedestrian candidate set according to the cost matrix. The pedestrians in the current frame are matched with the pedestrians in the pedestrian candidate set based on the matching principle of the maximum matching index S of all pedestrians.

[0061] Assume that there are M pedestrians in the current frame and N pedestrians in the pedestrian candidate set. The matching index between the pedestrian Ci in the current frame and the pedestrian Pj in the pedestrian candidate set is s ij , we can get the pedestrian matching index matrix S:

[0062]

[0063] Use k ij represents the matching status of the pedestrian Ci in the current frame and the pedestrian Pj in the pedestrian candidate set, and then the pedestrian matching status matrix K can be obtained. The matrix K also contains M rows and N columns. ij The assignment rules are as follows:

[0064]

[0065] The pedestrian matching problem is transformed into the following optimization problem:

[0066]

[0067]

[0068] By solving the above model through the Hungarian algorithm, the pedestrian matching state matrix K can be obtained, that is, the matching result between the pedestrian in the current frame and the pedestrian in the pedestrian candidate set can be obtained;

[0069] (3d) After the pedestrian matching of the frame to be matched is completed, the data in the pedestrian candidate set is updated according to the matching results, and pedestrians without matching results are added as new pedestrians to the candidate set, and the 6 sets of pedestrian information of the successfully matched pedestrians in the new frame are updated to the pedestrian candidate set, and pedestrians in the candidate set that have not been successfully matched for a long time are regarded as having left the picture area and removed from the candidate set, and then the next frame is matched, and steps (3a) to (3d) are repeated until all frames are matched;

[0070] (3e1) If the pedestrian in the pedestrian candidate set matches the pedestrian in the current frame successfully, the pedestrian information in the current frame is used to replace the corresponding pedestrian information in the pedestrian candidate set. For example, in the above example, pedestrian C1 in the current frame matches pedestrian P1 in the pedestrian candidate set successfully, then the pedestrian detection box position information, pedestrian detection box confidence, joint point position information and other attributes of pedestrian C1 are used to replace the corresponding attributes of pedestrian P1;

[0071] (3e2) If there is a pedestrian Px in the pedestrian candidate set that has not been successfully matched with the pedestrian in the current frame, then it can be divided into two cases: ① Pedestrian Px has not been successfully matched for a long time, for example, pedestrian Px has not updated its status for 100 frames or 5s in a row. At this time, the pedestrian may have left the camera's monitoring area, so the pedestrian should be removed from the pedestrian candidate set; ② Pedestrian Px has not been successfully matched for a short period of time, which may be due to the target detection algorithm. The pedestrian is just not detected in the current frame. At this time, the attributes of pedestrian Px should be kept unchanged;

[0072] (3e3) If there is a pedestrian Cx in the current frame that is not successfully matched with any pedestrian in the pedestrian candidate set, this may be because this pedestrian enters the camera's monitoring area for the first time and there is no information about this pedestrian in the pedestrian candidate set. Therefore, this pedestrian needs to be added to the pedestrian candidate set, and its pedestrian number is the maximum pedestrian number that has not been added to the previous pedestrian candidate set + 1.

[0073] Furthermore, the coverage time series algorithm described in step (4) comprises the following specific steps:

[0074] (4a) Based on the skeleton data of the pedestrian, the rectangular area where the pedestrian is located is delineated, and the coverage between the pedestrian and the rectangular area where the gate is located is calculated. The specific calculation formula for coverage is as follows:

[0075]

[0076] Among them, the rectangular frame area A is the gate area, the rectangular frame area B is the pedestrian detection frame range, and the rectangular frame area A∩B is the intersection of the two;

[0077] (4b) Using time signal processing methods, the time series of coverage is processed and the position relationship between pedestrians and gates is divided into two different states, namely, pedestrians inside the gate and pedestrians outside the gate. A threshold c is set. th To binarize the time series curve of coverage, let c th =0.2. z is used to represent the positional relationship between the pedestrian and the gate. When z=1, the pedestrian is within the gate range; when z=0, the pedestrian is not within the gate range. The specific calculation method of z is as follows:

[0078]

[0079] (4c) According to the change in the position relationship between pedestrians and gates, the time point for pedestrians to exit the gate is determined, and the exit interval between pedestrians is further calculated. The pedestrian exit interval T interval The calculation formula is as follows:

[0080] T interval =t out -t′ out

[0081] Among them, t out is the moment when the pedestrian leaves the gate area, t′ out Indicates the time when the previous pedestrian left the same gate area.

[0082] Furthermore, the threshold setting described in step (5) uses different threshold values ​​according to the gate type, and the behavior of the gate exit interval below the threshold is judged as tailgating fare evasion behavior, while the behavior above the threshold is judged as no tailgating fare evasion occurs.

[0083] Furthermore, the coverage described in step (4a) adopts the relative intersection-union ratio, that is, the area of ​​the intersection area between pedestrians and gates divided by the area of ​​the pedestrian area. The calculation method of coverage is similar to the calculation method of intersection-union ratio, the difference is that the denominator of coverage is A, while the denominator of intersection-union ratio is A∪B.

[0084] Furthermore, the time signal processing means described in step (4b) is a binarization algorithm, which converts the values ​​of pedestrian coverage that change over time, thereby extracting the positional relationship between the pedestrian and the gate into two states, namely, the pedestrian is inside the gate and the pedestrian is outside the gate.

[0085] The beneficial effects of the present invention are:

[0086] (1) A method for identifying tailgating fare evasion behavior is proposed, which can be continuously monitored in real time by surveillance cameras to solve the shortcomings of manual identification;

[0087] (2) Extract the human posture information of each frame from the original data, and use the human posture tracking algorithm to match the pedestrians in the previous and next frames to form a time series of the human posture information of each pedestrian, overcoming the interference of complex background and different clothing;

[0088] (3) The spatial-temporal characteristics of the pedestrian’s position in the picture, the human posture information, and the time series of pedestrian information changing over time are fully utilized to avoid misjudgment caused by the detection model based on single-frame information. The missing or erroneous results of a few frames do not affect the overall recognition results, and good recognition effects are still achieved in crowded and obstructed scenes.

[0089] (4) The exit interval of pedestrians is extracted from the original data, that is, the difference in time between two adjacent pedestrians leaving the gate. Too low an exit interval means that tailgating and fare evasion has occurred, which effectively realizes the accurate identification of tailgating and fare evasion. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0091] Figure 1 is a flow chart of a method for identifying tailgating fare evasion behavior according to Embodiment 1 of the present invention;

[0092] Figure 2 is a schematic diagram of the BODY_25 human skeleton model of OpenPose according to the first embodiment of the present invention;

[0093] Figure 3 is a human skeleton data result of a certain picture frame according to the first embodiment of the present invention;

[0094] Figure 4 is a flow chart of a human body posture tracking algorithm according to Embodiment 1 of the present invention;

[0095] Figure 5 is a schematic diagram of a flow chart of a human body posture tracking algorithm according to Embodiment 1 of the present invention;

[0096] Figure 6is the detection frame of the pedestrian C1 in a frame to be matched and the pedestrians P1 to P4 in the candidate set according to the first embodiment of the present invention;

[0097] Figure 7 is a detection frame of a joint point J1 of a pedestrian C1 in a frame to be matched and a pedestrian P1 in a candidate set according to the first embodiment of the present invention;

[0098] Figure 8 are successfully matched ORB feature points in the detection frames of the pedestrian C1 in a certain frame to be matched and the pedestrian P1 in the candidate set according to the first embodiment of the present invention (the two frames are connected by a line);

[0099] Fig. 9 is a successfully matched ORB feature point in the detection frame of the joint point J0 of the pedestrian C1 in a certain frame to be matched and the pedestrian P1 in the candidate set according to the first embodiment of the present invention (connected by a line in the two frames);

[0100] Fig.10 It is a flowchart of the coverage time series algorithm according to the first embodiment of the present invention.

[0101] Fig.11 It is a flowchart of the coverage time series algorithm according to the first embodiment of the present invention.

[0102] Fig.12 4 is a bar graph of experimental results of the pedestrian gate interval according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0103] The technical solution of the present invention will be clearly and completely described below with reference to the accompanying drawings and in combination with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0104] Embodiment 1:

[0105] like Figure 1 As shown, the method for identifying tailgating fare evasion behavior provided in the first embodiment of the present invention may include the following steps:

[0106] A method for identifying tailgating fare evasion behavior, characterized by comprising the following steps:

[0107] (1) Obtain the surveillance video of the gate entrance as input and convert it into image frames;

[0108] (2) using a human body posture estimation algorithm to obtain human body posture data in the image frame;

[0109] (3) Using the human posture tracking algorithm, the human posture data detected in each frame is matched to obtain the complete posture sequence of each pedestrian appearing in the surveillance video over time;

[0110] (4) Using the coverage time series algorithm, the time interval between adjacent pedestrians leaving the gate is obtained;

[0111] (5) A threshold is set, and pedestrians whose exit distance from the gate is lower than the threshold are judged as tailgating fare evaders.

[0112] Tailing fare evasion behavior involves two pedestrians in front and behind, and it is difficult to identify it by analyzing and detecting a single pedestrian alone. The above steps can accurately identify tailgating fare evasion behavior in a complex, crowded and obscured environment when tailgating fare evasion behavior samples are scarce, ensuring the order and stability of subway stations and the personal safety of passengers.

[0113] Preferably, the human body posture estimation algorithm described in step (2) refers to the OpenPose algorithm (conventional algorithm), using the BODY_25 human skeleton model, such as Figure 2 As shown in the figure, the key points of the human body are detected, and the key points belonging to the same pedestrian are connected to obtain the human skeleton data. The human skeleton data result of a certain picture frame is as follows: Figure 3 As shown, the specific steps are:

[0114] (2a): The first stage network generates a set of detection confidence maps S 1 =ρ 1 (F) and a set of joint affine field maps where ρ 1 and They are the CNN structures of the first stage. The input of each subsequent stage comes from the prediction results of the previous stage and the original image features F, producing more accurate prediction results.

[0115] (2b): According to the predicted confidence map, a discrete set of candidate joint points is obtained, and the formula is as follows:

[0116]

[0117] in,

[0118] represents the position of the mth joint point of the jth joint point, where j represents the joint point category;

[0119] (2c): Define the line set of all candidate joint points, the formula is as follows:

[0120]

[0121] in,

[0122] Candidate joint points Represents the position of the mth joint point of the j1th joint point;

[0123] Candidate joint points Represents the position of the nth joint point of the j2th joint point;

[0124] Represents candidate joint points and candidate joint points Is there a wired connection between them? Indicates that the two are connected by wire. Indicates that there is no connection between the two;

[0125] (2d): Consider all two candidate joint points j corresponding to limb c separately 1 and j 2 ,The optimal problem of graph matching with the highest total affinity value is solved by the Hungarian algorithm and the optimal matching ,is obtained. The optimization problem with the maximum total affinity is as follows:

[0126]

[0127]

[0128]

[0129] in,

[0130] E c represents affinity value, c represents each limb of human;

[0131] E mn Represents candidate joint points and candidate joint points Affinity between

[0132] (2e): When estimating the full body pose of multiple people, the K-point graph allocation problem is solved to assemble the links that share the same body parts into the full body pose of the human body. The formula is as follows:

[0133]

[0134] Among them, E represents the affinity value, c represents each human limb, and C represents the number of humans;

[0135] (2f): Set rules to clean the obtained human skeleton data. The formula is as follows:

[0136]

[0137] in,

[0138] Indicates the validity of the corresponding human skeleton data;

[0139] Indicates the number of joint points with a confidence greater than 0.2 in the human pose estimation result of the i-th pedestrian in the t-th frame image;

[0140] num J is the joint point number threshold.

[0141] Preferably, the human posture tracking algorithm described in step (3) is a Pose Flow algorithm (conventional algorithm), which performs frame-by-frame matching on pedestrians in the picture, such as Figure 4 and Figure 5 As shown, the specific steps are:

[0142] (3a) For a frame to be matched, based on the extracted human posture information and the data of the historical frames, a pedestrian candidate set is established to compare the similarity of features with the pedestrian to be matched, and the pedestrian posture information between the previous and next frames is matched. The pedestrian candidate set records the following six attributes of each pedestrian: the detection frame position, the detection frame position confidence, the joint point position, the joint point position confidence, the pedestrian number, and the time when the pedestrian last appeared in the picture;

[0143] (3b) Based on human posture data, similarity features between pedestrians in the image are extracted, including four groups of features: intersection-over-union ratio of detection frames between pedestrians, intersection-over-union ratio of detection frames between pedestrian joints, intersection-over-union ratio of ORB feature points of detection frames between pedestrian joints, and intersection-over-union ratio of ORB feature points of detection frames between pedestrian joints. The intersection-over-union ratio is used to describe the overlap rate between detection frames of two pedestrians or between rectangular detection frames centered on joints. The larger the intersection-over-union ratio, the higher the degree of overlap between the two pedestrians. ORB feature points are special feature points in an image, most of which are contour points. The Hamming distance is used when matching ORB feature points of two images. The smaller the Hamming distance, the higher the degree of match between the feature points corresponding to the two descriptors. The intersection-over-union ratio of ORB feature points is the intersection-over-union ratio of the number of matching ORB feature points between pedestrian detection frames or joint point detection frames. All features will be weighted averaged to measure the similarity between the pedestrian to be matched and the pedestrians in the candidate set.

[0144] (3b1) Define the intersection-over-union (IOU) ratio between the detection boxes of pedestrians box , the formula is as follows:

[0145]

[0146] in,

[0147] A and B are the areas where the two pedestrian detection frames are located. The detection frame areas of pedestrian C1 in a frame to be matched and pedestrians P1 to P4 in the candidate set are as follows: Figure 6 As shown;

[0148] (3b2) Define the intersection-over-union (IOU) ratio of the detection boxes between the pedestrian joint points pose_box The specific calculation process is as follows: With the joint point as the center, a square with a side length of l is constructed as the detection frame of the joint point; the intersection and union ratio of the joint point detection frames of the two pedestrians corresponding to the joint points is calculated. Where i = 0, 1, ..., 24; the 25 joint point detection frame intersection-over-union calculation results are sorted from large to small according to the confidence of the joint point detection results, and the corresponding elements with a confidence of 0 are directly eliminated; the average of the first num joint point detection frame intersection-over-union ratios after sorting is taken as the joint point detection frame intersection-over-union ratio (IOU) between pedestrians pose_box , the detection box area of ​​the joint point J1 of the pedestrian C1 in a certain frame to be matched and the pedestrian P1 in the candidate set is as follows Figure 7 shown.

[0149] (3b3) Define the ORB feature point intersection between the detection boxes of pedestrians box , the formula is as follows:

[0150]

[0151] in,

[0152] Kp A represents the set of ORB feature points in the detection box A, Kp B represents the set of ORB feature points in the detection box B, Kp A ∩Kp B Indicates the set of ORB feature points that are successfully matched in detection box A and detection box B, num(Kp A ∩Kp B ) represents the set Kp A ∩Kp B The number of feature points in the detection frame of pedestrian C1 in a certain frame to be matched and pedestrian P1 in the candidate set successfully matched ORB feature points (connected by a line in the two frames) Figure 8 As shown;

[0153] (3b4) Define the ORB feature point intersection and union ratio of the detection box between pedestrian joint points pose_box The calculation process is as follows: take the joint point as the center, construct a square with a side length of l as the detection frame of the joint point; calculate the intersection and union ratio of the feature points of the joint point detection frames of the two pedestrians corresponding to the joint points Where i = 0, 1, ..., 24; the calculated results of the feature point intersection and union ratio of the 25 joint point detection frames are sorted from large to small according to the confidence of the joint point detection results, and the corresponding elements with a confidence of 0 are directly eliminated; the average value of the feature point intersection and union ratio of the first num joint point detection frames after sorting is taken as the ORB feature point intersection and union ratio of the detection frame between the pedestrian joint points pose_box , the successfully matched ORB feature points in the detection frame of the pedestrian C1 in a certain frame to be matched and the joint point J0 of the pedestrian P1 in the candidate set (connected by a line in the two frames) Fig. 9 shown.

[0154] (3b5) Set the similarity S of pedestrian matching, and its formula is as follows:

[0155]

[0156] in,

[0157] w 1 、w 2 、w 3 、w 4 Feature IOU box , IOU pose_box 、ORB box 、ORB pose_box The weight of

[0158] (3c) According to the extracted pedestrian features, the pedestrians to be matched in the frame are matched with the pedestrians in the candidate set. For example, the Hungarian algorithm is used. Each pedestrian can be matched successfully or not. The Hungarian algorithm is used to match the pedestrians detected in the current frame with the pedestrians in the pedestrian candidate set according to the cost matrix. The pedestrians in the current frame are matched with the pedestrians in the pedestrian candidate set based on the matching principle of the maximum matching index S of all pedestrians.

[0159] Assume that there are M pedestrians in the current frame and N pedestrians in the pedestrian candidate set. The matching index between the pedestrian Ci in the current frame and the pedestrian Pj in the pedestrian candidate set is s ij , we can get the pedestrian matching index matrix S:

[0160]

[0161] Use k ij represents the matching status of the pedestrian Ci in the current frame and the pedestrian Pj in the pedestrian candidate set, and then the pedestrian matching status matrix K can be obtained. The matrix K also contains M rows and N columns. ij The assignment rules are as follows:

[0162]

[0163] The pedestrian matching problem is transformed into the following optimization problem:

[0164]

[0165]

[0166] By solving the above model through the Hungarian algorithm, the pedestrian matching state matrix K can be obtained, that is, the matching result between the pedestrian in the current frame and the pedestrian in the pedestrian candidate set can be obtained;

[0167] (3d) After the pedestrian matching of the frame to be matched is completed, the data in the pedestrian candidate set is updated according to the matching results, and pedestrians without matching results are added as new pedestrians to the candidate set, and the 6 sets of pedestrian information of the successfully matched pedestrians in the new frame are updated to the pedestrian candidate set, and pedestrians in the candidate set that have not been successfully matched for a long time are regarded as having left the picture area and removed from the candidate set, and then the next frame is matched, and steps (3a) to (3d) are repeated until all frames are matched;

[0168] (3e1) If the pedestrian in the pedestrian candidate set matches the pedestrian in the current frame successfully, the pedestrian information in the current frame is used to replace the corresponding pedestrian information in the pedestrian candidate set. For example, in the above example, pedestrian C1 in the current frame matches pedestrian P1 in the pedestrian candidate set successfully, then the pedestrian detection box position information, pedestrian detection box confidence, joint point position information and other attributes of pedestrian C1 are used to replace the corresponding attributes of pedestrian P1;

[0169] (3e2) If there is a pedestrian Px in the pedestrian candidate set that has not been successfully matched with the pedestrian in the current frame, then it can be divided into two cases: ① Pedestrian Px has not been successfully matched for a long time, for example, pedestrian Px has not updated its status for 100 frames or 5s in a row. At this time, the pedestrian may have left the camera's monitoring area, so the pedestrian should be removed from the pedestrian candidate set; ② Pedestrian Px has not been successfully matched for a short period of time, which may be due to the target detection algorithm. The pedestrian is just not detected in the current frame. At this time, the attributes of pedestrian Px should be kept unchanged;

[0170] (3e3) If there is a pedestrian Cx in the current frame that is not successfully matched with any pedestrian in the pedestrian candidate set, this may be because this pedestrian enters the camera's monitoring area for the first time and there is no information about this pedestrian in the pedestrian candidate set. Therefore, this pedestrian needs to be added to the pedestrian candidate set, and its pedestrian number is the maximum pedestrian number that has not been added to the previous pedestrian candidate set + 1.

[0171] Specifically, Fig.10 and Fig.11 As shown, the coverage time series algorithm described in step (4) may include the following steps:

[0172] (4a) Based on the skeleton data of the pedestrian, the rectangular area where the pedestrian is located is delineated, and the coverage between the pedestrian and the rectangular area where the gate is located is calculated. The specific calculation formula for coverage is as follows:

[0173]

[0174] Among them, the rectangular frame area A is the gate area, the rectangular frame area B is the pedestrian detection frame range, and the rectangular frame area A∩B is the intersection of the two, such as Fig.11 As shown in the "gate area" figure and the "coverage" figure;

[0175] (4b) Use time signal processing methods to process the time series of coverage, such as Fig.11 As shown in the “Coverage Time Series” figure, the positional relationship between pedestrians and gates is divided into two different states, namely, pedestrians inside the gate and pedestrians outside the gate, as shown in Fig.11 In the figure "Coverage after Binarization", set a threshold c th To binarize the time series curve of coverage, let c th =0.2. z is used to represent the positional relationship between the pedestrian and the gate. When z=1, the pedestrian is within the gate range; when z=0, the pedestrian is not within the gate range. The specific calculation method of z is as follows:

[0176]

[0177] (4c) Determine the time when the pedestrian exits the gate according to the change in the position relationship between the pedestrian and the gate, and further calculate the interval between pedestrians exiting the gate. Fig.11 The "Pedestrian Exit Gate Interval" graph shows the pedestrian exit situation at different times. The horizontal axis is time, and the vertical axis indicates whether there are pedestrians leaving the gate area at the corresponding time. 1 means someone is leaving the station at the corresponding time, and 0 means no one is leaving the station at the corresponding time. The point marked with a small box indicates that a pedestrian exited the station through the gate at 146s, that is, the time t when the pedestrian left the gate area out =146. Then the pedestrian exit interval T interval The calculation formula is as follows:

[0178] T interval =t out -t′ out

[0179] Among them, t out is the moment when the pedestrian leaves the gate area, t′ outThe time interval between two adjacent pedestrians leaving the same gate area indicates the moment when the previous pedestrian left the same gate area. This indicator can well distinguish between pedestrians who are fare-dodging and other normal pedestrians. Based on the test data in the experimental environment, the inventor found that the interval between the gates of the tailgaters was significantly lower than that of other pedestrians. The two types of pedestrians can be easily defined and divided, such as Fig.12 shown.

[0180] Preferably, the threshold value set in step (5) adopts different threshold values ​​according to the gate type, and the behavior of gate exit interval below the threshold is judged as tailgating fare evasion behavior, while the behavior above the threshold is judged as no tailgating fare evasion occurs.

[0181] Preferably, the coverage described in step (4a) adopts the relative intersection-union ratio, that is, the area of ​​the intersection area between pedestrians and gates divided by the area of ​​the pedestrian area. The calculation method of coverage is similar to the calculation method of intersection-union ratio, the difference is that the denominator of coverage is A, while the denominator of intersection-union ratio is A∪B.

[0182] Preferably, the time signal processing means described in step (4b) is a binarization algorithm, which converts the values ​​of pedestrian coverage changing over time, thereby extracting the positional relationship between the pedestrian and the gate into two states, namely, the pedestrian is inside the gate and the pedestrian is outside the gate.

[0183] The tailgating and fare evasion behaviors at the entrance of the ticket gate occur from time to time, threatening the safety of pedestrians and equipment and disrupting the normal operation order. The means of human monitoring will consume additional costs and cannot comprehensively and accurately identify the fare evasion behaviors. Based on the existing surveillance cameras, the tailgating and fare evasion behavior identification method proposed in the method of the present invention can realize all-round and all-time automatic detection by means of artificial intelligence, which is convenient for operation management personnel to perform further operations on fare evaders, curb the occurrence of such abnormal behaviors, and avoid the potential risks brought by them.

[0184] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0186] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0187] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0188] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A method for identifying tailgating fare evasion behavior, characterized in that: The following steps are involved: (1) Obtain the surveillance video of the gate entrance as input and convert it into image frames; (2) using a human body posture estimation algorithm to obtain human body posture data in the image frame; (3) Using the human posture tracking algorithm, the human posture data detected in each frame is matched to obtain the complete posture sequence of each pedestrian appearing in the surveillance video over time; (4) Using the coverage time series algorithm, the time interval between adjacent pedestrians leaving the gate is obtained; The coverage time series algorithm described in step (4) comprises the following specific steps: (4a) Based on the skeleton data of the pedestrian, the rectangular area where the pedestrian is located is delineated, and the coverage between the pedestrian and the rectangular area where the gate is located is calculated. The specific calculation formula for coverage is as follows: Among them, the rectangular frame area A is the gate area, the rectangular frame area B is the pedestrian detection frame range, and the rectangular frame area A∩B is the intersection of the two; (4b) Using time signal processing methods, process the time series of coverage in step (4a), and divide the position relationship between the pedestrian and the gate into two different states, namely, the pedestrian is inside the gate and the pedestrian is outside the gate, and set a threshold ct h To binarize the time series curve of coverage, let c th =0.2; z is used to represent the positional relationship between the pedestrian and the gate. When z=1, the pedestrian is within the gate range; when z=0, the pedestrian is not within the gate range. The specific calculation method of z is as follows: (4c) According to the change in the position relationship between the pedestrian and the gate in step (4b), the time point for the pedestrian to exit the gate is determined, and the exit interval between pedestrians is further calculated. The pedestrian exit interval T interval The calculation formula is as follows: T interval =t out -t′ out Among them, t out is the moment when the pedestrian leaves the gate area, t′ out Indicates the time when the previous pedestrian left the same gate area; (5) A threshold is set, and pedestrians whose exit distance from the gate is lower than the threshold are judged as tailgating fare evaders.

2. The method for identifying tailgating fare evasion behavior according to claim 1, characterized in that: The human posture estimation algorithm described in step (2) refers to the OpenPose algorithm, which uses the BODY_25 human skeleton model to detect the key points of the human body and connect the key points belonging to the same pedestrian to obtain the human skeleton data. The specific steps are as follows: (2a): The first stage network generates a set of detection confidence maps S 1 =ρ 1 (F) and a set of joint affine field maps where ρ 1 and They are the CNN structures of the first stage. The input of each subsequent stage comes from the prediction results of the previous stage and the original image features F, producing more accurate prediction results. (2b): According to the predicted confidence map, a discrete set of candidate joint points is obtained, and the formula is as follows: in, represents the position of the mth joint point of the jth joint point, where j represents the joint point category; (2c): Define the line set of all candidate joint points, the formula is as follows: in, Candidate joint points Represents the position of the mth joint point of the j1th joint point; Candidate joint points Represents the position of the nth joint point of the j2th joint point; Represents candidate joint points and candidate joint points Is there a wired connection between them? Indicates that the two are connected by wire. Indicates that there is no connection between the two; (2d): Consider the two candidate joint points j1 and j2 corresponding to the limb c separately, and solve the optimal problem of the graph matching method with the highest total affinity value through the Hungarian algorithm to obtain the optimal matching. The optimization problem with the maximum total affinity is as follows: in, E c represents affinity value, c represents each limb of human; E mn Represents candidate joint points and candidate joint points Affinity between (2e): When estimating the full body pose of multiple people, the K-point graph allocation problem is solved to assemble the links that share the same body parts into the full body pose of the human body. The formula is as follows: Among them, E represents the affinity value, c represents each human limb, and C represents the number of humans; (2f): Set rules to clean the obtained human skeleton data. The formula is as follows: in, Indicates the validity of the corresponding human skeleton data; Indicates the number of joint points with a confidence greater than 0.2 in the human pose estimation result of the i-th pedestrian in the t-th frame image; num is the joint point number threshold.

3. The method for identifying tailgating fare evasion behavior according to claim 1, characterized in that: The human posture tracking algorithm described in step (3) is the Pose Flow algorithm, which performs frame-by-frame matching of pedestrians in the picture. The specific steps are as follows: (3a) For a frame to be matched, based on the extracted human posture information and the data of the historical frames, a pedestrian candidate set is established to compare the similarity of features with the pedestrian to be matched, and the pedestrian posture information between the previous and next frames is matched. The pedestrian candidate set records the following six attributes of each pedestrian: the detection frame position, the detection frame position confidence, the joint point position, the joint point position confidence, the pedestrian number, and the time when the pedestrian last appeared in the picture; (3b) Based on human posture data, similarity features between pedestrians in the picture are extracted; Specifically, it includes four groups of features: the intersection-and-union ratio of detection frames between pedestrians, the intersection-and-union ratio of detection frames between pedestrian joints, the intersection-and-union ratio of ORB feature points of detection frames between pedestrian joints, and the intersection-and-union ratio of ORB feature points of detection frames between pedestrian joints. The intersection-and-union ratio is used to describe the overlap rate between the detection frames of two pedestrians or between rectangular detection frames centered on joints. The larger the intersection-and-union ratio, the higher the degree of overlap between the two pedestrians. ORB feature points are special feature points in an image, most of which are contour points. The Hamming distance is used when matching ORB feature points of two images. The smaller the Hamming distance, the higher the degree of matching of the feature points corresponding to the two descriptors. The intersection-and-union ratio of ORB feature points is the intersection-and-union ratio of the number of matching ORB feature points between pedestrian detection frames or joint point detection frames. All features will be weighted averaged to measure the similarity between the pedestrian to be matched and the pedestrians in the candidate set. (3c) According to the extracted pedestrian similarity features, the pedestrians to be matched in the frame are matched with the pedestrians in the candidate set. Each pedestrian can be matched successfully or not. The Hungarian algorithm is used to match the pedestrians detected in the current frame with the pedestrians in the pedestrian candidate set according to the cost matrix. The pedestrians in the current frame are matched with the pedestrians in the pedestrian candidate set based on the matching principle of the maximum matching index S of all pedestrians. Assume that there are M pedestrians in the current frame and N pedestrians in the pedestrian candidate set. The matching index between the pedestrian Ci in the current frame and the pedestrian Pj in the pedestrian candidate set is s ij , then we get the pedestrian matching index matrix S: Use k ij represents the matching status of the pedestrian Ci in the current frame and the pedestrian Pj in the pedestrian candidate set, and then obtains the pedestrian matching status matrix K, which also contains M rows and N columns; where k ij The assignment rules are as follows: The pedestrian matching problem is transformed into the following optimization problem: The above model is solved by the Hungarian algorithm to obtain the pedestrian matching state matrix K, that is, the matching result between the pedestrian in the current frame and the pedestrian in the pedestrian candidate set is obtained; (3d) After the pedestrian matching of the frame to be matched is completed, the data in the pedestrian candidate set is updated according to the matching results, and pedestrians without matching results are added as new pedestrians to the candidate set, and the 6 sets of pedestrian information of the successfully matched pedestrians in the new frame are updated to the pedestrian candidate set, and pedestrians in the candidate set that have not been successfully matched for a long time are regarded as having left the picture area and removed from the candidate set, and then the next frame is matched, and steps (3a) to (3d) are repeated until all frames are matched; There are three processing results: (3e1) If the pedestrian in the pedestrian candidate set matches the pedestrian in the current frame successfully, the pedestrian information in the current frame is used to replace the corresponding pedestrian information in the pedestrian candidate set; specifically, if the pedestrian C1 in the current frame matches the pedestrian P1 in the pedestrian candidate set successfully, the pedestrian detection box position information, pedestrian detection box confidence, and joint point position information attributes of the pedestrian C1 are used to replace the corresponding attributes of the pedestrian P1; (3e2) If there is a pedestrian Px in the pedestrian candidate set that has not been successfully matched with the pedestrian in the current frame, there are two cases: ① Pedestrian Px has not been successfully matched for a long time, and the pedestrian is removed from the pedestrian candidate set; ② Pedestrian Px has not been successfully matched for a short time, and the attributes of pedestrian Px are maintained unchanged; (3e3) If there is a pedestrian Cx in the current frame that has not been successfully matched with any pedestrian in the pedestrian candidate set, and this pedestrian enters the camera's monitoring area for the first time, the pedestrian is added to the pedestrian candidate set, and its pedestrian number is the maximum pedestrian number that has not been added to the previous pedestrian candidate set + 1.

4. The method for identifying tailgating fare evasion behavior according to claim 1, characterized in that: Step (3b) is implemented by the following algorithm: (3b1) Define the intersection-over-union (IOU) ratio between the detection boxes of pedestrians box , the formula is as follows: Among them, A and B are the areas where the two pedestrian detection frames are located; (3b2) Define the intersection-over-union (IOU) ratio of the detection boxes between the pedestrian joint points pose_box The specific calculation process is as follows: With the joint point as the center, a square with a side length of l is constructed as the detection frame of the joint point; the intersection and union ratio of the joint point detection frames of the two pedestrians corresponding to the joint points is calculated. Where i = 0, 1, ..., 24; the 25 joint point detection frame intersection and union ratio calculation results are sorted from large to small according to the confidence of the joint point detection results, and the corresponding elements with a confidence of 0 are directly eliminated; the average of the first num joint point detection frame intersection and union ratios after sorting is taken as the joint point detection frame intersection and union ratio IOU between pedestrian C1 and pedestrian P1 pose_box ; (3b3) Define the ORB feature point intersection between the detection boxes of pedestrians box , the formula is as follows: Among them, Kp A represents the set of ORB feature points in the detection box A, Kp B represents the set of ORB feature points in the detection box B, Kp A ∩Kp B Indicates the set of ORB feature points that are successfully matched between detection box A and detection box B, num(Kp A ∩Kp B ) represents the set Kp A ∩Kp B The number of feature points in ; (3b4) Define the ORB feature point intersection and union ratio of the detection box between pedestrian joint points pose_box The calculation process is as follows: take the joint point as the center, construct a square with a side length of l as the detection frame of the joint point; calculate the intersection and union ratio of the feature points of the joint point detection frames of the two pedestrians corresponding to the joint points Where i = 0, 1, ..., 24; the calculated results of the feature point intersection and union ratio of the 25 joint point detection frames are sorted from large to small according to the confidence of the joint point detection results, and the corresponding elements with a confidence of 0 are directly eliminated; the average value of the feature point intersection and union ratio of the first num joint point detection frames after sorting is used as the ORB feature point intersection and union ratio of the detection frame between the pedestrian joint points pose_box ; (3b5) Set the similarity S of pedestrian matching, and its formula is as follows: Among them, w1, w2, w3, and w4 are feature IOUs respectively. box , IOU pose_box 、ORB box 、ORB pose_box The weight of .

5. The method for identifying tailgating fare evasion behavior according to claim 1, characterized in that: The coverage described in step (4a) of the coverage time series algorithm adopts the relative intersection-union ratio, that is, the area of ​​the intersection of pedestrians and gates divided by the area of ​​the pedestrian area. The calculation method of coverage is similar to that of intersection-union ratio, the difference is that the denominator of coverage is A, while the denominator of intersection-union ratio is A∪B.

6. The method for identifying tailgating fare evasion behavior according to claim 1, characterized in that: The time signal processing method described in step (4b) of the coverage time series algorithm is a binarization algorithm, which converts the values ​​of pedestrian coverage that change over time, thereby extracting the positional relationship between the pedestrian and the gate into two states, namely, the pedestrian is inside the gate and the pedestrian is outside the gate.

7. The method for identifying tailgating fare evasion behavior according to claim 1, characterized in that: The threshold setting in step (5) uses different threshold values ​​according to the gate type, and the behavior of gate exit interval below the threshold is judged as tailgating fare evasion behavior, while the behavior above the threshold is judged as no tailgating fare evasion.

Images