Method and system for covert tailgating detection based on gaze and gait state behavior

Abstract

The application discloses a secret tailing detection method and system based on gaze and walking state behavior, and the method comprises the following steps: collecting a multi-person monitoring video dataset comprising real tailing behavior and ordinary walking behavior; extracting the walking trajectory of pedestrians in the multi-person monitoring video dataset, and obtaining a walking state behavior mode according to the walking trajectory of the pedestrians; obtaining a gaze behavior mode of the pedestrians; obtaining a secret tailing identification model; inputting the obtained walking state behavior mode and gaze behavior mode into the secret tailing identification model to obtain the identification result of real secret tailers and ordinary pedestrians. The secret tailing detection method based on gaze and walking state behavior provided by the application simultaneously considers the walking state behavior mode and the gaze behavior mode, accurately reveals secret tailers, is suitable for mining complex secret human behavior activities with unclear visual features, and provides an efficient information processing method for the public security field.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human behavior recognition, in particular to a secret tailing detection method and system based on gaze and walking state behavior. BACKGROUND

[0002] Human behavior recognition is currently a major research hotspot in the field of video understanding and has potential great value in monitoring security. Secret tailing is a special tailing behavior with characteristics such as secrecy and regularity, and it is essentially a behavior understanding. It is not an ongoing criminal activity, but a preparatory activity for many illegal and criminal acts that endanger national security. Therefore, timely detection of this secret abnormal behavior can provide early warning for a series of criminal and terrorist incidents that endanger national security, and plays an extremely important role in public security.

[0003] In recent years, most of the traditional tailing detection methods are based on trajectories, mainly judging whether a pedestrian is following a target object according to the trajectories. For example, in 2008, Andersson et al. of Lund University published a paper in GeoInformatica, which gave a natural definition of the "one object leading other objects" pattern, and developed a method to capture the spatio-temporal relationship between moving objects that constitute leadership. In 2013, Li et al. of Pennsylvania State University proposed a "following pattern" definition based on a local sequence minimizer in the ICDM conference, and used a local sequence alignment algorithm to detect the tailing period in the trajectory based on the definition. However, in crowded public places, many people are walking on the same path, and trajectory similarity occurs from time to time, making it difficult to accurately detect secret and truly dangerous tailing behavior. In order to solve this problem, in 2021, Xu et al. of Wuhan University first proposed a secret tailing detection framework using a spatio-temporal co-occurrence detection and attention model in the ACM MM conference, and used a model to mine the attention patterns of pedestrians to distinguish secret tailers. In 2022, Xu et al. of Wuhan University proposed using gaze patterns and distance patterns to identify secret tailers in the ACM MM conference. However, this method has the following limitations: 1) the threshold binary coarse-grained representation of attention behavior lacks a fine-grained expression of gaze angles, and cannot truly and effectively represent the complex gaze patterns of pedestrians. 2) Secret tailers try to make their speed and direction similar when tracking target pedestrians, so the speed state of pedestrians cannot comprehensively represent the changes in the walking state of pedestrians.

[0004] SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the above background art, and to provide a secret tailing detection method and system based on gaze and walking state behavior.

[0006] In a first aspect, the application provides a secret tailing detection method based on gaze and walking state behavior, comprising the following steps:

[0007] Collecting a multi-person monitoring video dataset including real tailing behavior and ordinary walking behavior;

[0008] Extracting the walking trajectory of pedestrians in the multi-person monitoring video dataset by a multi-target tracking algorithm, and obtaining a walking state behavior pattern according to the walking trajectory of the pedestrians;

[0009] Obtaining a gaze behavior pattern of the pedestrians from the multi-person monitoring video dataset by a gaze detection model;

[0010] Training a time series classification model by taking the walking state behavior pattern and the gaze behavior pattern of the real secret tailer and the ordinary pedestrian as positive samples and negative samples respectively, and obtaining a secret tailing recognition model;

[0011] Inputting the obtained walking state behavior pattern and gaze behavior pattern into the secret tailing recognition model to obtain the recognition result of the real secret tailer and the ordinary pedestrian.

[0012] According to the first aspect, in a first possible implementation manner of the first aspect, after the step S1 of collecting the multi-person monitoring video dataset including real tailing behavior and ordinary walking behavior, the method further comprises the following steps:

[0013] Converting the collected multi-person monitoring video dataset into an image under each frame.

[0014] According to the first possible implementation manner of the first aspect, in a second possible implementation manner of the first aspect, the step S2 of extracting the walking trajectory of pedestrians in the multi-person monitoring video dataset by a multi-target tracking algorithm, and obtaining a walking state behavior pattern according to the walking trajectory of the pedestrians, specifically comprises the following steps:

[0015] Using a target detection and tracking algorithm to extract the pixel coordinates of people in each frame from the converted image of the collected multi-person monitoring video dataset, converting the image into a top view, and calculating and obtaining the top two-dimensional position coordinates of the pedestrians;

[0016] Obtaining the expression of the walking state behavior pattern by calculating the top two-dimensional coordinates of the pedestrians.

[0017] According to the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, the step of obtaining the expression of the walking state behavior pattern by calculating the top two-dimensional coordinates of the pedestrians, specifically comprises the following steps:

[0018] The distance of each pair of pedestrians, the distance sequence of each pair of pedestrians, the distance range of each pair of pedestrians, the average distance of each pair of pedestrians, the relative speed between each pair of pedestrians, and the walking speed similarity of the two pedestrians in each pair of pedestrians are calculated through the two-dimensional position coordinates of the top view of the pedestrians.

[0019] According to the first aspect, in a first possible implementation manner of the first aspect, the step of obtaining the gaze behavior pattern of the pedestrian from the multi-person monitoring video dataset by the gaze detection model specifically includes the following steps:

[0020] Obtain the attention sequence of the multi-person monitoring video dataset;

[0021] According to the obtained attention sequence, obtain the attention estimation of the covert follower;

[0022] According to the obtained attention of the covert follower, calculate the attention estimation frequency and the attention estimation density, and take the obtained attention estimation frequency and the attention estimation density as the gaze behavior pattern expression of the pedestrian.

[0023] According to the fourth possible implementation manner of the first aspect, in a fifth possible implementation manner of the first aspect, the step of obtaining the attention sequence of the multi-person monitoring video dataset specifically includes the following steps:

[0024] Select the most suitable gaze pattern mapping function through experiments;

[0025] According to the selected mapping function, obtain the attention sequence of the multi-person monitoring video dataset.

[0026] According to the fifth possible implementation manner of the first aspect, in a sixth possible implementation manner of the first aspect, after the step of obtaining the attention sequence of the multi-person monitoring video dataset according to the selected mapping function, the method further includes the following steps:

[0027] Exponentially average the attention in the obtained attention sequence of the multi-person monitoring video dataset to obtain the exponentially averaged attention sequence of the multi-person monitoring video dataset.

[0028] According to the first aspect, in a seventh possible implementation manner of the first aspect, the step of taking the walking state behavior pattern and the gaze behavior pattern of the real covert follower and the ordinary pedestrian as positive samples and negative samples respectively to train the time sequence classification model to obtain the covert follower identification model specifically includes the following steps:

[0029] Concatenate the walking state behavior pattern and the gaze behavior pattern of the positive samples and the negative samples into a six-dimensional vector;

[0030] The concatenated six-dimensional vector is cut into sequences of the same length and input into a time series classification model for training to obtain the covert tailing identification model.

[0031] In a second aspect, the application provides a covert tailing detection system based on gaze and walking state behavior, comprising:

[0032] A video dataset acquisition module is configured to acquire a multi-person monitoring video dataset including real tailing behavior and normal walking behavior.

[0033] A walking state behavior pattern acquisition module is in communication connection with the video dataset acquisition module and is configured to extract a walking trajectory of a pedestrian in the multi-person monitoring video dataset by using a multi-target tracking algorithm and acquire a walking state behavior pattern according to the walking trajectory of the pedestrian.

[0034] A pedestrian gaze behavior pattern acquisition module is in communication connection with the video dataset acquisition module and is configured to acquire a gaze behavior pattern of a pedestrian in the multi-person monitoring video dataset by using a gaze detection model.

[0035] A covert tailing identification model acquisition module is in communication connection with the walking state behavior pattern acquisition module and the pedestrian gaze behavior pattern acquisition module and is configured to train a time series classification model by taking the walking state behavior pattern and the gaze behavior pattern of a real covert tailer and a normal pedestrian as positive samples and negative samples respectively and acquire a covert tailing identification model.

[0036] A tailing detection result acquisition module is in communication connection with the covert tailing identification model acquisition module and is configured to input the acquired walking state behavior pattern and gaze behavior pattern into the covert tailing identification model and acquire an identification result of the real covert tailer and the normal pedestrian.

[0037] In a first possible implementation manner of the second aspect, the application further comprises:

[0038] A frame image conversion module is in communication connection with the video dataset acquisition module and is configured to convert the collected multi-person monitoring video dataset into images under each frame.

[0039] Compared with the prior art, the application has the following advantages:

[0040] The covert tailing detection method based on gaze and walking state behavior provided by the application simultaneously considers the walking state behavior pattern and the gaze behavior pattern, accurately reveals a covert tailer and is suitable for mining complex human behavior activities with unclear visual features, thereby providing an efficient information processing method for the field of public security. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1is a method flowchart of a secret tailing detection method based on gaze and walking state behavior of an embodiment of the present application.

[0042] Figure 2 is a function module block diagram of a secret tailing detection system based on gaze and walking state behavior of an embodiment of the present application. DETAILED DESCRIPTION

[0043] Reference will now be made in detail to the specific embodiments of the present application, examples of which are illustrated in the accompanying drawings. While the present application will be described in conjunction with the specific embodiments, it will be understood that the present application is not intended to be limited to the described embodiments. On the contrary, the present application is intended to cover modifications, alterations, and equivalents, which are within the scope of the present application as defined by the appended claims. It should be noted that the method steps described herein can all be implemented by any functional block or functional arrangement, and any functional block or functional arrangement can be implemented as a physical entity or a logical entity, or a combination of both.

[0044] In order for those skilled in the art to better understand the present application, the present application will be further described in detail below in conjunction with the drawings and specific embodiments.

[0045] Note: The examples to be introduced next are only one specific example, and are not as a limitation on the embodiments of the present application must be as follows specific steps, values, conditions, data, order, etc. Those skilled in the art can use the concept of the present application to construct more embodiments not mentioned in the present specification by reading the present specification.

[0046] Most of the traditional tailing detection methods are simply based on pedestrian trajectory data for analysis, lack of fine expression of pedestrian gaze angle, cannot truly and effectively represent the complex gaze mode of pedestrians, do not consider the speed state of pedestrian shape, cannot fully represent the walking state change of pedestrians, and cannot accurately and efficiently identify secret tailers.

[0047] Therefore, the present application provides a secret tailing detection method based on gaze and walking state behavior, which is used to solve the technical problems of lack of fine expression of pedestrian gaze angle, inability to truly and effectively represent the complex gaze mode of pedestrians, inability to consider the speed state of pedestrian shape, inability to fully represent the walking state change of pedestrians, inability to accurately and efficiently identify secret tailers, and unsuitability for mining complex human behavior activities with non-obvious visual features.

[0048] In a first aspect, referring to Figure 1 The present application provides a secret tailing detection method based on gaze and walking state behavior, comprising the following steps:

[0049] Step S1, collect a multi-person monitoring video dataset including more than or equal to 2 people of real tailing behavior and ordinary walking behavior;

[0050] Step S2, extracting the walking trajectory of the pedestrian in the multi-person monitoring video dataset by a multi-target tracking algorithm, and obtaining the walking state behavior pattern according to the walking trajectory of the pedestrian;

[0051] Step S3, obtaining the gaze behavior pattern of the pedestrian from the multi-person monitoring video dataset by a gaze detection model;

[0052] Step S4, training a time sequence classification model by taking the walking state behavior pattern and the gaze behavior pattern of the real secret follower and the ordinary pedestrian as positive samples and negative samples respectively, obtaining a secret following recognition model, inputting the walking state behavior pattern and the gaze behavior pattern obtained into the secret following recognition model, and obtaining the recognition result of the real secret follower and the ordinary pedestrian.

[0053] The secret following detection method based on gaze and walking state behavior provided in the application simultaneously considers the walking state behavior pattern and the gaze behavior pattern, is suitable for mining complex secret human behavior activities with unclear visual features, accurately reveals secret followers, and provides an efficient information processing method for the public security field.

[0054] In an embodiment, after the step S1 of collecting the multi-person monitoring video dataset including real following behavior and ordinary walking behavior, the following steps are further included:

[0055] The gaze behavior pattern and the walking state pattern of the real secret follower with following intention in the multi-person monitoring video dataset of the mobile phone are taken as negative samples, the gaze behavior pattern and the walking state pattern of the ordinary pedestrian without following intention are taken as positive samples, and the collected multi-person monitoring video dataset is converted into an image under each frame.

[0056] In an embodiment, the step S2 of extracting the walking trajectory of the pedestrian in the multi-person monitoring video dataset by a multi-target tracking algorithm, and obtaining the walking state behavior pattern according to the walking trajectory of the pedestrian, specifically includes the following steps:

[0057] The pixel coordinates of the person in each frame are extracted from the converted image of the collected multi-person monitoring video dataset by using the TraDes model target detection and tracking algorithm, the image is converted into a top view, the top two-dimensional position coordinates of the pedestrian are calculated and obtained, and the walking state behavior pattern expression is calculated and obtained through the top two-dimensional coordinates of the pedestrian.

[0058] In an embodiment, the step S3 of calculating and obtaining the walking state behavior pattern expression through the top two-dimensional coordinates of the pedestrian, specifically includes the following steps:

[0059] The distance between each pair of pedestrians, the distance sequence of each pair of pedestrians, the distance range of each pair of pedestrians, the average distance of each pair of pedestrians, the relative speed between each pair of pedestrians, and the similarity of the walking speed between the two pedestrians of each pair of pedestrians are calculated by the top-down two-dimensional position coordinates of the pedestrians.

[0060] In a more specific embodiment, the step of calculating the distance between each pair of pedestrians, the distance sequence of each pair of pedestrians, the distance range of each pair of pedestrians, the average distance of each pair of pedestrians, the relative speed between each pair of pedestrians, and the similarity of the walking speed between the two pedestrians of each pair of pedestrians by the top-down two-dimensional position coordinates of the pedestrians is specifically implemented as:

[0061] The image is converted into a top-down view, and finally the top-down two-dimensional position coordinates of the pedestrians are calculated. Specifically, a perspective transformation matrix is calculated by corresponding the pixel coordinates of four points in the image to the ground coordinates in the top-down view, and then the pixel coordinates of the pedestrians are converted into top-down two-dimensional position coordinates by using the perspective transformation matrix where i represents the ith pedestrian, and t represents the time at which the current frame is located. Then the trajectory sequence of n pedestrians in a video is

[0062] The specific calculation formula of the walking state behavior pattern expression is as follows:

[0063] Given a video with a length of L seconds, it is divided into several video segments with a length of K seconds, and each segment contains RxK frames, where R represents the frame rate.

[0064] The distance between each pair of pedestrians is calculated by the top-down two-dimensional position coordinates of the pedestrians, and the distance sequence is obtained Distance range Average distance

[0065] The speed of the pedestrian P at time t can be expressed as

[0066] The speed of the pedestrian P is The speed of the pedestrian M is The size of the relative speed between the pedestrian P and the pedestrian M can be expressed as:

[0067]

[0068] The similarity of the speed of the pedestrian P and the pedestrian M can be expressed as:

[0069]

[0070] Where |·| represents the modulus of the vector.

[0071] In an embodiment, considering the complexity of the gaze state of the covert follower, the gaze behavior is converted into a continuously changing gaze spectrum, and the step of obtaining the gaze behavior pattern of the pedestrian from the multi-person monitoring video dataset by the gaze detection model specifically comprises the following steps:

[0072] Obtain the attention sequence of the multi-person monitoring video dataset;

[0073] According to the obtained attention sequence, obtain the attention estimation of the covert follower;

[0074] According to the obtained attention of the covert follower, calculate the attention estimation frequency and attention estimation density, and take the obtained attention estimation frequency and attention estimation density as the gaze behavior pattern of the pedestrian.

[0075] The present application proposes a method for estimating the continuous gaze degree of pedestrians in continuous time in monitoring video, which skillfully converts the gaze angle of pedestrians into continuously dynamic gaze state, can effectively describe the complex gaze behavior of pedestrians, and obtain the relevance of the covert follower and the followed target object in walking speed, providing more detailed data information basis for revealing the covert follower.

[0076] In an embodiment, the step of obtaining the attention sequence of the multi-person monitoring video dataset specifically comprises the following steps:

[0077] Detect the head box of the pedestrian from the multi-person monitoring video dataset by the DensePose model, then combine the head box with the pedestrian tracking result of the TraDeS model to obtain the position coordinates of the pedestrian, and detect the gaze direction of the pedestrian by the Gaze360 model;

[0078] According to the gaze direction, extract the attention sequence in the multi-person monitoring video dataset.

[0079] In an embodiment, the step of extracting the attention sequence in the multi-person monitoring video dataset according to the gaze direction specifically comprises the following steps:

[0080] Given a time period t, two pedestrians P and Q, g pq represent the frequency of P gazing at Q, and the gaze estimation g pq is calculated as follows.

[0081] Δx = x q -x p , Δy = y q -y p

[0082]

[0083]

[0084] where (x p , y p ) and (x q , y q ) are the positions of P and Q in the scene, is the two-dimensional gaze direction of P, φ represents the maximum attention angle deviation from the central gaze direction, and φ takes a value of 90. f(α) is a mapping function, which takes a value related to the current attention angle and the historical attention angle, and follows the rule that the smaller the attention angle, the closer f(α) is to 1. When the pedestrian Q is within the attention range (g pq ± φ), the pedestrian Q has a chance to be attended by P, and then the attention degree g pq of the pedestrian P to the pedestrian Q is f(α), otherwise, it cannot be attended by P, and thus g pq = 0.

[0085] In an embodiment, based on the principle that the closer the gaze angle is to a person, the greater the possibility of gazing at the person, a suitable mapping function is selected to convert the gaze angle into a continuous gaze degree estimation to effectively represent the complex and variable gaze behavior of pedestrians. The step of obtaining the attention degree sequence of the multi-person monitoring video data set specifically includes the following steps:

[0086] Selecting the most suitable gaze mode mapping function through experiments;

[0087] Obtaining the attention degree sequence of the multi-person monitoring video data set according to the selected mapping function.

[0088] In an embodiment, the step of selecting the most suitable gaze mode mapping function through experiments specifically includes the following steps:

[0089] Selecting four different mapping functions, which have different gaze state estimations in the gaze direction of the box:

[0090] y1 = cos(x),

[0091] Then, the most suitable gaze mode when a covert follower tracks a target pedestrian is selected using the experimental results.

[0092] In an embodiment, it is not very accurate to judge whether a covert follower P has an attention behavior according to a certain moment, and attention should be a continuous process, so an exponential average is added to associate the attention degree at a moment t with the historical attention degree. After the step of obtaining the attention degree sequence of the multi-person monitoring video data set according to the selected mapping function, the following steps are further included:

[0093] The attention of the attention sequence of the obtained multi-person monitoring video dataset is exponentially averaged as shown in the following formula, and the exponentially averaged attention sequence of the obtained multi-person monitoring video dataset is obtained:

[0094]

[0095] Wherein, β is a smoothing coefficient, and in the present work, it is set to 0.95.

[0096] In an embodiment, the step of obtaining the attention estimation frequency and the attention estimation density according to the obtained attention of the covert follower, and expressing the gazing behavior pattern of the pedestrian by the obtained attention estimation frequency and the attention estimation density, specifically comprises the following steps:

[0097] We calculate the frequency of attention estimation (Gf) and the density of attention estimation (Gd) to express the gazing behavior pattern of the pedestrian, and the specific calculation formula is as follows: Given a video of L seconds, divide it into several video segments of K seconds, and each segment contains RxK frames, where R represents the frame rate.

[0098]

[0099] In an embodiment, the step of obtaining the covert follower recognition model by training the time series classification model with the real covert follower and the walking state behavior pattern and the gazing behavior pattern of the ordinary pedestrian as positive and negative samples, respectively, specifically comprises the following steps:

[0100] The walking state behavior pattern and the gazing behavior pattern of the positive and negative samples are spliced into a six-dimensional vector;

[0101] The spliced six-dimensional vector is cut into sequences of the same length and substituted into the time series classification model for training to obtain the covert follower recognition classification model.

[0102] The covert follower detection method based on gazing and walking state behavior provided in the present application focuses on mining complex human behavior activities with inconspicuous visual features; and takes into account the complexity and variability of the gazing state of the covert follower, converts the gazing behavior into a continuously changing gazing spectrum; and finds the relevance of the walking speed of the covert follower and the target object. Specifically, the present application proposes a method for estimating the continuous gazing degree of pedestrians in continuous time under monitoring video, which ingeniously converts the gazing angle of pedestrians into a continuously and dynamically changing gazing state, which can effectively describe the complex gazing behavior of pedestrians. And the speed amplitude change and direction of the pedestrian are introduced to more deeply represent the walking state of the covert follower. The present application makes full use of the gazing and walking state features between pedestrians, effectively improves the accuracy of covert follower detection, and effectively reduces the detection error.

[0103] The applicant runs the method of the embodiment on a computer of Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz, 2080Ti GPU, and compares the method with a leading baseline method using a real scene hidden following data set Real-HFD, and the identification accuracy, recall rate and F1-score are greatly improved, so that the method can be applied to the field of hidden following detection.

[0104] In a second aspect, based on the same application concept, please refer to Figure 2 The application provides a hidden following detection system based on gaze and walking state behavior, which comprises a video data set acquisition module 100, a walking state behavior mode acquisition module 200, a pedestrian gaze behavior mode acquisition module 300, a hidden following identification model acquisition module 400 and a following detection result acquisition module 500. The video data set acquisition module 100 is used to collect a multi-person monitoring video data set comprising real following behavior and ordinary walking behavior. The walking state behavior mode acquisition module 200 is in communication connection with the video data set acquisition module 100, and is used to extract the walking trajectory of a pedestrian in the multi-person monitoring video data set through a multi-target tracking algorithm, and acquire a walking state behavior mode according to the walking trajectory of the pedestrian. The pedestrian gaze behavior mode acquisition module 300 is in communication connection with the video data set acquisition module 100, and is used to acquire the gaze behavior mode of the pedestrian from the multi-person monitoring video data set through a gaze detection model. The hidden following identification model acquisition module 400 is in communication connection with the walking state behavior mode acquisition module 200 and the pedestrian gaze behavior mode acquisition module 300, and is used to train a time sequence classification model by taking the walking state behavior mode and the gaze behavior mode of a real hidden follower and an ordinary pedestrian as positive samples and negative samples respectively, and acquire a hidden following identification model. The following detection result acquisition module 500 is in communication connection with the hidden following identification model acquisition module, and is used to input the acquired walking state behavior mode and gaze behavior mode into the hidden following identification model, and acquire the identification result of the real hidden follower and the ordinary pedestrian.

[0105] In an embodiment, the application provides a hidden following detection system based on gaze and walking state behavior, which further comprises a frame image conversion module in communication connection with the video data set acquisition module, and used to convert the collected multi-person monitoring video data set into an image under each frame.

[0106] Based on the same application concept, the embodiment of the application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize all method steps or part of the method steps of the above method.

[0107] The application implements all or part of the processes in the above method, and can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer readable storage medium, and the computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form. The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in a jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include an electrical carrier signal and a telecommunication signal.

[0108] Based on the same application concept, the embodiments of the application also provide an electronic device, including a memory and a processor, the memory stores a computer program running on the processor, and the processor implements all method steps or part of the method steps in the above method when executing the computer program.

[0109] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The processor is the control center of the computer device, and connects all parts of the computer device through various interfaces and lines.

[0110] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application program (such as a sound playing function, an image playing function, etc.) required by a function; and the data storage area can store data (such as audio data, video data, etc.) created according to the use of the mobile phone. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0111] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, a server or a computer program product. Therefore, the present application can adopt a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer usable program codes.

[0112] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), servers and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0113] These computer program instructions can also be stored in a computer readable memory that can guide the computer or other programmable data processing apparatus to work in a specific way, so that the instructions stored in the computer readable memory produce a product including instruction apparatus, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 an apparatus that carries out the functions specified in one or more flows and / or blocks.

[0114] These computer program instructions can also be loaded into a computer or other programmable data processing devices, so that a series of operational steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the functions specified in the flowchart Figure 1 one or more flows and / or blocks Figure 1 one or more blocks or steps of the functions specified in the flowchart

[0115] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for detecting covert tailing based on gaze and walking behavior, characterized in that, Includes the following steps: Collect a dataset of surveillance videos of multiple people, including real tailing behavior and ordinary walking behavior; The walking trajectories of pedestrians in a multi-person surveillance video dataset are extracted using a multi-object tracking algorithm, and walking status behavior patterns are obtained based on the pedestrians' walking trajectories. The gaze behavior patterns of pedestrians were obtained from a multi-person surveillance video dataset using a gaze detection model. The walking behavior patterns and gaze behavior patterns of real stalkers and ordinary pedestrians are used as positive and negative samples, respectively, to train a time series classification model and obtain a stalking identification model. The acquired walking behavior patterns and gaze behavior patterns are input into the covert tailing identification model to obtain the identification results of real covert tails and ordinary pedestrians. Following the step of collecting a dataset of multiple surveillance videos including actual tailing behavior and normal walking behavior, the following steps are also included: The collected dataset of multi-person surveillance videos is converted into images for each frame. The step of extracting pedestrian walking trajectories from a multi-person surveillance video dataset using a multi-target tracking algorithm and obtaining walking status behavior patterns based on these trajectories specifically includes the following steps: The pixel coordinates of people in each frame are extracted from the images after the collection of multi-person surveillance video dataset is transformed using object detection and tracking algorithms. The images are then converted into a top view, and the top-view two-dimensional position coordinates of the pedestrians are calculated. The pedestrian's walking state behavior pattern is obtained by calculating the pedestrian's top-down two-dimensional coordinates. The step of obtaining pedestrian gaze behavior patterns from a multi-person surveillance video dataset using a gaze detection model specifically includes the following steps: Obtain the attention sequence of a multi-person surveillance video dataset; Based on the obtained attention sequence, obtain the attention estimate of the hidden follower; Based on the attention levels of the surreptitious followers, the estimated frequency and density of attention levels are calculated, and these are used as expressions of the pedestrians' gaze behavior patterns.

2. The covert tailing detection method based on gaze and walking state behavior as described in claim 1, characterized in that, The step of calculating and obtaining the walking state behavior pattern expression through the pedestrian's top-down two-dimensional coordinates specifically includes the following steps: The distance between each pair of pedestrians, the distance sequence between each pair of pedestrians, the distance range between each pair of pedestrians, the average distance between each pair of pedestrians, the relative walking speed between each pair of pedestrians, and the similarity of walking speed between the two pedestrians in each pair of pedestrians are obtained by calculating the two-dimensional position coordinates of each pair of pedestrians from above.

3. The covert tailing detection method based on gaze and walking state behavior as described in claim 1, characterized in that, The step of obtaining the attention sequence of the multi-person monitoring video dataset specifically includes the following steps: The most suitable gaze pattern mapping function was selected through experiments; Based on the selected mapping function, obtain the attention sequence of the multi-person surveillance video dataset.

4. The covert tailing detection method based on gaze and walking state behavior as described in claim 3, characterized in that, After obtaining the attention sequence of the multi-person monitoring video dataset according to the selected mapping function, the following steps are also included: The attention scores in the attention score sequence of the acquired multi-person surveillance video dataset are exponentially averaged to obtain the attention score sequence of the multi-person surveillance video dataset after exponential averaging.

5. The covert tailing detection method based on gaze and walking state behavior as described in claim 1, characterized in that, The step of training a time-series classification model by using real stalker and ordinary pedestrian walking behavior patterns and gaze behavior patterns as positive and negative samples, respectively, to obtain a stalker identification model, specifically includes the following steps: The walking state behavior patterns and gaze behavior patterns of positive and negative samples are concatenated into a six-dimensional vector; The concatenated six-dimensional vector is trimmed into sequences of the same length and substituted into a time series classification model for training to obtain a hidden tail recognition classification model.

6. A covert tailing detection system based on gaze and walking behavior, characterized in that, include: The video dataset acquisition module is used to acquire multi-person surveillance video datasets from mobile phones, including real tailing behavior and ordinary walking behavior. The walking state behavior pattern acquisition module is communicatively connected to the video dataset acquisition module. It is used to extract the walking trajectory of pedestrians in the multi-person monitoring video dataset through a multi-target tracking algorithm, and to acquire the walking state behavior pattern based on the pedestrian's walking trajectory. The pedestrian gaze behavior pattern acquisition module is communicatively connected to the video dataset acquisition module and is used to acquire pedestrian gaze behavior patterns from a multi-person surveillance video dataset through a gaze detection model. The stealth tailing identification model acquisition module is communicatively connected to the walking state behavior pattern acquisition module and the pedestrian gaze behavior pattern acquisition module. It is used to train the time series classification model by using the walking state behavior patterns and gaze behavior patterns of real stealth tailers and ordinary pedestrians as positive and negative samples, respectively, to obtain the stealth tailing identification model. The tailing detection result acquisition module is communicatively connected to the covert tailing recognition model acquisition module. It is used to input the acquired walking state behavior pattern and gaze behavior pattern into the covert tailing recognition model to obtain the identification results of real covert tailers and ordinary pedestrians. The frame image conversion module is communicatively connected to the video dataset acquisition module and is used to convert the collected multi-person monitoring video dataset into images for each frame. The step of extracting pedestrian walking trajectories from a multi-person surveillance video dataset using a multi-target tracking algorithm and obtaining walking state behavior patterns based on the pedestrian walking trajectories further includes: The pixel coordinates of people in each frame are extracted from the images after the collection of multi-person surveillance video dataset is transformed using object detection and tracking algorithms. The images are then converted into a top view, and the top-view two-dimensional position coordinates of the pedestrians are calculated. The pedestrian's walking state behavior pattern is obtained by calculating the pedestrian's top-down two-dimensional coordinates. The step of obtaining pedestrian gaze behavior patterns from a multi-person surveillance video dataset using a gaze detection model further includes: Obtain the attention sequence of a multi-person surveillance video dataset; Based on the obtained attention sequence, obtain the attention estimate of the hidden follower; Based on the attention levels of the surreptitious followers, the estimated frequency and density of attention levels are calculated, and these are used as expressions of the pedestrians' gaze behavior patterns.

Images