Pedestrian loitering detection method, apparatus, device, and medium

Abstract

The embodiment of the application provides a kind of pedestrian loitering detection method, device, equipment and medium, and the method specifically includes: obtaining the video stream corresponding to video area;The video stream is tracked to obtain the track contained in the video stream;The track corresponds to track feature;The track feature includes: pedestrian position feature and pedestrian identity feature;According to the pedestrian identity feature, the track contained in the video stream is associated to corresponding pedestrian;According to the pedestrian corresponding pedestrian position feature, the track distribution information corresponding to pedestrian is determined;According to the track distribution information, it is judged whether the pedestrian is candidate loitering target.The embodiment of the application can improve the accuracy of pedestrian loitering detection.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device, and medium for detecting pedestrian loitering. Background Technology

[0002] Pedestrian loitering detection technology is an important technology in the field of computer vision. When a pedestrian continuously and repeatedly moves within a video area, they can be considered to be in a loitering state. Pedestrian loitering detection technology can detect pedestrian behavior within a video area and issue timely warnings. Pedestrian loitering detection technology can be applied in many application scenarios such as ponds, reservoirs, and banks.

[0003] Current pedestrian loitering detection methods typically perform location-based matching and tracking of targets appearing within a video area, record the duration of the tracked targets, and select targets whose duration exceeds a set time threshold as candidate loitering targets.

[0004] In practical applications, setting time thresholds is easily cracked. If the time thresholds are cracked, it will give criminals an opportunity to exploit the system, thereby affecting the accuracy of pedestrian loitering detection. Summary of the Invention

[0005] This application provides a method for detecting pedestrian loitering, which can improve the accuracy of pedestrian loitering detection.

[0006] Accordingly, embodiments of this application also provide a pedestrian loitering detection device, an electronic device, and a machine-readable medium to ensure the implementation and application of the above methods.

[0007] To address the aforementioned problems, this application discloses a pedestrian loitering detection method, the method comprising:

[0008] Get the video stream corresponding to the video region;

[0009] Pedestrian tracking is performed on the video stream to obtain the trajectories contained in the video stream; the trajectories correspond to trajectory features; the trajectory features include: pedestrian position features and pedestrian identity features;

[0010] Based on the pedestrian's identity characteristics, the trajectory contained in the video stream is associated with the corresponding pedestrian;

[0011] Based on the pedestrian location characteristics, determine the trajectory distribution information of the pedestrian;

[0012] Based on the trajectory distribution information, determine whether the pedestrian is a candidate loitering target.

[0013] To address the aforementioned problems, this application discloses a pedestrian loitering detection device, the device comprising:

[0014] The video stream acquisition module is used to acquire the video stream corresponding to a video region;

[0015] A pedestrian tracking module is used to track pedestrians in the video stream to obtain trajectories contained in the video stream; the trajectories correspond to trajectory features; the trajectory features include: pedestrian position features and pedestrian identity features;

[0016] The trajectory-pedestrian association module is used to associate the trajectory contained in the video stream with the corresponding pedestrian based on the pedestrian's identity characteristics;

[0017] The trajectory distribution information determination module is used to determine the trajectory distribution information of a pedestrian based on the pedestrian's location characteristics.

[0018] The judgment module is used to determine whether the pedestrian is a candidate loitering target based on the trajectory distribution information.

[0019] Optionally, the trajectory distribution information determination module includes:

[0020] A mapping module is used to map the pedestrian's location features onto a preset grid area to obtain the trajectory points of the pedestrian in the preset grid area; the preset grid area includes n×n grids;

[0021] The entropy information determination module is used to determine the entropy information of the pedestrian trajectory in the preset network area based on the number of times the trajectory point corresponding to the pedestrian appears in the grid, and use it as the trajectory distribution information corresponding to the pedestrian.

[0022] Optionally, the entropy information determination module includes:

[0023] The probability of occurrence determination module is used to determine the probability of occurrence of a trajectory point in a grid based on the number of times the trajectory point corresponding to a pedestrian trajectory appears in the grid.

[0024] The single-trajectory entropy information determination module is used to determine the entropy information corresponding to a pedestrian trajectory based on the probability of occurrence of trajectory points in multiple grids.

[0025] The first weighted average module is used to perform a weighted average of the entropy information corresponding to multiple pedestrian trajectories to obtain the entropy information of the pedestrian trajectory in the preset network area.

[0026] Optionally, the judgment module is specifically used to determine whether the pedestrian is a candidate loitering target based on the trajectory distribution information, the repeated appearance information of the pedestrian in the image acquisition device, and / or the duration information of the pedestrian trajectory.

[0027] Optionally, the device further includes:

[0028] The recurrence information determination module is used to determine the recurrence information of a pedestrian in multiple image acquisition devices based on the number of times the pedestrian trajectory appears in each of these devices; and / or

[0029] The duration information determination module is used to determine the duration information of the pedestrian trajectory based on the start and end detection points corresponding to the pedestrian trajectory.

[0030] Optionally, the determination module includes:

[0031] The second weighted average module is used to perform a weighted average of the trajectory distribution information, the information on the repeated appearance of pedestrians in the image acquisition device, and / or the duration information of the pedestrian trajectory, so as to obtain the pedestrian's wandering score.

[0032] The sorting module is used to sort multiple pedestrians according to their respective wandering scores;

[0033] The judgment module based on the sorting result is used to determine whether the pedestrian is a candidate loitering target based on the sorting result.

[0034] Optionally, the trajectory pedestrian association module includes:

[0035] The first creation module is used to create a new pedestrian and associate a first trajectory with the pedestrian.

[0036] A matching degree determination module is used to determine the matching degree between a first trajectory and a second trajectory; wherein the second trajectory is different from the first trajectory;

[0037] The association module is used to associate a second trajectory that matches a preset condition with the pedestrian.

[0038] Optionally, the device further includes:

[0039] Add a module to add a track contained in at least one video stream to the track library;

[0040] The trajectory pedestrian association module includes:

[0041] A construction module is used to build a pedestrian database; the pedestrian database contains at least one pedestrian.

[0042] The query module is used to query whether there is a first pedestrian in the pedestrian database that matches the first trajectory in the trajectory database. If there is, the first trajectory is associated with the first pedestrian.

[0043] The second new module is used to create a second pedestrian if there is no matching first pedestrian and associate the first trajectory with the second pedestrian.

[0044] Optionally, the pedestrian location features include: location features corresponding to the pedestrian bounding box in the trajectory.

[0045] Optionally, the pedestrian identity features include: pedestrian identity features of the image corresponding to the pedestrian bounding box in the trajectory.

[0046] This application also discloses an electronic device, including: a processor; and a memory storing executable code thereon, which, when executed, causes the processor to perform the method described in this application.

[0047] This application also discloses a machine-readable medium storing executable code thereon, which, when executed, causes a processor to perform the method described in this application.

[0048] The embodiments of this application have the following advantages:

[0049] In the technical solution of this application embodiment, the trajectory distribution information corresponding to pedestrians can characterize the distribution information of pedestrian trajectories, such as walking or standing still. Based on this trajectory distribution information, this application embodiment determines whether the pedestrian is a candidate loitering target. Because the trajectory distribution information corresponding to pedestrians has high flexibility and the ability to perceive actual scenes, the trajectory distribution information can change with changes in the actual scene; that is, different actual scenes can correspond to different trajectory distribution information. Thus, the trajectory distribution information of this application embodiment is not easily cracked. Therefore, this application embodiment can, to a certain extent, avoid the problem of trajectory distribution information being cracked affecting the accuracy of pedestrian loitering detection, thereby improving the accuracy of pedestrian loitering detection. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the steps of a pedestrian loitering detection method according to an embodiment of this application;

[0051] Figures 2(a) and 2(b) are schematic diagrams of pedestrian trajectories in a preset grid area according to an embodiment of this application.

[0052] Figure 3 This is a flowchart illustrating the steps of a pedestrian loitering detection method according to an embodiment of this application;

[0053] Figure 4 This is a schematic diagram of the structure of a pedestrian loitering detection device according to an embodiment of this application;

[0054] Figure 5This is a schematic diagram of the structure of an apparatus provided in one embodiment of this application. Detailed Implementation

[0055] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] The embodiments of this application can be applied to intelligent security scenarios to detect pedestrians who may be loitering. Loitering is a suspicious behavior that often leads to criminal acts such as pickpocketing and terrorist attacks; therefore, intelligent security scenarios have a certain need for the identification of loitering behavior.

[0057] Current pedestrian loitering detection methods typically record the duration of the tracked target's presence and identify targets whose presence time exceeds a set time threshold as candidate loitering targets. However, this time threshold is easily cracked. If the time threshold is compromised, it will provide opportunities for malicious actors, thereby affecting the accuracy of pedestrian loitering detection.

[0058] To address the technical problem that the accuracy of pedestrian loitering detection is affected by the cracking of the set time threshold, this application provides a pedestrian loitering detection method. The method specifically includes: acquiring a video stream corresponding to a video region; performing pedestrian tracking on the video stream to obtain trajectories contained in the video stream; each trajectory has corresponding trajectory features; these trajectory features specifically include pedestrian position features and pedestrian identity features; associating the trajectories contained in the video stream with the corresponding pedestrians based on the pedestrian identity features; determining the trajectory distribution information corresponding to the pedestrian based on the pedestrian position features; and determining whether the pedestrian is a candidate loitering target based on the trajectory distribution information.

[0059] In this embodiment, the trajectory distribution information corresponding to a pedestrian can characterize the distribution information of pedestrian trajectories, such as walking or standing still. Based on this trajectory distribution information, this embodiment determines whether the pedestrian is a candidate loitering target. Because the trajectory distribution information corresponding to a pedestrian has high flexibility and the ability to perceive real-world scenarios, the trajectory distribution information can change with the actual scenario; that is, different actual scenarios can correspond to different trajectory distribution information. Thus, the trajectory distribution information in this embodiment is difficult to crack. Therefore, this embodiment can, to a certain extent, avoid the problem of trajectory distribution information being cracked affecting the accuracy of pedestrian loitering detection, thereby improving the accuracy of pedestrian loitering detection.

[0060] Method Example 1

[0061] refer to Figure 1 The diagram illustrates a step-by-step flowchart of a pedestrian loitering detection method according to an embodiment of this application. The method may specifically include the following steps:

[0062] Step 101: Obtain the video stream corresponding to the video region;

[0063] Step 102: Perform pedestrian tracking on the video stream to obtain the trajectory contained in the video stream; the trajectory may correspond to trajectory features; the trajectory features may specifically include: pedestrian position features and pedestrian identity features;

[0064] Step 103: Based on the pedestrian's identity characteristics, associate the trajectory contained in the video stream with the corresponding pedestrian;

[0065] Step 104: Determine the trajectory distribution information of the pedestrians based on their location characteristics;

[0066] Step 105: Based on the trajectory distribution information, determine whether the pedestrian is a candidate loitering target.

[0067] In step 101, the video stream can be acquired by an image acquisition device such as a camera. Figure 1 The method shown can be performed by a pedestrian loitering detection device. The video region can characterize the area captured by the image acquisition device.

[0068] The pedestrian loitering detection device can be located at the acquisition end of the image acquisition device, or the image acquisition device can be located at the video decoding end. The acquisition end can encode the acquired video stream to obtain a video bitstream. The video decoding end can decode the video bitstream to obtain a video stream. A video stream can include multiple frames in formats such as RGB (Red, Green, Blue).

[0069] The video stream set in this application embodiment can be denoted as C = {c(1), c(2), ..., c(k) ..., c(n)}, that is, the video stream set includes n video streams, where k ∈ {1, 2, 3, ..., n}, and k and n can both be positive integers.

[0070] In step 102, a pedestrian tracking method can be used to track pedestrians in the video stream in order to obtain the trajectories contained in the video stream.

[0071] This application does not limit the specific pedestrian tracking method.

[0072] For example, one pedestrian tracking method uses two separate models. First, the detection model locates the bounding boxes of targets such as pedestrians in the frame image. Then, the association model extracts pedestrian identity features such as Re-ID (Re-identification) features from each bounding box and associates the bounding boxes with an existing tracking result based on the pedestrian identity features.

[0073] For example, one pedestrian tracking method performs both target detection and pedestrian identity feature extraction. The core idea is to complete target detection and identity embedding functions simultaneously in a single network, thereby reducing inference time by sharing most of the computation.

[0074] In practical implementation, the frame images contained in the video stream can first be fed into an encoder-decoder network to extract high-resolution feature maps. Then, the high-resolution feature maps are input into a parallel first detection head and a second detection head. The first detection head is used to predict pedestrian bounding boxes, and the second detection head is used to predict pedestrian identity features. Next, pedestrian bounding boxes can be temporally concatenated based on the pedestrian identity features. The aforementioned high-resolution feature maps allow for better alignment between pedestrian identity features and pedestrian bounding boxes. The parallel first and second detection heads reduce computation time and improve the robustness of feature matching.

[0075] In practical applications, pedestrian tracking methods can track pedestrians in a video stream at preset time intervals. The preset time interval can be determined by those skilled in the art based on actual application requirements. The number of detection points can be determined based on the preset time interval. Specifically, the number of detection points can be determined based on the ratio of the video stream duration to the preset time interval. An example of a preset time interval could be 0.1 seconds; in the case of a 100-second video stream, the number of detection points could be 1000. The detection result corresponding to one detection point includes: the presence of a pedestrian or the absence of a pedestrian.

[0076] The trajectories contained in the video stream can be included in a trajectory set. The trajectory set tc = {t(1), t(2), ..., t(j) ...}, j ∈ {1, 2, 3, ..., ∞}, where t(j) represents a trajectory, and t(j) = {(box(1), fea(1), c), (box(2), fea(2), c) ... (box(i), fea(i), c) ...}, i ∈ {1, 2, 3, ..., ∞}. box(i) represents the corresponding pedestrian bounding box detected at the i-th detection point of this trajectory, with i ordered chronologically. box(i) = (x1, y1, x2, y2), where (x1, y1) represents the coordinates of the upper left vertex of the bounding box, and (x2, y2) represents the coordinates of the lower right vertex. fea(i) represents the pedestrian identity features of the frame image corresponding to this pedestrian bounding box. Pedestrian identity features can be determined using pedestrian tracking methods. Pedestrian identity features are used to distinguish different pedestrians. Pedestrian identity features can be vectors of 512 dimensions or higher.

[0077] In practical applications, pedestrian location features can be the location features corresponding to the pedestrian bounding box in the trajectory. These location features can be the positional information of the pedestrian bounding box within the frame image. Assuming the frame image has a width of W and a height of H, the coordinates of the top-left vertex of the frame image can be (0, 0), and the coordinates of the bottom-right vertex of the frame image can be (W, H). box(i) = (x1, y1, x2, y2) can be determined based on the coordinate range corresponding to the frame image.

[0078] For each box(i) in a trajectory t(j), the pedestrian position feature v(i) can be determined based on the center point of the pedestrian bounding box, and box(i) can be replaced with v(i).

[0079] For a trajectory t(j), the mean of all fea(i) can be taken as the pedestrian identity feature fea_avg. Therefore, the trajectory feature corresponding to a trajectory t(j) can be denoted as (v, fea_avg, c), where v = {v(1), v(2)……v(i)……}, i∈{1,2,3.......∞}, v(i) represents the pedestrian position feature corresponding to the i-th detection point, fea_avg represents the pedestrian identity feature, and c represents the identifier of the video stream in which the trajectory is located.

[0080] In a specific implementation, trajectories can be added to a trajectory library T. Optionally, trajectories in a terminated state can be added to the trajectory library. For a trajectory, whether it is in a terminated state can be determined based on the detection results of its subsequent X detection points. X can be a positive integer. The subsequent X detection points can be detection points arranged in chronological order. For example, if the detection results of the subsequent X detection points are all "not containing pedestrians," then the trajectory can be determined to be in a terminated state. Optionally, trajectories whose trajectory points exceed a first threshold can be added to the trajectory library. Here, trajectory points can be detection points contained in the trajectory whose detection results indicate the presence of pedestrians. The first threshold can be determined by those skilled in the art based on actual application requirements; for example, the first threshold can be a positive integer such as 5.

[0081] In step 103, based on the pedestrian's identity characteristics, the trajectories contained in the video stream are associated with the corresponding pedestrians, so that a pedestrian contains one or more trajectories.

[0082] In practical applications, for each trajectory in the trajectory library T, the corresponding pedestrian can be identified.

[0083] In one implementation, the process of associating the trajectory contained in the video stream with the corresponding pedestrian based on the pedestrian's identity characteristics may specifically include: creating a new pedestrian and associating a first trajectory with the pedestrian; determining the matching degree between the first trajectory and a second trajectory; wherein the second trajectory is different from the first trajectory; and associating the second trajectory with a matching degree that meets a preset condition with the pedestrian.

[0084] In this model, both the first and second trajectories can be trajectories unassociated with pedestrians. The matching degree between the first and second trajectories can be the matching degree between the first pedestrian identity features corresponding to the first trajectory and the second pedestrian identity features corresponding to the second trajectory. When the first and second pedestrian identity features are vectors, the matching degree can be a measure such as the cosine similarity between vectors. The matching degree meets the preset conditions, such as the cosine similarity matching degree exceeding a second threshold.

[0085] In another implementation, at least one trajectory contained in a video stream can be added to a trajectory library. The process of associating the trajectory in the video stream with the corresponding pedestrian based on the pedestrian's identity characteristics can specifically include: constructing a pedestrian library; the pedestrian library contains at least one pedestrian; for a first trajectory in the trajectory library, querying whether a matching first pedestrian exists in the pedestrian library; if so, associating the first trajectory with the first pedestrian; if no matching first pedestrian exists, creating a second pedestrian and associating the first trajectory with the second pedestrian.

[0086] The process of querying whether there is a first pedestrian matching the first trajectory in the pedestrian database can be as follows: determine the matching degree between the identity features of the first pedestrian corresponding to the first trajectory and the identity features of the third pedestrian corresponding to the third trajectory in the pedestrian database. If the matching degree exceeds the second threshold, the first trajectory and the third trajectory can be considered to match.

[0087] After associating the first trajectory with the second pedestrian, the matching degree between the first trajectory and the second trajectory can be determined, and the second trajectory that meets the preset matching conditions can be associated with the second pedestrian.

[0088] In practical applications, the embodiments of this application can obtain a real-time pedestrian database P = {p(1), p(2), ..., p(i) ...}, i ∈ {1, 2, 3.......}, where p(i) represents any pedestrian, p(i) = {t(1), t(2), ..., t(j) ...}, j ∈ {1, 2, 3.......}, where t(j) is any trajectory generated by p(i), t(j) = (v, fea_avg, c), where v = {v(1), v(2), ..., v(i) ...}, v(i) is the pedestrian position coordinates corresponding to each detection point, fea_avg is the pedestrian identity feature, and c is the identifier of the video stream where the trajectory is located.

[0089] In step 104, the trajectory distribution information corresponding to pedestrians can characterize the distribution information of pedestrian trajectories, such as walking or standing still. A pedestrian trajectory can refer to one or more trajectories contained within a single pedestrian.

[0090] To facilitate the determination of pedestrian trajectory distribution information, the pedestrian location features can be mapped to a preset grid area to obtain the trajectory points of the pedestrian in the preset grid area. The preset grid area may include n×n grids, where n can be a positive integer, and the value of n can be determined by those skilled in the art according to actual application requirements. n can be an even number greater than or equal to 12, for example, n can be 12, 16, etc.

[0091] In practical applications, the region corresponding to the frame image can be divided to obtain a preset grid region. The grid coordinates of the upper left corner of the preset grid region can be (0,0), the grid coordinates of the lower right corner can be (n,n), and so on for the other grids. The pedestrian position coordinates v(i) corresponding to the original pixel of the frame image can be updated to the corresponding grid coordinates based on which grid the pedestrian position coordinates v(i) fall into. Alternatively, the coordinate range corresponding to the original pixel of the frame image can be mapped to the coordinate range corresponding to the preset grid region. That is, the coordinates (0,0) of the original pixel of the frame image can be mapped to (0,0) of the preset grid region, and the coordinates (W, H) of the original pixel of the frame image can be mapped to (n,n) of the preset grid region. Similarly, mapping can be performed for other original pixels.

[0092] Therefore, the process of determining the trajectory distribution information corresponding to a pedestrian in this embodiment of the application may specifically include: mapping the pedestrian's location features to a preset grid area to obtain the trajectory point corresponding to the pedestrian in the preset grid area; the preset grid area specifically includes n×n grids; and determining the entropy information of the pedestrian's trajectory in the preset grid area based on the number of times the trajectory point corresponding to the pedestrian appears in the grid, as the trajectory distribution information corresponding to the pedestrian.

[0093] The entropy information of the pedestrian trajectory within the preset network area can describe the amount of movement of the pedestrian in the video area. Generally speaking, if a pedestrian stays in one position for a long time, the entropy information value will decrease; or, if the pedestrian continues to move, the entropy information value will increase.

[0094] The process of determining the entropy information of a pedestrian trajectory in the preset network area may specifically include: determining the probability of a trajectory point appearing in a grid based on the number of times the trajectory point corresponding to a pedestrian trajectory appears in the grid; determining the entropy information corresponding to a pedestrian trajectory based on the probability of the trajectory points appearing in multiple grids; and performing a weighted average of the entropy information corresponding to multiple pedestrian trajectories to obtain the entropy information of the pedestrian trajectory in the preset network area.

[0095] Referring to Figures 2(a) and 2(b), schematic diagrams of pedestrian trajectories in a preset grid area according to an embodiment of this application are shown respectively. In these diagrams, grids containing numbers correspond to trajectory points, and the numbers in the grid represent the number of times a trajectory point appears in the network. Assuming a pedestrian trajectory contains F detection points, and the number of times a trajectory point appears in a network is fb, then the probability of the trajectory appearing in this grid is: pb = fb / F. The entropy information value of a pedestrian trajectory in the preset network area can be expressed as:

[0096]

[0097] Where pb(i) represents fb corresponding to the i-th grid, and he represents the amount of movement of the pedestrian in the video area. Generally speaking, if the pedestrian moves in the video area, it will increase the entropy information value; or, if the pedestrian stands still in the video area, it will decrease the entropy information value.

[0098] When a pedestrian corresponds to a pedestrian trajectory, the entropy information corresponding to a pedestrian trajectory can be used as the entropy information of the pedestrian trajectory in the preset network area.

[0099] When a pedestrian corresponds to multiple pedestrian trajectories, the entropy information corresponding to the multiple pedestrian trajectories can be weighted and averaged to obtain the entropy information of the pedestrian trajectory in the preset network area.

[0100] For example, for any pedestrian p, assuming there are m pedestrian trajectories and F(j) detection points for each trajectory, the weight of each trajectory can be expressed as:

[0101]

[0102] The process of weighted averaging of entropy information corresponding to multiple pedestrian trajectories can be represented as:

[0103]

[0104] Where he(j) represents the entropy information value corresponding to the j-th pedestrian trajectory.

[0105] In step 105, the trajectory distribution information can be used as a basis for determining whether a pedestrian is a candidate loitering target.

[0106] Figures 2(a) and 2(b) show pedestrian trajectories corresponding to different behaviors. In Figure 2(b), the trajectory points appear 21, 22, or 25 times in some grid cells, while in Figure 2(a), the maximum number of occurrences is 5. Furthermore, based on entropy calculations, the entropy value of Figure 2(b) is greater than that of Figure 2(a), therefore the pedestrian in Figure 2(b) can be identified as a candidate loitering target. In practical applications, the pedestrian in Figure 2(a) simply passes through the video area; therefore, the pedestrian in Figure 2(a) is not within the range of candidate loitering targets.

[0107] In determining whether a pedestrian is a candidate loitering target, in addition to using trajectory distribution information, other information can also be used, such as the recurrence information of the pedestrian in the image acquisition device and / or the duration information of the pedestrian trajectory.

[0108] Accordingly, the process of determining whether the pedestrian is a candidate loitering target may specifically include: determining whether the pedestrian is a candidate loitering target based on the trajectory distribution information, the repeated appearance information of the pedestrian in the image acquisition device, and / or the duration information of the pedestrian trajectory.

[0109] The process of determining the recurrence information may include: determining the recurrence information of the pedestrian in the image acquisition devices based on the number of times the pedestrian trajectory appears in each of the multiple image acquisition devices.

[0110] Recurrence Information (RAS) is used to describe how much a pedestrian lingers in the image capture device. Generally, if a pedestrian lingers in the video area, they will typically appear repeatedly in multiple cameras; or, if a pedestrian passes through the video area, they will typically appear in one camera.

[0111] Assuming the number of image acquisition devices corresponding to the video area is k, and assuming that for any pedestrian p, for any camera, pedestrian p has a possible trajectories, then RA can represent the occurrence information corresponding to one image acquisition device. When a is greater than 0, RA = 2. a-1 And when a = 0, RA = 0. Assuming RA(i) represents the occurrence information corresponding to the i-th camera, then the repeated occurrence information RAS can be represented as:

[0112]

[0113] The process of determining duration information may include: determining the duration information of the pedestrian trajectory based on the start and end detection points corresponding to the pedestrian trajectory.

[0114] The starting detection point can be the first detection point in the pedestrian trajectory, and the ending detection point can be the last detection point in the pedestrian trajectory. Therefore, the duration information of a pedestrian trajectory can be the time interval between the last detection point and the first detection point. Assuming the first detection point is numbered 1 and the last detection point is numbered M, the time interval can be (M-1) or the product of M and a preset time interval.

[0115] The duration information DS corresponding to a pedestrian can be represented as:

[0116]

[0117] Where m is the number of pedestrian trajectories, and D(j) represents the duration information of the j-th pedestrian trajectory.

[0118] In a specific implementation, the process of determining whether a pedestrian is a candidate loitering target based on the trajectory distribution information, the repeated appearance information of the pedestrian in the image acquisition device, and / or the duration information of the pedestrian trajectory can specifically include: performing a weighted average of the trajectory distribution information, the repeated appearance information of the pedestrian in the image acquisition device, and / or the duration information of the pedestrian trajectory to obtain a loitering score corresponding to the pedestrian; sorting multiple pedestrians according to the loitering score corresponding to the pedestrian; and determining whether the pedestrian is a candidate loitering target based on the sorting result.

[0119] The wandering fraction can be expressed as:

[0120] LS = a*RAS + b*WES + c*DS

[0121]

[0122] The determination process of LS differs for different loitering detection scenarios because loitering detection scenarios have different purposes and different contexts. Therefore, the weights a, b, and c can have different values ​​in different application scenarios.

[0123] In practical applications, the wandering score LS of all pedestrians in the pedestrian database P can be calculated in real time. Pedestrians are then sorted according to their wandering scores from highest to lowest LS, and the top N pedestrians with the highest wandering scores are selected as candidate wandering targets. The top N wandering scores can be the N highest-ranking pedestrians in descending order of their wandering scores. Alternatively, pedestrians with wandering scores exceeding a third threshold could also be considered as candidate wandering targets.

[0124] For candidate loitering targets, embodiments of this application can output corresponding warning messages, or can continuously monitor candidate loitering targets.

[0125] In summary, the pedestrian loitering detection method of this application embodiment uses the trajectory distribution information corresponding to pedestrians to characterize the distribution information of pedestrian trajectories, such as walking or standing still. Based on this trajectory distribution information, this application embodiment determines whether a pedestrian is a candidate loitering target. Because the trajectory distribution information corresponding to pedestrians has high flexibility and the ability to perceive real-world scenarios, the trajectory distribution information can change with the actual scene; that is, different actual scenes can correspond to different trajectory distribution information. Thus, the trajectory distribution information of this application embodiment is difficult to crack. Therefore, this application embodiment can, to a certain extent, avoid the problem of trajectory distribution information being cracked affecting the accuracy of pedestrian loitering detection, thereby improving the accuracy of pedestrian loitering detection.

[0126] Method Example 2

[0127] refer to Figure 3 The diagram illustrates a step-by-step flowchart of a pedestrian loitering detection method according to an embodiment of this application. The method may specifically include the following steps:

[0128] Step 301: Acquire the video stream corresponding to the video region; a video stream may include: multiple frame images;

[0129] Step 302: Divide the frame image into a grid to obtain the preset grid region corresponding to the frame image;

[0130] Step 303: Perform pedestrian tracking on the video stream and add the obtained trajectories to the trajectory library;

[0131] Step 304: Based on the pedestrian identity characteristics corresponding to the trajectory, associate the trajectories contained in the trajectory database with the corresponding pedestrians, and add the associated pedestrians to the pedestrian database;

[0132] Step 305: Based on the trajectory distribution information of pedestrians in the pedestrian database, determine the wandering score of pedestrians in the pedestrian database in real time;

[0133] Step 306: Sort the pedestrians in the pedestrian database according to their wandering scores from highest to lowest;

[0134] Step 307: Based on the sorting results, determine whether the pedestrians in the pedestrian database are candidate loitering targets.

[0135] In summary, the pedestrian loitering detection method of this application determines whether a pedestrian is a candidate loitering target based on the trajectory distribution information. Since the trajectory distribution information corresponding to a pedestrian has high flexibility and the ability to perceive the actual scene, the trajectory distribution information can change with the actual scene; that is, different actual scenes can correspond to different trajectory distribution information. Thus, the trajectory distribution information of this application is not easily cracked. Therefore, this application can, to a certain extent, avoid the problem of trajectory distribution information being cracked affecting the accuracy of pedestrian loitering detection, thereby improving the accuracy of pedestrian loitering detection.

[0136] Furthermore, in determining whether a pedestrian is a candidate loitering target, this embodiment of the application can utilize other information besides trajectory distribution information, such as the pedestrian's recurrence information in the image acquisition device and / or the duration of the pedestrian's trajectory. The recurrence information can be the pedestrian's appearance information in multiple cameras; therefore, this embodiment of the application can be applied to multi-camera application scenarios.

[0137] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0138] Based on the above embodiments, this embodiment also provides a pedestrian loitering detection device, referring to... Figure 4 The pedestrian loitering detection device may specifically include: a video stream acquisition module 401, a pedestrian tracking module 402, a trajectory pedestrian association module 403, a trajectory distribution information determination module 404, and a judgment module 405.

[0139] Among them, the video stream acquisition module 401 is used to acquire the video stream corresponding to the video region;

[0140] The pedestrian tracking module 402 is used to track pedestrians in the video stream to obtain the trajectory contained in the video stream; the trajectory corresponds to trajectory features; the trajectory features include: pedestrian position features and pedestrian identity features;

[0141] The trajectory-pedestrian association module 403 is used to associate the trajectory contained in the video stream with the corresponding pedestrian based on the pedestrian's identity characteristics.

[0142] The trajectory distribution information determination module 404 is used to determine the trajectory distribution information of a pedestrian based on the pedestrian's location characteristics.

[0143] The judgment module 405 is used to determine whether the pedestrian is a candidate loitering target based on the trajectory distribution information.

[0144] Optionally, the trajectory distribution information determination module 404 may specifically include:

[0145] A mapping module is used to map the pedestrian's location features onto a preset grid area to obtain the trajectory points of the pedestrian in the preset grid area; the preset grid area may specifically include n×n grids;

[0146] The entropy information determination module is used to determine the entropy information of the pedestrian trajectory in the preset network area based on the number of times the trajectory point corresponding to the pedestrian appears in the grid, and use it as the trajectory distribution information corresponding to the pedestrian.

[0147] Optionally, the entropy information determination module may specifically include:

[0148] The probability of occurrence determination module is used to determine the probability of occurrence of a trajectory point in a grid based on the number of times the trajectory point corresponding to a pedestrian trajectory appears in the grid.

[0149] The single-trajectory entropy information determination module is used to determine the entropy information corresponding to a pedestrian trajectory based on the probability of occurrence of trajectory points in multiple grids.

[0150] The first weighted average module is used to perform a weighted average of the entropy information corresponding to multiple pedestrian trajectories to obtain the entropy information of the pedestrian trajectory in the preset network area.

[0151] Optionally, the judgment module is specifically used to determine whether the pedestrian is a candidate loitering target based on the trajectory distribution information, the repeated appearance information of the pedestrian in the image acquisition device, and / or the duration information of the pedestrian trajectory.

[0152] Optionally, the device may further include:

[0153] The recurrence information determination module is used to determine the recurrence information of a pedestrian in multiple image acquisition devices based on the number of times the pedestrian trajectory appears in each of these devices; and / or

[0154] The duration information determination module is used to determine the duration information of the pedestrian trajectory based on the start and end detection points corresponding to the pedestrian trajectory.

[0155] Optionally, the determination module 405 may specifically include:

[0156] The second weighted average module is used to perform a weighted average of the trajectory distribution information, the information on the repeated appearance of pedestrians in the image acquisition device, and / or the duration information of the pedestrian trajectory, so as to obtain the pedestrian's wandering score.

[0157] The sorting module is used to sort multiple pedestrians according to their respective wandering scores;

[0158] The judgment module based on the sorting result is used to determine whether the pedestrian is a candidate loitering target based on the sorting result.

[0159] Optionally, the trajectory pedestrian association module 403 may specifically include:

[0160] The first creation module is used to create a new pedestrian and associate a first trajectory with the pedestrian.

[0161] A matching degree determination module is used to determine the matching degree between a first trajectory and a second trajectory; wherein the second trajectory is different from the first trajectory;

[0162] The association module is used to associate a second trajectory that matches a preset condition with the pedestrian.

[0163] Optionally, the device may further include:

[0164] Add a module to add a track contained in at least one video stream to the track library;

[0165] The trajectory pedestrian association module 403 may specifically include:

[0166] A construction module is used to build a pedestrian database; the pedestrian database contains at least one pedestrian.

[0167] The query module is used to query whether there is a first pedestrian in the pedestrian database that matches the first trajectory in the trajectory database. If there is, the first trajectory is associated with the first pedestrian.

[0168] The second new module is used to create a second pedestrian if there is no matching first pedestrian and associate the first trajectory with the second pedestrian.

[0169] Optionally, the pedestrian location features may specifically include: location features corresponding to the pedestrian bounding box in the trajectory.

[0170] Optionally, the pedestrian identity features may specifically include: pedestrian identity features of the image corresponding to the pedestrian bounding box in the trajectory.

[0171] In summary, the pedestrian loitering detection device of this application determines whether a pedestrian is a candidate loitering target based on the trajectory distribution information. Since the trajectory distribution information corresponding to the pedestrian has high flexibility and the ability to perceive the actual scene, the trajectory distribution information can change with the actual scene; that is, different actual scenes can correspond to different trajectory distribution information. Thus, the trajectory distribution information of this application is not easily cracked. Therefore, this application can, to a certain extent, avoid the problem of trajectory distribution information being cracked affecting the accuracy of pedestrian loitering detection, thereby improving the accuracy of pedestrian loitering detection.

[0172] Furthermore, in determining whether a pedestrian is a candidate loitering target, this embodiment of the application can utilize other information besides trajectory distribution information, such as the pedestrian's recurrence information in the image acquisition device and / or the duration of the pedestrian's trajectory. The recurrence information can be the pedestrian's appearance information in multiple cameras; therefore, this embodiment of the application can be applied to multi-camera application scenarios.

[0173] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0174] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more of the methods described in the above embodiments. In this application, the electronic device includes various types of devices such as terminal devices and servers (clusters).

[0175] The embodiments of this disclosure can be implemented as an apparatus configured as desired using any suitable hardware, firmware, software, or any combination thereof, including electronic devices such as terminal devices and servers (clusters). Figure 5 An exemplary apparatus 1100 is schematically shown that can be used to implement the various embodiments described in this application.

[0176] In one embodiment, Figure 5An exemplary device 1100 is shown, which includes one or more processors 1102, a control module (chipset) 1104 coupled to at least one of the processors 1102, a memory 1106 coupled to the control module 1104, a non-volatile memory (NVM) / storage device 1108 coupled to the control module 1104, one or more input / output devices 1110 coupled to the control module 1104, and a network interface 1112 coupled to the control module 1104.

[0177] Processor 1102 may include one or more single-core or multi-core processors, and processor 1102 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 1100 can serve as a terminal device, server (cluster), or other device as described in the embodiments of this application.

[0178] In some embodiments, apparatus 1100 may include one or more computer-readable media (e.g., memory 1106 or NVM / storage device 1108) having instructions 1114 and one or more processors 1102 that are combined with the one or more computer-readable media and configured to execute instructions 1114 to implement a module thereby performing the actions described in this disclosure.

[0179] In one embodiment, the control module 1104 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1102 and / or any suitable device or component communicating with the control module 1104.

[0180] The control module 1104 may include a memory controller module to provide an interface to the memory 1106. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0181] Memory 1106 may be used, for example, to load and store data and / or instructions 1114 for device 1100. In one embodiment, memory 1106 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 1106 may include double data rate type quad synchronous dynamic random access memory (DDR4 SDRAM).

[0182] In one embodiment, the control module 1104 may include one or more input / output controllers to provide interfaces to the NVM / storage device 1108 and (one or more) input / output devices 1110.

[0183] For example, NVM / storage device 1108 may be used to store data and / or instructions 1114. NVM / storage device 1108 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0184] NVM / storage device 1108 may include storage resources that are physically part of a device on which device 1100 is mounted, or that can be accessed by the device without needing to be part of the device. For example, NVM / storage device 1108 may be accessed via a network via one or more input / output devices 1110.

[0185] One or more input / output devices 1110 may provide an interface for device 1100 to communicate with any other suitable device. Input / output devices 1110 may include communication components, audio components, sensor components, etc. Network interface 1112 may provide an interface for device 1100 to communicate via one or more networks. Device 1100 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0186] In one embodiment, at least one of the processors 1102 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1104. In one embodiment, at least one of the processors 1102 may be logically packaged with one or more controllers of the control module 1104 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1102 may be integrated with the logic of one or more controllers of the control module 1104 on the same die. In one embodiment, at least one of the processors 1102 may be integrated with the logic of one or more controllers of the control module 1104 on the same die to form a system-on-a-chip (SoC).

[0187] In various embodiments, device 1100 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 1100 may have more or fewer components and / or different architectures. For example, in some embodiments, device 1100 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0188] The detection device can use a main control chip as a processor or control module, and sensor data, position information, etc. can be stored in a memory or NVM / storage device. The sensor group can be used as an input / output device, and the communication interface can include a network interface.

[0189] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0190] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0191] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0192] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0193] These computer program instructions may also be loaded onto a computer or other programmable data processing terminal equipment to cause a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable terminal equipment, provide steps for implementing the functions specified in one or more flowcharts and / or one or more blocks of a block diagram.

[0194] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0195] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0196] The foregoing has provided a detailed description of a pedestrian loitering detection method and apparatus, an electronic device, and a machine-readable medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for detecting pedestrian loitering, characterized in that, The method includes: Get the video stream corresponding to the video region; Pedestrian tracking is performed on the video stream to obtain the trajectories contained in the video stream; the trajectories correspond to trajectory features; the trajectory features include: pedestrian position features and pedestrian identity features; Based on the pedestrian's identity characteristics, the trajectory contained in the video stream is associated with the corresponding pedestrian; Based on the pedestrian location characteristics, determine the trajectory distribution information of the pedestrian; Based on the trajectory distribution information, determine whether the pedestrian is a candidate loitering target; The determination of the trajectory distribution information corresponding to the pedestrian includes: The pedestrian's location features are mapped onto a preset grid area to obtain the trajectory points of the pedestrian in the preset grid area; the preset grid area includes n×n grids; Based on the number of times the trajectory points corresponding to the pedestrian appear in the grid, the entropy information of the pedestrian trajectory in the preset grid area is determined, which serves as the trajectory distribution information corresponding to the pedestrian.

2. The method according to claim 1, characterized in that, The determination of the entropy information of the pedestrian trajectory in the preset grid area includes: The probability of a trajectory point appearing in a grid is determined by the number of times the trajectory point corresponding to a pedestrian trajectory appears in the grid. Based on the probability of occurrence of trajectory points in multiple grids, determine the entropy information corresponding to a pedestrian trajectory; The entropy information corresponding to multiple pedestrian trajectories is weighted and averaged to obtain the entropy information of the pedestrian trajectory in the preset grid area.

3. The method according to any one of claims 1 to 2, characterized in that, The determination of whether the pedestrian is a candidate loitering target includes: Based on the trajectory distribution information, the information on the repeated appearance of the pedestrian in the image acquisition device, and / or the duration information of the pedestrian trajectory, it is determined whether the pedestrian is a candidate loitering target.

4. The method according to claim 3, characterized in that, The method further includes: Based on the frequency of pedestrian trajectories appearing in multiple image acquisition devices, determine the recurrence information of the pedestrian in the image acquisition devices; and / or The duration of the pedestrian trajectory is determined based on the starting and ending detection points.

5. The method according to claim 3, characterized in that, The step of determining whether a pedestrian is a candidate loitering target based on the trajectory distribution information, the pedestrian's recurrence information in the image acquisition device, and / or the duration information of the pedestrian's trajectory includes: The trajectory distribution information, the information on the repeated appearance of pedestrians in the image acquisition device, and / or the duration information of the pedestrian trajectory are weighted and averaged to obtain the pedestrian's wandering score. Based on the pedestrians' loitering scores, the multiple pedestrians are ranked. Based on the sorting results, determine whether the pedestrian is a candidate loitering target.

6. The method according to any one of claims 1 to 2, characterized in that, The step of associating the trajectory contained in the video stream with the corresponding pedestrian based on the pedestrian's identity characteristics includes: Create a new pedestrian and associate the first trajectory with the pedestrian; Determine the matching degree between the first trajectory and the second trajectory; wherein the second trajectory is different from the first trajectory; The second trajectory, which matches the preset conditions, is associated with the pedestrian.

7. The method according to any one of claims 1 to 2, characterized in that, The method further includes: Add at least one track contained in a video stream to the track library; The step of associating the trajectory contained in the video stream with the corresponding pedestrian based on the pedestrian's identity characteristics includes: Construct a pedestrian database; the pedestrian database contains at least one pedestrian. For the first trajectory in the trajectory database, query the pedestrian database to see if there is a first pedestrian that matches it. If there is, associate the first trajectory with the first pedestrian. If no first pedestrian matches the first pedestrian, a second pedestrian will be created, and the first trajectory will be associated with the second pedestrian.

8. The method according to any one of claims 1 to 2, characterized in that, The pedestrian location features include: the location features corresponding to the pedestrian bounding box in the trajectory.

9. The method according to any one of claims 1 to 2, characterized in that, The pedestrian identity features include: pedestrian identity features of the image corresponding to the pedestrian bounding box in the trajectory.

10. A pedestrian loitering detection device, characterized in that, The device includes: The video stream acquisition module is used to acquire the video stream corresponding to a video region; A pedestrian tracking module is used to track pedestrians in the video stream to obtain trajectories contained in the video stream; the trajectories correspond to trajectory features; the trajectory features include: pedestrian position features and pedestrian identity features; The trajectory-pedestrian association module is used to associate the trajectory contained in the video stream with the corresponding pedestrian based on the pedestrian's identity characteristics; The trajectory distribution information determination module is used to determine the trajectory distribution information of a pedestrian based on the pedestrian's location characteristics. The judgment module is used to determine whether the pedestrian is a candidate loitering target based on the trajectory distribution information; The trajectory distribution information determination module includes: A mapping module is used to map the pedestrian's location features onto a preset grid area to obtain the trajectory points of the pedestrian in the preset grid area; the preset grid area includes n×n grids; The entropy information determination module is used to determine the entropy information of the pedestrian trajectory in the preset grid area based on the number of times the trajectory point corresponding to the pedestrian appears in the grid, and use it as the trajectory distribution information corresponding to the pedestrian.

11. The apparatus according to claim 10, characterized in that, The entropy information determination module includes: The probability of occurrence determination module is used to determine the probability of occurrence of a trajectory point in a grid based on the number of times the trajectory point corresponding to a pedestrian trajectory appears in the grid. The single-trajectory entropy information determination module is used to determine the entropy information corresponding to a pedestrian trajectory based on the probability of occurrence of trajectory points in multiple grids. The first weighted average module is used to perform a weighted average of the entropy information corresponding to multiple pedestrian trajectories to obtain the entropy information of the pedestrian trajectory in the preset grid area.

12. An electronic device, characterized in that, include: processor; and A memory having executable code stored thereon, which, when executed, causes the processor to perform the method as described in any one of claims 1-9.

13. A machine-readable medium having executable code stored thereon, which, when executed, causes a processor to perform the method as described in any one of claims 1-9.

Images