Pedestrian entry and exit state detection method based on video electronic fence

Abstract

The application provides a video electronic fence-based pedestrian entering and exiting state detection method. For each received image, the position area of the pedestrians in the current image is obtained based on the position of the pedestrians in the previous image. Then, for a pedestrian in the previous image, the pedestrian image in the corresponding position area is compared with the image of the pedestrian. If there is an image corresponding to the pedestrian, the position corresponding to the image is added to the position sequence list corresponding to the pedestrian and put into an intermediate table. Otherwise, the state of the pedestrian relative to the electronic fence is obtained based on the position sequence list corresponding to the pedestrian. Finally, after comparing all pedestrians, if there are pedestrians not appearing in the previous image, the position sequence list of these pedestrians is constructed. In this way, the specific movement trajectory of each human body near the electronic fence can be accurately and quickly known.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to a method for detecting the entry and exit status of pedestrians based on video electronic fences. Background Art

[0002] Traditional electronic fences require specialized equipment, are costly, and can only detect when an object is entering or exiting a fence. They cannot automatically determine whether the object is a pedestrian or whether the pedestrian is entering or exiting the fence. Summary of the Invention

[0003] In response to the above technical problems, the present invention provides a method for detecting the entry and exit status of pedestrians based on video electronic fences. This method adopts a video-based "electronic fence" and combines it with deep learning technology to automatically determine the entry and exit status of pedestrians. It has low hardware cost and good effect.

[0004] The technical solution adopted in the present invention is:

[0005] An embodiment of the present invention provides a method for detecting the entry and exit status of pedestrians based on a video electronic fence, which is used to detect the status of pedestrians in a target monitoring area, where a camera is provided. The method comprises the following steps:

[0006] S100, receiving the image sent by the shooting device in real time, and identifying the pedestrian information in the received i-th frame image to obtain the identification information table R i , R i The jth row in the ij , P ij , G ij ), ID ij is the ID of the jth pedestrian identified in the i-th frame image, P ij is the image data of the jth pedestrian identified in the i-th frame image, G ij is the position of the jth pedestrian identified in the i-th frame image in the set coordinate system, where j ranges from 1 to m(i), where m(i) is the number of pedestrians in the i-th frame image; i is a natural number greater than or equal to 1; if the number of frames of the currently identified image i ≥ 2, execute S200;

[0007] S200, based on the recognition information table R of the i-1th frame image i-1 , get the candidate region table Z in the i-th frame image i =(Z i1 , Z i2 ,…,Z ik ,…,Z im(i-1) );Z ikis the candidate region determined in the i-th frame based on the position of the k-th pedestrian in the i-1-th frame image, where k ranges from 1 to m(i-1), and m(i-1) is the number of pedestrians in the i-1-th frame image;

[0008] S300, for R i-1 For any row k in , if k≤m(i-1), get Z ik Pedestrian image dataset (P 1 ik , P 2 ik ,…,P r ik ,…,P r(ik) ik );P r ik Z ik The image data of the rth pedestrian in Z, r ranges from 1 to r(ik), r(ik) is Z ik The number of the r-th pedestrian in ; executing S400;

[0009] S400, obtaining similarity set D i-1 ik =(D i-1 ik 1, D i-1 ik 2,…,D i-1 ik r,…,D i-1 ik r(ik)),D i-1 ik r is P r ik and the image data P of the kth pedestrian in the i-1th frame image (i-1)k similarity between

[0010] S500, if max(D i-1 ik )≥D, max(D i-1 ik ) The corresponding pedestrian position is stored in P (i-1)k The next position in the corresponding position sequence list, and max(D i-1 ik ) The corresponding pedestrian ID is stored in the intermediate set M i , execute S700; otherwise, execute S600; D is the set similarity threshold;

[0011] S600, based on P (i-1)k The corresponding position sequence table and the position information of the electronic fence in the set coordinate system are obtained. (i-1)kThe state of the corresponding pedestrian relative to the electronic fence; the electronic fence is a closed curve drawn in a set area of ​​the image; execute S700;

[0012] S700, set k=k+1; execute S300;

[0013] S800, based on R i and M i , get R i Does not belong to M i The ID of the pedestrian is obtained, and a corresponding position sequence table is set for each pedestrian ID obtained, and the position corresponding to each pedestrian ID is stored in the first position of the corresponding position sequence table.

[0014] The present invention also provides a non-transitory computer-readable storage medium, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by a processor to implement the aforementioned method.

[0015] The present invention has at least the following beneficial effects:

[0016] The method for detecting pedestrian entry and exit status using video-based electronic fences, provided by an embodiment of the present invention, first predicts the location area where these pedestrians appear in the current frame based on the pedestrian positions in the previous frame. Next, for a pedestrian in the previous frame, the pedestrian images in the corresponding predicted location area are compared with the pedestrian image. If an image corresponding to the pedestrian exists, the position corresponding to the image is added to the pedestrian's corresponding position sequence table and placed in an intermediate table. Otherwise, a complete position sequence table for the pedestrian is generated, and the pedestrian's status relative to the electronic fence is calculated. Finally, after comparing all pedestrians in the previous frame, the pedestrians identified in the current frame are compared with those in the intermediate table. If there are pedestrians not in the intermediate table, a new position sequence table is constructed for these pedestrians. In this way, the specific movement trajectory of each person near the electronic fence can be accurately and quickly determined. Specifically, by combining human detection and target tracking technologies, the movement trajectory of each person can be generated, which can then be used to determine the person's movement logic in conjunction with the electronic fence, enabling accurate statistics on pedestrian entry and exit. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 This is a flowchart of a method for detecting pedestrian entry and exit based on video electronic fences provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0020] Figure 1 This is a flowchart of a method for detecting pedestrian entry and exit based on video electronic fences provided by an embodiment of the present invention.

[0021] An embodiment of the present invention provides a method for detecting the entry and exit status of pedestrians based on a video electronic fence, which is used to detect the status of pedestrians in a target monitoring area to determine the status of pedestrians relative to the electronic fence. A shooting device is provided in the target monitoring area. In an embodiment of the present invention, the electronic fence is a closed curve drawn in a set area of ​​an image. In one embodiment of the present invention, the position information of the electronic fence is determined based on manual annotation. For example, a polygon can be drawn on the image. The polygon can be regular or irregular and can be customized. In another embodiment of the present invention, the position information of the electronic fence can be obtained based on the position of the set area corresponding to the electronic fence in the world coordinate system. For example, when the corresponding set area is a door, the position of the electronic fence in the set coordinate system can be obtained based on the position of the door in the real world coordinate system mapped to the image coordinate system.

[0022] In an exemplary embodiment, the photographing device may be, for example, a camera.

[0023] like Figure 1 As shown, the method may include the following steps:

[0024] S100, receiving the image sent by the shooting device in real time, and identifying the pedestrian information in the received i-th frame image to obtain the identification information table R i , R i The jth row in the ij , P ij , G ij ), ID ij is the ID of the jth pedestrian identified in the i-th frame image, P ij is the image data of the jth pedestrian identified in the i-th frame image, G ijThe position of the j-th pedestrian identified in the i-th frame image in the set coordinate system, where j ranges from 1 to m(i), m(i) is the number of pedestrians in the i-th frame image; i is a natural number greater than or equal to 1; if the number of frames of the currently identified image i ≥ 2, execute S200.

[0025] In this embodiment of the present invention, pedestrian IDs can be customized or generated in chronological order. Pedestrian image data can be high-dimensional feature vectors extracted by an image feature extractor. Those skilled in the art will appreciate that extracting image feature vectors to form an image feature vector set can be accomplished using existing techniques, such as deep neural networks.

[0026] Furthermore, in the embodiment of the present invention, the set coordinate system may be an image coordinate system, that is, an image coordinate system in the monitoring screen.

[0027] Furthermore, in the embodiment of the present invention, each pedestrian image identified in each frame is marked by a detection frame, that is, a rectangular frame is added to each pedestrian image for marking. ij G is the midpoint of the bottom edge of the detection box of the jth pedestrian identified in the i-th frame image. ij =(x ij ,y ij ), x ij is the horizontal coordinate of the jth pedestrian image coordinate system in the i-th frame image, y ij is the vertical coordinate of the j-th pedestrian image in the i-th frame image in the coordinate system, in pixels.

[0028] Those skilled in the art will appreciate that, in an embodiment of the present invention, if the number of frames of the currently recognized image is i=1, a position sequence table will be constructed for each recognized pedestrian. In an exemplary embodiment, the position sequence table of the jth pedestrian in the i-th frame image may be in the form of L j i =(L j1 i , L j2 i ,…,L jn(j) i ), L j i The stored data is ordered and initialized to a list containing the header element. The length of the list can be expanded.

[0029] It should be noted that for each frame of image, there may be no pedestrians. In this case, the data in the recognition information table will be empty.

[0030] S200, based on the recognition information table R of the i-1th frame image i-1 , get the candidate region table Z in the i-th frame imagei =(Z i1 , Z i2 ,…,Z ik ,…,Z im(i-1) );Z ik is the position G of the kth pedestrian in the i-1th frame image (i-1)k In the candidate area determined in the i-th frame image, the value of k ranges from 1 to m(i-1), and m(i-1) is the number of pedestrians in the i-1-th frame image.

[0031] In the embodiment of the present invention, Z ik G c jk A circular area is drawn with a set radius r as the center. r can be obtained based on experiments. For example, r can be 10 pixels. c ik =F ik *G (i-1)k +w (i-1)k ; F ik is the state transformation vector of the kth pedestrian identified in the i-1th frame image at the current moment, w (i-1)k F is the Gaussian noise of the motion coordinates of the kth pedestrian identified in the i-1th frame image. ik The spatial coordinates of the current time point can be predicted by the historical data of the real spatial coordinates of the target human body, and then the image coordinates can be obtained by affine transformation through camera calibration. (i-1)k The setting may be made based on experience. In an exemplary embodiment, the noise corresponding to each pedestrian may be the same.

[0032] S300, for R i-1 For any row k in , if k≤m(i-1), get Z ik Pedestrian image dataset (P 1 ik , P 2 ik ,…,P r ik ,…,P r(ik) ik );P r ik Z ik The image data of the rth pedestrian in Z, r ranges from 1 to r(ik), r(ik) is Z ik The number of the r-th pedestrian in ; execute S400.

[0033] S400, obtaining similarity set D i-1 ik =(D i-1 ik 1, D i-1ik 2,…,D i-1 ik r,…,D i-1 ik r(ik)),D i-1 ik r is P r ik and the image data P of the kth pedestrian in the i-1th frame image (i-1)k The similarity between them.

[0034] In the embodiment of the present invention, the similarity between two image data may be calculated using a predetermined similarity calculation method, such as cosine distance, Euclidean distance, and Hamming distance.

[0035] In the embodiment of the present invention, D may be determined based on existing methods. For example, D may be 0.5 to 0.95.

[0036] S500, if max(D i-1 ik )≥D, max(D i-1 ik ) The corresponding pedestrian position is stored in P (i-1)k The next position in the corresponding position sequence list, and max(D i-1 ik ) The corresponding pedestrian ID is stored in the intermediate set M i , execute S700; otherwise, execute S600; D is the set similarity threshold.

[0037] If max(D i-1 ik )≥D, indicating that in Z ik Find the (i-1)k The corresponding pedestrian indicates that the pedestrian's trajectory has not yet ended, and the current position of the pedestrian is stored in the corresponding position sequence table. i The stored data is ordered and initialized to a list containing the header element. The length of the list can be expanded.

[0038] If max(D i-1 ik )<D, indicating that Z ik No results found for P (i-1)k The corresponding pedestrian corresponds to the pedestrian, which means that the pedestrian's movement ends within the camera's field of view, and the pedestrian's movement trajectory can be generated based on the image captured by the camera.

[0039] S600, based on P (i-1)k The corresponding position sequence table and the position information of the electronic fence in the set coordinate system are obtained.(i-1)k The status of the corresponding pedestrian relative to the electronic fence; execute S700.

[0040] Furthermore, based on P (i-1)k The corresponding position sequence table and the position information of the electronic fence in the set coordinate system are obtained. (i-1)k The corresponding pedestrian status relative to the geo-fence may specifically include the following steps:

[0041] S601: Acquire a reference area formed by the outline of the electronic fence based on the location information of the electronic fence. Those skilled in the art will appreciate that acquiring a reference area formed by the outline of the electronic fence based on the location information of the electronic fence is an existing technique.

[0042] S602, based on P (i-1)k The position sequence table corresponding to each ID in the corresponding position sequence table generates a trajectory segment corresponding to each ID. The trajectory segment corresponding to each pedestrian can be formed into a broken line segment by multiple segments.

[0043] S603: Calculate the number of intersections between the trajectory segment corresponding to each ID and the reference area.

[0044] S604: Determine the state of the pedestrian corresponding to each ID relative to the electronic fence based on the starting and ending positions and the number of intersections of the pedestrian corresponding to the ID in the set coordinate system.

[0045] In an embodiment of the present invention, the status of a pedestrian relative to the electronic fence may be divided based on actual conditions, for example, may include entering the electronic fence, leaving the electronic fence, crossing the electronic fence, and passing the electronic fence.

[0046] For example, if a pedestrian's trajectory segment has no intersection with the geo-fence, the pedestrian is considered to have merely passed through the geo-fence, but not crossed it. If a pedestrian's trajectory segment has one intersection with the geo-fence, and the pedestrian's starting position is inside the geo-fence, the pedestrian is considered to have crossed the geo-fence from the inside out, i.e., left the geo-fence. If a pedestrian's trajectory segment has one intersection with the geo-fence, and the pedestrian's starting position is outside the geo-fence, the pedestrian is considered to have crossed the geo-fence from the outside in, i.e., entered the geo-fence. If a pedestrian's trajectory segment has two intersections with the geo-fence, and the pedestrian's starting and ending positions are outside the geo-fence, the pedestrian is considered to have crossed the geo-fence from the outside in and then from the inside out.

[0047] S700, set k=k+1; execute S300.

[0048] S800, based on R i and M i , get R iDoes not belong to M i The ID of the pedestrian is obtained, and a corresponding position sequence table is set for each pedestrian ID obtained, and the position corresponding to each pedestrian ID is stored in the first position of the corresponding position sequence table.

[0049] Setting a corresponding position sequence table for each obtained pedestrian ID may be performed in the same manner as constructing a position sequence table for the pedestrian in the first frame image.

[0050] In summary, the method for detecting pedestrian entry and exit status using video-based electronic fences provided by the present invention predicts the location area where these pedestrians appear in the current frame based on the pedestrian positions in the previous frame. Next, for a pedestrian in the previous frame, the pedestrian images in the corresponding predicted location area are compared with the pedestrian image. If an image corresponding to the pedestrian exists, the position corresponding to the image is added to the position sequence table corresponding to the pedestrian and placed in the intermediate table. Otherwise, a complete position sequence table for the pedestrian is generated, and the pedestrian's status relative to the electronic fence is calculated. Finally, after comparing all pedestrians in the previous frame, the pedestrians identified in the current frame are compared with those in the intermediate table. If there are pedestrians not in the intermediate table, a new position sequence table is constructed for these pedestrians. In this way, the specific movement trajectory of each human being near the electronic fence can be accurately and quickly determined. Specifically, by combining human detection and target tracking technology, the movement trajectory of each human can be generated, which can then be used to determine the movement logic of the person in conjunction with the electronic fence, enabling accurate statistics.

[0051] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to implementing a method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiment.

[0052] An embodiment of the present invention further provides a computer program product comprising program code. When the program product is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the method according to various exemplary embodiments of the present invention described above in this specification.

[0053] Although some specific embodiments of the present invention have been described in detail by way of example, it will be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present invention. It will also be understood by those skilled in the art that various modifications may be made to the embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims

1. A method for detecting pedestrian entry and exit based on video electronic fence, characterized in that: The method is used to detect the status of pedestrians in a target monitoring area, wherein a photographing device is provided in the target monitoring area; the method comprises the following steps: S100, receiving the image sent by the shooting device in real time, and identifying the pedestrian information in the received i-th frame image to obtain the identification information table R i , R i The jth row in the ij , P ij , G ij ), ID ij is the ID of the jth pedestrian identified in the i-th frame image, P ij is the image data of the jth pedestrian identified in the i-th frame image, G ij The position of the jth pedestrian identified in the i-th frame image in the set coordinate system, where j ranges from 1 to m(i), where m(i) is the number of pedestrians in the i-th frame image; i is a natural number greater than or equal to 1; if the number of frames of the currently identified image i ≥ 2, execute S200; S200, based on the recognition information table R of the i-1th frame image i-1 , get the candidate region table Z in the i-th frame image i =(Z i1 , Z i2 ,…,Z ik ,…,Z im（i-1） ); Z ik is the candidate region determined in the i-th frame based on the position of the k-th pedestrian in the i-1-th frame image, where k ranges from 1 to m(i-1), and m(i-1) is the number of pedestrians in the i-1-th frame image; S300, for R i-1 For the kth pedestrian in , if k≤m(i-1), then get Z ik Pedestrian image dataset (P 1 ik , P 2 ik ,…,P r ik ,…,P r（ik） ik );P r ik Z ik The image data of the rth pedestrian in the image, r ranges from 1 to r(ik), r(ik) is Z ik The number of pedestrians in the vehicle; executing S400; S400, obtaining similarity set D i-1 ik =(D i-1 ik 1, D i-1 ik 2,…,D i-1 ik r,…,D i-1 ik r(ik)), D i-1 ik r is P r ik and the image data P of the kth pedestrian in the i-1th frame image （i-1）k similarity between S500, if max(D i-1 ik )≥D, max(D i-1 ik ) The corresponding pedestrian position is stored in P （i-1）k The next position in the corresponding position sequence list, and max (D i-1 ik ) The corresponding pedestrian ID is stored in the intermediate set M i At the corresponding position, execute S700; Otherwise, execute S600; D is the set similarity threshold; S600, based on P （i-1）k The corresponding position sequence table and the position information of the electronic fence in the set coordinate system are obtained. （i-1）k The status of the corresponding pedestrian relative to the electronic fence; The electronic fence is a closed curve drawn in a set area of ​​the image; execute S700; S700, set k=k+1; execute S300; S800, based on R i and M i , get R i Does not belong to M i The ID of the pedestrian is obtained, and a corresponding position sequence table is set for each pedestrian ID obtained, and the position corresponding to each pedestrian ID is stored in the first position of the corresponding position sequence table; Among them, Z ik G c ik As the center of the circle, a circular area is drawn with a set radius r; where G c ik =F ik ×G （i-1）k+ w （i-1）k ; G （i-1）k is the position of the kth pedestrian in the i-1th frame image, F ik is the state transformation vector of the position of the kth pedestrian in the i-1th frame image at the current moment, w （i-1）k is the Gaussian noise of the motion coordinates of the k-th pedestrian in the i-1-th frame image; S600 includes the following steps: S601, acquiring a reference area formed by the outline of the electronic fence based on the position information of the electronic fence; S602, based on P （i-1）k The position sequence table corresponding to each ID in the corresponding position sequence table generates a trajectory segment corresponding to each ID; S603, calculating the number of intersections between the trajectory segment corresponding to each ID and the reference area; S604: Determine the state of the pedestrian corresponding to each ID relative to the electronic fence based on the starting and ending positions and the number of intersections of the pedestrian corresponding to the ID in the set coordinate system.

2. The method according to claim 1, characterized in that The set coordinate system is an image coordinate system.

3. The method according to claim 2, characterized in that The image of each pedestrian identified in each frame is marked by a detection box; Among them, G ij is the midpoint of the bottom edge of the detection box of the j-th pedestrian identified in the i-th frame image.

4. The method according to claim 1, wherein The location information of the electronic fence is determined based on manual marking.

5. The method according to claim 1, wherein The position information of the electronic fence is obtained based on the position of the set area corresponding to the electronic fence in the world coordinate system.

6. The method according to claim 1, characterized in that The shooting device is a camera.

7. A non-transitory computer-readable storage medium, wherein the storage medium stores at least one instruction or at least one program, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the method according to any one of claims 1 to 6.

Images