A vector topology integrated passenger flow calculation method and system based on fisheye probe

Abstract

The present invention provides a vector topology integrated passenger flow calculation method and system based on fisheye probes. The method includes the following steps: obtaining video stream data collected by a target fisheye probe; obtaining a first pedestrian trajectory image based on the video stream data; obtaining a target vector pedestrian feature corresponding to the first pedestrian trajectory image based on the regional topology of the first pedestrian trajectory image and the target fisheye probe; performing a topological search on the target vector pedestrian feature based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image; and obtaining passenger flow results based on the second pedestrian trajectory image. The present invention can maintain system stability in scenarios where pedestrians change rapidly, effectively reduce imaging interference caused by fisheye probes, improve the overall robustness of the system, and can stably and effectively output passenger flow results.

Description

Technical Field

[0001] The present invention relates to the technical field of visual monitoring, and in particular to a vector topology integrated passenger flow calculation method and system based on a fisheye probe. Background Art

[0002] Fisheye cameras are extremely wide-angle lenses, optical imaging devices with a large aperture and an extremely large visual field. Their wide field of view eliminates the need for rotation and scanning, allowing for gaze-based image capture, significantly reducing the number of surveillance devices installed and subsequent maintenance costs. These characteristics have led to their widespread use in security settings, offline shopping malls, and retail brand stores. Pedestrian counting solutions developed with fisheye cameras have become a popular and leading computer vision-based solution for passenger flow problems.

[0003] The introduction of fisheye sensors has also brought challenges to this type of passenger counting system that are different from those of previous systems based on oblique illumination sensors:

[0004] (1) Image distortion: When a fisheye probe acquires an ultra-wide-angle field of view, it often causes severe image distortion. Therefore, a certain amount of computing power is required to correct the image distortion. Due to the characteristics of this distortion, the effective utilization rate of the captured images of the data acquired by the front-end device will be greatly reduced. As a result, the real objects captured by the oblique illumination probe at the same pixel level have higher resolution, which places higher analysis and accuracy requirements on the passenger flow counting system.

[0005] (2) Image semantic information: Due to the distortion and distortion of fisheye probes, and in order to give full play to their ultra-wide-angle characteristics, in actual production environments, fisheye probes are often installed perpendicular to the ground. This results in different capture angles when the captured object is in different positions of the probe, causing the graphic semantic information to change almost all the time. In particular, when the object is located directly below the probe, the image effect is similar to a top-down view, which is difficult to handle for recognition and tracking.

[0006] (3) Image style: In an actual production environment, the installation and deployment of the probes need to be closely coordinated with the environment. This will result in the probes in the same location actually having different image parameters, resulting in different visual styles. In addition, the images captured by the same probe will have completely different image styles at the center and edge of the picture, which is a considerable challenge to the overall passenger flow statistics.

[0007] Therefore, due to the imaging interference of the fisheye probe, the passenger flow system is not stable and cannot make accurate calculations of the passenger flow results, resulting in large fluctuations in the number of passenger flow detections. Summary of the Invention

[0008] In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a vector topology integrated passenger flow calculation method and system based on a fisheye probe, which is used to solve the problem in the prior art that due to the imaging interference of the fisheye probe, the passenger flow system is unstable and cannot make accurate calculations of the passenger flow results, resulting in large fluctuations in the number of passenger flow detections.

[0009] To achieve the above-mentioned objectives and other related objectives, the present invention provides a vector topology integrated passenger flow calculation method based on a fisheye probe, and the vector topology integrated passenger flow calculation method based on a fisheye probe includes the following steps: obtaining video stream data collected by a target fisheye probe; obtaining a first pedestrian trajectory image based on the video stream data; obtaining a target vector pedestrian feature corresponding to the first pedestrian trajectory image based on the regional topology of the first pedestrian trajectory image and the target fisheye probe; performing topological retrieval on the target vector pedestrian feature based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image; and obtaining passenger flow results based on the second pedestrian trajectory image.

[0010] In one embodiment of the present invention, obtaining the first pedestrian trajectory image based on the video stream data includes the following steps: performing image preprocessing on the video stream data to obtain a perspective image; detecting the pedestrian in the perspective image based on a target detection method, and obtaining a pedestrian torso candidate image corresponding to the pedestrian; when there is a pedestrian torso candidate image corresponding to the pedestrian at the previous moment, associating the pedestrian torso candidate image with the pedestrian torso candidate image at the previous moment based on a tracking algorithm to obtain the first pedestrian trajectory image; if there is no pedestrian torso candidate image at the previous moment, using the pedestrian torso candidate image as the first pedestrian trajectory image.

[0011] In one embodiment of the present invention, obtaining the target vector pedestrian features corresponding to the first pedestrian trajectory image based on the regional topology of the first pedestrian trajectory image and the target fisheye probe includes the following steps: sampling the first pedestrian trajectory image to obtain the pedestrian features corresponding to the first pedestrian trajectory image; dividing the pedestrian features according to the regional topology of the target fisheye probe to obtain the target vector pedestrian features.

[0012] In one embodiment of the present invention, sampling the first pedestrian trajectory image includes: when the pedestrian information in the first pedestrian trajectory image is complete, dense sampling the first pedestrian trajectory image; when the pedestrian information in the first pedestrian trajectory image is incomplete, sparse sampling the first pedestrian trajectory image.

[0013] In one embodiment of the present invention, the target vector pedestrian features are topologically retrieved based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image, including the following steps: obtaining the non-target vector pedestrian features corresponding to the non-target fisheye probe before the current moment; screening the non-target vector pedestrian features to obtain candidate vector pedestrian features; performing feature sparse matching calculation on the target vector pedestrian features and the candidate vector pedestrian features, and selecting the candidate vector pedestrian features with the highest sparse matching result as the trajectory matching result; associating the trajectory matching result with the target vector pedestrian features based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image.

[0014] In one embodiment of the present invention, the feature sparse matching calculation formula is:

[0015] S 稀疏匹配分数 =a×S1+(1-a)×S2;

[0016] Among them, a is an adjustable coefficient, S1 represents the average score of dense sampling, S2 represents the average score of sparse sampling, and S 稀疏匹配分数 The calculation formulas for the trajectory matching results, the dense sampling average score and the sparse sampling average score are as follows:

[0017]

[0018] Where n=1 means that the number of trajectories corresponding to the candidate vector pedestrian feature is 1, m means the maximum number of trajectories corresponding to the candidate vector pedestrian feature, S x Indicates the similarity of features at the same angle, S y Indicates the similarity of adjacent angle features, S 采样平均分数 Expressed as the sample mean score.

[0019] In one embodiment of the present invention, the regional topology of the target fisheye probe is to divide the monitoring screen of the target fisheye probe into different areas according to the installation point and monitoring area of ​​the target fisheye probe, and then topologically connect the different areas; the spatial topology of the target fisheye probe and the non-target fisheye probe is to topologically connect the regional topology of the target fisheye probe with the regional topology of the non-target fisheye probe again.

[0020] The present invention provides a vector topology integrated passenger flow calculation system based on a fisheye probe, which includes: an image collection module, a first pedestrian trajectory image acquisition module, a vector pedestrian feature module, a pedestrian retrieval module, and a passenger flow output module; the image collection module is used to obtain video stream data collected by a target fisheye probe; the first pedestrian trajectory image acquisition module is used to obtain a first pedestrian trajectory image based on the video stream data; the vector pedestrian feature module is used to obtain a target vector pedestrian feature corresponding to the first pedestrian trajectory image based on the regional topology of the first pedestrian trajectory image and the target fisheye probe; the pedestrian retrieval module is used to perform topological retrieval on the target vector pedestrian feature based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image; and the passenger flow output module is used to obtain passenger flow results based on the second pedestrian trajectory image.

[0021] The present invention provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned vector topology integrated passenger flow calculation method based on fisheye probe is realized.

[0022] The present invention provides a terminal, comprising: a processor and a memory; the memory is used to store computer programs; the processor is used to execute the computer programs stored in the memory, so that the terminal executes the above-mentioned vector topology integrated passenger flow calculation method based on fisheye probe.

[0023] As described above, the vector topology integrated passenger flow calculation method and system based on fisheye probes of the present invention have the following beneficial effects:

[0024] (1) Compared with the existing technology, the present invention proposes a complete passenger flow solution, which combines open source pedestrian recognition, tracking, and re-identification algorithms. A specific sampling method is used to capture the pedestrian trajectory. At the same time, based on the actual position of the fisheye probe in three-dimensional space, the spatial topological relationship between the probes is constructed, and combined with the pedestrian tracking results, vector pedestrian features are obtained. Based on the above sampling and topological logic, real-time sparse matching is performed. The overall coordination of the system is high, which effectively reduces the disadvantages brought by the fisheye probe.

[0025] (2) The present invention uses the real position of the fisheye probe in three-dimensional space to construct the spatial topological relationship between the probes, and combines it with pedestrian recognition and tracking algorithms to improve the overall robustness of the system and can stably and effectively output passenger flow results. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 Shown is a flow chart of an embodiment of a vector topology integrated passenger flow calculation method based on a fisheye probe according to the present invention.

[0027] Figure 2 Shown is a flow chart of obtaining a first pedestrian trajectory image in one embodiment of the present invention.

[0028] Figure 3 Shown is a flow chart of obtaining a target vector pedestrian feature corresponding to a first pedestrian trajectory image in one embodiment of the present invention.

[0029] Figure 4 Shown is a structural schematic diagram of an embodiment of the regional topology of the fisheye probe monitoring screen of the present invention.

[0030] Figure 5 Shown is a flow chart of obtaining a second pedestrian trajectory image in one embodiment of the present invention.

[0031] Figure 6 Shown is a structural diagram of a vector topology integrated passenger flow calculation system based on a fisheye probe in one embodiment of the present invention.

[0032] Figure 7 Shown is a schematic structural diagram of a terminal in one embodiment of the present invention.

[0033] Label Description

[0034] 4 Fisheye probe image coverage area

[0035] 41 Target Fisheye Probe

[0036] 411 Area 1A

[0037] 412 Area 1B

[0038] 413 Area 1C

[0039] 42 Non-target fisheye probe

[0040] 421 Area 2A

[0041] 422 Area 2B

[0042] 61 Image Collection Module

[0043] 62 First pedestrian trajectory image acquisition module

[0044] 63 Vector Pedestrian Feature Module

[0045] 64 Pedestrian Retrieval Module

[0046] 65 Passenger flow output module

[0047] 7 Terminal

[0048] 71 processing units

[0049] 72 Memory

[0050] 721 Random Access Memory

[0051] 722 Cache Memory

[0052] 723 Storage System

[0053] 724 Programs / Utilities

[0054] 7241 Program Module

[0055] 73 bus

[0056] 74 Input / Output Interfaces

[0057] 75 Network Adapter

[0058] 8 External devices

[0059] 9 Display

[0060] Steps S1 to S5

[0061] Steps S21-S22

[0062] Steps S31-S32

[0063] Steps S41 to S44 DETAILED DESCRIPTION

[0064] The following describes the embodiments of the present invention through specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through different specific embodiments. The details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other unless they conflict.

[0065] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0066] Compared to existing technologies, the present invention's fisheye probe-based vector topology integrated passenger flow calculation method and system utilizes the actual position of the fisheye probe in three-dimensional space to construct the spatial topological relationship between probes. This, combined with pedestrian recognition and tracking algorithms, imparts realistic vector variations to pedestrian trajectories, resulting in a complete, full-field trajectory of pedestrians captured by different probes. The coordinated and complementary nature of each module effectively reduces imaging interference introduced by the fisheye probe, improves the overall robustness of the system, and enables stable and efficient passenger flow output.

[0067] The present invention will be described below using flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine such that when these computer program instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowcharts and / or block diagrams.

[0068] These computer program instructions may also be stored in a computer-readable medium, which causes a computer, other programmable data processing apparatus, or other device to operate in a specific manner, so that the instructions stored in the computer-readable medium produce an article of manufacture that includes instructions for implementing the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0069] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide a process for implementing the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0070] like Figure 1 As shown, in one embodiment, the vector topology integrated passenger flow calculation method based on the fisheye probe of the present invention includes the following steps:

[0071] Step S1: Obtain video stream data collected by a target fisheye probe.

[0072] It should be noted that the video stream data collected by the target fisheye probe can be scenes with rapidly changing pedestrians such as offline shopping malls and retail brand points, or video stream data from public places such as residential areas. Video stream data can be understood as a sequence of image frames, that is, multiple frames of images constitute the video data.

[0073] Step S2: Acquire a first pedestrian trajectory image based on the video stream data.

[0074] like Figure 2 As shown, in one embodiment, obtaining a first pedestrian trajectory image based on the video stream data includes the following steps:

[0075] S21. Perform image preprocessing on the video stream data to obtain a perspective image.

[0076] Specifically, the acquired video stream data is subjected to image correction, noise filtering and other processing to obtain a perspective image that conforms to human visual senses.

[0077] It should be noted that in the actual installation of probes, the height and strength of the light field of each probe are different, so relevant adjustments will be made to the image, including but not limited to the control of basic image processing such as contrast, brightness, and saturation; after obtaining a stable and clear monitoring image, the fisheye probe image needs to be distorted and corrected, and finally a perspective image that conforms to human visual perception can be obtained.

[0078] In order to capture images in the video stream data at different angles, the fisheye probe is installed vertically. The vertically installed fisheye probe can not only maximize the ultra-wide-angle shooting characteristics of the fisheye probe, but also expand the coverage area of ​​the fisheye probe while ensuring that pedestrians can be cleared and stably captured.

[0079] S22: Detect a pedestrian in the perspective image based on a target detection method, and obtain a pedestrian torso candidate image corresponding to the pedestrian.

[0080] Specifically, the acquired perspective image is subjected to a target detection method using a neural network to detect pedestrians in the image, thereby obtaining candidate pedestrian torso images corresponding to all pedestrians in the image.

[0081] It should be noted that the target detection method of the neural network used in this embodiment can be various replaceable popular open source target detection networks, such as yolo3, yolo4, Efficient Net, etc.

[0082] S23. When there is a pedestrian torso candidate image corresponding to the pedestrian at the previous moment, the pedestrian torso candidate image is associated with the pedestrian torso candidate image at the previous moment based on the tracking algorithm to obtain the first pedestrian trajectory image; if there is no pedestrian torso candidate image at the previous moment, the pedestrian torso candidate image is used as the first pedestrian trajectory image.

[0083] Specifically, at the current moment T, the picture captured by the target fisheye probe is subjected to image preprocessing and target detection to obtain a large number of pedestrian torso candidate images. These pedestrian torso candidate images need to be associated with the pedestrian torso candidate images obtained at the previous moment (T-1) to obtain the complete first pedestrian trajectory image.

[0084] It should be noted that in actual production environments, since the number of candidate pedestrian torso images is often very large, this method will use a multi-target tracking method such as the open source algorithm DeepSORT to match and associate the detection results of two adjacent times, thereby obtaining the pedestrian trajectory under the same probe, that is, the first pedestrian trajectory image.

[0085] Step S3: Based on the first pedestrian trajectory image and the regional topology of the target fisheye probe, a target vector pedestrian feature corresponding to the first pedestrian trajectory image is obtained.

[0086] It should be noted that the first pedestrian trajectory image obtained is the pedestrian trajectory under the target fisheye probe. To obtain a complete passenger flow estimate from the time the pedestrian enters the scene to the time he leaves the scene, it is necessary to correlate the pedestrian trajectory image under the target fisheye probe corresponding to the pedestrian with the pedestrian trajectory image under the non-target fisheye probe. Due to the influence of factors such as the installation requirements of the fisheye probe and the actual installation site environment, both are core requirements of the pedestrian feature matching algorithm across fisheye probes. Therefore, pedestrian trajectory matching across fisheye probes is also a core part of the passenger flow algorithm.

[0087] Specifically, if Figure 3 As shown, in one embodiment, based on the first pedestrian trajectory image and the regional topology of the target fisheye probe, obtaining the target vector pedestrian feature corresponding to the first pedestrian trajectory image includes the following steps:

[0088] S31: Sampling the first pedestrian trajectory image to obtain pedestrian features corresponding to the first pedestrian trajectory image.

[0089] Specifically, the first pedestrian trajectory image is sampled according to the sampling requirements, and the sampled image is subjected to feature extraction by a feature extraction neural network to obtain corresponding pedestrian features.

[0090] It should be noted that the pedestrian feature generated by the first pedestrian trajectory image is the first pedestrian trajectory feature, which is the image information and position information corresponding to the pedestrian.

[0091] It should be noted that the feature extraction neural network based on pedestrian re-identification used in this embodiment adopts a more complex network structure and a deeper network depth to obtain pedestrian features with high information content and stability, so as to ensure that the pedestrian features can have higher matching accuracy and recall rate under cross-probe matching, thereby ensuring that the pedestrian trajectory can be completely matched.

[0092] The feature extraction neural network used in this embodiment usually uses a larger one such as ResNet50 or VGG16 as a backbone, and is subsequently supplemented with specific network results to ultimately output higher-dimensional pedestrian features. However, such a design often consumes more computing power resources than other modules.

[0093] Therefore, in order to save computing resources and ensure the matching effect, while taking into account the capture effect of the vertical fisheye probe at different angles, the pedestrian torso image under the trajectory is specifically sampled, and the sampled image will be used for feature extraction.

[0094] Specifically, sampling the first pedestrian trajectory image includes:

[0095] i) When the pedestrian information in the first pedestrian trajectory image is complete, dense sampling is performed on the first pedestrian trajectory image.

[0096] It should be noted that when the angle is moderate, the pedestrian's torso information is relatively complete. At this time, the pedestrian features contain more comprehensive body information and good generalization, which can be better matched with the features obtained by other probes. Such images are collected at relatively dense time intervals.

[0097] ii) When the pedestrian information in the first pedestrian trajectory image is incomplete, sparse sampling is performed on the first pedestrian trajectory image.

[0098] It should be noted that when the angle of the captured pedestrian trajectory image deviates from the optimal shooting angle or is captured at the edge, the image quality is poor and the information of the pedestrian torso features in the image is severely lost. Such images will not be completely discarded. Often, such specialized features can also provide a certain matching contribution, so only partial collection is required at a larger time interval.

[0099] S32. Divide the pedestrian features according to the regional topology of the target fisheye probe to obtain the target vector pedestrian features.

[0100] Specifically, the pedestrian features are assigned corresponding entry and exit direction vectors to the pedestrian trajectory according to the regional topology of the target fisheye probe, thereby obtaining a target vector pedestrian feature composed of pedestrian features corresponding to the pedestrian trajectory image.

[0101] It should be noted that the regional topology of the target fisheye probe is based on the actual installation position of the target fisheye probe and the corresponding monitoring area, dividing the monitoring screen into topological structures of different area sizes.

[0102] like Figure 4 As shown, in one embodiment, in order to make the captured image of the fisheye probe cover the area where people flow, and to form channels with different flow directions in the image of the fisheye probe, the monitoring image of the fisheye probe is divided into regional topology.

[0103] Figure 4 In the figure, N is the store area, the fisheye probe's image coverage area 4 is the gray part, and the target fisheye probe 41's capture area can completely cover the bifurcation hub shown in the figure, so that the capture area can be well divided into area 1A411, area 1B412 and area 1C413. Therefore, there are three topological relationships in the pedestrian trajectory passed by the target fisheye probe 41: area 1A411-area 1B412, area 1A411-area 1C413, and area 1B412-area 1C413. , and according to the direction of pedestrian flow, the pedestrian trajectory is assigned corresponding entry and exit direction vectors; for example, a pedestrian enters the capture area of ​​area 1A411 of the target fisheye probe 41 at time 1, and finally leaves the capture area of ​​area 1B412 of the target fisheye probe 41 at time 2. During this period, effective trajectory sampling features F1, F2, F3, ... FN are generated. Then the trajectory generated by the pedestrian under the target fisheye probe 41 will be output as structured data as shown in Table 1 - target vector pedestrian features.

[0104] Pedestrian characteristics F1, F2, F3, ...FN Probe number 41 Enter the area Area A Out of area Area B Track duration Time 1 - Time 2

[0105] Step S4: performing a topological search on the target vector pedestrian feature based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image.

[0106] It should be noted that when the target vector pedestrian feature corresponding to the first pedestrian trajectory image is obtained, real-time topology retrieval will begin.

[0107] Specifically, if Figure 5 As shown, in one embodiment, performing a topological search on the target vector pedestrian feature based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image includes the following steps:

[0108] Step S41: Obtain the non-target vector pedestrian features corresponding to the non-target fisheye probe before the current moment.

[0109] It should be noted that in order to obtain a complete trajectory image of a pedestrian, it is necessary to retrieve all vector pedestrian features of the pedestrian entering the monitoring area of ​​the fisheye probe. Therefore, it is necessary to obtain the non-target vector pedestrian features corresponding to the pedestrian under the non-target fisheye probe before the current moment.

[0110] Step S42: Screen the non-target vector pedestrian features to obtain candidate vector pedestrian features.

[0111] It should be noted that each set of vector pedestrian features has a corresponding vector pedestrian trajectory.

[0112] Specifically, the target vector pedestrian feature of the pedestrian at time T under the target fisheye probe has a corresponding vector pedestrian trajectory F. The candidate vector pedestrian features that participate in pedestrian retrieval together with it need to meet the following conditions:

[0113] (1) Time condition: The output time of the candidate vector feature must be before time T;

[0114] (2) Topological condition: the exit region of the candidate vector pedestrian feature must have a direct topological connection with the entry region of the vector pedestrian trajectory F;

[0115] (3) Deduplication: The candidate vector pedestrian features have formed a complete trajectory, and finally generated features out of the region from the edge of the current topology, and can no longer participate in the retrieval.

[0116] Step S43 : performing feature sparse matching calculation on the target vector pedestrian feature and the candidate vector pedestrian feature, and selecting the candidate vector pedestrian feature with the highest sparse matching result as the trajectory matching result.

[0117] It should be noted that since different vector pedestrian features will have corresponding different vector pedestrian trajectories, it is necessary to perform sparse matching on the candidate vector pedestrian features; and due to the capture limitations of the fisheye probe, the closer the pedestrian is to the fisheye probe, the larger the angle between the vertical line formed with it will be, and the information of the pedestrian's torso will be blocked by his or her head and shoulders. This loss of image semantic information will seriously affect the distance between pedestrian features, thereby leading to incorrect matching or classification of pedestrian features.

[0118] Therefore, in one embodiment, sparse matching is proposed, and the results of dense acquisition and sparse acquisition in the pedestrian feature extraction process are combined to coordinate the perspective differences between different trajectories.

[0119] Specifically, the feature sparse matching calculation formula is as follows:

[0120] S 稀疏匹配分数=a×S1+(1-a)×S2;

[0121] Among them, a is an adjustable coefficient, usually 0.8, that is, the image feature matching score of dense sampling is higher, S1 is the average score of dense sampling, S2 is the average score of sparse sampling, S 稀疏匹配分数 The trajectory matching result.

[0122] The calculation formulas for the dense sampling average score and the sparse sampling average score are as follows:

[0123]

[0124] Among them, n=1 means that the number of trajectories corresponding to the candidate vector pedestrian feature is 1, m represents the maximum number of trajectories corresponding to the candidate vector pedestrian feature, S x Indicates the similarity of features at the same angle, S y Indicates the similarity of adjacent angle features, S 采样平均分数 Expressed as the sample mean score.

[0125] Specifically, when the two have the same angle, the single feature of the same angle is used to calculate the similarity of the Euclidean distance; for angles that do not exist between the two, the feature of the other party that is closest to the angle is used for matching, and the number of features with the largest number of trajectories between the two is traversed several times to calculate the sampling average score; finally, among the candidate vector pedestrian features, the feature with the highest sparse matching score is the trajectory matching result.

[0126] Step S44: Associating the trajectory matching result with the target vector pedestrian feature based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image.

[0127] It should be noted that the spatial topology of the target fisheye probe and the non-target fisheye probe is a topological connection of the regional topology of the target fisheye probe and the regional topology of the non-target fisheye probe.

[0128] like Figure 4As shown, in one embodiment, for example, the vector pedestrian feature of a pedestrian under the non-target fisheye probe 42 is expressed as: entering from the capture area of ​​area 2B422 of the non-target fisheye probe 42 at time 1, and leaving from the capture area of ​​area 2A421 of the fisheye probe 42 at time 2; the target vector pedestrian feature under the target fisheye probe 41 is expressed as: entering from the capture area of ​​area 1A411 of the target fisheye probe 41 at time 3, and finally leaving from the capture area of ​​area 1B412 of the target fisheye probe 41 at time 4, then the complete pedestrian trajectory of the pedestrian at time 1-4 is: area 2B422-area 2A421-area 1A411-area 1B412, and the corresponding capture images are associated to obtain a second pedestrian trajectory image.

[0129] Step S5: Obtain passenger flow results based on the second pedestrian trajectory image.

[0130] Specifically, the second pedestrian trajectory image corresponding to each pedestrian is marked as an ID, and the number of IDs corresponds to the passenger flow calculation result.

[0131] It should be noted that the protection scope of the vector topology integrated passenger flow calculation method based on fisheye probe described in the present invention is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the existing technology based on the principles of the present invention are included in the protection scope of the present invention.

[0132] like Figure 6 As shown, in one embodiment, the vector topology integrated passenger flow calculation system based on the fisheye probe of the present invention includes: an image collection module 61, a first pedestrian trajectory image acquisition module 62, a vector pedestrian feature module 63, a pedestrian retrieval module 64, and a passenger flow output module 65.

[0133] The image collection module 61 is used to obtain the video stream data collected by the target fisheye probe;

[0134] The first pedestrian trajectory image acquisition module 62 is used to acquire a first pedestrian trajectory image based on the video stream data;

[0135] The vector pedestrian feature module 63 is configured to obtain a target vector pedestrian feature corresponding to the first pedestrian trajectory image based on the first pedestrian trajectory image and the regional topology of the target fisheye probe;

[0136] The pedestrian retrieval module 64 is configured to perform a topological retrieval on the target vector pedestrian feature based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image;

[0137] The passenger flow output module 65 is configured to obtain passenger flow results based on the second pedestrian trajectory image.

[0138] It should be noted that the structure and principle of the vector topology integrated passenger flow calculation system based on fisheye probes correspond one to one with the steps in the above-mentioned vector topology integrated passenger flow calculation method based on fisheye probes, so they will not be repeated here.

[0139] It should be understood that the division of the modules in the above system is merely a division of logical functions. In actual implementation, they may be fully or partially integrated into a single physical entity or physically separated. Furthermore, these modules may be implemented entirely in software called by a processing element, or entirely in hardware. Alternatively, some modules may be implemented in software called by a processing element, while others may be implemented in hardware. For example, module x may be a separate processing element, or integrated into a chip in the above system. Furthermore, it may be stored in the form of program code in the memory of the above system, called by a processing element in the system to perform the functions of module x. The implementation of other modules is similar. Furthermore, these modules may be fully or partially integrated or implemented independently. The processing element described herein may be an integrated circuit with signal processing capabilities. During implementation, the steps of the above method or the modules above may be performed by hardware integrated logic circuits in the processor element or by software instructions.

[0140] For example, the above modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0141] It should be noted that the vector topology integrated passenger flow calculation system based on fisheye probe of the present invention can realize the vector topology integrated passenger flow calculation method based on fisheye probe of the present invention, but the implementation device of the vector topology integrated passenger flow calculation method based on fisheye probe of the present invention includes but is not limited to the structure of the vector topology integrated passenger flow calculation system based on fisheye probe listed in this embodiment. All structural deformations and replacements of the prior art made according to the principles of the present invention are included in the protection scope of the present invention.

[0142] The storage medium of the present invention stores a computer program that, when executed by a processor, implements the above-mentioned fisheye probe-based vector topology integrated passenger flow calculation method. The storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), a magnetic disk, a USB flash drive, a memory card, or an optical disk.

[0143] Any combination of one or more storage media may be used. The storage medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, RAM, ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0144] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0145] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0146] The computer program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0147] The terminal of the present invention includes a processor and a memory.

[0148] The memory is used to store computer programs; preferably, the memory includes: ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk and other media that can store program codes.

[0149] The processor is connected to the memory and is used to execute the computer program stored in the memory, so that the terminal executes the above-mentioned vector topology integrated passenger flow calculation method based on the fisheye probe.

[0150] Preferably, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0151] Figure 7 Shown is a block diagram of an exemplary terminal 7 suitable for implementing embodiments of the present invention.

[0152] Figure 7 The terminal 7 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0153] like Figure 7As shown, the terminal 7 is in the form of a general-purpose computing device. Components of the terminal 7 may include, but are not limited to, one or more processors or processing units 71, a memory 72, and a bus 73 connecting different system components (including the memory 72 and the processing unit 71).

[0154] Bus 73 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of such architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MCA) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0155] The terminal 7 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the terminal 7, including volatile and non-volatile media, removable and non-removable media.

[0156] The memory 72 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 721 and / or cache memory 722. The terminal 7 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 723 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 Not shown, often called a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 73 via one or more data medium interfaces. Memory 72 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0157] A program / utility 724 having a set (at least one) of program modules 7241 may be stored, for example, in memory 72. Such program modules 7241 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 7241 generally implement the functions and / or methods of the embodiments described herein.

[0158] The terminal 7 may also communicate with one or more external devices 8 (e.g., a keyboard, a pointing device, a display 9, etc.), one or more devices that enable a user to interact with the terminal 7, and / or any device that enables the terminal 7 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 74. Furthermore, the terminal 7 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 75. Figure 7 As shown, the network adapter 75 communicates with other modules of the terminal 7 via the bus 73. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with the terminal 7, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0159] In summary, the present invention proposes a vector topology integrated passenger flow calculation method and system based on fisheye probes for the shopping mall passenger flow system based on the installation of fisheye probes, and proposes a complete passenger flow solution, which combines open source pedestrian recognition, tracking, and re-identification algorithms. A specific sampling method is used to capture the pedestrian trajectory. At the same time, according to the real position of the fisheye probe in three-dimensional space, the spatial topological relationship between the probes is constructed, and the vector pedestrian features are obtained by matching with the pedestrian tracking results. Based on the above sampling and topological logic, real-time sparse matching is performed. The overall coordination of the system is high, which effectively reduces the disadvantages brought by the fisheye probe. The overall robustness of the system can stably and effectively output passenger flow results; therefore, the present invention effectively overcomes the various shortcomings of the existing technology and has a high industrial utilization value.

[0160] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Anyone skilled in the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by one of ordinary skill in the art without departing from the spirit and technical principles disclosed herein are intended to be covered by the claims of the present invention.

Claims

1. A vector topology integrated passenger flow calculation method based on fisheye probe, characterized in that: The following steps are involved: Obtain the video stream data collected by the target fisheye probe; Based on the video stream data, obtaining a first pedestrian trajectory image; Acquire a target vector pedestrian feature corresponding to the first pedestrian trajectory image based on the first pedestrian trajectory image and the regional topology of the target fisheye probe; Based on the spatial topology of the target fisheye probe and the non-target fisheye probe, a topological search is performed on the target vector pedestrian feature to obtain a second pedestrian trajectory image; including: obtaining the non-target vector pedestrian feature corresponding to the non-target fisheye probe before the current moment; screening the non-target vector pedestrian feature to obtain a candidate vector pedestrian feature; performing feature sparse matching calculation on the target vector pedestrian feature and the candidate vector pedestrian feature, and selecting the candidate vector pedestrian feature with the highest sparse matching result as the trajectory matching result; correlating the trajectory matching result with the target vector pedestrian feature based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image; obtaining a passenger flow result based on the second pedestrian trajectory image; The feature sparse matching calculation formula is as follows: S 稀疏匹配分数 =a×S1+(1-a)×S2; Among them, a is an adjustable coefficient, S1 represents the average score of dense sampling, S2 represents the average score of sparse sampling, and S 稀疏匹配分数 The calculation formulas for the trajectory matching results, the dense sampling average score and the sparse sampling average score are as follows: Where n=1 means that the number of trajectories corresponding to the candidate vector pedestrian feature is 1, m means the maximum number of trajectories corresponding to the candidate vector pedestrian feature, S x Indicates the similarity of features at the same angle, S y Indicates the similarity of adjacent angle features, S 采样平均分数 Expressed as the sampling mean score; When the pedestrian information in the first pedestrian trajectory image is complete, dense sampling is performed on the first pedestrian trajectory image; When the pedestrian information in the first pedestrian trajectory image is incomplete, sparse sampling is performed on the first pedestrian trajectory image.

2. The vector topology integrated passenger flow calculation method based on fisheye probe according to claim 1 is characterized in that: The step of obtaining a first pedestrian trajectory image based on the video stream data comprises the following steps: Performing image preprocessing on the video stream data to obtain a perspective image; Detecting a pedestrian in the perspective image based on a target detection method, and obtaining a pedestrian torso candidate image corresponding to the pedestrian; When there is a pedestrian torso candidate image corresponding to the pedestrian at the previous moment, the pedestrian torso candidate image is associated with the pedestrian torso candidate image at the previous moment based on the tracking algorithm to obtain the first pedestrian trajectory image; if there is no pedestrian torso candidate image at the previous moment, the pedestrian torso candidate image is used as the first pedestrian trajectory image.

3. The vector topology integrated passenger flow calculation method based on fisheye probe according to claim 1 is characterized in that: The acquiring of a target vector pedestrian feature corresponding to the first pedestrian trajectory image based on the first pedestrian trajectory image and the regional topology of the target fisheye probe comprises the following steps: Sampling the first pedestrian trajectory image to obtain pedestrian features corresponding to the first pedestrian trajectory image; The pedestrian features are divided according to the regional topology of the target fisheye probe to obtain the target vector pedestrian features.

4. The vector topology integrated passenger flow calculation method based on fisheye probe according to claim 1 is characterized in that: The regional topology of the target fisheye probe is to divide the monitoring screen of the target fisheye probe into different areas according to the installation point and monitoring area of ​​the target fisheye probe, and then topologically connect the different areas; the spatial topology of the target fisheye probe and the non-target fisheye probe is to topologically connect the regional topology of the target fisheye probe with the regional topology of the non-target fisheye probe again.

5. A vector topology integrated passenger flow calculation system based on fisheye probe, characterized in that: include: Image collection module, first pedestrian trajectory image acquisition module, vector pedestrian feature module, pedestrian retrieval module, passenger flow output module; The image collection module is used to obtain video stream data collected by the target fisheye probe; The first pedestrian trajectory image acquisition module is used to acquire a first pedestrian trajectory image based on the video stream data; The vector pedestrian feature module is used to obtain a target vector pedestrian feature corresponding to the first pedestrian trajectory image based on the first pedestrian trajectory image and the regional topology of the target fisheye probe; The pedestrian retrieval module is used to perform a topological search on the target vector pedestrian feature based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image; including: obtaining the non-target vector pedestrian feature corresponding to the non-target fisheye probe before the current moment; screening the non-target vector pedestrian feature to obtain a candidate vector pedestrian feature; performing a feature sparse matching calculation on the target vector pedestrian feature and the candidate vector pedestrian feature, and selecting the candidate vector pedestrian feature with the highest sparse matching result as the trajectory matching result; correlating the trajectory matching result with the target vector pedestrian feature based on the spatial topology of the target fisheye probe and the non-target fisheye probe to obtain a second pedestrian trajectory image; The passenger flow output module is used to obtain passenger flow results based on the second pedestrian trajectory image; The feature sparse matching calculation formula is as follows: S 稀疏匹配分数 =a×S1+(1-a)×S2; Among them, a is an adjustable coefficient, S1 represents the average score of dense sampling, S2 represents the average score of sparse sampling, and S 稀疏匹配分数 The calculation formulas for the trajectory matching results, the dense sampling average score and the sparse sampling average score are as follows: Where n=1 means that the number of trajectories corresponding to the candidate vector pedestrian feature is 1, m means the maximum number of trajectories corresponding to the candidate vector pedestrian feature, S x Indicates the similarity of features at the same angle, S y Indicates the similarity of adjacent angle features, S 采样平均分数 Expressed as the sampling mean score; When the pedestrian information in the first pedestrian trajectory image is complete, dense sampling is performed on the first pedestrian trajectory image; When the pedestrian information in the first pedestrian trajectory image is incomplete, sparse sampling is performed on the first pedestrian trajectory image.

6. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the vector topology integrated passenger flow calculation method based on fisheye probe according to any one of claims 1 to 4 is implemented.

7. A terminal, characterized in that: include: processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program stored in the memory, so that the terminal executes the vector topology integrated passenger flow calculation method based on fisheye probe according to any one of claims 1 to 4.

Images