Pass following detection method based on pure vision multi-view image fusion

Abstract

The invention provides a pass following detection method based on pure vision multi-view image fusion, and the method comprises the steps: obtaining a pass region image synchronously collected by a multi-view monocular RGB camera array, so as to obtain video stream data of a plurality of different views; performing geometric transformation and ground plane mapping on the video stream data of each view angle to obtain ground plane track data; performing space-time geometric consistency analysis on the ground plane track data to determine the actual number of the passing targets, and performing passing following detection according to the actual number of the passing targets; therefore, high-precision anti-trailing detection can be realized with low-cost hardware.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method for detecting trailing behavior based on pure visual multi-view image fusion. Background Technology

[0002] With the continuous growth of global air passenger traffic, airport security is facing increasing pressure. As the first critical line of physical security, the intelligence level of security gates directly affects the airport's operational efficiency and security level. Especially in key areas such as boarding gates, staff access routes, and VIP lounges, preventing unauthorized personnel from illegally entering through tailgating or piggybacking is of paramount importance for security work.

[0003] Currently, the anti-tailgating detection technologies applied to access gates mainly fall into the following categories: (1) Infrared beam technology: Multiple pairs of infrared sensors are deployed in the channel to form a light curtain; the technology is mature, the cost is extremely low, and the response speed is fast, making it the mainstream configuration for current economic gates; however, it cannot identify the shape of objects and is easily interfered with by luggage, clothing, etc., leading to false alarms; two people standing close together, jumping, or walking slowly can easily deceive the sensors, resulting in missed alarms. The security level is low. (2) 2D vision technology: A single camera (usually top-view) is used for human detection and counting; as an upgraded version of the infrared solution, it is used in some mid-range products and can provide intelligent analysis capabilities; however, there are serious visual blind spots and occlusion problems. When two people are close together or side by side, they are easily identified as a single target in the 2D image, leading to missed detection. At the same time, it is sensitive to changes in lighting and shadows and has poor environmental adaptability. (3) 3D vision technology (ToF / structured light): This technology uses a specially designed 3D camera to acquire depth information of a scene for 3D analysis. It is currently the most advanced solution and is being applied in high-security scenarios such as airports. It achieves high accuracy through 3D contour analysis; however, the hardware cost is high. The price of 3D cameras and dedicated processing chips is several times or even dozens of times that of ordinary RGB cameras, which greatly limits its popularization in a wider market (such as office buildings, subways, and parks). At the same time, some low-end 3D solutions still face challenges when dealing with extreme scenarios such as close embraces and people stacked with luggage. Summary of the Invention

[0004] This application aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, this application proposes a method for detecting tailgating based on pure visual multi-view image fusion, which can achieve high-precision anti-tailgating detection with low-cost hardware.

[0005] To achieve the above objectives, this application proposes a method for detecting trailing traffic based on pure visual multi-view image fusion. The method includes the following steps: acquiring images of the passage area synchronously collected by a multi-view monocular RGB camera array to obtain video stream data from multiple different perspectives; performing geometric transformation and ground plane mapping on the video stream data from each perspective to obtain ground plane trajectory data; performing spatiotemporal geometric consistency analysis on the ground plane trajectory data to determine the actual number of passing targets, and performing trailing traffic detection based on the actual number of passing targets.

[0006] The tail-following detection method based on pure visual multi-view image fusion proposed in this invention has the advantage of enabling high-precision anti-tail-following detection with low-cost hardware.

[0007] In addition, the traffic tailing detection method based on pure visual multi-view image fusion proposed in the above embodiments of this application may also have the following additional technical features: Optionally, after obtaining video stream data from multiple different perspectives, the method further includes: generating a time stamp for each frame of the video stream data from multiple different perspectives to establish a time correspondence between the multi-view images.

[0008] Optionally, the multi-view monocular RGB camera array includes at least one top-view camera and two oblique-view cameras.

[0009] Optionally, geometric transformation and ground plane mapping are performed on the video stream data of each viewpoint to obtain ground plane trajectory data, including: pedestrian detection and tracking of the video stream data of each viewpoint to assign a target trajectory in the current viewpoint to each detected pedestrian; and the target trajectory of each viewpoint is projected onto a unified ground plane coordinate system using a homography matrix to obtain ground plane trajectory data.

[0010] Optionally, pedestrian detection and tracking are performed on the video stream data of each viewpoint to assign a target trajectory in the current viewpoint to each detected pedestrian. This includes: inputting the video stream data of each viewpoint into a pre-trained real-time target detection model to output the pedestrian bounding box for each viewpoint; adopting a detection-based tracking paradigm to predict the target's position in the next frame using a Kalman filter, and associating the detection box of the current frame with the existing trajectory using a Hungarian algorithm or IoU matching to assign a target trajectory in the current viewpoint to each detected pedestrian.

[0011] Optionally, a homography matrix is ​​used to project the target trajectory from each viewpoint onto a unified ground plane coordinate system to obtain ground plane trajectory data. This includes: placing a calibration plate on the gate floor in advance and taking photos from each camera viewpoint to calculate the homography transformation matrix between the image plane and the ground plane from each camera viewpoint using the OpenCV vision library; for the pedestrian bounding box tracked in each viewpoint, taking the foot position point and projecting it onto the ground plane coordinates through the homography transformation matrix to obtain ground plane trajectory data.

[0012] Optionally, the multi-view monocular RGB camera array includes 2-4 head-up cameras mounted on both sides of the channel.

[0013] Optionally, geometric transformation and ground plane mapping are performed on the video stream data of each viewpoint to obtain ground plane trajectory data, including: using inverse perspective transformation to fuse the images in the video stream data of each viewpoint into a unified ground plane top-view image; and performing pedestrian detection and tracking on the ground plane top-view image to obtain ground plane trajectory data.

[0014] Optionally, the image fusion employs a weighted feathering fusion algorithm to achieve a smooth transition in the stitching area.

[0015] Optionally, spatiotemporal geometric consistency analysis is performed on the ground plane trajectory data to determine the actual number of passing targets, and tailing detection is performed based on the actual number of passing targets. This includes: clustering trajectory points from different viewpoints and with similar physical locations on the ground plane under the same time marker to obtain fusion candidate clusters; calculating the centroid of all points within each fusion candidate cluster, and calculating the average or maximum distance from each point to the centroid; if the average or maximum distance is less than a preset compactness threshold, the geometric verification is considered consistent, and it is determined to be a fusion object; otherwise, it is considered that there are multiple physical entities within the cluster, and the cluster is split into multiple fusion objects according to the distribution of points within the cluster before a tailing alarm is triggered. For a fusion object that passes geometric verification, its bounding box in each original viewpoint is traced back and compared with a preset single-person model threshold. If it is identified as an abnormal body shape, a tailing alarm is triggered. After being identified as a normal body shape, a tracking filter spanning multiple frames is established for each fusion object to predict its position, velocity, and size at the next moment. If a stable single-person trajectory splits into two trajectories at time t, and the observation at time t-1 cannot explain this split, it is determined as a tailing event, and a tailing alarm is triggered. If the two trajectories merge into one trajectory at time t, and the pseudo-3D volume of the trajectory is equal to the sum of the two trajectories, it is marked as a close-fitting state, the multi-person count is maintained, and a tailing alarm is triggered. Attached Figure Description

[0016] Figure 1This is a flowchart illustrating the method for detecting tailing based on pure visual multi-view image fusion according to an embodiment of this application. Figure 2 This is a diagram illustrating the overall architecture of the system according to an embodiment of the present application, which is logically composed of a physical device layer and a core processing layer. Figure 3 This is a schematic diagram of the spatiotemporal geometric consistency fusion process according to an embodiment of this application; Figure 4 This is a schematic diagram of homography projection according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating the principle of inverse perspective transformation according to an embodiment of this application. Detailed Implementation

[0017] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0018] To better understand the above technical solutions, exemplary embodiments of this application will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of this application and to fully convey the scope of this application to those skilled in the art.

[0019] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0020] It should be noted that the method of this application is applied to a pedestrian tailing detection device based on pure visual multi-view image fusion, such as... Figure 2 As shown, the device of this application is logically composed of two main parts: the physical device layer and the core processing layer. Its overall workflow is universal and compatible with different specific implementation methods.

[0021] The physical equipment layer consists of a multi-view camera array containing N (N≥2) time-synchronized monocular RGB cameras. The physical deployment of this array is flexible and can be selected based on the site environment, such as a combination of "top-view + oblique-view" or a combination of "multi-channel eye-level wide-angle," with the common goal of achieving seamless coverage of the passageway. The core processing layer is responsible for intelligent analysis of the acquired image information, and its workflow is as follows: Image acquisition and synchronization module: responsible for receiving video streams from various cameras and adding a precise timestamp to each frame to ensure the time alignment of multi-view data.

[0022] Pedestrian detection and tracking module: Performs pedestrian detection independently on each video stream and tracks the target on its respective 2D image plane.

[0023] The geometric transformation and ground plane mapping module is key to standardization. It uses specific geometric transformations (such as homography projection or inverse perspective transformation) to unify data (detection points or the image itself) from different perspectives and in different forms into a shared ground plane coordinate system.

[0024] The spatiotemporal geometric consistency fusion module receives standardized ground plane data and uses complex algorithmic logic (such as clustering, associating, and triple-checking ground plane trajectories from different perspectives) to finally output accurate number of people and their status.

[0025] Decision and Alarm Module: Based on the fusion results and access control authorization status, make the final decision on whether to grant passage or trigger an alarm.

[0026] Figure 1 This is a flowchart illustrating a method for detecting trailing pedestrians based on pure visual multi-view image fusion according to an embodiment of this application. Figure 1 As shown, the method for detecting trailing pedestrians based on pure visual multi-view image fusion in this application includes the following steps: S101: Acquire images of the passage area simultaneously captured by a multi-view monocular RGB camera array to obtain video stream data from multiple different perspectives.

[0027] As an example, after obtaining video stream data from multiple different perspectives, the method further includes: generating a time stamp for each frame of the video stream data from multiple different perspectives to establish a time correspondence between the multi-view images.

[0028] In other words, the image acquisition and synchronization module ensures that the data processed by the subsequent fusion algorithm is strictly corresponding in the time dimension.

[0029] As a specific implementation, NTP (Network Time Protocol) or PTP (Precise Time Protocol) is used to synchronize the time of each camera or processing node. When hardware synchronization cannot be achieved, a software synchronization mechanism is used. After receiving each video frame, the module buffers and reorders them according to their arrival time and built-in timestamps, and outputs a group of synchronized frames with timestamp differences within the allowable error (e.g., 5ms).

[0030] S102, geometric transformation and ground plane mapping are performed on the video stream data of each viewpoint to obtain ground plane trajectory data.

[0031] There are several ways to perform geometric transformations and ground plane mapping on the video stream data from each viewpoint to obtain ground plane trajectory data.

[0032] As an example, a multi-view monocular RGB camera array includes at least one top-view camera and two oblique-view cameras.

[0033] In other words, in this embodiment, the "multi-view camera array" is specifically defined as a combination of at least one top-view (Cam-T) and two oblique-view (Cam-L, Cam-R) cameras.

[0034] Geometric transformation and ground plane mapping are performed on the video stream data of each viewpoint to obtain ground plane trajectory data. This includes: pedestrian detection and tracking of the video stream data of each viewpoint to assign a target trajectory in the current viewpoint to each detected pedestrian; and using a homography matrix to project the target trajectories of each viewpoint onto a unified ground plane coordinate system to obtain ground plane trajectory data.

[0035] Specifically, pedestrian detection and tracking are performed on the video stream data of each viewpoint to assign a target trajectory in the current viewpoint to each detected pedestrian. This includes: inputting the video stream data of each viewpoint into a pre-trained real-time target detection model to output the pedestrian bounding box for each viewpoint; adopting a detection-based tracking paradigm, predicting the target's position in the next frame using a Kalman filter, and associating the detection box of the current frame with the existing trajectory using a Hungarian algorithm or IoU matching to assign a target trajectory in the current viewpoint to each detected pedestrian.

[0036] As a specific implementation, a pedestrian detection and tracking module is used to identify and continuously track human targets in each independent 2D viewpoint. For pedestrian detection, lightweight real-time target detection algorithms such as YOLOv5s and MobileNet-SSD are employed. The model is pre-trained on pedestrian datasets (such as MOT and CrowdHuman) and can be fine-tuned according to the actual scene. For each frame, a pedestrian bounding box B_i = (x, y, w, h) is output, where i is the viewpoint index. For target tracking, a detection-by-detection paradigm is adopted. Algorithms such as SORT or DeepSORT are used to predict the target's position in the next frame using a Kalman filter. Then, the detection box in the current frame is associated with existing trajectories using the Hungarian algorithm or a simpler IoU matching, assigning a unique trajectory ID to each detected pedestrian in the current viewpoint.

[0037] Specifically, a homography matrix is ​​used to project the target trajectory from each viewpoint onto a unified ground plane coordinate system to obtain ground plane trajectory data. This includes: placing a calibration plate on the gate floor in advance and taking photos from each camera viewpoint to calculate the homography transformation matrix between the image plane and the ground plane from each camera viewpoint using the OpenCV vision library; for the pedestrian bounding box tracked in each viewpoint, taking the foot position point and projecting it onto the ground plane coordinates through the homography transformation matrix to obtain ground plane trajectory data.

[0038] As a specific embodiment, the geometric transformation and ground plane mapping module is implemented as a homography projection module; it projects the foot points of each 2D bounding box onto the ground plane through a homography matrix H, forming discrete trajectory points. (Hibgraphy projection illustration) Figure 4 As shown, a one-time calibration is first performed during system deployment. A calibration board (such as a checkerboard) is placed on the ground of the turnstile, and photos are taken from the perspective of each camera. Using vision libraries such as OpenCV, the homography transformation matrix H_i between the image plane and the ground plane from each camera's perspective can be calculated. During runtime, for the pedestrian bounding box B_i tracked in each viewpoint i, its foot position point p_i = (u, v) (usually the midpoint of the bottom edge of the bounding box) is taken. It is projected onto the ground plane coordinates P_i = (X, Y) using the formula P_i = H_i * p_i. This module outputs a set of ground plane trajectory points, where each point contains its source viewpoint, original trajectory ID, and timestamp information.

[0039] Therefore, the process in this example is relatively simple and the amount of calculation is small.

[0040] As another example, a multi-view monocular RGB camera array includes 2-4 head-up cameras mounted on either side of the channel.

[0041] In other words, in this embodiment, the "multi-view camera array" is specifically defined as installing 2 to 4 wide-angle monocular cameras on both sides of the channel, which are at eye level or slightly downward, to ensure that the fields of view are fully overlapped.

[0042] Geometric transformation and ground plane mapping are performed on the video stream data from each viewpoint to obtain ground plane trajectory data. This includes: using inverse perspective transformation to fuse the images in the video stream data from each viewpoint into a unified ground plane top-view image; performing pedestrian detection and tracking on the ground plane top-view image to obtain ground plane trajectory data; wherein, the image fusion adopts a weighted feathering fusion algorithm to achieve a smooth transition in the stitching area.

[0043] As a specific implementation, the geometric transformation and ground plane mapping module is actually implemented as a more powerful inverse perspective transformation and image fusion module. It directly processes the original image and outputs a complete God's-eye view. For example... Figure 5 As shown, the Inverse Perspective Transformation (IPM) aims to restore the perspective-projected image to a top view in the world coordinate system. The camera imaging process can be simplified to: p = K * [R|t] * P_w. Here, p is the pixel coordinate, P_w is the world coordinate, K is the camera intrinsic parameter, and [R|t] is the extrinsic parameter. When the ground is the Z_w=0 plane, this relationship can be simplified to a 3x3 homography matrix H. IPM is the inverse transformation H_ipm. p_pers = H * P_gp (from the ground plane to the camera) P_gp = H_ipm * p_pers (from the camera to the ground plane, H_ipm = H^-1).

[0044] Image fusion combines local top-down views from multiple cameras, after IPM transformation, in overlapping regions to generate a seamless, complete "carpet map." A commonly used fusion algorithm is weighted feathering fusion; where, at any point in the overlapping region, the final pixel value I_final = w_1 * I_1_warped + w_2 * I_2_warped. Here, w_1 and w_2 are weights, and w_1 + w_2 = 1. The magnitude of the weight depends on the distance of the point from its respective image center; the closer to the center, the higher the weight, thus achieving a smooth transition.

[0045] In this example, the detection and tracking module processes the top-down view image output by the geometric transformation and ground plane mapping module. The pedestrian detection and tracking algorithm is then run directly on this stitched, uniform image.

[0046] Therefore, this example solves the installation limitations that may have existed in the previous example and provides higher detection accuracy.

[0047] S103 performs spatiotemporal geometric consistency analysis on the ground plane trajectory data to determine the actual number of passing targets, and performs passing tail detection based on the actual number of passing targets.

[0048] As an embodiment, spatio-temporal geometric consistency analysis is performed on the ground plane trajectory data to determine the actual number of passing targets, and passing following detection is performed according to the actual number of passing targets, including: clustering trajectory points that are from different perspectives and are physically close in the ground plane position under the same time identifier to obtain a fusion candidate cluster; calculating the centroid of all points within each fusion candidate cluster, and calculating the average distance or the maximum distance from each point to the centroid; if the average distance or the maximum distance is less than the preset compactness threshold, it is considered that the geometric verification is consistent and determined as a fusion object; otherwise, it is considered that there are multiple physical entities within the cluster, and it is split into multiple fusion objects according to the distribution of the points within the cluster and then following alarms are issued; for a fusion object that passes the geometric verification, trace back its bounding box in each original perspective and compare it with the preset single-person model threshold, and issue a following alarm if it is identified as an abnormal body shape; after being identified as a normal body shape, establish a tracking filter that spans multiple frames for each fusion object to predict its position, speed, and size at the next moment. If a stable single-person trajectory splits into two trajectories at time t and the observations at time t-1 cannot explain this split, it is determined as a following event and a following alarm is issued; if two trajectories merge into one trajectory at time t and the pseudo-three-dimensional volume of this trajectory is equal to the sum of the two trajectories, it is marked as a close state, the multiple-person count is maintained, and a following alarm is issued.

[0049] As a specific embodiment, as Figure 3 shown, it includes the following steps: (1) Trajectory association (clustering): Cluster trajectory points that are from different perspectives and are physically close in the ground plane position under the same time stamp t. The association basis is the Euclidean distance on the ground plane.

[0050] Formula: For any two trajectory points P_i = (X_i, Y_i) and P_j = (X_j, Y_j), their distance Dist(P_i, P_j) = sqrt((X_i - X_j)^2 + (Y_i - Y_j)^2). When Dist < D_thresh, the two points are considered related.

[0051] (2) Geometric consistency verification: Verify each "fusion candidate cluster". The projections of a single physical entity in multiple perspectives should form a compact cluster on the ground plane.

[0052] Index: Calculate the centroid of all points within the cluster, and calculate the average distance or the maximum distance from each point to the centroid. If this value is less than the compactness threshold C_thresh, it is considered that the geometry is consistent and determined as a fusion object.

[0053] Processing: If there is a discrepancy, it is assumed that there are multiple physical entities within the cluster, and the cluster is split into multiple fused objects based on the distribution of points within the cluster (e.g., using K-Means clustering).

[0054] (3) Pseudo-3D volume estimation: to help distinguish between “fat man” and “two people standing close together”, as well as “person” and “luggage”.

[0055] Principle: For a fused object that has passed geometric verification, backtrack its 2D bounding box B_i = (x, y, w, h) in each original viewpoint (especially eye level or oblique view).

[0056] Formula: Volume_Proxy = Area_GP * Height_AVG. Where Area_GP is the area of ​​the convex hull of the object's projection point on the ground plane, and Height_AVG is the average value of its bounding box height h in each original viewpoint. Abnormal body shapes can be identified by comparing Volume_Proxy and Height_AVG with preset thresholds for single-person models.

[0057] (4) Time consistency verification: Utilize the continuity of the target's motion to make a final decision in the time dimension.

[0058] Principle: A tracking filter (such as a Kalman filter) is established for each fused object across multiple frames to predict its position, velocity, and size at the next moment.

[0059] Split detection: If a stable single-person trajectory T_A splits into two trajectories T_A1 and T_A2 at time t, and the observation at time t-1 cannot explain this split, it is determined to be a trailing event.

[0060] Merging detection: If two trajectories T_A and T_B merge into a single trajectory T_C at time t, and the pseudo-3D volume of T_C is approximately equal to the sum of T_A and T_B, the system will mark it as "closely attached" and maintain multi-person counting.

[0061] Therefore, this application uses a widely available, low-cost monocular RGB camera as the sole sensor, resulting in extremely low hardware costs. Through innovative software algorithms, it achieves highly robust detection in complex scenes, reaching performance far exceeding traditional 2D solutions and approaching that of high-end 3D solutions. Through multi-view data fusion, information obstructed from one viewpoint can be compensated for by other viewpoints. Both detection point fusion and image-level fusion effectively decompose occlusion. This application provides at least two implementation methods. One can choose a combination of "top view + oblique view," or a completely top-viewless "multi-level wide-angle" combination. The latter, by generating a virtual God's-eye view, completely eliminates dependence on ceiling installation conditions, greatly expanding application scenarios. Ground plane projection and spatiotemporal consistency verification mechanisms ensure that detection results do not depend on the instantaneous state of a single image, but are based on continuous observation over a spatiotemporal period, providing stronger resistance to interference such as sudden changes in illumination and shadow movement. The "spatiotemporal geometric consistency fusion module" constructs a "pseudo-3D" verification system without requiring real 3D point clouds, achieving accurate counting and state estimation of passing targets.

[0062] In summary, the tailgating detection method based on pure visual multi-view image fusion according to the embodiments of this application aims to achieve high-precision anti-tailgating detection with low-cost hardware. This application includes at least two implementation methods. The first implementation method uses a combination of top-view and oblique-view cameras, and projects the detection points from each viewpoint onto the ground plane for fusion through homography transformation. The second implementation method uses multiple level-view wide-angle cameras, which are stitched together into a complete "virtual God's-eye view" image through inverse perspective transformation (IPM) technology before detection. Both methods ultimately rely on the core "spatiotemporal geometric consistency fusion module," which associates, verifies, and fuses the target trajectory on the ground plane. Through triple verification of geometric consistency, pseudo-3D volume estimation, and temporal consistency, the actual number of people passing through is accurately determined. Thus, this application effectively solves the problems of missed detection and false detection caused by occlusion in traditional 2D vision solutions, avoids the high hardware cost of 3D vision solutions, and provides flexible deployment options to adapt to different installation environments.

[0063] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0064] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0065] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0066] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0067] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. This application can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0068] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0069] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0070] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0071] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0072] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0073] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. The illustrative expressions of the above terms in this specification should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0074] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A method for tailgating detection based on pure vision multi-view image fusion, characterized in that, The method comprises the following steps: Obtaining images of a passing area collected synchronously by a multi-view monocular RGB camera array to obtain video stream data of multiple different views; Performing geometric transformation and ground plane mapping on the video stream data of each view to obtain ground plane trajectory data; Performing spatio-temporal geometric consistency analysis on the ground plane trajectory data to determine the actual number of passing targets and performing passing tail detection according to the actual number of passing targets.

2. The method of claim 1, wherein the method is based on pure vision multi-view image fusion for tailgating detection. After obtaining the video stream data of multiple different views, the method further comprises: Generating a time identifier for each frame of image in the video stream data of multiple different views to establish a time correspondence relationship between multi-view images.

3. The method of claim 1, wherein the method is based on pure vision multi-view image fusion for tailgating detection. The multi-view monocular RGB camera array comprises at least one overhead camera and two oblique cameras.

4. The method of claim 3, wherein the method is based on pure vision multi-view image fusion for tailgating detection. The method of performing geometric transformation and ground plane mapping on the video stream data of each view to obtain ground plane trajectory data comprises: Performing pedestrian detection and tracking on the video stream data of each view to assign a target trajectory under the current view to each detected pedestrian; Projecting the target trajectories of each view into a unified ground plane coordinate system by using a homography matrix to obtain the ground plane trajectory data.

5. The method of claim 4, wherein the method is based on pure vision multi-view image fusion for tailgating detection. The method of performing pedestrian detection and tracking on the video stream data of each view to assign a target trajectory under the current view to each detected pedestrian comprises: Inputting the video stream data of each view into a pre-trained real-time target detection model to output a pedestrian bounding box of each view; Using a detection-based tracking paradigm to predict the position of a target in the next frame by using a Kalman filter and to associate the detection frame of the current frame with an existing trajectory by using a Hungarian algorithm or an IoU matching to assign a target trajectory under the current view to each detected pedestrian.

6. The method of claim 4, wherein the method is based on pure vision multi-view image fusion for tailgating detection. The method of projecting the target trajectories of each view into a unified ground plane coordinate system by using a homography matrix to obtain the ground plane trajectory data comprises: Placing a calibration board on the ground of a gate and taking a photo from each camera view to calculate a homographic transformation matrix between the image plane under each camera view and the ground plane by using an OpenCV vision library; Projecting the foot position point of the pedestrian bounding box tracked in each view to the ground plane coordinate by using the homographic transformation matrix to obtain the ground plane trajectory data.

7. The method of claim 1, wherein the method is based on pure vision multi-view image fusion for tailgating detection. The multi-view monocular RGB camera array comprises 2-4 overhead cameras installed on both sides of a channel.

8. The method of claim 7, wherein the method is based on pure vision multi-view image fusion for tailgating detection. The method of performing geometric transformation and ground plane mapping on the video stream data of each view to obtain ground plane trajectory data comprises: Fusing the images in the video stream data of each view into a unified ground plane god view image by using inverse perspective transformation; Performing pedestrian detection and tracking on the ground plane god view image to obtain the ground plane trajectory data.

9. The method of claim 8, wherein the method is based on pure vision multi-view image fusion for tailgating detection. The image fusion adopts a weighted feathering fusion algorithm to achieve smooth transition in a splicing area.

10. The method of claim 2, wherein the method is based on pure vision multi-view image fusion for tailgating detection. The method of performing spatio-temporal geometric consistency analysis on the ground plane trajectory data to determine the actual number of passing targets and performing passing tail detection according to the actual number of passing targets comprises: Trajectory points from different views and similar physical locations on the ground plane are clustered to obtain fusion candidate clusters under the same time label; The centroid of all points in each fusion candidate cluster is calculated, and the average distance or maximum distance of each point to the centroid is calculated; If the average distance or maximum distance is less than the preset compactness threshold, it is considered that the geometric verification is consistent, and it is determined as a fusion object; otherwise, it is considered that there are multiple physical entities in the cluster, which are split into multiple fusion objects according to the distribution of points in the cluster, and tailing alarm is performed; For a fusion object that passes the geometric verification, the bounding box of the fusion object in each original view is traced back, and compared with the preset single person model threshold. If it is identified as an abnormal body shape, tailing alarm is performed; After being identified as a normal body shape, a tracking filter spanning multiple frames is established for each fusion object to predict its position, speed and size at the next time. If a stable single person trajectory is split into two trajectories at time t, and the observation value at time t-1 cannot explain the split, it is determined as a tailing event, and tailing alarm is performed. If two trajectories are merged into one at time t, and the pseudo three-dimensional volume of the trajectory is equal to the sum of the two trajectories, it is marked as close state, the multi-person count is maintained, and tailing alarm is performed.

Images