Low-view-angle aerial view sensing method and system applied to pedestrian gate

Abstract

The invention provides a low-view-angle aerial view perception method and system applied to a pedestrian gate. The method comprises the following steps: acquiring a plurality of low-view-angle scene images; performing feature extraction on the scene image to obtain a 2D feature map; performing geometric structure processing on the 2D feature map to obtain a BEV feature map with 3D geometric structure information; performing context information processing on the 2D feature map to obtain a BEV feature map with context information; fusing the two BEV feature maps to obtain a target BEV feature map; and performing data enhancement processing on the target BEV feature map to obtain a target BEV map. By applying the method provided by the invention, the 3D BEV feature map with geometric structure information and context information can be obtained, finally data enhancement processing is carried out to obtain a smooth and high-quality BEV map, and the method effectively improves the BEV perception precision, robustness and system deployability in a low-view scene.

Description

Technical Field

[0001] This invention relates to the field of BEV sensing technology, and in particular to a low-angle bird's-eye view sensing method and system for pedestrian turnstiles. Background Technology Bird's-eye-view (BEV) based perception technology has achieved significant results in fields such as autonomous driving by providing a unified, unobstructed scene representation. However, when directly transferring this technology to near-field dense scenes with limited hardware and unique perspectives (such as pedestrian turnstiles and security passages), existing methods have obvious limitations, mainly in the following aspects: Perspective challenges in extreme scenarios: In near-field, high-density, narrow-channel environments, cameras are typically mounted at low heights (approximately 1 meter), causing drastic changes in the scale and shape of targets (such as pedestrians) in the image, making it difficult to maintain geometric consistency. Simultaneously, severe occlusion, or even complete overlap, between pedestrians makes projection methods relying solely on visual appearance highly susceptible to failure. Furthermore, unidirectional viewing angles lead to incomplete information, further increasing perceptual uncertainty. Summary of the Invention

[0002] This invention aims to at least solve one of the technical problems existing in the prior art. Therefore, one objective of this invention is to propose a low-view bird's-eye view perception method and system for pedestrian turnstiles. This method, through geometric structure processing and contextual information processing of 2D feature maps, can generate high-precision and high-completeness BEV perception results even when the image is occluded.

[0003] A low-angle bird's-eye view perception method for pedestrian turnstiles, the method comprising: Acquire scene images of the target scene from multiple low-angle perspectives, captured by the image acquisition device at the current moment; Feature extraction is performed on the scene image to obtain a 2D feature map; The 2D feature map is input into the trained BEV perception framework to perform a pre-defined image processing procedure to obtain the target BEV map. The preset image processing procedure includes: Geometric structure processing is performed on the 2D feature map to obtain the first BEV feature map, which is a BEV feature map with 3D geometric structure information. Contextual information processing is performed on the 2D feature map to obtain a second BEV feature map, which is a BEV feature map with contextual information. The first BEV feature map and the second BEV feature map are fused to obtain the target BEV feature map; Data augmentation processing is performed on the target BEV feature map to obtain the target BEV map.

[0004] Optionally, the above method may further include: The target BEV map is input into at least one preset task detection device to obtain the environmental perception result of the target scene at the current moment. The environmental perception result includes at least the motion trajectory of the target object in the target scene at the current moment and / or the current environmental data of the target scene.

[0005] Optionally, the process of training the BEV perception framework, as described above, includes: Construct a digital twin scene corresponding to the target scene, and obtain simulation data of the digital twin scene under different environmental parameters. Each of the simulation data is an image with BEV tags. Acquire unlabeled video streams from realistic low-angle scenes; The BEV perception framework is iteratively trained using simulation data and unlabeled video streams to obtain the total loss function corresponding to the BEV perception framework. If the current iteration of training has not ended, adjust the parameters of the BEV perception framework based on the total loss function; Once the current iteration of training has ended, complete the training of the BEV perception framework.

[0006] Optionally, the above method involves iteratively training the BEV perception framework using simulation data and unlabeled video streams to obtain the total loss function corresponding to the BEV perception framework, including: The simulation data and the unlabeled video stream are input into the BEV perception framework to obtain the first prediction result corresponding to the simulation data and the second prediction result corresponding to the unlabeled video stream output by the BEV perception framework. The BEV labels of the simulation data are compared with the first prediction result, and the supervision loss is calculated based on the comparison result. The supervision loss includes detection loss and segmentation loss. Self-supervised loss calculations were performed on the unlabeled video stream and the second prediction result to obtain the photometric consistency loss corresponding to the unlabeled video stream and the cross-frame feature optical flow anchoring loss corresponding to the second prediction result. Based on detection loss, segmentation loss, photometric consistency loss, and cross-frame feature optical flow anchoring loss, the total loss function corresponding to the BEV perception framework is obtained.

[0007] Optionally, the above method involves performing geometric structure processing on the 2D feature map to obtain the first BEV feature map, including: A preset depth prediction network is applied to predict the probability distribution of the depth of each pixel in the 2D feature map along the ray direction corresponding to the image acquisition device to which the 2D feature map belongs. The outer product operation is performed on the 2D feature map and the probability distribution of each pixel to obtain multiple 3D feature points. Each 3D feature point is used to characterize the depth position of the planar features of the 2D feature map in 3D space. Each 3D feature point is projected onto a pre-set 3D voxel space to obtain a BEV space feature map. The BEV spatial feature map is compressed according to a preset compression direction to obtain a first BEV feature map, which is a BEV planar map with geometric structure information.

[0008] Optionally, the above method may involve performing contextual information processing on the 2D feature map to obtain a second BEV feature map, including: Obtain a pre-defined BEV query grid, which consists of multiple query points, each representing a location in the BEV space; The BEV query grid is used as the Query, and the 2D feature map is used as the Key and Value input to the preset deformable cross attention module to obtain the second BEV feature map with contextual information output by the deformable cross attention module. The deformable cross-attention module is used to predict multiple sampling points and the offset of each sampling point on the 2D feature map for each query point; based on each sampling point and its offset, feature sampling is performed for each query point to obtain the attention weight of each sampling point according to the sampled features; the attention weights are weighted and summed to output a second BEV feature map with context information.

[0009] Optionally, in the above method, fusing the first BEV feature map and the second BEV feature map to obtain the target BEV feature map includes: The first BEV feature map and the second BEV feature map are concatenated to obtain a concatenated feature map; The spliced ​​feature map is input into a preset fusion network for deep fusion to obtain the target BEV feature map.

[0010] Optionally, the above method involves performing data augmentation processing on the target BEV feature map to obtain the target BEV map, including: The target BEV feature map is input into a preset cyclic spatiotemporal memory fusion module, which then fuses the BEV feature map with historical BEV feature maps to obtain the target BEV map.

[0011] A low-angle bird's-eye view perception system for pedestrian turnstiles includes: Multiple image acquisition devices are used to synchronously acquire scene images of the target scene from a low angle and send the scene images to the edge computing unit; the image acquisition range of each image acquisition device covers the entire target scene, and the image acquisition ranges of some or all image acquisition devices have overlapping fields of view; The edge computing unit is used to execute the aforementioned low-angle bird's-eye view perception method applied to pedestrian turnstiles.

[0012] Optionally, in the above system, each image acquisition device synchronously acquires scene images of the target scene at a low angle via the PTP protocol; or, Each image acquisition device includes a master device and multiple slave devices; when the master device triggers an image acquisition command, it broadcasts the image acquisition command to each slave device, so that the master device and each slave device can synchronously acquire scene images of the target scene from a low angle.

[0013] Compared with the prior art, the present invention has the following advantages: This invention provides a low-view bird's-eye view perception method for pedestrian turnstiles. The method includes: acquiring scene images of a target scene from multiple low-view perspectives, captured by an image acquisition device at a current moment; extracting features from the scene images to obtain 2D feature maps; inputting the 2D feature maps into a trained BEV perception framework to perform a preset image processing procedure to obtain a target BEV map; wherein, performing the preset image processing procedure includes: performing geometric structure processing on the 2D feature maps to obtain a first BEV feature map, wherein the first BEV feature map is a BEV feature map with 3D geometric structure information; performing context information processing on the 2D feature maps to obtain a second BEV feature map, wherein the second BEV feature map is a BEV feature map with context information; fusing the first BEV feature map and the second BEV feature map to obtain a target BEV feature map; and performing data augmentation processing on the target BEV feature map to obtain the target BEV map.

[0014] In the low-view bird's-eye view perception method for pedestrian turnstiles provided by this invention, the BEV perception framework is first trained using simulated data and real unlabeled video streams. During training, the BEV perception framework is jointly optimized through supervised and self-supervised processes, enabling it to complete the basic training process based on labeled simulated data. Simultaneously, it undergoes self-optimization and scene adaptation by incorporating unlabeled video streams to obtain a higher-precision BEV perception framework. After training, the BEV perception framework is applied to a real target scene. Multiple image acquisition devices are deployed in the target scene to acquire scene images in real time. After obtaining the scene images at the current moment, feature extraction is performed to obtain 2D feature maps. The 2D feature maps are input into the BEV perception framework. Through the dual-stream heterogeneous view conversion engine within the BEV perception framework, explicit geometric prior stream and implicit context aggregation stream processing are performed on the 2D feature maps to obtain 3D BEV feature maps with geometric structure information and contextual information. Finally, through the cyclic spatiotemporal memory fusion module within the BEV perception framework, historical data is fused with the current BEV feature map to obtain a smooth, high-quality BEV map. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of a low-angle bird's-eye view perception method for pedestrian turnstiles, provided by an embodiment of the present invention. Figure 2 This is a schematic diagram illustrating the training process of the BEV perception framework provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the structure of the dual-stream heterogeneous view conversion engine provided in an embodiment of the present invention; Figure 4 A flowchart of an explicit geometric prior flow provided in an embodiment of the present invention; Figure 5 A flowchart of the implicit context aggregation flow provided in an embodiment of the present invention; Figure 6 A global flowchart of each execution stage provided in the embodiments of the present invention; Figure 7 This is a system structure diagram of a low-angle bird's-eye view perception system for pedestrian turnstiles, provided as an embodiment of the present invention.

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] In this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0019] This invention can be used in multiple scenarios, such as low-angle scenes captured by image acquisition devices in intelligent pedestrian passages, subways, buildings, and park turnstiles.

[0020] This invention provides a low-view bird's-eye view perception method for pedestrian turnstiles. The method is applied to an edge computing unit, and the flowchart of the method is as follows. Figure 1 Specifically, it includes: S1: Acquire scene images of the target scene from multiple low-angle perspectives at the current moment, captured by the image acquisition device; S2: Extract features from the scene image to obtain a 2D feature map; S3: Input the 2D feature map into the trained BEV perception framework to perform a preset image processing procedure to obtain the target BEV map; The preset image processing procedure includes: Geometric structure processing is performed on the 2D feature map to obtain the first BEV feature map, which is a BEV feature map with 3D geometric structure information. Contextual information processing is performed on the 2D feature map to obtain a second BEV feature map, which is a BEV feature map with contextual information. The first BEV feature map and the second BEV feature map are fused to obtain the target BEV feature map; Data augmentation processing is performed on the target BEV feature map to obtain the target BEV map.

[0021] Based on the content of steps S1-S3 above, it can be seen that in the low-angle bird's-eye view perception method for pedestrian turnstiles provided in this embodiment of the invention, step S3 includes the following implementation process: S31: Perform geometric structure processing on the 2D feature map to obtain a first BEV feature map, wherein the first BEV feature map is a BEV feature map with 3D geometric structure information. S32: Perform context information processing on the 2D feature map to obtain a second BEV feature map, wherein the second BEV feature map is a BEV feature map with context information; S33: Fuse the first BEV feature map and the second BEV feature map to obtain the target BEV feature map; S34: Perform data augmentation processing on the target BEV feature map to obtain the target BEV map.

[0022] Based on the above steps S1-S3 and S31-S34, the following specific explanations are provided: S1: Acquire scene images of the target scene at the current moment from multiple low-angle perspectives, captured by the image acquisition device.

[0023] The scene images captured by the image acquisition device can be a single photograph or at least one video frame extracted from a video stream. Image acquisition devices can be digital cameras, industrial cameras, camcorders, webcams, scanners, etc. To meet the requirements of a wide near-field perspective, a fixed-focus lens with a large field of view can be selected. To clearly capture pedestrian dynamics, the image acquisition device needs to support 1080p resolution and a frame rate of 30fps, and preferably employs global shutter technology to avoid the "jelly effect" artifacts caused by fast-moving pedestrians. All image acquisition devices need to undergo precise joint calibration of intrinsic and extrinsic parameters before use to obtain their accurate spatial pose in a unified world coordinate system, which is the basis for subsequent geometric view transformation algorithms.

[0024] Each image acquisition device covers the entire target scene, and some or all of the image acquisition devices have overlapping fields of view. For example, each gate in a smart pedestrian passage is equipped with four industrial-grade RGB cameras. The cameras are grouped and installed on the pillars on both sides of the gate's entrance and exit, at a height of approximately 1.0-1.2 meters. Each camera is angled towards the center of the passage and slightly tilted downwards, forming a stereoscopic visual layout with overlapping fields of view covering the entire passage area. When acquiring images, each image acquisition device needs to complete exposure and imaging synchronously. To ensure that all image acquisition devices expose and image at exactly the same time, a hardware-level time synchronization solution can be adopted. For example, network synchronization can be achieved through an Ethernet switch supporting the IEEE 1588 Precision Time Protocol (PTP), or the exposure signal of the master image acquisition device can be broadcast to all slave image acquisition devices via a physical trigger cable, achieving microsecond-level synchronization accuracy.

[0025] S2: Extract features from the scene image to obtain a 2D feature map.

[0026] In this invention, after acquiring multiple images, each image acquisition device sends them to a multi-view image encoder with shared weights to extract a high-dimensional 2D feature map containing rich semantic information. This multi-view image encoder is a neural network model with a CNN (Convolutional Neural Networks) structure, specifically a ResNet (Residual Neural Network).

[0027] S3: Input the 2D feature map into the trained BEV perception framework to perform a preset image processing procedure to obtain the target BEV map.

[0028] It should be noted that the BEV perception framework of this invention is Orthogonal View-space Geometry and Context Inference (OV-GCI). This framework, trained offline, utilizes a large model. After training the BEV perception framework, this model is applied online to perform image processing on 2D feature maps. The trained BEV perception framework includes a Heterogeneous View Transformation Engine (DHV-Engine) and a Recurrent Spatiotemporal Memory Fusion Module (RSTM-Module). The DHV-Engine combines structural processing and contextual information into the 2D feature maps to obtain a first BEV feature map with 3D geometric structure information and a second BEV feature map with contextual information. The RSTM-Module then fuses the two BEV feature maps and performs data augmentation to obtain the target BEV feature map, thus completing the image processing.

[0029] Based on the above steps S1-S3, it can be seen that the implementation process of the low-angle bird's-eye view perception method for pedestrian turnstiles provided in this embodiment of the invention involves an offline training stage and an online application stage. The offline training stage involves training the BEV perception framework, and the online application stage involves image acquisition, feature extraction, view transformation, and temporal memory fusion. The view transformation and temporal memory fusion processes are image processing processes performed using the BEV perception framework that has been trained.

[0030] (I) Offline Training Phase (Reference) Figure 2 ): Construct a digital twin scene corresponding to the target scene and acquire simulation data of the digital twin scene under different environmental parameters. Each simulation data is an image with a BEV label. Acquire unlabeled video streams in real low-view scenes. Apply the simulation data and unlabeled video streams to iteratively train the BEV perception framework to obtain the total loss function corresponding to the BEV perception framework. If the current iteration training number is not completed, adjust the parameters of the BEV perception framework based on the total loss function. If the current iteration training number is completed, complete the training of the BEV perception framework. Each training process is as follows: Simulated data and unlabeled video streams are input into the BEV perception framework to obtain the first prediction result corresponding to the simulated data and the second prediction result corresponding to the unlabeled video stream. The BEV labels of the simulated data are compared with the first prediction result, and the supervised loss, including detection loss and segmentation loss, is calculated based on the comparison result. Self-supervised loss is calculated for both the unlabeled video stream and the second prediction result to obtain the photometric consistency loss corresponding to the unlabeled video stream and the cross-frame feature optical flow anchoring loss corresponding to the second prediction result. Based on the detection loss, segmentation loss, photometric consistency loss, and cross-frame feature optical flow anchoring loss, the total loss function corresponding to the BEV perception framework is obtained.

[0031] In this invention, the digital twin scene can be constructed using professional 3D simulation software (such as CARLA, Unreal Engine, etc.). A high-fidelity digital twin scene is built based on various scene parameters (such as geometric dimensions, materials, and lighting environment) of the target scene (such as a specific turnstile channel). During the acquisition of simulation data, environmental parameters of the virtual environment in the digital twin scene (such as lighting conditions, congestion level, pedestrian appearance, and behavioral paths) can be changed to simulate the changes of the target scene in a real environment, and corresponding simulation data is collected according to different environmental parameters. Each set of simulation data contains images from virtual multi-cameras and perfectly matched, pixel-level accurate BEV spatial ground truth (i.e., BEV labels), such as 2D / 3D detection boxes, instance segmentation maps, and motion trajectories of pedestrians. The unlabeled video stream in the real low-view scene is video data collected from the real target scene or other real environments similar to the target scene, and this data does not have BEV labels.

[0032] After obtaining simulation data and unlabeled data, the BEV perception framework is trained. The training process of the BEV perception framework is divided into two branches: a self-supervised loss branch and a supervised loss branch. All losses calculated through the two branches are weighted and summed to obtain the total loss function. Backpropagation is then performed based on the total loss function to optimize the parameters of the BEV perception framework, thereby fine-tuning the framework and further improving the accuracy of data processing. Training ends when multiple training iterations are performed or the total loss function converges.

[0033] For the supervised loss branch: After inputting simulation data into the BEV perception framework, the first prediction result output by the BEV perception framework is the label prediction result of the BEV perception framework on the simulation data. The BEV labels in the simulation data generated in the original digital twin scene are compared with the first prediction result to determine the difference between them, and the supervised loss is calculated based on the difference, including detection loss and segmentation loss. The formula for calculating the detection loss is as follows:

[0034] in, As an improved cross-entropy loss, it addresses the problem of extreme imbalance between positive and negative samples (target and background) in detection tasks. It reduces the weight of correctly classified simple samples, allowing the BEV perception framework to focus more on learning difficult-to-classify samples. The parameters, such as the center point coordinates and size of the regression prediction detection box, are used to calculate the sum of the absolute values ​​of the differences between the predicted and the true values. As an improved Intersection over Union (IoU) loss, compared to the traditional IoU loss, it not only cares about the overlap area between the predicted box and the ground truth box, but also the distance and shape between them. It can provide effective gradients even when the two boxes do not overlap, which helps to converge faster and more stably. To balance the hyperparameters of different loss terms.

[0035] The formula for calculating the segmentation loss is as follows:

[0036] in, Dice Loss (image segmentation loss) is used to measure the degree of overlap between the predicted mask and the real mask, and is suitable for handling foreground / background imbalance. We use pixel-level cross-entropy loss to ensure supervised stability for each pixel category. The optional boundary alignment loss constrains the predicted edges to fit the true contour, improving detail quality. These are the hyperparameters that balance the loss of each segmentation, and can be tuned according to the scenario (when there are no boundary constraints, they can be set as follows). ).

[0037] For the self-supervised loss branch: After inputting the unlabeled video stream into the BEV perception framework, the second prediction result output by the BEV perception framework is the prediction result of the BEV perception framework for label prediction of each video frame in the unlabeled video stream (i.e., identifying the target detection object and background region in the video frame). Since each video frame of the unlabeled video stream does not carry relevant data labels, but the temporal consistency present in the video stream can serve as a supervision signal. Therefore, after inputting the unlabeled video stream into the BEV perception framework, not only is the second prediction result output, but some intermediate products are also exposed, such as the "predicted depth map" used to calculate photometric loss, and the "BEV temporal features" used to calculate feature consistency. At this time, a pre-set photometric consistency loss algorithm can be applied to perform self-supervised loss calculation on the unlabeled video stream, and the pixel differences between two adjacent video frames in the video stream can be identified through this algorithm. For example: if the 3D structure of the scene is known (which can be approximated by the depth map predicted by the BEV perception framework), then the image I_{t-1} of the previous frame should be able to synthesize an image that is very similar to the current frame I_t after 3D transformation and reprojection. The pixel difference between these two images is the photometric consistency loss. Simultaneously, a cross-frame feature optical flow anchoring loss algorithm can be applied to calculate the self-supervised loss of the second prediction result, specifically targeting the static background region. The high-dimensional features of static background regions in the BEV space should theoretically remain unchanged across consecutive frames. The cross-frame feature optical flow anchoring loss is achieved by calculating the difference in static region features between two consecutive BEV feature maps. It acts as an "anchor," stabilizing the feature representation space of the BEV perception framework and making it more robust to changes in illumination.

[0038] The formula for calculating the photometric consistency loss algorithm is as follows:

[0039] in, They represent the current time. and the previous moment Real camera images. The frame is the current image. Predicted pixel-level dense depth map. This represents a reprojection function. It utilizes the predicted depth map. Using the known camera intrinsic and extrinsic parameters, the image from the previous moment... Each pixel is "distorted" or "rendered" to the current moment based on its 3D position. From the camera's perspective, a composite image of the current state is generated. . It is the L1 norm, used to calculate the sum of the absolute values ​​of all pixel differences between two images, as a measure of their inconsistency.

[0040] The calculation formula for the cross-frame feature optical flow anchoring loss algorithm is as follows:

[0041] This represents all pixels in the BEV space that are identified as static background. Take the expected value (or average value). This indicates the current state of the BEV perception framework. For BEV pixels The extracted high-dimensional feature vector. This represents a coordinate transformation function. Since the camera and scene are fixed, theoretically the BEV feature maps between two frames are aligned. This function is used to handle possible minor jitter or correction errors. It is the square of the L2 norm, i.e., the square of the Euclidean distance, used to measure the difference between the feature vector of the current frame and the transformed feature vector of the previous frame.

[0042] In this invention, after obtaining various loss functions (e.g., detection loss) Segmentation loss Loss of photometric uniformity and cross-frame feature optical flow anchoring loss Then, the individual losses are weighted and summed to obtain the total loss function. The formula for calculating the total loss function is as follows:

[0043] (include: , , and The ) corresponds to the learnable parameters of the task, representing the uncertainty (or noise level) of the task. This represents the weighted average of the loss. The significance of this term lies in the fact that the more uncertain the BEV perception framework's prediction for a certain task (i.e., the more uncertain it is...), the more likely it is to result in a weighted average. The larger the value, the smaller the weight of the task loss, thus dynamically reducing the gradient impact of "difficult" or "noisy" tasks during training. This is a regularization term used to prevent all... The values ​​increase without limit (which would cause the total loss to approach 0, making learning impossible). Minimizing this is equivalent to encouraging the framework as a whole to reduce uncertainty for all tasks and seek more confident predictions.

[0044] In the offline training phase of this invention, the BEV perception framework is jointly optimized using real and simulated data, as well as through supervised and self-supervised learning. On one hand, ideal labeled data is generated on a large scale in a simulated environment to pre-train the framework's basic capabilities. On the other hand, self-supervised signals such as the designed "photometric consistency loss" and "cross-frame feature optical flow anchoring loss" enable the framework to continuously self-optimize and adapt to scenes using unlabeled real video streams, greatly reducing the reliance on expensive manual annotation. In addition, the loss obtained through a combination of supervised and self-supervised methods is used to optimize all learnable parameters of the framework through a unified backpropagation. This joint optimization strategy allows the framework to learn accurate semantic knowledge from simulated data and generalize to cope with the complexities of the real world from unlabeled real data, greatly improving the framework's accuracy.

[0045] (II) Online Application Stage (corresponding to steps S1~S3 provided in the embodiments of the present invention): 1) Image acquisition: Acquire scene images captured by image acquisition devices.

[0046] 2) Feature extraction: Extract image features from scene images through a multi-view image encoder to obtain 2D feature maps.

[0047] 3) View Transformation: The dual-stream heterogeneous engine in the BEV perception framework is applied to transform the 2D feature map into a feature representation from the BEV's perspective. (Reference) Figure 3 The dual-stream heterogeneous view conversion engine contains two parallel execution paths: an explicit geometric prior flow and an implicit context aggregation flow. These two execution paths are used to implement the processes corresponding to steps S31 and S32 above, in order to obtain a first BEV feature map with geometric structure information and a second BEV feature map with context information. Finally, the two feature maps are fused to obtain the target BEV feature map.

[0048] 4) Temporal memory fusion: The target BEV feature map obtained in 3) above is augmented by the cyclic spatiotemporal memory fusion module to obtain the target BEV map.

[0049] This invention addresses view transformation and timing-based fusion processes in online application phases, and provides the following specific details: For the explicit geometric prior flow of the dual-stream heterogeneous view transformation engine, it strictly follows the "camera-world" projection geometry relationship. Its workflow is similar to a "pseudo-LiDAR" system, aiming to construct a geometrically reliable BEV feature basis. Therefore, as Figure 4 As shown, the process of performing geometric structure processing on the 2D feature map to obtain the first BEV feature map is as follows: S311: Apply a preset depth prediction network to predict the probability distribution of the depth of each pixel in the 2D feature map along the ray direction corresponding to the image acquisition device to which the 2D feature map belongs.

[0050] A depth prediction network can be a lightweight neural network that predicts a probability distribution of a series of discrete depth values ​​along the corresponding camera ray direction for each pixel in the input 2D feature map. This differs from directly regressing a single depth value; instead, it provides a probability distribution, making it more adaptable to uncertainty.

[0051] S312: Perform an outer product operation on the 2D feature map and the probability distribution of each pixel to obtain multiple 3D feature points.

[0052] Each 3D feature point is used to represent the depth position of the planar features of the 2D feature map in 3D space. An outer product operation is performed on the 2D feature map and the probability distribution of each pixel to map the planar features of the 2D image to different depth positions in 3D space according to their depth probabilities, thus forming a series of weighted 3D feature points.

[0053] S313: Project each 3D feature point onto a pre-set 3D voxel space to obtain a BEV space feature map.

[0054] Voxel sputtering is used to project or sputter individual 3D feature points into a predefined, unified global 3D voxel space V. The same voxel mesh may receive multiple point features from different viewpoints and different pixels, which are then summed together.

[0055] S314: Compress the BEV spatial feature map according to the preset compression direction to obtain the first BEV feature map.

[0056] The first BEV feature map is a BEV planar map with geometric structure information. The compression direction can be the Z-axis direction. In step S314, the 3D voxel space V, which contains all scene information, is redundant for the BEV task and is compressed. Specifically, a learnable compression module (such as a 1x1 3D convolution) is used to compress and fuse the features in the vertical (Z-axis) direction, ultimately "flattening" it into a BEV planar map, outputting a BEV feature map containing rich 3D geometric structure information.

[0057] Based on the above steps S311-S314, the calculation formula corresponding to the explicit geometric prior flow in this embodiment of the invention is as follows:

[0058]

[0059] This is a high-dimensional feature vector extracted from a pixel by the backbone network of a 2D feature map. For a lightweight network, the first pixel along its camera ray direction is predicted. The probability of a discrete depth interval. . It is a "context-aware" 3D point feature obtained by weighted summation of depth probabilities. It is no longer an isolated 2D feature, but a 3D feature containing depth information. In the final constructed global 3D voxel space, located at coordinates The characteristics contained in voxels. This represents all 3D feature points that fall within the voxel after coordinate transformation. Features Perform sum pooling.

[0060] The implicit context aggregation stream of the dual-stream heterogeneous view transformation engine borrows the "query-key-value" concept from Transformer, aggregating information in a more flexible, top-down manner to compensate for the shortcomings of EGP streams in occluded and weakly textured areas. Therefore, as... Figure 5 As shown, the process of performing contextual information processing on the 2D feature map to obtain the second BEV feature map includes: S321: Obtain the pre-defined BEV query grid.

[0061] The BEV query grid consists of multiple query points, each representing a location in the BEV space.

[0062] In the BEV query grid, each query point is responsible for retrieving the required information from the image.

[0063] S322: Input the BEV query grid as the Query and the 2D feature map as the Key and Value into the preset deformable cross-attention module to obtain the second BEV feature map with contextual information output by the deformable cross-attention module.

[0064] The deformable cross-attention module is used to predict multiple sampling points K on the 2D feature map for each query point and the offset of each sampling point; based on each sampling point and its offset, feature sampling is performed for each query point to obtain the attention weight of each sampling point according to the sampled features; the attention weights are weighted and summed to output a second BEV feature map with context information.

[0065] Based on the above steps S321 and S322, the calculation formula corresponding to the implicit context aggregation stream in this embodiment of the invention is as follows:

[0066] The coordinates in BEV space are The grid cells are ultimately aggregated into the feature vector. This represents a pre-defined, learnable query grid in the BEV space. This is its updated state. The total number of image acquisition devices. The number of sampling points sampled per image for each BEV query point is a preset hyperparameter (such as 4 or 8). Indicates the network is the first The attention weights are dynamically learned for each sampling point, and the points with larger weights contribute more to the final result. Indicates the first 2D image feature map of an image acquisition device. Represents BEV grid points A corresponding 3D reference point is projected onto the camera using camera parameters. Two-dimensional coordinates on an image. Indicates the network is the first The two-dimensional offset learned by each sampling point. This allows the sampling points to "jump out" of their fixed projection positions and flexibly search for useful information in other parts of the image, which is the core of deformable attention.

[0067] This invention combines explicit geometric prior flow and implicit context aggregation flow in parallel through an original "Dual-Stream Heterogeneous View Conversion Engine (DHV-Engine)". The former provides a reliable geometric foundation, while the latter compensates for deficiencies in occlusion and contextual information. The complementary advantages of the two enable this invention to generate high-precision and high-completeness BEV perception results even under low-viewpoint and severe occlusion conditions, far exceeding any single technical approach.

[0068] like Figure 3As shown, after obtaining the first and second BEV feature maps through explicit geometric prior flow and implicit context aggregation flow respectively, the two are fused and augmented to obtain a BEV map with geometric structure information and contextual information. Specifically, the fusion process involves concatenating the first and second BEV feature maps to obtain a concatenated feature map; this concatenated feature map is then input into a pre-defined fusion network for deep fusion to obtain the target BEV feature map. The fusion network can be a convolutional network. The resulting fused BEV feature map ensures both the geometric realism of the physical world and possesses strong contextual understanding and robustness against occlusion.

[0069] After converting the 2D feature map into a target BEV feature map with geometric structure information and contextual information through the dual-stream heterogeneous view conversion engine, in order to make the output BEV map smoother and obtain a more stable and smoother perception result in time, a temporal memory fusion process is used to fuse historical data with the currently output BEV feature map, providing high-quality input for subsequent BEV map applications.

[0070] Specifically, during the temporal memory fusion process, the target BEV feature map is input into a preset cyclic spatiotemporal memory fusion module. The cyclic spatiotemporal memory fusion module then fuses the BEV feature map with historical BEV feature maps to obtain the target BEV map. The cyclic spatiotemporal memory fusion module acts like a sponge with short-term memory, absorbing current information and fusing it with historical memory, effectively suppressing jitter in single-frame detection or missed detections caused by brief occlusions.

[0071] Optionally, in the online application stage of this invention, after obtaining the BEV graph through the temporal memory fusion process, a multi-task prediction and output process, as well as downstream application and decision-making processes, can be further executed. For example... Figure 6As shown, the multi-task prediction and output process involves inputting the target BEV map output by the cyclic spatiotemporal memory fusion module into multiple parallel task prediction heads. These prediction heads are task detection devices, each responsible for different perception tasks (e.g., detecting the movement trajectory of pedestrians in the target scene at the current moment and / or detecting the current environmental data of the target scene), and outputting perception results related to the perception task, such as outputting the pedestrian's BEV detection box (position and size), segmentation mask, and cross-frame tracking ID. The downstream application and decision-making process uses the output perception results to serve the upper-layer application logic. For example, a Trajectory State Machine (TSM) can determine whether there are abnormal behaviors such as tailgating, going against traffic, or intrusion (Behavior Analysis Zone, BAZ) based on the pedestrian's real-time trajectory, speed, and mutual positional relationships, and make decisions accordingly, such as triggering an audible and visual alarm or remaining inactive.

[0072] The low-view bird's-eye view perception method for pedestrian turnstiles provided in this invention first trains the BEV perception framework using simulated data and real unlabeled video streams. During training, a supervised and self-supervised learning process is used for joint optimization, enabling the BEV perception framework to complete basic training based on labeled simulated data. Simultaneously, it incorporates unlabeled video streams for self-optimization and scene adaptation, resulting in a higher-precision BEV perception framework. After training, the BEV perception framework is applied to a real-world target scene. Multiple image acquisition devices are deployed in the target scene to acquire real-time scene images. After obtaining the scene images at the current moment, feature extraction is performed to obtain 2D feature maps. These 2D feature maps are input into the BEV perception framework and processed by a dual-stream heterogeneous view conversion engine. This process involves explicit geometric prior stream and implicit context aggregation stream processing to obtain a 3D BEV feature map with geometric and contextual information. Finally, a cyclic spatiotemporal memory fusion module fuses historical data with the current BEV feature map to obtain a smooth, high-quality BEV map. The entire technical solution of this invention is specifically designed to solve the problem of low-viewpoint perception. It does not rely on expensive sensors such as LiDAR, and can achieve excellent results using only ordinary RGB cameras. Therefore, this invention is characterized by low cost and easy deployment, and can be widely applied to various real-world scenarios where ideal high-level deployment is not feasible, such as various turnstiles, security checkpoints, and passageways.

[0073] Optional, such as Figure 6As shown, after completing the entire online application phase, this invention can also perform a corresponding closed-loop self-optimization phase for the BEV perception framework. That is, after inputting the processing results output by the BEV perception framework in the online application phase (e.g., the target BEV feature map in this invention) into the multi-task prediction head to obtain the corresponding perception results, the perception results are applied to optimize the trained BEV perception framework. During the execution of the online application phase in this embodiment, the trained BEV perception framework does not merely output results unidirectionally. It continuously mines and backfeeds "hard cases" encountered in real-world scenarios, such as samples where the model prediction is inaccurate or uncertain. These unlabeled real data are calculated using a specific self-supervised loss function (e.g., cross-frame feature optical flow anchoring loss FFA-Loss) to generate gradient signals for model optimization. This allows for incremental fine-tuning of the pre-trained BEV perception framework online or offline without manual annotation, enabling it to continuously adapt to changes in the real-world environment and forming a continuously self-improving data closed loop.

[0074] and Figure 1 Corresponding to the method described above, this embodiment of the invention also provides a low-angle bird's-eye view perception system for use in pedestrian turnstiles, for... Figure 1 The specific implementation of the method is illustrated in the schematic diagram of the low-angle bird's-eye view perception system applied to pedestrian turnstiles provided in this embodiment of the invention, as follows: Figure 7 As shown, it specifically includes: Multiple image acquisition devices 100 are used to synchronously acquire scene images of the target scene from a low angle and send the scene images to the edge computing unit 200; the image acquisition range of each image acquisition device 100 covers the entire target scene, and the image acquisition ranges of some or all image acquisition devices have overlapping fields of view.

[0075] Specifically, each image acquisition device 100 synchronously acquires scene images of the target scene from a low angle via the PTP protocol; or, each image acquisition device 100 includes a master device and multiple slave devices; when the master device triggers an image acquisition command, it broadcasts the image acquisition command to each slave device, so that the master device and each slave device synchronously acquire scene images of the target scene from a low angle.

[0076] The edge computing unit 200 is used to execute the specific process of a low-angle bird's-eye view perception method for pedestrian turnstiles provided in the above embodiments.

[0077] The system also includes auxiliary equipment, including an industrial-grade PoE (Power over Ethernet) switch, which provides data transmission and stable power to each image acquisition device 100 and edge computing unit 200 simultaneously.

[0078] The specific working process of the image acquisition device 100 and the edge computing unit 200 in the low-angle bird's-eye view perception system for pedestrian turnstiles disclosed in the above embodiments of the present invention can be found in the corresponding content of the low-angle bird's-eye view perception method for pedestrian turnstiles disclosed in the above embodiments of the present invention, and will not be repeated here.

[0079] In this invention, the edge computing unit 200 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a programmable logic device, or other general-purpose or special-purpose hardware processing unit.

[0080] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0081] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both.

[0082] To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described in terms of functionality above. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0083] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A low-angle bird's-eye view perception method applied to pedestrian turnstiles, characterized in that, The method includes: Acquire scene images of the target scene from multiple low-angle perspectives, captured by the image acquisition device at the current moment; Feature extraction is performed on the scene image to obtain a 2D feature map; The 2D feature map is input into the trained BEV perception framework to perform a pre-defined image processing procedure to obtain the target BEV map. The preset image processing procedure includes: Geometric structure processing is performed on the 2D feature map to obtain the first BEV feature map, which is a BEV feature map with 3D geometric structure information. Contextual information processing is performed on the 2D feature map to obtain a second BEV feature map, which is a BEV feature map with contextual information. The first BEV feature map and the second BEV feature map are fused to obtain the target BEV feature map; Data augmentation processing is performed on the target BEV feature map to obtain the target BEV map.

2. The low-angle bird's-eye view perception method for pedestrian turnstiles according to claim 1, characterized in that, The method further includes: The target BEV map is input into at least one preset task detection device to obtain the environmental perception result of the target scene at the current moment. The environmental perception result includes at least the motion trajectory of the target object in the target scene at the current moment and / or the current environmental data of the target scene.

3. The low-angle bird's-eye view perception method for pedestrian turnstiles according to claim 1, characterized in that, The process of training the BEV perception framework includes: Construct a digital twin scene corresponding to the target scene, and obtain simulation data of the digital twin scene under different environmental parameters. Each of the simulation data is an image with BEV tags. Acquire unlabeled video streams from realistic low-angle scenes; The BEV perception framework is iteratively trained using simulation data and unlabeled video streams to obtain the total loss function corresponding to the BEV perception framework. If the current iteration of training has not ended, adjust the parameters of the BEV perception framework based on the total loss function; Once the current iteration of training has ended, complete the training of the BEV perception framework.

4. The low-angle bird's-eye view perception method for pedestrian turnstiles according to claim 3, characterized in that, The application simulation data and unlabeled video stream are used to iteratively train the BEV perception framework to obtain the total loss function corresponding to the BEV perception framework, including: The simulation data and the unlabeled video stream are input into the BEV perception framework to obtain the first prediction result corresponding to the simulation data and the second prediction result corresponding to the unlabeled video stream output by the BEV perception framework. The BEV labels of the simulation data are compared with the first prediction result, and the supervision loss is calculated based on the comparison result. The supervision loss includes detection loss and segmentation loss. Self-supervised loss calculations were performed on the unlabeled video stream and the second prediction result to obtain the photometric consistency loss corresponding to the unlabeled video stream and the cross-frame feature optical flow anchoring loss corresponding to the second prediction result. Based on detection loss, segmentation loss, photometric consistency loss, and cross-frame feature optical flow anchoring loss, the total loss function corresponding to the BEV perception framework is obtained.

5. The low-angle bird's-eye view perception method for pedestrian turnstiles according to claim 1, characterized in that, The geometrical processing of the 2D feature map to obtain the first BEV feature map includes: A preset depth prediction network is applied to predict the probability distribution of the depth of each pixel in the 2D feature map along the ray direction corresponding to the image acquisition device to which the 2D feature map belongs. The outer product operation is performed on the 2D feature map and the probability distribution of each pixel to obtain multiple 3D feature points. Each 3D feature point is used to characterize the depth position of the planar features of the 2D feature map in 3D space. Each 3D feature point is projected onto a pre-set 3D voxel space to obtain a BEV space feature map. The BEV spatial feature map is compressed according to a preset compression direction to obtain a first BEV feature map, which is a BEV planar map with geometric structure information.

6. The low-angle bird's-eye view perception method for pedestrian turnstiles according to claim 1, characterized in that, Contextual information processing is performed on the 2D feature map to obtain the second BEV feature map, including: Obtain a pre-defined BEV query grid, which consists of multiple query points, each representing a location in the BEV space; The BEV query grid is used as the Query, and the 2D feature map is used as the Key and Value input to the preset deformable cross attention module to obtain the second BEV feature map with contextual information output by the deformable cross attention module. The deformable cross-attention module is used to predict multiple sampling points and the offset of each sampling point on the 2D feature map for each query point; based on each sampling point and its offset, feature sampling is performed for each query point to obtain the attention weight of each sampling point according to the sampled features; the attention weights are weighted and summed to output a second BEV feature map with context information.

7. The low-angle bird's-eye view perception method for pedestrian turnstiles according to claim 1, characterized in that, The step of fusing the first BEV feature map and the second BEV feature map to obtain the target BEV feature map includes: The first BEV feature map and the second BEV feature map are concatenated to obtain a concatenated feature map; The spliced ​​feature map is input into a preset fusion network for deep fusion to obtain the target BEV feature map.

8. The low-angle bird's-eye view perception method for pedestrian turnstiles according to claim 1, characterized in that, The step of performing data augmentation processing on the target BEV feature map to obtain the target BEV map includes: The target BEV feature map is input into a preset cyclic spatiotemporal memory fusion module, which then fuses the BEV feature map with historical BEV feature maps to obtain the target BEV map.

9. A low-angle bird's-eye view perception system for pedestrian turnstiles, characterized in that, include: Multiple image acquisition devices are used to synchronously acquire scene images of the target scene from a low angle and send the scene images to the edge computing unit; the image acquisition range of each image acquisition device covers the entire target scene, and the image acquisition ranges of some or all image acquisition devices have overlapping fields of view; An edge computing unit is used to execute the low-angle bird's-eye view perception method for pedestrian turnstiles as described in any one of claims 1-8.

10. The low-angle bird's-eye view perception system for pedestrian turnstiles according to claim 9, characterized in that, Each image acquisition device synchronously acquires scene images of the target scene from a low angle via the PTP protocol; or, Each image acquisition device includes a master device and multiple slave devices; when the master device triggers an image acquisition command, it broadcasts the image acquisition command to each slave device, so that the master device and each slave device can synchronously acquire scene images of the target scene from a low angle.

Images