Multi-view crowd tracking method, device and equipment with view-ground interaction and medium

Abstract

The invention discloses a multi-view crowd tracking method and device with view-ground interaction, equipment and a medium. The method comprises the following steps: projecting a multi-scale single-view feature to a ground plane to obtain a current frame ground feature; interacting the multi-view feature of the previous frame and the ground feature of the previous frame to generate a tracking query; determining a current frame feature query based on the current frame ground feature, and performing time sequence interaction on the tracking query and the current frame feature query to obtain a current frame tracking query; motion offset and a crowd thermodynamic map are predicted for multi-view crowd tracking based on a current frame tracking query and a current frame feature query. According to the method, the tracking query is generated by interacting the multi-view feature of the previous frame and the ground feature of the previous frame, and the tracking query and the ground feature of the current frame are interacted to generate the tracking query of the current frame, so that interaction between the ground feature of the current frame and the previous ground feature and interaction between the ground feature and the view feature are realized; and the tracking stability of severely shielded and complex scenes is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a multi-view crowd tracking method, device, equipment and medium with viewpoint-ground interaction. Background Technology

[0002] Multi-view crowd tracking aims to continuously estimate the identity and movement trajectory of pedestrians by observing the same scene from multiple cameras within a unified spatial representation (typically a ground plane or a bird's-eye view, BEV). This technology has significant application value in fields such as intelligent transportation, public safety, and smart cities.

[0003] Existing multi-view crowd tracking methods are mainly based on convolutional neural network (CNN) architectures, which perform target detection from different perspectives and complete the association within a unified space. While these methods perform well in simple scenarios, they are prone to problems such as trajectory interruption and identity switching in real-world scenarios with dense pedestrians, severe occlusion, and long time spans, thus limiting their practical application capabilities.

[0004] Therefore, the existing technology still needs to be improved and enhanced. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a multi-view crowd tracking method, device, equipment and medium with viewpoint-ground interaction, which addresses the shortcomings of the existing technology.

[0006] To address the aforementioned technical problems, the first aspect of this application provides a multi-view crowd tracking method with viewpoint-ground interaction, wherein the multi-view crowd tracking method with viewpoint-ground interaction specifically includes: Acquire the multi-view image of the current frame, as well as the multi-view features and ground features of the previous frame; Extract multi-scale single-view features from each single-view image in the multi-view image, and project the multi-scale single-view features onto the ground plane to obtain the ground features of the current frame; The multi-view features and ground features of the previous frame are interacted to generate a tracking query; Based on the ground features of the current frame, a feature query for the current frame is determined. Based on the location of the crowd in the previous frame, the tracking query and the feature query for the current frame are interacted in a time sequence to obtain the tracking query for the current frame. Based on the current frame, the tracking query predicts motion offset, and based on the current frame features, the query predicts crowd heatmap; The position of the crowd in the current frame is predicted based on the motion offset and the crowd heatmap to achieve multi-view crowd tracking.

[0007] The multi-view crowd tracking method with viewpoint-ground interaction, wherein projecting multi-scale single-view features onto the ground plane to obtain ground features of the current frame specifically includes: For each voxel of a single-scale single-view feature in a multi-scale single-view feature, the vertex coordinates of the voxel are projected onto the image plane to obtain the pixel coordinates corresponding to each vertex coordinate. Based on the pixel coordinates, multi-height bilinear sampling is used to determine the feature value corresponding to the pixel coordinates to obtain the single-view ground feature corresponding to the single-scale single-view feature. All single-view ground features corresponding to each scale are fused to obtain the single-scale ground features of the current frame, and all single-scale ground features of the current frame are used as the ground features of the current frame.

[0008] The multi-view crowd tracking method with viewpoint-ground interaction, wherein the interaction between the multi-view features of the previous frame and the ground features of the previous frame to generate a tracking query specifically includes: The previous frame feature query is determined based on the ground features of the previous frame; Discrete sampling is performed on the previous frame feature query based on the crowd position in the previous frame to construct an initial tracking query; A viewpoint query is constructed based on the multi-view features of the previous frame, and a cross-attention mechanism is used to interact the initial tracking query and the viewpoint query to generate a tracking query.

[0009] The multi-view crowd tracking method with viewpoint-ground interaction, wherein the step of using a cross-attention mechanism to interact the initial tracking query and the viewpoint query to generate a tracking query specifically includes: The initial tracking query and the view query are refined using a feedforward neural network to obtain a refined tracking query and a refined view query. The refined tracking query is used as the query vector, and the refined perspective query is used as the key vector and value vector. The tracking query is generated through a cross-attention mechanism.

[0010] The multi-view crowd tracking method with viewpoint-ground interaction is described above, wherein the previous frame feature query and the current frame feature query are both generated by a Transformer encoder.

[0011] The multi-view crowd tracking method with viewpoint-ground interaction, wherein the step of performing a temporal interaction between the tracking query and the current frame feature query based on the crowd position of the previous frame to obtain the current frame tracking query specifically includes: Driven by the tracking query of the previous frame and with the target position of the previous frame as the reference point, a multi-scale deformable attention structure is used to model the temporal correspondence between the feature query of the previous frame and the feature query of the current frame in order to generate the tracking query of the current frame.

[0012] The multi-view crowd tracking method with viewpoint-ground interaction, wherein the multi-view features of the previous frame include single-scale single-view features of each single-view image; the generation process of the multi-view features of the previous frame specifically includes: Obtain multi-scale single-view features of each previous single-view image in the previous frame of multi-view image. The multi-scale single-view features of each previous single-view image are input into the feature pyramid network, and the single-scale single-view features of the previous single-view image are output through the feature pyramid network to generate the multi-view features of the previous frame.

[0013] A second aspect of this application provides a multi-view crowd tracking device with viewpoint-ground interaction, wherein the multi-view crowd tracking device with viewpoint-ground interaction specifically includes: The feature extraction and multi-view fusion module is used to acquire the current frame multi-view image, the previous frame multi-view features and the previous frame ground features, and to extract the multi-scale single-view features of each single-view image in the multi-view image, and project the multi-scale single-view features onto the ground plane to obtain the current frame ground features. The multi-view tracking encoding module is used to interact with the multi-view features of the previous frame and the ground features of the previous frame to generate a tracking query, and to determine the feature query of the current frame based on the ground features of the current frame. The multi-view tracking decoding module is used to perform temporal interaction between the tracking query and the current frame feature query based on the crowd position in the previous frame to obtain the current frame tracking query, predict motion offset based on the current frame tracking query, predict crowd heat map based on the current frame feature query, and predict the current frame crowd position based on the motion offset and the crowd heat map, so as to realize multi-view crowd tracking.

[0014] A third aspect of this application provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the multi-view crowd tracking method with view-ground interaction as described above.

[0015] A fourth aspect of this application provides a terminal device, which includes: a processor and a memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps in any of the above-described multi-view crowd tracking methods with viewpoint-ground interaction.

[0016] Beneficial effects: 1. This application generates a tracking query by interacting with the multi-view features of the previous frame and the ground features of the previous frame. Then, the tracking query is interacted with the ground features of the current frame to generate a tracking query for the current frame. This achieves information interaction between the ground features of the current frame and the ground features of the previous frame, as well as information interaction between the ground features and the view features, thereby improving tracking stability in severely occluded and complex scenes.

[0017] 2. This application combines motion offset and crowd heatmap when performing crowd tracking, which can further improve the accuracy of crowd trajectory prediction and reduce identity switching rate and tracking loss.

[0018] 3. This application introduces the Transformer architecture into large-scale multi-view crowd tracking, making full use of its ability to model complex spatial-temporal relationships, thereby significantly improving the performance of multi-view crowd tracking in large-scale and complex scenarios. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart of a multi-view crowd tracking method with viewpoint-ground interaction provided in an embodiment of this application.

[0021] Figure 2 A flowchart illustrating the principle of a multi-view crowd tracking method with viewpoint-ground interaction provided in this application embodiment.

[0022] Figure 3 A schematic diagram of a multi-view crowd tracking device with viewpoint-ground interaction provided in an embodiment of this application.

[0023] Figure 4 A schematic block diagram of the terminal device provided in the embodiments of this application. Detailed Implementation

[0024] This application provides a multi-view crowd tracking method, apparatus, device, and medium with viewpoint-ground interaction. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0025] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0026] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0027] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0028] Research has shown that multi-view crowd tracking aims to continuously estimate pedestrian identities and movement trajectories by observing the same scene from multiple cameras within a unified spatial representation (typically a ground plane or a bird's-eye view, BEV). This technology has significant application value in fields such as intelligent transportation, public safety, and smart cities.

[0029] Existing multi-view crowd tracking methods are mainly based on convolutional neural network (CNN) architectures, which perform target detection from different perspectives and complete the association within a unified space. While these methods perform well in simple scenarios, they are prone to problems such as trajectory interruption and identity switching in real-world scenarios with dense pedestrians, severe occlusion, and long time spans, thus limiting their practical application capabilities.

[0030] To address the aforementioned issues, in this embodiment, the following steps are taken: First, a multi-view image of the current frame, multi-view features of the previous frame, and ground features of the previous frame are acquired. Second, multi-scale single-view features of each single-view image in the multi-view image are extracted, and these features are projected onto the ground plane to obtain the ground features of the current frame. Third, the multi-view features of the previous frame and the ground features of the previous frame are interacted to generate a tracking query. Fourth, a current frame feature query is determined based on the current frame ground features. Finally, the tracking query and the current frame feature query are time-series interacted based on the crowd position of the previous frame to obtain a current frame tracking query. Fifth, motion offset is predicted based on the current frame tracking query, and a crowd heatmap is predicted based on the current frame feature query. Sixth, the current frame crowd position is predicted based on the motion offset and the crowd heatmap to achieve multi-view crowd tracking. This application's embodiments generate a tracking query by interacting with the multi-view features and ground features of the previous frame. This tracking query is then interacted with the ground features of the current frame to generate a tracking query for the current frame. This achieves information interaction between the current frame's ground features and the previous frame's ground features, as well as between ground features and viewpoint features, thereby improving tracking stability in severely occluded and complex scenes. Furthermore, by combining motion offset and crowd heatmaps when performing crowd tracking, the accuracy of crowd trajectory prediction can be further improved, reducing identity switching rates and tracking loss.

[0031] The application content will be further explained below with reference to the accompanying drawings and the description of the embodiments.

[0032] This embodiment provides a multi-view crowd tracking method with viewpoint-ground interaction. This method can apply a multi-view crowd tracking model, which includes a feature extraction and multi-view fusion module, a multi-view tracking encoding module, and a multi-view tracking decoding module. The feature extraction and multi-view fusion module includes a feature extraction unit and a projection unit. The feature extraction unit is used to extract multi-scale single-view features from each single-view image in the multi-view image, and to determine the single-scale view features of the single-view image based on the multi-scale single-view features of each single-view image to obtain multi-view features. The projection unit is used to project the multi-scale single-view features onto the ground plane to obtain the ground features of the current frame. The multi-view tracking encoding module includes a Transform encoder and a viewpoint encoder. - The ground interaction unit, the Transform encoder, is used to determine the current frame feature query based on the current frame ground features. The view-ground interaction unit is used to interact with the previous frame multi-view features and the previous frame ground features to generate a tracking query. The multi-view tracking decoding module includes an offset decoding branch and a crowd heatmap decoding branch. The offset decoding branch is used to perform temporal interaction between the tracking query and the current frame feature query based on the crowd position in the previous frame to obtain the current frame tracking query and predict motion offset based on the current frame tracking query. The crowd heatmap decoding branch is used to predict the crowd heatmap based on the current frame feature query and predict the current frame crowd position based on the motion offset and the crowd heatmap to achieve multi-view crowd tracking.

[0033] like Figure 1 and 2 As shown, the multi-view crowd tracking method with viewpoint-ground interaction provided in this embodiment specifically includes: S10. Obtain the current frame multi-view image, the previous frame multi-view features, and the previous frame ground features. Extract the multi-scale single-view features of each single-view image in the multi-view image, and project the multi-scale single-view features onto the ground plane to obtain the current frame ground features.

[0034] Specifically, the current frame multi-view image is obtained by capturing the same scene. For example, multiple calibrated camera angles can be deployed in a scene, and the scene can be captured by capturing images from these multiple calibrated camera angles to obtain the current frame multi-view image. The number of camera angles can be set according to the actual scene requirements, such as 2, 4, 8, etc.

[0035] The multi-view features and ground features of the previous frame are determined during multi-view crowd tracking based on the previous frame's multi-view image. For the current frame's multi-view image, the multi-view features and ground features of the previous frame are known. That is, when performing multi-view crowd tracking based on the previous frame's multi-view image, after acquiring the multi-view features and ground features of the previous frame, these features can be stored so that they can be directly retrieved when performing multi-view crowd tracking based on the current frame's multi-view image. The current frame's multi-view image and the previous frame's multi-view image can be view images acquired at two consecutive acquisition times, or view images acquired at a preset time interval. The specific settings can be determined according to the frame rate requirements of the actual application scenario. For example, if one multi-view image is acquired every 100ms, then the time interval between the current frame and the previous frame is 100ms.

[0036] Multi-scale single-view features are obtained by extracting multi-scale features from each single-view image in a multi-view image. For example, features can be extracted from single-view images using pre-trained feature extraction networks (such as ResNet, VGG, etc.) to obtain the corresponding multi-scale single-view features. For example, sampling time... The corresponding current frame multi-view image is The process of extracting multi-scale single-view features from each single-view image in the current frame's multi-view image can be represented as: , in, express Time The single-view characteristic of camera perspective, Represents multi-scale quantities. Indicates the number of camera angles. This represents a feature extraction network. Indicates the camera view index. express Time Single-view image from the camera's perspective. Indicates scale index.

[0037] It should be noted that, as Figure 2As shown, after obtaining the multi-scale single-view features of each single-view image, viewpoint prediction can be performed based on these features. Specifically, when using the multi-scale single-view features of each single-view image for viewpoint prediction, the single-scale single-view features of each single-view image can be determined first based on these multi-scale single-view features. Then, viewpoint prediction is performed based on these single-scale single-view features to obtain the predicted crowd image corresponding to the camera viewpoint of that single-view image. Furthermore, the single-scale single-view features of the single-view images can be used to construct the multi-view features of the current frame's multi-view images, which can be used to generate tracking queries for the next frame.

[0038] It can be understood that the multi-view features of the previous frame include the single-scale single-view features of each single-view image in the previous frame, and these single-scale single-view features are determined based on the multi-scale single-view features of the single-view images in the previous frame. Specifically, the generation process of the multi-view features of the previous frame includes: Obtain multi-scale single-view features of each previous single-view image in the previous frame of multi-view image. The multi-scale single-view features of each previous single-view image are input into the feature pyramid network, and the single-scale single-view features of the previous single-view image are output through the feature pyramid network to generate the multi-view features of the previous frame.

[0039] Specifically, the multi-view features of the previous frame are used for viewpoint detection in multi-view crowd tracking based on the previous frame's multi-view image, and for generating tracking queries in multi-view crowd tracking based on the current frame's multi-view image. The feature pyramid network is used to fuse multi-scale single-view features to generate single-scale single-view features. For example, the feature pyramid network uses a top-down feature fusion approach, fusing high-resolution, low-semantic-information single-view features with low-resolution, high-semantic-information single-view features, and adjusting the number of channels through convolutional operations to output single-scale single-view features. This single-scale single-view feature retains key semantic information while reducing feature dimensionality, facilitating subsequent viewpoint detection and cross-frame information interaction.

[0040] Furthermore, the ground features of the current frame in step S10 above are obtained by projecting multi-scale single-view features onto the ground plane. The ground features of the current frame include ground features of multiple scales, that is, the ground features of the current frame are multi-scale ground features. For example, each single-scale single-view feature in the multi-scale single-view features can be projected onto the ground plane to obtain multi-scale ground features.

[0041] In one embodiment, projecting multi-scale single-view features onto the ground plane to obtain ground features of the current frame specifically includes: For each voxel of a single-scale single-view feature in a multi-scale single-view feature, the vertex coordinates of the voxel are projected onto the image plane to obtain the pixel coordinates corresponding to each vertex coordinate. Based on the pixel coordinates, multi-height bilinear sampling is used to determine the feature value corresponding to the pixel coordinates to obtain the single-view ground feature corresponding to the single-scale single-view feature. All single-view ground features corresponding to each scale are fused to obtain the single-scale ground features of the current frame, and all single-scale ground features of the current frame are used as the ground features of the current frame.

[0042] Specifically, for each voxel of a single-scale, single-view feature, the vertex coordinates of its eight vertices are extracted, and camera parameters (including intrinsic and extrinsic matrices) are obtained to form a projection matrix. Then, homogeneous coordinates are constructed for each vertex coordinate, and the homogeneous projection coordinates of these coordinates onto the image plane (i.e., the ground plane) are calculated using the projection matrix. Finally, the homogeneous projection coordinates are normalized to obtain the pixel coordinates of each vertex in the corresponding camera image. The homogeneous projection coordinates can be expressed as: , in, Represents the homogeneous projection coordinates. Represents the projection matrix. Represents the intrinsic parameter matrix, Represents the extrinsic parameter matrix. Represents the rotation matrix. Represents the translation matrix. Homogeneous coordinates representing vertex coordinates Represents the vertex coordinates.

[0043] Furthermore, after obtaining the pixel coordinates corresponding to the vertex coordinates, the depth direction of the voxel corresponding to the pixel coordinates is divided into multiple sampling height layers, which cover the feature distribution of the voxel in the depth direction. Then, using the pixel coordinates as a reference, bilinear sampling is performed on the feature values ​​corresponding to the sampling height layer on a single-scale, single-view feature, to obtain the feature value of the vertex coordinates at that sampling height layer. Finally, the feature values ​​of all sampling height layers are fused (e.g., by pre-configuring weights for each sampling height layer and then fusing them according to the weights) to obtain the feature value corresponding to the pixel coordinates, thereby obtaining the single-view ground feature corresponding to the single-scale, single-view feature.

[0044] However, for each scale, the single-view ground features of each viewpoint at that scale are obtained to obtain all single-view ground features corresponding to that scale, and all single-view ground features corresponding to that scale are fused (e.g., using convolution to fuse between viewpoints) to obtain the ground features corresponding to that scale, thus obtaining multi-scale ground features.

[0045] This application utilizes multi-height bilinear sampling to determine ground features, effectively capturing feature information at different height levels. This improves robustness to long-distance perception and crowded scenes, and overcomes the feature loss problem that may occur with traditional single-height sampling. The projected ground features more comprehensively reflect the spatial distribution of pedestrians in the scene. For example, when pedestrians are in different height states (such as standing, bending over, or carrying items), multi-height sampling can extract features at the corresponding heights, ensuring the sensitivity of ground features to changes in pedestrian posture. Furthermore, by fusing all single-view ground features at each scale, observation information from different perspectives can be integrated, reducing the impact of single-view occlusion or viewpoint deviation, and improving the robustness and accuracy of the ground features in the current frame.

[0046] S20. Interact with the multi-view features of the previous frame and the ground features of the previous frame to generate a tracking query.

[0047] Specifically, the multi-view features and ground features from the previous frame interact to integrate complementary information from the ground and camera perspectives, thereby better representing each tracked person. In this interaction, a cross-attention mechanism can be used to achieve information fusion between the two.

[0048] In one embodiment, the interaction between the previous frame's multi-view features and the previous frame's ground features to generate a tracking query specifically includes: The previous frame feature query is determined based on the ground features of the previous frame; Discrete sampling is performed on the previous frame feature query based on the crowd position in the previous frame to construct an initial tracking query; A viewpoint query is constructed based on the multi-view features of the previous frame, and a cross-attention mechanism is used to interact the initial tracking query and the viewpoint query to generate a tracking query.

[0049] Specifically, the previous frame feature query is obtained by encoding the ground features of the previous frame. For example, the ground features of the previous frame are input into a Transformer encoder, which independently processes the ground features of the previous frame to generate the previous frame feature query. The Transformer encoder can use a multi-scale deformable attention module to aggregate information from multiple scales. Furthermore, the process of obtaining the current frame feature query in subsequent step S30 can be the same as that of obtaining the previous frame feature query, both being obtained through encoding by the Transformer encoder.

[0050] Therefore, the feature query of the previous frame and the feature query of the current frame can be represented as follows: , , in, This indicates a feature query from the previous frame. Indicates the current frame feature query. Indicates Transformer encoder, This indicates the ground features of the previous frame. Indicates the first of the ground features in the previous frame Scale of ground features, Indicates the ground features of the current frame. Indicates the first of the ground features in the current frame Scale of ground features, Indicates the sampling time of the current frame. Indicates the sampling time of the previous frame. Represents multi-scale quantities. Indicates scale index.

[0051] Furthermore, after obtaining the feature query from the previous frame, the feature query from the previous frame is discretely sampled at the location of the crowd in the previous frame on the ground plane to construct an initial tracking query, wherein the initial tracking query can be expressed as: , in, Indicates the initial tracking query. Indicates discrete sampling. This indicates a feature query from the previous frame. This indicates the position of the crowd in the previous frame.

[0052] The initial tracking query in this embodiment is obtained by sampling the feature query of the previous frame with the crowd position as a constraint. This can accurately locate the spatial position of the target pedestrian, avoiding background noise interference that may be caused by unconstrained sampling, and providing more targeted initial information for the generation of subsequent tracking queries. Specifically, the crowd position in the previous frame can be obtained from the tracking or detection results of the previous frame. For example, if several pedestrians have been tracked in the previous frame, the center coordinates or bounding box center coordinates of each pedestrian can be used as sampling points. By extracting local features from the feature query of the previous frame at these sampling points, the initial tracking query can effectively carry the pedestrian's identity information and spatial features in the previous frame, laying the foundation for cross-frame association.

[0053] After obtaining the initial tracking query, to better represent each tracked person by integrating complementary information from the ground and camera perspectives, the initial tracking query can be interacted with the multi-view features of the previous frame. To this end, after obtaining the initial tracking query, a viewpoint query can be constructed based on the multi-view image features of the previous frame. Then, a cross-attention mechanism is used to interact the initial tracking query and the viewpoint query to generate a tracking query, enabling the tracking query to integrate complementary information from the ground and camera perspectives.

[0054] Since the multi-view features of the previous frame contain single-scale, single-view features from each camera's perspective, these features capture scene information from different observation angles. Therefore, when constructing a view query, a set of view-specific queries can be sampled from the multi-view features of the previous frame, and then the sampled view feature queries are fused to form the view query. This view query can be represented as: , in, Indicates a viewpoint query. Indicates a connection operation. Indicates the first feature from the multi-view features of the previous frame. View-specific queries sampled from single-scale, single-view features corresponding to each camera viewpoint.

[0055] After acquiring the viewpoint query, a cross-attention mechanism is used to interact with the initial tracking query and the viewpoint query to generate a new tracking query. This cross-attention mechanism allows the initial tracking query (as the query vector) to actively focus on information relevant to itself within the viewpoint query (as the key and value vectors). During the interaction based on the cross-attention mechanism, the attention weights of each feature unit in the initial tracking query and the viewpoint query are determined. Higher attention weights indicate that the feature information contained in that feature unit is more relevant to the tracking target (e.g., a person) corresponding to the initial tracking query. By utilizing the cross-attention mechanism to interact with the initial tracking query and the viewpoint query, this application can filter complementary features related to the tracking target corresponding to the initial tracking query from the multi-view features of the previous frame. This ensures that the generated tracking query not only includes historical information about the target from the previous frame but also integrates observation information from multiple viewpoints. This addresses the problem that discrete sampling from the ground representation of the previous frame may not fully capture the appearance of a person, improving the discriminative power and robustness of the tracking query and enabling it to better handle target tracking in complex scenes.

[0056] In one embodiment, to further improve the discriminative power and robustness of the tracking query, the initial tracking query and the viewpoint query can be refined before interacting with each other using the cross-attention mechanism. Based on this, the interaction between the initial tracking query and the viewpoint query using the cross-attention mechanism to generate the tracking query specifically includes: The initial tracking query and the view query are refined using a feedforward neural network to obtain a refined tracking query and a refined view query. The refined tracking query is used as the query vector, and the refined perspective query is used as the key vector and value vector. The tracking query is generated through a cross-attention mechanism.

[0057] Specifically, a feed-forward network (FFN) consists of linear layers, activation functions (such as ReLU), and dropout layers. The initial tracking query and view query are input into the feed-forward network, which processes them (such as dimensionality reduction, dimensionality increase, or feature reorganization) to enhance the expressive power of features and filter out redundant or noisy information, so that the subsequent cross-attention mechanism can more effectively focus on key features.

[0058] After obtaining the refined tracking query and refined perspective query, the refined tracking query is used as the query vector of the cross-attention module, and the refined perspective query is used as the key and value vectors. The cross-attention mechanism is then used to interact the refined tracking query and refined perspective query to generate a tracking query. This tracking query integrates the spatial positioning information of the ground features in the previous frame and the appearance details of the multi-view image features, providing a more comprehensive and robust query basis for multi-view crowd tracking in the current frame.

[0059] For example, a tracking query can be represented as: , in, Indicates a tracking query. This represents the cross-attention mechanism. This represents a feedforward neural network. Indicates the initial tracking query. Indicates a viewpoint query.

[0060] S30. Determine the current frame feature query based on the ground features of the current frame, and perform a time-series interaction between the tracking query and the current frame feature query based on the crowd position of the previous frame to obtain the current frame tracking query.

[0061] Specifically, the current frame feature query is obtained by encoding the ground features of the current frame. Its acquisition process is the same as that of the previous frame feature query, both being obtained through Transformer encoder encoding. For details, please refer to the description of the acquisition process for the previous frame feature query. This current frame feature query also aggregates feature information from multi-scale ground features of the current frame, providing feature information for target tracking in the current frame.

[0062] After obtaining the tracking query and the current frame feature query, a temporal interaction is performed between the tracking query and the current frame feature query based on the crowd location in the previous frame. The tracking information from the previous frame is transmitted to the current frame and combined with the ground feature information of the current frame to obtain the current frame tracking query. The crowd location in the previous frame is used as a spatial anchor point for temporal association, ensuring that the interaction process focuses on areas where the tracking target may exist, reducing interference from irrelevant areas.

[0063] In one embodiment, the step of performing a time-series interaction between the tracking query based on the crowd location in the previous frame and the feature query in the current frame to obtain the tracking query for the current frame specifically includes: Driven by the tracking query of the previous frame and with the target position of the previous frame as the reference point, a multi-scale deformable attention structure is used to model the temporal correspondence between the feature query of the previous frame and the feature query of the current frame in order to generate the tracking query of the current frame.

[0064] Specifically, the multi-scale deformable attention structure can dynamically adjust the position and weight of sampling points based on the crowd position in the previous frame, and perform flexible feature aggregation on current frame feature queries at different scales. For example, for each previous frame tracking query, the multi-scale deformable attention module generates an offset around the crowd position in the previous frame on multiple scale feature maps of the current frame feature query. This offset is used to determine the actual sampling position. Then, by weighted summing of the features at the sampling position, the possible position of the tracked target in the previous frame and its feature changes in the current frame can be captured, thereby realizing the temporal transmission of the tracking query. This application uses the crowd position in the previous frame as a reference point and utilizes the multi-scale deformable attention structure to model the temporal correspondence between the previous frame feature query and the current frame feature query. It not only considers the spatial continuity of the target but also enhances the adaptability to complex situations such as target scale changes and pose changes through the fusion of multi-scale features, making the generated current frame tracking query more accurately reflect the target's state in the current frame.

[0065] For example, the current frame tracking query can be represented as: , in, Indicates current frame tracking query, This represents a multi-scale deformable attention structure. Indicates a tracking query. Indicates the current frame feature query. This indicates the position of the crowd in the previous frame.

[0066] In addition, in practical applications, time-series interaction can also be implemented in other ways, such as dynamically fusing historical information from tracking queries and new information from current frame feature queries through gating mechanisms.

[0067] S40. Based on the current frame, track and query to predict motion offset, based on the current frame features, query to predict crowd heatmap, and predict the current frame crowd position based on the motion offset and the crowd heatmap, so as to realize multi-view crowd tracking.

[0068] Specifically, the current frame tracking query contains historical motion information of the tracked target (i.e., the target person) and feature association information of the current frame. By decoding the current frame tracking query, the motion offset of the tracked target relative to the position of the previous frame can be predicted. This motion offset is predicted using the motion continuity of the tracked target and is used to provide an initial position estimate for crowd location prediction. This motion offset can be predicted using a regression head (such as a lightweight trajectory continuity low-population-ratio P-head), which includes the lateral and longitudinal offsets of the tracked target in the image coordinate system or world coordinate system. For example, if the crowd position in the previous frame was... The predicted motion offset is ( , Then, the preliminary predicted position of the tracked target in the current frame can be represented as: .

[0069] For example, motion offset can be expressed as: , in, Indicates motion offset. The P-head indicates the proportion of people with low continuity in lightweight trajectories. Indicates current frame tracking query, Indicates the lateral offset. This indicates the vertical offset.

[0070] Meanwhile, the current frame feature query aggregates global information about the ground features of the current frame, and based on this query, a crowd heatmap for the current frame can be predicted. The crowd heatmap is a two-dimensional density map, where the value of each pixel represents the probability or confidence of a pedestrian's presence at that location. The crowd heatmap can be generated by upsampling and convolutional operations on the current frame feature query using a convolutional regression head (i.e., using a convolutional neural network as a decoding head), resulting in a crowd heatmap with the same or higher resolution as the input ground features. Peak regions on the crowd heatmap correspond to possible pedestrian locations, and by performing post-processing operations such as non-maximum suppression (NMS) on the crowd heatmap, candidate pedestrian center locations can be extracted.

[0071] Furthermore, in practical applications, when predicting crowd heatmaps based on current frame feature queries, a pyramid network can be used to sample and fuse the current frame feature queries to the highest spatial resolution. Then, the fused current frame feature queries are used to predict the crowd heatmap. Based on this, the crowd heatmap can be represented as: , in, Represents a population heatmap. Indicates the convolutional regression head. Represents a pyramid network. This indicates a feature query for the current frame.

[0072] Furthermore, after predicting the motion offset and crowd heatmap, the motion offset can be used to determine the initial crowd position, and the crowd heatmap can be used to determine the candidate crowd position. Then, the initial crowd position and the candidate crowd position are fused to determine the final crowd position in the current frame. For example, the initial crowd position predicted by the motion offset can be used as the center, and the peak point with the highest confidence in the corresponding area on the crowd heatmap can be searched as the crowd position in the current frame; or, the initial crowd position provided by the motion offset can be used as a priori to weight the features near the initial crowd position in the crowd heatmap to enhance the response of the real target. The embodiments of this application predict the crowd position in the current frame by combining motion offset and crowd heatmap, combining the temporal coherence of target motion and the spatial discriminativeness of current frame features, which can effectively improve the accuracy of crowd position prediction. Especially in complex situations such as target occlusion, rapid movement, or posture changes, more robust multi-view crowd tracking can be achieved through the complementary information of motion offset and crowd heatmap. For example, when motion offset prediction has a large error due to rapid movement, crowd heatmaps can provide more accurate spatial positioning clues; conversely, when heatmaps are blurred due to occlusion, motion offsets can provide the general movement trend of the target and guide the selection of the correct location.

[0073] In summary, this embodiment provides a multi-view crowd tracking method with viewpoint-ground interaction. The method includes acquiring a current frame multi-view image, previous frame multi-view features, and previous frame ground features; extracting multi-scale single-view features from each single-view image in the multi-view image, and projecting the multi-scale single-view features onto a ground plane to obtain the current frame ground features; interacting with the previous frame multi-view features and previous frame ground features to generate a tracking query; determining a current frame feature query based on the current frame ground features, and performing a temporal interaction between the tracking query and the current frame feature query based on the crowd position in the previous frame to obtain a current frame tracking query; predicting motion offset based on the current frame tracking query, and predicting a crowd heatmap based on the current frame feature query; and predicting the current frame crowd position based on the motion offset and the crowd heatmap to achieve multi-view crowd tracking. This application's embodiments generate a tracking query by interacting with the multi-view features and ground features of the previous frame. This tracking query is then interacted with the ground features of the current frame to generate a tracking query for the current frame. This achieves information interaction between the current frame's ground features and the previous frame's ground features, as well as between ground features and viewpoint features, thereby improving tracking stability in severely occluded and complex scenes. Furthermore, by combining motion offset and crowd heatmaps when performing crowd tracking, the accuracy of crowd trajectory prediction can be further improved, reducing identity switching rates and tracking loss.

[0074] Furthermore, compared to existing multi-view crowd tracking methods based on convolutional neural networks (such as EarlyBird and TrackTacular), the multi-view crowd tracking method with view-ground interaction provided in this application introduces a Transformer architecture, which can more effectively model the global dependencies between multiple views and in the time dimension, thereby significantly improving the accuracy and stability of tracking in complex scenes. As shown in Table 1, the method of this application outperforms existing technologies on two large-scale real-world scene datasets, MVCrowdTrack and CityTrack. Specifically, in the MVCrowdTrac dataset, under the same crowd heatmap supervision as the EarlyBird method, the method of this application significantly leads in key indicators such as multi-target tracking accuracy and identity preservation performance; compared to the MVFlow method that only relies on weakly supervised motion information, the method of this application performs more stably in long-term tracking and effectively reduces trajectory interruption problems. Furthermore, compared to CNN methods (such as TrackTacular) that model solely through stacking historical information, our method introduces a viewpoint-ground interaction mechanism to fuse multi-viewpoint and temporal features in a unified space. This allows the model to maintain high identity association and trajectory prediction accuracy even under conditions of high-density crowds and severe occlusion. Ablation experiments further validate the significant improvement effect of this interaction mechanism on multi-target tracking accuracy and identity preservation performance. In summary, our method demonstrates higher accuracy and stability compared to state-of-the-art techniques under complex large-scale scenes, multiple occlusions, and long temporal sequences, while reducing reliance on complex post-processing, resulting in better generalization ability and practical application value.

[0075] Meanwhile, the multi-view crowd tracking method proposed in this application, featuring viewpoint-ground interaction, can effectively fuse multi-view spatial information and long-range temporal dependencies under conditions of large-scale real-world scenes, high-density crowds, and severe occlusion, thereby achieving more accurate and stable multi-view crowd tracking. For the multi-view crowd tracking task, as shown in Table 1, the proposed method is compared with existing methods, including the MVFlow method based on weakly supervised motion prediction, the EarlyBird method based on early BEV fusion combined with ReID and Kalman filtering for trajectory association, and the TrackTacular method based on 3D BEV stacking and regressing motion offset through adjacent frame world representations. Experimental results show that the proposed method achieves state-of-the-art performance on both the MVCrowdTrack and CityTrack large-scale real-world scene datasets. Specifically, on the MVCrowdTrack dataset, the multi-target tracking accuracy reaches 63.87%, the identity preservation performance reaches 59.06%, the proportion of people with high trajectory continuity increases to 42.85%, and the proportion of people with low trajectory continuity decreases to 8.16%, which is better than the TrackTacular and EarlyBird methods in all indicators. On the CityTrack dataset with more severe occlusion and denser crowds, the multi-target tracking accuracy of the method in this application reaches 55.39%, which is 6.54 and 12.02 percentage points higher than the EarlyBird and TrackTacular methods, respectively, showing a more significant advantage and fully verifying the adaptability and robustness of the method in complex and large-scale scenes.

[0076] Table 1 Experimental Results

[0077] Furthermore, this application also conducted ablation experiments to analyze multi-view information fusion and view-ground plane interaction. As shown in Table 2, in the ablation experiments, when only the crowd heatmap decoding branch of the 2D camera view was added without introducing the view-ground interaction unit, the performance of the multi-view crowd tracking model in this application decreased rather than improved (multi-target tracking accuracy was 53.17%). However, after further introducing the view-ground interaction unit proposed in this application, the performance of the multi-view crowd tracking model in this application was significantly improved, ultimately reaching the optimal result (multi-target tracking accuracy of 55.39% and identity preservation performance of 34.41%). The above results fully demonstrate that the view-ground interaction unit effectively fuses complementary information from multiple perspectives and improves the overall multi-view crowd tracking performance.

[0078] Table 2 Tracking performance of ablation experiments on the CityTrack dataset

[0079] Based on the aforementioned multi-view crowd tracking method with viewpoint-ground interaction, this embodiment provides a multi-view crowd tracking device with viewpoint-ground interaction, such as... Figure 3 As shown, the multi-view crowd tracking device with viewpoint-ground interaction specifically includes: The feature extraction and multi-view fusion module 100 is used to acquire the current frame multi-view image, the previous frame multi-view features and the previous frame ground features, and to extract the multi-scale single-view features of each single-view image in the multi-view image, and project the multi-scale single-view features onto the ground plane to obtain the current frame ground features. The multi-view tracking encoding module 200 is used to interact with the multi-view features of the previous frame and the ground features of the previous frame to generate a tracking query, and to determine the feature query of the current frame based on the ground features of the current frame. The multi-view tracking decoding module 300 is used to perform time-series interaction between the tracking query and the current frame feature query based on the crowd position of the previous frame to obtain the current frame tracking query, predict motion offset based on the current frame tracking query, predict crowd heat map based on the current frame feature query, and predict the current frame crowd position based on the motion offset and the crowd heat map, so as to realize multi-view crowd tracking.

[0080] Based on the above-described multi-view crowd tracking method with viewpoint-ground interaction, this embodiment provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps in the multi-view crowd tracking method with viewpoint-ground interaction as described in the above embodiment.

[0081] Based on the aforementioned multi-view crowd tracking method with viewpoint-ground interaction, this application also provides a terminal device, such as... Figure 4 As shown, it includes at least one processor 20; a display screen 21; and a memory 22, and may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via the bus 24. The display screen 21 is configured to display a preset user guide interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can invoke logical instructions in the memory 22 to execute the methods described in the above embodiments.

[0082] Furthermore, the logical instructions in the aforementioned memory 22 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0083] The memory 22, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of this disclosure. The processor 20 executes functional applications and data processing by running the software programs, instructions, or modules stored in the memory 22, thereby implementing the methods in the above embodiments.

[0084] The memory 22 may include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 22 may include high-speed random access memory (RAM) and non-volatile memory. Examples include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, as well as transient storage media.

[0085] Furthermore, the specific process of loading and executing multiple instruction processors in the aforementioned storage medium and terminal device has been described in detail in the above method, and will not be repeated here.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multi-view crowd tracking method with viewpoint-ground interaction, characterized in that, The aforementioned multi-view crowd tracking method with viewpoint-ground interaction specifically includes: Acquire the multi-view image of the current frame, as well as the multi-view features and ground features of the previous frame; Extract multi-scale single-view features from each single-view image in the multi-view image, and project the multi-scale single-view features onto the ground plane to obtain the ground features of the current frame; The multi-view features and ground features of the previous frame are interacted to generate a tracking query; Based on the ground features of the current frame, a feature query for the current frame is determined. Based on the location of the crowd in the previous frame, the tracking query and the feature query for the current frame are interacted in a time sequence to obtain the tracking query for the current frame. Based on the current frame, the tracking query predicts motion offset, and based on the current frame features, the query predicts crowd heatmap; The position of the crowd in the current frame is predicted based on the motion offset and the crowd heatmap to achieve multi-view crowd tracking.

2. The multi-view crowd tracking method with viewpoint-ground interaction according to claim 1, characterized in that, The step of projecting multi-scale single-view features onto the ground plane to obtain the ground features of the current frame specifically includes: For each voxel of a single-scale single-view feature in a multi-scale single-view feature, the vertex coordinates of the voxel are projected onto the image plane to obtain the pixel coordinates corresponding to each vertex coordinate. Based on the pixel coordinates, multi-height bilinear sampling is used to determine the feature value corresponding to the pixel coordinates to obtain the single-view ground feature corresponding to the single-scale single-view feature. All single-view ground features corresponding to each scale are fused to obtain the single-scale ground features of the current frame, and all single-scale ground features of the current frame are used as the ground features of the current frame.

3. The multi-view crowd tracking method with viewpoint-ground interaction according to claim 1, characterized in that, The interaction between the multi-view features of the previous frame and the ground features of the previous frame to generate a tracking query specifically includes: The previous frame feature query is determined based on the ground features of the previous frame; Discrete sampling is performed on the previous frame feature query based on the crowd position in the previous frame to construct an initial tracking query; A viewpoint query is constructed based on the multi-view features of the previous frame, and a cross-attention mechanism is used to interact the initial tracking query and the viewpoint query to generate a tracking query.

4. The multi-view crowd tracking method with viewpoint-ground interaction according to claim 3, characterized in that, The step of using a cross-attention mechanism to interact with the initial tracking query and the view query to generate a tracking query specifically includes: The initial tracking query and the view query are refined using a feedforward neural network to obtain a refined tracking query and a refined view query. The refined tracking query is used as the query vector, and the refined perspective query is used as the key vector and value vector. The tracking query is generated through a cross-attention mechanism.

5. The multi-view crowd tracking method with viewpoint-ground interaction according to claim 3, characterized in that, Both the previous frame feature query and the current frame feature query are generated by the Transformer encoder.

6. The multi-view crowd tracking method with viewpoint-ground interaction according to claim 1, characterized in that, The step of performing a time-series interaction between the tracking query and the current frame feature query based on the crowd position in the previous frame to obtain the current frame tracking query specifically includes: Driven by the tracking query of the previous frame and with the target position of the previous frame as the reference point, a multi-scale deformable attention structure is used to model the temporal correspondence between the feature query of the previous frame and the feature query of the current frame in order to generate the tracking query of the current frame.

7. The multi-view crowd tracking method with viewpoint-ground interaction according to claim 1, characterized in that, The previous frame multi-view features include single-scale single-view features of each single-view image; The process of generating the multi-view features of the previous frame specifically includes: Obtain the multi-scale single-view features of each previous single-view image in the previous frame of the multi-view image. The multi-scale single-view features of each previous single-view image are input into the feature pyramid network, and the single-scale single-view features of the previous single-view image are output through the feature pyramid network to generate the multi-view features of the previous frame.

8. A multi-view crowd tracking device with viewpoint-ground interaction, characterized in that, The aforementioned multi-view crowd tracking device with view-to-ground interaction specifically includes: The feature extraction and multi-view fusion module is used to acquire the current frame multi-view image, the previous frame multi-view features and the previous frame ground features, and to extract the multi-scale single-view features of each single-view image in the multi-view image, and project the multi-scale single-view features onto the ground plane to obtain the current frame ground features. The multi-view tracking encoding module is used to interact with the multi-view features of the previous frame and the ground features of the previous frame to generate a tracking query, and to determine the feature query of the current frame based on the ground features of the current frame. The multi-view tracking decoding module is used to perform temporal interaction between the tracking query and the current frame feature query based on the crowd position in the previous frame to obtain the current frame tracking query, predict motion offset based on the current frame tracking query, predict crowd heat map based on the current frame feature query, and predict the current frame crowd position based on the motion offset and the crowd heat map, so as to realize multi-view crowd tracking.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the multi-view crowd tracking method with view-ground interaction as described in any one of claims 1-7.

10. A terminal device, characterized in that, include: Processor and memory; The memory stores a computer-readable program that can be executed by the processor; When the processor executes the computer-readable program, it implements the steps of the multi-view crowd tracking method with viewpoint-ground interaction as described in any one of claims 1-7.

Images