Multi-view construction personnel tracking method and system based on attention perception

Abstract

The invention discloses a multi-view construction personnel tracking method and system based on attention perception. The method comprises the following steps: giving synchronous images from S cameras, and inputting the synchronous images into an encoder for feature extraction to obtain a multi-view feature map; transforming the multi-view feature map into a unified aerial view space by using perspective projection, and aggregating features after projection transformation of all views by using a convolutional layer; inputting the aggregated aerial view features into a decoder for decoding; a cross attention module is introduced, bird's-eye view features of a current frame and an adjacent frame are processed through 3D position coding, instance tokens are extracted as query, keys and values, an affinity matrix is generated through CNN coding similarity, features are propagated through matrix multiplication, and the bird's-eye view features of the current frame are updated. According to the method, the multi-view feature map is projected to the aerial view to realize early fusion, and a cross-frame attention mechanism is introduced, so that the problem of appearance feature distortion caused by perspective transformation is solved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent monitoring technology. Specifically, it is a method and system for tracking construction workers based on attention perception multi-view, which is particularly suitable for complex scenarios with severe occlusion. Background Technology

[0002] In existing technologies, early fusion strategies detect construction workers by projecting multi-view feature maps onto a common ground plane (such as a bird's-eye view BEV), but this has the following drawbacks: 1) Perspective distortion problem: Ground projection causes serious distortion of the appearance features of construction workers, affecting the accuracy of cross-frame association; 2) Feature degradation problem: Traditional motion models such as Kalman filtering have difficulty maintaining feature consistency in occluded scenes; 3) Computational efficiency bottleneck: The computational complexity of existing attention mechanisms (such as Transformer) increases quadratically in long sequence tracking. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a method and system for tracking construction workers based on attention perception multi-view. It projects multi-view feature maps onto a bird's-eye view to achieve early fusion and introduces a cross-frame attention mechanism to solve the problem of appearance feature distortion caused by perspective transformation.

[0004] Therefore, the present invention adopts the following technical solution.

[0005] In a first aspect, the present invention provides a method for tracking construction workers based on attention-aware multi-view methods, comprising the following steps:

[0006] 1) Given synchronized images from S cameras, input them into an encoder for feature extraction to obtain multi-view feature maps;

[0007] 2) Transform the multi-view feature maps to a unified bird's-eye view (BEV) space using perspective projection, and aggregate the features after all view projections are aggregated using convolutional layers;

[0008] 3) Input the aggregated bird's-eye view features into the decoder for decoding;

[0009] 4) Introduce a cross-attention module to process the bird's-eye view features of the current frame and adjacent frames through 3D position encoding, extract instance tokens as queries, keys and values, generate affinity matrices through CNN encoding similarity, and propagate features through matrix multiplication to update the bird's-eye view features of the current frame in order to handle the boundary situation of people entering and leaving the scene.

[0010] In this invention, 3D position encoding refers to an encoding method that converts spatial coordinates (x, y) and timestamp t into high-dimensional feature vectors using sine and cosine functions.

[0011] Furthermore, step 1) specifically includes:

[0012] The encoder uses ResNet to extract feature maps from synchronized images provided by S cameras. Each image is downsampled stepwise through multiple ResNet blocks. The features of each layer are upsampled and fused with the output of the previous layer. Then, they are processed through convolutional layers to finally obtain multi-view feature maps.

[0013] Furthermore, step 2) specifically includes:

[0014] Based on the pinhole camera model, 3D points are mapped to 2D pixel coordinates on the image plane through a perspective transformation matrix, and 3D features are mapped to the ground plane with z=0 to simplify the projection process. All multi-view feature maps are projected onto a common ground plane and discretized into a grid of predetermined dimensions based on the size of the labeled region. Finally, all view features are aggregated into a single feature representation through a convolutional layer.

[0015] Furthermore, step 3) specifically includes:

[0016] A ResNet-18 decoder is used. In this decoder architecture, each layer of the ResNet block downsamples the bird's-eye view features, then upsamples the output of each layer to match the size of the output of the previous layer, concatenates the two features during lateral connections, and uses deformable convolutions for feature fusion.

[0017] Furthermore, the deformable convolution uses a 3×3 deformable convolution with a stride of 1 and a padding of 1. By adaptively adjusting the receptive field at each position, it enhances flexibility, improves the multi-scale feature alignment effect in the fusion stage, and more accurately aggregates the position and appearance features of construction workers.

[0018] Furthermore, step 4) specifically includes:

[0019] 41) Input the decoded bird's-eye view features of the current frame and adjacent frames into the cross-attention module, apply 3D position encoding to enrich spatial and temporal information; extract the construction personnel features of each detection center as tokens, use the current frame token as a query, the previous frame token as a key, and the original token as a value.

[0020] Furthermore, step 4) specifically also includes:

[0021] 42) Encoding the similarity between the query and the key using CNN: The query and the key are expanded into token-level pairs, combined element-wise by multiplication, and then input into the CNN to generate an inner matrix that encodes the intimacy between construction workers in different frames.

[0022] Furthermore, step 4) specifically also includes:

[0023] 43) To handle the boundary situation of construction workers entering and exiting the video between two frames, extra rows and columns are added to the inner matrix to obtain the first matrix and the second matrix; column-direction softmax is applied to the first matrix and row-direction softmax is applied to the second matrix; the two inner matrices are multiplied to obtain the inner affinity matrix, and the extra rows and columns are processed by multiplying them by the mean of the corresponding rows and columns of the inner affinity matrix to obtain the affinity matrix.

[0024] Furthermore, step 4) specifically also includes:

[0025] 44) The cross-attention mechanism guides the propagation of instance features between frames through the affinity matrix. The affinity matrix is ​​multiplied with the value tokens so that each query focuses on the most similar key-value pairs. The attention result is added to the original position of the query token in the current frame to generate a refined bird's-eye view feature that incorporates information from the previous frame.

[0026] Secondly, the present invention provides an attention-aware multi-view construction worker tracking system for implementing the above-mentioned construction worker tracking method, comprising:

[0027] Feature extraction unit: Extracts features from synchronized images from S cameras and outputs multi-view feature maps;

[0028] BEV Projection Unit: Transforms multi-view feature maps to a unified bird's-eye view space through perspective projection, and aggregates the features after all view projection transformations using convolutional layers;

[0029] Feature decoding unit: Inputs the aggregated bird's-eye view features into the decoder for decoding;

[0030] Cross-frame association unit: Introduces a cross-attention module, processes the bird's-eye view features of the current frame and adjacent frames through 3D position encoding, extracts instance tokens as query, key and value, generates an affinity matrix through CNN encoding similarity, and propagates features through matrix multiplication to update the bird's-eye view features of the current frame.

[0031] The beneficial effects of this invention are as follows: This invention projects multi-view feature maps onto a bird's-eye view, achieving early fusion, and introduces a cross-frame attention mechanism to solve the problem of appearance feature distortion caused by perspective transformation. This invention proposes a cross-attention-based construction worker feature association strategy, improving the robust propagation of construction worker features across multiple frames; it employs an adaptive feature fusion module (deformable convolution) to reduce the impact of ground projection distortion on detection accuracy; and it enriches spatiotemporal information through 3D position encoding and generates an affinity matrix using CNN encoding similarity, optimizing cross-frame feature association. Attached Figure Description

[0032] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0033] Figure 1 This is a flowchart of a multi-view construction worker tracking method based on attention perception according to the present invention;

[0034] Figure 2 This is a schematic diagram of the attention module of the present invention;

[0035] Figure 3 This is a comparison chart of the tracking performance of the model corresponding to the method of the present invention and the baseline model on the complete Wildtrack test set; (a) is the trajectory of the model corresponding to the method of the present invention, (b) is the trajectory of the baseline model, and (c) is the trajectory of the baseline true value.

[0036] Figure 4 This is a structural diagram of a construction worker tracking system based on attention perception multi-view according to the present invention. Detailed Implementation

[0037] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1

[0039] This embodiment describes a method for tracking construction workers based on attention-based multi-view observation. Figure 1 As shown, the steps are as follows:

[0040] Step 1): Given synchronized images from S cameras, input them into the encoder for feature extraction to obtain multi-view feature maps.

[0041] The encoder uses ResNet to extract feature maps from synchronized images provided by S cameras. Each image is 3×H×W, where H represents the height and W represents the width. The encoder contains three ResNet blocks, progressively downsampling the images by a factor of 2. The features from each layer are upsampled and fused with the output of the previous layer, then processed through convolutional layers to obtain multi-view feature maps, with each view having a feature dimension of C. f ×H f ×W f , where H f=H / 4, W f =W / 4, C f H represents the number of output feature channels. f W represents the height of the multi-view feature map. f This represents the width of the multi-view feature map.

[0042] Step 2) Transform the multi-view feature maps to a unified bird's-eye view (BEV) space using perspective projection, and then aggregate the features after all view projections are applied using convolutional layers, as follows:

[0043] Using a pinhole camera model, a 3×4 perspective transformation matrix P = K[R|t] is used to map 3D points (x,y,z) to 2D pixel coordinates (u,v) on the image plane, as shown below:

[0044]

[0045] Where d represents a scaling factor used to account for downsampling of multi-view feature maps, and [R|t] is the extrinsic parameter matrix, and K is the intrinsic parameter matrix.

[0046] In this embodiment, the 3D features are mapped to the plane where z = 0, thereby simplifying the projection process, as follows:

[0047]

[0048] in, It is a 3×3 matrix that transforms the features captured by camera s, with the third column of matrix P removed. Feature maps from all views are projected onto the same common plane and discretized into a predetermined dimension [H] based on the size of the labeled region. g W g A grid of [[...]] is used. Each location in the grid corresponds to a 10cm x 10cm area in the real world, and to ensure efficient memory usage, it is further downsampled by a factor of 4. Finally, a convolutional layer is applied to aggregate the features after all view projection transformations into a grid of size C. g ×H g ×W g The single feature representation, C g H represents the number of feature channels after aggregation; g W represents the height of the aggregated features. g This indicates the width of the feature after aggregation.

[0049] Step 3) Input the aggregated bird's-eye view features into the decoder for decoding.

[0050] In the decoder architecture, a ResNet-18 decoder is used. In this architecture, each layer of the ResNet block downsamples the BEV features by a factor of 2. Then, the output of each layer is upsampled to match the size of the output of the previous layer. The two features are concatenated and aggregated, and then processed through deformable convolutions. Unlike the traditional Feature Pyramid Network (FPN) structure that uses 1×1 convolutions for feature fusion, this embodiment uses 3×3 deformable convolutions with a stride of 1 and padding of 1. By adaptively adjusting the receptive field at each position, it provides greater flexibility, thereby improving the multi-scale feature alignment effect in the fusion stage and more accurately aggregating the location and appearance features of construction workers. In this embodiment, although the output and input of the Feature Pyramid Network have the same dimension C... g ×H g ×W g However, it expands the receptive field of each spatial location.

[0051] Step 4), introduce a cross-attention module, such as... Figure 2 As shown, the bird's-eye view features of the current frame and adjacent frames are processed by 3D position encoding, and instance tokens are extracted as queries, keys and values. Affinity matrices are generated by CNN encoding similarity. Features are propagated by matrix multiplication and the bird's-eye view features of the current frame are updated to handle the boundary situation of construction workers entering or leaving the video between two frames.

[0052] 41) Decode the BEV features of the current frame and adjacent frames (in this embodiment, the current frame is X). t Adjacent frames are X t-τ The input of 1≤τ≤3) to the cross-attention module is applied to enrich the spatial and temporal information in the BEV features using 3D positional encoding, resulting in position-encoded BEV features Pe(X). t ) and Pe(X t-τ Extract the construction worker characteristics of each testing center as a token, and the token Pe(X) of the current frame. t Used for querying The token Pe(X) from the previous frame t-τ ) represents the key Original representation of token X t Used as a value

[0053] 3D positional encoding refers to an encoding method that converts spatial coordinates (x, y) and a timestamp t into a high-dimensional feature vector using sine and cosine functions. The specific formula for 3D positional encoding is as follows:

[0054] PE(x,t)=[sin(x / 10 k ),cos(x / 10 k sin(t / 10) m ),cos(t / 10m )]

[0055] Where k and m are learnable parameters.

[0056] 42) Use CNN to efficiently encode the similarity between the query and the key. To do this, expand the query and key to copies of their elements, creating token-level pairs, resulting in... and Then they are combined using element-wise multiplication, represented as:

[0057]

[0058] The generated inner matrix The closeness of the construction workers between different frames was encoded.

[0059] 43) To handle construction workers entering or leaving the video between frames (i.e., handling boundary situations where construction workers enter or leave the scene), this embodiment adds extra rows and columns to the inner matrix A, where the padding value v = γⅠ, where γ is a learned parameter optimized during training, and Ⅰ is a one-dimensional vector with all values ​​of 1, thus obtaining two matrices. sum matrix Then, applying column-direction softmax to matrix A1 yields A′1, which represents adjacent frames X. t-τ Each construction worker and all N t +1 identity association. Similarly, applying row direction softmax to matrix A2 yields A′2, which represents the current frame X. t Each construction worker and all N t-τ +1 Identity Association.

[0060] In this embodiment, matrices A′1 and A′2 are multiplied to obtain the internal affinity matrix. in The additional rows and columns are respectively obtained through the inner affinity matrix A aff The mean of the corresponding rows and columns is multiplied to ensure they match their distribution. Finally, the affinity matrix is ​​obtained by adding adjusted extra rows and columns. Next, the cross-attention mechanism uses an internal affinity matrix. Guides the propagation of instances between frames. It is multiplied by the value token V, as follows:

[0061]

[0062] 44) The cross-attention mechanism guides inter-frame instance feature propagation through an affinity matrix. The affinity matrix is ​​multiplied by value tokens, and each query (features of construction workers in the current frame) focuses on the most similar key-value pair (features of construction workers in the previous frame). The corresponding token obtained from the attention is then added to the current frame X.t The original location of the query token was determined, which yielded refined BEV characteristics. It enhances the features of each construction worker by incorporating relevant information from the previous frame. Furthermore, this distinguishes the method of this invention from the state-of-the-art EarlyBird method, which relies on appearance features in a single frame to differentiate construction workers. The refined BEV features are provided as input to the attention mechanism of subsequent frames and provide the CNN with more robust and discriminative features, enabling it to better estimate the affinity between each query-key pair.

[0063] Application examples

[0064] This invention evaluates the performance of DSR-AT on multiple datasets under various domain offset settings. Two datasets were used: Wildtrack and MultiviewX.

[0065] The input image resolution is 720×1280 pixels. During training, a series of enhancements are applied to the images to avoid overfitting and improve generalization. Each image is randomly adjusted within a scale range of [0.8, 1.2], randomly offset from the center, and cropped to maintain its original size. The camera intrinsic matrix K is adjusted accordingly to preserve multi-view accuracy. Figure 1 Consistency. To avoid detector overfitting, some noise was introduced into the translation vector t of the extrinsic parameter matrix during training, following the approach in [35,36]. In the training setup, the Adam optimizer was used in conjunction with a recurrent learning rate scheduler, with a maximum learning rate set to 1×10^-3. The encoder and decoder networks were initialized using weights pre-trained on ImageNet-1K.

[0066] Table 1 shows the model evaluation results compared to state-of-the-art methods on the Wildtrack and MultiviewX datasets. Our invention achieves state-of-the-art results on both datasets, demonstrating significant improvements over three state-of-the-art methods (EarlyBird, MVFlow, and TrackTacular) for tracking in the BEV space. On the Wildtrack dataset, it improves IDF1 and MOTA by +0.8% and +0.9%, respectively. On the MultiviewX dataset, our invention shows improvement on IDF1, with TrackTacular being the only method to achieve a higher score on MOTA. Furthermore, our invention improves MOTP on both datasets, with a +2.2% improvement on Wildtrack and a +0.7% improvement on MultiviewX.

[0067] Table 1: Model evaluation results compared with state-of-the-art methods on the Wildtrack and MultiviewX datasets. The best and second-best performing models are indicated by bold and underline, respectively.

[0068]

[0069] Table 2 evaluates the impact of each component in the model corresponding to the method of this invention (hereinafter referred to as the model of this invention). The first row shows the performance of the baseline model, which is used only for detection and uses a Kalman filter to link the detection results. The second row shows the performance of adding a cross-attention module to the model, an improvement that significantly enhances tracking performance, enabling the model to outperform existing state-of-the-art methods. The last row shows the impact of using an optimized decoder, which introduces deformable convolutions in the fusion stage of the Feature Pyramid Network (FPN) structure. The optimized decoder primarily improves detection performance, increasing the MOTA from 92.1% to 92.7%, as MOTA is more sensitive to detection accuracy.

[0070] Table 2: Analysis of each component in the model corresponding to the method of this invention

[0071] Configuration IDF1↑ MOTA↑ MOTP↑ Baseline model 94.2 90.4 89.0 + Cross-attention module 95.9 92.1 89.2 +Optimized decoder 96.1 92.7 88.8

[0072] Figure 3 The tracking performance of this invention is compared with the baseline model on the full Wildtrack test set. The figure shows the predicted trajectories corresponding to different construction worker identities, as well as the ground-based trajectories. The areas highlighted by the dashed ellipse indicate that the model of this invention performs better in maintaining tracking consistency. Specifically, the model of this invention, through the use of a cross-attention module, is more effective in associating and propagating construction worker features, thereby reducing identity switching and trajectory breakage. This demonstrates that the method of this invention has higher robustness and accuracy when handling complex scenes and occlusion conditions.

[0073] Example 2

[0074] This embodiment provides an attention-aware multi-view construction worker tracking system, such as... Figure 4 As shown, the method described in Embodiment 1 consists of a feature extraction unit, a BEV projection unit, a feature decoding unit, and a cross-frame association unit.

[0075] Feature extraction unit: Extracts features from synchronized images from S cameras and outputs multi-view feature maps;

[0076] BEV Projection Unit: Transforms multi-view feature maps to a unified bird's-eye view space through perspective projection, and aggregates the features after all view projection transformations using convolutional layers;

[0077] Feature decoding unit: Inputs the aggregated bird's-eye view features into the decoder for decoding;

[0078] Cross-frame association unit: Introduces a cross-attention module, processes the bird's-eye view features of the current frame and adjacent frames through 3D position encoding, extracts instance tokens as query, key and value, generates an affinity matrix through CNN encoding similarity, and propagates features through matrix multiplication to update the bird's-eye view features of the current frame.

[0079] It should be noted that each unit in the aforementioned attention-based multi-view construction worker tracking system can be implemented entirely or partially through software, hardware, or a combination thereof. These units can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each unit. For specific limitations regarding the attention-based multi-view construction worker tracking system, please refer to the limitations of the attention-based multi-view construction worker tracking method (i.e., Embodiment 1) above; both have the same function and role, and will not be repeated here.

[0080] The above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made based on the essence of the content of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for tracking construction workers based on attention-based multi-view perception, characterized in that, Including the following steps: 1) Given synchronized images from S cameras, input them into an encoder for feature extraction to obtain multi-view feature maps; 2) Transform the multi-view feature maps to a unified bird's-eye view space using perspective projection, and aggregate the features after all view projections are transformed using convolutional layers; 3) Input the aggregated bird's-eye view features into the decoder for decoding; 4) Introduce a cross-attention module to process the bird's-eye view features of the current frame and adjacent frames through 3D position encoding, extract instance tokens as queries, keys and values, generate affinity matrices through CNN encoding similarity, and propagate features through matrix multiplication to update the bird's-eye view features of the current frame in order to handle the boundary situation of construction workers entering and leaving the scene.

2. The method for tracking construction workers based on attention perception multi-view according to claim 1, characterized in that, Step 1) specifically includes: The encoder uses ResNet to extract feature maps from synchronized images provided by S cameras. Each image is downsampled stepwise through multiple ResNet blocks. The features of each layer are upsampled and fused with the output of the previous layer. Then, they are processed through convolutional layers to finally obtain multi-view feature maps.

3. The method for tracking construction workers based on attention perception in multiple views according to claim 1, characterized in that, Step 2) specifically includes: Based on the pinhole camera model, 3D points are mapped to 2D pixel coordinates on the image plane through a perspective transformation matrix, and 3D features are mapped to the ground plane with z=0 to simplify the projection process. All view feature maps are projected onto a common ground plane and discretized into a grid of predetermined dimensions based on the size of the labeled area. Finally, all view features are aggregated into a single feature representation through a convolutional layer.

4. The method for tracking construction workers based on attention perception in multiple views according to claim 1, characterized in that, Step 3) specifically includes: A ResNet-18 decoder is used. In this decoder architecture, each layer of the ResNet block downsamples the bird's-eye view features, then upsamples the output of each layer to match the size of the output of the previous layer, concatenates the two features during lateral connections, and uses deformable convolutions for feature fusion.

5. The method for tracking construction workers based on attention perception multi-view according to claim 4, characterized in that, The deformable convolution uses a 3×3 deformable convolution with a stride of 1 and a padding of 1. By adaptively adjusting the receptive field at each position, it enhances flexibility, improves the multi-scale feature alignment effect in the fusion stage, and more accurately aggregates the position and appearance features of construction workers.

6. The method for tracking construction workers based on attention perception in multiple views according to claim 1, characterized in that, Step 4) specifically includes: 41) Input the decoded bird's-eye view features of the current frame and adjacent frames into the cross-attention module, apply 3D position encoding to enrich spatial and temporal information; extract the construction personnel features of each detection center as tokens, use the current frame token as a query, the previous frame token as a key, and the original token as a value.

7. The method for tracking construction workers based on attention perception in multiple views according to claim 6, characterized in that, Step 4) specifically also includes: 42) Encoding the similarity between the query and the key using CNN: The query and the key are expanded into token-level pairs, combined element-wise by multiplication, and then input into the CNN to generate an inner matrix that encodes the intimacy between construction workers in different frames.

8. The method for tracking construction workers based on attention perception in multiple views according to claim 7, characterized in that, Step 4) specifically also includes: 43) To handle the boundary conditions of construction personnel entering and exiting the scene, add extra rows and columns to the inner matrix to obtain the first matrix and the second matrix; apply column-direction softmax to the first matrix and row-direction softmax to the second matrix; multiply the two inner matrices to obtain the inner affinity matrix, and process the extra rows and columns by multiplying them by the mean of the corresponding rows and columns of the inner affinity matrix to finally obtain the affinity matrix.

9. A method for tracking construction workers based on attention perception in multiple views according to claim 8, characterized in that, Step 4) specifically also includes: 44) The cross-attention mechanism guides the propagation of instance features between frames through the affinity matrix. The affinity matrix is ​​multiplied with the value tokens so that each query focuses on the most similar key-value pairs. The attention result is added to the original position of the query token in the current frame to generate a refined bird's-eye view feature that incorporates information from the previous frame.

10. A construction worker tracking system based on attention perception multi-view, used to implement the method described in any one of claims 1-9, characterized in that, include: Feature extraction unit: Extracts features from synchronized images from S cameras and outputs multi-view feature maps; BEV Projection Unit: Transforms multi-view feature maps to a unified bird's-eye view space through perspective projection, and aggregates the features after all view projection transformations using convolutional layers; Feature decoding unit: Inputs the aggregated bird's-eye view features into the decoder for decoding; Cross-frame association unit: Introduces a cross-attention module, processes the bird's-eye view features of the current frame and adjacent frames through 3D position encoding, extracts instance tokens as query, key and value, generates an affinity matrix through CNN encoding similarity, and propagates features through matrix multiplication to update the bird's-eye view features of the current frame.

Images