A Multi-View Single-Target Tracking Method and System Based on Adaptive 3D Feature Sampling

Abstract

This invention discloses a multi-view single-target tracking method and system based on adaptive 3D feature sampling, belonging to the field of computer vision and image processing technology. The method includes: acquiring multi-view video sequences; extracting search and reference image features from specific viewpoints using a backbone network; constructing an occlusion perception module to predict occlusion probabilities and generate soft masks; modulating the search features to suppress occlusion noise; simultaneously using a template aggregation module to obtain a global reference aggregation vector across viewpoints; projecting the modulated features into a 3D voxel space and compressing them into bird's-eye view features; using an adaptive 3D feature sampling module to establish a 2D-3D physical mapping through prior enhancement and 3D position encoding; and using a cross-attention mechanism to adaptively sample 3D features from the bird's-eye view space to enhance the search features, finally outputting the detection result. This invention can significantly improve the target tracking accuracy and robustness in complex dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of computer vision and image processing technology, and in particular to a multi-view single-target tracking method and system based on adaptive 3D feature sampling. Background Technology

[0002] In recent years, with the widespread application of the Transformer architecture in the field of visual object tracking, multi-view single-object tracking algorithms that combine 3D voxel feature fusion (such as MITracker) have shown the potential to outperform traditional single-view tracking. Such methods typically project the two-dimensional features of each viewpoint into a unified three-dimensional voxel space for aggregation to overcome the occlusion and blind spot problems in single-view tracking.

[0003] However, existing technologies still have the following main drawbacks when handling multi-view feature fusion: First, multi-view fusion lacks the ability to perceive and resist occlusion: when severe or even complete occlusion occurs, the features extracted from the occluded viewpoint often contain a large amount of background interference. If these invalid features are projected into the 3D voxel space indiscriminately, it will seriously pollute the construction of 3D feature information.

[0004] Second, insufficient utilization of cross-view template prior information: Existing methods typically only implicitly utilize reference frames during the joint encoding stage, without explicitly aggregating template information from multiple views, resulting in the inability to construct a comprehensive global target appearance representation when the target pose changes.

[0005] Third, the mapping and sampling between 2D features and 3D space is inaccurate: Existing technologies typically compress the fused 3D features into a vector and then simply concatenate it with the 2D features. This crude fusion method not only leads to severe spatial information loss but also fails to establish an accurate physical correspondence between 2D image pixels and 3D space. 2D image features cannot intelligently extract enhanced features from 3D space according to their own needs.

[0006] Therefore, there is an urgent need for a multi-view tracking method that can actively sense and filter occlusion noise, and achieve high-precision, adaptive alignment and sampling of two-dimensional and three-dimensional features. Summary of the Invention

[0007] The purpose of this invention is to provide a multi-view single-target tracking method and system based on adaptive 3D feature sampling, so as to solve the problems of occlusion feature pollution of 3D space, insufficient utilization of template information, and inaccurate 2D and 3D feature mapping in the prior art.

[0008] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On the one hand, a multi-view single-target tracking method based on adaptive 3D feature sampling is provided, the method comprising the following steps: S1. Obtain video sequences containing search images and reference images from multiple perspectives in the same scene; S2. Input the video sequence into the backbone network and extract the search feature sequence and reference feature sequence for each viewpoint respectively; S3. The template aggregation module aggregates the reference feature sequences of each viewpoint into a unified reference aggregation vector. The occlusion perception module predicts the occlusion degree of the search feature map corresponding to the search feature sequence of each viewpoint and generates a feature mask. The feature mask is used to modulate the original search feature map element by element to obtain the occlusion perception feature vector. S4. Project the occlusion perception feature vectors from each viewpoint onto a unified three-dimensional voxel space for fusion, and compress them along the height dimension to generate bird's-eye view features. S5. Using the adaptive 3D feature sampling module, the search features of each viewpoint are enhanced a priori based on the reference aggregation vector and the occlusion perception feature vector. Combined with 3D position encoding, feature sampling is performed from the bird's-eye view features through a cross-attention mechanism to obtain spatial perception search features, which are then sent to the detection head to output the target prediction and tracking results.

[0009] Optionally, the processing steps of the template aggregation module include: Introduce a learnable fusion token and viewpoint position encoding for each viewpoint; Add the reference feature sequences of all valid viewpoints to the corresponding viewpoint position codes, and concatenate them with the fusion token to form the input sequence; The input sequence is fed into a Transformer-based self-attention network for global interaction; Extract the output corresponding to the fusion token from the output sequence and perform a residual concatenation with the initial fusion token to obtain a global reference aggregation vector shared across perspectives.

[0010] Optionally, the occlusion sensing module adopts a UNet-like encoding and decoding structure, and the specific steps are as follows: The encoder continuously downsamples the two-dimensional search feature map from a specific perspective, extracts deep semantic information in the bottleneck layer, and obtains the bottleneck layer features. The bottleneck layer features are mapped to occlusion-aware feature vectors through global average pooling and fully connected layers, and the continuous occlusion prediction results are output through a linear layer for regression loss supervision. The decoder performs symmetrical upsampling and skip connections on the bottleneck layer features, outputs a feature mask, and obtains a channel-wise and spatially position-wise soft mask after Sigmoid activation. The soft mask is multiplied element-wise with the original search feature map to achieve adaptive suppression of the occluded region features.

[0011] Optionally, the adaptive 3D feature sampling module performs feature sampling by including: The occlusion-aware feature vector, the reference aggregation vector, and the search feature sequence are concatenated and then enhanced prior through a self-attention layer. A first 3D location code is generated for the enhanced search features, and a second 3D location code is generated for the bird's-eye view features; The enhanced search features are added to the first three-dimensional location code to form the query vector, and the bird's-eye view features are added to the second three-dimensional location code to form the key vector and value vector; The cross-attention mechanism enables 2D search features to adaptively address and aggregate 3D information in the bird's-eye view space; The sampling results are residually concatenated with the original search features and further processed to obtain spatially perceptual search features.

[0012] Optionally, the generation steps of the first three-dimensional position code are as follows: For each pixel position on the two-dimensional search feature map, multiple depth candidate values ​​are uniformly sampled along the depth direction; The pixel coordinates at the depth candidate values ​​are back-projected to the reference 3D coordinate system using the camera's intrinsic and extrinsic parameters. After normalizing the 3D coordinates according to the scene boundary, the coordinates are then subjected to inverse Sigmoid transformation and multilayer perceptron encoding, and the mean value of the depth candidate axes is taken to obtain the 3D position code corresponding to the pixel position.

[0013] Optionally, the model employs a two-stage training strategy: Phase 1: Basic training using single-view images to optimize the backbone network and detection head for specific view feature extraction; The second stage involves fine-tuning the entire framework using multi-view images, introducing a bird's-eye view loss function to supervise the construction of 3D spatial features, and introducing an occlusion prediction regression loss function to explicitly supervise the occlusion perception module.

[0014] On the other hand, a multi-view single-target tracking system based on adaptive 3D feature sampling is provided to implement the method described in any of the above embodiments, the system comprising: The image acquisition module is used to acquire video sequences containing search images and reference images from multiple perspectives in the same scene; The feature extraction module is used to input the video sequence into the backbone network and extract the search feature sequence and reference feature sequence for each viewpoint respectively; The template aggregation module is used to aggregate reference feature sequences from various perspectives into a unified reference aggregation vector. The occlusion perception module is used to predict the degree of occlusion of the search feature map corresponding to the search feature sequence of each view and generate a feature mask. The original search feature map is then modulated element by element using the feature mask to obtain the occlusion perception feature vector. The projection fusion module is used to project the occlusion perception feature vectors from various viewpoints onto a unified three-dimensional voxel space for fusion, and compress them along the height dimension to generate bird's-eye view features. An adaptive 3D feature sampling module is used to perform prior enhancement on the search features of each viewpoint based on the reference aggregation vector and the occlusion perception feature vector, and combine it with 3D position encoding to sample features from the bird's-eye view features through a cross-attention mechanism to obtain spatial perception search features. The detection module is used to send the spatially perceived search features into the detection head and output the predicted tracking results of the target.

[0015] On the other hand, an electronic device is provided, the electronic device comprising: processor; The memory stores computer-readable instructions, which, when loaded and executed by the processor, implement the steps of the multi-view single-target tracking method based on adaptive 3D feature sampling as described above.

[0016] On the other hand, a computer-readable storage medium is provided, wherein program code is stored in the computer-readable storage medium, and the program code can be called by a processor to execute the steps of the multi-view single-target tracking method based on adaptive 3D feature sampling described above.

[0017] The beneficial effects of the technical solution provided by this invention include at least the following: (1) Strong robustness against occlusion: By introducing an explicit occlusion degree regression supervision task and an occlusion perception module, the model can spontaneously identify and suppress the feature response of the occluded area, fundamentally preventing background noise from polluting the subsequent 3D fusion space.

[0018] (2) Improved global target appearance representation capability: The template aggregation module breaks the limitation of incomplete single-view appearance and uses the global perception capability of Transformer to construct a reference representation that integrates multi-angle information, providing more powerful prior guidance for the search phase.

[0019] (3) Refined and low-loss feature sampling: The traditional coarse feature stitching is replaced by three-dimensional position encoding and cross-attention mechanism. This adaptive three-dimensional feature sampling module enables two-dimensional pixels to autonomously retrieve missing information in the three-dimensional bird's-eye view space based on precise geometric coordinates, maximizing the preservation of spatial semantics and improving positioning accuracy. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart of a multi-view single-target tracking method based on adaptive 3D feature sampling provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall network architecture of the multi-view single-target tracking method provided in this embodiment of the invention; Figure 3 This is a schematic diagram of the network architecture of the occlusion perception module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the network architecture of the adaptive three-dimensional feature sampling module provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a multi-view single-target tracking system based on adaptive three-dimensional feature sampling provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0023] Example 1: Multi-view Single Target Tracking Method Flow

[0024] This invention provides a multi-view single-target tracking method based on adaptive 3D feature sampling. This method can be implemented by an electronic device, which can be a terminal or a server. (Reference) Figure 1 and Figure 2 As shown, the processing flow of this method may include the following steps: S1. Obtain video sequences containing search images and reference images from multiple perspectives within the same scene.

[0025] This step involves acquiring image data. Multiple camera sensors are used to set visible light image-level camera parameters for each viewing angle, and this process is responsible for image data acquisition and preprocessing.

[0026] S2. Input the video sequence into the backbone network and extract the search feature sequence and reference feature sequence for each viewpoint.

[0027] This step implements feature extraction and prior interaction. A Vision Transformer (ViT) initialized with DINOv2 is used as the backbone network. For the input N viewpoints of the search and reference images, the tokens of the reference and search images are concatenated and jointly encoded, allowing the prior information of the target's appearance to interact with the search region in the shallow layers. The final output shape is... Two-dimensional feature map .

[0028] S3. The template aggregation module aggregates the reference feature sequences of each viewpoint into a unified reference aggregation vector. The occlusion perception module predicts the occlusion degree of the search feature map corresponding to the search feature sequence of each viewpoint and generates a feature mask. The feature mask is used to modulate the original search feature map element by element to obtain the occlusion perception feature vector.

[0029] This step implements reference feature aggregation and occlusion-aware modulation, performing two operations in parallel: First, it uses the template aggregation module to extract the global reference aggregation vector. On the other hand, for each perspective ,Will Input occlusion perception module to extract occlusion perception feature vector And generate a soft mask Then, through element-wise multiplication... Modulate the original search feature map.

[0030] S4. Project the occlusion perception feature vectors from each viewpoint onto a unified three-dimensional voxel space for fusion, and compress them along the height dimension to generate bird's-eye view features.

[0031] This step achieves 3D feature projection and fusion. The modulated... Projected onto a unified [projection method] via a differentiable three-dimensional voxel backprojection operation. In 3D voxel space, after fusion, compression is performed along the Z-axis dimension to generate a bird's-eye view (BEV) feature. .

[0032] S5. Using the adaptive 3D feature sampling module, the search features of each viewpoint are enhanced a priori based on the reference aggregation vector and the occlusion perception feature vector. Combined with 3D position encoding, feature sampling is performed from the bird's-eye view features through a cross-attention mechanism to obtain spatial perception search features, which are then sent to the detection head to output the target prediction and tracking results.

[0033] This step achieves adaptive and accurate feature sampling. For each viewpoint... ,use and Prior enhancements are applied to the search features. Then, a physical mapping between the 2D token and 3D space is established using 3D positional encoding, and the BEV features are extracted through a cross-attention mechanism. Adaptive sampling is performed. The search features fused with spatial awareness are fed into the detection head to output the predicted bounding box of the target.

[0034] In this embodiment, a four-stage progressive processing architecture is adopted: First, in the feature extraction and prior interaction stage, the Vision Transformer is used to extract reference features and search features from each perspective.

[0035] Secondly, in the reference feature aggregation and occlusion-aware modulation stages, an occlusion-aware UNet module was designed. This module can predict the occlusion probability of the viewpoint, extract the global occlusion-aware feature vector, and generate a soft mask to modulate the search feature map element by element, actively filtering the interference features of the occluded area. At the same time, a template aggregation module based on self-attention was designed to fuse a global reference aggregation vector shared across viewpoints.

[0036] Next, in the 3D feature projection and fusion stage, the clean features that have been filtered out by occlusion are projected onto the 3D voxel space and fused and compressed into bird's-eye view (BEV) features.

[0037] Finally, in the adaptive precise feature sampling stage, the 2D search features are first enhanced a priori using the aforementioned occlusion-aware feature vectors and reference aggregation vectors. Then, a 3D position code (3D PE) is generated using camera parameters to physically map 2D pixels to a 3D coordinate system. Based on this code, a cross-attention mechanism is used to enable the 2D search features to actively address and adaptively sample from the BEV space, achieving fine-grained alignment and fusion of 2D and 3D information, which is ultimately fed into the detection head to output high-precision tracking results.

[0038] Compared with existing technologies, this invention can effectively filter out occlusion noise, achieve fine-grained and accurate alignment of two-dimensional and three-dimensional features, and significantly improve the target tracking accuracy and robustness in complex dynamic environments.

[0039] Example 2: Template aggregation module and occlusion detection module

[0040] The processing steps of the template aggregation module include: A learnable fusion token and viewpoint position codes for each viewpoint are introduced; the reference feature sequences of all effective viewpoints are added to the corresponding viewpoint position codes and concatenated with the fusion token to form the input sequence; the input sequence is fed into a Transformer-based self-attention network for global interaction; the output corresponding to the fusion token in the output sequence is extracted and residually connected with the initial fusion token to obtain a global reference aggregation vector shared across viewpoints.

[0041] Specifically, the template aggregation module targets N viewpoint reference sequences for the same target. Introducing learnable fusion tokens and viewpoint position encoding Construct the input sequence The sequence is input to a 3-layer TransformerBlock and interacts with it, extracting the output at position 0 and performing residual connections. This yields the global reference aggregation vector.

[0042] The occlusion perception module adopts a UNet-like encoding and decoding structure, and the specific steps are as follows: The encoder continuously downsamples the 2D search feature map from a specific viewpoint, extracts deep semantic information at the bottleneck layer, and obtains the bottleneck layer features. The bottleneck layer features are mapped through global average pooling and fully connected layers to obtain occlusion-aware feature vectors, and the continuous occlusion prediction results are output through a linear layer for regression loss supervision. The decoder performs symmetrical upsampling and skip connections on the bottleneck layer features, outputs a feature mask, and obtains a channel-wise and spatially-wise soft mask after Sigmoid activation. The soft mask is multiplied element-wise with the original search feature map to achieve adaptive suppression of occluded region features.

[0043] Specifically, refer to Figure 3 As shown, the occlusion perception module adopts a UNet-like structure. The encoder reduces the resolution to [a certain value] through convolution and max pooling. At the bottleneck layer, features are mapped through global average pooling and a fully connected layer to obtain occlusion-aware feature vectors. This vector is simultaneously connected to the prediction head output occlusion prediction. The decoder is supervised by continuous occlusion ratio labels in the range of [0, 1]. The decoder upsamples through transposed convolutions and skips connections to encoder features, then generates a mask via Sigmoid activation. .

[0044] Example 3: Adaptive 3D Feature Sampling Module

[0045] The adaptive 3D feature sampling module performs the following steps for feature sampling: The occlusion-aware feature vector, reference aggregation vector, and search feature sequence are concatenated and enhanced prior through a self-attention layer. A first 3D positional code is generated for the enhanced search features, and a second 3D positional code is generated for the bird's-eye view features. The enhanced search features are added to the first 3D positional code to obtain the query vector, and the bird's-eye view features are added to the second 3D positional code to obtain the key and value vectors. A cross-attention mechanism enables the 2D search features to adaptively address and aggregate 3D information in the bird's-eye view space. The sampling results are residually concatenated with the original search features and further processed to obtain the spatially aware search features.

[0046] The generation steps of the first three-dimensional position code are as follows: For each pixel position on the 2D search feature map, multiple depth candidate values ​​are uniformly sampled along the depth direction; the pixel coordinates at the depth candidate values ​​are back-projected to the reference 3D coordinate system using the camera's intrinsic and extrinsic parameters; after normalizing the 3D coordinates according to the scene boundary, they are encoded by inverse Sigmoid transformation and multilayer perceptron, and the mean value is taken over the depth candidate axis to obtain the 3D position code corresponding to the pixel position.

[0047] Specifically, refer to Figure 4 As shown, this module avoids the traditional simple vector concatenation.

[0048] Step S51: Prior Enhancement. Concatenate the occlusion perception vector, template aggregation vector, and search feature sequence. Prior augmented search features are obtained after interaction with the Transformer Block. .

[0049] Step S52: Generate 3D Position Encoding (3D PE). Eight candidate values ​​are uniformly sampled along the depth direction, back-projected to 3D world coordinates using camera parameters, normalized, and processed using inverse sigmoid and MLP. The mean value along the depth axis is then used to obtain the 3D position encoding of the search features. Simultaneously, location codes for BEV features are generated based on the BEV's physical coordinates. .

[0050] Step S53: Cross-attention sampling. For querying, with As key-value pairs, interactive computation is performed through the Transformer Decoder. This enables "precise addressing" of two-dimensional pixels in BEV.

[0051] Example 4: Model Training Strategy

[0052] This invention employs a two-stage training method.

[0053] Phase 1: Single-view basic training. The ViT encoder and detector head are optimized using a large-scale single-view dataset with a high learning rate until the model converges.

[0054] Phase Two: Fine-tuning of Multi-View Integration. Using only multi-view data, the batch size is adjusted to 2. Weights are introduced. BEV loss and weights The occlusion prediction regression loss (mean squared error loss) supervises the model, forcing the network to understand the three-dimensional geometric relationships and optimize the mask generation capability of the occlusion perception module.

[0055] Example 5: Multi-view Single Target Tracking System

[0056] Embodiments of the present invention also provide a multi-view single-target tracking system based on adaptive 3D feature sampling, referencing... Figure 5 As shown, the system includes: The image acquisition module is used to acquire video sequences containing search images and reference images from multiple perspectives in the same scene; The feature extraction module is used to input the video sequence into the backbone network and extract the search feature sequence and reference feature sequence for each viewpoint respectively; The template aggregation module is used to aggregate reference feature sequences from various perspectives into a unified reference aggregation vector. The occlusion perception module is used to predict the degree of occlusion of the search feature map corresponding to the search feature sequence of each view and generate a feature mask. The original search feature map is then modulated element by element using the feature mask to obtain the occlusion perception feature vector. The projection fusion module is used to project the occlusion perception feature vectors from various viewpoints onto a unified three-dimensional voxel space for fusion, and compress them along the height dimension to generate bird's-eye view features. An adaptive 3D feature sampling module is used to perform prior enhancement on the search features of each viewpoint based on the reference aggregation vector and the occlusion perception feature vector, and combine it with 3D position encoding to sample features from the bird's-eye view features through a cross-attention mechanism to obtain spatial perception search features. The detection module is used to send the spatially perceived search features into the detection head and output the predicted tracking results of the target.

[0057] For ease of explanation, Figure 5 Only the main components of the system are shown. The system of this embodiment can be used to perform... Figure 1 The technical solutions of the method embodiments shown are similar in principle and in effect, and will not be described again here.

[0058] Example 6: Electronic device and storage medium

[0059] In an exemplary embodiment, the present invention also provides an electronic device, the electronic device comprising: processor; The memory stores computer-readable instructions, which, when loaded and executed by the processor, implement the steps of the multi-view single-target tracking method based on adaptive 3D feature sampling as described above.

[0060] Specifically, the electronic device can be a server, a desktop computer, an in-vehicle computing platform, or an embedded device. The electronic device includes: Processor: Can be a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), or field-programmable gate array (FPGA). When processing the deep learning networks involved in this invention (such as the Vision Transformer backbone network, the occlusion-aware UNet network, and cross-attention matrix operations), it is preferable to use a GPU with powerful parallel computing capabilities (such as an Nvidia RTX 3090 or A40 architecture graphics card) for accelerated computation to meet the memory and computing power requirements of multi-view high-dimensional feature tensor processing.

[0061] Memory: Used to store computer programs and temporary data (such as multi-view image frames, generated soft mask matrices, high-dimensional three-dimensional voxel feature maps, and model weight parameters, etc.), and may include high-speed random access memory (RAM) and non-volatile memory (such as solid-state drives and flash memory).

[0062] Communication interface: Used to receive multi-view video data streams synchronously input from multiple external camera sensors, and output the final target tracking prediction coordinates to peripheral systems (such as security terminals or autonomous driving decision systems).

[0063] In an exemplary embodiment, the present invention also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the steps of the multi-view single-target tracking method based on adaptive 3D feature sampling described above. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0064] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0065] The use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0066] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0067] In the embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0068] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0069] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0070] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-view single object tracking method based on adaptive three-dimensional feature sampling, characterized in that, Includes the following steps: S1. Obtain video sequences containing search images and reference images from multiple perspectives in the same scene; S2. Input the video sequence into the backbone network and extract the search feature sequence and reference feature sequence for each viewpoint respectively; S3. The template aggregation module aggregates the reference feature sequences of each viewpoint into a unified reference aggregation vector. The occlusion perception module predicts the occlusion degree of the search feature map corresponding to the search feature sequence of each viewpoint and generates a feature mask. The feature mask is used to modulate the original search feature map element by element to obtain the occlusion perception feature vector. S4. Project the occlusion perception feature vectors from each viewpoint onto a unified three-dimensional voxel space for fusion, and compress them along the height dimension to generate bird's-eye view features. S5. Using the adaptive 3D feature sampling module, the search features of each viewpoint are enhanced a priori based on the reference aggregation vector and the occlusion perception feature vector. Combined with 3D position encoding, feature sampling is performed from the bird's-eye view features through a cross-attention mechanism to obtain spatial perception search features, which are then sent to the detection head to output the target prediction and tracking results.

2. The multi-view single object tracking method based on adaptive three-dimensional feature sampling according to claim 1, wherein, The processing steps of the template aggregation module include: Introduce a learnable fusion token and viewpoint position encoding for each viewpoint; Add the reference feature sequences of all valid viewpoints to the corresponding viewpoint position codes, and concatenate them with the fusion token to form the input sequence; The input sequence is fed into a Transformer-based self-attention network for global interaction; Extract the output corresponding to the fusion token from the output sequence and perform a residual concatenation with the initial fusion token to obtain a global reference aggregation vector shared across perspectives.

3. The multi-view single object tracking method based on adaptive three-dimensional feature sampling according to claim 1, wherein, The occlusion perception module adopts a UNet-like encoding and decoding structure, and the specific steps are as follows: The encoder continuously downsamples the two-dimensional search feature map from a specific perspective, extracts deep semantic information in the bottleneck layer, and obtains the bottleneck layer features. The bottleneck layer features are mapped to occlusion-aware feature vectors through global average pooling and fully connected layers, and the continuous occlusion prediction results are output through a linear layer for regression loss supervision. The decoder performs symmetrical upsampling and skip connections on the bottleneck layer features, outputs a feature mask, and obtains a channel-wise and spatially position-wise soft mask after Sigmoid activation. The soft mask is multiplied element-wise with the original search feature map to achieve adaptive suppression of the occluded region features.

4. The multi-view single object tracking method based on adaptive three-dimensional feature sampling according to claim 1, wherein, The adaptive 3D feature sampling module performs the following steps for feature sampling: The occlusion-aware feature vector, the reference aggregation vector, and the search feature sequence are concatenated and then enhanced prior through a self-attention layer. A first 3D location code is generated for the enhanced search features, and a second 3D location code is generated for the bird's-eye view features; The enhanced search features are added to the first three-dimensional location code to form the query vector, and the bird's-eye view features are added to the second three-dimensional location code to form the key vector and value vector; The cross-attention mechanism enables 2D search features to adaptively address and aggregate 3D information in the bird's-eye view space; The sampling results are residually concatenated with the original search features and further processed to obtain spatially perceptual search features.

5. The multi-view single object tracking method based on adaptive three-dimensional feature sampling according to claim 4, characterized in that, The steps for generating the first three-dimensional position code are as follows: For each pixel position on the two-dimensional search feature map, multiple depth candidate values ​​are uniformly sampled along the depth direction; The pixel coordinates at the depth candidate values ​​are back-projected to the reference 3D coordinate system using the camera's intrinsic and extrinsic parameters. After normalizing the 3D coordinates according to the scene boundary, the coordinates are then subjected to inverse Sigmoid transformation and multilayer perceptron encoding, and the mean value of the depth candidate axes is taken to obtain the 3D position code corresponding to the pixel position.

6. The multi-view single-target tracking method based on adaptive 3D feature sampling according to claim 1, characterized in that, The model employs a two-stage training strategy: Phase 1: Basic training using single-view images to optimize the backbone network and detection head for specific view feature extraction; The second stage involves fine-tuning the entire framework using multi-view images, introducing a bird's-eye view loss function to supervise the construction of 3D spatial features, and introducing an occlusion prediction regression loss function to explicitly supervise the occlusion perception module.

7. A multi-view single-target tracking system based on adaptive 3D feature sampling, the system being used to implement the method as described in any one of claims 1 to 6, characterized in that, The system includes: The image acquisition module is used to acquire video sequences containing search images and reference images from multiple perspectives in the same scene; The feature extraction module is used to input the video sequence into the backbone network and extract the search feature sequence and reference feature sequence for each viewpoint respectively; The template aggregation module is used to aggregate reference feature sequences from various perspectives into a unified reference aggregation vector. The occlusion perception module is used to predict the degree of occlusion of the search feature map corresponding to the search feature sequence of each view and generate a feature mask. The original search feature map is then modulated element by element using the feature mask to obtain the occlusion perception feature vector. The projection fusion module is used to project the occlusion perception feature vectors from various viewpoints onto a unified three-dimensional voxel space for fusion, and compress them along the height dimension to generate bird's-eye view features. An adaptive 3D feature sampling module is used to perform prior enhancement on the search features of each viewpoint based on the reference aggregation vector and the occlusion perception feature vector, and combine it with 3D position encoding to sample features from the bird's-eye view features through a cross-attention mechanism to obtain spatial perception search features. The detection module is used to send the spatially perceived search features into the detection head and output the predicted tracking results of the target.

8. An electronic device, characterized in that, The electronic device includes: processor; A memory storing computer-readable instructions that, when loaded and executed by the processor, implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 6.

Images