Methods, devices, equipment, and media for multi-view collaborative target object tracking

By projecting the image features of the current viewpoint and the auxiliary viewpoint into the same preset space in multi-view collaborative target detection and tracking, and performing spatial offset resampling and feature fusion, the problem of insufficient cross-viewpoint alignment accuracy is solved, and high-precision tracking of target objects is achieved.

CN123118701APending Publication Date: 2026-11-17AEROSPACE INFORMATION RES INST CAS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611458266.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-18
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

In multi-view collaborative target detection and tracking, existing technologies suffer from severe imaging distortion under different flight states of UAVs, resulting in insufficient cross-view geometric alignment accuracy and affecting target recognition and tracking accuracy.

Method used

By projecting the image features of the current viewpoint and the auxiliary viewpoint onto the same preset feature metric space, determining the spatial offset and resampling, the auxiliary viewpoint features are aligned with the current viewpoint features. Feature fusion is then achieved by combining depthwise separable convolution and a hybrid expert model, thus realizing semantic-level alignment and spatial correction of cross-viewpoint features.

Benefits of technology

It improves the accuracy of cross-view alignment, reduces spatial misalignment and fusion noise, and enhances the accuracy and reliability of target tracking.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118701A_ABST
    Figure CN123118701A_ABST
Patent Text Reader

Abstract

This application provides a method, apparatus, device, and medium for multi-view collaborative target object tracking, applied in the field of computer vision. The method includes: acquiring a current-view image and an auxiliary-view image of the target region; projecting the current-view search region features of the current-view image and the auxiliary-view search region features of the auxiliary-view image onto the same preset feature metric space, respectively, to obtain current-view projection features and auxiliary-view projection features; determining the spatial offset of the current-view based on the current-view projection features and the auxiliary-view projection features, and resampling the auxiliary-view projection features based on the spatial offset to obtain spatially aligned auxiliary-view search region features; and mapping the sub-search features that match the template features from the search features obtained by fusing the current-view search region features and the spatially aligned auxiliary-view search region features to the current-view image to obtain the tracking position of the target object.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of computer vision, and more specifically to a method, apparatus, device, and medium for tracking target objects in a multi-view collaborative manner. Background Technology

[0002] Multi-view object tracking (MVOT) technology is an important research direction in intelligent sensing systems, with applications in autonomous driving, intelligent security, drone inspection, emergency search and rescue, and intelligent transportation. As intelligent sensing tasks evolve from single-platform to multi-platform and from fixed-viewpoint to dynamic multi-viewpoint, collaborative sensing from multiple sensor sources is becoming a trend. Particularly in air-ground collaborative sensing systems composed of drones, ground cameras, and mobile robots, reliably fusing visual information from different altitudes, imaging modes, and postures to achieve continuous cross-viewpoint tracking has been a major research focus in recent years.

[0003] However, in related technologies, all perspectives are usually mapped onto a plane through a unified projection. But due to the severe image distortion between the aerial perspective imaging of UAVs at different flight states, such as different altitudes and attitudes, and the ground perspective imaging of ground cameras and mobile robots, the accuracy of cross-perspective geometric alignment is insufficient when performing projection mapping, which affects the target recognition and tracking accuracy. Summary of the Invention

[0004] In view of the above problems, this application provides a method, apparatus, device and medium for target object tracking oriented to multi-view collaboration.

[0005] According to the first aspect of this application, a target object tracking method for multi-view collaboration is provided, comprising: in response to detecting a target object in a target region, acquiring a current-view image and an auxiliary-view image of the target region from a current view and an auxiliary view, respectively, wherein the current view and the auxiliary view are different views; projecting the current-view search region features of the current-view image and the auxiliary-view search region features of the auxiliary-view image onto the same preset feature metric space, respectively, to obtain current-view projection features and auxiliary-view projection features, wherein in the preset feature metric space, the similarity between cross-view projection features at the same geographical location is greater than the similarity between cross-view projection features at different geographical locations; determining the spatial offset of the current view relative to the auxiliary view based on the current-view projection features and the auxiliary-view projection features, and resampling the auxiliary-view projection features based on the spatial offset to transform the auxiliary-view projection features into a state aligned with the current-view search region features, thereby obtaining spatially aligned auxiliary-view search region features; mapping the sub-search features that match the template features in the search features obtained by fusing the current-view search region features and the spatially aligned auxiliary-view search region features to the current-view image to obtain the tracking position of the target object, wherein the template features represent the attribute features of the target object under the current view.

[0006] According to an embodiment of this application, the current view search region features of the current view image and the auxiliary view search region features of the auxiliary view image are projected onto the same preset feature metric space to obtain the current view projection features and the auxiliary view projection features, respectively. This includes: inputting the current view search region features and the auxiliary view search region features into a pre-trained projection network, and outputting the current view projection features and the auxiliary view projection features accordingly. The projection network is pre-trained through the following steps: for the sample regions where multiple sample objects are located, current view sample images and auxiliary view sample images of the sample regions are collected from the current view and the auxiliary view, respectively, to obtain multiple current view sample images and multiple auxiliary view sample images; feature extraction is performed on each current view sample image and each auxiliary view sample image to obtain the corresponding current view sample search region features. The system searches for regional features using both current and auxiliary viewpoint samples. Based on these regional features, positive and negative sample feature pairs are constructed. Positive feature pairs indicate the same sample object, while negative feature pairs indicate different sample objects. These pairs are then input into an initial projection network to project both pairs onto the same preset feature metric space, outputting the corresponding current viewpoint sample projection features and auxiliary viewpoint sample projection features. The initial projection network is trained with the training objective of maximizing the similarity between the current viewpoint sample projection features and auxiliary viewpoint sample projection features corresponding to positive feature pairs, and minimizing the similarity between the current viewpoint sample projection features and auxiliary viewpoint sample projection features corresponding to negative feature pairs.

[0007] According to an embodiment of this application, determining the spatial offset of the current view relative to the auxiliary view based on the current view projection features and the auxiliary view projection features includes: stitching the current view projection features and the auxiliary view projection features to obtain stitched features; downsampling the stitched features to obtain primary stitched features; performing channel-level feature processing on the primary stitched features to obtain the horizontal offset and the vertical offset of the current view relative to the auxiliary view; and mapping the horizontal offset and the vertical offset to their respective ranges to obtain the spatial offset.

[0008] According to an embodiment of this application, the auxiliary view projection features are resampled based on the spatial offset to transform the auxiliary view projection features into a state aligned with the current view search region features, thereby obtaining spatially aligned auxiliary view search region features. This includes: constructing a reference coordinate grid of the auxiliary view projection features based on the auxiliary view projection features; correcting the reference coordinate grid based on the spatial offset to obtain a sampling position grid; and resampling the auxiliary view projection features according to the sampling position grid to obtain spatially aligned auxiliary view search region features.

[0009] According to embodiments of this application, the target object tracking method for multi-view collaboration further includes: using template features as convolution kernels to perform depthwise separable convolution operations on spatially aligned auxiliary view search region features, and outputting a similarity heatmap corresponding to the auxiliary view search region; using the similarity heatmap as spatial attention weights, performing channel-dimension weighted multiplication of the similarity heatmap and the spatially aligned auxiliary view search region features at the same spatial location to obtain enhanced auxiliary view search region features; and fusing the current view search region features and the enhanced auxiliary view search region features to obtain fused search features.

[0010] According to an embodiment of this application, the current viewpoint search region features and the enhanced auxiliary viewpoint search region features are fused to obtain fused search features. This includes: inputting the current viewpoint search region features and the enhanced auxiliary viewpoint search region features into a pre-trained hybrid expert model, and outputting the fused search features after performing the following operations. The hybrid expert model includes a classifier, a multi-branch encoder, and a gating network: using the classifier, classifying the current viewpoint search region features and the enhanced auxiliary viewpoint search region features to obtain probability values ​​that match multiple tracking scene attribute features of the target region. The tracking scene attribute features include the target object's motion speed features, the target region's background complexity features, and the target object's position. Occlusion probability features and target area illumination variation features are used. Based on the probability values ​​of matching multiple tracking scene attribute features of the target area, a preset number of encoders are selected from the multi-branch encoders. Using the preset number of encoders, the current view search area features and the enhanced auxiliary view search area features are encoded respectively, and the encoded current view search area features and the encoded enhanced auxiliary view search area features are fused respectively to obtain a preset number of fused features. Using a gating network, the preset number of fused features are weighted and fused to obtain the fused search features. The weight values ​​of the preset number of fused features are determined based on the matching degree between the fused features and the tracking scene attribute features.

[0011] According to an embodiment of this application, the sub-search features that match the template features in the search features obtained by fusing the current viewpoint search region features and the spatially aligned auxiliary viewpoint search region features are mapped to the current viewpoint image to obtain the tracking position of the target object. This includes: performing spatial similarity measurement on each sub-search feature in the template features and search features to generate a similarity response map; determining the sub-search feature with the largest spatial similarity measurement with the template features in the similarity response map as the sub-search feature that matches the template features; and mapping the spatial position of the sub-search feature that matches the template features in the search features to the spatial position in the current viewpoint image to obtain the tracking position.

[0012] A second aspect of this application provides a target object tracking device for multi-view collaboration, comprising: an acquisition module, configured to acquire, in response to detecting a target object within a target area, a current-view image and an auxiliary-view image of the target area from a current viewpoint and an auxiliary viewpoint, respectively, wherein the current viewpoint and the auxiliary viewpoint are different viewpoints; and a projection module, configured to project the current-view search region features of the current-view image and the auxiliary-view search region features of the auxiliary-view image onto the same preset feature metric space, respectively, to obtain current-view projection features and auxiliary-view projection features, wherein in the preset feature metric space, the similarity between cross-view projection features at the same geographical location is greater than that between cross-view projection features at different geographical locations. The system includes: a similarity module; an alignment module, which determines the spatial offset of the current view relative to the auxiliary view based on the current view projection features and the auxiliary view projection features, and resamples the auxiliary view projection features based on the spatial offset to transform the auxiliary view projection features into a state aligned with the current view search region features, thus obtaining the spatially aligned auxiliary view search region features; and a mapping module, which maps the sub-search features that match the template features in the search features obtained by fusing the current view search region features and the spatially aligned auxiliary view search region features to the current view image to obtain the tracking position of the target object. The template features represent the attribute features of the target object under the current view.

[0013] A third aspect of this application provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0014] A fourth aspect of this application also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0015] According to embodiments of this application, by projecting the current view search region features and the auxiliary view search region features onto the same preset feature metric space, and constraining the similarity between cross-view projected features at the same geographical location in the preset feature metric space to be greater than the similarity between different geographical locations, semantic-level alignment of cross-view features can be achieved. This alleviates the geometric mismatch problem caused by different flight states of the UAV and enhances the accuracy of cross-view alignment in scenarios with large view differences. Resampling the auxiliary view projected features using spatial offsets aligns the auxiliary view projected features with the current view search region features spatially, reducing spatial misalignment between the two views, avoiding fusion noise introduced by spatial inconsistency, and reducing the complexity of feature fusion. Furthermore, by establishing semantic alignment relationships between features from different views and performing spatial correction, semantic mismatch problems introduced by spatial transformation before feature semantic alignment can be avoided. This allows cross-view alignment to progress layer by layer from semantic to spatial, providing semantically consistent and spatially matched feature inputs, thereby improving the accuracy of cross-view alignment. The current viewpoint search region features are fused with the spatially aligned auxiliary viewpoint search region features and then matched with the template features. The fusion process integrates the feature information of the two viewpoints, so that the fused search features contain the original spatial information of the current viewpoint and the complementary information of the auxiliary viewpoint after semantic alignment and spatial correction. After fusion, the target tracking position is output after matching with the template features and mapping back to the current viewpoint image, which can obtain reliable target localization and achieve the accuracy of target tracking. Attached Figure Description

[0016] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 A flowchart illustrating a target object tracking method for multi-view collaboration according to an embodiment of this application is shown schematically.

[0018] Figure 2 A data flow diagram of a training projection network according to an embodiment of this application is illustrated schematically;

[0019] Figure 3 A schematic diagram of a projection network according to an embodiment of this application is shown.

[0020] Figure 4 A data flow diagram illustrating the determination of spatial offsets according to an embodiment of this application is shown schematically.

[0021] Figure 5 This illustration schematically shows a data flow diagram for determining the enhanced auxiliary view search region features according to an embodiment of this application;

[0022] Figure 6 A schematic diagram illustrating the structure of a hybrid expert model according to an embodiment of this application is shown.

[0023] Figure 7 This diagram schematically illustrates a target object tracking framework according to an embodiment of the present application.

[0024] Figure 8 This schematically illustrates a structural block diagram of a multi-view collaborative target object tracking device according to an embodiment of this application; and

[0025] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a multi-view collaborative target object tracking method according to an embodiment of this application. Detailed Implementation

[0026] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0027] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0028] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0029] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0030] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0031] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this application all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0032] Figure 1 A flowchart illustrating a target object tracking method for multi-view collaboration according to an embodiment of this application is shown.

[0033] like Figure 1 As shown, the target object tracking method for multi-view collaboration in this embodiment includes operations S110 to S140.

[0034] In operation S110, in response to detecting a target object in the target area, the current view image and the auxiliary view image of the target area are acquired from the current view and the auxiliary view, respectively.

[0035] In operation S120, the current view search region features of the current view image and the auxiliary view search region features of the auxiliary view image are projected onto the same preset feature metric space, respectively, to obtain the current view projection features and the auxiliary view projection features.

[0036] In operation S130, based on the current view projection features and the auxiliary view projection features, the spatial offset of the current view relative to the auxiliary view is determined, and based on the spatial offset, the auxiliary view projection features are resampled to transform the auxiliary view projection features into a state aligned with the current view search area features, thus obtaining the spatially aligned auxiliary view search area features.

[0037] In operation S140, the sub-search features that match the template features in the search features obtained by fusing the current view search region features and the spatially aligned auxiliary view search region features are mapped to the current view image to obtain the tracking position of the target object.

[0038] In this embodiment, to achieve continuous tracking of the target object's position, after detecting the target object within the target area, an image is acquired from the current viewpoint and an image is acquired from the auxiliary viewpoint. The current viewpoint and the auxiliary viewpoint can be understood as two observation angles from different shooting positions. For example, in an air-to-ground collaborative scenario, the viewpoint of an aerial drone and the viewpoint of a ground-based fixed camera differ in altitude, pitch angle, or imaging attitude. It can be understood that when the aerial drone's viewpoint is used as the current viewpoint, the ground-based fixed camera's viewpoint is the auxiliary viewpoint; conversely, when the ground-based fixed camera's viewpoint is used as the current viewpoint, the aerial drone's viewpoint is the auxiliary viewpoint.

[0039] Feature extraction is performed on both the current viewpoint image and the auxiliary viewpoint image to obtain the current viewpoint search region features and the auxiliary viewpoint search region features, respectively. These features are then projected onto the same preset feature metric space to obtain the current viewpoint projection features and the auxiliary viewpoint projection features. Within this preset feature metric space, the similarity between cross-viewpoint projection features corresponding to the same geographical location is greater than the similarity between cross-viewpoint projection features corresponding to different geographical locations. Therefore, the resulting current viewpoint projection features and auxiliary viewpoint projection features, after projection, can be used to indicate the search region features for the same geographical location.

[0040] By measuring the offsets of the current viewpoint projection features and the auxiliary viewpoint projection features within the aforementioned preset feature metric space, the spatial offset of the current viewpoint relative to the auxiliary viewpoint can be determined. After obtaining the spatial offset, the auxiliary viewpoint projection features are resampled to obtain spatially aligned auxiliary viewpoint search region features. Resampling can be performed at the pixel level on the feature map within the aforementioned preset feature metric space using the specific value of the spatial offset. For example, bilinear interpolation can be used to resample feature values, aligning the spatial coordinates of the auxiliary viewpoint projection features with the spatial coordinates of the current viewpoint search region features.

[0041] The search features are obtained by fusing the current viewpoint search region features with the spatially aligned auxiliary viewpoint search region features. Fusion methods may include, but are not limited to, element-wise summation and channel concatenation. Essentially, the fused search features contain both the original spatial information of the current viewpoint and complementary information from the spatially aligned auxiliary viewpoint. The fused search features are then matched with template features to obtain sub-search features. Template features can be understood as the attribute features of the target object in the current viewpoint when the target object appears in the target region. Template features characterize the appearance, scale, and other attributes of the target object in the current viewpoint. The matched sub-search features are then mapped onto the current viewpoint image to obtain the tracking position of the target object. The tracking position can be understood as the bounding box position and scale of the target object in the current viewpoint image, i.e., the horizontal coordinate, vertical coordinate, width, and height.

[0042] According to embodiments of this application, by projecting the current view search region features and the auxiliary view search region features onto the same preset feature metric space, and constraining the similarity between cross-view projected features at the same geographical location in the preset feature metric space to be greater than the similarity between different geographical locations, semantic-level alignment of cross-view features can be achieved. This alleviates the geometric mismatch problem caused by different flight states of the UAV and enhances the accuracy of cross-view alignment in scenarios with large view differences. Resampling the auxiliary view projected features using spatial offsets aligns the auxiliary view projected features with the current view search region features spatially, reducing spatial misalignment between the two views, avoiding fusion noise introduced by spatial inconsistency, and reducing the complexity of feature fusion. Furthermore, by establishing semantic alignment relationships between features from different views and performing spatial correction, semantic mismatch problems introduced by spatial transformation before feature semantic alignment can be avoided. This allows cross-view alignment to progress layer by layer from semantic to spatial, providing semantically consistent and spatially matched feature inputs, thereby improving the accuracy of cross-view alignment. The current viewpoint search region features are fused with the spatially aligned auxiliary viewpoint search region features and then matched with the template features. The fusion process integrates the feature information of the two viewpoints, so that the fused search features contain the original spatial information of the current viewpoint and the complementary information of the auxiliary viewpoint after semantic alignment and spatial correction. After fusion, the target tracking position is output after matching with the template features and mapping back to the current viewpoint image, which can obtain reliable target localization and achieve the accuracy of target tracking.

[0043] Figure 2 A data flow diagram of a training projection network according to an embodiment of this application is illustrated schematically.

[0044] like Figure 2As shown, multiple current viewpoint sample images and multiple auxiliary viewpoint sample images are input into the projection network. Feature extraction is performed on these multiple images to obtain multiple current viewpoint sample search region features and multiple auxiliary viewpoint sample search region features. For example, in the figure, the blue triangles, blue squares, and blue circles represent different current viewpoint sample search region features, and the green triangles, green squares, and green circles represent different auxiliary viewpoint sample search region features.

[0045] Based on the search region features of multiple current viewpoint samples and multiple auxiliary viewpoint samples, the corresponding projection features of the current viewpoint samples and the projection features of the auxiliary viewpoint samples are output through comparative learning. That is, the positive sample feature pairs shown in the figure can indicate the projection features of the current viewpoint samples and the projection features of the auxiliary viewpoint samples for the same sample object.

[0046] Figure 3 A schematic diagram of a projection network according to an embodiment of this application is shown.

[0047] like Figure 3 As shown, the projection network can include a backbone network stacked N times, a Dropout regularization layer, and an L2 Norm (L2 norm normalization) layer. The backbone network can include Convolutional layers, ReLU activation functions (Rectified Linear Units), and Batch Normalization (BatchNorm). These three sub-units in the backbone network are stacked N times in a complete loop, serving as the backbone for feature extraction. The number of stacks, N, can be customized according to the target task. The Dropout regularization layer enables the projection network to learn general features, suppressing overfitting and preventing feature redundancy. The L2 Norm (L2 norm normalization) layer can be applied to feature matching, eliminating the influence of feature magnitude, unifying feature scale, and making the distance calculation between features more accurate.

[0048] According to an embodiment of this application, the current view search region features of the current view image and the auxiliary view search region features of the auxiliary view image are projected onto the same preset feature metric space to obtain the current view projection features and the auxiliary view projection features, respectively. This includes: inputting the current view search region features and the auxiliary view search region features into a pre-trained projection network, and outputting the current view projection features and the auxiliary view projection features accordingly. The projection network is pre-trained through the following steps: for the sample regions where multiple sample objects are located, current view sample images and auxiliary view sample images of the sample regions are collected from the current view and the auxiliary view, respectively, to obtain multiple current view sample images and multiple auxiliary view sample images; feature extraction is performed on each current view sample image and each auxiliary view sample image to obtain the corresponding current view sample search region features. The system searches for regional features using both current and auxiliary viewpoint samples. Based on these regional features, positive and negative sample feature pairs are constructed. Positive feature pairs indicate the same sample object, while negative feature pairs indicate different sample objects. These pairs are then input into an initial projection network to project both pairs onto the same preset feature metric space, outputting the corresponding current viewpoint sample projection features and auxiliary viewpoint sample projection features. The initial projection network is trained with the training objective of maximizing the similarity between the current viewpoint sample projection features and auxiliary viewpoint sample projection features corresponding to positive feature pairs, and minimizing the similarity between the current viewpoint sample projection features and auxiliary viewpoint sample projection features corresponding to negative feature pairs.

[0049] In this embodiment, the current viewpoint search region features and the auxiliary viewpoint search region features are respectively input into a pre-trained projection network, and the projection network outputs the current viewpoint projection features and the auxiliary viewpoint projection features accordingly. The pre-training process of the projection network can be implemented through the following steps:

[0050] For each sample region containing multiple sample objects, current-view sample images and auxiliary-view sample images of the sample region are acquired from both the current viewpoint and auxiliary viewpoint, resulting in multiple current-view sample images and multiple auxiliary-view sample images. The sample objects can be different target objects or pedestrians. For each sample object, current-view sample images and auxiliary-view sample images of the aforementioned sample object are acquired simultaneously from both viewpoints.

[0051] Feature extraction is performed on each current viewpoint sample image and each auxiliary viewpoint sample image to obtain the corresponding search region features for the current viewpoint sample and the search region features for the auxiliary viewpoint sample. In one example, feature extraction can be performed as follows: Figure 3The backbone network shown is stacked N times. In another example, feature extraction can also employ a backbone network that shares weights with the tracking network, such as ResNet50 (Residual Network 50, a 50-layer residual network), to transform the sample image data into a high-dimensional feature tensor.

[0052] Based on the search region features of multiple current-view samples and multiple auxiliary-view samples, positive and negative sample feature pairs are constructed. Positive sample feature pairs indicate the same sample object; that is, they can be understood as pairing the current-view sample search region features of the same sample object with the auxiliary-view sample search region features. Negative sample feature pairs indicate different sample objects; that is, they can be understood as pairing the current-view sample search region features of different sample objects with the auxiliary-view sample search region features. For example, the current-view sample search region features of sample object A and the auxiliary-view sample search region features of sample object B constitute a negative sample feature pair.

[0053] Positive and negative sample feature pairs are input into the initial projection network. The initial projection network projects all features from both pairs onto the same preset feature metric space, outputting the corresponding current viewpoint sample projection features and auxiliary viewpoint sample projection features. In one example, the projection network structure can be as follows: Figure 3 As shown. In another example, the projection network can employ a multi-layer perceptron (MLP), such that the features of the current viewpoint and the features of the auxiliary viewpoint are mapped to the same predefined feature metric space through the same transformation.

[0054] The initial projection network training objective can be to maximize the similarity between the projected features of positive sample pairs corresponding to the current viewpoint sample projection features and the projected features of auxiliary viewpoint sample projection features, and minimize the similarity between the projected features of negative sample pairs corresponding to the current viewpoint sample projection features and the projected features of auxiliary viewpoint sample projection features. The similarity can be measured using cosine similarity or dot product similarity, and the loss function used during training can be contrastive loss or Information Noise-Contrastive Estimation (InfoNCE) loss. By iteratively optimizing the parameters of the initial projection network, the distance between positive sample feature pairs in the same predefined feature metric space is reduced, while the distance between negative sample pairs in the projection space is increased, resulting in the trained projection network. After the above training, the projection network can map any input current viewpoint features and auxiliary viewpoint features to the same predefined feature metric space. Cross-viewpoint features corresponding to the same spatial location in the same predefined feature metric space have high similarity, while cross-viewpoint features corresponding to different locations have low similarity.

[0055] According to embodiments of this application, a projection network is trained by constructing positive and negative feature pairs of multiple sample objects under the current and auxiliary viewpoints, and by maximizing the similarity of positive sample pairs in the projection space and minimizing the similarity of negative sample pairs in the projection space. This enables the projection network to map the features of any two input viewpoints to the same preset feature metric space. In this space, the cross-viewpoint features of the same target object are brought closer, while the cross-viewpoint features of different target objects are pushed further apart, alleviating the geometric distortion and viewpoint mismatch problems caused by changes in UAV altitude, pitch angle, and roll angle. After training, the projection network can complete the cross-viewpoint feature mapping through forward propagation during the inference phase without introducing significant computational overhead, meeting the requirements of real-time tracking tasks.

[0056] Figure 4 A data flow diagram illustrating the determination of spatial offsets according to an embodiment of this application is shown schematically.

[0057] like Figure 4 As shown, the projection features of the current viewpoint and the projection features of the auxiliary viewpoint are concatenated to obtain the concatenated features. The concatenated features are then downsampled to obtain the primary concatenated features. Channel-level feature processing is performed on the primary concatenated features to obtain the horizontal and vertical offsets of the current viewpoint relative to the auxiliary viewpoint. The horizontal and vertical offsets are then mapped to their respective ranges to obtain the spatial offset. .

[0058] According to an embodiment of this application, determining the spatial offset of the current view relative to the auxiliary view based on the current view projection features and the auxiliary view projection features includes: stitching the current view projection features and the auxiliary view projection features to obtain stitched features; downsampling the stitched features to obtain primary stitched features; performing channel-level feature processing on the primary stitched features to obtain the horizontal offset and the vertical offset of the current view relative to the auxiliary view; and mapping the horizontal offset and the vertical offset to their respective ranges to obtain the spatial offset.

[0059] In this embodiment, the projection features of the current viewpoint and the projection features of the auxiliary viewpoint can be stitched together along the channel dimension to obtain the stitched features. The stitched features can be understood as a fusion of the projection features of the two viewpoints, providing a joint input for calculating the spatial offset.

[0060] The concatenated features are downsampled to obtain initial concatenated features. Downsampling can be implemented using PooL pooling layers, such as adaptive average pooling or max pooling, to reduce the spatial resolution of the features and expand the receptive field, allowing for the perception of a wider range of contextual spatial relationships when calculating spatial offsets. It is understood that downsampling can also be implemented using convolutional layers with a stride greater than 1; the specific implementation method is not limited here.

[0061] Channel-level feature processing is performed on the initial stitched features to obtain the horizontal and vertical offsets of the current viewpoint relative to the auxiliary viewpoint. Channel-level feature processing can be implemented using Conv convolutional layers. For example, convolutional layers can be used to transform the initial stitched features, progressively regressing the offset values ​​in both the horizontal and vertical directions. It is understood that the channel-level feature processing process can involve multiple stacked convolutional and activation layers, such as a Conv convolutional layer followed by a ReLU activation function, extracting features step-by-step through multiple transformations. The specific implementation method is not limited here.

[0062] The spatial offset is obtained by mapping the ranges of the horizontal and vertical offsets. This range mapping can be implemented using the Tanh activation function, which restricts the numerical ranges of the horizontal and vertical offsets to a preset interval; for example, normalizing the offsets to a given interval. or k is a preset constant, which constrains the effective range of the spatial offset and prevents feature space misalignment after subsequent resampling of the auxiliary view projection features due to extreme offset values. It is understandable that the range mapping can also be implemented using the Sigmoid function in conjunction with scaling and translation; the specific implementation method is not limited here.

[0063] According to embodiments of this application, by stitching together the current view projection features and the auxiliary view projection features and then downsampling, the feature space resolution can be reduced without losing cross-view joint information, thus reducing the computational load of spatial offsets. Channel-level feature processing is performed on the initial stitched features to extract horizontal and vertical offsets. Value range mapping is then applied to the horizontal and vertical offsets, constraining the spatial offset values ​​within a fixed range. This prevents extreme offset values ​​from causing coordinate out-of-bounds errors or feature space misalignment during subsequent resampling, ensuring that resampling operations are performed within the effective pixel range, thereby improving the stability and reliability of spatial alignment. Completing the calculation of spatial offsets meets the requirements of real-time tracking tasks and provides an accurate spatial transformation basis for subsequent resampling of the auxiliary view projection features.

[0064] According to an embodiment of this application, the auxiliary view projection features are resampled based on the spatial offset to transform the auxiliary view projection features into a state aligned with the current view search region features, thereby obtaining spatially aligned auxiliary view search region features. This includes: constructing a reference coordinate grid of the auxiliary view projection features based on the auxiliary view projection features; correcting the reference coordinate grid based on the spatial offset to obtain a sampling position grid; and resampling the auxiliary view projection features according to the sampling position grid to obtain spatially aligned auxiliary view search region features.

[0065] In this embodiment, a reference coordinate grid for the auxiliary view projection features is constructed based on the auxiliary view projection features. The auxiliary view projection features can be understood as feature maps with spatial dimensions. Each position on the feature map corresponds to a pixel or spatial region on the image plane, and the reference coordinate grid is the original coordinate mapping of each position in the feature map in the spatial coordinate system. Specifically, the reference coordinate grid records the two-dimensional coordinate values ​​corresponding to each spatial position in the auxiliary view projection features. The coordinate values ​​are typically arranged at equal intervals along the height and width directions of the feature map.

[0066] Based on the spatial offset, the reference coordinate grid is corrected to obtain the sampling position grid. The spatial offset includes displacement values ​​in the horizontal and vertical directions. These displacement values ​​are superimposed on the corresponding coordinate values ​​of the reference coordinate grid, causing the coordinates of each point in the reference coordinate grid to shift and form a new coordinate distribution, thus obtaining the sampling position grid.

[0067] The auxiliary viewpoint projection features are resampled according to the sampling position grid to obtain spatially aligned auxiliary viewpoint search region features. Resampling can be performed on the feature map representing the auxiliary viewpoint projection features based on the coordinates of each position in the sampling position grid. Since the coordinates in the sampling position grid may be floating-point numbers and cannot be directly mapped to integer pixel positions on the feature map, bilinear interpolation can be used. Specifically, on the feature map representing the auxiliary viewpoint projection features, the feature values ​​of the four adjacent integer positions around the sampling coordinate are weighted and summed to obtain the feature value at that sampling coordinate. This process is repeated point-by-point through all coordinate positions in the sampling position grid, generating a feature map representing the spatially aligned auxiliary viewpoint search region features. The spatial coordinate system of this feature map is consistent with the spatial coordinate system of the current viewpoint search region features, enabling spatial alignment transformation of the auxiliary viewpoint projection features to obtain the spatially aligned auxiliary viewpoint search region features.

[0068] According to embodiments of this application, a reference coordinate grid is constructed based on the auxiliary view projection features, establishing a clear spatial position reference for the auxiliary view projection features and providing an operable coordinate basis for resampling operations. The sampling position grid is obtained by correcting the reference coordinate grid based on spatial offsets, enabling independent positioning and adjustment of each spatial position of the auxiliary view projection features in both horizontal and vertical directions. This preserves the relative spatial structure relationships between different regions within the auxiliary view projection features and avoids spatial distortion caused by coarse-grained alignment. Resampling the auxiliary view projection features according to the sampling position grid ensures accurate matching between the spatially aligned auxiliary view search region features and the current view search region features in the spatial coordinate system, thereby reducing the spatial differences between the features of the two views.

[0069] Figure 5 The diagram illustrates a data flow graph for determining the enhanced auxiliary view search region features according to an embodiment of this application.

[0070] like Figure 5 As shown, the red boxes represent template features, and the blue boxes represent spatially aligned auxiliary view search region features. Using the template features as convolution kernels, depthwise separable convolution operations are performed on the spatially aligned auxiliary view search region features. The MLP outputs a similarity heatmap corresponding to the auxiliary view search region. The similarity heatmap is then multiplied by the spatially aligned auxiliary view search region features to obtain the enhanced auxiliary view search region features.

[0071] According to embodiments of this application, the target object tracking method for multi-view collaboration further includes: using template features as convolution kernels to perform depthwise separable convolution operations on spatially aligned auxiliary view search region features, and outputting a similarity heatmap corresponding to the auxiliary view search region; using the similarity heatmap as spatial attention weights, performing channel-dimension weighted multiplication of the similarity heatmap and the spatially aligned auxiliary view search region features at the same spatial location to obtain enhanced auxiliary view search region features; and fusing the current view search region features and the enhanced auxiliary view search region features to obtain fused search features.

[0072] In this embodiment, the depthwise separable convolution operation uses the template features as the convolution kernel and performs a sliding window operation on the spatially aligned auxiliary viewpoint search region features to calculate the similarity between the template features and the local regions of the spatially aligned auxiliary viewpoint search region features. Specifically, the depthwise separable convolution operation is performed independently in each channel; that is, each channel of the template features is cross-correlated with the corresponding channel of the spatially aligned auxiliary viewpoint search region features, maintaining the independence of each channel's information. After the depthwise separable convolution operation, a similarity heatmap corresponding to the auxiliary viewpoint search region is output. The spatial size of the similarity heatmap is consistent with the spatial size of the search region features. The value at each position on the heatmap represents the similarity response value between the local region at that position and the template features; the higher the response value, the more likely that the position contains the target object.

[0073] Using a similarity heatmap as spatial attention weights, the similarity heatmap and the spatially aligned auxiliary view search region features are multiplied by channel dimension at the same spatial location to obtain enhanced auxiliary view search region features. Specifically, the value at a certain location in the similarity heatmap is used as a global scaling factor for that spatial location, and all channels at the same location in the spatially aligned auxiliary view search region features are uniformly weighted. After weighting, regions with high similarity to template features are enhanced, while background or irrelevant regions with low similarity to template features are suppressed, thus obtaining enhanced auxiliary view search region features.

[0074] The search features of the current viewpoint search region and the enhanced auxiliary viewpoint search region features are fused to obtain the fused search features. The fusion method can include splicing, gating fusion, and attention fusion, etc., and is not limited here. It can be understood that the fused search features contain the original spatial information of the current viewpoint and the complementary information of the enhanced auxiliary viewpoint, which can be used for matching and locating with template features to track the target object.

[0075] According to embodiments of this application, template features are used as convolution kernels to perform depthwise separable convolution operations on the spatially aligned auxiliary view search region features. Each channel performs independent cross-correlation calculations, which can accurately measure the similarity between each local region of the spatially aligned auxiliary view search region and the template features while maintaining the semantic independence of each channel. The output similarity heatmap reflects the confidence that the target object is contained at each spatial location. The similarity heatmap is used as a spatial attention weight and multiplied with the spatially aligned auxiliary view search region features at the same spatial location in a channel-dimensional weighted manner. This enhances the feature values ​​of high-response regions and suppresses the feature values ​​of low-response regions, effectively suppressing background and noise interference while highlighting the relevant regions of the target object. The current view search region features and the enhanced auxiliary view search region features are fused, integrating the original spatial information of the current view and the complementary information of the auxiliary view after importance filtering. This allows the fused search features to combine the expressive advantages of both perspectives, providing richer information and less interference in the feature input, thus improving the accuracy of target localization.

[0076] Figure 6 A schematic diagram of a hybrid expert model according to an embodiment of this application is shown.

[0077] like Figure 6 As shown, the hybrid expert model includes a classifier, a multi-branch encoder, and a gating network. The classifier includes PooL pooling layers and fully connected (FC) layers. The multi-branch encoder consists of multiple encoders, each including a Conv layer stacked N times and a ReLU activation function. The gating network includes a Conv layer and a Sigmoid activation function.

[0078] According to an embodiment of this application, the current viewpoint search region features and the enhanced auxiliary viewpoint search region features are fused to obtain fused search features. This includes: inputting the current viewpoint search region features and the enhanced auxiliary viewpoint search region features into a pre-trained hybrid expert model, and outputting the fused search features after performing the following operations. The hybrid expert model includes a classifier, a multi-branch encoder, and a gating network: using the classifier, classifying the current viewpoint search region features and the enhanced auxiliary viewpoint search region features to obtain probability values ​​that match multiple tracking scene attribute features of the target region. The tracking scene attribute features include the target object's motion speed features, the target region's background complexity features, and the target object's position. Occlusion probability features and target area illumination variation features are used. Based on the probability values ​​of matching multiple tracking scene attribute features of the target area, a preset number of encoders are selected from the multi-branch encoders. Using the preset number of encoders, the current view search area features and the enhanced auxiliary view search area features are encoded respectively, and the encoded current view search area features and the encoded enhanced auxiliary view search area features are fused respectively to obtain a preset number of fused features. Using a gating network, the preset number of fused features are weighted and fused to obtain the fused search features. The weight values ​​of the preset number of fused features are determined based on the matching degree between the fused features and the tracking scene attribute features.

[0079] In this embodiment, the current viewpoint search region features and the enhanced auxiliary viewpoint search region features are input into a pre-trained hybrid expert model. The hybrid expert model then performs the following operations and outputs the fused search features. The hybrid expert model consists of three components: a classifier, a multi-branch encoder, and a gating network.

[0080] A classifier is used to classify the features of the current viewpoint search region and the enhanced auxiliary viewpoint search region, obtaining probability values ​​that match each of the multiple tracking scene attribute features of the target region. These tracking scene attribute features include, but are not limited to, the target object's motion speed, the target region's background complexity, the target object's occlusion probability, and the target region's illumination variation. The classifier can employ a lightweight convolutional network or a fully connected network. The input to the classifier is the current viewpoint search region features and the enhanced auxiliary viewpoint search region features, and the output is multiple probability values, each representing the confidence level that the current tracking scene belongs to a certain attribute category.

[0081] For example, a higher probability value for the motion speed feature indicates that the target object is moving faster; a higher probability value for the background complexity feature indicates that the background of the current tracking scene is more cluttered; a higher probability value for the occlusion probability feature indicates that the target object is more likely to be occluded; and a higher probability value for the illumination change feature indicates that the illumination change of the current tracking scene is more drastic.

[0082] Based on the above probability values, a preset number of encoders are selected from the multi-branch encoders. A multi-branch encoder contains multiple encoder branches with identical structures but independent parameters, each branch possessing differentiated processing capabilities for different types of tracking scenarios. Based on the probability values ​​output by the classifier, the multi-branch encoders are selected, choosing several encoders that best match the tracking scenario attribute features of the target region; the selection number is a preset value.

[0083] For example, if the occlusion probability value is high, the encoder that is good at handling occluded scenes is selected; if the motion speed probability value is high, the encoder that is good at handling fast motion scenes is selected.

[0084] Each selected encoder independently encodes and transforms the features of the current viewpoint search region and the enhanced auxiliary viewpoint search region, mapping the features of the current viewpoint and the auxiliary viewpoint to a high-dimensional representation space. The encoded features are then merged in either the channel dimension or the spatial dimension to output a fused feature. Because the parameters of different encoders are independent, the output fused features are adapted to different tracking scene attribute features in their feature representation, thus obtaining a preset number of fused features.

[0085] A gating network is used to weight and fuse a preset number of features to obtain the fused search features. The gating network receives the fused features output from the multi-branch encoder as input, calculates a weight score for each fused feature, and the weight score reflects the degree of fit between the fused feature and the current tracking scene; the fused feature that fits the current scene better receives a higher weight. The gating network can adopt a lightweight network structure, multiplying each fused feature by its corresponding weight and then summing the results to obtain the fused search features.

[0086] According to embodiments of this application, a classifier classifies the features of the current viewpoint search region and the enhanced auxiliary viewpoint search region to classify the tracking scene attributes, obtaining probability values ​​for multiple dimensions such as target motion speed, background complexity, occlusion probability, and illumination changes. This enables multi-dimensional quantitative description of the current tracking scene, providing a basis for selecting encoder branches to determine the tracking scene. Based on the probability values ​​of each tracking scene attribute, a preset number of encoders are selected from the multi-branch encoders, activating the encoding branch that best matches the current tracking scene for computation, while inactive branches do not participate in forward propagation, reducing computational overhead. Each encoder has independent parameters, encoding and fusing features from two different viewpoints separately. The resulting multiple fused features express the current tracking scene from different dimensions, enriching the diversity of feature representation. The gating network determines weight values ​​based on the matching degree between each fused feature and the tracking scene attribute features and performs weighted summation, giving greater weight to fused features with higher adaptability to the current tracking scene. This achieves adaptive dynamic weighted fusion of the tracking scene, improving the accuracy and tracking stability of the fused search features in complex dynamic scenes such as rapid motion, occlusion, and illumination changes.

[0087] According to an embodiment of this application, the sub-search features that match the template features in the search features obtained by fusing the current viewpoint search region features and the spatially aligned auxiliary viewpoint search region features are mapped to the current viewpoint image to obtain the tracking position of the target object. This includes: performing spatial similarity measurement on each sub-search feature in the template features and search features to generate a similarity response map; determining the sub-search feature with the largest spatial similarity measurement with the template features in the similarity response map as the sub-search feature that matches the template features; and mapping the spatial position of the sub-search feature that matches the template features in the search features to the spatial position in the current viewpoint image to obtain the tracking position.

[0088] In this embodiment, spatial similarity is measured between the template features and each sub-search feature in the fused search features to generate a similarity response map. The fused search features have spatial dimensions and can be divided into multiple sub-regions, each corresponding to a sub-search feature with the same spatial dimensions as the template features. Similarity calculation is performed between the template features and each sub-search feature using cross-correlation. The template features are used as convolution kernels and slide across the search features, calculating the dot product similarity or normalized cross-correlation value between the template features and the sub-search features within the window coverage area position by position. After traversing all spatial positions of the search features, the similarity values ​​corresponding to all positions constitute a two-dimensional response map, i.e., the similarity response map. The response value at each position in the similarity response map reflects the degree of matching between the sub-search feature and the template feature at that position.

[0089] The sub-search feature with the highest similarity measure to the template feature space in the similarity response map is identified as the sub-search feature that matches the template feature. The response value at each position in the similarity response map represents the confidence that a target object exists at that position. The position with the largest response value indicates that the local region at that position has the highest similarity to the template feature, and is determined to be the position where the target object is most likely to appear in that frame. The sub-search feature corresponding to this position is the sub-search feature that matches the template feature.

[0090] The spatial position of the sub-search feature matching the template feature is mapped to its spatial position in the current view image to obtain the tracking position. Since the current view search region feature is extracted from the current view image by sampling at a certain step size and scale, the spatial coordinates of the sub-search feature are mapped back to the pixel coordinate system of the current view image to obtain the bounding box position of the target object in the current view image. The output format is horizontal coordinate, vertical coordinate, width, and height, thus completing the determination of the tracking position.

[0091] According to embodiments of this application, by measuring the spatial similarity between template features and each sub-search feature in the fused search features, a similarity response map is generated, enabling parallel similarity calculation for all spatial locations with high matching efficiency. The similarity response map preserves the spatial location correspondence, and the response value represents the matching confidence of each sub-search feature with the template feature. Target localization can be achieved without classification or regression networks, simplifying the inference process, without introducing additional parameters and computational overhead, and ensuring the interpretability and stability of target determination. The spatial location of the sub-search feature matching the template feature in the search features is mapped back to the spatial location in the current view image. Utilizing the fixed geometric correspondence between the feature map and the input image, the bounding box coordinates of the target object in the original image can be obtained through reverse mapping. This mapping method has low computational cost and controllable accuracy, and the output target tracking position in the current view image is highly efficient, achieving accuracy in the tracking task.

[0092] Figure 7 A schematic diagram of a target object tracking framework according to an embodiment of this application is shown.

[0093] like Figure 7As shown, the template images of the current viewpoint and the auxiliary viewpoint, along with the search region image, are input into the feature extraction network for feature extraction. The extracted features are then processed through information exchange. This information exchange includes a cross-viewpoint alignment module based on contrastive learning, an importance inference and enhancement module, and a multi-branch fusion module. The cross-viewpoint alignment module uses a projection network, and the multi-branch fusion module uses a hybrid expert model. It can be understood that the cross-viewpoint alignment module obtains the spatially aligned auxiliary viewpoint search region features, the importance inference and enhancement module obtains the enhanced auxiliary viewpoint search region features, and the multi-branch fusion module obtains the fused search features. The fused search features are then input into the relevant module to generate a similarity response map. After passing through the prediction head, the tracking position is output.

[0094] Based on the aforementioned multi-view collaborative target object tracking method, this application also provides a multi-view collaborative target object tracking device. The following will combine... Figure 8 The device is described in detail.

[0095] Figure 8 The schematic diagram illustrates a structural block diagram of a target object tracking device for multi-view collaboration according to an embodiment of this application.

[0096] like Figure 8 As shown, the target object tracking device 800 for multi-view collaboration in this embodiment includes an acquisition module 810, a projection module 820, an alignment module 830, and a mapping module 840.

[0097] The acquisition module 810 is used to acquire images of the target area from the current viewpoint and the auxiliary viewpoint respectively in response to the detection of a target object within the target area. The current viewpoint and the auxiliary viewpoint are different viewpoints. In one embodiment, the acquisition module 810 can be used to perform the operation S110 described above, which will not be repeated here.

[0098] The projection module 820 is used to project the current view search region features of the current view image and the auxiliary view search region features of the auxiliary view image onto the same preset feature metric space, respectively, to obtain the current view projection features and the auxiliary view projection features. In the preset feature metric space, the similarity between cross-view projection features of the same geographical location is greater than the similarity between cross-view projection features of different geographical locations. In one embodiment, the projection module 820 can be used to perform the operation S120 described above, which will not be repeated here.

[0099] Alignment module 830 is used to determine the spatial offset of the current viewpoint relative to the auxiliary viewpoint based on the projection features of the current viewpoint and the projection features of the auxiliary viewpoint, and to resample the projection features of the auxiliary viewpoint based on the spatial offset, so as to transform the projection features of the auxiliary viewpoint to a state aligned with the features of the search region of the current viewpoint, thereby obtaining the spatially aligned features of the search region of the auxiliary viewpoint. In one embodiment, alignment module 830 can be used to perform the operation S130 described above, which will not be repeated here.

[0100] The mapping module 840 is used to map the sub-search features that match the template features from the search features obtained by fusing the current viewpoint search region features and the spatially aligned auxiliary viewpoint search region features to the current viewpoint image, thereby obtaining the tracking position of the target object. The template features represent the attribute features of the target object in the current viewpoint. The mapping module 840 can be used to perform the operation S140 described above, which will not be repeated here.

[0101] According to an embodiment of this application, the projection module 820 includes a first input submodule and a training submodule.

[0102] The first input submodule is used to input the current view search region features and the auxiliary view search region features into the pre-trained projection network, and output the current view projection features and the auxiliary view projection features respectively.

[0103] The training submodule is used to pre-train the projection network.

[0104] According to an embodiment of this application, the training submodule includes an acquisition unit, an extraction unit, a construction unit, a projection unit, and a training unit.

[0105] The acquisition unit is used to acquire current view sample images and auxiliary view sample images of the sample regions where multiple sample objects are located from the current view and auxiliary view respectively, so as to obtain multiple current view sample images and multiple auxiliary view sample images.

[0106] The extraction unit is used to extract features from each current viewpoint sample image and each auxiliary viewpoint sample image to obtain the corresponding current viewpoint sample search region features and auxiliary viewpoint sample search region features.

[0107] The building unit is used to construct positive sample feature pairs and negative sample feature pairs based on the search region features of multiple current viewpoint samples and multiple auxiliary viewpoint samples. Positive sample feature pairs indicate the same sample object, while negative sample feature pairs indicate different sample objects.

[0108] The projection unit is used to input positive sample feature pairs and negative sample feature pairs into the initial projection network so that both positive and negative sample feature pairs are projected onto the same preset feature metric space, and output the corresponding current viewpoint sample projection features and auxiliary viewpoint sample projection features.

[0109] The training unit is used to train the initial projection network with the training objective of maximizing the similarity between the projection features of positive sample features and the projection features of the corresponding current view sample and the projection features of the auxiliary view sample, and minimizing the similarity between the projection features of negative sample features and the projection features of the corresponding current view sample and the projection features of the auxiliary view sample, so as to obtain the trained projection network.

[0110] According to an embodiment of this application, the alignment module 830 includes a splicing submodule, a downsampling submodule, a processing submodule, and a mapping submodule.

[0111] The stitching submodule is used to stitch together the projection features of the current viewpoint and the projection features of the auxiliary viewpoint to obtain the stitched features.

[0112] The downsampling submodule is used to downsample the splicing features to obtain the initial splicing features.

[0113] The processing submodule is used to perform channel-level feature processing on the primary stitching features to obtain the horizontal offset of the current viewpoint relative to the auxiliary viewpoint and the vertical offset of the current viewpoint relative to the auxiliary viewpoint.

[0114] The mapping submodule is used to map the ranges of the horizontal and vertical offsets to obtain the spatial offset.

[0115] According to embodiments of this application, the alignment module 830 further includes a construction submodule, a correction submodule, and a resampling submodule.

[0116] A submodule is constructed to build a reference coordinate grid for the auxiliary view projection features based on the auxiliary view projection features.

[0117] The correction submodule is used to correct the reference coordinate grid based on the spatial offset to obtain the sampling position grid.

[0118] The resampling submodule is used to resample the auxiliary view projection features according to the sampling position grid to obtain the spatially aligned auxiliary view search region features.

[0119] According to embodiments of this application, the multi-view collaborative target object tracking device 800 further includes a convolution module, a computation module, and a fusion module.

[0120] The convolution module is used to perform depthwise separable convolution operations on the spatially aligned auxiliary view search region features using template features as convolution kernels, and outputs a similarity heatmap corresponding to the auxiliary view search region.

[0121] The calculation module is used to use the similarity heatmap as spatial attention weights, and to perform channel-dimensional weighted multiplication of the similarity heatmap and the spatially aligned auxiliary view search region features at the same spatial location to obtain the enhanced auxiliary view search region features.

[0122] The fusion module is used to fuse the features of the current view search area and the enhanced auxiliary view search area to obtain the fused search features.

[0123] According to an embodiment of this application, the fusion module includes a second input submodule.

[0124] The second input submodule is used to input the current viewpoint search region features and the enhanced auxiliary viewpoint search region features into a pre-trained hybrid expert model so as to output the fused search features after performing the following operations. The hybrid expert model includes a classifier, a multi-branch encoder and a gating network.

[0125] According to an embodiment of this application, the second input submodule includes a classification unit, a filtering unit, an encoding unit, and a fusion unit.

[0126] The classification unit is used to classify the features of the current view search area and the enhanced auxiliary view search area using a classifier, and obtain the probability values ​​that match each of the multiple tracking scene attribute features of the target area. The tracking scene attribute features include the target object's motion speed feature, the target area's background complexity feature, the target object's occlusion probability feature, and the target area's illumination change feature.

[0127] The filtering unit is used to filter a preset number of encoders from the multi-branch encoders based on the probability values ​​that match the various tracking scene attribute features of the target region.

[0128] The encoding unit is used to encode the current view search region features and the enhanced auxiliary view search region features using a preset number of encoders, and then fuse the encoded current view search region features and the encoded enhanced auxiliary view search region features to obtain a preset number of fused features.

[0129] The fusion unit is used to use a gating network to perform weighted fusion of a preset number of fusion features to obtain fused search features. The weight values ​​of the preset number of fusion features are determined based on the matching degree between the fusion features and the tracking scene attribute features.

[0130] According to an embodiment of this application, the mapping module 840 includes a generation submodule, a determination submodule, and a tracking submodule.

[0131] The generation submodule is used to measure the spatial similarity of each sub-search feature in the template features and search features, and generate a similarity response map.

[0132] The determination submodule is used to identify the sub-search features with the highest similarity measure to the template feature space in the similarity response map as the sub-search features that match the template features.

[0133] The tracking submodule is used to map the spatial position of the sub-search feature that matches the template feature to the spatial position of the current view image to obtain the tracking position.

[0134] According to embodiments of this application, any plurality of modules among the acquisition module 810, projection module 820, alignment module 830, and mapping module 840 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this application, at least one of the acquisition module 810, projection module 820, alignment module 830, and mapping module 840 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any appropriate combination of any of these three implementation methods. Alternatively, at least one of the acquisition module 810, projection module 820, alignment module 830, and mapping module 840 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0135] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a multi-view collaborative target object tracking method according to an embodiment of this application.

[0136] like Figure 9As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this application.

[0137] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 executes various operations of the method flow according to embodiments of this application by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also execute various operations of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0138] According to embodiments of this application, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output (I / O) interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output (I / O) interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0139] This application also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0140] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0141] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0142] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this application embodiment. According to the embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0143] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0144] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this application embodiment. According to the embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0145] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0147] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

[0148] The embodiments of this application have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of this application. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Without departing from the scope of this application, those skilled in the art can make various substitutions and modifications, all of which should fall within the scope of this application.

Claims

1. A target object tracking method for multi-view collaborative operation, characterized in that, The method includes: In response to the detection of a target object within a target area, images of the target area from the current viewpoint and the auxiliary viewpoint are acquired from the current viewpoint and the auxiliary viewpoint, respectively, wherein the current viewpoint and the auxiliary viewpoint are different viewpoints; The current view search region features of the current view image and the auxiliary view search region features of the auxiliary view image are projected onto the same preset feature metric space to obtain the current view projection features and the auxiliary view projection features. In the preset feature metric space, the similarity between cross-view projection features of the same geographical location is greater than the similarity between cross-view projection features of different geographical locations. Based on the current view projection features and the auxiliary view projection features, the spatial offset of the current view relative to the auxiliary view is determined, and based on the spatial offset, the auxiliary view projection features are resampled to transform the auxiliary view projection features into a state aligned with the current view search region features, thereby obtaining the spatially aligned auxiliary view search region features. The sub-search features that match the template features in the search features obtained by fusing the current view search region features and the spatially aligned auxiliary view search region features are mapped to the current view image to obtain the tracking position of the target object. The template features represent the attribute features of the target object under the current view.

2. The method according to claim 1, characterized in that, The step of projecting the current view search region features of the current view image and the auxiliary view search region features of the auxiliary view image onto the same preset feature metric space to obtain the corresponding current view projection features and auxiliary view projection features includes: The current view search region features and the auxiliary view search region features are respectively input into a pre-trained projection network, and the current view projection features and the auxiliary view projection features are output accordingly. The projection network is pre-trained through the following steps: For the sample regions where multiple sample objects are located, current view sample images and auxiliary view sample images of the sample regions are acquired from the current view and the auxiliary view respectively, to obtain multiple current view sample images and multiple auxiliary view sample images; Feature extraction is performed on each current viewpoint sample image and each auxiliary viewpoint sample image to obtain the corresponding current viewpoint sample search region features and auxiliary viewpoint sample search region features; Based on multiple current viewpoint sample search region features and multiple auxiliary viewpoint sample search region features, positive sample feature pairs and negative sample feature pairs are constructed. The positive sample feature pairs indicate the same sample object, and the negative sample feature pairs indicate different sample objects. The positive sample feature pairs and the negative sample feature pairs are input into the initial projection network so that both the positive sample feature pairs and the negative sample feature pairs are projected to the same preset feature metric space, and the corresponding current viewpoint sample projection features and auxiliary viewpoint sample projection features are output. The initial projection network is trained with the training objective of maximizing the similarity between the positive sample features and the corresponding projection features of the current viewpoint sample and the auxiliary viewpoint sample, and minimizing the similarity between the negative sample features and the corresponding projection features of the current viewpoint sample and the auxiliary viewpoint sample, to obtain the trained projection network.

3. The method according to claim 1, characterized in that, Determining the spatial offset of the current viewpoint relative to the auxiliary viewpoint based on the current viewpoint projection features and the auxiliary viewpoint projection features includes: The current view projection feature and the auxiliary view projection feature are stitched together to obtain the stitched feature; The splicing features are downsampled to obtain primary splicing features; Channel-level feature processing is performed on the primary stitching features to obtain the horizontal offset of the current viewpoint relative to the auxiliary viewpoint and the vertical offset of the current viewpoint relative to the auxiliary viewpoint. The spatial offset is obtained by mapping the value range of the horizontal offset and the vertical offset.

4. The method according to claim 1, characterized in that, The step of resampling the auxiliary view projection features based on the spatial offset to transform the auxiliary view projection features into a state aligned with the current view search region features, thereby obtaining spatially aligned auxiliary view search region features, includes: Based on the auxiliary view projection features, a reference coordinate grid for the auxiliary view projection features is constructed; Based on the spatial offset, the reference coordinate grid is corrected to obtain the sampling position grid; The auxiliary view projection features are resampled according to the sampling position grid to obtain the spatially aligned auxiliary view search region features.

5. The method according to claim 1, characterized in that, The method further includes: Using the template features as convolution kernels, depth-separable convolution operations are performed on the spatially aligned auxiliary view search region features to output a similarity heatmap corresponding to the auxiliary view search region. Using the similarity heatmap as spatial attention weight, the similarity heatmap and the spatially aligned auxiliary view search region features are multiplied by channel dimension at the same spatial location to obtain the enhanced auxiliary view search region features. The current viewpoint search region features and the enhanced auxiliary viewpoint search region features are fused to obtain the fused search features.

6. The method according to claim 5, characterized in that, The step of fusing the current viewpoint search region features and the enhanced auxiliary viewpoint search region features to obtain the fused search features includes: The current viewpoint search region features and the enhanced auxiliary viewpoint search region features are input into a pre-trained hybrid expert model to output the fused search features after performing the following operations: The hybrid expert model includes a classifier, a multi-branch encoder, and a gating network. Using the classifier, the features of the current view search region and the enhanced auxiliary view search region are classified to obtain the probability values ​​that match each of the multiple tracking scene attribute features of the target region. The tracking scene attribute features include the target object's motion speed feature, the target region's background complexity feature, the target object's occlusion probability feature, and the target region's illumination change feature. Based on the probability values ​​that match the various tracking scene attribute features of the target region, a preset number of encoders are selected from the multi-branch encoders. Using the preset number of encoders, the current view search region features and the enhanced auxiliary view search region features are encoded respectively, and the encoded current view search region features and the encoded enhanced auxiliary view search region features are fused respectively to obtain the preset number of fused features; Using the gating network, the preset number of fusion features are weighted and fused to obtain the fused search features, wherein the weight values ​​of the preset number of fusion features are determined based on the matching degree between the fusion features and the tracking scene attribute features.

7. The method according to claim 1, characterized in that, The process of fusing the current viewpoint search region features and the spatially aligned auxiliary viewpoint search region features to obtain the sub-search features that match the template features in the search features obtained by fusing them with the template features, and mapping them to the current viewpoint image to obtain the tracking position of the target object, includes: Spatial similarity measurement is performed on the template features and each sub-search feature in the search features to generate a similarity response map; The sub-search feature with the highest similarity measure to the template feature space in the similarity response map is determined as the sub-search feature that matches the template feature; The spatial position of the sub-search feature that matches the template feature in the search feature is mapped to the spatial position of the current view image to obtain the tracking position.

8. A target object tracking device for multi-view collaborative operation, characterized in that, The device includes: The acquisition module is used to acquire images of the target area from the current viewpoint and the auxiliary viewpoint respectively in response to the detection of a target object in the target area. The current viewpoint and the auxiliary viewpoint are different viewpoints. The projection module is used to project the current view search region features of the current view image and the auxiliary view search region features of the auxiliary view image onto the same preset feature measurement space, respectively, to obtain the current view projection features and the auxiliary view projection features. In the preset feature measurement space, the similarity between cross-view projection features of the same geographical location is greater than the similarity between cross-view projection features of different geographical locations. An alignment module is used to determine the spatial offset of the current viewpoint relative to the auxiliary viewpoint based on the current viewpoint projection features and the auxiliary viewpoint projection features, and to resample the auxiliary viewpoint projection features based on the spatial offset to transform the auxiliary viewpoint projection features into a state aligned with the current viewpoint search region features, thereby obtaining spatially aligned auxiliary viewpoint search region features; and The mapping module is used to map the sub-search features that match the template features in the search features obtained by fusing the current view search region features and the spatially aligned auxiliary view search region features to the current view image to obtain the tracking position of the target object. The template features represent the attribute features of the target object under the current view.

9. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.