Multi-view object detection and tracking method based on view-angle-space joint attention

Abstract

The application discloses a multi-view target detection and tracking method based on view-angle-space joint attention, and the steps are as follows: acquiring multi-view images aligned at the same time, extracting two-dimensional features and projecting the two-dimensional features to a bird's eye view plane; calculating effective observation regions of each view angle; weighting and fusing multi-view features by using a view-angle-space joint attention module and an uncertainty perception fusion module, and gain modulating in combination with a region prior heat map; inputting the fused features into a decoding network to obtain detection results, and then inputting the detection results into a tracker to complete target association and track state updating; in a training stage, a teacher network and a student network are adopted, and the network is trained by using a supervision constraint and a cross-view self-supervised consistency constraint. The application can improve the stability of multi-view target tracking in a complex scene, and has the advantages of strong robustness and convenience for engineering deployment.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and intelligent video analysis technology, and further relates to a multi-view target detection and tracking method based on viewpoint-space joint attention, which can be used in scenarios such as multi-camera collaborative monitoring, smart security and smart transportation. Background Technology

[0002] Multi-view target tracking technology, by integrating visual information from multiple cameras, can effectively solve problems such as single-view occlusion and limited field of view, and has irreplaceable application value in fields such as security monitoring and intelligent transportation. In recent years, related patent technologies in this field have been rapidly iterating. US Patent US10176405B2 proposes a cross-view vehicle feature inference method based on neural networks.

[0003] Chinese patents CN121414783A and CN117611638B explored cross-view trajectory matching and multi-sensor fusion tracking methods, respectively. While these technologies have promoted the development of multi-view tracking, significant technical bottlenecks remain in the two core areas of fine-grained modeling through feature fusion and robust optimization through multi-camera collaboration, making it difficult to meet the high-precision tracking requirements in complex scenarios.

[0004] At the feature fusion level, the shortcomings of existing mainstream methods are mainly reflected in their coarse weighting strategies, lack of uncertainty modeling, and insufficient utilization of prior information. Early fusion methods, represented by MVDet, directly sum and convolve the features projected from each viewpoint onto the bird's-eye view (BEV) representation, without considering the differences in imaging quality between different cameras in local areas. When there is occlusion or noise interference in a certain viewpoint, low-quality features will be indiscriminately integrated into the global features, leading to false positives and false negatives. SHOT and its improved algorithm Booster-SHOT achieve high-dimensional feature filtering through a soft selection module, but are still limited to single-dimensional weighting and cannot dynamically adjust the viewpoint weights for each BEV grid position. Some patented technologies use splicing or average pooling fusion strategies, which ignore the spatial geometric relationship and semantic redundancy between viewpoints and fail to balance the consistency and complementarity of multi-view features, resulting in a large loss of discriminative information. More importantly, existing technologies generally do not model the uncertainty of viewpoint observation, treating all viewpoint features as equally reliable, making them susceptible to being misled by blurred or low-resolution viewpoints in complex environments. Meanwhile, most solutions do not incorporate prior scene information, wasting a lot of computing resources in non-key areas, while the feature representation capabilities of core monitoring areas such as cash registers and traffic intersections are insufficient, further reducing tracking accuracy.

[0005] At the level of multi-camera collaborative mechanisms, the shortcomings of existing technologies are mainly manifested in imperfect self-supervised training strategies, weak adaptability to missing viewpoints, and insufficient tracking and matching accuracy.

[0006] The cross-view trajectory matching scheme proposed in Chinese patent CN121414783A, on the one hand, explicitly decomposes cross-camera association into a fusion criterion of spatial consistency and appearance consistency, which facilitates modular deployment and threshold parameter tuning in engineering implementation; on the other hand, by mapping the trajectory to a spatial coordinate system and performing alignment analysis, cross-camera association does not completely depend on appearance features, which can alleviate the appearance drift problem caused by differences in lighting and viewpoints between cross-cameras to a certain extent, and adapt to the cross-camera tracking needs of multi-category targets such as pedestrians and vehicles. The method has shortcomings: firstly, cross-camera association is based on local trajectory quality. In occluded and dense scenes, single-camera trajectories are easily fragmented, and the number of candidate trajectory pairs expands with increasing fragmentation. Threshold judgment is more sensitive to errors. Once mismatch or omission occurs, subsequent segmented fusion will solidify the error into a global trajectory error, manifesting as cross-camera relay failure and ID (identity) errors. First, the switching is frequent. Second, the spatial position consistency coefficient is obtained by statistically analyzing the Euclidean distance between the aligned position points. This essentially assumes that the spatial mapping and observation noise are relatively stable. However, actual multi-camera systems often have calibration errors, non-strictly flat ground, holes in the edge field of view projection, and occlusion causing jitter in the detection center. These factors can cause the same target to have a systematic shift or increased variance after cross-camera mapping, thereby reducing the reliability of position consistency and causing mismatches. Third, the appearance similarity fluctuates greatly under cross-camera resolution, exposure, and attitude changes. This scheme lacks a fine-grained mechanism for allocating viewpoint confidence based on spatial position. The appearance features of unreliable viewpoints in occluded areas may dominate the matching confidence, inducing mismatches. Fourth, although the comprehensive matching confidence integrates position and appearance, the claims do not reflect a mechanism for explicitly injecting observation uncertainty into the gating and filtering covariance. When occlusion occurs frequently, two types of problems can easily occur: overly strict gating leading to mismatches and overly loose gating leading to mismatches. Ultimately, this manifests as trajectory breakage and global ID instability. Meanwhile, cross-camera appearance representation is significantly affected by domain differences such as viewing angle, lighting, resolution and occlusion. Appearance similarity is easily distorted when crossing scenes, resulting in unstable cross-camera association thresholds. The Re-Identification field also regards cross-camera domain differences as an important factor restricting generalization. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of the existing technologies by proposing a multi-view target detection and tracking method based on view-space joint attention. This method aims to solve the problems of inaccurate fusion, easy introduction of noise, and insufficient generalization ability of existing methods under conditions of occlusion, view difference, limited data, and missing view.

[0008] To achieve the above objectives, this invention employs the following technical solution: First, feature extraction is performed on multi-view images taken at the same time, and the two-dimensional features of each view are projected onto the bird's-eye view representation plane based on camera calibration information; second, an effective observation area mask (FoV mask) is constructed to constrain the effective visible area of ​​each view at each BEV grid position, shielding projection holes and invalid areas, thereby reducing noise features entering the fusion process from the source; further, a view-spatial attention gating (VAG) module is used to generate per-view and per-grid attention weights, enabling position-related view selection, automatically reducing the weight of occluded or poorly imaged views in the corresponding areas, thus mitigating the root cause of mismatch caused by unstable appearance and position in trajectory-level fusion, and combining this with uncertainty-aware fusion (UAF). The Fusion module modifies the attention logits, reducing the contribution of high-uncertainty observations to the fusion process and yielding multi-view fusion features. Then, a region prior heatmap is introduced to modulate the fusion features with position-related gain, strengthening target responses in high-priority regions and suppressing the interference of false detections in low-priority regions. The modulated features are then input into the decoding network, outputting target detection results. During the inference phase, the detection results are input into the tracker, outputting the final multi-target tracking results. In the training phase, a teacher-student structure is adopted, using supervised training constraints and cross-view self-supervised consistency (CVC) constraints to train the learnable networks related to feature extraction, fusion, and decoding. During the inference phase, the trained student network parameters are loaded, outputting the final multi-view target detection and multi-target tracking results. This invention, through its overall design of effective region constraints, position-related view weighting, uncertainty perception suppression—regional prior modulation, consistency and lack-view robust training, and tracking association, improves the stability and generalization ability of cross-camera tracking.

[0009] To achieve the above objectives, the specific implementation steps of the present invention include the following:

[0010] Step 1: Extract the two-dimensional features of the current image from each camera and project the two-dimensional features onto the BEV plane;

[0011] Step 2: Calculate the effective observation area for each viewpoint at each BEV grid location;

[0012] Step 3: Calculate the weights for each view using the constructed view-space joint attention module (VAG);

[0013] Step 4: Use the constructed uncertainty-aware fusion module UAF to correct the viewpoint weights and obtain the BEV fusion features;

[0014] Step 5: Utilize the region prior heatmap to perform gain modulation on the BEV fusion features;

[0015] Step 6: Input the modulated fused features into the decoding network and output the center heatmap, offset, scale, orientation prediction, and identity feature of the target in the current frame after decoding.

[0016] Step 7: Copy the learnable network structure related to the extracted two-dimensional features, feature fusion, and decoded detection results into a teacher network with the same structure. Use the original network as the student network. Input the same training data into the student network and the teacher network respectively. Train the student network and the teacher network through the supervised training constraints and the cross-view self-supervised consistency constraints. After training, retain the trained student network parameters.

[0017] Step 8: In the inference phase, the trained student network parameters are loaded into the networks corresponding to Steps 1 to 6, Steps 1 to 6 are re-executed, and the decoded detection results output in Step 6 are input into the tracker to output the final multi-view target detection results and multi-target tracking results.

[0018] Furthermore, the step of extracting the two-dimensional features of the current image of each camera is as follows:

[0019] The first step is to align each video stream corresponding to each camera viewpoint at a unified time to obtain the current frame image corresponding to each viewpoint, and then combine the current frame images at all unified alignment times into a current frame image set.

[0020] The second step involves inputting the current frame image of each channel into the feature extraction sub-network of a convolutional neural network with shared parameters, and outputting two-dimensional features of the current camera image with a preset channel dimension; the feature extraction sub-network includes a backbone convolutional coding layer and a feature transformation layer.

[0021] Furthermore, the projection of the two-dimensional features onto the BEV plane is obtained by the following formula:

[0022] ;

[0023] in, Indicates the first Group Sample No. The camera in the Two-dimensional features of the image projected onto the BEV plane at each time step. This indicates a projection operation. Indicates the first Group Sample No. The camera in the Two-dimensional features of an image at a given time point. express Indicates the first Group Sample No. The camera in the The projection matrix from image feature coordinates to BEV grid coordinates at each moment is determined by the camera's intrinsic and extrinsic parameters and the reference bird's-eye view plane.

[0024] Furthermore, the effective observation area for each viewpoint at each BEV grid location is obtained by the following formula:

[0025] ;

[0026] in, Indicates the first Group Sample No. The camera in the Image BEV coordinates at each time point The mask for the effective observation region, whose values ​​range from {0,1}. This indicates an indicator function; the value is 1 if the condition is true, and 0 otherwise. Denotes the first digit of the projected all-one two-dimensional matrix. Group Sample No. The camera in the Image BEV coordinates at each time point The value of , The threshold representing the minimum value is 0.5. In the middle, when The value is greater than When the time is right, it is recorded as the effective observation area, that is... Otherwise, it is recorded as an invalid observation area, i.e. .

[0027] Furthermore, the steps for calculating the weights of each viewpoint using the constructed view-space joint attention module (VAG) are as follows:

[0028] The first step is to construct a View-Spatial Joint Attention Module (VAG). The VAG module shares a lightweight convolutional sub-network for each viewpoint, used to generate attention logits from BEV features. The convolutional sub-networks are configured in a concatenated order, including: a 1x1 convolutional layer used to separate the channel dimensions... Mapped to intermediate channel Then, a non-linear activation layer is connected in series, using the ReLU activation function; finally, a 1*1 convolutional layer is connected in series to convert the intermediate channels. Mapped to a single-channel logits output;

[0029] The second step, calculating the weight of each viewpoint, involves inputting the two-dimensional features of the effective observation area of ​​each viewpoint at each BEV grid position into the viewpoint-space joint attention module (VAG), and then combining the output with the effective observation area mask and performing softmax normalization in the viewpoint dimension to obtain the weight corresponding to each viewpoint at each BEV grid position.

[0030] Furthermore, the Uncertainty Awareness Fusion Module (UAF) and the View-Space Joint Attention Module (VAG) are configured in parallel; the UAF includes an uncertainty estimation branch, a logarithmic mapping correction unit, and a view weight calculation unit; wherein,

[0031] The uncertainty estimation branch takes the BEV features of each viewpoint as input and includes, in a concatenated order: a 3*3 convolutional layer to map the channel dimension from the input channel to the intermediate channel; followed by a non-linear activation layer using the ReLU activation function; and then a 1*1 convolutional layer to output the uncertainty logarithm map corresponding to each viewpoint and each BEV grid position.

[0032] The logarithmic mapping correction unit obtains the uncertainty quantity by exponential mapping the uncertainty logarithmic graph, and applies it to the attention logits through a penalty correction method to obtain the corrected final logits.

[0033] The viewpoint weight calculation unit performs softmax normalization on the corrected final logits in the viewpoint dimension to recalculate the weights of each viewpoint at each BEV grid position, and performs element-wise weighted summation and viewpoint dimension summation on the multi-view BEV features based on the weights to obtain uncertainty-aware BEV fusion features.

[0034] Furthermore, the step of correcting the viewpoint weights using the constructed uncertainty-aware fusion module UAF is as follows:

[0035] The first step is to weight the BEV features for each viewpoint and calculate the attention logits:

[0036] ;

[0037] in, Indicates the first Group Sample No. The camera in the The attention logits of the image at each time step; Represents the viewpoint-space joint attention module (VAG);

[0038] The second step is to use a FoV mask to suppress invalid regions in logits:

[0039] ;

[0040] in, Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point Attention logits after being suppressed by the FoV mask; This represents a positive constant used to reduce the logits of invalid regions to a very small value.

[0041] The third step involves using the uncertainty estimation branch to output an uncertainty logarithm plot for each viewpoint BEV feature, and then performing an exponential mapping on it to obtain the uncertainty quantity:

[0042] ;

[0043] in, For the first Group Sample No. The camera in the The uncertainty is obtained by exponentially mapping the uncertainty estimation branch results at each time step. This indicates an exponential operation with the natural constant e as the base. Indicates the uncertainty estimation branch;

[0044] Fourth step, utilize right Adjust the penalty accordingly, and obtain And use it as input to softmax:

[0045] ;

[0046] in, Indicates the first Group Sample No. The camera in the The final logits after UAF correction at each time step; Indicates the penalty coefficient. Represents the logarithmic function with base 10. This represents a numerically stable term, a very small positive constant whose value takes a value of 100%. ;

[0047] Fifth step: Apply softmax to the corrected logits in the view dimension to obtain the weights:

[0048] ;

[0049] in, Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point Perspective weighting on the top Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point The final logits after UAF correction; This represents the summation operation. The number indicating the viewpoint. =1,2,...., , Indicates the total number of viewpoints;

[0050] Step 6: Using weights, weight each element of the BEV features from each perspective:

[0051] ;

[0052] in, Indicates the first Group Sample No. The camera in the The weighted BEV characteristics at each time point, This indicates element-wise multiplication. Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point Corresponding perspective weights;

[0053] Step 7: Summate the weighted multi-view features along the view dimension to obtain the BEV fusion features:

[0054] ;

[0055] in, Indicates the first Group of samples in the first BEV features after multi-view fusion at each moment.

[0056] Furthermore, the step of using the region prior heatmap to perform gain modulation on the fused features is as follows:

[0057] The first step is to obtain the count distribution on the BEV grid offline and normalize it to form a priori heatmap of the region;

[0058] ;

[0059] in, Represents BEV coordinates The region of prior values, Indicates position The cumulative count; This indicates the location of all BEV grid points. Take the maximum value;

[0060] The second step involves applying the region's prior heatmap as a gain term to the weighted multi-view BEV features to achieve location-dependent gain modulation.

[0061] ;

[0062] in, Indicates the first Group of samples in the first BEV characteristics after multi-view fusion at each moment. This represents the assignment / update symbol. This represents the gain coefficient, with a value of 0.1.

[0063] Furthermore, the training data refers to training samples composed of multi-view images and their corresponding annotations synchronously acquired by multiple fixed cameras at the same time; wherein, each training sample includes at least the current frame image corresponding to each camera view at the same time, the center position annotation, position offset annotation, scale, orientation annotation, and target identity annotation of the target in the BEV plane; the current frame image corresponding to each camera view at the same timestamp is used as input data shared by the student network and the teacher network, and the center position annotation, position offset annotation, scale, orientation annotation, and target identity annotation are used to construct the supervised training constraints in step 7.

[0064] Furthermore, the learnable network parameters for training the student network and teacher network through the supervised training constraints and the cross-view self-supervised consistency constraints include: the feature extraction network parameters in step 1, the view-space joint attention module parameters in step 3, the uncertainty perception fusion module parameters in step 4, and the decoding network parameters in step 6.

[0065] Furthermore, the steps for training the student network and teacher network using the supervised training constraints and the cross-perspective self-supervised consistency constraints are as follows:

[0066] The first step is to perform the feature extraction, projection, fusion and decoding processes in steps 1 to 6 on the same batch of multi-view inputs, using the student network and the teacher network respectively, to obtain the decoding detection results output by the student network and the multi-view BEV features corresponding to the student network and the teacher network.

[0067] The second step involves defining an effective BEV grid set based on the union of the FoV masks. A supervised loss is constructed based on the decoding detection results output by the student network, and a consistency loss is constructed based on the multi-view BEV features corresponding to the student and teacher networks. The supervised loss consists of the difference between the student network's decoding results and the training sample annotations, and includes at least a center heatmap supervision term, a position offset supervision term, a scale and orientation supervision term, and an identity feature supervision term, used to constrain the consistency between the student network's output detection results and the annotation results. The consistency loss is used to constrain the multi-view BEV features output by the student and teacher networks under the same training sample input to remain consistent within the effective BEV grid set, thereby reducing the feature differences between the teacher and student networks at corresponding spatial locations.

[0068] ;

[0069] in, This indicates a loss of consistency. Indicates the number of elements in the set. Indicates the BEV coordinates Summing the effective grid positions in the middle; This represents the L2 norm operation. Indicates the student network in BEV grid coordinates BEV feature vector at location; Indicates the teacher network in BEV grid coordinates BEV feature vector at location;

[0070] The third step is to weightedly sum the supervision loss and consistency loss to update the student network parameters. Specifically, the student network parameters are: Teacher network parameters are .

[0071] Student parameters are updated from student network parameters and teacher network parameters according to EMA rules:

[0072] ;

[0073] in, Indicates the total training loss. Indicates monitoring losses, This represents the consistency loss weighting coefficient, with a value of 0.1;

[0074] Fourth, during training, the student network updates its parameters by minimizing the total training loss, while the teacher network parameters are updated using an exponential moving average of the student network parameters. This ensures that the teacher network outputs a more stable feature representation than the student network.

[0075] ;

[0076] in, is the EMA attenuation coefficient, a hyperparameter set to 0.9995.

[0077] Compared with the prior art, the present invention has the following advantages:

[0078] First, this invention introduces View-Spatial Joint Attention (VAG) in the multi-view BEV fusion stage, which can adaptively weight the feature contributions of different cameras at the BEV grid level and suppress the response of views with local occlusion, illumination changes or image quality degradation. This helps to reduce the interference of noisy views on the fusion results and improve the accuracy and stability of information fusion in the multi-view target detection and tracking process.

[0079] Secondly, this invention projects the image features from each viewpoint onto a unified BEV plane through camera calibration parameters for spatial alignment and information aggregation. This enables the representation of targets at the same ground position under different viewpoints to be processed in a unified coordinate system, thereby enhancing cross-viewpoint geometric consistency and alleviating the problem of incomplete detection caused by single-viewpoint occlusion.

[0080] Third, this invention introduces uncertainty estimation and uncertainty perception fusion UAF in the attention weight calculation process, explicitly models the observation uncertainty of each view at each BEV grid position, and uses the uncertainty to correct the attention logits, thereby helping to reduce the impact of unreliable observations on the fusion results and improve the robustness of multi-view fusion in complex scenes.

[0081] Fourth, this invention introduces a region prior heatmap to perform position-related gain modulation on the fused features, so that the high-frequency occurrence area of ​​the target or the business-related area obtains a stronger feature response, and the invalid response of the low prior area is suppressed, thereby improving the relevance of the detection results and improving the feature utilization efficiency in the overall reasoning process.

[0082] Fifth, this invention adopts a combination of teacher network and student network in the training phase and introduces cross-view self-supervised consistency learning to constrain the consistency of BEV feature representation under synchronous multi-view input, thereby improving the adaptability of student network to multi-view feature changes, reducing the instability of model training under limited data scale, and improving the generalization ability of model in multi-view target detection and tracking tasks. Attached Figure Description

[0083] Figure 1 This is a flowchart of an embodiment of the present invention;

[0084] Figure 2 This is a sample diagram of the multi-channel camera input in an embodiment of the present invention;

[0085] Figure 3 This is a schematic diagram of the two-dimensional feature extraction results in an embodiment of the present invention;

[0086] Figure 4 This is a schematic diagram showing the result of projecting multi-view features onto the BEV plane in an embodiment of the present invention;

[0087] Figure 5 This is a schematic diagram of the multi-target detection and tracking results in an embodiment of the present invention;

[0088] Figure 6 This is a schematic diagram of FoV mask generation according to an embodiment of the present invention;

[0089] Figure 7 This is a schematic diagram of the View-Spatial Joint Attention (VAG) and Uncertainty Branching (UAF) modules in an embodiment of the present invention.

[0090] Figure 8 This is a schematic diagram of the viewpoint weight and effective observation area results in an embodiment of the present invention;

[0091] Figure 9 This is a flowchart illustrating the generation process of the prior map of the sensing region in an embodiment of the present invention.

[0092] Figure 10 This is a flowchart of the cross-perspective consistency learning training process according to an embodiment of the present invention;

[0093] Figure 11 This is a schematic diagram of the decoder network structure according to an embodiment of the present invention;

[0094] Figure 12 This is a flowchart illustrating the tracker update and cascading association process according to an embodiment of the present invention. Detailed Implementation

[0095] The present invention will be further described in detail below through specific implementation examples to make it easier to understand. Some operations related to the present invention may not be mentioned or described in detail in this specification, in order to avoid the inventive solutions proposed by the present invention being overlooked. For those skilled in the art, they can understand the relevant operations based on the design of the solutions mentioned in the present invention and their own general technical skills.

[0096] refer to Figure 1 The specific implementation steps of the embodiments of the present invention will be described in further detail below.

[0097] Step 100: Obtain the synchronized image at time t from the multiple cameras, such as... Figure 2 As shown.

[0098] Step 200: Apply the encoder to extract features from images at various viewpoints, obtaining two-dimensional feature vectors. The encoded features can then be visualized, such as... Figure 3 As shown.

[0099] Step 300: Construct a perspective transformation matrix based on camera intrinsic and extrinsic parameters, project the 2D features from each viewpoint onto the BEV plane to obtain multi-view BEV features, enhancing visual consistency, such as... Figure 4 As shown.

[0100] Step 400: Calculate the effective observation area of ​​each viewpoint in the BEV plane and generate a FoV mask to constrain the subsequent fusion stage to aggregate multi-view information only in the effective area, such as... Figure 6 As shown.

[0101] Step 500: Introduce the View-Spatial Joint Attention Module (VAG), calculate multi-view weights for each BEV grid location, and then... Figure 7 As shown.

[0102] Step 600: Introduce an uncertainty estimation branch into the attention weight calculation to output an uncertainty heatmap, and perform uncertainty penalty correction on the attention logits to achieve uncertainty-aware fusion (UAF). Then, perform weighted fusion of multi-view BEV features to output weighted fused BEV features. Simultaneously, the weight distribution of different views can be visualized, such as... Figures 7-8 As shown.

[0103] Step 700: Introduce a regional prior heatmap to modulate the location-related gain of the fused features, making the model focus more on the target in high-prior regions and suppress false alarms in low-prior regions, such as... Figure 10 As shown.

[0104] Step 800: Input the fused BEV features into the decoding network and output the target center heatmap, offset, scale, orientation, and identity features through multiple decoupling heads. Decode the target coordinates and ID information, and output the final trajectory result based on the tracking update strategy, such as... Figures 11-12 As shown in the image. Meanwhile, the visualization results of the BEV prediction are as follows: Figure 5 As shown in the figure, the coordinates of the BEV viewpoint are the corresponding bright spots, which represent the predicted position of the target. The left side shows the actual value and the right side shows the predicted result.

[0105] refer to Figure 6In embodiments of the present invention, the FoV mask is used to characterize the effective observable region of each camera on the BEV plane. Specifically, feature maps with the same two-dimensional feature dimensions as each viewpoint and all values ​​set to 1 can be mapped to the BEV plane through the same perspective transformation as feature projection, and the projection result is thresholded to obtain a 0-1 mask. The FoV mask is used in subsequent fusion to mask invalid regions and prevent invalid projections from introducing noisy features into the fusion process.

[0106] refer to Figure 7 In an embodiment of the present invention, the VAG module assigns view confidence to each BEV grid position during the BEV fusion stage and uses the final attention logits corrected by UAF.

[0107] The first step is to weight the BEV features for each viewpoint and calculate the attention logits:

[0108] ;

[0109] in, Indicates the first Group Sample No. The camera in the The attention logits of the image at each time step; Represents the viewpoint-space joint attention module (VAG).

[0110] The second step is to use a FoV mask to suppress invalid regions in logits:

[0111] ;

[0112] in, Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point Attention logits suppressed by the FoV mask; This represents a positive constant used to reduce the logits of invalid regions to a very small value.

[0113] The third step involves using the uncertainty estimation branch to output an uncertainty logarithm plot for each viewpoint BEV feature, and then performing an exponential mapping on it to obtain the uncertainty quantity:

[0114] ;

[0115] in, For the first Group Sample No. The camera in the The uncertainty is obtained by exponentially mapping the uncertainty estimation branch results at each time step. This indicates an exponential operation with the natural constant e as the base. This indicates the uncertainty estimation branch.

[0116] Fourth step, utilize right Adjust the penalty accordingly, and obtain And use it as input to softmax:

[0117] ;

[0118] in, Indicates the first Group Sample No. The camera in the The final logits after UAF correction at each time step; Indicates the penalty coefficient. Represents the logarithmic function with base 10. This represents a numerically stable term, a very small positive constant whose value takes a value of 100%. .

[0119] Fifth step: Apply softmax to the corrected logits in the view dimension to obtain the weights:

[0120] ;

[0121] in, Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point Perspective weighting on the top Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point The final logits after UAF correction; This represents the summation operation. The number indicating the viewpoint. =1,2,...., , Indicates the total number of viewpoints.

[0122] Step 6: Using weights, weight each element of the BEV features from each perspective:

[0123] ;

[0124] in, Indicates the first Group Sample No. The camera in the The weighted BEV characteristics at each time point, This indicates element-wise multiplication. Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point The corresponding viewpoint weights.

[0125] Step 7: Summate the weighted multi-view features along the view dimension to obtain the BEV fusion features:

[0126] ;

[0127] in, Indicates the first Group of samples in the first BEV features after multi-view fusion at each moment.

[0128] Therefore, VAG can adaptively reduce the corresponding view weight in occluded or poorly imaged areas, thereby improving the accuracy of multi-view fusion; UAF explicitly injects view uncertainty into the fusion weight, making the contribution of views with higher uncertainty in local areas smaller, thereby reducing multi-view fusion conflicts and improving robustness.

[0129] refer to Figure 8 This image shows the weighted heatmap of each camera at its corresponding position in the BEV plane. By comparison, it can be observed that the weight of some cameras decreases while the weight of other cameras increases in the occluded area, thus demonstrating the interpretability of the fusion.

[0130] refer to Figure 9 In embodiments of the present invention, the regional prior heatmap can be generated from historical trajectory statistics or labeled data and stored as a matrix of the same size as the BEV grid. During the fusion stage, this prior is used as a location gain term to lightly modulate the fusion features, enabling the network to focus more on target response in high-priority regions while reducing false detections in low-priority regions.

[0131] Specifically, the step of using the region prior heatmap to modulate the gain of the fused features is as follows:

[0132] The first step is to obtain the count distribution on the BEV grid offline and normalize it to form a priori heatmap of the region:

[0133] ;

[0134] in, Represents BEV coordinates The region of prior values, Indicates position The cumulative count; This indicates all BEV grid locations. Take the maximum value.

[0135] The second step involves applying the region's prior heatmap as a gain term to the weighted multi-view BEV features to achieve location-dependent gain modulation.

[0136] ;

[0137] in, Indicates the first Group of samples in the first BEV characteristics after multi-view fusion at each moment. This represents the assignment / update symbol. This represents the gain coefficient, with a value of 0.1.

[0138] refer to Figure 10 In an embodiment of the present invention, the steps for training the student network and the teacher network using the supervised training constraints and the cross-perspective self-supervised consistency constraints are as follows:

[0139] The first step involves performing the feature extraction, projection, fusion, and decoding processes in steps 1 to 6 on the same batch of multi-view inputs, using both the student network and the teacher network respectively, to obtain the decoding detection results output by the student network and the multi-view BEV features corresponding to the student network and the teacher network.

[0140] The second step involves defining an effective BEV grid set based on the union of the FoV masks. A supervised loss is constructed based on the decoding detection results output by the student network, and a consistency loss is constructed based on the multi-view BEV features corresponding to the student and teacher networks. The supervised loss consists of the difference between the student network's decoding results and the training sample annotations, and includes at least a center heatmap supervision term, a position offset supervision term, a scale and orientation supervision term, and an identity feature supervision term, used to constrain the consistency between the student network's output detection results and the annotation results. The consistency loss is used to constrain the multi-view BEV features output by the student and teacher networks under the same training sample input to remain consistent within the effective BEV grid set, thereby reducing the feature differences between the teacher and student networks at corresponding spatial locations.

[0141] ;

[0142] in, This indicates a loss of consistency. Indicates the number of elements in the set. Indicates the BEV coordinates Summing the effective grid positions in the middle; This represents the L2 norm operation. Indicates the student network in BEV grid coordinates BEV feature vector at location; Indicates the teacher network in BEV grid coordinates The BEV feature vector at that location.

[0143] The third step is to weightedly sum the supervision loss and consistency loss to update the student network parameters. Specifically, the student network parameters are: Teacher network parameters are .

[0144] Student parameters are updated from student network parameters and teacher network parameters according to EMA rules:

[0145] ;

[0146] in, Indicates the total training loss. Indicates monitoring losses, This represents the consistency loss weighting coefficient, with a value of 0.1.

[0147] Fourth, during training, the student network updates its parameters by minimizing the total training loss, while the teacher network parameters are updated using an exponential moving average of the student network parameters. This ensures that the teacher network outputs a more stable feature representation than the student network.

[0148] ;

[0149] in, is the EMA attenuation coefficient, a hyperparameter set to 0.9995.

[0150] refer to Figure 11 The decoding network in this embodiment of the invention adopts a structure that combines a shared backbone with a multi-branch decoupled prediction head to decode the target detection results of the fused BEV features and provide target coordinates, confidence and identity information for subsequent multi-target tracking.

[0151] Specifically, the fused BEV feature map is first input into the shared part of the decoding network. This shared part further encodes the multi-view fused features, extracting high-level semantic features suitable for target localization, attribute estimation, and identity representation. Based on this, the decoding network splits into multiple prediction branches with different functions, which are used to output the target center heatmap, position offset, target size, target orientation, and identity features, respectively. Specifically, the center prediction head provides the candidate center position of the target in the BEV grid; the offset prediction head refines and compensates the discrete grid center to reduce localization errors caused by grid quantization; the size prediction head estimates the geometric scale of the target in the bird's-eye view plane; the orientation prediction head outputs the target's rotation or orientation information; and the identity feature prediction head outputs identity embedding features that can be used for cross-frame association.

[0152] Furthermore, the image-level auxiliary branch shown on the right side of the figure is an optional training branch, which can output image-level center prediction, image-level offset, size prediction and image-level identity features, and participate in auxiliary loss calculation during the training phase to improve the stability of the multi-view learning process; during the inference phase, the image-level auxiliary branch may not participate in the final detection result output.

[0153] Therefore, the decoding network transforms the fused BEV features output by the preceding modules into structured results that can be used for target detection and tracking. It then feeds the decoded target coordinates, confidence levels, and identity information into the tracker for subsequent target association and trajectory state updates. This structure achieves decoupled prediction of target position, geometric attributes, and identity information, while also providing a unified and directly usable input for the subsequent tracking stage.

[0154] refer to Figure 12 In this embodiment of the invention, the tracker update and cascade association process is used to perform temporal association and trajectory state maintenance on the current frame detection results output by the decoding network, thereby obtaining the valid trajectory of the current frame and its corresponding target identity, location and status information.

[0155] Specifically, the tracker's input includes the target coordinates, target confidence score, and identity features from the current frame's detection results. First, the current frame's detection results are filtered by confidence score, retaining only targets with confidence scores above a detection threshold to construct candidate detection trajectories. Then, historical trajectories are divided into confirmed trajectories, lost trajectories, removed trajectories, and unconfirmed trajectories. Confirmed trajectories and lost trajectories are combined to form a set of trajectories to be associated. To improve the temporal continuity of the association, Kalman filtering is used to predict the state of each trajectory in the set of trajectories to be associated before association, thereby obtaining the estimated positions of each historical trajectory in the current frame.

[0156] In an embodiment of the present invention, the trajectory association adopts a cascaded matching method.

[0157] In the first stage, identity features are prioritized for association with motion information. Specifically, the identity feature distance between historical trajectories and current frame candidate detections is first calculated. Then, motion consistency constraints are applied based on Kalman prediction results to construct the first association cost matrix, and the matching results are solved through linear allocation. For trajectories that are successfully matched in this stage, if the current state of the trajectory is tracked, the trajectory is updated using the current detection results; if the trajectory was previously lost, it is reactivated, returning it to the valid tracking set.

[0158] For confirmed trajectories and remaining detected targets that still do not match after the first-stage association, a second-stage association is performed. In this stage, a second association cost matrix is ​​constructed mainly based on the center distance between the predicted trajectory location and the detection location, and matching is completed again through linear allocation. For trajectories that are successfully matched in this stage, their status is updated using the current detection results; for historical trajectories that still do not match in this stage, they are marked as lost trajectories to be retrieved in subsequent frames.

[0159] Furthermore, for the set of unconfirmed trajectories, i.e. newly created trajectories that have not yet been sufficiently verified by historical observations, a separate association process is used for processing. Specifically, a cost matrix is ​​constructed using the center distance between the unconfirmed trajectories and the remaining detected targets. After matching is completed, the successfully matched unconfirmed trajectories are updated and transformed into valid trajectories; unconfirmed trajectories that are still not matched are directly removed to reduce unstable trajectories introduced by occasional false detections.

[0160] For detected targets that are still not matched after the above two-stage association and unconfirmed trajectory processing, if their detection confidence is not lower than a set threshold, they are initialized as new trajectories and assigned new target identification information. At the same time, for trajectories in the lost trajectory set, if the number of frames elapsed since the last successful match exceeds a preset time threshold, they are marked as removed trajectories to avoid long-term invalid trajectories continuously occupying tracking resources.

[0161] Finally, the tracked, lost, and removed trajectories are aggregated, updated, and duplicate trajectories are cleaned up to remove trajectories that may repeatedly represent the same target. The valid trajectories active in the current frame are then output. The output includes at least the trajectory identifier, target location, and trajectory status information. Thus, the tracker achieves stable matching between the current frame detection results and historical trajectories through a cascaded association method that prioritizes identity features, supplements location distance, and assists motion prediction. This helps reduce the impact of target occlusion, short-term loss, and local false detections on the continuity of target tracking across frames.

[0162] The effects of this invention will be further illustrated below with simulation experiments:

[0163] 1. Simulation experimental conditions.

[0164] The simulation experiment hardware platform of this invention is as follows: the CPU processor is an Intel Xeon 4210R multi-core processor, the graphics card is an NVIDIA RTX 4090 with 24GB of video memory, and the memory size is 64GB.

[0165] The simulation experiment software platform of this invention is: Ubuntu 20.04, CUDA and CUDNN versions 11.2 and 8.1.1 respectively, based on the PyTorch 13.1 deep learning framework.

[0166] The dataset used in the simulation experiments of this invention is the publicly available multi-view pedestrian tracking dataset Wildtrack. This dataset was acquired synchronously by multiple fixed cameras at the same time and includes synchronized multi-view pedestrian images and corresponding annotations, which can be used to verify the performance of multi-view target detection and tracking algorithms. The evaluation metrics used in the simulation experiments of this invention are: Multi-target detection accuracy (MODA), Multi-target detection precision (MODP), Multi-target tracking accuracy (MOTA), Multi-target tracking precision (MOTP), and Track Precision.

[0167] 2. Simulation content and result analysis.

[0168] The simulation experiment of this invention consists of two parts.

[0169] Simulation Experiment 1: Performance comparison experiment between the method of this invention and the baseline method.

[0170] Simulation Experiment 1 of this invention uses the method of this invention and the existing baseline method to conduct multi-view target detection and tracking experiments on the Wildtrack dataset, and obtains MODA, MODP, MOTA, MOTP and Track_Precision results respectively. The results are then plotted in tables or figures for comparison and explanation.

[0171] In simulation experiment 1, the existing baseline method used is the EarlyBird multi-target tracking algorithm proposed by Torben Teepe, Philipp Wolters, Johannes Gilg, Fabian Herzog, and Gerhard Rigoll in their paper "EarlyBird: Early-Fusion for Multi-View Tracking in the Bird's Eye View" (Proceedings of the 2024 IEEE / CVF Winter Conference on Applications of Computer Vision Workshops, 2024: 102-111). During the experiment, the predicted pedestrian coordinates in the BEV plane were used as the object of detection and tracking evaluation.

[0172] Among them, detection performance is measured by the correspondence between the predicted target and the real target in the BEV plane; tracking performance is measured by the consistency between the target association results in each frame and the real identity annotation.

[0173] Simulation Experiment 2: Ablation experiments of each innovative module of this invention.

[0174] Simulation Experiment 2 of this invention is based on the existing baseline method. It introduces the View-Space Joint Attention Module (VAG), the Uncertainty Perception Fusion Module (UAF), the Cross-View Self-Supervised Consistency Training Module, and the Region Prior Mechanism step by step. Five evaluation metrics (Multi-Target Detection Accuracy (MODA), Multi-Target Detection Precision (MODP), Multi-Target Tracking Accuracy (MOTA), Multi-Target Tracking Precision (MOTP), and Tracking Precision (Track_Precision)) are used to compare the detection and tracking performance under different module combinations. The experimental results are listed in Table 1.

[0175]

[0176]

[0177]

[0178]

[0179]

[0180] The simulation experiments in Experiment 2 included the following methods: Baseline, +VAG, +VAG+UAF, +VAG+UAF+CVC, and +ALL. By comparing the experimental results of each group, the contribution of each component module in this invention to the final performance can be demonstrated.

[0181] 3. Explanation of the effectiveness of the invention based on simulation experiment results.

[0182] The experimental results show that after introducing the viewpoint-space joint attention module, the model's ability to effectively utilize information in the multi-view feature fusion process is enhanced, and both detection and tracking metrics are improved. After further introducing the uncertainty-aware fusion module, the model can suppress observations with high uncertainty, thereby improving the fusion effect in complex scenes. After adding cross-view self-supervised consistency training, the model's consistency in feature representations under different viewpoint inputs is enhanced, and the tracking performance is further improved. After comprehensively introducing all the improved modules, such as region prior, the method of this invention achieves better overall performance results.

[0183] Table 1. Comparison of simulation results between the method of this invention, the baseline method, and combinations of various improved modules.

[0184]

[0185] As shown in Table 1, compared with existing baseline methods, the method of this invention achieves better results in terms of multi-target detection accuracy, detection precision, tracking accuracy, and overall tracking performance. Specifically, the method incorporating all improved modules outperforms the baseline method in MODA, MODP, MOTA, and Track Precision, with MODA reaching 93.27%, MODP 81.47%, MOTA 92.43%, and MOTP 11.66%. This demonstrates that the multi-view fusion and training strategy proposed in this invention can effectively improve the performance of multi-view target detection and tracking.

[0186] The simulation experiments above show that by introducing view-space joint attention, uncertainty perception fusion, region prior modulation, and teacher-student consistency training mechanism into the multi-view target detection and tracking process, this invention can improve the effectiveness of multi-view information fusion and enhance the stability and robustness of target detection and tracking in complex scenarios. Therefore, it is an effective multi-view target detection and tracking method.

Claims

1. A multi-view target detection and tracking method based on view-space joint attention, characterized in that, The steps of this method include the following: Step 1: Extract the two-dimensional features of the current image from each camera and project the two-dimensional features onto the BEV plane; Step 2: Calculate the effective observation area for each viewpoint at each BEV grid location; Step 3: Calculate the weights for each view using the constructed view-space joint attention module (VAG); Step 4: Use the constructed uncertainty-aware fusion module UAF to correct the viewpoint weights and obtain the BEV fusion features; Step 5: Use the region prior heatmap to perform gain modulation on the BEV fusion features; Step 6: Input the modulated fused features into the decoding network and output the center heatmap, offset, scale / orientation prediction, and identity feature detection results of the target in the current frame. Step 7: Copy the learnable network structure related to the extracted two-dimensional features, feature fusion, and decoded detection results into a teacher network with the same structure. Use the original network as the student network. Input the same training data into the student network and the teacher network respectively. Train the student network and the teacher network through the supervised training constraints and the cross-view self-supervised consistency constraints. After training, retain the trained student network parameters. Step 8: In the inference phase, the trained student network parameters are loaded into the networks corresponding to Steps 1 to 6, Steps 1 to 6 are re-executed, and the decoded detection results output in Step 6 are input into the tracker to output the final multi-view target detection results and multi-target tracking results.

2. The multi-view target detection and tracking method according to claim 1, characterized in that, The steps for extracting the two-dimensional features of the current image from each camera in step 1 are as follows: The first step is to align each video stream corresponding to each camera viewpoint at a unified time to obtain the current frame image corresponding to each viewpoint, and then combine the current frame images at all unified alignment times into a current frame image set. The second step involves inputting the current frame image of each channel into the feature extraction sub-network of a convolutional neural network with shared parameters, and outputting two-dimensional features of the current camera image with a preset channel dimension; the feature extraction sub-network includes a backbone convolutional coding layer and a feature transformation layer.

3. The multi-view target detection and tracking method according to claim 2, characterized in that, The projection of the two-dimensional features onto the BEV plane described in step 1 is obtained by the following formula: ； in, Indicates the first Group Sample No. The camera in the Two-dimensional features of the image projected onto the BEV plane at each time step. This indicates a projection operation. Indicates the first Group Sample No. The camera in the Two-dimensional features of an image at a given time point. express Indicates the first Group Sample No. The camera in the The projection matrix from image feature coordinates to BEV grid coordinates at each moment is determined by the camera's intrinsic and extrinsic parameters and the reference bird's-eye view plane.

4. The multi-view target detection and tracking method according to claim 3, characterized in that, The calculation of the effective observation area for each viewpoint at each BEV grid location in step 2 is obtained by the following formula: ； in, Indicates the first Group Sample No. The camera in the Image BEV coordinates at each time point The mask for the effective observation region, whose values ​​range from {0,1}. This indicates an indicator function; the value is 1 if the condition is true, and 0 otherwise. Denotes the first digit of the projected all-one two-dimensional matrix. Group Sample No. The camera in the Image BEV coordinates at each time point The value of , The threshold representing the minimum value is 0.

5. In the middle, when The value is greater than When the time is right, it is recorded as the effective observation area, that is... Otherwise, it is recorded as an invalid observation area, i.e. .

5. The multi-view target detection and tracking method according to claim 4, characterized in that, The steps in step 3 for calculating the weights of each viewpoint using the constructed view-space joint attention module (VAG) are as follows: The first step is to construct a View-Spatial Joint Attention Module (VAG). The VAG module shares a lightweight convolutional sub-network for each viewpoint, used to generate attention logits from BEV features. The convolutional sub-networks are configured in a concatenated order, including: a 1x1 convolutional layer used to separate the channel dimensions... Mapped to intermediate channel Then, a non-linear activation layer is connected in series, using the ReLU activation function; finally, a 1*1 convolutional layer is connected in series to convert the intermediate channels. Mapped to a single-channel logits output; The second step, calculating the weight of each viewpoint, involves inputting the two-dimensional features of the effective observation area of ​​each viewpoint at each BEV grid position into the viewpoint-space joint attention module (VAG), and then combining the output with the effective observation area mask and performing softmax normalization in the viewpoint dimension to obtain the weight corresponding to each viewpoint at each BEV grid position.

6. The multi-view target detection and tracking method according to claim 1, characterized in that, The Uncertainty Awareness Fusion Module (UAF) described in step 4 is configured in parallel with the View-Space Joint Attention Module (VAG); the UAF includes an uncertainty estimation branch, a logarithmic mapping correction unit, and a view weight calculation unit; wherein, The uncertainty estimation branch takes the BEV features of each viewpoint as input and includes, in a concatenated order: a 3*3 convolutional layer to map the channel dimension from the input channel to the intermediate channel; followed by a non-linear activation layer using the ReLU activation function; and then a 1*1 convolutional layer to output the uncertainty logarithm map corresponding to each viewpoint and each BEV grid position. The logarithmic mapping correction unit obtains the uncertainty quantity by exponential mapping the uncertainty logarithmic graph, and applies it to the attention logits through a penalty correction method to obtain the corrected final logits. The viewpoint weight calculation unit performs softmax normalization on the corrected final logits in the viewpoint dimension to recalculate the weights of each viewpoint at each BEV grid position, and performs element-wise weighted summation and viewpoint dimension summation on the multi-view BEV features based on the weights to obtain uncertainty-aware BEV fusion features.

7. The multi-view target detection and tracking method according to claim 4, characterized in that, The steps in step 4, which involve using the constructed uncertainty-aware fusion module UAF to correct the viewpoint weights, are as follows: The first step is to weight the BEV features for each viewpoint and calculate the attention logits: ； in, Indicates the first Group Sample No. The camera in the The attention logits of the image at each time step; Represents the viewpoint-space joint attention module (VAG); The second step is to use a FoV mask to suppress invalid regions in logits: ； in, Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point Attention logits after being suppressed by FoVmask; This represents a positive constant used to reduce the logits of invalid regions to a very small value. The third step involves using the uncertainty estimation branch to output an uncertainty logarithm plot for each viewpoint BEV feature, and then performing an exponential mapping on it to obtain the uncertainty quantity: ； in, For the first Group Sample No. The camera in the The uncertainty is obtained by exponentially mapping the uncertainty estimation branch results at each time step. This indicates an exponential operation with the natural constant e as the base. Indicates the uncertainty estimation branch; Fourth step, utilize right Adjust the penalty accordingly, and obtain And use it as input to softmax: ； in, Indicates the first Group Sample No. The camera in the The final logits after UAF correction at each time step; Indicates the penalty coefficient. Represents the logarithmic function with base 10. This represents a numerically stable term, a very small positive constant whose value takes a value of 100%. ; Fifth step: Apply softmax to the corrected logits in the view dimension to obtain the weights: ； in, Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point Perspective weighting on the top Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point The final logits after UAF correction; This represents the summation operation. The number indicating the viewpoint. =1,2,...., , Indicates the total number of viewpoints; Step 6: Using weights, weight each element of the BEV features from each perspective: ； in, Indicates the first Group Sample No. The camera in the The weighted BEV characteristics at each time point, This indicates element-wise multiplication. Indicates the first Group Sample No. The camera in the BEV coordinates of the image at each time point Corresponding perspective weights; Step 7: Summate the weighted multi-view features along the view dimension to obtain the BEV fusion features: ； in, Indicates the first Group of samples in the first BEV features after multi-view fusion at each moment.

8. The multi-view target detection and tracking method according to claim 6, characterized in that, The step of using the region prior heatmap to modulate the gain of the fused features in step 5 is as follows: The first step is to obtain the count distribution on the BEV grid offline and normalize it to form a priori heatmap of the region; ； in, Represents BEV coordinates The region of prior values, Indicates position The cumulative count; This indicates the location of all BEV grid points. Take the maximum value; The second step involves applying the region's prior heatmap as a gain term to the weighted multi-view BEV features to achieve location-dependent gain modulation. ； in, Indicates the first Group of samples in the first BEV characteristics after multi-view fusion at each moment. This represents the assignment / update symbol. This represents the gain coefficient, with a value of 0.

1.

9. The multi-view target detection and tracking method according to claim 8, characterized in that, The training data mentioned in step 7 refers to training samples consisting of multi-view images and their corresponding annotations synchronously acquired by multiple fixed cameras at the same time. Each training sample includes at least the current frame image corresponding to each camera viewpoint at the same time, the center position annotation, position offset annotation, scale / orientation annotation, and target identity annotation of the target in the BEV plane. The current frame images corresponding to each camera viewpoint at the same timestamp serve as input data shared by the student network and the teacher network. The center position annotation, position offset annotation, scale / orientation annotation, and target identity annotation are used to construct the supervised training constraints in step 7.

10. The multi-view target detection and tracking method according to claim 8, characterized in that, The learnable network parameters for training the student network and teacher network through the supervised training constraints and the cross-view self-supervised consistency constraints in step 7 include: the feature extraction network parameters in step 1, the view-space joint attention module parameters in step 3, the uncertainty perception fusion module parameters in step 4, and the decoding network parameters in step 6.

11. The multi-view target detection and tracking method according to claim 8, characterized in that, The steps in step 7 for training the student network and teacher network using the supervised training constraints and the cross-perspective self-supervised consistency constraints are as follows: The first step is to perform the feature extraction, projection, fusion and decoding processes in steps 1 to 6 on the same batch of multi-view inputs, using the student network and the teacher network respectively, to obtain the decoding detection results output by the student network and the multi-view BEV features corresponding to the student network and the teacher network. The second step involves defining an effective BEV grid set based on the union of the FoV masks. A supervisory loss is constructed based on the decoding detection results output by the student network, and a consistency loss is constructed based on the multi-view BEV features corresponding to the student and teacher networks. The supervisory loss consists of the difference between the student network's decoding results and the training sample annotations, and includes at least a center heatmap supervision term, a position offset supervision term, a scale / orientation supervision term, and an identity feature supervision term, used to constrain the consistency between the detection results output by the student network and the annotation results. The consistency loss is used to constrain the multi-view BEV features output by the student and teacher networks under the same training sample input to remain consistent within the effective BEV grid set, thereby reducing the feature differences between the teacher and student networks at corresponding spatial locations. ； in, This indicates a loss of consistency. Indicates the number of elements in the set. Indicates the BEV coordinates Summing the effective grid positions in the middle; This represents the L2 norm operation. Indicates the student network in BEV grid coordinates BEV feature vector at location; Indicates the teacher network in BEV grid coordinates BEV feature vector at location; The third step is to weightedly sum the supervision loss and consistency loss to update the student network parameters. Specifically, the student network parameters are: Teacher network parameters are ; Student parameters are updated from student network parameters and teacher network parameters according to EMA rules: ； in, Indicates the total training loss. Indicates monitoring losses, This represents the consistency loss weighting coefficient, with a value of 0.1; Fourth, during training, the student network updates its parameters by minimizing the total training loss, while the teacher network parameters are updated using an exponential moving average of the student network parameters. This ensures that the teacher network outputs a more stable feature representation than the student network. ； in, is the EMA attenuation coefficient, a hyperparameter set to 0.9995.

Images