A multi-view target tracking detection method, device, terminal and medium

Abstract

The application discloses a multi-view target tracking detection method and device, a terminal and a medium. The scheme provided by the application first inputs multi-view data obtained to an encoder module containing a specific feature reconstruction pyramid model, wherein the feature reconstruction pyramid model comprises a bottom-up network unit, a top-down network unit and a feature reconstruction unit, the feature reconstruction unit is arranged on a transverse connection channel between the bottom-up network unit and the top-down network unit, information is more efficiently fused through the feature reconstruction pyramid model, important spatial features and refined channel features are extracted, feature representation of a key region is optimized, thereby eliminating redundant information, improving the accuracy of feature aggregation, and further improving the target detection and tracking capability in a multi-view scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine vision, and in particular to a multi-view target tracking detection method and device, a terminal and a medium. BACKGROUND

[0002] A multi-view system uses multiple overlapping cameras to observe a target from different angles, effectively reducing the problem of occlusion, but at the same time brings the challenge of how to effectively aggregate multi-view features. At present, the mainstream multi-view feature aggregation strategy is an early fusion strategy, and the specific operation logic is as follows: first, project all views onto the ground plane, and then process them for detection. The multi-view tracking method based on the early fusion strategy follows the opposite association process, first time-associates according to the detection results of each view, and then performs cross-view matching.

[0003] However, in actual application scenarios, complementary information and redundant information between different views are intertwined, resulting in deviation in feature aggregation, which seriously affects the target detection and tracking capability in a multi-view scene. SUMMARY

[0004] The present application provides a multi-view target tracking detection method, device, terminal and medium, which is used to solve the technical problem of insufficient target detection and tracking capability in a multi-view scene in the prior art.

[0005] To solve the above technical problems, the first aspect of the present application provides a multi-view target tracking detection method, comprising:

[0006] Obtaining multi-view data to be detected;

[0007] Inputting the multi-view data into a preset encoder module to obtain multi-view aggregated features corresponding to the multi-view data through processing of the encoder module, wherein the encoder module contains a preset feature reconstruction pyramid model, and the feature reconstruction pyramid model comprises a bottom-up network unit, a top-down network unit and a feature reconstruction unit, wherein the feature reconstruction unit is arranged on a transverse connection channel between the bottom-up network unit and the top-down network unit, and is used for eliminating redundant information and extracting key features according to feature information output by the bottom-up network unit;

[0008] Based on the multi-view aggregated features, projection, aggregation and decoding processing are sequentially performed to obtain a target tracking detection result.

[0009] Preferably, the processing mode of the feature reconstruction unit comprises:

[0010] According to the image features output by each layer of the bottom-up network unit, the image features are subjected to convolution processing.

[0011] The image features after convolution processing are combined with two weights generated by adaptive gating logic to obtain two sets of weighted image features, and the two sets of weighted image features are cross-reconstructed to obtain reconstructed features.

[0012] According to the channel dimension, the reconstructed features are split into two sets of input features, and the two sets of input features are respectively subjected to convolution processing, and then the processed features are fused, and the channel weight value of each channel dimension is calculated based on the fused features.

[0013] According to the product of the channel weight value and the reconstructed feature, a channel weighted feature is obtained.

[0014] The channel weighted feature is sequentially subjected to convolution and activation processing and then input to the top-down network unit to obtain a multi-view aggregation feature.

[0015] Preferably, combining the image features after convolution processing with two weights generated by adaptive gating logic to obtain two sets of weighted image features comprises:

[0016] The image features after convolution processing are subjected to group normalization processing to obtain normalized features.

[0017] The normalized features are multiplied by a normalized related weight value to be reweighted, and then the product of the normalized features and the normalized related weight value is mapped to a preset binary interval through a Sigmoid function, and then two weights are obtained through adaptive gating logic.

[0018] According to the product of the image features and the two weights, two sets of weighted image features are respectively obtained.

[0019] Preferably, according to the channel dimension, the reconstructed features are split into two sets of input features, and the two sets of input features are respectively subjected to convolution processing, and then the processed features are fused, and the channel weight value of each channel dimension is calculated based on the fused features.

[0020] According to the channel dimension, the reconstructed features are split into a first input feature and a second input feature.

[0021] The first input feature is subjected to feature extraction through group convolution and point-by-point convolution respectively, and then the features obtained through the two kinds of convolution processing are summed to obtain a first intermediate feature.

[0022] The second input feature is subjected to feature extraction through point-by-point convolution, and then the extracted feature is spliced with the second input feature to obtain a second intermediate feature.

[0023] The first intermediate feature and the second intermediate feature are fused by a global average pooling processing mode to obtain compressed channel features, and then based on the compressed channel features, a convolution and a nonlinear activation function are used to determine the channel weight value corresponding to each channel dimension.

[0024] Preferably, before the projection processing based on the multi-view aggregation feature further includes:

[0025] The multi-view aggregation feature is extracted by a separate convolution processing and a plurality of preset expansion rate parameters to obtain multi-scale features, and then a 1*1 convolution layer is used to aggregate the multi-scale features and the multi-view aggregation features to obtain multi-scale aggregation features.

[0026] According to the image space resolution parameters of the multi-view aggregation feature, the high element feature and the wide element feature in the image space resolution parameters are respectively adaptively pooled to obtain corresponding pooled features, and then the attention score is generated according to the product of the two groups of pooled features combined with the Sigmoid function.

[0027] The multi-scale aggregation weighted feature is obtained by multiplying the multi-scale aggregation feature and the attention score, and then the multi-scale aggregation weighted feature and the original multi-scale aggregation feature are added to obtain the final multi-view aggregation feature.

[0028] Preferably, the feature reconstruction pyramid model further includes a scale aggregation unit for connecting the features of multiple scales output by the top-down network unit to a unified representation.

[0029] Preferably, the scale aggregation unit is a feature pyramid network unit with a top-down structure.

[0030] The second aspect of the present application provides a multi-view target tracking detection device, which comprises:

[0031] A multi-view data acquisition unit is configured to acquire multi-view data to be detected.

[0032] A multi-view feature processing unit is configured to input the multi-view data into a preset encoder module to obtain multi-view aggregation features corresponding to the multi-view data through processing of the encoder module, wherein the encoder module comprises a preset feature reconstruction pyramid model, and the feature reconstruction pyramid model comprises a bottom-up network unit, a top-down network unit, and a feature reconstruction unit.

[0033] a target detection unit, configured to sequentially perform projection, aggregation and decoding processing based on the multi-view aggregated features, to obtain a target tracking detection result.

[0034] The third aspect of the present application provides a multi-view target tracking detection terminal, comprising a memory and a processor.

[0035] The memory is configured to store program code for implementing the multi-view target tracking detection method according to the first aspect of the present application.

[0036] The processor is configured to read and execute the program code.

[0037] The fourth aspect of the present application provides a computer readable storage medium, which stores program code for being read and executed by a processor to implement the multi-view target tracking detection method according to the first aspect of the present application.

[0038] From the above technical solutions, the present application has the following advantages:

[0039] The scheme provided by the present application first inputs the obtained multi-view data into an encoder module comprising a specific feature reconstruction pyramid model, wherein the feature reconstruction pyramid model comprises a bottom-up network unit, a top-down network unit and a feature reconstruction unit, the feature reconstruction unit is arranged on a transverse connection channel between the bottom-up network unit and the top-down network unit, the feature reconstruction pyramid model more efficiently fuses information, extracts important spatial features and refines channel features, optimizes feature representation of key regions, thereby eliminating redundant information, improving the accuracy of feature aggregation, and further improving the target detection and tracking capability in a multi-view scene. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0041] Figure 1 The logic flowchart of the existing multi-view target tracking detection based on EarlyBird.

[0042] Figure 2 The flowchart of the multi-view target tracking detection method embodiment provided by the present application.

[0043] Figure 3 A logic flow chart of an embodiment of a multi-view object tracking detection method provided in the present application.

[0044] Figure 4 A feature reconstruction pyramid model block diagram in a multi-view object tracking detection method provided in the present application.

[0045] Figure 5 A logic block diagram of a feature reconstruction unit in a feature reconstruction pyramid model provided in the present application.

[0046] Figure 6 A logic block diagram of an adaptive spatial enhancer in a feature reconstruction unit provided in the present application.

[0047] Figure 7 A logic block diagram of a channel refinement module in a feature reconstruction unit provided in the present application.

[0048] Figure 8 A logic block diagram of a multi-scale context enhancement processing in a multi-view object tracking detection method provided in the present application.

[0049] Figure 9 A structural schematic diagram of an embodiment of a multi-view object tracking detection device provided in the present application.

[0050] Figure 10 A structural schematic diagram of an embodiment of a multi-view object tracking detection terminal provided in the present application. DETAILED DESCRIPTION

[0051] As shown in Figure 1 Earlybird is a method based on early fusion strategy, which combines 2D detection with cross-view association, directly identifies targets in the bird's eye view (BEV) space, and learns strong Re-ID features for each detection to perform temporal association. The following is the main implementation scheme of Earlybird:

[0052] 1. Encoder: encode the synchronous RGB input images from S cameras using ResNet or SwinTransformer network, and finally reduce the image by 4 times.

[0053] 2. Projection: project the image features onto the ground plane using view transformation, and calculate the translation between the three-dimensional position (x, y, z) and the two-dimensional image pixel coordinates (u, v):

[0054] (1)

[0055] where s is a real-valued scaling factor, P = K[R|t] is a 3x4 view transformation matrix, K is a camera intrinsic matrix, and [R|t] is a 3x4 extrinsic parameter matrix. After that, all pixels are projected to the ground plane z = 0, then the projection can be simplified as:

[0056] (2)

[0057] where P0denotes a 3x3 view transformation matrix that does not contain the third column of P. Equation (2) is applied to project the features of S cameras into a predefined size ground plane grid [Hg, Wg]. Each grid position represents a 10cmx10cm area, and the annotation grid is further reduced by 4 samples.

[0058] 3. Aggregation: The aggregation stage combines the features of S cameras into one feature, i.e., reduces the S dimension of the BEV feature map. All feature maps are concatenated along the channel dimension to generate a high-dimensional BEV feature map. The high-dimensional BEV feature is reduced to a certain channel size by two 2D convolutions.

[0059] 4. Decoder: After aggregation, the BEV feature is input into a ResNet-18 decoder. The distortion caused by perspective projection causes pedestrian features to spread out from their actual positions on the ground plane, so a decoder is used to obtain a large ground plane receptive field. Specifically, in each layer of ResNet, the BEV feature is down-sampled by 2. Then, a pyramid network architecture is used to up-sample the output of each layer to the size of the previous larger output. Finally, the two features are concatenated in the channel dimension and a two-dimensional convolution is applied.

[0060] 5. Detection and re-identification head: The detection architecture follows CenterNet, with an added center detection head that generates a heatmap on the ground plane. Another head is added for offset prediction, which helps to more accurately predict the location as it mitigates the quantization error of the ground grid. A detection head is added for image features that predict the center of the 2D bounding box, and the position of the foot is estimated at the bottom center of the bounding box, helping the image features to have higher activation at the location of each pedestrian.

[0061] The re-identification head aims to generate features that can distinguish individual pedestrians. Ideally, the affinity between different pedestrians should be smaller than the affinity between the same pedestrian. To achieve this, Re-ID features are learned through a classification task and a metric learning task.

[0062] However, this approach is difficult to effectively distinguish between redundant information and key features, resulting in complementary information and redundant information between different views being intertwined in multi-view data processing, leading to biased feature aggregation. This severely affects the multi-view detection and tracking capabilities.

[0063] Therefore, the embodiment of the present application provides a multi-view target tracking detection method, device, terminal and medium, which is used to solve the technical problem of insufficient target detection and tracking capability of the prior art in a multi-view scene.

[0064] In order to make the application purpose, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the following described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0065] First, a detailed description of an embodiment of a multi-view target tracking detection method provided by the present application is given as follows.

[0066] Please refer to Figure 2 and Figure 3 The multi-view target tracking detection method provided by the embodiment of the present application comprises the following steps.

[0067] Step 101: acquiring multi-view data to be detected;

[0068] Step 102: inputting the multi-view data into a preset encoder module to obtain multi-view aggregated features corresponding to the multi-view data through processing of the encoder module.

[0069] The encoder module comprises a preset feature reconstruction pyramid model, and the feature reconstruction pyramid model comprises a bottom-up network unit, a top-down network unit and a feature reconstruction unit. The feature reconstruction unit is arranged on a transverse connection channel between the bottom-up network unit and the top-down network unit, and is used to eliminate redundant information and extract key features according to feature information output by the bottom-up network unit.

[0070] Step 103: sequentially performing projection, aggregation and decoding processing based on the multi-view aggregated features to obtain a target tracking detection result.

[0071] It should be noted that the bottom-up network unit refers to a network structure for extracting image features and reducing resolution through convolution layers level by level, which can be implemented by a backbone network such as ResNet, and is used to capture bottom-level detail features of multi-view data. The top-down network unit refers to a network structure for restoring feature resolution through upsampling operation, such as the top path in FPN, and is used to fuse semantic information of different scales. The feature reconstruction unit refers to a processing module arranged on the transverse connection channel, which can be implemented by convolution operation combined with adaptive gating logic, and is used to eliminate spatial redundancy and extract key features of the features output by the bottom-up network.

[0072] Specifically, when the multi-view data is processed by the encoder module, the bottom-up network extracts local features of each view layer by layer, and the feature reconstruction unit performs spatial enhancement on the features through convolution and adaptive weight assignment, eliminating background interference irrelevant to the target. Subsequently, the top-down network fuses high-level semantic information with the reconstructed bottom-level features to form multi-scale aggregated features. In the projection and decoding stage, the aggregated features are further mapped to the target detection space, and the target position and motion trajectory are output through the decoder.

[0073] The feature reconstruction pyramid model can filter key spatial features layer by layer and refine channel weights by embedding feature reconstruction units in the lateral connection channel, thereby suppressing the interference of redundant information and enhancing the feature response of the target region in the feature fusion stage. Through the above technical solution, the present application realizes more accurate feature aggregation in a multi-view scene, eliminates redundant information and strengthens the expression of key region features, thereby improving the robustness of multi-target detection and tracking in complex occlusion environments, especially in dense crowd or dynamic background scenes, which can effectively distinguish overlapping targets and reduce the false detection rate.

[0074] Based on the above basic embodiment, the feature reconstruction pyramid model further includes a scale aggregation unit, which is specifically a top-down structure feature pyramid network unit, for connecting multiple scale features output by the top-down network unit to a unified representation.

[0075] The scale aggregation unit refers to a structure for integrating features of different scales, which can be implemented by a network module with a multi-level cascade structure. This module forms a unified feature representation by layer-by-layer fusion of feature maps of different resolutions.

[0076] The top-down structure feature pyramid network unit refers to a network architecture that starts with high-resolution features and gradually samples down and fuses with low-level features. Cross-level feature transmission can be achieved through lateral connection and upsampling operations, thereby enhancing the semantic consistency of multi-scale features.

[0077] Specifically, as shown in Figure 4 In the feature reconstruction pyramid model provided in this embodiment, multiple scale features output by the top-down network unit are input to the scale aggregation unit. This unit fuses high-resolution features with low-level features step by step through a top-down hierarchical structure, for example, aligns and superimposes high-level semantic features and bottom-level detail features through upsampling operations. In this process, the lateral connection channel splices or weights the features of different levels, and finally generates a unified feature representation containing multi-scale information. This processing method enables the spatial position and size variation of the target under different viewing angles to be captured more comprehensively, thereby improving the robustness of detection and tracking.

[0078] This solution introduces a top-down feature pyramid network unit, which dynamically adjusts the contribution of features at different scales, enhancing semantic consistency while preserving detailed information, thereby more accurately eliminating redundant information across views. Through this technical solution, this application addresses the problem of inconsistent representation of features at different scales in multi-view scenarios. By enhancing the spatial alignment capability of cross-view features through a hierarchical feature fusion mechanism, the stability and accuracy of object detection and tracking results are significantly improved in complex occlusion or viewpoint changing scenarios.

[0079] In some embodiments, this application further proposes a processing method for the feature reconstruction unit, including: performing convolution processing on the image features output by each layer of the bottom-up network unit; combining the convolutional image features with two weights generated by adaptive gating logic to obtain two sets of weighted image features; performing cross-reconstruction processing on the two sets of weighted image features to obtain reconstructed features; splitting the reconstructed features into two sets of input features according to the channel dimension and performing convolution processing on each set; performing feature fusion on the processed features and calculating the channel weight values ​​of each channel dimension based on the fused features; obtaining channel-weighted features by multiplying the channel weight values ​​with the reconstructed features; and performing convolution and activation processing on the channel-weighted features in sequence to obtain multi-view aggregated features.

[0080] The adaptive gating logic refers to a mechanism that generates two weights through normalization and the Sigmoid function. Specifically, it can be implemented by combining group normalization with the Sigmoid activation function to dynamically adjust the feature weights of different spatial regions. Cross-reconstruction processing refers to the interactive fusion of two sets of weighted image features, which can be implemented through element-wise multiplication or addition to enhance the feature representation of key regions. Channel splitting refers to the operation of dividing the reconstructed features into two parts along the channel dimension, which can be implemented by equal or proportional partitioning to process different feature subsets in parallel. Channel weight values ​​are the importance coefficients of each channel calculated based on the fused features. This can be implemented by combining global average pooling with convolutional layers to highlight the feature information of key channels.

[0081] Specifically, such as Figure 5 As shown, the image features after convolution are first subjected to group normalization. The normalized features are multiplied by the normalized weights and then mapped to a binary interval using the Sigmoid function, generating two weights. The image features are then multiplied by the two weights to obtain two sets of weighted image features. These two sets of features are then fused through cross-reconstruction to generate the reconstructed feature X. sThe reconstructed feature is split into a first input feature and a second input feature along the channel dimension, the first input feature is summed after being extracted by grouped convolution and point-wise convolution, and the second input feature is extracted by point-wise convolution and spliced with the original feature. The fused intermediate feature is processed by global average pooling, convolution and nonlinear activation function to generate channel weight values, and finally the multi-view aggregation feature X' is generated by channel weighting and convolution activation.

[0082] The scheme dynamically adjusts the spatial weight through adaptive gating logic, combines cross reconstruction and channel refinement processing, eliminates redundant information while retaining key features, solves the interference problem caused by multi-view feature interweaving, can effectively distinguish complementary information and redundant information in different views, optimizes feature representation through spatial enhancement and channel refinement, and improves the accuracy of target detection and tracking in a multi-view scene.

[0083] Further, the above embodiment proposes to perform convolution processing on the image features output by each layer of the bottom-up network unit, obtain two groups of weighted image features from the two weight values generated by the adaptive gating logic after combining the convolution-processed image features, and perform cross reconstruction processing on the two groups of weighted image features to obtain the reconstructed feature. The process can specifically include:

[0084] The convolution-processed image features are normalized in groups to obtain normalized features, the normalized features are reweighted according to the normalized correlation weight, the product is mapped to a preset binary interval through a Sigmoid function, two weight values are obtained through adaptive gating logic, two groups of weighted image features are obtained according to the product of the image features and the two weight values, and cross reconstruction processing is performed on the two groups of weighted image features to obtain the reconstructed feature.

[0085] It should be noted that this process corresponds to the adaptive spatial enhancer in Figure 5 , which aims to optimize the feature representation of key regions while minimizing redundant information and enhancing spatial perception. The group normalization processing refers to the normalization operation after grouping the features along the channel dimension, which can be implemented by dividing the features into a preset number of groups and calculating the mean and variance of each group to eliminate the difference in feature distribution between different views. The normalized correlation weight refers to a learnable parameter matrix associated with the normalized feature, which can be generated through a fully connected layer or a convolution layer to adjust the feature response strength of different spatial positions. The Sigmoid function refers to an activation function that compresses the input value to the 0-1 interval. The adaptive gating logic refers to a mechanism for dynamically generating weights based on input features, which can be constructed through a multi-layer perception or a gated recurrent unit to capture the nonlinear relationship between features.

[0086] Specifically, as shown in Figure 6As shown, first, the relative importance of different features is evaluated by using the scale factor in Group Normalization (GN), and the normalized feature X norm The specific expression is as follows:

[0087] (3)

[0088] wherein, and are the mean and standard deviation of the input feature X, is a positive number to ensure numerical stability, and are trainable affine transformation parameters. The normalized correlation weight representing the relative importance of different features can be calculated as follows:

[0089] (4)

[0090] where i, j are channel indices, C is the number of channels, is the channel weight, is the trainable affine transformation parameter in Group Normalization.

[0091] Next, the normalized feature X norm is reweighted by multiplying and mapped to the range [0, 1] through the Sigmoid function. An adaptive gating mechanism with an initial threshold of 0.5 can be used to generate two different weights: W1 and W2, corresponding to values above and below the threshold, respectively, which can be represented as:

[0092] (5)

[0093] The weighted features X1 and X2 are obtained by applying weights W1 and W2 to the original input feature X. Specifically, X1 contains most of the spatial information, while X2 contains relatively less information. To improve information utilization, cross-reconstruction is performed on the two features, and each feature is divided into two parts, which are merged by cross-adding. After reconstruction, the two features are concatenated to restore the original size. At the same time, a nonlinear activation function is used to generate X s to enrich the feature representation.

[0094] The scheme realizes dynamic modeling of feature space distribution through group normalization and an adaptive double weight generation mechanism. For example, in a crowded scene, the mechanism can automatically suppress the interference features of the background area while enhancing the feature response intensity of the target subject under different perspectives, effectively solving the problem of spatial information redundancy in the multi-view feature aggregation process and enhancing the spatial discrimination ability of feature representation. In the cross-camera target tracking scene, the scheme can accurately extract the common features of the target under different perspectives. For example, for a partially occluded target, the double weight mechanism can be used to retain the feature information of the visible parts, thereby improving the accuracy of cross-view target association.

[0095] Further, the above embodiment proposes to split the reconstructed features into two groups of input features according to the channel dimension, and perform convolution processing on the two groups of input features respectively. After feature fusion of the processed features, the channel weight values of each channel dimension are calculated based on the fused features, and the channel weighted features are obtained according to the product of the channel weight values and the reconstructed features. The process can specifically include:

[0096] The reconstructed features are split into two groups of input features according to the channel dimension, and the two groups of input features are processed by convolution respectively. Then, the processed features are fused, and the channel weight values of each channel dimension are calculated based on the fused features. The process includes: splitting the reconstructed features into first input features and second input features according to the channel dimension; extracting features from the first input features by group convolution and point-by-point convolution respectively, and then summing the features obtained by the two convolution processes to obtain first intermediate features; extracting features from the second input features by point-by-point convolution, and then concatenating the extracted features with the second input features to obtain second intermediate features; fusing the first intermediate features and the second intermediate features by global average pooling processing, and then determining the channel weight values corresponding to each channel dimension based on the fused intermediate features by convolution and nonlinear activation function processing.

[0097] It should be noted that this process corresponds to the channel refinement module in Figure 5 , which aims to enhance the interconnection between channels while preserving key channel features. The group convolution refers to dividing the channels of the input feature map into several groups and performing convolution operation on each group. Specifically, the channel number can be evenly divided into four or eight groups to reduce the computational complexity and enhance the local feature extraction capability. The point-by-point convolution refers to the convolution operation using a 1x1 convolution kernel for channel dimension transformation. Specifically, the output channel number can be adjusted to realize cross-channel information interaction. The global average pooling refers to the average compression of the spatial dimension of the feature map to generate a channel description vector. Specifically, the height and width dimensions of the feature map can be averaged to realize the operation, which can capture global statistical information.

[0098] Specifically, as shown in Figure 7As shown, first, the input feature X s is divided into two equal parts in the channel dimension for efficient feature extraction. To improve computational efficiency, the feature channels are further compressed to obtain X s1 and X s2 . Specifically, X s1 is first extracted using GWConv (Groupwise Convolution). Subsequently, the extracted feature is combined with the feature obtained from Pointwise Convolution (PWConv) to obtain the first intermediate feature Y1, which is expressed as follows:

[0099] (6)

[0100] For X s2 , point-wise features are extracted using PWConv and then concatenated with X s2 to obtain a more comprehensive feature representation, i.e., the second intermediate feature Y2, without additional computational cost, which is expressed as follows:

[0101] (7)

[0102] where denotes the concatenation operation.

[0103] Next, feature fusion is performed. Specifically, Y1 and Y2 are merged to obtain Y, and a global average pooling layer is used to extract global information to obtain compressed channel features , which is expressed as follows:

[0104] (8)

[0105] where denotes the compressed channel features, Pooling denotes the global average pooling layer, and H and W correspond to the spatial resolution of the input image.

[0106] The next step is to learn the channel dependency using compressed information. First, a convolution layer and a nonlinear activation function are applied to determine the weight of each channel to capture the relationship between channels. The convolution layer reduces the dimension of the channel to minimize computational complexity. Then the output is processed by the ReLU activation function to enhance the feature representation. The resulting feature is then passed through a second convolution layer to restore the channel size to match the original input X s . Finally, the output is constrained to the range [0, 1] using the Sigmoid function, i.e.:

[0107] (9)

[0108] where represents a ReLU function, represents a Sigmoid function, represents the weight of each channel.

[0109] Then the weight vector s is applied to the input feature X s . For each channel, the weight is multiplied by the corresponding channel feature, and then the weighted features are aggregated to obtain the output feature X c of the channel refinement module, represented as:

[0110] (10)

[0111] As can be understood, as shown in Figure 5 , the output feature X c of the channel refinement module is further processed by two layers of convolution and activation functions, etc., to obtain the feature quantity output by the feature reconstruction unit to the top-down network unit, so as to obtain the multi-view aggregated feature finally output by the encoder module through subsequent processing such as the top-down network unit.

[0112] The scheme can more effectively enhance the interconnection relationship between channels by combining the processing of grouped convolution and point-by-point convolution, and combining the global statistical feature fusion mechanism, while retaining key channel features that are discriminative for target recognition. The scheme solves the problems of channel feature redundancy and key information loss in the multi-view feature aggregation process, and realizes fine selection of channel features through multi-dimensional convolution combination and global statistical fusion mechanism, so that the subsequent feature aggregation process can accurately focus on the channel features that are discriminative, thereby improving the robustness of multi-view target detection and tracking.

[0113] Further, after the encoder module outputs the multi-view aggregated feature, and before projection processing based on the multi-view aggregated feature, the scheme can further include: performing feature extraction on the multi-view aggregated feature through separated convolution processing and multiple dilation rate parameters to obtain multi-scale features, and then aggregating the multi-scale features and the multi-view aggregated feature through a 1*1 convolution layer to obtain multi-scale aggregated features; performing adaptive pooling processing on the high element feature and the wide element feature according to the image space resolution parameters of the multi-view aggregated feature to obtain corresponding pooled features, and then generating an attention score according to the product of the two groups of pooled features combined with a Sigmoid function; multiplying the multi-scale aggregated features and the attention score to obtain multi-scale aggregated weighted features, and then adding the multi-scale aggregated weighted features and the original multi-scale aggregated features to obtain the final multi-view aggregated feature.

[0114] Separate convolution processing refers to decomposing standard convolution into depthwise convolution and pointwise convolution, which can be implemented using a depthwise separable convolution structure to improve computational efficiency by reducing the number of parameters. The dilation parameter controls the sampling interval of the convolution kernel; for example, a combination of dilation rates of 1, 3, and 5 can be used to capture features within different receptive fields. Adaptive pooling dynamically adjusts the pooling window size based on the spatial dimension of the input features; for example, global average pooling or max pooling can be used to extract key information in the spatial dimension. Attention scores are weight coefficients calculated based on the correlation between features; for example, they can be generated using channel attention or spatial attention mechanisms to dynamically weight features at multiple scales.

[0115] Specifically, such as Figure 8 As shown, using DWConv with different dilation rates enhances the extraction of detail and contextual features, effectively expanding the receptive field without introducing additional parameters. These multi-scale features are then aggregated with the input multi-view features X. a Combine, and then through The convolutional layers perform aggregation. Then, we can obtain the multi-scale features F, as shown below:

[0116] (11)

[0117] Where d represents different expansion rates.

[0118] Next, the attention score is calculated. Adaptive pooling layers are applied to both the height and width. Then, the two pooling features are multiplied and processed using a sigmoid function to generate the attention score F. att ,Right now:

[0119] (12)

[0120] Where Sigmoid is the activation function, H and W are the spatial resolution parameters of the corresponding input image, representing the height and width of the image respectively, and x(i,j) corresponds to the element in the i-th row and j-th column of the input image.

[0121] Next, the attention score is multiplied element-wise with the multi-scale features, and the original feature information is preserved through residual connections to form an enhanced multi-view aggregated feature that generates the final output. This is to emphasize key feature areas.

[0122] This approach utilizes depthwise separable convolutions with different dilation rates and attention mechanisms to extract local and global features, effectively capturing global contextual information, enhancing feature representation, and thus improving the stability of multi-target tracking.

[0123] The above is a detailed description of an embodiment of the multi-view target tracking detection method provided in the application. The following is a detailed description of an embodiment of a multi-view target tracking detection device provided in the application.

[0124] Referring to Figure 9 The multi-view target tracking detection device provided in the application comprises:

[0125] The multi-view data acquisition unit 201 is configured to acquire multi-view data to be detected.

[0126] The multi-view feature processing unit 202 is configured to input the multi-view data to a preset encoder module, so as to obtain multi-view aggregated features corresponding to the multi-view data through processing of the encoder module. The encoder module comprises a preset feature reconstruction pyramid model, which comprises a bottom-up network unit, a top-down network unit, and a feature reconstruction unit. The feature reconstruction unit is arranged on a lateral connection channel between the bottom-up network unit and the top-down network unit, and is configured to eliminate redundant information and extract key features according to feature information output by the bottom-up network unit.

[0127] The target detection unit 203 is configured to sequentially perform projection, aggregation, and decoding processing based on the multi-view aggregated features, so as to obtain a target tracking detection result.

[0128] Specifically, the multi-view data acquisition unit collects image or video data of different angles from multiple cameras and transmits the data to the multi-view feature processing unit. The bottom-up network unit in the encoder module extracts multi-level features from the input data to generate a feature map containing spatial details. The feature reconstruction unit receives the features output by the bottom-up network unit through lateral connection, performs spatial weighting on the features by using adaptive gating logic, and realizes feature screening in the channel dimension by using grouping convolution and channel attention mechanism, and finally outputs key features after eliminating redundancy. The top-down network unit up-samples the reconstructed features and fuses them with the bottom-layer features to form multi-scale aggregated features. The target detection unit reduces the dimension of the aggregated features through projection operation, and generates position and category information of the target through a decoder to complete the tracking detection task.

[0129] As Figure 10 shown, the application further provides a multi-view target tracking detection terminal, which comprises a memory 33 and a processor 31, wherein the memory 33 and the processor 31 are connected through a communication bus 34.

[0130] The memory 33 is configured to store program code, and the program code is configured to implement a multi-view target tracking detection method provided in the above embodiments.

[0131] The processor 31 is configured to read and execute the program code.

[0132] The memory refers to a hardware device for storing program codes, which can be implemented by a flash memory, a solid state disk or a disk memory, and functions to provide instructions and data basis required by algorithm execution for the running of the processor. The processor refers to an operation unit for executing program codes, which can be implemented by a central processing unit, a graphics processing unit or a special neural network acceleration chip, and functions to quickly complete feature extraction, aggregation and decoding operations through parallel computing capability, thereby reducing the computing delay of multi-view data processing. The program codes refer to a set of computer instructions including an encoder module, a feature reconstruction pyramid model and decoding logic, which can be implemented by combining model files generated by a deep learning framework and self-defined algorithm modules, and functions to convert multi-view data into target tracking detection results while reducing redundant information interference through optimization of the feature aggregation process.

[0133] Specifically, the terminal controls the processor to execute the following operations through program codes stored in the memory: first, multi-view data from multiple cameras are acquired, and then the data are input into an encoder module to generate multi-view aggregated features, wherein the encoder module adopts a pyramid structure including a bottom-up network, a top-down network and a transverse feature reconstruction unit, for eliminating redundant features and enhancing key region representation. Next, the processor performs multi-scale context enhancement processing on the aggregated features, separates convolution and fuses local and global features through an attention mechanism, and finally outputs target position and motion trajectory through decoding processing. In this process, the coordinated work of the memory and the processor enables complex algorithms to run in real time on terminal devices, for example, achieving a processing speed of more than 30 frames per second in embedded devices.

[0134] In some specific embodiments, the memory can be configured as a non-volatile memory that dynamically allocates storage space, for example, adjusting the cache space of model parameters according to the real-time computing requirements of the processor. The processor can integrate a multi-core architecture, for example, adopting a big-little core design to process feature extraction and trajectory prediction tasks respectively, further optimizing the utilization rate of computing resources.

[0135] The fourth aspect of the present application provides a computer readable storage medium, which has program codes saved therein, and the program codes are used to be read and executed by a processor to implement a multi-view target tracking detection method provided by the above-described embodiments.

[0136] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the terminal, device and unit described above can refer to the corresponding processes in the foregoing method embodiments, which will not be described herein.

[0137] In several embodiments provided in the present application, it should be understood that the disclosed terminal, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. For example, the division of the units is merely a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0138] The terms "first", "second", "third", "fourth", and the like used in the description and the accompanying drawings of the application (if any) are used to distinguish similar objects, and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein, for example, can be implemented in an order other than that illustrated or described herein. In addition, the terms "comprise" and "have" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a list of steps or units as an example, does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to such processes, methods, products, or apparatus.

[0139] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that there are three cases of only A, only B, and A and B at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions means any combination of these items, including single item or any combination of multiple items. For example, at least one of a, b or c, can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0140] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or can be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0141] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0142] If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0143] The above, the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-view object tracking detection method, characterized by, The method comprises the following steps: obtaining multi-view data to be detected; inputting the multi-view data into a preset encoder module to obtain multi-view aggregated features corresponding to the multi-view data through processing of the encoder module, wherein the encoder module comprises a preset feature reconstruction pyramid model, and the feature reconstruction pyramid model comprises a bottom-up network unit, a top-down network unit and a feature reconstruction unit, wherein the feature reconstruction unit is arranged on a lateral connection channel between the bottom-up network unit and the top-down network unit; based on the multi-view aggregated features, sequentially performing projection, aggregation and decoding processing to obtain a target tracking detection result; wherein the processing mode of the feature reconstruction unit comprises: performing convolution processing on image features output by each layer of the bottom-up network unit; combining the convolution-processed image features with two weights generated by adaptive gating logic to obtain two groups of weighted image features, and performing cross-reconstruction processing on the two groups of weighted image features to obtain reconstruction features; according to the channel dimension, the reconstruction features are split into two groups of input features, and convolution processing is performed on the two groups of input features respectively, and then feature fusion is performed on the processed features, and the channel weight values of each channel dimension are calculated based on the fused features; according to the product of the channel weight values and the reconstruction features, channel weighted features are obtained; after the channel weighted features are sequentially subjected to convolution and activation processing, the channel weighted features are input into the top-down network unit to obtain multi-view aggregated features.

2. The multi-view object tracking and detection method of claim 1, wherein, combining the convolution-processed image features with two weights generated by adaptive gating logic to obtain two groups of weighted image features comprises: performing group normalization processing on the convolution-processed image features to obtain normalized features; according to the normalized features multiplied by the normalized correlation weight, the product is mapped to a preset binary interval through a Sigmoid function, and then two weights are obtained through adaptive gating logic; according to the product of the image features and the two weights, two groups of weighted image features are obtained respectively.

3. The multi-view object tracking and detection method of claim 1, wherein, according to the channel dimension, the reconstruction features are split into two groups of input features, and convolution processing is performed on the two groups of input features respectively, and then feature fusion is performed on the processed features, and the channel weight values of each channel dimension are calculated based on the fused features, which comprises: according to the channel dimension, the reconstruction features are split into first input features and second input features; feature extraction is performed on the first input features through grouped convolution and point-by-point convolution respectively, and then the features obtained through the two kinds of convolution processing are summed to obtain first intermediate features; feature extraction is performed on the second input features through point-by-point convolution, and then the extracted features are spliced with the second input features to obtain second intermediate features; the first intermediate features and the second intermediate features are fused through a global average pooling processing mode to obtain compressed channel features, and then based on the compressed channel features, the channel weight values corresponding to each channel dimension are determined through convolution and nonlinear activation function processing.

4. The multi-view object tracking and detection method of claim 1, wherein, The projection processing based on the multi-view aggregated feature further comprises: Through separate convolution processing and a plurality of preset dilation rate parameters, the multi-view aggregated feature is extracted to obtain multi-scale features, and then the multi-scale features are aggregated with the multi-view aggregated feature through a preset 1*1 convolution layer to obtain multi-scale aggregated features; According to the image spatial resolution parameters of the multi-view aggregated feature, adaptive pooling processing is respectively performed on high element features and wide element features in the image spatial resolution parameters to obtain corresponding pooled features, and then attention scores are generated according to the product of the two groups of pooled features and in combination with a Sigmoid function; The multi-scale aggregated features and the attention scores are multiplied to obtain multi-scale aggregated weighted features, and then the multi-scale aggregated weighted features and the original multi-scale aggregated features are added to obtain final multi-view aggregated features.

5. The multi-view object tracking and detection method of claim 1, wherein, The feature reconstruction pyramid model further comprises a scale aggregation unit configured to connect the features of multiple scales output by the top-down network unit to a unified representation.

6. The multi-view object tracking and detection method of claim 5, wherein, The scale aggregation unit is specifically a feature pyramid network unit in a top-down structure.

7. A multi-view object tracking and detection apparatus, characterized by comprising: The method comprises: a multi-view data acquisition unit configured to acquire multi-view data to be detected; a multi-view feature processing unit configured to input the multi-view data to a preset encoder module to obtain multi-view aggregated features corresponding to the multi-view data through processing of the encoder module, wherein the encoder module comprises a preset feature reconstruction pyramid model, and the feature reconstruction pyramid model comprises a bottom-up network unit, a top-down network unit, and a feature reconstruction unit, wherein the feature reconstruction unit is arranged on a transverse connection channel between the bottom-up network unit and the top-down network unit; a target detection unit configured to sequentially perform projection, aggregation, and decoding processing based on the multi-view aggregated features to obtain a target tracking detection result. The processing mode of the feature reconstruction unit comprises: performing convolution processing on image features output by each layer of the bottom-up network unit; combining two weight values generated by adaptive gating logic with the image features after convolution processing to obtain two groups of weighted image features, and performing cross reconstruction processing on the two groups of weighted image features to obtain reconstruction features; splitting the reconstruction features into two groups of input features according to channel dimensions, performing convolution processing on the two groups of input features respectively, performing feature fusion on the processed features, and calculating channel weight values of each channel dimension based on the fused features; obtaining channel weighted features according to the product of the channel weight values and the reconstruction features; inputting the channel weighted features to the top-down network unit after sequential convolution and activation processing to obtain multi-view aggregated features.

8. A multi-view target tracking detection terminal, characterized by, The method comprises: a memory and a processor; the memory is configured to store program code for implementing the multi-view target tracking detection method according to any one of claims 1 to 6; the processor is configured to read and execute the program code.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program codes, which are read and executed by the processor to implement the multi-view target tracking detection method according to any one of claims 1 to 6.

Images