A multi-view target detection and tracking method and apparatus based on adaptive fusion and temporal correlation

Abstract

This invention discloses a multi-view target detection and tracking method and apparatus based on adaptive fusion and temporal correlation, relating to the field of visual tracking technology. The method includes: simultaneously acquiring input images of the current frame from multiple camera devices; performing multi-scale feature extraction and adaptive multi-scale feature fusion on each input image to determine the scale fusion features; mapping the scale fusion features to a bird's-eye view space and then stitching them together to generate current bird's-eye view information; inputting the current bird's-eye view information and historical bird's-eye view information from adjacent historical frames into an improved bird's-eye view temporal Transformer to output bird's-eye view features; and decoding the bird's-eye view features and determining the bird's-eye view detection and tracking result using a detection head and a tracking head. Adaptive multi-scale feature fusion reduces artifacts while enhancing feature detail expression. The use of a bird's-eye view temporal Transformer for pixel-level matching between adjacent frames enhances fine-grained motion estimation, contributing to improved detection and tracking accuracy.

Description

Technical Field

[0001] This invention relates to the field of visual tracking technology, and in particular to a multi-view target detection and tracking method and apparatus based on adaptive fusion and temporal correlation. Background Technology

[0002] Visual target tracking aims to continuously determine the position of targets in a video sequence. In practical applications, although single-camera multi-target tracking technology has made significant progress, ... Figure 1 As shown, if the target object is obstructed or interfered with by objects or environmental factors, it is easy to cause the target to be lost and the identity to be switched.

[0003] To alleviate the limitations of shading, such as Figure 2 As shown, multi-camera systems achieve continuous tracking by utilizing complementary information from overlapping viewpoints. Traditional multi-view target detection and tracking methods mainly rely on strategies of first-view-then-temporal correlation or first-temporal-then-view correlation. First-view-then-temporal correlation (e.g.) Figure 3 (As shown) refers to obtaining the 2D detection for each view, associating the detections of different views within the same time step across views, and associating the detection results across time steps, first time-series then view association (e.g.) Figure 4 (As shown) refers to cross-view matching between views after temporal correlation based on detection results within each view; however, this is easily affected by inaccuracies introduced during the detection phase. In recent years, some methods have attempted to project features onto a bird's-eye view to directly identify targets (such as...). Figure 5 (As shown) To mitigate the impact of error propagation, feature extraction mainly relies on dilated convolution or multi-scale feature concatenation. However, dilated convolution expands the receptive field at the expense of local detail information, leading to a decrease in the accuracy of edge feature extraction. Furthermore, multi-scale feature concatenation cannot fully utilize the complementary information between features at different levels, which may introduce artifacts. In addition, data association is closely related to achieving accurate cross-frame target matching in multi-view object detection and tracking. However, distance-based association methods are usually used for cross-frame target matching, which may be inaccurate when there are many targets or a long time frame.

[0004] Therefore, existing multi-view target detection and tracking methods need to improve their detection and tracking accuracy. Summary of the Invention

[0005] This invention provides a multi-view target detection and tracking method and apparatus based on adaptive fusion and temporal correlation, which is used to solve the technical problem that existing multi-view target detection and tracking methods need to improve detection and tracking accuracy.

[0006] The first aspect of this invention provides a multi-view target detection and tracking method based on adaptive fusion and temporal correlation, comprising:

[0007] Synchronously acquire input images of current frames of multiple camera devices;

[0008] Respectively perform multi-scale feature extraction and adaptive multi-scale feature fusion on each of the input images to determine scale fusion features of each of the input images;

[0009] Map each of the scale fusion features to an aerial view space and then splice them to generate current aerial view information;

[0010] Input the current aerial view information and historical aerial view information of adjacent historical frames into an improved aerial view time sequence Transformer to perform motion estimation and output aerial view features;

[0011] After decoding the aerial view features, determine aerial view detection and tracking results through a detection head and a tracking head.

[0012] Optionally, the determination process of the scale fusion features comprises:

[0013] Input any of the input images into a backbone network to perform multi-scale feature extraction and determine three layers of scale features;

[0014] After upsampling and feature reorganization of the scale features of the bottom layer to determine first refined features, input the first refined features and the scale features of the middle layer into a fusion block to perform adaptive multi-scale feature fusion and construct first fusion features;

[0015] After upsampling and feature reorganization of the first fusion features to determine second refined features, input the second refined features and the scale features of the top layer into the fusion block to perform adaptive multi-scale feature fusion and generate second fusion features;

[0016] Perform feature reorganization on the second fusion features to output scale fusion features;

[0017] The feature reorganization process comprises using a cascaded 3x3 convolution layer and a 1x1 convolution layer to perform feature reorganization.

[0018] Optionally, the fusion block comprises an adaptive global pooling layer and a sigmoid activation function, and the processing process of the fusion block comprises:

[0019] After splicing the input refined features and the scale features, perform pooling on the refined features through the adaptive global pooling layer to generate compressed features;

[0020] Uniformly divide the compressed features into first sub-compressed features and second sub-compressed features;

[0021] Use the sigmoid activation function to respectively map the first sub-compressed features and the second sub-compressed features to generate corresponding first channel importance vectors and second channel importance vectors;

[0022] After multiplying the input refined feature and the first channel importance vector, and multiplying the input scale feature and the second channel importance vector respectively, a fusion output fusion feature is obtained.

[0023] Optionally, the improved bird's eye view time series Transformer comprises an improved multi-head cross attention layer, a residual regularization layer and a fully connected feedforward network layer.

[0024] The historical bird's eye view information of the adjacent historical frame of the current frame is obtained.

[0025] In the improved multi-head cross attention layer, the current bird's eye view information is linearly projected into a query matrix through each head of the improved multi-head cross attention layer, and the historical bird's eye view information is projected into a key matrix and a value matrix respectively.

[0026] The attention feature is output after the head feature of each head is output by subtracting the associated query matrix and key matrix, performing softmax operation and multiplying element by element with the corresponding value matrix.

[0027] The historical bird's eye view information and the attention feature are subjected to feature addition and regularization processing through the residual regularization layer to determine the intermediate feature.

[0028] The intermediate feature is subjected to feature processing through the cascaded fully connected feedforward network layer and the residual regularization layer to output the bird's eye view feature.

[0029] The second aspect of the present application provides a multi-view target detection and tracking device based on adaptive fusion and time series association, comprising:

[0030] An image acquisition module is configured to synchronously acquire input images of current frames of a plurality of camera devices.

[0031] A feature extraction module is configured to perform multi-scale feature extraction and adaptive multi-scale feature fusion on each input image to determine scale fusion features of each input image.

[0032] A feature projection module is configured to map each scale fusion feature to a bird's eye view space and then perform splicing to generate current bird's eye view information.

[0033] A motion correlation module is configured to input an improved bird's eye view time series Transformer with the current bird's eye view information and historical bird's eye view information of adjacent historical frames to perform motion estimation and output bird's eye view features.

[0034] The feature decoding module is configured to decode the aerial view features, and determine aerial view detection and tracking results by means of the detection head and the tracking head.

[0035] Optionally, the determination process of the scale fusion features comprises:

[0036] Any of the input images is input into a backbone network to extract multi-scale features and determine three layers of scale features.

[0037] After upsampling and feature reorganization of the scale features of the bottom layer, first refined features are determined, and the scale features of the middle layer are input into a fusion block to perform adaptive multi-scale feature fusion, thereby constructing first fusion features.

[0038] The first fusion features are upsampled and reorganized to determine second refined features, and the scale features of the top layer are input into a fusion block to perform adaptive multi-scale feature fusion, thereby generating second fusion features.

[0039] The second fusion features are reorganized to output scale fusion features.

[0040] The feature reorganization process comprises using cascaded 3x3 convolution layers and 1x1 convolution layers to reorganize features.

[0041] Optionally, the fusion block comprises an adaptive global pooling layer and a sigmoid activation function, and the processing process of the fusion block comprises:

[0042] After the input refined features and the scale features are spliced, the adaptive global pooling layer is used to perform pooling to generate compressed features.

[0043] The compressed features are evenly divided into first sub-compressed features and second sub-compressed features.

[0044] The sigmoid activation function is used to respectively map the first sub-compressed features and the second sub-compressed features to generate corresponding first channel importance vectors and second channel importance vectors.

[0045] The input refined features and the first channel importance vectors are respectively multiplied element by element, and the input scale features and the second channel importance vectors are respectively multiplied element by element, thereby outputting fusion features.

[0046] A computer device provided in a third aspect of the present application comprises a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the multi-view target detection and tracking method based on adaptive fusion and time sequence association according to any one of the above aspects.

[0047] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed to realize the multi-view target detection tracking method based on adaptive fusion and time sequence association.

[0048] The fifth aspect of the present application provides a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to realize the multi-view target detection tracking method based on adaptive fusion and time sequence association.

[0049] From the above technical solutions, the present application has the following advantages:

[0050] The multi-view target detection tracking method based on adaptive fusion and time sequence association provided by the present application comprises the following steps: synchronously acquiring input images of current frames of a plurality of camera devices; performing multi-scale feature extraction and adaptive multi-scale feature fusion on each input image respectively to determine scale fusion features of the input images; splicing the scale fusion features after mapping each scale fusion feature to a bird's eye view space to generate current bird's eye view information; inputting the current bird's eye view information and historical bird's eye view information of adjacent historical frames into an improved bird's eye view time sequence Transformer for motion estimation to output bird's eye view features; and determining a bird's eye view detection tracking result through a detection head and a tracking head after decoding the bird's eye view features. Based on the above scheme, the adaptive multi-scale feature fusion strategy is designed to reduce the artifacts caused by feature extraction while enhancing the feature detail expression, the improved bird's eye view time sequence Transformer is used in the bird's eye view angle to perform pixel-level matching between adjacent frames in time sequence to enhance fine-grained motion estimation, which helps to improve the detection tracking accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0052] Figure 1 The existing single-camera identity switching schematic diagram provided for the embodiments of the present application;

[0053] Figure 2 The existing multi-camera identity switching schematic diagram provided for the embodiments of the present application;

[0054] Figure 3 The architecture schematic diagram of the existing view-first time sequence association method provided for the embodiments of the present application;

[0055] Figure 4 The architecture schematic diagram of the prior view correlation method before time sequence provided for the embodiment of the present application is shown in the figure.

[0056] Figure 5 The architecture schematic diagram of the prior bird's eye view detection method provided for the embodiment of the present application is shown in the figure.

[0057] Figure 6 The step flow chart of the multi-view target detection tracking method based on adaptive fusion and time sequence correlation provided for the embodiment of the present application is shown in the figure.

[0058] Figure 7 The flow chart of the multi-view target detection tracking method based on adaptive fusion and time sequence correlation provided for the embodiment of the present application is shown in the figure.

[0059] Figure 8 The architecture schematic diagram of the multi-scale feature extraction and adaptive multi-scale feature fusion provided for the embodiment of the present application is shown in the figure.

[0060] Figure 9 The architecture schematic diagram of the improved bird's eye view time sequence Transformer provided for the embodiment of the present application is shown in the figure.

[0061] Figure 10 The bird's eye view detection result schematic diagram provided for the embodiment of the present application is shown in the figure.

[0062] Figure 11 The bird's eye view tracking result schematic diagram provided for the embodiment of the present application is shown in the figure.

[0063] Figure 12 The structure block diagram of the multi-view target detection tracking device based on adaptive fusion and time sequence correlation provided for the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0064] The multi-view target detection tracking method and device based on adaptive fusion and time sequence correlation provided by the embodiment of the present application are used to solve the technical problem that the existing multi-view target detection tracking method needs to improve the detection tracking accuracy.

[0065] In order to make the inventive purpose, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the embodiments described below are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. It should be understood that the terms "first", "second", etc. are used to distinguish similar objects, not necessarily to describe a specific order or sequence.

[0066] Please refer toFigure 6 and Figure 7 The embodiment of the present application provides a multi-view target detection and tracking method based on adaptive fusion and timing association, which can be applied to video monitoring, pedestrian behavior analysis, traffic monitoring and other static monitoring scenes of camera equipment, and comprises the following steps:

[0067] Step 101, synchronously acquiring input images of current frames of multiple camera equipment.

[0068] Camera equipment refers to equipment for capturing images or videos, such as a camera.

[0069] Current frame refers to a frame being displayed, processed or analyzed.

[0070] Input image refers to an image captured by camera equipment.

[0071] It should be noted that at a certain moment, the input images captured by the current frame of the multiple camera equipment are synchronously read , is an index of the camera equipment, wherein is the number of image channels, is the image height, is the image width, is a real space, and each camera equipment can provide image information of an observation object from different perspectives.

[0072] Step 102, respectively performing multi-scale feature extraction and adaptive multi-scale feature fusion on each input image to determine scale fusion features of the input image.

[0073] It should be noted that for any input image, after multi-scale feature extraction, the adaptive multi-scale feature fusion strategy designed in the embodiment is used for feature aggregation, and the obtained scale fusion features retain the detail information.

[0074] In one specific embodiment of the embodiment, as shown in Figure 8 , the determination process of the scale fusion features comprises the following steps.

[0075] Any input image is input into a backbone network to perform multi-scale feature extraction and determine three layers of scale features.

[0076] After upsampling and feature recombination of the scale features of the bottom layer to determine the first refined features, the first refined features are input into a fusion block with the scale features of the middle layer to perform adaptive multi-scale feature fusion and construct first fusion features.

[0077] After upsampling and feature recombination of the first fusion features to determine the second refined features, the second refined features are input into a fusion block with the scale features of the top layer to perform adaptive multi-scale feature fusion and generate second fusion features.

[0078] perform feature reorganization on the second fusion feature to output a scale fusion feature;

[0079] The process of feature reorganization includes using a cascaded 3x3 convolution layer and a 1x1 convolution layer to perform feature reorganization.

[0080] In one more specific implementation of the embodiment, as shown in FIG. 2, the fusion block includes an adaptive global pooling layer and a sigmoid activation function, and the processing procedure of the fusion block includes: Figure 8

[0081] After the input refined feature and the scale feature are spliced, the adaptive global pooling layer is used to perform pooling to generate compressed features;

[0082] The compressed features are evenly divided into a first sub-compressed feature and a second sub-compressed feature;

[0083] The sigmoid activation function is used to respectively map the first sub-compressed feature and the second sub-compressed feature to generate corresponding first channel importance vectors and second channel importance vectors;

[0084] The input refined feature and the first channel importance vector are respectively multiplied element by element, and the input scale feature and the second channel importance vector are respectively multiplied element by element, and then the fusion output fusion features.

[0085] It should be noted that when determining the scale fusion feature of the input image, multi-scale feature extraction is first performed through the backbone network, that is, feature extraction is performed at different levels to obtain scale features of multiple scales, and the output quantity of the scale features can be set according to the feature extraction requirement. Such a backbone network usually has multiple stages, each stage outputs a scale feature. Exemplarily, a convolutional neural network backbone network can be used. In specific implementation, the embodiment generates scale features through the backbone network , is a feature scale index, and is the number of feature scales, wherein corresponds to a different scale of channel number; more specifically, in the backbone network, the spatial resolution of different scales can be gradually reduced by 4, 8 and 16 times compared to the original spatial resolution , that is, the input image is first down-sampled by four times to obtain the scale feature of the top layer to reduce memory consumption;

[0086] ​Next, the scale features extracted from the backbone network are aggregated, and an adaptive multi-scale feature fusion strategy is proposed in this embodiment, which can fully utilize the complementary information between different levels of features and reduce the generation of artifacts, effectively alleviate the limitations of existing methods, and improve the robustness and discriminability of feature representation. Specifically, the architecture of adaptive multi-scale feature fusion is as shown in Figure 8

[0087] 1) The scale features of the bottom layer are up-sampled by bilinear interpolation, so as to be spatially aligned with Specifically, the up-sampling multiple corresponds to the down-sampling multiple in the backbone network, such as 2 times up-sampling, secondly, the up-sampled features are refined by a 3x3 convolution layer to eliminate the artifacts introduced by up-sampling, and then a 1x1 convolution layer is used to reduce the channel dimension and realize cross-channel information interaction to obtain the first refined feature , wherein is a convolution layer with a kernel size of , and is up-sampling.

[0088] 2) Then, an adaptive fusion strategy is introduced in the fusion block to combine the scale features of the intermediate layer and , instead of simply adding or splicing, specifically:

[0089] First, the two features are spliced along the channel dimension, and then an adaptive global pooling layer is applied to collect spatial features, and the calculation process is as follows:

[0090]

[0091] , wherein is the compressed feature, is the adaptive global pooling layer, is the feature after splicing the refined feature and the scale feature, is the height index of the pixel value, is the width index of the pixel value.

[0092] Next, the compressed feature is divided into two identical parts to obtain the first sub-compressed feature and the second sub-compressed feature , and each sub-compressed feature obtains a channel-wise importance vector through a sigmoid activation function, that is:

[0093]

[0094]

[0095] , wherein​​​ is a first channel importance vector, is a second channel importance vector, is a natural constant;

[0096] Furthermore, the fusion feature is obtained by weighted fusion: , wherein, is a first fusion feature, is an element-wise multiplication;

[0097] 3) Similar to the determination process of the first fusion feature, the first fusion feature is continuously up-sampled and feature-reorganized, and then fused with the scale feature of the previous layer in the fusion block to output the second fusion feature, and then feature-reorganized after the fusion with the scale feature of the top layer is completed, so as to output the scale fusion feature, which provides rich multi-scale representation for the subsequent processing stage; this adaptive fusion strategy allows dynamic learning of the importance of different scale features, and in order to fuse the scale feature of the top layer, this fusion process can be applied recursively, which can be expressed as: , wherein, is a scale fusion feature, is a convolution layer used for feature reorganization.

[0098] Step 103, map each scale fusion feature to the bird's eye space respectively, and then splice to generate the current bird's eye information.

[0099] It should be noted that each scale fusion feature can be mapped to the bird's eye space by perspective transformation, and then aggregated and spliced to generate the current bird's eye information, so that the spatial correlation across views is performed while generating the bird's eye representation, thereby reducing the influence of error propagation.

[0100] Specifically, the process of perspective transformation can refer to the following:

[0101] First, the conversion between three-dimensional position and two-dimensional image pixel coordinates is as follows:

[0102]

[0103] , wherein, is a scale factor, is a two-dimensional image pixel coordinate, is a perspective transformation matrix, is an intrinsic matrix, is an extrinsic matrix, is a three-dimensional position coordinate, is a perspective transformation matrix element;

[0104] Then, all pixels are projected to the ground plane Therefore, the perspective transformation can be expressed as:

[0105]

[0106] where, is the ground plane perspective transformation matrix, i.e. does not include the third column; subsequently, the equation is used to project the features onto a quantized ground plane grid each grid location represents a 10cm x 10cm area, the annotated grid is 4x downsampled due to memory limitations.

[0107] Step 104, input the current bird's eye view information and the historical bird's eye view information of the adjacent historical frame into the improved bird's eye view time series Transformer for motion estimation, and output the bird's eye view feature.

[0108] In one specific implementation of the embodiment, as shown in Figure 9 the improved bird's eye view time series Transformer includes an improved multi-head cross attention layer, a residual regularization layer and a fully connected feedforward network layer; step 104 includes the following sub-steps:

[0109] obtain the historical bird's eye view information of the adjacent historical frame of the current frame;

[0110] In the improved multi-head cross attention layer, the current bird's eye view information is linearly projected into a query matrix by each head of the improved multi-head cross attention layer, and the historical bird's eye view information is projected into a key matrix and a value matrix respectively;

[0111] After subtracting the associated query matrix and the key matrix by each head respectively, performing softmax operation and element-wise multiplication with the corresponding value matrix, the head features of each head are output, and then the attention features are output after merging;

[0112] The historical bird's eye view information and the attention features are processed by the residual regularization layer for feature addition and regularization, to determine the intermediate features;

[0113] The intermediate features are processed by the cascaded fully connected feedforward network layer and the residual regularization layer, to output the bird's eye view features.

[0114] It should be noted that the embodiment introduces a new Transformer-based association architecture, called bird's eye view time series Transformer, which includes an improved multi-head cross attention layer, a residual regularization layer and a fully connected feedforward network layer, aiming to enhance fine-grained motion estimation by establishing pixel-level correspondence between adjacent frames in time series. The architecture effectively captures dense feature correspondence using multi-head cross attention mechanism, and simultaneously models explicit motion prior information through relative displacement calculation in bird's eye view space. Specifically:

[0115] In the improved multi-head cross attention layer, a plurality of heads are included, each of which linearly projects the current bird's eye information and the historical bird's eye information of the adjacent frame according to the current bird's eye information and the historical bird's eye information of the adjacent frame: , , , wherein, is a query matrix, is the current bird's eye information, is a frame index, is a learnable query projection matrix, is a key matrix, is the historical bird's eye information, is a learnable key projection matrix, is a value matrix, is a learnable value projection matrix; in order to calculate the attention score of each head, the motion prior is combined by calculating the relative displacement of the corresponding bird's eye coordinates, that is: , is a head index of the improved multi-head cross attention layer, is an attention weight of the i-th head, is a Softmax activation function, is a query matrix of the i-th head, is a key matrix of the i-th head; it can be understood that the Softmax activation function is used to convert a real number vector into a probability distribution, which allows the model to effectively focus on the most relevant information; then, the associated value matrix and the attention weight are multiplied element by element, and the results of multiple heads are combined, that is: , , wherein, is an attention weight of the i-th head, is an improved multi-head cross attention layer, is spliced along the channel dimension; after the output of the improved multi-head cross attention layer is connected with the input residual in the residual regularization layer, the regularization processing is performed, the intermediate feature is determined, and then the intermediate feature is processed by the cascaded fully connected feedforward network layer and the residual regularization layer to output the bird's eye feature; therefore, the processing process of the entire bird's eye time series Transformer can be represented as: , wherein, is a bird's eye feature, is a residual regularization layer, is a fully connected feedforward network layer;

[0116] ​​​​Therefore, the proposed bird's-eye view temporal Transformer directly learns the feature correspondence between the current frame and adjacent historical frames. At the same time, pedestrians move slowly, the movement distance between consecutive frames is short, and the overlapping view of the camera is large. By directly using the relationship between two adjacent frames for motion estimation, the misalignment problem caused by motion estimation error can be avoided. The introduction of the Transformer mechanism can capture long-distance dependencies and achieve low complexity. Therefore, it can effectively capture temporal dynamics and motion patterns, which helps to perform accurate motion estimation in bird's-eye view-based perception tasks.

[0117] Step 105: After decoding the bird's-eye view features, determine the bird's-eye view detection and tracking results using the detection head and tracking head.

[0118] It should be noted that decoding the bird's-eye view features through a preset decoder can expand the receptive field. Then, the decoded bird's-eye view features can be detected and tracked by a preset detection head and tracking head to obtain bird's-eye view detection and tracking results. It can be understood that the bird's-eye view detection and tracking results can include information such as the detection position and tracking trajectory of the observed object. The decoder can include, but is not limited to, a Transformer decoder, a decoder composed of a cascaded convolutional neural network and a feature pyramid, etc., as long as it can provide a larger receptive field for the bird's-eye view features. The detection head and tracking head can be any model that can convert the bird's-eye view features into specific detection or tracking results. This embodiment does not limit the structure of the decoder, detection head, and tracking head.

[0119] To better illustrate the technical effects of this embodiment, experiments were conducted on two pedestrian benchmark datasets, and the experimental results are as follows: Figure 10 and Figure 11 As shown: In Figure 10 In the diagram, each point corresponds to a specific location of a pedestrian in the bird's-eye view. It can be observed that the predicted values ​​are very close to the actual values, especially in densely populated central areas, demonstrating its powerful detection capability. Figure 11 In the diagram, each line represents the trajectory of a different pedestrian. It can be observed that the prediction shows high accuracy in densely populated central areas, indicating that the proposed method has excellent tracking performance under severe occlusion conditions.

[0120] In the embodiment of the present application, in view of the problem of local detail loss and artifact existing in the existing bird's eye view method, the method introduces an adaptive multi-scale fusion strategy, uses the backbone network to generate multi-scale features, and then uses channel attention to adaptively aggregate the features, which bypasses the constraint of fixed receptive field, retains the detail information, allows the input feature weights to interact, and adaptively generates more effective feature representation; in view of the limitation that the existing bird's eye view method does not accurately associate data, the method introduces a bird's eye view temporal Transformer, uses historical information to predict the potential position of the target in the previous frame, and then uses the Transformer for association, which combines the appearance-based and motion-based association strategies, and uses the Transformer to model the long-range dependency, realizes fine-grained motion estimation by establishing pixel-level feature matching between consecutive frames instead of object-level association, encodes motion prior information while reducing computation; compared with the traditional view-first and temporal association-first methods, the method can effectively alleviate the propagation of errors and improve the accuracy of detection and tracking, and compared with the existing bird's eye view detection method, the method can accurately associate the target even in the case of more targets and longer time, can maintain the consistency of target identity and the robustness of tracking in the scene with serious occlusion, thereby effectively alleviating the occlusion problem; therefore, the adaptive multi-scale feature fusion strategy designed can reduce the artifacts caused by feature extraction while enhancing the feature detail expression, and the improved bird's eye view temporal Transformer is used for pixel-level matching between adjacent frames in the temporal sequence under the bird's eye view, to enhance fine-grained motion estimation, which helps to improve the detection and tracking accuracy.

[0121] Please refer to Figure 12 The embodiment of the present application provides a multi-view target detection and tracking device based on adaptive fusion and temporal association, which comprises:

[0122] An image acquisition module 1201 is configured to synchronously acquire input images of current frames of a plurality of camera devices;

[0123] A feature extraction module 1202 is configured to perform multi-scale feature extraction and adaptive multi-scale feature fusion on each input image respectively, and determine scale fusion features of each input image;

[0124] A feature projection module 1203 is configured to map each scale fusion feature to a bird's eye view space respectively and then splice them to generate current bird's eye view information;

[0125] A motion association module 1204 is configured to input the current bird's eye view information and historical bird's eye view information of adjacent historical frames into an improved bird's eye view temporal Transformer to perform motion estimation, and output bird's eye view features;

[0126] The feature decoding module 1205 is configured to determine the bird's eye view detection and tracking result by detecting the head and tracking the head after decoding the bird's eye view feature.

[0127] Further, the determination process of the scale fusion feature includes:

[0128] Any input image is input into the backbone network to extract multi-scale features and determine three layers of scale features.

[0129] The first refined feature is determined by upsampling and feature reorganization of the bottom layer of scale features, and the adaptive multi-scale feature fusion is performed with the input of the fusion block of the middle layer of scale features to construct the first fusion feature.

[0130] The second refined feature is determined by upsampling and feature reorganization of the first fusion feature, and the adaptive multi-scale feature fusion is performed with the input of the fusion block of the top layer of scale features to generate the second fusion feature.

[0131] The scale fusion feature is output by feature reorganization of the second fusion feature.

[0132] The feature reorganization process includes using a cascaded 3x3 convolution layer and a 1x1 convolution layer for feature reorganization.

[0133] Further, the fusion block includes an adaptive global pooling layer and a sigmoid activation function, and the processing process of the fusion block includes:

[0134] After the input refined feature and the scale feature are spliced, the compression feature is generated by the adaptive global pooling layer.

[0135] The compression feature is uniformly divided into a first sub-compression feature and a second sub-compression feature.

[0136] The sigmoid activation function is used to respectively map the first sub-compression feature and the second sub-compression feature to generate corresponding first channel importance vectors and second channel importance vectors.

[0137] The input refined feature and the first channel importance vector are respectively multiplied element by element, and the input scale feature and the second channel importance vector are respectively multiplied element by element, and then the fusion output fusion feature is output.

[0138] Further, the improved bird's eye view time sequence Transformer includes an improved multi-head cross attention layer, a residual regularization layer, and a fully connected feedforward network layer; and the motion correlation module 1204 is specifically configured to:

[0139] Obtain historical bird's eye view information of the current frame and adjacent historical frames.

[0140] In the improved multi-head cross attention layer, the current bird's eye information is linearly projected into a query matrix by each head of the improved multi-head cross attention layer respectively, and the historical bird's eye information is projected into a key matrix and a value matrix respectively;

[0141] The associated query matrix and the key matrix are subtracted by each head respectively, and then the softmax operation is performed and multiplied element by element with the corresponding value matrix, and after outputting the head feature of each head, the attention feature is outputted by merging;

[0142] The historical bird's eye information and the attention feature are processed by the residual regularization layer to add and regularize the features, and the intermediate feature is determined.

[0143] The intermediate feature is processed by the cascaded fully connected feedforward network layer and the residual regularization layer, and the bird's eye feature is outputted.

[0144] The embodiment of the present application also provides a computer device, which comprises a memory and a processor, and the memory stores a computer program; when the computer program is executed by the processor, the processor executes the steps of the multi-view target detection and tracking method based on adaptive fusion and time sequence association of any one of the above-mentioned embodiments.

[0145] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the multi-view target detection and tracking method based on adaptive fusion and time sequence association of any one of the above-mentioned embodiments are realized.

[0146] The embodiment of the present application also provides a computer program product, which comprises a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the multi-view target detection and tracking method based on adaptive fusion and time sequence association of any one of the above-mentioned embodiments are realized.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and module can refer to the corresponding process in the foregoing method embodiment, and will not be repeated here.

[0148] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely illustrative, and the division of the modules is merely logical function division, and there can be another division manner in actual implementation. For example, a plurality of modules or features can be combined or integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different modules can be implemented by using some interfaces, and the indirect couplings or communication connections between different modules can be implemented in electrical, mechanical or other forms. The module or sub-module illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical modules. In actual implementation, some or all of the modules can be selected according to actual needs to achieve the purposes of the embodiments.

[0149] In addition, each function module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated module can be realized in the form of hardware or in the form of a software function module.

[0150] When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0151] The above embodiments are merely used to describe the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-view object detection and tracking method based on adaptive fusion and temporal correlation, characterized in that, The method comprises the following steps: synchronously acquiring input images of current frames of multiple camera devices; respectively performing multi-scale feature extraction and adaptive multi-scale feature fusion on each of the input images to determine scale fusion features of each of the input images; stitching the scale fusion features after respectively mapping them to a bird's eye view space to generate current bird's eye view information; inputting the current bird's eye view information and historical bird's eye view information of adjacent historical frames into an improved bird's eye view time series Transformer to perform motion estimation and output bird's eye view features; after decoding the bird's eye view features, determining a bird's eye view detection and tracking result through a detection head and a tracking head; the determination process of the scale fusion features comprises the following steps: inputting any of the input images into a backbone network to perform multi-scale feature extraction and determine three layers of scale features; after performing upsampling and feature reorganization on the scale features of the bottom layer to determine first refined features, inputting the first refined features and the scale features of the middle layer into a fusion block to perform adaptive multi-scale feature fusion and construct first fusion features; performing upsampling and feature reorganization on the first fusion features to determine second refined features, and inputting the second refined features and the scale features of the top layer into the fusion block to perform adaptive multi-scale feature fusion and generate second fusion features; performing feature reorganization on the second fusion features to output scale fusion features; the feature reorganization process comprises performing feature reorganization through cascaded 3*3 convolution layers and 1*1 convolution layers; the improved bird's eye view time series Transformer comprises an improved multi-head cross-attention layer, a residual regularization layer, and a fully connected feedforward network layer; and the determination process of the bird's eye view features comprises the following steps: acquiring historical bird's eye view information of adjacent historical frames of the current frame; in the improved multi-head cross-attention layer, linearly projecting the current bird's eye view information into a query matrix through each head of the improved multi-head cross-attention layer, and projecting the historical bird's eye view information into a key matrix and a value matrix respectively; subtracting the query matrix from the key matrix through each head, performing a softmax operation, and element-wise multiplying the value matrix to output head features of each head, and then combining and outputting attention features; performing feature addition and regularization processing on the historical bird's eye view information and the attention features through the residual regularization layer to determine intermediate features; performing feature processing on the intermediate features through the cascaded fully connected feedforward network layer and the residual regularization layer to output bird's eye view features.

2. The multi-view object detection and tracking method based on adaptive fusion and temporal correlation according to claim 1, characterized in that, The fusion block comprises an adaptive global pooling layer and a sigmoid activation function, and the processing process of the fusion block comprises the following steps: after stitching the input refined features and the scale features, performing pooling on the input refined features and the scale features through the adaptive global pooling layer to generate compressed features; uniformly dividing the compressed features into a first sub-compressed feature and a second sub-compressed feature; respectively mapping the first sub-compressed feature and the second sub-compressed feature through a sigmoid activation function to generate corresponding first channel importance vectors and second channel importance vectors; respectively element-wise multiplying the input refined features and the first channel importance vectors, and the input scale features and the second channel importance vectors to output fusion features.

3. A multi-view object detection and tracking device based on adaptive fusion and temporal correlation, characterized in that, The method comprises the following steps: An image acquisition module is configured to synchronously acquire input images of current frames of multiple camera devices; A feature extraction module is configured to perform multi-scale feature extraction and adaptive multi-scale feature fusion on each of the input images respectively, and determine scale fusion features of each of the input images; A feature projection module is configured to map each of the scale fusion features to an aerial view space respectively, and then perform splicing to generate current aerial view information; A motion correlation module is configured to input the current aerial view information and historical aerial view information of adjacent historical frames into an improved aerial view time series Transformer to perform motion estimation, and output aerial view features; A feature decoding module is configured to decode the aerial view features, and determine aerial view detection and tracking results through a detection head and a tracking head. The determination process of the scale fusion features includes: inputting any of the input images into a backbone network to perform multi-scale feature extraction and determine three layers of scale features; performing upsampling and feature reconstruction on the scale features of the bottom layer to determine first refined features, inputting the first refined features and the scale features of the middle layer into a fusion block to perform adaptive multi-scale feature fusion, and constructing first fusion features; performing upsampling and feature reconstruction on the first fusion features to determine second refined features, and inputting the second refined features and the scale features of the top layer into the fusion block to perform adaptive multi-scale feature fusion, and generating second fusion features; performing feature reconstruction on the second fusion features to output scale fusion features; The feature reconstruction process includes performing feature reconstruction through cascaded 3×3 convolution layers and 1×1 convolution layers; The improved aerial view time series Transformer includes an improved multi-head cross-attention layer, a residual regularization layer, and a fully connected feedforward network layer; and the motion correlation module is specifically configured to: acquire historical aerial view information of adjacent historical frames of the current frame; in the improved multi-head cross-attention layer, linearly project the current aerial view information into a query matrix through each head of the improved multi-head cross-attention layer, and project the historical aerial view information into a key matrix and a value matrix respectively; subtract the associated query matrix and key matrix through each head, perform softmax operation, and multiply with the corresponding value matrix element by element, output head features of each head, and then combine and output attention features; perform feature addition and regularization processing on the historical aerial view information and the attention features through the residual regularization layer to determine intermediate features; perform feature processing on the intermediate features through the cascaded fully connected feedforward network layer and the residual regularization layer to output aerial view features.

4. The multi-view object detection and tracking apparatus based on adaptive fusion and temporal correlation according to claim 3, characterized in that, The fusion block includes an adaptive global pooling layer and a sigmoid activation function, and the processing process of the fusion block includes: after splicing the input refined features and scale features, perform pooling through the adaptive global pooling layer to generate compressed features; uniformly divide the compressed features into first and second sub-compressed features; use the sigmoid activation function to respectively map the first and second sub-compressed features to generate corresponding first and second channel importance vectors; and After multiplying the input refined features with the first channel importance vector and the input scale features with the second channel importance vector respectively, the output fused features are fused.

5. A computer device, comprising: A computer readable storage medium storing the computer program product of claim 3.

6. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the steps of the adaptive fusion and temporal correlation based multi-view object detection and tracking method of any one of claims 1-2.

7. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions, when executed by a processor, implement the steps of the adaptive fusion and temporal correlation based multi-view object detection and tracking method of any one of claims 1-2. The computer program / instructions, when executed by a processor, implement the steps of the adaptive fusion and temporal correlation based multi-view object detection and tracking method of any one of claims 1-2.

Images