A 3D multi-target tracking method based on spatio-temporal adaptive attention

Abstract

This invention discloses a 3D multi-object tracking method based on spatiotemporal adaptive attention. The method includes: acquiring 2D multi-source data and 3D multi-source data of a target scene; extracting image features from the 2D multi-source data to obtain image features, and extracting point cloud features from the 3D multi-source data to obtain point cloud features; fusing the image features and the point cloud features to obtain fused features; determining the detection state of each object in the target scene based on the fused features; extracting cooperative features between the detected object and the predicted object based on the detection state; and updating the object detection and tracking state in the target scene in real time based on the cooperative features. This invention can effectively achieve better tracking performance and reduce inference time, and can be widely applied in the field of computer technology.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a 3D multi-target tracking method based on spatiotemporal adaptive attention. Background Technology

[0002] Multi-object tracking is widely used in drones, autonomous vehicles, visual surveillance, and other fields. For autonomous vehicles, accurate real-time 3D multi-object tracking systems can help them understand and perceive their environment more intelligently. However, multi-object tracking algorithms relying on existing single-modal (with camera or LiDAR systems) data cannot effectively integrate texture information extracted from images and spatial structure information extracted from point cloud data. Based on these aspects, some studies have shown that multimodal data fusion technology can improve the reliability and capability of perception tasks compared to single-modal perception systems, thereby enhancing the dynamism, reliability, and robustness of the entire autonomous driving system. Furthermore, multimodal perception systems can also extend temporal and spatial coverage, improving real-time performance and information utilization.

[0003] In existing 3D multi-object tracking algorithms, for per-detection tracking mechanisms, only with a well-designed object detection module can the ID switching and drift of tracked targets in point cloud data be avoided. However, joint tracking and detection mechanisms can collaboratively use learning detection and tracking modules. By mutually constraining and optimizing these two modules, a balance can be found between the detection and tracking modules to achieve better tracking performance. LSTM algorithms and detection-tracking correlation matrix-based methods are commonly used to match detected and tracked objects in spatiotemporal series. Although LSTM algorithms have proven to handle spatiotemporal data well and can be used as a typical timing sequence processing method for time series analysis tasks, they still require a large calibration dataset for training with low inference efficiency, which makes meeting the requirements of real-time systems challenging. Therefore, due to data limitations, methods based on calculating the detection-tracking correlation matrix are often used to associate detected targets with predicted targets for tracking tasks. However, some detection-tracking correlation-based methods often naively combine the features of detected and predicted objects, failing to intelligently achieve data association based solely on the features of the detected objects. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a 3D multi-target tracking method based on spatiotemporal adaptive attention to effectively obtain better tracking performance and reduce inference time.

[0005] One aspect of this invention provides a 3D multi-target tracking method based on spatiotemporal adaptive attention, comprising: acquiring 2D multi-source data and 3D multi-source data in a target scene; extracting image features from the 2D multi-source data to obtain image features; and extracting point cloud features from the 3D multi-source data to obtain point cloud features.

[0006] The image features and the point cloud features are fused to obtain fused features;

[0007] The detection status of each object in the target scene is determined based on the fusion features;

[0008] Based on the detection status, extract the collaborative features between the detected object and the predicted object;

[0009] The object detection and tracking status within the target scene is updated in real time based on the aforementioned collaborative features.

[0010] Optionally, the step of extracting image features from 2D multi-source data to obtain image features includes:

[0011] The 2D multi-source data is input into the image extraction network;

[0012] Deep texture features of the image in the 2D multi-source data are extracted by convolution in the image extraction network.

[0013] The extracted deep texture features of the image are linked together to obtain a two-dimensional fused feature tensor;

[0014] The image extraction network includes four image extractor blocks; each image extractor block includes a first convolutional layer, a batch normalization layer, a corrected linear unit layer, and a second convolutional layer.

[0015] Optionally, the step of extracting point cloud features from 3D multi-source data to obtain point cloud features includes:

[0016] The 3D multi-source data is input into the point cloud extraction network;

[0017] The encoder and decoder in the point cloud extraction network are used to obtain 3D fusion features from the 3D multi-source data.

[0018] The encoder consists of four set abstract blocks, and the decoder consists of four feature propagation blocks.

[0019] Optionally, the step of fusing the image features and the point cloud features to obtain fused features includes: using a shared weighted multilayer perceptron on the vectorization of the feature tensors of the image features and the point cloud features, and adding the corresponding vectorized feature tensors element by element;

[0020] Weighted 2D features can be obtained by using fully connected layers and the Sigmoid function to achieve weighted mapping.

[0021] The element-wise concatenation of 3D point cloud features and weighted 2D features is performed to calculate the final fused features.

[0022] Optionally, determining the detection state of each object in the target scene based on the fusion features includes:

[0023] Based on the final fusion features, a region proposal network is used to separate the background and the attention detection object;

[0024] The optimal 3D bounding box of an object is obtained through mapping functions; wherein, the mapping functions include translation mapping functions and scaling mapping functions;

[0025] Based on the optimal 3D bounding box, the detection state of each object in the target scene is obtained through the detection module.

[0026] Optionally, extracting the collaborative features between the detected object and the predicted object based on the detection state includes:

[0027] The correlation information between the predicted object and the historically tracked object is obtained through the spatiotemporal target singularity attention module;

[0028] An adaptive computation mechanism is used to obtain the correlation tensor of each object between two adjacent frames;

[0029] Based on the correlation information and the correlation tensor, a two-layer perceptron is used to unify re-identification, determine the unique identifier of each detected object, and obtain the collaborative features between the detected object and the predicted object.

[0030] Optionally, the step of updating the object detection and tracking status in the target scene in real time based on the cooperative features includes:

[0031] The affinity matrix between the detected object and the predicted object is calculated based on the class-specific affinity magnitude.

[0032] Based on the affinity matrix, the objects in the target scene are optimized by combining the object detection confidence and the object management probability, and the association between objects is constrained by class specialization to complete the update of the object detection and tracking status.

[0033] Another aspect of this invention provides a 3D multi-target tracking device based on spatiotemporal adaptive attention, comprising:

[0034] The first module is used to acquire 2D multi-source data and 3D multi-source data in the target scene, perform image feature extraction on the 2D multi-source data to obtain image features, and perform point cloud feature extraction on the 3D multi-source data to obtain point cloud features.

[0035] The second module is used to fuse the image features and the point cloud features to obtain fused features;

[0036] The third module is used to determine the detection status of each object in the target scene based on the fusion features;

[0037] The fourth module is used to extract the collaborative features between the detected object and the predicted object based on the detection status.

[0038] The fifth module is used to update the object detection and tracking status in the target scene in real time based on the collaborative features.

[0039] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0040] The memory is used to store programs;

[0041] The processor executes the program to implement the method described above.

[0042] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0043] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0044] Embodiments of the present invention acquire 2D multi-source data and 3D multi-source data in a target scene, extract image features from the 2D multi-source data to obtain image features, and extract point cloud features from the 3D multi-source data to obtain point cloud features; fuse the image features and the point cloud features to obtain fused features; determine the detection state of each object in the target scene based on the fused features; extract cooperative features between the detected object and the predicted object based on the detection state; and update the object detection and tracking state in the target scene in real time based on the cooperative features. The present invention can effectively obtain better tracking performance and reduce inference time. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating the overall steps of an embodiment of the present invention;

[0047] Figure 2 This is an architecture diagram of the adaptive spatial guidance fusion module provided in an embodiment of the present invention;

[0048] Figure 3 This is a block diagram of a spatiotemporal adaptive attention 3D tracker provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] To address the problems existing in the prior art, one aspect of the present invention provides a 3D multi-target tracking method based on spatiotemporal adaptive attention, comprising:

[0051] Acquire 2D and 3D multi-source data of the target scene, extract image features from the 2D multi-source data to obtain image features, and extract point cloud features from the 3D multi-source data to obtain point cloud features.

[0052] The image features and the point cloud features are fused to obtain fused features;

[0053] The detection status of each object in the target scene is determined based on the fusion features;

[0054] Based on the detection status, extract the collaborative features between the detected object and the predicted object;

[0055] The object detection and tracking status within the target scene is updated in real time based on the aforementioned collaborative features.

[0056] Optionally, the step of extracting image features from 2D multi-source data to obtain image features includes:

[0057] The 2D multi-source data is input into the image extraction network;

[0058] Deep texture features of the image in the 2D multi-source data are extracted by convolution in the image extraction network.

[0059] The extracted deep texture features of the image are linked together to obtain a two-dimensional fused feature tensor;

[0060] The image extraction network includes four image extractor blocks; each image extractor block includes a first convolutional layer, a batch normalization layer, a corrected linear unit layer, and a second convolutional layer.

[0061] Optionally, the step of extracting point cloud features from 3D multi-source data to obtain point cloud features includes:

[0062] The 3D multi-source data is input into the point cloud extraction network;

[0063] The encoder and decoder in the point cloud extraction network are used to obtain 3D fusion features from the 3D multi-source data.

[0064] The encoder consists of four set abstract blocks, and the decoder consists of four feature propagation blocks.

[0065] Optionally, the step of fusing the image features and the point cloud features to obtain fused features includes: using a shared weighted multilayer perceptron on the vectorization of the feature tensors of the image features and the point cloud features, and adding the corresponding vectorized feature tensors element by element;

[0066] Weighted 2D features can be obtained by using fully connected layers and the Sigmoid function to achieve weighted mapping.

[0067] The element-wise concatenation of 3D point cloud features and weighted 2D features is performed to calculate the final fused features.

[0068] Optionally, determining the detection state of each object in the target scene based on the fusion features includes:

[0069] Based on the final fusion features, a region proposal network is used to separate the background and the attention detection object;

[0070] The optimal 3D bounding box of an object is obtained through mapping functions; wherein, the mapping functions include translation mapping functions and scaling mapping functions;

[0071] Based on the optimal 3D bounding box, the detection state of each object in the target scene is obtained through the detection module.

[0072] Optionally, extracting the collaborative features between the detected object and the predicted object based on the detection state includes:

[0073] The correlation information between the predicted object and the historically tracked object is obtained through the spatiotemporal target singularity attention module;

[0074] An adaptive computation mechanism is used to obtain the correlation tensor of each object between two adjacent frames;

[0075] Based on the correlation information and the correlation tensor, a two-layer perceptron is used to unify re-identification, determine the unique identifier of each detected object, and obtain the collaborative features between the detected object and the predicted object.

[0076] Optionally, the step of updating the object detection and tracking status in the target scene in real time based on the cooperative features includes:

[0077] The affinity matrix between the detected object and the predicted object is calculated based on the class-specific affinity magnitude.

[0078] Based on the affinity matrix, the objects in the target scene are optimized by combining the object detection confidence and the object management probability, and the association between objects is constrained by class specialization to complete the update of the object detection and tracking status.

[0079] Another aspect of this invention provides a 3D multi-target tracking device based on spatiotemporal adaptive attention, comprising:

[0080] The first module is used to acquire 2D multi-source data and 3D multi-source data in the target scene, perform image feature extraction on the 2D multi-source data to obtain image features, and perform point cloud feature extraction on the 3D multi-source data to obtain point cloud features.

[0081] The second module is used to fuse the image features and the point cloud features to obtain fused features;

[0082] The third module is used to determine the detection status of each object in the target scene based on the fusion features;

[0083] The fourth module is used to extract the collaborative features between the detected object and the predicted object based on the detection status.

[0084] The fifth module is used to update the object detection and tracking status in the target scene in real time based on the collaborative features.

[0085] Another aspect of the present invention provides an electronic device, including a processor and a memory;

[0086] The memory is used to store programs;

[0087] The processor executes the program to implement the method described above.

[0088] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the methods described above.

[0089] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0090] The following describes in detail the specific implementation process of the 3D multi-target tracking method based on spatiotemporal adaptive attention according to an embodiment of the present invention, with reference to the accompanying drawings:

[0091] Reference Figure 1 This invention provides a feature-adaptive and mutually guided multi-source information fusion classification method, which includes the following steps:

[0092] S1. Image and point cloud features are extracted using feature extraction branches based on 2D and 3D multi-source data, and then the detection state is obtained from the fused feature tensor based on two different modalities.

[0093] S2. A shared weighted self-attention module is used to extract collaborative features between the detected and predicted objects, and the correlation matrix is ​​calculated in an adaptive weighted manner.

[0094] S3. Use special linear programming and Kalman filters to constrain data associations and manage and update tracking states.

[0095] As a further preferred embodiment of this method, the step of extracting features from the point cloud and image using 2D+3D dual-stream branches, and then obtaining the detection state from the fused feature tensor based on two different modalities, specifically includes:

[0096] The S11, 2D feature extraction branch takes the image as network input and extracts deep texture features of the image through a series of commonly used convolutions.

[0097] Specifically, the image extractor block (I EB) contains one convolutional layer (kernel size set to 3×3, stride set to 1), a batch normalization (BN) layer, a corrected linear unit (ReLU) layer, and another convolutional layer (kernel size set to 3×3, stride set to 1). The image extraction module (I EM) consists of four I EBs in total. The generation of each extractor block is upsampled to the input size through a deconvolutional layer (Deconv(·)). Finally, the four recovered feature tensors are combined in a concatenated manner to obtain a two-dimensional fused feature tensor.

[0098]

[0099] Where 2×i represents the kernel size and the stride of the deconvolution layer. The feature tensors extracted at different stages can be obtained from the following formula:

[0100]

[0101] in, This represents the original image data.

[0102] The S12 3D feature extraction branch uses point clouds as network input and Point-RCNN as the backbone. Its branch structure borrows from U-Net and uses skip connections to connect corresponding blocks in the encoder (EC) and decoder (DC). The encoder consists of four Set Abstraction (SA) blocks, and the decoder consists of four Feature Propagation (FP) blocks. Let F... pc F′ pc and These represent the original point cloud data, coded point cloud features, and 3D fused features, respectively. The final 3D fused features... This can be expressed as follows:

[0103]

[0104] Similarly, It can be represented as:

[0105]

[0106]

[0107]

[0108] in, and Represents the output of different encoding stages, and It can be represented as follows:

[0109]

[0110] S13. Improved LiDAR-Image fusion module, introducing an adaptive spatially guided fusion module for full computation. and F pc To reduce the correlation between them and reduce computational complexity, the architecture of the adaptive spatial guidance fusion module is as follows: Figure 2 As shown.

[0111] Specifically, the adaptive spatially guided fusion module consists of three main steps. First, a shared weighted multilayer perceptron (MLP) layer is used on the vectorization of the feature tensor, instead of a 3D convolutional layer on the original feature tensor. Then, the two MLP features are element-wise added, and a weighted mapping is obtained through a fully connected layer and a sigmoid (·) function. Finally, the final fused feature F at time t is computed through element-wise connections between the 3D features and the weighted 2D features. u

[0112]

[0113] Once the fusion matrix is ​​obtained, a Region Proposal Network (RPN) is used to separate the background and the objects to be detected, and the optimal 3D bounding boxes of the objects are obtained through mapping functions such as translation and scaling. The detection module obtains the detection state (ds∈R) of each object in the scene. 8×M This includes classification information and a 3D bounding box.

[0114]

[0115] Where M represents the total number of objects to be detected. (x,y,z) represents the position of the object; (l,w,h) represents the length, width, and height of the object; θ represents the viewing angle of the object; and cls represents the category of the object.

[0116] The next step, which uses a shared weighted self-attention module to extract collaborative features between the detected and predicted objects and to compute the correlation matrix in an adaptive weighted manner, specifically includes:

[0117] S21. A spatiotemporal target singularity attention module is proposed to infer the correlation between an object detected at time t and historically tracked objects in the previous frame. An overview of the spatiotemporal target singularity attention module is as follows: Figure 3 Specifically, the spatiotemporal target singularity attention module intelligently pairs objects based on their relative spatial positions throughout the time series. The intelligent fusion strategy is determined by the spatiotemporal sequence correlation between the tracked and detected objects. Therefore, the detection phase... and prediction stage The weighted object feature tensor is as follows:

[0118]

[0119]

[0120] in, and It can be viewed as an activation function that can obtain the correlation and similarity distribution between each object.

[0121] S22. An adaptive computation mechanism is used to obtain the correlation tensor Cor∈R for each object between two adjacent frames. M ×N×D ;

[0122] Specifically

[0123]

[0124] Here, λ1 and λ2 are adaptively learnable parameters.

[0125] S23. After obtaining the correlation matrix, we use a 2-layer MLP to unify the re-identification and start-to-end estimation modules. The re-identification and start-to-end estimation block is a branch network that judges and statistically analyzes the relationship between the detected object and the predicted object, thereby giving each detected object a unique ID.

[0126] As a further preferred embodiment of this method, the use of a special linear programming paradigm and a Kalman filter to constrain data association and manage and update the tracking state. This step specifically includes:

[0127] S31. The affinity matrix X between the detected object and the predicted object is calculated by introducing a special affinity magnitude. aff

[0128] Specifically, affinity consists of four parts: first, re-identifying features to calculate F. re-id The obtained appearance features are: first, the category attribute obtained by object category matching; second, the spatial distance between the detected object and the predicted object; and third, the overlapping area between the 3D bounding boxes of the detected object and the predicted object.

[0129]

[0130] Among them ts cls and ds cls These represent the categories of the tracked and detected objects, respectively. Furthermore, the classification threshold is set to 0.85.

[0131] S32. Use the widely used mixed-integer programming optimizer Google OR-Tools to find the optimal solution in a cost-efficient manner.

[0132] Specifically, in this way, we can estimate the tracking state Y, including the matching / mismatch and missed objects at time t.

[0133] Y = [y cls ,y re-id ,y start ,y end ]

[0134] Among them, y clsIndicates whether the object category is accurately classified. re-id Indicates whether the object successfully matched the predicted object. start This indicates whether a new ID should be assigned. end This indicates whether the object's trajectory should be closed. Regarding the optimization problem between the detected object at time t and the tracked object at time t-1, the model can be constructed as follows:

[0135]

[0136]

[0137] These two constraint functions can be represented using the associated data of a linear problem, as shown below:

[0138]

[0139] By using precondition constraints, tracking information such as accurate tracking, lost tracking, and monitoring completion can be statistically analyzed throughout the lifecycle. The tracking results are used as the predicted state at time t+1. Subsequent tracking tasks continue until all objects are accurately tracked.

[0140] S33. Based on the designed tracker, propose the corresponding loss function.

[0141] Specifically, as mentioned earlier, the algorithm incorporates detection and tracking modules for collaborative training. Therefore, the total loss function must include multiple sets of information from both the detection and tracking states to jointly optimize the final model. Thus, the loss function of the proposed tracker can be defined as follows:

[0142] L = L det +L track =L rpn +L PointRCNN +L re-id +L start +L end

[0143] Among them, L rpn and L PointRCNN By classification ( and ) and regression ( and Composed of ) can be represented as follows:

[0144]

[0145] for and The classification loss uses the Sigmoid Focal and Binary Cross Entropy loss functions as follows:

[0146]

[0147]

[0148] Where ε and γ are hyperparameters. In this invention, ε = 2 and γ = 0.25. i Let p(y) represent the true value of the i-th sample. i ) indicates that it belongs to y i The probability of.

[0149] and The regression loss function expression is the same, and can be expressed as follows:

[0150]

[0151] Re-identification loss L re-id And start-end estimated loss L start and L end The L1 loss function was used. These three parts enable objects in the spatiotemporal sequence to be associated in a more common-sense way, preventing mismatches between distant spatial objects over shorter time spans.

[0152] In summary, this invention provides a novel spatiotemporal adaptive attention 3D tracker. The main design steps of this tracker include extracting features from point clouds and images using 2D+3D dual-stream branches, and then obtaining the detection state from the fused feature tensor based on two different modalities. A shared weighted self-attention module is used to extract cooperative features between the detected and predicted objects, and the correlation matrix is ​​calculated in an adaptive weighted manner. Special-type linear programming and Kalman filters are used to constrain data associations and manage and update the tracking state. The tracker includes: detection state statistics, object association with a spatiotemporal target singularity attention module, and tracking state statistics. By using this invention, the feature complementarity of image and point cloud data can be fully exploited, and feature data fusion from multiple data source types can be compatible, demonstrating strong practical application value.

[0153] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0154] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0155] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0157] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0158] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0159] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0160] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0161] The above is a detailed description of the preferred embodiments of the present invention, but the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A 3D multi-target tracking method based on spatiotemporal adaptive attention, characterized in that, include: Acquire 2D and 3D multi-source data of the target scene, extract image features from the 2D multi-source data to obtain image features, and extract point cloud features from the 3D multi-source data to obtain point cloud features. The image features and the point cloud features are fused to obtain fused features; The detection status of each object in the target scene is determined based on the fusion features; Based on the detection status, extract the collaborative features between the detected object and the predicted object; The object detection and tracking status within the target scene is updated in real time based on the aforementioned collaborative features; The step of extracting image features from 2D multi-source data to obtain image features includes: The 2D multi-source data is input into the image extraction network; Deep texture features of the image in the 2D multi-source data are extracted by convolution in the image extraction network. The extracted deep texture features of the image are linked together to obtain a two-dimensional fused feature tensor; The image extraction network includes four image extractor blocks; each image extractor block includes a first convolutional layer, a batch normalization layer, a corrected linear unit layer, and a second convolutional layer. The step of extracting point cloud features from 3D multi-source data to obtain point cloud features includes: The 3D multi-source data is input into the point cloud extraction network; The encoder and decoder in the point cloud extraction network are used to obtain 3D fusion features from the 3D multi-source data. The encoder consists of four set abstract blocks, and the decoder consists of four feature propagation blocks. The step of fusing the image features and the point cloud features to obtain fused features includes: A shared weighted multilayer perceptron is used to vectorize the feature tensors of the image features and the point cloud features, and the corresponding vectorized feature tensors are added element by element. Weighted 2D features are obtained by using a fully connected layer and a sigmoid function to achieve a weighted mapping. The element-wise concatenation of 3D point cloud features and weighted 2D features is performed to calculate the final fused features.

2. The 3D multi-target tracking method based on spatiotemporal adaptive attention according to claim 1, characterized in that, Determining the detection state of each object in the target scene based on the fusion features includes: Based on the final fusion features, a region proposal network is used to separate the background and the attention detection object; The optimal 3D bounding box of an object is obtained through mapping functions; wherein, the mapping functions include translation mapping functions and scaling mapping functions; Based on the optimal 3D bounding box, the detection state of each object in the target scene is obtained through the detection module.

3. The 3D multi-target tracking method based on spatiotemporal adaptive attention according to claim 1, characterized in that, The step of extracting collaborative features between the detected object and the predicted object based on the detection state includes: The correlation information between the predicted object and the historically tracked object is obtained through the spatiotemporal target singularity attention module; An adaptive computation mechanism is used to obtain the correlation tensor of each object between two adjacent frames; Based on the correlation information and the correlation tensor, a two-layer perceptron is used to unify re-identification, determine the unique identifier of each detected object, and obtain the collaborative features between the detected object and the predicted object.

4. The 3D multi-target tracking method based on spatiotemporal adaptive attention according to claim 1, characterized in that, The step of updating the object detection and tracking status in the target scene in real time based on the cooperative features includes: The affinity matrix between the detected object and the predicted object is calculated based on the class-specific affinity magnitude. Based on the affinity matrix, the objects in the target scene are optimized by combining the object detection confidence and the object management probability, and the association between objects is constrained by class specialization to complete the update of the object detection and tracking status.

5. A 3D multi-target tracking device based on spatiotemporal adaptive attention, characterized in that, include: The first module is used to acquire 2D multi-source data and 3D multi-source data in the target scene, perform image feature extraction on the 2D multi-source data to obtain image features, and perform point cloud feature extraction on the 3D multi-source data to obtain point cloud features. The step of extracting image features from 2D multi-source data to obtain image features includes: The 2D multi-source data is input into the image extraction network; Deep texture features of the image in the 2D multi-source data are extracted by convolution in the image extraction network. The extracted deep texture features of the image are linked together to obtain a two-dimensional fused feature tensor; The image extraction network includes four image extractor blocks; each image extractor block includes a first convolutional layer, a batch normalization layer, a corrected linear unit layer, and a second convolutional layer. The step of extracting point cloud features from 3D multi-source data to obtain point cloud features includes: The 3D multi-source data is input into the point cloud extraction network; The encoder and decoder in the point cloud extraction network are used to obtain 3D fusion features from the 3D multi-source data. The encoder consists of four set abstract blocks, and the decoder consists of four feature propagation blocks. The second module is used to fuse the image features and the point cloud features to obtain fused features; The step of fusing the image features and the point cloud features to obtain fused features includes: A shared weighted multilayer perceptron is used to vectorize the feature tensors of the image features and the point cloud features, and the corresponding vectorized feature tensors are added element by element. Weighted 2D features are obtained by using a fully connected layer and a sigmoid function to achieve a weighted mapping. The element-wise concatenation of 3D point cloud features and weighted 2D features is performed to calculate the final fused features; The third module is used to determine the detection status of each object in the target scene based on the fusion features; The fourth module is used to extract the collaborative features between the detected object and the predicted object based on the detection status. The fifth module is used to update the object detection and tracking status in the target scene in real time based on the collaborative features.

6. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 4.

Images