Multi-target tracking method and device, storage medium and electronic device

Abstract

The present disclosure relates to the technical field of data processing, and in particular, to a multi-target tracking method, device, storage medium and electronic equipment. The method comprises: determining image data sequence and point cloud data information of a current time in a current scene, wherein the elements in the image data sequence are moving targets; projecting the point cloud data information onto the corresponding elements in the image data sequence to generate a target sequence; generating a correlation matrix according to the feature information of each element in the target sequence; and matching the latest point cloud data information and the correlation matrix to obtain a tracking result. In this way, the point cloud data information corresponding to the three-dimensional space can be projected into the image data sequence corresponding to the two-dimensional space, the elements in the image data sequence are used to filter part of the point cloud, the calculation amount when generating the correlation matrix is reduced, the target tracking efficiency is improved, and meanwhile, the uniformity of the image data sequence and the point cloud data information in space and time is ensured, and the accuracy of the target tracking is improved.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more specifically, to a multi-target tracking method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the development of technology, the problem of multi-target tracking in complex environments has received increasing attention from researchers. Image-assisted millimeter-wave radar tracking can solve the problem that millimeter-wave radar cannot distinguish nearby targets, thus enabling the tracking of nearby targets' tracks. How to organically combine these two methods has always been a hot research topic in the field of tracking.

[0003] Currently, most image-assisted millimeter-wave radar tracking methods are based on joint probabilistic data association algorithms that use time-series statistical analysis of image measurements and radar measurements, followed by distance threshold measurement. However, this method consumes excessive computational resources when describing data association due to the construction of a large and complex probabilistic association matrix, and the determined data association is easily affected by the motion disturbances of various targets in the scene, leading to target tracking loss. Summary of the Invention

[0004] The purpose of this disclosure is to provide a multi-target tracking method, apparatus, storage medium, and electronic device to improve the accuracy and efficiency of target tracking.

[0005] To achieve the above objectives, a first aspect of this disclosure provides a multi-target tracking method, comprising:

[0006] Determine the image data sequence and point cloud data information at the current moment in the current scene, wherein the elements in the image data sequence are moving targets in the current scene;

[0007] The point cloud data information is projected onto the corresponding elements in the image data sequence to generate the target sequence;

[0008] Generate an association matrix based on the feature information of each element in the target sequence;

[0009] The latest point cloud data is matched with the correlation matrix to obtain the tracking results.

[0010] Optionally, determining the image data sequence includes:

[0011] The image of the current scene at the current moment is input into a pre-trained image target detection model to obtain the image data sequence output by the image target detection model. The feature information of each element in the image data sequence includes the image position of the moving target, the target category, and the category confidence.

[0012] Optionally, point cloud data information is determined, including:

[0013] Based on clustering algorithms, the point cloud data information is determined according to the radar data at the current moment in the current scene. The feature information of each point cloud in the point cloud data information includes velocity, distance and azimuth.

[0014] Optionally, projecting the point cloud data information onto corresponding elements in the image data sequence to generate a target sequence includes:

[0015] The point cloud data and the image data sequence are input into a pre-trained projection model to obtain the target sequence output by the projection model.

[0016] Optionally, the feature information of each element in the target sequence includes the target index, target category, category confidence, image location, velocity, distance, and azimuth angle; the generation of the correlation matrix includes:

[0017] The target sequence is input into a pre-trained multi-objective Bernoulli filter based on a random finite set to obtain the correlation matrix output by the multi-objective Bernoulli filter.

[0018] Optionally, matching the latest point cloud data information with the correlation matrix to obtain the tracking result includes:

[0019] Based on the Hungarian matching algorithm, the latest point cloud data information is matched with the correlation matrix to obtain the tracking result.

[0020] Optionally, matching the latest point cloud data information with the correlation matrix to obtain the tracking result includes:

[0021] For each cluster center point cloud in the latest point cloud data, determine the deviation distance or intersection-union ratio between the center point cloud and the center point cloud corresponding to each element in the association matrix.

[0022] If multiple deviation distances are less than a distance threshold, or multiple intersection-union ratios are greater than an intersection-union ratio threshold, then the target center point cloud corresponding to the center point cloud in the association matrix is ​​determined based on the velocity and azimuth of the center point cloud and each center point cloud in the association matrix.

[0023] Based on the feature information of the central point cloud, the feature information of the target central point cloud is updated, and each cluster central point cloud in the latest point cloud data information is traversed to obtain the tracking result at the current moment.

[0024] Optionally, the method further includes:

[0025] The tracking results are input into a pre-trained neural network model to obtain the corrected tracking results output by the neural network model.

[0026] A second aspect of this disclosure provides a multi-target tracking device, comprising:

[0027] The first determining module is used to determine the image data sequence and point cloud data information at the current moment in the current scene, wherein the elements in the image data sequence are moving targets in the current scene;

[0028] The first generation module is used to project the point cloud data information onto the corresponding elements in the image data sequence to generate a target sequence;

[0029] The second generation module is used to generate an association matrix based on the feature information of each element in the target sequence;

[0030] The matching module is used to match the latest point cloud data information with the correlation matrix to obtain the tracking result.

[0031] Optionally, the first determining module includes:

[0032] The first determining submodule is used to input the image of the current scene at the current moment into a pre-trained image target detection model to obtain the image data sequence output by the image target detection model, wherein the feature information of each element in the image data sequence includes the image position of the moving target, the target category and the category confidence.

[0033] Optionally, the first determining module includes:

[0034] The second determining submodule is used to determine the point cloud data information based on the radar data at the current moment in the current scene using a clustering algorithm. The feature information of each point cloud in the point cloud data information includes velocity, distance, and azimuth.

[0035] Optionally, the first generation module is used to generate the target sequence in the following manner:

[0036] The point cloud data and the image data sequence are input into a pre-trained projection model to obtain the target sequence output by the projection model.

[0037] Optionally, the feature information of each element in the target sequence includes the target index, target category, category confidence, image location, velocity, distance, and azimuth angle; the second generation module is used to generate the correlation matrix in the following manner:

[0038] The target sequence is input into a pre-trained multi-objective Bernoulli filter based on a random finite set description to obtain the correlation matrix output by the multi-objective Bernoulli filter.

[0039] Optionally, the matching module is used to obtain the tracking results in the following manner:

[0040] Based on the Hungarian matching algorithm, the latest point cloud data information is matched with the correlation matrix to obtain the tracking result.

[0041] Optionally, the matching module includes:

[0042] The third determination submodule is used to determine the deviation distance or intersection-union ratio between the center point cloud of each cluster in the latest point cloud data information and the center point cloud corresponding to each element in the association matrix.

[0043] The fourth determining submodule is used to determine the target center point cloud corresponding to the center point cloud in the association matrix based on the velocity and azimuth angle of the center point cloud and each center point cloud in the association matrix if multiple deviation distances are less than a distance threshold or multiple intersection-union ratios are greater than an intersection-union ratio threshold.

[0044] The update submodule is used to update the feature information of the target center point cloud based on the feature information of the center point cloud, and to traverse the center point cloud of each cluster in the latest point cloud data information to obtain the tracking result at the current moment.

[0045] Optionally, the device further includes:

[0046] The second determining module is used to input the tracking results into a pre-trained neural network model to obtain the corrected tracking results output by the neural network model.

[0047] A third aspect of this disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the steps of the method provided in the first aspect of this disclosure.

[0048] A fourth aspect of this disclosure provides an electronic device, comprising:

[0049] A memory on which computer programs are stored;

[0050] A controller, when the computer program is executed by the controller, implements the steps of the method provided in the first aspect of this disclosure.

[0051] The above technical solution projects point cloud data onto corresponding elements in an image data sequence to generate a target sequence. An association matrix is ​​generated based on the feature information of each element in the target sequence. The latest point cloud data is then matched with the association matrix to obtain the tracking result. This allows point cloud data in three-dimensional space to be projected onto an image data sequence in two-dimensional space. Elements in the image data sequence are used to filter out invalid and interfering point clouds, reducing the computational load during association matrix generation and matching, thus improving target tracking efficiency. Simultaneously, it ensures the spatial and temporal consistency of the image data sequence and point cloud data, improving the accuracy of target tracking.

[0052] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0053] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings:

[0054] Figure 1 This is a flowchart of a multi-target tracking method provided in an exemplary embodiment of this disclosure.

[0055] Figure 2 This is a flowchart of a multi-target tracking method provided in an exemplary embodiment of this disclosure.

[0056] Figure 3 This is a block diagram of a multi-target tracking device provided in an exemplary embodiment of the present disclosure.

[0057] Figure 4 This is a block diagram of an electronic device provided in an exemplary embodiment of the present disclosure. Detailed Implementation

[0058] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0059] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0060] Figure 1 This is a flowchart of a multi-target tracking method provided in an exemplary embodiment of this disclosure. Figure 1 As shown, the method may include S101 to S104.

[0061] S101, determine the image data sequence and point cloud data information at the current moment in the current scene.

[0062] The elements in the image data sequence represent moving targets in the current scene.

[0063] For example, images and radar data can be acquired simultaneously via a camera and millimeter-wave radar, respectively. The corresponding image data sequence can be determined from the images. The moving target is an object moving within the current scene; it can be a person or a vehicle. Typically, multiple moving targets may be present in the same scene. The method provided in this disclosure can track multiple moving targets in the current scene, with each moving target corresponding to an element in the image data sequence. Each moving target (an element in the image data sequence) can correspond to a bounding box on the image. The size of this bounding box can be adapted to the size of the moving target in the image. The feature information of each element in the image data sequence can include the image position of the bounding box corresponding to the moving target, the target category of the moving target, and the category confidence. The target category can include vehicles and people, or vehicles can be further subdivided by vehicle model. Point cloud data information can be determined based on radar data. The point cloud data information can include feature information from multiple point clouds. The feature information of each point cloud can include velocity, distance, and azimuth. The point cloud can be the center point cloud of a cluster determined based on a clustering algorithm.

[0064] S102, Project the point cloud data information onto the corresponding elements in the image data sequence to generate the target sequence.

[0065] In millimeter-wave radar output, due to frame drops, noise, and target overlap, previously labeled point clouds may be reclassified as new point clouds and target IDs may be reassigned, or multiple IDs may be assigned to point clouds at the same location, leading to tracking failure or redundant labeling. Projecting the point cloud data information corresponding to the three-dimensional space onto the corresponding image data sequence in the two-dimensional space can, on the one hand, utilize elements in the image data sequence (i.e., the recognition results of image perception) to filter out part of the point cloud, reducing the computational load when generating the correlation matrix and improving target tracking efficiency; on the other hand, it can also ensure that the image data sequence and point cloud data information spatially construct the positional relationship between moving targets and point clouds. Simultaneously, fusing target visual features also facilitates the description of the correlation between consecutive frames, achieving temporal consistency, improving the accuracy of target tracking, increasing robustness, and also better estimating lateral moving targets and static targets.

[0066] The feature information of each element in the target sequence may include the target category, category confidence, and image location associated with the image, and the velocity, range, and azimuth of the point cloud associated with the radar data. From the image perspective, the feature information corresponding to each bounding box may include the image location, target category, category confidence, and the velocity, range, and azimuth of each point cloud within the bounding box. A single bounding box may contain multiple point clouds. For example, during projection, if a point cloud is not projected into any bounding box, it can be deleted to achieve filtering. Similarly, if a point cloud is determined to be a small target or a low-intensity reflected wave, it can be deleted to achieve filtering. To facilitate the differentiation of different elements in the target sequence, the feature information of each element may also include a target index.

[0067] S103, Generate an association matrix based on the feature information of each element in the target sequence.

[0068] For example, the correlation matrix is ​​used to describe the similarity between two consecutive frames of images. The correlation matrix can be determined using the feature information of each element in the target sequence corresponding to the previous and current time steps. Specifically, it can be described using a multi-Bernoulli random finite set to approximate the posterior probability density of the multi-target state set, thus ensuring that the correlation matrix between adjacent frames in time is dynamically updated. In this way, the determined correlation matrix can be adapted to the target sequence at the current time step.

[0069] S104 matches the latest point cloud data with the correlation matrix to obtain the tracking results.

[0070] For example, the Hungarian algorithm can be used to perform data association verification in the association matrix to determine the point cloud in the association matrix that has the highest similarity to point cloud a in the latest point cloud data information. By connecting the two point clouds with the highest similarity, the tracking path of point cloud a can be obtained, i.e., the tracking result.

[0071] The above technical solution projects point cloud data onto corresponding elements in an image data sequence to generate a target sequence. An association matrix is ​​generated based on the feature information of each element in the target sequence. The latest point cloud data is then matched with the association matrix to obtain the tracking result. This allows point cloud data in three-dimensional space to be projected onto an image data sequence in two-dimensional space. Elements in the image data sequence are used to filter out invalid and interfering point clouds, reducing the computational load during association matrix generation and matching, thus improving target tracking efficiency. Simultaneously, it ensures the spatial and temporal consistency of the image data sequence and point cloud data, improving the accuracy of target tracking.

[0072] Optionally, in S101, determining the image data sequence may include:

[0073] The image of the current scene at the current moment is input into a pre-trained image object detection model to obtain the image data sequence output by the image object detection model.

[0074] The feature information of each element in the image data sequence includes the image location of the moving target, the target category, and the category confidence level.

[0075] For example, the image object detection model can be a model trained based on the YOLOv5 network model structure. The current image of the current scene at the current time is input into the pre-trained image object detection model, and the model outputs an image data sequence. The prior bounding boxes in the image object detection model can output the categories of definite moving objects. The category with the highest probability among multiple prior bounding boxes can be determined as the target category, and the probability corresponding to the target category can be determined as the category confidence. This model can be stored locally for local access each time it is used, or it can be stored on a third-party platform for access each time it is used; no specific limitation is made here.

[0076] Optionally, in S101, determining the point cloud data information may include:

[0077] Based on clustering algorithms, point cloud data information is determined according to radar data at the current moment in the current scene.

[0078] The feature information of each point cloud in the point cloud data includes velocity, distance, and azimuth.

[0079] For example, clustering algorithms can include any one of the following: kmean clustering, DBSCAN clustering, and Euclidean clustering. Clustering algorithms can group scattered point clouds together to identify multiple point cloud clusters. The central point cloud of each cluster is then used as the point cloud in the overall point cloud data, effectively reducing the computational load during the projection process.

[0080] Optionally, in S102, projecting the point cloud data information onto the corresponding elements in the image data sequence to generate the target sequence may include:

[0081] Point cloud data and image data sequences are input into a pre-trained projection model to obtain the target sequence output by the projection model.

[0082] For example, point cloud data and image data sequences can be input into a pre-trained projection model, and the model output is the target sequence. The model can detect and classify moving targets in the scene, and then project the point cloud onto the corresponding image target using jointly calibrated extrinsic parameters. These extrinsic parameters are coordinate transformation matrices generated through joint calibration of images acquired by a camera and millimeter-wave radar data. It's worth noting that the projection model can be a machine learning model trained on image and radar data of the current scene, capable of projecting from point cloud data and image data sequences. This projection model can be stored locally for local access each time it's used, or stored on a third-party platform for access each time it's used; no specific limitation is made here.

[0083] The feature information of each element in the target sequence may include target index, target category, category confidence, image location, velocity, distance, and azimuth.

[0084] Optionally, in S103, generating the correlation matrix may include:

[0085] The target sequence is input into a pre-trained multi-objective Bernoulli filter based on a random finite set description to obtain the correlation matrix of the multi-objective Bernoulli filter output.

[0086] For example, the states and measurements of moving targets in the current scene can be defined as random finite sets. During the filtering process, the posterior probabilities of multiple targets are modeled using a multi-Bernoulli distribution, and iterative filtering is performed based on Bayesian principles to achieve tracking of multiple targets of unknown and varying numbers. The states of moving targets may include target index, target category, category confidence, and image position; the measurements of moving targets may include velocity, distance, and azimuth.

[0087] In this process, the effective target results in the association matrix updated and predicted by the multi-Bernoulli filter are matched pairwise with maximum similarity, and the point clouds with effective matching (maximum matching) are added with corresponding categories and indices, thereby realizing the tracking of point cloud targets.

[0088] The set of state vectors of multiple moving targets can be denoted as X, where each element x in the set is the state vector of a single moving target; the set of measurement vectors of multiple moving targets can be denoted as Z, where each element z in the set is the measurement vector of a single moving target, i.e., Where F(X) and F(Z) represent finite subsets of the state space and measurement space, respectively, k is the number of elements, and f is the posterior probability density function of the multi-objective domain. k|k-1 (X k |Z 1:k-1 = can be described as:

[0089] f k|k-1 (X k |Z 1:k-1 )=∫φ k|k-1 (X k |X 1:k-1 )·f k|k-1 (X k-1 |Z 1:k-1 )dX k-1

[0090] The update steps are as follows:

[0091]

[0092] Where, φ k|k-1 (·) is the Markov state transition density function, g k (·) is the multi-objective likelihood function.

[0093] In this way, the state transition (i.e., posterior probability) can be determined based on the observation state at the current moment and the observation state at the previous moment, and the correlation matrix can be dynamically updated. Then, the confidence of the set to which the current correlation matrix belongs can be updated using the maximum confidence in the current row of the correlation matrix.

[0094] Optionally, in S104, the latest point cloud data information and the correlation matrix are matched to obtain the tracking result, which may include:

[0095] Based on the Hungarian matching algorithm, the latest point cloud data information and the correlation matrix are matched to obtain the tracking results.

[0096] For example, for each cluster center point cloud in the latest point cloud data, the deviation distance between that center point cloud and the center point cloud corresponding to each element in the association matrix can be determined; if the minimum deviation distance is less than the distance threshold, then the center point cloud corresponding to the minimum distance in the association matrix is ​​determined as the target center point cloud; based on the feature information of that center point cloud, the feature information of the target center point cloud is updated, and each cluster center point cloud in the latest point cloud data is traversed to obtain the tracking result at the current time.

[0097] For example, for each cluster center point cloud in the latest point cloud data, the intersection-union ratio (IUR) between that center point cloud and the center point cloud corresponding to each element in the association matrix can be determined. If the maximum IUR is greater than the IUR threshold, the center point cloud in the association matrix corresponding to the maximum IUR can be determined as the target center point cloud. Based on the feature information of that center point cloud, the feature information of the target center point cloud is updated, and each cluster center point cloud in the latest point cloud data is traversed to obtain the tracking result at the current time.

[0098] Based on the Hungarian matching algorithm, data association verification and similarity comparison can be performed to match and ensure the accuracy of tracking results.

[0099] Optionally, in S104, the latest point cloud data information and the correlation matrix are matched to obtain the tracking result, which may include:

[0100] For each cluster center point cloud in the latest point cloud data, determine the deviation distance or intersection-union ratio between the center point cloud and the center point cloud corresponding to each element in the association matrix;

[0101] If multiple deviation distances are less than the distance threshold, or multiple intersection-union ratios are greater than the intersection-union ratio threshold, then the target center point cloud corresponding to the center point cloud in the correlation matrix is ​​determined based on the velocity and azimuth of the center point cloud and each center point cloud in the correlation matrix.

[0102] Based on the feature information of the central point cloud, the feature information of the target central point cloud is updated, and each cluster central point cloud in the latest point cloud data is traversed to obtain the tracking result at the current moment.

[0103] The displacement of the same moving target between adjacent frames will not change too much. Distance, velocity and orientation can be used as measures and constraints on the change of the same target between adjacent frames to obtain matching results.

[0104] For example, taking cluster center point cloud 'a' from the latest point cloud data as an example, if the distance between center point cloud 'a' and center point clouds 'b' and 'c' in the association matrix is ​​less than a preset distance threshold, then the target point cloud can be determined based on the similarity of velocity and azimuth between center point clouds 'a' and 'b', and between center point clouds 'a' and 'c'. For instance, if the velocity difference and azimuth deviation between center point clouds 'a' and 'b' are less than the velocity difference and azimuth deviation between center point clouds 'a' and 'c', then center point cloud 'b' can be used as the target center point cloud. As another example, the similarity can be determined by a weighted sum of distance, velocity, and azimuth. The azimuth, velocity, and distance of center point cloud 'a' can be updated based on the azimuth, velocity, and distance of center point cloud 'a'. In this way, by traversing the center point clouds of each cluster in the latest point cloud data, the tracking result at the current moment can be obtained.

[0105] In this way, the target point cloud with the highest similarity to each central point cloud in the latest point cloud data can be determined from the correlation matrix, thus achieving accuracy of the latest point cloud data and the correlation matrix.

[0106] Optionally, the multi-target tracking method provided in this disclosure may further include:

[0107] The tracking results are input into a pre-trained neural network model to obtain the corrected tracking results output by the neural network model.

[0108] For example, the neural network model can be a Long Short-Term LSTM network with memory capabilities, which can observe and track the motion state of a single target based on time series data to ensure the accuracy of tracking results for the same target between consecutive frames. Thus, inputting the tracking results into a pre-trained neural network model can yield more accurate and corrected tracking results.

[0109] Figure 2 This is a flowchart of a multi-target tracking method provided in an exemplary embodiment of this disclosure. Figure 2 This allows for a clearer understanding of the implementation process of the multi-target tracking method provided in this disclosure. For example... Figure 2 As shown, the method may include S201 to S212.

[0110] S201, acquire image and radar data for the current moment in the current scene.

[0111] S202, input the image into the pre-trained image target detection model to obtain the image data sequence output by the image target detection model.

[0112] In this image data sequence, each element represents a moving target, and the feature information of each element includes the image location, target category, and category confidence level of the moving target.

[0113] S203, based on a clustering algorithm, determines point cloud data information according to radar data at the current moment in the current scene.

[0114] The feature information of each point cloud in the point cloud data includes velocity, distance, and azimuth.

[0115] S204: Input the point cloud data information and image data sequence into the pre-trained projection model to obtain the target sequence output by the projection model.

[0116] The feature information of each element in the target sequence includes the target index, target category, category confidence, image location, velocity, distance, and azimuth.

[0117] S205, input the target sequence into a pre-trained multi-objective Bernoulli filter based on a random finite set description, and obtain the correlation matrix of the multi-objective Bernoulli filter output.

[0118] S206, based on the Hungarian matching algorithm, matches the latest point cloud data information with the correlation matrix to obtain the tracking results.

[0119] S207, input the tracking results into the pre-trained neural network model to obtain the corrected tracking results output by the neural network model.

[0120] By projecting point cloud data from three-dimensional space onto a corresponding image data sequence in two-dimensional space, and filtering out parts of the point cloud using elements from the image data sequence, the computational load when generating the correlation matrix is ​​reduced, thus improving target tracking efficiency. Simultaneously, it ensures the spatial and temporal consistency of the image data sequence and point cloud data, enhancing target tracking accuracy. Furthermore, through the Hungarian matching algorithm and neural network models, multi-target tracking can be achieved more efficiently and accurately.

[0121] Based on the same inventive concept, this disclosure also provides a multi-target tracking device. Figure 3 This is a block diagram of a multi-target tracking device 300 provided in an exemplary embodiment of this disclosure. (Refer to...) Figure 3 The multi-target tracking device 300 may include:

[0122] The first determining module 301 is used to determine the image data sequence and point cloud data information at the current moment in the current scene, wherein the elements in the image data sequence are moving targets;

[0123] The first generation module 302 is used to project the point cloud data information onto the corresponding elements in the image data sequence to generate a target sequence;

[0124] The second generation module 303 is used to generate an association matrix based on the feature information of each element in the target sequence;

[0125] The matching module 304 is used to match the latest point cloud data information with the correlation matrix to obtain the tracking result.

[0126] The above technical solution projects point cloud data onto corresponding elements in an image data sequence to generate a target sequence. An association matrix is ​​generated based on the feature information of each element in the target sequence. The latest point cloud data is then matched with the association matrix to obtain the tracking result. This allows point cloud data in three-dimensional space to be projected onto an image data sequence in two-dimensional space. Elements in the image data sequence are used to filter out invalid and interfering point clouds, reducing the computational load during association matrix generation and matching, thus improving target tracking efficiency. Simultaneously, it ensures the spatial and temporal consistency of the image data sequence and point cloud data, improving the accuracy of target tracking.

[0127] Optionally, the first determining module 301 includes:

[0128] The first determining submodule is used to input the image of the current scene at the current moment into a pre-trained image target detection model to obtain the image data sequence output by the image target detection model, wherein the feature information of each element in the image data sequence includes the image position of the moving target, the target category and the category confidence.

[0129] Optionally, the first determining module 301 includes:

[0130] The second determining submodule is used to determine the point cloud data information based on the radar data at the current moment in the current scene using a clustering algorithm. The feature information of each point cloud in the point cloud data information includes velocity, distance, and azimuth.

[0131] Optionally, the first generation module 302 is used to generate the target sequence in the following manner:

[0132] The point cloud data and the image data sequence are input into a pre-trained projection model to obtain the target sequence output by the projection model.

[0133] Optionally, the feature information of each element in the target sequence includes the target index, target category, category confidence, image location, velocity, distance, and azimuth angle; the second generation module 303 is used to generate the correlation matrix in the following manner:

[0134] The target sequence is input into a pre-trained multi-objective Bernoulli filter based on a random finite set description to obtain the correlation matrix output by the multi-objective Bernoulli filter.

[0135] Optionally, the matching module 304 is used to obtain the tracking result in the following manner:

[0136] Based on the Hungarian matching algorithm, the latest point cloud data information is matched with the correlation matrix to obtain the tracking result.

[0137] Optionally, the matching module 304 includes:

[0138] The third determination submodule is used to determine the deviation distance or intersection-union ratio between the center point cloud of each cluster in the latest point cloud data information and the center point cloud corresponding to each element in the association matrix.

[0139] The fourth determining submodule is used to determine the target center point cloud corresponding to the center point cloud in the association matrix based on the velocity and azimuth angle of the center point cloud and each center point cloud in the association matrix if multiple deviation distances are less than a distance threshold or multiple intersection-union ratios are greater than an intersection-union ratio threshold.

[0140] The update submodule is used to update the feature information of the target center point cloud based on the feature information of the center point cloud, and to traverse the center point cloud of each cluster in the latest point cloud data information to obtain the tracking result at the current moment.

[0141] Optionally, the device further includes:

[0142] The second determining module is used to input the tracking result into a pre-trained neural network model to obtain the corrected tracking result output by the neural network model.

[0143] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0144] Figure 4 This is a block diagram illustrating an electronic device 700 according to an exemplary embodiment. Figure 4 As shown, the electronic device 700 may include a processor 701 and a memory 702. The electronic device 700 may also include one or more of a multimedia component 703, an input / output (I / O) interface 704, and a communication component 705.

[0145] The processor 701 controls the overall operation of the electronic device 700 to complete all or part of the steps in the multi-target tracking method described above. The memory 702 stores various types of data to support the operation of the electronic device 700. This data may include, for example, instructions for any application or method operating on the electronic device 700, and application-related data such as contact data, sent and received messages, pictures, audio, video, etc. The memory 702 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 703 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 702 or transmitted via communication component 705. The audio component also includes at least one speaker for outputting audio signals. I / O interface 704 provides an interface between processor 701 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 705 is used for wired or wireless communication between the electronic device 700 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, 4G, NB-IoT, eMTC, or other 5G technologies, or combinations thereof, is not limited here. Therefore, the corresponding communication component 705 may include: a Wi-Fi module, a Bluetooth module, an NFC module, etc.

[0146] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the multi-target tracking method described above.

[0147] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the multi-target tracking method described above. For example, the computer-readable storage medium may be the memory 702 including program instructions described above, which may be executed by the processor 701 of the electronic device 700 to complete the multi-target tracking method described above.

[0148] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a programmable device, the computer program having a code portion for performing the multi-target tracking method described above when executed by the programmable device.

[0149] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0150] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0151] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A multi-target tracking method, characterized in that, include: Determine the image data sequence and point cloud data information at the current moment in the current scene, wherein the elements in the image data sequence are moving targets in the current scene; The point cloud data information is projected onto the corresponding elements in the image data sequence to filter the point cloud using the elements in the image data sequence and generate a target sequence. The feature information of each element in the target sequence includes target index, target category, category confidence, image position, velocity, distance, and azimuth. Generate an association matrix based on the feature information of each element in the target sequence; The latest point cloud data information is matched with the correlation matrix to obtain the tracking results; The process of generating an association matrix based on the feature information of each element in the target sequence includes: The target sequence is input into a pre-trained multi-objective Bernoulli filter based on a random finite set description to obtain the correlation matrix output by the multi-objective Bernoulli filter. The step of matching the latest point cloud data information with the correlation matrix to obtain the tracking result includes: For each cluster center point cloud in the latest point cloud data, determine the deviation distance or intersection-union ratio between the center point cloud and the center point cloud corresponding to each element in the association matrix. If multiple deviation distances are less than a distance threshold, or multiple intersection-union ratios are greater than an intersection-union ratio threshold, then the target center point cloud corresponding to the center point cloud in the association matrix is ​​determined based on the velocity and azimuth of the center point cloud and each center point cloud in the association matrix. Based on the feature information of the central point cloud, the feature information of the target central point cloud is updated, and each cluster central point cloud in the latest point cloud data is traversed to obtain the tracking result at the current moment.

2. The method according to claim 1, characterized in that, Determine the image data sequence, including: The image of the current scene at the current moment is input into a pre-trained image target detection model to obtain the image data sequence output by the image target detection model. The feature information of each element in the image data sequence includes the image position of the moving target, the target category, and the category confidence.

3. The method according to claim 1, characterized in that, Determine point cloud data information, including: Based on a clustering algorithm, the point cloud data information is determined according to the radar data at the current moment in the current scene. The feature information of each point cloud in the point cloud data information includes velocity, distance and azimuth.

4. The method according to claim 1, characterized in that, The step of projecting the point cloud data information onto the corresponding elements in the image data sequence to filter the point cloud using the elements in the image data sequence and generate a target sequence includes: The point cloud data and the image data sequence are input into a pre-trained projection model to filter the point cloud using elements in the image data sequence, thereby obtaining the target sequence output by the projection model.

5. The method according to claim 1, characterized in that, The step of matching the latest point cloud data information with the correlation matrix to obtain the tracking result includes: Based on the Hungarian matching algorithm, the latest point cloud data information is matched with the correlation matrix to obtain the tracking result.

6. The method according to claim 1, characterized in that, The method further includes: The tracking results are input into a pre-trained neural network model to obtain the corrected tracking results output by the neural network model.

7. A multi-target tracking device, characterized in that, The device includes: The first determining module is used to determine the image data sequence and point cloud data information at the current moment in the current scene, wherein the elements in the image data sequence are moving targets in the current scene; The first generation module is used to project the point cloud data information onto the corresponding elements in the image data sequence, so as to filter the point cloud using the elements in the image data sequence and generate a target sequence. The feature information of each element in the target sequence includes target index, target category, category confidence, image position, velocity, distance and azimuth angle. The second generation module is used to generate an association matrix based on the feature information of each element in the target sequence; The matching module is used to match the latest point cloud data information with the correlation matrix to obtain the tracking result; The second generation module is used to generate the association matrix in the following manner: The target sequence is input into a pre-trained multi-objective Bernoulli filter based on a random finite set description to obtain the correlation matrix output by the multi-objective Bernoulli filter. The matching module includes: The third determination submodule is used to determine the deviation distance or intersection-union ratio between the center point cloud of each cluster in the latest point cloud data information and the center point cloud corresponding to each element in the association matrix. The fourth determining submodule is used to determine the target center point cloud corresponding to the center point cloud in the association matrix based on the velocity and azimuth angle of the center point cloud and each center point cloud in the association matrix if multiple deviation distances are less than a distance threshold or multiple intersection-union ratios are greater than an intersection-union ratio threshold. The update submodule is used to update the feature information of the target center point cloud based on the feature information of the center point cloud, and to traverse the center point cloud of each cluster in the latest point cloud data information to obtain the tracking result at the current moment.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method described in any one of claims 1-6.

9. An electronic device, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1-6.

Images