Target tracking method and device fusing multi-view images and three-dimensional point clouds

Abstract

The application discloses a target tracking method and device fusing multi-view images and three-dimensional point clouds, which comprises the following steps: firstly, fusing multi-view multi-modal data to predict the three-dimensional position of a target in a scene; then, using the three-dimensional detection result to obtain the three-dimensional space information of the target and the appearance features of the target in the multi-view images, calculating a matching score matrix, and finally obtaining an accurate three-dimensional target tracking result. The application uses the appearance features of the current object and the appearance features of the historical to-be-restored trajectory to match and restore the trajectory. The three-dimensional target tracking method can greatly eliminate the problems of trajectory exchange and target loss in a single modal tracking result, and can obtain a high-accuracy, good-continuity and strong-robustness long-period tracking trajectory of the target, thereby providing convenience for scene perception and security monitoring.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision perception, in particular to a target tracking method and device fusing multi-view images and three-dimensional point clouds. BACKGROUND

[0002] In the field of three-dimensional vision, accurate three-dimensional target detection and tracking are the basis for scene visual perception and analysis. In recent years, the hot development of automatic driving, scene monitoring and other fields has put forward urgent demand for high-precision target tracking algorithm, so three-dimensional target tracking has become the most concerned research direction in computer vision. The detection and tracking task takes the original data of the sensor as input and outputs accurate target position and tracking id, which is the basis for subsequent links such as path planning and is also an essential part of the whole system. In the face of the demand for accurate three-dimensional positioning and tracking, the depth camera or multi-camera sensor has low precision, short positioning distance, and is greatly affected by light. Laser radar has the characteristics of long distance, high precision and strong stability, but it also has the disadvantages of high price, sparse point cloud and lack of texture features of the target. The existing three-dimensional target tracking technology directly obtains the tracking trajectory of the target based on the input of a single modal sensor, and the result is limited by the defects of the single sensor used, and it is difficult to obtain high-quality tracking results in complex and diverse scenes. Therefore, the fusion of two-dimensional images and three-dimensional point cloud multi-modal data can eliminate the shortcomings of a single sensor, enhance the robustness of the tracking technology, and greatly improve the accuracy of the tracking result.

[0003] In the existing three-dimensional target tracking technology, there are still the following limitations and deficiencies: the three-dimensional tracking technology based on multi-view images obtains two-dimensional target detection results by identifying images, and then relies on a well-calibrated multi-camera system to calculate the three-dimensional position of the target using epipolar geometry constraints, and finally matches and connects to obtain the three-dimensional target tracking trajectory. Due to the calibration error and occlusion, it is difficult for the technology based on multi-view images to obtain the accurate three-dimensional position of the target, and it also lacks spatial position information, so the occlusion of the target can easily lead to tracking loss and id matching errors. The three-dimensional tracking technology based on three-dimensional point cloud uses laser radar to collect high-precision point cloud data, obtains accurate three-dimensional detection results of the target, calculates a matching score matrix in three-dimensional space, and connects to obtain a three-dimensional tracking trajectory. Due to the sparsity of point cloud data, the target lacks texture features, and the matching score calculation and trajectory generation rely on the geometric position of the target, so when multiple targets gather in three-dimensional space, it is easy to cause the exchange of tracking ids.

[0004] With the continuous development of three-dimensional imaging systems, the target tracking technology based on traditional single modal data acquisition systems cannot meet the needs of robust and high-precision tracking. SUMMARY

[0005] The present application aims to at least solve one of the technical problems in the related art to some extent.

[0006] To this end, the present application proposes a target tracking method fusing multi-view images and three-dimensional point clouds. The present application is divided into two stages in the whole, the first stage fuses multi-view images and three-dimensional point cloud data for three-dimensional target detection, and the second stage matches and connects the output target tracking trajectory based on the spatial information and appearance features of the detected target. In the first stage, the present application simultaneously inputs two-dimensional images and three-dimensional point clouds from multiple perspectives in the scene, and performs feature extraction and fusion. For two-dimensional images, the image features from multiple perspectives are obtained by using a two-dimensional feature extraction network, and then the feature maps are projected to a three-dimensional bird's eye view according to different camera parameters for splicing and fusion, so as to obtain a multi-perspective fused two-dimensional feature map; for three-dimensional point clouds, the point clouds from different perspectives of laser radars are directly fused after spatio-temporal registration to obtain multi-angle dense point clouds, and then a three-dimensional feature extraction network is used to obtain a point cloud feature map. The two-dimensional feature map and the point cloud feature map are spliced and fused according to the spatial position to obtain a multi-modal feature map, and then a detector is used to obtain an accurate three-dimensional detection result as the input of the second stage. In the second stage, the present application uses the detection result of the first stage for matching and connection to obtain the final tracking trajectory. The present application calculates the intersection over union of the target detection box in the three-dimensional space as the relationship measurement of the target in the space. Further, in order to fuse the multi-modal features, the present application uses the three-dimensional target detection box to project to the input multi-perspective two-dimensional images to obtain the target region on the two-dimensional images, and extracts the appearance features of the target on the two-dimensional images, calculates the cosine distance of the appearance features between different targets as the relationship measurement of the target in the appearance features. After fusing the two relationship measurement matrices, the present application uses a matching algorithm to match and connect the detection boxes, and outputs the final tracking trajectory of the target. For target objects that leave the scene or are severely occluded, the detection algorithm may miss detection, resulting in a broken tracking trajectory. When such a target is re-identified in the scene, the present application uses the appearance features of the current object and the appearance features of the historical to-be-restored trajectory to match and repair the trajectory, thereby improving the integrity of the trajectory.

[0007] Another object of the present application is to propose a target tracking device fusing multi-view images and three-dimensional point clouds.

[0008] To achieve the above object, the present application proposes, in one aspect, a target tracking method fusing multi-view images and three-dimensional point clouds, comprising:

[0009] obtaining two-dimensional feature maps and point cloud feature maps based on multi-perspective two-dimensional images and three-dimensional point clouds respectively;

[0010] The multi-modal feature map obtained by fusing the two-dimensional feature map and the point cloud feature map is input into a pre-trained multi-modal fusion target detection model for detection to obtain a target detection frame; wherein the target detection frame comprises a three-dimensional target detection result of a target trajectory;

[0011] A spatial position matching matrix is obtained according to a comparison result of the target detection frame and a detection frame of a historical time's trajectory tracking result, and an appearance feature matching matrix is obtained by calculating a two-dimensional image feature of the target trajectory and the historical time's trajectory tracking result, and the spatial position matching matrix and the appearance feature matching matrix are aggregated to obtain a final relationship measurement matrix;

[0012] The three-dimensional target detection result and the historical time's trajectory tracking result are matched based on the final relationship measurement matrix and a preset matching algorithm, and a tracking trajectory result of the target trajectory at the current time is obtained according to a matching result.

[0013] In addition, the target tracking method for fusing multi-view images and three-dimensional point clouds according to the above-mentioned embodiments of the present application can also have the following additional technical features:

[0014] Further, in an embodiment of the present application, the two-dimensional feature map and the point cloud feature map are obtained based on multi-view two-dimensional images and three-dimensional point clouds, respectively, which comprises: acquiring multi-view two-dimensional images and three-dimensional point clouds; using an image processing network to extract the multi-view two-dimensional images to obtain an initial feature map, and projecting the initial feature map to a three-dimensional bird's eye view for splicing and fusion according to different camera parameters to obtain a multi-view fused two-dimensional feature map; and using a point cloud processing network to fuse the three-dimensional point clouds after spatio-temporal registration to obtain multi-angle dense point clouds, and extracting features of the multi-angle dense point clouds to obtain a point cloud feature map.

[0015] Further, in an embodiment of the present application, the point cloud processing network comprises a feature extraction network and a backbone network; the backbone network comprises a first sub-network and a second sub-network; the image processing network comprises a convolutional neural network, and the method further comprises: inputting the multi-view three-dimensional point clouds into the feature extraction network for point cloud conversion to obtain a pseudo image, inputting the pseudo image into the first sub-network for feature extraction of different spatial resolution feature maps; inputting the features extracted from the different spatial resolution feature maps into the second sub-network for deconvolution operation and then concatenating to obtain the point cloud feature map; and inputting the multi-view two-dimensional images into the convolutional neural network to calculate a multi-channel feature map, using projection transformation of the multi-channel feature map for multi-camera information aggregation, so as to project the multi-channel feature map to a three-dimensional bird's eye view for splicing and fusion to obtain the two-dimensional feature map.

[0016] Further, in an embodiment of the present application, the method further comprises: obtaining sample data of the multi-modal feature map, performing data labeling of the target detection frame using the multi-sample data, training the multi-modal fusion target detection model using the labeled data; inputting the two-dimensional image and the three-dimensional point cloud to be tracked into the trained multi-modal fusion target detection model to detect the target trajectory, and obtaining the three-dimensional target detection result of each prediction time.

[0017] Further, in an embodiment of the present application, the method further comprises: performing matching score calculation according to the detection frame geometric information of the three-dimensional target detection result of the current time and the detection frame geometric information in the trajectory tracking result matched from the historical time to obtain the spatial position matching matrix; projecting the three-dimensional target detection frame to the multi-view two-dimensional image, and performing appearance feature cosine distance calculation of the target trajectory and the trajectory tracking result of the historical time to obtain the appearance feature matching matrix; and adding and fusing the spatial position matching matrix and the appearance feature matching matrix to obtain the final relationship measurement matrix of the current time target and the trajectory tracking result of the historical time.

[0018] Further, in the tracking trajectory repair processing process, for the target that is tracked in the scene and is re-identified, the embodiment of the present application uses the appearance feature of the current object and the appearance feature matching of the historical trajectory to be restored for trajectory repair, avoids the situation that the trajectory of the same object is broken, improves the trajectory integrity to obtain more continuous and complete tracking results.

[0019] To achieve the above purpose, another aspect of the present application provides a target tracking device fusing multi-view images and three-dimensional point clouds, comprising:

[0020] A feature extraction module is configured to obtain a two-dimensional feature map and a point cloud feature map based on the multi-view two-dimensional image and the three-dimensional point cloud, respectively;

[0021] A target detection module is configured to input the multi-modal feature map obtained by fusing the two-dimensional feature map and the point cloud feature map into a pre-trained multi-modal fusion target detection model to obtain a target detection frame; wherein the target detection frame comprises a three-dimensional target detection result of a target trajectory.

[0022] A feature fusion module is configured to obtain a spatial position matching matrix according to the comparison result of the target detection frame and the detection frame of the trajectory tracking result of the historical time, and obtain an appearance feature matching matrix by calculating the two-dimensional image features of the target trajectory and the trajectory tracking result of the historical time, and aggregate the spatial position matching matrix and the appearance feature matching matrix to obtain a final relationship measurement matrix.

[0023] The matching output module is configured to match the three-dimensional target detection result and the track tracking result at the historical moment based on the final relationship degree matrix and a preset matching algorithm, and obtain a tracking trajectory result of the target trajectory at the current moment according to a matching result.

[0024] The target tracking method and device for fusing multi-view images and three-dimensional point clouds can greatly eliminate problems such as trajectory exchange and target loss in a single modal tracking result, and obtain a target long-period tracking trajectory with high accuracy, good continuity and strong robustness, thereby providing convenience for scene perception and security monitoring.

[0025] The present application has the following advantages:

[0026] 1) Compared with a conventional single modal multi-target recognition method, the present application fuses two-dimensional images with rich appearance information and three-dimensional point clouds with accurate spatial information, and performs fusion of feature levels of the two modalities in a bird's eye view, thereby greatly improving the performance of target recognition and detection in a scene.

[0027] 2) A traditional three-dimensional multi-target tracking algorithm based on point clouds only uses spatial information of detection to obtain a tracking trajectory in a tracking stage, and cannot distinguish different targets from appearance. The present application introduces appearance features of a target in a multi-view image in the tracking stage, calculates a similarity of the appearance, enhances a distinguishing degree of matching of different targets, and thereby obtains a more accurate, continuous and robust tracking trajectory.

[0028] 3) For an object that disappears in a scene and then reappears, the present application matches appearance features of the current object and appearance features of a historical to-be-restored trajectory, performs trajectory restoration, avoids a situation that a trajectory of the same object is broken, and obtains a more continuous and complete tracking result.

[0029] 4) One of typical application scenarios of the present application is to obtain information of a player in motion. By using multi-view multi-modal data, accurate positions of players on a court and tracking trajectories can be obtained, and data such as a running distance and a speed of the players can be calculated, so as to facilitate event analysis.

[0030] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0031] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0032] Figure 1 is a flowchart of a target tracking method for fusing multi-view images and three-dimensional point clouds according to an embodiment of the present application;

[0033] Figure 2 is a target tracking method architecture diagram of fusing multi-view images and three-dimensional point clouds according to an embodiment of the present application;

[0034] Figure 3 is a target tracking architecture diagram based on spatial information and appearance features according to an embodiment of the present application;

[0035] Figure 4 is a structural schematic diagram of a target tracking device fusing multi-view images and three-dimensional point clouds according to an embodiment of the present application. DETAILED DESCRIPTION

[0036] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0037] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should belong to the scope of protection of the present application.

[0038] The target tracking method and device fusing multi-view images and three-dimensional point clouds according to the embodiments of the present application will be described below with reference to the accompanying drawings.

[0039] Figure 1 is a flowchart of the target tracking method fusing multi-view images and three-dimensional point clouds according to an embodiment of the present application.

[0040] As shown in Figure 1 , the method includes but is not limited to the following steps:

[0041] S1, obtaining a two-dimensional feature map and a point cloud feature map based on a multi-view two-dimensional image and a three-dimensional point cloud, respectively;

[0042] S2, inputting a multi-modal feature map obtained by fusing the two-dimensional feature map and the point cloud feature map into a pre-trained multi-modal fusion target detection model for detection to obtain a target detection frame; wherein the target detection frame includes a three-dimensional target detection result of a target trajectory;

[0043] S3, obtaining a spatial position matching matrix according to a comparison result of the target detection frame and a detection frame of a historical time trajectory tracking result, and calculating an appearance feature matching matrix based on a two-dimensional image feature of the target trajectory and the historical time trajectory tracking result, and aggregating the spatial position matching matrix and the appearance feature matching matrix to obtain a final relationship measurement matrix;

[0044] S4, matching the three-dimensional target detection result and the track tracking result at the historical moment based on the final relationship measurement matrix and a preset matching algorithm, and obtaining a tracking trajectory result of the target trajectory at the current moment according to a matching result.

[0045] The target tracking method fusing multi-view images and three-dimensional point clouds according to the embodiment of the application uses appearance feature matching of a current object and a historical to-be-restored trajectory to perform trajectory repair. The three-dimensional target tracking method can greatly eliminate problems such as trajectory exchange and target loss in a single modal tracking result, and obtain a target long-period tracking trajectory with high accuracy, good continuity and strong robustness, thereby providing convenience for scene perception and security monitoring.

[0046] The target tracking method fusing multi-view images and three-dimensional point clouds according to the embodiment of the application will be described in detail below with reference to the accompanying drawings.

[0047] First, the following symbols appearing below will be described: t: the current moment of algorithm processing, S: a relationship measurement matrix, S1: a spatial position image matrix, S2: an appearance feature matching matrix, An image block of the target of the i th camera at the t th moment, An appearance feature of the image block of the target of the i th camera at the t th moment.

[0048] Specifically, the application is divided into two stages in the whole, which are obtaining a three-dimensional target detection result and target tracking based on spatial information and appearance features. As shown in Figure 2 and Figure 3 .

[0049] First stage: multi-view multi-modal three-dimensional target detection.

[0050] It can be understood that for the multi-view multi-modal three-dimensional target detection task, the application simultaneously inputs two-dimensional images and three-dimensional point clouds of multiple views in a scene to perform feature extraction and fusion. For the two-dimensional images, two-dimensional feature extraction networks are used to obtain image features from multiple views, and then the feature maps are projected to a three-dimensional bird's eye view according to different camera parameters to perform splicing and fusion, thereby obtaining a multi-view fused two-dimensional feature map; for the three-dimensional point cloud, the point clouds from different views of the laser radar are directly fused after spatiotemporal registration to obtain a multi-angle dense point cloud, and then a three-dimensional feature extraction network is used to obtain a point cloud feature map. The two-dimensional feature map and the point cloud feature map are spliced and fused according to the spatial position to obtain a multi-modal feature map, and then a detection head is used to obtain an accurate three-dimensional target detection result.

[0051] Specifically, a multi-modal fusion target detection deep learning network is constructed. The point cloud data and the camera data have modal differences, and the present application uses different feature extraction networks and backbone networks to process the two kinds of data. The obtained two kinds of features are spliced and fused according to the spatial position to obtain a multi-modal feature map, and then a detection head is used to obtain an accurate three-dimensional target detection result.

[0052] As an example, a point cloud processing network is constructed. The point cloud processing network designed by the present application includes two parts: a feature extraction network and a backbone network, wherein the point cloud feature extraction network is used for point cloud feature coding, and the backbone network is used for further processing of the extracted features.

[0053] Specifically, the feature extraction network is used to convert the point cloud into a pseudo image and extract the point cloud features. First, the input point cloud is divided into multiple units, each unit being a 3D grid obtained by dividing the point cloud in the X-Y plane (Cartesian coordinate system) at a certain step size. Due to the sparsity of point cloud data, many units may not contain point clouds or contain a small number of point clouds. In view of the problem of computational complexity, the number of units is limited to a maximum of P non-empty units, and each unit contains a maximum of N point cloud feature vectors. If the number of point clouds is greater than N, N point clouds are selected by random sampling. If the number of point clouds is less than N, zero padding is used to fill N point clouds. Through the above method, a frame of point cloud data is encoded into a dense tensor with dimensions (, P, N). Next, a three-dimensional convolutional network is used for processing to generate a tensor with dimensions (, P, N). Then, maximum pooling is performed on each unit to obtain a tensor with dimensions (, P). Finally, a (C,, ) pseudo image is generated by a scatter operator.

[0054] As an example, the backbone network is composed of a 2D convolutional neural network, which is used to extract high-dimensional features from the pseudo image output by the feature extraction network. The backbone network is divided into two sub-networks: a top-down sub-network for extracting features from smaller and smaller spatial resolution feature maps, and another sub-network for upsampling the features extracted from different resolution feature maps to the same dimension size through deconvolution operation and then concatenating them.

[0055] As an example, an image processing network is constructed. The input of the image processing network is multiple images captured by cameras with different viewing angles. A convolutional neural network is used to process the input images and extract features. The convolutional neural network calculates the C-channel feature maps of the M input images respectively, and shares the weights in all calculations. In order to maintain a high spatial resolution of the feature maps, the last three layers of convolution are replaced by dilated convolution. Before projection, the size of the N feature maps is adjusted to a fixed size [H, W] (H and W represent the height and width of the feature map).

[0056] Further, the multi-camera information aggregation is performed by using the projection transformation of the feature map, and the multi-view feature maps are projected to a three-dimensional bird's eye view for splicing and fusion. According to the camera parameters, the corresponding relationship between the picture pixels and the coordinates on the ground can be obtained. Through the correspondence between a set of ground coordinates (z = 0) and a set of image pixels, the feature map can be projected. In order to strengthen the perception of the network to the spatial position, the spliced and fused Mx C+2 channel bird's eye view feature map is obtained by splicing and fusing the plurality of projected feature maps and X-Y coordinate maps. The multi-modal feature map is obtained by splicing and fusing the two-dimensional feature map and the point cloud feature map according to the spatial position, and the accurate three-dimensional target detection result is obtained through a detection head.

[0057] Further, the training and inference of the multi-modal fusion target detection model. The multi-view multi-modal data is collected by using the multi-modal data acquisition system, the three-dimensional target frame is manually labeled for the collected data, and the multi-modal fusion target detection model is trained by using the labeled data. In the inference, the multi-view image and point cloud data to be tracked are obtained by using the similar multi-modal data acquisition system, and the three-dimensional target detection result of each prediction time in the point cloud and image sequence is obtained by using the above trained multi-modal fusion target detection deep learning network.

[0058] The second stage is target tracking based on spatial information and appearance features.

[0059] Specifically, the three-dimensional target detection result of the current prediction time to be tracked in the first stage is obtained, and the trajectory tracking result of the previous time is matched. The intersection over union of the target and the tracking result of the previous time is calculated as a spatial relationship measure. The three-dimensional target detection frame is projected to the input multi-view two-dimensional image, and the cosine distance of the appearance features of the target and the trajectory tracking result of the previous time is calculated as a relationship measure on the appearance. After the average aggregation of the two relationship measure matrices, the Hungarian matching algorithm is used to match the detection result of the current time and the trajectory tracking result of the previous time, and finally the tracking trajectory of the target at the current time is obtained.

[0060] Further, the current prediction time is represented by t time, and the previous time is represented by t-1 time. The distance intersection ratio is used as the relationship measure between the detection frames in the present application. The number of detection frames of the t time detection result is m, and the number of detection frames in the tracking result matched at the t-1 time is n, so the matching score calculated is a matrix of m*n, which is represented by S1.

[0061] Further, based on the accurate three-dimensional spatial information of the three-dimensional point cloud, the fusion multi-modal detection result obtained in the first stage has accurate spatial information, but the detection boxes of close and clustered targets may overlap, and only using three-dimensional spatial positions for tracking matching may easily lead to errors in trajectory matching. In addition to the spatial information, the application uses multi-view two-dimensional images to extract rich appearance features, calculates the cosine distance between the appearance features of different targets, and improves the accuracy of tracking matching.

[0062] As an example, the application projects the three-dimensional detection box obtained in the first stage onto a two-dimensional image using a projection matrix from the world coordinate system to the camera coordinate system to obtain a two-dimensional detection box of the target on the image. The obtained two-dimensional image detection box is subjected to image cropping to obtain an image block of each target at different viewing angles. For a target of the i-th camera at time t, an image block of the target is obtained A two-dimensional image feature extraction deep learning network is used to calculate the appearance features of the image block For each target obtained in the first stage, the application extracts the appearance features of the target at k viewing angles and the K appearance features of the historical time recorded by the trajectory tracking result of the previous time The cosine similarity is calculated. For a pair of detection targets at time t and time t-1, a k*k size cosine similarity matrix can be calculated, and finally the appearance feature similarity is obtained by matching score screening and averaging as the matching score of the appearance feature.

[0063] If the number of detection boxes of the detection result obtained in the first stage at time t is m, and the number of detection boxes in the tracking result obtained by matching at time t-1 is n, then the calculated matching score is a matrix of m*n, denoted as S2.

[0064] Further, the spatial position matching matrix S1 and the appearance feature matching matrix S2 obtained above are added and fused to obtain the final relationship measurement matrix S of the matching of the target at the current time and the tracking result at the previous time. The application uses a bipartite matching algorithm to connect the detection target at the current time and the tracking trajectory. After completing the trajectory matching connection at time t, the appearance features of the target at time t on the two-dimensional image obtained above are added to the historical appearance feature library of the tracking trajectory to which the target belongs, for use in the matching at the next time.

[0065] Further, the tracking trajectory repair process, due to the departure of the target from the scene or the occlusion of the target by an obstacle, the target tracking trajectory is easily lost, and when the target detection returns to normal, the historical trajectory should be restored. However, the traditional tracking algorithm often regards it as a new target and assigns a new id, so the tracking algorithm needs to repair the reappeared tracking trajectory. The specific method is as follows:

[0066] After the matching of the next time point is completed, three kinds of trajectories are obtained, the first kind is the trajectory successfully matched and connected in the detection result of the t time point, the second kind is the historical trajectory that fails to find a matching pair in the detection result of the t time point, and the third kind is the detection result of the t time point without matching.

[0067] For the historical trajectory segment that fails to successfully obtain matching, it is not directly considered that the trajectory immediately disappears, but the corresponding appearance feature of the trajectory is continued to be retained as a historical trajectory to be recovered.

[0068] For the detection target of the t time point without successful matching, the appearance feature relationship measurement method is used to calculate the appearance feature score between the unmatched detection target and the historical trajectory to be recovered, if the score is greater than a threshold, it is considered that the detection target and the historical trajectory to be recovered belong to the same object, and the two are connected, the reappeared target tracking trajectory is repaired, and the continuity of the same object trajectory is improved.

[0069] According to the target tracking method for fusing multi-view images and three-dimensional point clouds, the problems of trajectory exchange and target loss in single modal tracking results can be greatly eliminated, and the target long-period tracking trajectory with high accuracy, good continuity and strong robustness is obtained, which provides convenience for scene perception and security monitoring.

[0070] In order to realize the above-mentioned embodiments, as shown in the figure, the embodiment further provides a target tracking device 10 for fusing multi-view images and three-dimensional point clouds. Figure 4 The device 10 includes a feature extraction module 100, a target detection module 200, a feature fusion module 300 and a matching output module 400.

[0071] The feature extraction module 100 is used to obtain a two-dimensional feature map and a point cloud feature map based on a multi-view two-dimensional image and a three-dimensional point cloud respectively.

[0072] The target detection module 200 is used to input the multi-modal feature map obtained by fusing the two-dimensional feature map and the point cloud feature map to a pre-trained multi-modal fusion target detection model for detection to obtain a target detection frame; wherein the target detection frame includes a three-dimensional target detection result of a target trajectory.

[0073] The feature fusion module 300 is used to obtain a spatial position matching matrix according to the comparison result of the target detection frame and the detection frame of the historical time point trajectory tracking result, and calculate the appearance feature matching matrix of the target trajectory and the two-dimensional image feature of the historical time point trajectory tracking result, and aggregate the spatial position matching matrix and the appearance feature matching matrix to obtain a final relationship measurement matrix.

[0074] The matching output module 400 is configured to match the three-dimensional target detection result and the track tracking result at the historical moment based on the final relationship degree matrix and a preset matching algorithm, and obtain the tracking trajectory result of the target trajectory at the current moment according to the matching result.

[0075] Further, the feature extraction module 100 is further configured to:

[0076] acquire the multi-view two-dimensional image and the three-dimensional point cloud;

[0077] extract the multi-view two-dimensional image to obtain an initial feature map by using an image processing network, project the initial feature map to a three-dimensional bird's eye view according to different camera parameters, perform splicing and fusion to obtain a multi-view fused two-dimensional feature map, and

[0078] fuse the multi-angle dense point cloud obtained by spatiotemporal registration of the three-dimensional point cloud by using a point cloud processing network to obtain a multi-angle dense point cloud, and extract a point cloud feature map from the multi-angle dense point cloud.

[0079] Further, the point cloud processing network includes a feature extraction network and a backbone network; the backbone network includes a first subnetwork and a second subnetwork; the image processing network includes a convolutional neural network, and the feature extraction module is further configured to:

[0080] input the multi-view three-dimensional point cloud into the feature extraction network to perform point cloud conversion to obtain a pseudo image, input the pseudo image into the first subnetwork to perform feature extraction on different spatial resolution feature maps, input the features extracted from the different spatial resolution feature maps into the second subnetwork to perform deconvolution operation and then concatenate to obtain the point cloud feature map, and

[0081] input the multi-view two-dimensional image into the convolutional neural network to calculate a multi-channel feature map, perform multi-camera information aggregation by using projection transformation of the multi-channel feature map, project the multi-channel feature map to a three-dimensional bird's eye view to perform splicing and fusion to obtain a two-dimensional feature map.

[0082] Further, the device 10 further includes a model pre-training module configured to:

[0083] acquire sample data of the multi-modal feature map, perform data labeling on a target detection frame by using the multi-sample data, and train the multi-modal fusion target detection model by using the labeled data;

[0084] input the two-dimensional image and the three-dimensional point cloud to be tracked into the trained multi-modal fusion target detection model to obtain a three-dimensional target detection result at each predicted moment.

[0085] Further, the device 10 further includes a matrix calculation module configured to:

[0086] The matching score is calculated based on the geometric information of the detection frame of the three-dimensional target detection result at the current time and the geometric information of the detection frame in the trajectory tracking result matched at the historical time, so as to obtain a spatial position matching matrix;

[0087] The three-dimensional target detection frame is projected to the multi-view two-dimensional image, and the appearance feature cosine distance is calculated between the target trajectory and the trajectory tracking result at the historical time, so as to obtain an appearance feature matching matrix; and

[0088] The spatial position matching matrix and the appearance feature matching matrix are added and fused, so as to obtain a final relationship measurement matrix of the matching between the target at the current time and the trajectory tracking result at the historical time.

[0089] The target tracking device for fusing multi-view images and three-dimensional point clouds according to the embodiment of the application can greatly eliminate the problems such as trajectory exchange and target loss in the single modal tracking result, and obtain a target long-period tracking trajectory with high accuracy, good continuity and strong robustness, thereby providing convenience for scene perception and security monitoring.

[0090] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the different embodiments or examples described in the present specification and the features of the different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0091] In addition, the terms "first", "second" are only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

Claims

1. A target tracking method of fusing multi-view images and three-dimensional point clouds, characterized by, The method comprises the following steps: obtaining a two-dimensional feature map and a point cloud feature map based on multi-view two-dimensional images and three-dimensional point clouds respectively; inputting a multi-modal feature map obtained by fusing the two-dimensional feature map and the point cloud feature map into a pre-trained multi-modal fusion target detection model to obtain a target detection frame through detection; wherein the target detection frame comprises a three-dimensional target detection result of a target trajectory; obtaining a spatial position matching matrix according to a comparison result of the target detection frame and a detection frame of a historical time trajectory tracking result, and obtaining an appearance feature matching matrix by calculating a two-dimensional image feature of the target trajectory and the historical time trajectory tracking result, and aggregating the spatial position matching matrix and the appearance feature matching matrix to obtain a final relationship measurement matrix; matching the three-dimensional target detection result and the historical time trajectory tracking result based on the final relationship measurement matrix and a preset matching algorithm, and obtaining a tracking trajectory result of the target trajectory at the current time according to a matching result; the method comprises the following steps: obtaining a two-dimensional feature map and a point cloud feature map based on multi-view two-dimensional images and three-dimensional point clouds respectively; obtaining a two-dimensional feature map and a point cloud feature map based on multi-view two-dimensional images and three-dimensional point clouds respectively; obtaining a two-dimensional feature map and a point cloud feature map based on multi-view two-dimensional images and three-dimensional point clouds respectively; the point cloud processing network comprises a feature extraction network and a backbone network; the backbone network comprises a first sub-network and a second sub-network; the image processing network comprises a convolutional neural network, and the method further comprises: inputting the multi-view three-dimensional point cloud into the feature extraction network to obtain a pseudo image through point cloud conversion, inputting the pseudo image into the first sub-network to extract features of different spatial resolution feature maps, inputting the features extracted by the different spatial resolution feature maps into the second sub-network to perform deconvolution operation and then concatenate to obtain the point cloud feature map; and inputting the multi-view two-dimensional image into the convolutional neural network to calculate a multi-channel feature map, using projection transformation of the multi-channel feature map to aggregate multi-camera information, and projecting the multi-channel feature map to a three-dimensional bird's eye view to obtain the two-dimensional feature map through splicing and fusion; the method further comprises: obtaining sample data of the multi-modal feature map, using the sample data to perform data labeling of the target detection frame, and using the labeled data to train the multi-modal fusion target detection model; inputting the two-dimensional image and the three-dimensional point cloud to be tracked into the trained multi-modal fusion target detection model to detect the target trajectory, and obtaining a three-dimensional target detection result at each predicted time; the method further comprises: performing matching score calculation according to detection frame geometric information of the three-dimensional target detection result at the current time and detection frame geometric information in the trajectory tracking result matched at the historical time, to obtain the spatial position matching matrix; projecting the three-dimensional target detection frame to the multi-view two-dimensional image, and performing appearance feature cosine distance calculation on the target trajectory and the trajectory tracking result at the historical moment to obtain an appearance feature matching matrix; and adding and fusing the spatial position matching matrix and the appearance feature matching matrix to obtain a final relationship measurement matrix of the target at the current moment and the trajectory tracking result at the historical moment.

2. A target tracking apparatus that fuses a multi-view image and a three-dimensional point cloud, characterized by, Comprise: The feature extraction module is used for obtaining two-dimensional feature maps and point cloud feature maps based on multi-view two-dimensional images and three-dimensional point clouds respectively; The target detection module is used for inputting the multi-modal feature map obtained after fusing the two-dimensional feature map and the point cloud feature map into the pre-trained multi-modal fusion target detection model for detection to obtain a target detection frame; wherein the target detection frame comprises a three-dimensional target detection result of a target trajectory; The feature fusion module is used for obtaining a spatial position matching matrix according to a comparison result of the target detection frame and the detection frame of the trajectory tracking result at the historical moment, and obtaining an appearance feature matching matrix by calculating the two-dimensional image features of the target trajectory and the trajectory tracking result at the historical moment, and aggregating the spatial position matching matrix and the appearance feature matching matrix to obtain a final relationship measurement matrix; The matching output module is used for matching the three-dimensional target detection result and the trajectory tracking result at the historical moment based on the final relationship measurement matrix and a preset matching algorithm, and obtaining a tracking trajectory result of the target trajectory at the current moment according to the matching result; The feature extraction module is also used for: Obtaining multi-view two-dimensional images and three-dimensional point clouds; Using an image processing network to extract the multi-view two-dimensional images to obtain initial feature maps, projecting the initial feature maps to a three-dimensional bird's eye view according to different camera parameters for splicing and fusion to obtain multi-view fused two-dimensional feature maps; and Using a point cloud processing network to fuse the three-dimensional point clouds after spatio-temporal registration to obtain multi-angle dense point clouds, and extracting features of the multi-angle dense point clouds to obtain a point cloud feature map; The point cloud processing network comprises a feature extraction network and a backbone network; the backbone network comprises a first sub-network and a second sub-network; the image processing network comprises a convolutional neural network; the feature extraction module is also used for: Inputting the multi-view three-dimensional point clouds into the feature extraction network to convert the point clouds to obtain pseudo images, inputting the pseudo images into the first sub-network to extract features of different spatial resolution feature maps, inputting the features extracted from the different spatial resolution feature maps into the second sub-network for deconvolution operation and then concatenating to obtain the point cloud feature map; and Inputting the multi-view two-dimensional images into the convolutional neural network to calculate multi-channel feature maps, using projection transformation of the multi-channel feature maps to aggregate multi-camera information, and projecting the multi-channel feature maps to a three-dimensional bird's eye view for splicing and fusion to obtain the two-dimensional feature maps; The device further comprises a model pre-training module, which is used for: Obtaining sample data of multi-modal feature maps, performing data labeling of target detection frames using the sample data, and training the multi-modal fusion target detection model using the labeled data; Input the two-dimensional image and the three-dimensional point cloud to be tracked into the trained multi-modal fusion target detection model to obtain a three-dimensional target detection result at each prediction time; The device further comprises a matrix calculation module configured to: Calculate a matching score based on the bounding box geometric information of the three-dimensional target detection result at the current time and the bounding box geometric information in the trajectory tracking result obtained by matching the historical time, to obtain the spatial position matching matrix; Project the three-dimensional target detection box to the multi-view two-dimensional image, and calculate the appearance feature cosine distance between the target trajectory and the trajectory tracking result at the historical time to obtain an appearance feature matching matrix; and Add the spatial position matching matrix and the appearance feature matching matrix to obtain a final relationship measurement matrix of the current target and the trajectory tracking result at the historical time.

Images