Multi-view image tracking method, device and equipment, storage medium and vehicle

Abstract

The invention relates to a multi-view image tracking method and device, equipment, a storage medium and a vehicle, and the method comprises the steps: obtaining at least two groups of image sequences which are acquired from at least two views at the same time; for each group of image sequences in the at least two groups of image sequences, performing tracking processing on each object contained in each frame of image in each group of image sequences to obtain a first tracking result of each object corresponding to each group of image sequences; and carrying out fusion processing on the plurality of first tracking results corresponding to the at least two groups of image sequences to obtain second tracking results of each object under the at least two visual angles. According to the method, object tracking is carried out in the single-view image sequence, and then multiple tracking results of the multi-view image sequence are fused, so that multi-view multi-target tracking can be realized without three-dimensional labeling, the requirement on the computing power is relatively low, the configuration threshold of the function is reduced, and meanwhile, the overall processing speed is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image technology, and in particular to a multi-view image tracking method, device, equipment, storage medium and vehicle. Background Art

[0002] Multiple Object Tracking (MOT) is a key common technology in application scenarios such as autonomous driving, smart transportation, and smart security. Its purpose is to associate the object frames detected in the scene in time sequence to form a trajectory, and to track the real-time position of the object. Multi-view MOT can obtain richer and more comprehensive information. Existing multi-view MOT often uses bird's eye view (BEV) visual images to process multi-target tracking results. It is necessary to annotate the three-dimensional detection frames corresponding to the objects in the bird's eye view visual images, which requires high computing power. Therefore, how to reduce the requirements of multi-view MOT on computing resources is a technical problem that needs to be solved. Summary of the invention

[0003] In order to solve the above technical problems, the present disclosure provides a multi-view image tracking method, device, equipment, storage medium and vehicle.

[0004] A first aspect of an embodiment of the present disclosure provides a multi-view image tracking method, the method comprising:

[0005] Acquire at least two sets of image sequences, where the at least two sets of image sequences are simultaneously acquired from at least two viewing angles;

[0006] For each of the at least two groups of image sequences, tracking processing is performed on each object contained in each frame of the image in each group of image sequences to obtain a first tracking result of each object corresponding to each group of image sequences;

[0007] A plurality of first tracking results corresponding to the at least two groups of image sequences are fused to obtain second tracking results of the respective objects under the at least two viewing angles.

[0008] A second aspect of an embodiment of the present disclosure provides a multi-view image tracking device, the device comprising:

[0009] An acquisition module, used to acquire at least two sets of image sequences, wherein the at least two sets of image sequences are acquired simultaneously from at least two viewing angles;

[0010] A tracking module, configured to perform tracking processing on each object contained in each frame of each image sequence in the at least two image sequences, and obtain a first tracking result of each object corresponding to each image sequence;

[0011] The fusion module is used to fuse the multiple first tracking results corresponding to the at least two groups of image sequences to obtain the second tracking results of each object under the at least two viewing angles.

[0012] A third aspect of an embodiment of the present disclosure provides a computer device, comprising a memory and a processor, and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the multi-view image tracking method as described in the first aspect above is implemented.

[0013] A fourth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the multi-view image tracking method as described in the first aspect above is implemented.

[0014] A fifth aspect of an embodiment of the present disclosure provides a vehicle, comprising a memory and a processor, and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the multi-view image tracking method as described in the first aspect above is implemented.

[0015] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0016] In the multi-view image tracking method, device, equipment, storage medium and vehicle provided in the embodiments of the present disclosure, at least two groups of image sequences are obtained by acquiring the at least two groups of image sequences, which are acquired simultaneously from at least two viewpoints; for each group of image sequences in the at least two groups of image sequences, each object contained in each frame of the image sequences is tracked to obtain a first tracking result for each object corresponding to the image sequences; and a plurality of first tracking results corresponding to the at least two groups of image sequences are fused to obtain a second tracking result for each object under the at least two viewpoints. By performing multi-target tracking on objects in the multiple groups of images obtained from different viewpoints, multi-target tracking from multiple viewpoints can be achieved without the need to annotate the three-dimensional detection frame, and the computing power requirement is relatively low, which can reduce the configuration threshold of the function and improve the overall processing speed, thereby obtaining tracking results faster and meeting the application requirements of scenes with high requirements for tracking timeliness. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 is a flow chart of a multi-view image tracking method provided by an embodiment of the present disclosure;

[0020] Figure 2 is a flow chart of a single-view image tracking method provided by an embodiment of the present disclosure;

[0021] Figure 3 is a flow chart of a method for matching detection frames provided by an embodiment of the present disclosure;

[0022] Figure 4 is a flow chart of a method for removing a detection frame provided by an embodiment of the present disclosure;

[0023] Figure 5 is a flowchart of a method for fusing multi-view tracking results provided by an embodiment of the present disclosure;

[0024] Figure 6 is a flow chart of a method for optimizing a similarity matrix provided by an embodiment of the present disclosure;

[0025] Figure 7 is a structural schematic diagram of a multi-view image tracking device provided by an embodiment of the present disclosure;

[0026] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0028] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] Figure 1 FIG. 1 is a flowchart of a multi-view image tracking method provided by an embodiment of the present disclosure, and the method can be executed by a multi-view image tracking device. Figure 1 As shown, the multi-view image tracking method provided in this embodiment includes the following steps:

[0033] S101. Acquire at least two sets of image sequences, where the at least two sets of image sequences are simultaneously acquired from at least two viewing angles.

[0034] The image sequence in the embodiment of the present disclosure can be understood as an image sequence with a time-series relationship between image frames. For example, the image sequence can be obtained by frame processing of the original video data collected by the camera, or it can be obtained by extracting frames from the original video according to certain rules, which is not limited here.

[0035] In the embodiment of the present disclosure, the multi-view image tracking device can obtain at least two groups of image sequences collected simultaneously from at least two viewpoints.

[0036] In an exemplary implementation of the disclosed embodiment, a multi-view image tracking device can obtain at least two groups of image sequences simultaneously captured by at least two cameras for capturing images from different viewpoints. The cameras and the image sequences can correspond one to one, and each camera is used to capture a group of image sequences. The at least two cameras can be installed at different positions. For example, four cameras are respectively installed at four positions on the outside of a vehicle, front, back, left, and right, to capture four groups of image sequences on the front, back, left, and right sides of the vehicle. Optionally, in order to increase the image capture viewing angle, the camera can use a fisheye lens.

[0037] S102: For each image sequence in the at least two image sequences, perform tracking processing on each object contained in each frame of the image sequence to obtain a first tracking result for each object corresponding to each image sequence.

[0038] The object in the embodiments of the present disclosure may be understood as an object to be tracked. For example, the object may be a human body, a vehicle, or other stationary or moving object, which is not limited here.

[0039] The tracking process in the disclosed embodiment can be understood as determining the position of the target tracked object at each time, and determining the movement trajectory of the object according to the position at each time. For multi-target tracking, an ID can be assigned to each of the multiple target tracked objects, and the position of the object corresponding to each ID at each time and the movement trajectory of the object corresponding to each ID can be determined.

[0040] The first tracking result in the embodiment of the present disclosure may be understood as tracking each object contained in each frame of the image sequence, and obtaining a tracking result including movement trajectories of each object.

[0041] In the disclosed embodiment, the multi-view image tracking device can process each group of image sequences separately after obtaining at least two groups of image sequences of at least two viewpoints. Specifically, the multi-view image tracking device can track each object contained in each frame of the target processed image sequence separately, obtain the movement trajectory of each object in the current image sequence, and determine it as the first tracking result corresponding to the group of image sequences, so as to obtain the first tracking result corresponding to each group of image sequences after performing the above-mentioned processing on each group of image sequences.

[0042] In an exemplary implementation of the disclosed embodiment, the multi-view image tracking device may input each group of image sequences into a pre-trained tracking model respectively, and the tracking model may track each object contained in each frame image in each group of image sequences to obtain a first tracking result corresponding to each group of image sequences. The tracking model may be based on a multi-domain learning network (MDNet), a fully convolutional network (FCN) or other network structures, which are not limited here.

[0043] In another exemplary implementation of the disclosed embodiment, the multi-view image tracking device may use a SORT (Simple Online and Realtime Tracking) algorithm to implement tracking processing of each object contained in each frame image in each group of image sequences, and obtain a first tracking result for each object corresponding to each group of image sequences.

[0044] In another exemplary implementation of the disclosed embodiment, the multi-view image tracking device may, after obtaining at least two groups of image sequences, perform format conversion on each frame image in each group of image sequences, from YUV format to RGB mode, and then perform subsequent tracking processing.

[0045] S103: Fusing the multiple first tracking results corresponding to the at least two groups of image sequences to obtain second tracking results for each object under the at least two viewing angles.

[0046] The second tracking result in the embodiment of the present disclosure can be understood as a tracking result determined by tracking and fusing each object contained in each frame image in multiple groups of image sequences, including the movement trajectory of each object under multiple viewing angles.

[0047] In the disclosed embodiment, the multi-view image tracking device can fuse the multiple first tracking results after obtaining the multiple first tracking results corresponding to at least two groups of image sequences. Specifically, the multi-view image tracking device can fuse two or more movement trajectories corresponding to objects that appear simultaneously in two or more groups of image sequences, so that in the final second tracking result, the same object corresponds to only one movement trajectory.

[0048] In an exemplary implementation of the embodiments of the present disclosure, the multi-view image tracking device can fuse and splice multiple scenes corresponding to multiple first tracking results, determine the actual position of each object in the multiple first tracking results in the world coordinate system based on the internal and external parameters of the camera, and then determine that objects located at the same actual position in different groups of image sequences are the same object, and fuse the movement trajectories belonging to the same object, so as to obtain second tracking results for each object contained in at least two groups of image sequences under at least two perspectives.

[0049] In another exemplary implementation of the disclosed embodiment, the multi-view image tracking device may select one of the multiple viewpoints as the main viewpoint, project the first tracking results corresponding to the other viewpoints to the main viewpoint, and use a Scale Invariant Feature Transform (SIFT) algorithm to detect and compensate for the dynamic projection deviation of the images between the multiple viewpoints to improve the accuracy of the dynamic projection, thereby fusing two or more movement trajectories corresponding to objects that appear simultaneously in two or more image sequences, so that in the final second tracking result, the same object corresponds to only one movement trajectory.

[0050] The disclosed embodiment acquires at least two sets of image sequences, which are acquired from at least two perspectives simultaneously; for each set of image sequences in the at least two sets of image sequences, tracks each object contained in each frame of the image sequences to obtain a first tracking result for each object corresponding to the image sequences; and fuses multiple first tracking results corresponding to the at least two sets of image sequences to obtain a second tracking result for each object under the at least two perspectives. By performing multi-target tracking on objects in multiple sets of images obtained from different perspectives, multi-target tracking from multiple perspectives can be achieved without the need for annotating three-dimensional detection frames, and the computing power requirements are relatively low, which can reduce the configuration threshold of the function and improve the overall processing speed, thereby obtaining tracking results faster and meeting the application requirements of scenes with high requirements for tracking timeliness.

[0051] Figure 2 is a flow chart of a method for single-view image tracking provided by an embodiment of the present disclosure, such as Figure 2 As shown, based on the above embodiment, single-view image tracking can be performed by the following method.

[0052] S201: Perform image detection on each frame of the image in each group of image sequences to obtain an actual detection frame.

[0053] The detection frame in the embodiment of the present disclosure can be understood as a rectangular frame set to identify the position of the target detection object in the image. The detection frame can be circumscribed to the target detection object, and the actual detection frame can be understood as a detection frame used to identify the actual position of the target detection object in the image.

[0054] In the embodiment of the present disclosure, after obtaining at least two groups of image sequences, the multi-view image tracking device can perform image detection on the object of target detection for each frame image in each group of image sequences, wherein the object of target detection can be set to human body, vehicle, football, animal, etc. as needed, and no limitation is made here, to obtain an actual detection frame for identifying the position of the object of target detection in the image.

[0055] In an exemplary implementation of the disclosed embodiments, a multi-view image tracking device may input each frame of an image in each group of image sequences into a preset deep learning-based image detection model, wherein the detection model is trained based on pre-input image data containing annotation results, and the model can detect the input image and obtain the actual identification box of the target detection object in the image.

[0056] In another exemplary implementation of the disclosed embodiment, the multi-view image tracking device may obtain the coordinates of the actual detection frame of the target detected object in the image after performing image detection on each frame of the image.

[0057] S202: Extract features of each object in the actual detection frame to obtain a first feature.

[0058] In the disclosed embodiment, the multi-view image tracking device can extract features of each object inside the actual detection frame after determining the actual detection frame in each image, and obtain the first feature corresponding to each actual detection frame.

[0059] In an exemplary implementation of the disclosed embodiment, the multi-view image tracking device may use a re-identification (Re-identification, ReID) model to extract features of each object in the actual detection frame to obtain a first feature, wherein the ReID model may use a residual neural network (Residual Network, ResNet) as a feature extraction network, calculate the joint loss of cross entropy loss, center loss, and ternary loss, and improve the feature discrimination.

[0060] S203 . Starting from the second frame image in each group of image sequences, based on the actual detection frame in the previous frame image, a Kalman filter is used to predict the predicted detection frame of each object in the current frame image.

[0061] The predicted detection box in the embodiment of the present disclosure may be understood as a detection box used to identify the predicted position of an object in an image.

[0062] In the disclosed embodiment, after obtaining the image sequence, the multi-view image tracking device can start from the second frame image in each group of image sequences and use Kalman filtering to predict the predicted detection frame of each object in the current frame image based on the actual detection frame detected in the previous frame image. Specifically, the predicted detection frame of each object in the current frame image can be predicted based on the target detection frame corresponding to the object in the previous frame image.

[0063] S204: In the current frame image, extract features of each object in the predicted detection box to obtain a second feature.

[0064] In the disclosed embodiment, the multi-view image tracking device can perform feature extraction on each object in the predicted detection box of the current frame image after determining the predicted detection box in the current frame image to obtain a second feature. The specific extraction method is similar to S202 and will not be repeated here.

[0065] S205: Perform feature matching on the first feature and the second feature extracted from the current frame image.

[0066] In the disclosed embodiment, the multi-view image tracking device may perform feature matching processing on the first features and the second features after obtaining the first features and the second features corresponding to the current frame image, and determine the second features that match the respective first features.

[0067] In an exemplary implementation of the disclosed embodiment, the multi-view image tracking device may use the Hungarian algorithm to perform feature matching on the first feature and the second feature.

[0068] S206. Determine the target detection frame corresponding to each object in the actual detection frame included in the current frame image based on the feature matching result, until the last frame image in each group of image sequences is matched, to obtain the first tracking result.

[0069] In the disclosed embodiment, the multi-view image tracking device can determine the target detection frame of the object corresponding to each second feature in the current frame image from multiple actual detection frames in the current frame image based on the matching result of the first feature and the second feature. Specifically, the actual detection frame to which the first feature matching the second feature belongs can be determined as the target detection frame of the object corresponding to the predicted detection frame to which the second feature belongs in the current frame image, and then continue to use Kalman filtering to predict the predicted detection frame of the object in the next frame image based on the target detection frame of the object in the current frame, extract the second feature therein and match it with the first feature of the actual detection frame in the next frame image, and so on, until the matching is completed for the last frame image in each group of image sequences, and obtain multiple target detection frames corresponding to each object at each time in each frame image in each group of image sequences, and then obtain the first tracking result of each object contained in each frame image in each group of image sequences.

[0070] The disclosed embodiment performs image detection on each frame image in each group of image sequences to obtain an actual detection frame; extracts features from each object in the actual detection frame to obtain a first feature; starting from the second frame image in each group of image sequences, based on the actual detection frame in the previous frame image, Kalman filtering is used to predict the predicted detection frame of each object in the current frame image; in the current frame image, features are extracted from each object in the predicted detection frame to obtain a second feature; feature matching is performed on the first feature extracted from the current frame image and the second feature; in the actual detection frame contained in the current frame image, the target detection frame corresponding to each object is determined based on the feature matching result, until the last frame image in each group of image sequences is matched to obtain the first tracking result. Through feature matching, the tracking processing of multiple objects contained in each frame image in a single-view image sequence can be realized, thereby improving the tracking accuracy.

[0071] Figure 3is a flow chart of a method for matching detection frames provided by an embodiment of the present disclosure, such as Figure 3 As shown, based on the above embodiment, the detection frame can be matched by the following method.

[0072] S301: Perform intersection-over-union matching on the actual detection box and the predicted detection box corresponding to the first feature and the second feature for which feature matching fails.

[0073] In the disclosed embodiment, after feature matching is performed on the first feature and the second feature, the multi-view image tracking device can determine, for the first feature and the second feature that are successfully matched, the target detection frame of the object corresponding to each second feature in the current frame image; for the first feature and the second feature that are unmatched, the intersection-and-union ratio of the actual detection frame corresponding to each first feature and the predicted detection frame corresponding to each second feature is calculated, and the matching first feature and second feature are determined based on the intersection-and-union ratio.

[0074] In an exemplary implementation of the disclosed embodiment, the multi-view image tracking device may use the Hungarian algorithm to perform intersection-and-union matching on the actual detection frame and the predicted detection frame corresponding to the first feature and the second feature that failed to match.

[0075] S302: In the actual detection frame included in the current frame image, determine the target detection frame corresponding to each object based on the feature matching result and the intersection-over-union matching result.

[0076] In the disclosed embodiment, the multi-view image tracking device can determine the target detection frame corresponding to each predicted detection frame from the actual detection frame contained in the current frame image based on the feature matching results and the intersection-and-union matching results. The specific determination method is similar to S206 and will not be repeated here.

[0077] In an exemplary implementation of the disclosed embodiment, for an actual detection frame in which both feature matching and IoU matching fail, it is determined that the object contained in the actual detection frame is a newly appeared object, a new ID is created for it, and its predicted detection frame in the next frame image is predicted based on the actual detection frame, and steps S203-S206 are executed. For the predicted detection frame in which both matching fails, it is determined that the object corresponding to the predicted detection frame has left the image acquisition area.

[0078] The disclosed embodiment performs intersection-over-union matching on the actual detection frame and predicted detection frame corresponding to the first feature and the second feature where feature matching fails; in the actual detection frame contained in the current frame image, the target detection frame corresponding to each object is determined based on the feature matching result and the intersection-over-union matching result. It is possible to further perform intersection-over-union matching on the basis of feature matching, reduce the missed detection rate, and further improve the tracking accuracy.

[0079] Figure 4 is a flow chart of a method for removing a detection frame provided by an embodiment of the present disclosure, such as Figure 4 As shown, based on the above embodiment, the detection frame can be removed by the following method.

[0080] S401: Determine the position of the target key point of each object in the actual detection frame in the world coordinate system.

[0081] The target key point in the embodiment of the present disclosure can be understood as a key structural point of an object. For example, for a human object, the target key point may be the head, feet, etc., and for a vehicle, the target key point may be the left front wheel, etc., which is not limited here.

[0082] In the disclosed embodiment, the multi-view image tracking device can, after detecting the actual detection frame in each frame of the image, further perform key point detection on the objects in the actual detection frame to determine the coordinates of the target key points of each object in the image, and then combine the intrinsic and extrinsic parameters of the camera that captures the image to perform a projection transformation on the target key points of each object in the image to obtain the position of the target key points of each object in the world coordinate system.

[0083] In an exemplary implementation of the disclosed embodiment, the multi-view image tracking device may use a pre-trained key point detection model to perform key point detection. The model may use a deep learning architecture to complete initial training through training images containing key point annotation results.

[0084] S402: Determine a first distance between a position of a target key point of each object in a world coordinate system and an image acquisition position of an image where the target key point is located.

[0085] In the disclosed embodiment, the multi-view image tracking device can determine the image acquisition position of the image where the target key points of each object are located in the world coordinate system based on the external parameters of the image acquisition device corresponding to the image where the target key points of each object are located, and then determine the first distance between the position of the target key point in the world coordinate system and the image acquisition position of the image where the target key point is located based on the position of the target key point in the world coordinate system obtained by projection transformation.

[0086] S403: Eliminate the actual detection frame corresponding to the object to which the target key point whose first distance is greater than the preset distance belongs.

[0087] The preset distance in the embodiment of the present disclosure can be understood as a pre-set distance threshold that is considered to be far away from the image acquisition device and does not need to be included in the tracking range. The specific value of the first distance can be set as needed.

[0088] In the embodiment of the present disclosure, the multi-view image tracking device can compare each first distance with a preset distance after determining the first distance corresponding to the target key point of each object, and remove the actual detection frame corresponding to the object to which the target key point whose first distance is greater than the preset distance belongs from the actual detection frame included in the current frame.

[0089] The disclosed embodiment determines the position of the target key points of each object in the actual detection frame in the world coordinate system; determines the first distance between the position of the target key points of each object in the world coordinate system and the image acquisition position of the image where the target key points are located; and removes the actual detection frame corresponding to the object to which the target key points belong whose first distance is greater than a preset distance. The tracking and monitoring distance around the image acquisition device can be freely set, which reduces the amount of calculation while meeting user needs, and further improves the overall processing speed.

[0090] Figure 5 is a flowchart of a method for fusing multi-view tracking results provided by an embodiment of the present disclosure. Figure 5 As shown, based on the above embodiment, the multi-view tracking result fusion can be performed by the following method.

[0091] S501. Extract a third feature of each object based on the multiple first tracking results.

[0092] The third feature in the embodiment of the present disclosure may be understood as a feature extracted from each frame of the image where each object is located.

[0093] In the disclosed embodiment, after obtaining a plurality of first tracking results, the multi-view image tracking apparatus may extract, for each object included in each first tracking result, a third feature of the object from each frame of the image where the object is located.

[0094] In an exemplary implementation of the disclosed embodiment, the multi-view image tracking device may extract features of the object in each frame of the image in which the object is located, and determine an average value of the extracted multiple features as the third feature of the object.

[0095] S502: Calculate the similarity between the third features of each object contained in each frame image in different groups of image sequences to obtain a similarity matrix.

[0096] In the disclosed embodiment, after obtaining the third features of each object contained in each frame image in multiple groups of image sequences, the multi-view image tracking device can calculate the similarity between the third features of any two objects in different groups of image sequences, and obtain a similarity matrix for all similarity combinations.

[0097] S503 , performing agglomerative hierarchical clustering on each object contained in each frame image in different groups of image sequences based on the similarity matrix, and determining the same object contained in each frame image in different groups of image sequences.

[0098] In the disclosed embodiment, after obtaining the similarity matrix of any two objects in the image sequences of different groups, the multi-view image tracking device can perform agglomerative hierarchical clustering based on the similarity matrix to determine whether any two objects in the image sequences of different groups are the same object.

[0099] In an exemplary implementation of the disclosed embodiments, the multi-view image tracking device can, when performing similarity calculation, calculate the similarity between the third features of only the objects contained in each frame image in the image sequence of adjacent viewpoints, and when performing agglomerative hierarchical clustering based on the similarity matrix, only perform agglomerative hierarchical clustering on the image sequences of adjacent viewpoints, such as the left viewpoint and the front viewpoint, the left viewpoint and the rear viewpoint, the right viewpoint and the front viewpoint, and the right viewpoint and the rear viewpoint.

[0100] S504: performing splicing processing on the first tracking results of the same object contained in each frame image in the different groups of image sequences to obtain the second tracking results of each object contained in each frame image in the different groups of image sequences under the at least two viewing angles.

[0101] In the embodiment of the present disclosure, the multi-view image tracking device can, after determining that the same object is contained in each frame image in different groups of image sequences, perform splicing processing on the movement trajectory of the same object in different groups of image sequences, and combine the movement trajectories of other objects corresponding to different groups of image sequences to obtain a second tracking result of multiple objects under at least two perspectives.

[0102] The disclosed embodiment extracts the third features of the objects based on the multiple first tracking results; calculates the similarity between the third features of the objects contained in the frames of the different groups of image sequences to obtain a similarity matrix; performs agglomerative hierarchical clustering on the objects contained in the frames of the different groups of image sequences based on the similarity matrix to determine the same object contained in the frames of the different groups of image sequences; performs splicing processing on the first tracking results of the same object contained in the frames of the different groups of image sequences to obtain the second tracking results of the objects contained in the frames of the different groups of image sequences under the at least two perspectives. It is possible to perform fusion processing on multiple tracking results under multiple perspectives, and to perform multi-object tracking from multiple perspectives in real time and accurately on a chip with low computing power, so that the tracking results from each perspective can be displayed simultaneously, and only one tracking result is displayed for the same object, which reduces the configuration threshold of this function.

[0103] Figure 6 is a flow chart of a method for optimizing a similarity matrix provided by an embodiment of the present disclosure, such as Figure 6 As shown, based on the above embodiment, the similarity matrix can be optimized by the following method.

[0104] S601. Based on the positions of the target key points of the respective objects in the world coordinate system, calculate the second distances between the positions of the target key points of the respective objects contained in the respective frame images in different groups of image sequences in the world coordinate system to obtain a distance matrix.

[0105] In the embodiment of the present disclosure, the multi-view image tracking device can calculate the second distance between the positions of the target key points of any two objects in different groups of image sequences in the world coordinate system after obtaining the positions of the target key points of each object in the world coordinate system through projection transformation, and combine all the second distances to obtain a distance matrix.

[0106] S602: Optimize the similarity matrix based on the distance matrix.

[0107] In the embodiment of the present disclosure, the multi-view image tracking device can modify the value of the similarity matrix according to the distance matrix after obtaining the distance matrix, so as to optimize the similarity matrix based on the actual position. The specific modification method can be expressed by the following formula:

[0108]

[0109] Among them, S represents the similarity matrix, D represents the distance matrix, i and j represent two objects in two image sequences, S ij Represents the similarity between the two objects, α ijRepresents the pre-set distance coefficient between the two objects, α ij >0,D ij represents the second distance between the positions of the target key points of the two objects in the world coordinate system, and β represents a preset distance parameter threshold.

[0110] The disclosed embodiment calculates the second distance between the positions of the target key points of each object in each frame image of different groups of image sequences in the world coordinate system based on the positions of the target key points of each object in the world coordinate system to obtain a distance matrix; and optimizes the similarity matrix based on the distance matrix. The real position of each object can be considered when performing multi-view fusion, and the real position is important information to assist in object tracking. The introduction of the real position further improves the tracking accuracy.

[0111] In some embodiments of the present disclosure, the multi-view image tracking device may fuse multiple first tracking results corresponding to the at least two groups of image sequences to obtain second tracking results for each object under the at least two viewpoints, and then predict the trajectory of each object again under the at least two viewpoints, and correct the second tracking result based on the prediction result.

[0112] Specifically, after obtaining the second tracking results of each object under at least two perspectives, the multi-view image tracking device can predict the trajectory of each object again under at least two perspectives, and correct the moving trajectory corresponding to each object based on the prediction results. By way of example, the multi-view image tracking device can use filtering to predict the trajectory of each object again within the fusion range of at least two perspectives. By way of example, the filter can select Kalman filtering, predict the predicted position of the object in the nth frame based on the actual position of the object in the previous n-1 frames, and compare the predicted position with the actual position of the object in the nth frame. If the two are close (for example, the distance between the two is less than the preset distance), continue to predict the position of the object in the n+1 frame based on the actual position of the object in the previous n frames, and so on. If the two are far apart (for example, the distance between the two is greater than or equal to the preset distance), it is determined that the actual position of the nth frame corresponds to a new object, so it is necessary to correct the existing trajectory, thereby further improving the accuracy of multi-target tracking under multi-views.

[0113] Figure 7 is a schematic diagram of the structure of a multi-view image tracking device provided by an embodiment of the present disclosure, such as Figure 7As shown, the multi-view image tracking device 700 includes: an acquisition module 710, a tracking module 720, and a fusion module 730, wherein the acquisition module 710 is used to acquire at least two groups of image sequences, and the at least two groups of image sequences are simultaneously acquired from at least two perspectives; the tracking module 720 is used to track each object contained in each frame of each image sequence in the at least two groups of image sequences, and obtain a first tracking result for each object corresponding to the each image sequence; the fusion module 730 is used to fuse multiple first tracking results corresponding to the at least two groups of image sequences, and obtain a second tracking result for each object under the at least two perspectives.

[0114] Optionally, the tracking module 720 includes: a detection unit, used to perform image detection on each frame image in each group of image sequences to obtain an actual detection frame; a first extraction unit, used to perform feature extraction on each object in the actual detection frame to obtain a first feature; a prediction unit, used to start from the second frame image in each group of image sequences and, based on the actual detection frame in the previous frame image, use Kalman filtering to predict the predicted detection frame of each object in the current frame image; a second extraction unit, used to perform feature extraction on each object in the predicted detection frame in the current frame image to obtain a second feature; a first matching unit, used to perform feature matching on the first feature and the second feature extracted from the current frame image; a first determination unit, used to determine the target detection frame corresponding to each object in the actual detection frame contained in the current frame image based on the feature matching result, until the matching of the last frame image in each group of image sequences is completed to obtain the first tracking result.

[0115] Optionally, the tracking module 720 also includes: a second matching unit, used to perform intersection-over-union matching on the actual detection frame and the predicted detection frame corresponding to the first feature and the second feature where feature matching fails; and the first determination unit, specifically used to determine the target detection frame corresponding to each object in the actual detection frame contained in the current frame image based on the feature matching result and the intersection-over-union matching result.

[0116] Optionally, the tracking module 720 also includes: a second determination unit, used to determine the position of the target key points of each object in the actual detection frame in the world coordinate system; a third determination unit, used to determine the first distance between the position of the target key points of each object in the world coordinate system and the image acquisition position of the image where the target key points are located; and an elimination unit, used to eliminate the actual detection frame corresponding to the object to which the target key points belong whose first distance is greater than a preset distance.

[0117] Optionally, the fusion module 730 includes: a third extraction unit, used to extract the third features of the various objects based on the multiple first tracking results; a first calculation unit, used to calculate the similarity between the third features of the various objects contained in the various frame images in the image sequences of different groups to obtain a similarity matrix; a clustering unit, used to perform agglomerative hierarchical clustering on the various objects contained in the various frame images in the image sequences of different groups based on the similarity matrix to determine the same object contained in the various frame images in the image sequences of different groups; and a stitching unit, used to stitch the first tracking results of the same object contained in the various frame images in the image sequences of different groups to obtain the second tracking results of the various objects contained in the various frame images in the image sequences of different groups under the at least two perspectives.

[0118] Optionally, the fusion module 730 also includes: a second calculation unit, used to calculate the second distance between the positions of the target key points of each object contained in each frame image in different groups of image sequences in the world coordinate system based on the positions of the target key points of each object in the world coordinate system, to obtain a distance matrix; an optimization unit, used to optimize the similarity matrix based on the distance matrix.

[0119] Optionally, the multi-view image tracking device 700 further includes: a correction module, configured to perform trajectory prediction on each object again under the at least two view angles, and correct the second tracking result based on the prediction result.

[0120] The multi-view image tracking device provided in this embodiment can execute the method described in any of the above embodiments. Its execution method and beneficial effects are similar and will not be described in detail here.

[0121] Figure 8 It is a structural diagram of a computer device provided in an embodiment of the present disclosure.

[0122] like Figure 8 As shown, the computer device may include a processor 810 and a memory 820 storing computer program instructions.

[0123] Specifically, the processor 810 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0124] The memory 820 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 820 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 820 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 820 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 820 is a non-volatile solid-state memory. In a particular embodiment, the memory 820 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0125] The processor 810 reads and executes the computer program instructions stored in the memory 820 to perform the steps of the multi-view image tracking method provided in the embodiment of the present disclosure.

[0126] In one example, the computer device may further include a transceiver 830 and a bus 840. Figure 8 As shown, the processor 810, the memory 820 and the transceiver 830 are connected via a bus 840 and communicate with each other.

[0127] The bus 840 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 840 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0128] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the multi-view image tracking method provided by the embodiments of the present disclosure.

[0129] The above-mentioned storage medium may, for example, include a memory 820 of computer program instructions, and the above-mentioned instructions may be executed by the processor 810 of the multi-view image tracking device to complete the multi-view image tracking method provided by the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (CompactDisc ROM, CD-ROM), a tape, a floppy disk, and an optical data storage device, etc. The above-mentioned computer program may be written in any combination of one or more programming languages ​​to perform the program code of the operation of the embodiment of the present disclosure, and the programming language includes an object-oriented programming language, such as Java, C++, etc., and also includes a conventional procedural programming language, such as "C" language or a similar programming language. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device, partially on the remote computing device, or entirely on the remote computing device or server.

[0130] An embodiment of the present disclosure also provides a vehicle, which includes a memory, a processor, and a computer program, wherein the computer program is stored in the memory, and when the computer program is executed by the processor, the various processes and effects in the above-mentioned embodiments of the present disclosure can be implemented, which will not be elaborated here.

[0131] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-view image tracking method, characterized in that: include: Acquire at least two sets of image sequences, where the at least two sets of image sequences are simultaneously acquired from at least two viewing angles; For each of the at least two groups of image sequences, tracking processing is performed on each object contained in each frame of the image in each group of image sequences to obtain a first tracking result of each object corresponding to each group of image sequences; A plurality of first tracking results corresponding to the at least two groups of image sequences are fused to obtain second tracking results of the respective objects under the at least two viewing angles.

2. The method according to claim 1, characterized in that The tracking process is performed on each object contained in each frame of the image in each group of image sequences to obtain a first tracking result of each object corresponding to each group of image sequences, including: Performing image detection on each frame of the image in each group of image sequences to obtain an actual detection frame; Extracting features of each object in the actual detection frame to obtain a first feature; Starting from the second frame image in each group of image sequences, based on the actual detection frame in the previous frame image, a Kalman filter is used to predict the predicted detection frame of each object in the current frame image; In the current frame image, extracting features of each object in the predicted detection box to obtain a second feature; Performing feature matching on the first feature and the second feature extracted from the current frame image; In the actual detection frame contained in the current frame image, the target detection frame corresponding to each object is determined based on the feature matching result until the last frame image in each group of image sequences is matched to obtain the first tracking result.

3. The method according to claim 2, characterized in that After performing feature matching on the first feature and the second feature extracted from the current frame image, the method further includes: Performing intersection-over-union matching on the actual detection box and the predicted detection box corresponding to the first feature and the second feature for which feature matching fails; Determining the target detection frame corresponding to each object in the actual detection frame included in the current frame image based on the feature matching result includes: In the actual detection frame included in the current frame image, a target detection frame corresponding to each object is determined based on the feature matching result and the intersection-over-union matching result.

4. The method according to claim 2, characterized in that: After performing image detection on each frame of the image in each group of image sequences to obtain an actual detection frame, the method further includes: Determine the position of the target key point of each object in the actual detection frame in the world coordinate system; Determine a first distance between a position of a target key point of each object in a world coordinate system and an image acquisition position of an image where the target key point is located; The actual detection frame corresponding to the object to which the target key point whose first distance is greater than the preset distance belongs is eliminated.

5. The method according to claim 4, characterized in that The fusing the plurality of first tracking results corresponding to the at least two groups of image sequences to obtain the second tracking results of the respective objects under the at least two viewing angles includes: extracting a third feature of each object based on the plurality of first tracking results; Calculate the similarity between the third features of each object contained in each frame image in different groups of image sequences to obtain a similarity matrix; Based on the similarity matrix, each object contained in each frame image in the image sequences of different groups is clustered to determine the same object contained in each frame image in the image sequences of different groups; The first tracking results of the same object contained in each frame image in the different groups of image sequences are spliced ​​to obtain the second tracking results of each object contained in each frame image in the different groups of image sequences under the at least two viewing angles.

6. The method according to claim 5, characterized in that After calculating the similarity between the third features of the objects contained in the frames of the image sequences of different groups to obtain a similarity matrix, the method further includes: Based on the positions of the target key points of the respective objects in the world coordinate system, second distances between the positions of the target key points of the respective objects contained in the respective frame images in the image sequences of different groups in the world coordinate system are calculated to obtain a distance matrix; The similarity matrix is ​​optimized based on the distance matrix.

7. The method according to claim 1, characterized in that After fusing the plurality of first tracking results corresponding to the at least two groups of image sequences to obtain second tracking results of the respective objects under the at least two perspectives, the method further includes: Trajectory prediction is performed again on each of the objects under the at least two viewing angles, and the second tracking result is corrected based on the prediction result.

8. A multi-view image tracking device, characterized in that: include: An acquisition module, used to acquire at least two sets of image sequences, wherein the at least two sets of image sequences are acquired simultaneously from at least two viewing angles; A tracking module, configured to perform tracking processing on each object contained in each frame of each image sequence in the at least two image sequences, and obtain a first tracking result of each object corresponding to each image sequence; The fusion module is used to fuse the multiple first tracking results corresponding to the at least two groups of image sequences to obtain the second tracking results of each object under the at least two viewing angles.

9. A computer device, characterized in that: include: Memory; processor; and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A vehicle, characterized in that: include: Memory; processor; and a computer program; wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

Images