A three-dimensional group target trajectory matching method and device

Abstract

The application discloses a three-dimensional group target trajectory matching method, and belongs to the technical field of computer vision, which comprises the following steps: collecting the fundamental matrix between view one trajectory and view two trajectory corresponding to the same scene group target from two visual angles; calculating the epipolar distance cost vector corresponding to the view one trajectory and the view two trajectory according to the fundamental matrix, so as to construct a cost vector matrix, and then finding out multiple pairs of two-dimensional trajectories with successful matching from the view one trajectory and the view two trajectory, and finally obtaining a segmented trajectory set; reassociating the segmented trajectory set, so as to realize more accurate trajectory matching of the three-dimensional group target. The application is based on two-dimensional tracking trajectories, finds out multiple pairs of two-dimensional trajectories with successful matching from the view one trajectory and the view two trajectory based on the epipolar distance cost, finally obtains a segmented trajectory set, and then realizes the trajectory matching of the three-dimensional group target through reassociation, so as to solve the problems of trajectory breakage and target mismatching in the tracking of the three-dimensional group target.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and more particularly relates to a three-dimensional group target trajectory matching method and device. BACKGROUND

[0002] Target tracking is a key technology in computer vision. It is widely used in the fields of automatic driving, intelligent security, behavior recognition, etc. Researchers collect image and video data through devices and process images using computer technology, which can effectively extract targets in the video and obtain the trajectory information of the targets through data association to form tracking of the targets.

[0003] If the tracked objects have certain cooperation and similar appearance, they are classified as group target tracking. Group target tracking technology has a wide range of applications in military and life fields. In the military field, unmanned aerial vehicle cluster combat is moving from concept to reality and will become a new combat mode in the future. By observing the group behavior of ants, bees, birds, wolves and fish, a cluster cooperative motion model can be established to achieve the control of the bionic motion strategy of the unmanned aerial vehicle group. In production and life, group target tracking technology has important significance in ecological protection, animal husbandry, airport safety and other fields. For example, in bee breeding, opening the box operation will greatly disturb the activities of the bee group. Using group target tracking technology can achieve non-contact observation of bees and assist the development of bee breeding work. In addition, in the airport, bird groups pose a certain safety hazard to the take-off and landing process of aircraft. Using group target tracking technology to obtain the flight trajectory of the bird group and develop the corresponding driving strategy can achieve real-time monitoring and driving of the bird group.

[0004] Whether it is the study of biological cluster behavior or the real-time monitoring of group targets, the motion trajectory of the group targets needs to be obtained. However, in the cluster, the number of targets is large, the distance is small, and the shapes of the individuals are similar. Under optical observation conditions, the motion trajectories often intersect or block each other. Using current tracking methods, there is often serious tracking interference between target individuals, and stable tracking of their motion trajectories is a difficult problem that needs to be broken through. SUMMARY

[0005] In view of the above defects or improvement needs of the prior art, the present application provides a three-dimensional group target trajectory matching method and device, which aims to, when performing three-dimensional trajectory matching, based on a two-dimensional tracking trajectory, find a preliminary matching trajectory based on the epipolar distance cost, then find a step-type cost vector step point as a pre-segmentation point of the target two-dimensional trajectory based on the shape of the epipolar distance cost, and if the epipolar distance costs of the segments before and after each pre-segmentation point meet the preset conditions, it is regarded as a true segmentation point to determine the two-dimensional segmented trajectory, and then the optimized two-dimensional trajectory is used for three-dimensional trajectory matching, thereby solving the technical problem of inaccurate three-dimensional group target trajectory matching of the prior art.

[0006] To achieve the above objectives, according to one aspect of the present invention, a three-dimensional group target trajectory matching method is provided, comprising:

[0007] S1: Acquire a sequence of view pairs of objects in the same scene from two different perspectives, and obtain the view corresponding to the sequence of view pairs. Figure 1 The fundamental matrix F between the trajectory and the trajectory in view two;

[0008] S2: Traverse the sequence of view pairs and calculate the view based on the fundamental matrix F. Figure 1 The epipolar distance between the target corresponding to any trajectory i in the trajectory and the target corresponding to any trajectory j in the trajectory of the second view, is sequentially added to the epipolar distance cost vector costVector. ij Finally, construct the cost vector matrix CostMat;

[0009] S3: Traverse the cost vector matrix CostMat and use the epipolar distance cost vectors costVector ij Generate the element cost in the cost matrix ij ; By using the cost matrix through global optimization, the view Figure 1 Each trajectory in the trajectory is matched with each trajectory in the second view. If the view... Figure 1 The cost corresponding to the a-th trajectory in the trajectory and the b-th trajectory in the second view. ab If the cost threshold is less than the cost threshold, the a-th trajectory and the b-th trajectory are considered as a pair of successfully matched two-dimensional trajectories, and multiple pairs of successfully matched two-dimensional trajectories are obtained in the end; the epipolar line corresponding to any frame of the a-th trajectory in the second view trajectory is calculated, and the distance vector between it and the b-th trajectory is obtained; the step point that satisfies the threshold is found from the distance vector as the pre-segmentation point;

[0010] S4: Traverse the set of pre-segmentation points. If the epipolar distance cost of the segments before and after a pre-segmentation point meets the preset condition, it is regarded as a true segmentation point. Use two adjacent true segmentation points to determine the segmentation trajectory corresponding to the a-th trajectory and the b-th trajectory. Finally, obtain the segmentation trajectory corresponding to each pair of successfully matched two-dimensional matching trajectories to form a set of segmentation trajectories.

[0011] S5: Reassociate the segmented trajectory set to obtain a more accurate two-dimensional trajectory.

[0012] In one embodiment, S2 includes:

[0013] Traverse the sequence of view pairs and calculate the view based on the fundamental matrix F. Figure 1The polar line distance between the corresponding polar line of the target corresponding to any trajectory i in the trajectory and the target corresponding to any trajectory j in the view two trajectory is sequentially added to the polar line distance cost vector ij End;

[0014] determining the view Figure 1 The number of frame coincidences OFNum, the coincidence start frame number OSFNum, the coincidence end frame number OEFNum and the coincidence frame number sequence FNSeq of the trajectory and the view two trajectory are determined to construct the cost vector matrix CostMat.

[0015] In one embodiment, the element of the i-th row and the j-th column in the cost vector matrix CostMat is expressed as:

[0016] CostMat[i,j]=[costVector ij OFNum OSFNum OEFNum FNSeq];

[0017] Wherein, i=1,2,…,TrkNum1; j=1,2,…,TrkNum2, TrkNum1 represents the number of trajectories in the view Figure 1 The number of trajectories in the view two trajectory.

[0018] In one embodiment, the shape of the distance vector in S3 includes: V-shaped, monotonically increasing, monotonically decreasing and step-shaped; wherein the value of the step point in the step-shaped is greater than a first threshold value and the difference on both sides of the step point is less than the first threshold value.

[0019] In one embodiment, S4 includes:

[0020] Traverse the pre-segmentation point set, calculate the mean, standard deviation and variance of the polar line distance cost corresponding to the segments before and after each pre-segmentation point; if the above three are less than the respective second threshold value, it is considered as the true segmentation point;

[0021] The trajectory between two adjacent true segmentation points is taken as the segmentation trajectory corresponding to a pair of successfully matched two-dimensional matching trajectories, and the first and last point trajectories are added; wherein if there are m true segmentation points, the number of segmentation trajectories in the trajectory set is m-1+2.

[0022] In one embodiment, S5 includes:

[0023] S51: For each segmentation trajectory, calculate the mean and variance of the corresponding cost vector matrix to generate a mean cost matrix and a variance cost matrix;

[0024] S52: re-associate the rows and columns of the mean cost matrix and the variance cost matrix respectively, wherein each row represents a track in view one and each column represents a track in view two. Figure 1 When one of the tracks, each column represents a track in view two.

[0025] In one embodiment, the S52 comprises: traversing the rows and columns of the mean cost matrix and the variance cost matrix respectively, if the following conditions are met simultaneously:

[0026] A1, the number of elements whose mean cost and variance cost are both less than a third threshold is greater than 1;

[0027] A2, the inter-frame difference, the number of frame overlaps, and the spatial distance of the two segmented tracks are all less than the respective fourth threshold;

[0028] ID of the two segmented tracks are re-associated.

[0029] In one embodiment, the method further comprises:

[0030] evaluating the accuracy of the track set, the evaluation indicators including the number of tracks, the total length, and the average length; if the accuracy of the track set meets the preset requirements, it indicates that the preset parameters are reasonably set; if not, the preset parameters are adjusted, and steps S1-S5 are re-executed.

[0031] According to another aspect of the application, a three-dimensional group target track matching device is provided, comprising:

[0032] a determination module configured to acquire a view pair sequence of a group target in the same scene from two angles, obtain the view pair sequence corresponding to the view Figure 1 a basic matrix F between the tracks and the view two tracks;

[0033] a construction module configured to traverse the view pair sequence, calculate the view Figure 1 the epipolar distance between the corresponding target of any track i in the view one tracks and the corresponding target of any track j in the view two tracks, and sequentially add to the epipolar distance cost vector costVector ij at the end, to construct a cost vector matrix CostMat;

[0034] a matching module configured to traverse the cost vector matrix CostMat, utilize each of the epipolar distance cost vectors costVector ij to generate the elements cost ij in the cost matrix; and match each track in the view Figure 1 one tracks and each track in the view two tracks by globally optimizing the cost matrix. Figure 1The cost corresponding to the a-th trajectory in the trajectory and the b-th trajectory in the second view. ab If the cost threshold is less than the cost threshold, the a-th trajectory and the b-th trajectory are considered as a pair of successfully matched two-dimensional trajectories, and multiple pairs of successfully matched two-dimensional trajectories are obtained in the end; the epipolar line corresponding to any frame of the a-th trajectory in the second view trajectory is calculated, and the distance vector between it and the b-th trajectory is obtained; the step point that satisfies the threshold is found from the distance vector as the pre-segmentation point;

[0035] The determination module is used to traverse the set of pre-segmentation points. If the epipolar distance cost of the segments before and after a pre-segmentation point meets the preset condition, it is regarded as a real segmentation point. The segmentation trajectory corresponding to the a-th trajectory and the b-th trajectory is determined by using two adjacent real segmentation points. Finally, the segmentation trajectory corresponding to each pair of successfully matched two-dimensional matching trajectories is obtained to form a set of segmentation trajectories.

[0036] The association module is used to reassociate the segmented trajectory set to obtain a more accurate two-dimensional trajectory.

[0037] According to another aspect of the present invention, a computer vision device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0038] According to another aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0039] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0040] (1) This invention provides a three-dimensional group target trajectory matching method. When performing three-dimensional trajectory matching, based on the two-dimensional tracking trajectory, the method calculates the epipolar lines of the group targets in other views under multiple perspectives using epipolar constraints, and calculates the view... Figure 1 The cost matrix between the trajectory and the second view trajectory is used to obtain the initial matching trajectory based on minimizing the epipolar distance cost. The pre-segmentation point is determined according to the change characteristics of the distance cost vector between the target and the epipolar line. If the epipolar distance cost of the segment before and after each pre-segmentation point meets the preset conditions, it is regarded as the real segmentation point to determine the two-dimensional segmented trajectory. Then, the two-dimensional segmented trajectory is re-associated to obtain a more accurate two-dimensional trajectory, realizing the trajectory matching of three-dimensional group targets and solving the problems of trajectory breakage and target mismatch in three-dimensional group target tracking.

[0041] (2) This scheme uses epipolar geometry theory to calculate the fundamental matrix F between two viewpoints, and then calculates the viewpoint corresponding to the fundamental matrix F. Figure 1The epipolar distance cost of any track i in the sequence and any track j in the view two sequence ij and in combination with the view Figure 1 The frame coincidence number, coincidence start frame number, end frame number and coincidence frame number sequence of the track and view two track are used to construct a cost vector matrix CostMat, and the matching cost of the group target can be measured.

[0042] (3) The scheme directly uses the epipolar distance cost of the track and the view two track as the first element, and then combines the frame coincidence number, coincidence start frame number, end frame number and coincidence frame number sequence of the view sequences in the two views to construct a cost vector matrix, and the calculation complexity of track matching by traversing the cost vector matrix is low. Figure 1 The epipolar distance cost of the track and the view two track is directly used as the first element, and then the frame coincidence number, coincidence start frame number, end frame number and coincidence frame number sequence of the view sequences in the two views are combined to construct a cost vector matrix, and the calculation complexity of track matching by traversing the cost vector matrix is low.

[0043] (4) The shape of the cost vector corresponding to each costVector is determined in the scheme, and considering that the step-type cost vector often represents the wrong association between the tracks, the value of the step point of the step-type cost vector greater than the first threshold value and the difference on both sides of the step point less than the first threshold value are selected as the pre-segmentation point, which can reduce the wrong matching between the tracks after segmentation.

[0044] (5) The scheme calculates the mean, standard deviation and variance of the epipolar distance cost of each pre-segmentation point, and selects the true segmentation point if the value is greater than the threshold value, which can solve the problem of wrong association between the tracks and improve the matching accuracy.

[0045] (6) The scheme generates a mean cost matrix and a mean variance matrix by calculating the mean and variance of the cost vector matrix between the tracks after segmentation, and considering the bidirectional matching, the re-association matching is performed by traversing the cost matrix by row and by column, which can solve the problem that the Hungarian matching can only be used for one-to-one, has low calculation complexity and high matching accuracy.

[0046] (7) The scheme combines the tracks less than the set cost threshold value, time threshold value and space threshold value, which can solve the problem of broken real tracks in matching.

[0047] (8) The scheme evaluates the accuracy of the track set, and adjusts the preset parameters if it does not meet the requirements, thereby improving the matching accuracy; in addition, the number of tracks, total length and average length are included, which can evaluate the accuracy of the track set from multiple dimensions, and the evaluation accuracy is high. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The flowchart of the three-dimensional group target track matching method provided by an embodiment of the application.

[0049] Figure 1 The framework diagram of the three-dimensional group target track matching method provided by an embodiment of the application.

[0050] Figure 2a Flow chart of F matrix optimization provided by an embodiment of the present application.

[0051] Figure 2b Flow chart of cost matrix construction provided by an embodiment of the present application.

[0052] Figure 3 Schematic diagram of v-shaped cost vector shape provided by an embodiment of the present application.

[0053] Figure 4a Schematic diagram of monotone increasing and monotone decreasing cost vector shape provided by an embodiment of the present application.

[0054] Figure 4b Schematic diagram of step type cost vector shape provided by an embodiment of the present application.

[0055] Figure 4c Schematic diagram of other cost vector shape provided by an embodiment of the present application.

[0056] Figure 4d Flow chart of pre-segmentation point positioning algorithm provided by an embodiment of the present application.

[0057] Figure 5 Flow chart of real segmentation point positioning algorithm provided by an embodiment of the present application.

[0058] Figure 6 Flow chart of re-association algorithm provided by an embodiment of the present application.

[0059] Figure 7 Two-dimensional trajectory visualization diagram of Pair04 provided by an embodiment of the present application.

[0060] Figure 8 Three-dimensional trajectory matching result visualization diagram of Pair04 provided by an embodiment of the present application.

[0061] Figure 9 Schematic diagram of two bounding boxes of the same target provided by an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0063] As Figure 10As shown, the present invention provides a three-dimensional group target trajectory matching method, comprising:

[0064] S1: Acquire view pairs of the same group of targets in the scene from two perspectives, and obtain the view corresponding to the view pair sequence. Figure 1 The fundamental matrix F between the trajectory and the trajectory in view two;

[0065] S2: Traverse the sequence of view pairs and calculate the view based on the fundamental matrix F. Figure 1 The epipolar distances between the target corresponding to any trajectory i in the trajectory and the target corresponding to any trajectory j in the trajectory of the second view are sequentially added to the epipolar distance cost vector. ij Finally, construct the cost vector matrix CostMat;

[0066] S3: Traverse the cost vector matrix CostMat and use the cost vectors at each epipolar distance to the cost vector. ij Generate the element cost in the cost matrix ij By utilizing the cost matrix through global optimization, the view... Figure 1 Match each trajectory in the trajectory with each trajectory in view two. Figure 1 The cost corresponding to the a-th trajectory in the trajectory and the b-th trajectory in the second view. ab If the cost is less than the threshold, the a-th trajectory and the b-th trajectory are considered as a pair of successfully matched two-dimensional trajectories, and finally multiple pairs of successfully matched two-dimensional trajectories are obtained; calculate the epipolar line corresponding to any frame of the a-th trajectory in the second view trajectory, and obtain the distance vector between it and the b-th trajectory; find the step point that meets the threshold from the distance vector as the pre-segmentation point;

[0067] S4: Traverse the set of pre-segmentation points. If the epipolar distance cost of the segments before and after a pre-segmentation point meets the preset condition, it is regarded as a real segmentation point. Use two adjacent real segmentation points to determine the segmentation trajectory corresponding to the a-th trajectory and the b-th trajectory. Finally, obtain the segmentation trajectory corresponding to each pair of successfully matched two-dimensional matching trajectories, forming a set of segmentation trajectories.

[0068] S5: Reassociate the segmented trajectory set to obtain more accurate two-dimensional trajectories, thereby achieving more accurate trajectory matching for three-dimensional group targets.

[0069] Specifically, such as Figure 1 As shown, firstly, the estimation with the smallest error is found using the first n frames of the image sequence as the final F estimate. The specific optimization method is as follows: First, in the view... Figure 2aIn the first frame of the image, n reference points are selected. Then, the image coordinates of the corresponding reference points are found in the second view. One frame of the image corresponding to each time point is selected from each view as the input image for the LMDS algorithm. The F matrix of each candidate image pair is obtained, and the one with the smallest error is selected as the fundamental matrix F. The specific process is as follows: Figure 1 As shown. Then, the corresponding view is calculated using the fundamental matrix F. Figure 2b The cost vector of the epipolar distance between the i-th trajectory in view 1 and the j-th trajectory in view 2. ij Then, a cost vector matrix CostMat is constructed. The Cost vector matrix CostMat is traversed, and the mean of the distance cost vectors for each epipolar line is calculated to generate a cost matrix. Then, the Hungarian matching algorithm is used to minimize the total distance cost of matching the left and right view trajectories, generating two-dimensional trajectory matching pairs. Finally, the matching results are judged based on a cost threshold. Matches that meet the threshold form the final stereo correspondence, resulting in a preliminary matched three-dimensional trajectory.

[0070] In one embodiment, S2 includes:

[0071] Traverse the sequence of view pairs and calculate the view based on the fundamental matrix F. Figure 1 The epipolar distances between the target corresponding to any trajectory i in the trajectory and the target corresponding to any trajectory j in the trajectory of the second view are sequentially added to the epipolar distance cost vector. ij end;

[0072] Determine the view Figure 1 The cost vector matrix CostMat is constructed by using the number of overlapping frames OFNum, the start frame number of overlap OSFNum, the end frame number OEFNum, and the sequence of overlapping frame numbers FNSeq for the trajectory and view 2 trajectory.

[0073] In one embodiment, the element in the i-th row and j-th column of the cost vector matrix CostMat is represented as: CostMat[i,j] = [costVector] ij OFNum OSFNum OEFNum FNSeq];

[0074] Where i = 1, 2, ..., TrkNum1; j = 1, 2, ..., TrkNum2, and TrkNum1 represents the view. Figure 1 The number of trajectories in the trajectory, TrkNum2 represents the number of trajectories in the trajectory of view two.

[0075] Specifically, the element structure in CostMat is defined as shown in the above formula. Calculating CostMat requires traversing the image sequence, calculating the epipolar distance cost of all candidate pairs in each frame, as well as the number of overlapping frames, the start frame number of the overlap, the end frame number, and the sequence of overlapping frame numbers for all candidate pairs, ultimately forming a cost vector matrix. The specific process is as follows: Figure 1 As shown.

[0076] In the above formula, TrkNum1 represents the view Figure 3 The number of trajectories in view 2, TrkNum2 represents the number of trajectories in view 2. Figure 1 Trajectory i in the sequence and trajectory j in the second view sequence, where costVector represents the cost vector of the two trajectories, OFNum (Overlap FrameNumber) represents the number of images that overlap in time, OSFNum (OverlapStartFrameNumber) represents the frame number of the image that starts to overlap in time, OEFNum (Overlap EndFrameNumber) represents the frame number of the image that ends to overlap, and FNSeq (FrameNumber Sequence) represents the sequence of overlapping frame numbers.

[0077] In one embodiment, each costVector in S3 ij The corresponding cost vector shapes include: V-shape, monotonically increasing shape, monotonically decreasing shape, and preset step shape; among them, the value of the step point of the preset step shape cost vector is greater than the first threshold and the difference between the two sides of the step point is less than the first threshold.

[0078] Among them, such as Figure 1 , Figure 4a , Figure 4b and Figure 4c The cost vector shapes are defined as shown, including V-shaped cost vectors, monotonically increasing or monotonically decreasing shapes, step-shaped cost vectors, and other shapes. By identifying the step-shaped cost vectors, the two-dimensional trajectory to be segmented is found.

[0079] Specifically, based on the step characteristics of the step cost vector, the positioning points in the curve are first located, and a pre-segmentation point positioning algorithm is designed. The flowchart of this algorithm is as follows: Figure 4d As shown in the diagram. In this algorithm, the difference between the two sides of the point where the step occurs must be less than a threshold to avoid interference caused by a single point and increase the robustness of the positioning; the threshold is set using an empirical value and can be set to 20.

[0080] In one embodiment, such as Figure 5As shown, S4 includes: traversing the pre-segmentation point set, calculating the mean, standard deviation and variance of the epipolar distance cost corresponding to each pre-segmentation point; if the above three are less than the respective second threshold, it is regarded as a true segmentation point; the trajectory between the adjacent two true segmentation points is regarded as a segmentation trajectory, and the first and last point trajectories are added; wherein, if there are m true segmentation points, the number of segmentation trajectories in the trajectory set is m-1+2.

[0081] Wherein, the second threshold lenTH, thVar, thStd are 3, 10, 3.3 respectively, which are empirical values. After the segmentation point positioning algorithm processing, the trajectories in the left and right views corresponding to the segmentation points are all segmented, and the trajectories with ID mismatch can be segmented.

[0082] In one of the embodiments, as shown in Figure 6 S5 includes:

[0083] S51: for each segmentation trajectory, calculate the mean and variance of the corresponding cost vector matrix to generate a mean cost matrix and a variance cost matrix;

[0084] S52: traverse the rows and columns of the mean cost matrix and the variance cost matrix respectively, and perform reassociation.

[0085] In one of the embodiments, S52 includes: traversing the rows and columns of the mean cost matrix and the variance cost matrix respectively, and if the following conditions are met simultaneously:

[0086] A1, the number of elements whose mean cost and variance cost are less than the third threshold is greater than 1;

[0087] A2, the inter-frame difference, frame overlap number and spatial distance of the two segmentation trajectories are all less than the respective fourth threshold;

[0088] ID reassociation of the two segmentation trajectories is regarded.

[0089] Specifically, step S52 includes the following steps:

[0090] A1. Calculate the mean and variance of the cost vector costVector in the original cost matrix to generate a mean cost matrix and a variance cost matrix;

[0091] A2. Traverse the rows of the cost matrix, extract the trajectories in each row that meet the cost threshold, and then perform reassociation judgment on these trajectories, the main judgment basis being whether the temporal and spatial distance between the two trajectories meets the threshold;

[0092] Specifically, in the two trajectory association processes, for the time distance, i.e., frame distance and frame overlap threshold, the threshold setting method of the application is as follows: (1) the time interval threshold, i.e., the interval between the image frame numbers of the two trajectories, is mainly set according to the empirical value; (2) the time overlap threshold, the application finds in the research that there is a case that the target detection network gives two bounding boxes for the same target in the detection process, such as Figure 7 The same target of three frames of images in time is shown, and Figure 10 The target detection network gives two bounding boxes in the middle image, and both satisfy the condition that the confidence is greater than 0.5, at this time, for the two-dimensional tracking algorithm, two IDs will be assigned to this target, and when the false alarm disappears, the target may continue to use any ID, so there is a case of ID mismatch, the application sets an overlap threshold, and the selection of the threshold is based on a large number of experimental analysis, and is set to 1.

[0093] In one of the embodiments, further comprising:

[0094] The accuracy of the trajectory set is evaluated, and the evaluation index includes the number of trajectories, the total length and the average length; if the accuracy of the trajectory set meets the preset requirement, it indicates that each preset parameter setting is reasonable; if not, each preset parameter is adjusted, and steps S1-S5 are re-executed.

[0095] Specifically, for the subjective evaluation method, the two-dimensional trajectories with stereo corresponding relationship obtained by the matching algorithm are visualized in two views respectively, the synchronous playback of the two views is used, and the position, motion direction, motion speed and attitude of the target and other information are combined for subjective judgment, and whether the stereo corresponding relationship of the target is correct is obtained. In the two-dimensional trajectory visualization process, different trajectories under the same view are represented by different colors; in the subjective evaluation, the same target under two views is represented by the same color, so that effective subjective evaluation can be carried out.

[0096] (1) The number of trajectories TrkNum: the number of matched trajectories is obtained by counting the matched three-dimensional motion trajectories, and the number of trajectories TrkNum can reflect the number of targets in the video sequence. The TrkNum index is not the larger the better, but it can assist in judging the performance of the algorithm, for example, the number of matched trajectories is increased after segmentation under normal circumstances; but the number of matched trajectories is reduced after re-association under normal circumstances, so the evaluation of the matching result should be dialectically analyzed.

[0097] (2) Total length TotalLen: the length of all matched trajectories is summed. If the matched trajectories are correct through subjective evaluation, then the TotalLen index is the larger the better.

[0098] (3) MeanLen: the index can evaluate the length of the matched trajectory, if the matched trajectory is correct through subjective evaluation, then the greater the average length is, the higher the completeness of the matched trajectory is, and the better the matching quality is.

[0099] In summary, the application corrects the ID mismatch problem of the target in the two-dimensional tracking process according to the motion constraint using the three-dimensional trajectory, improves the accuracy in the two-dimensional tracking process, and iteratively corrects the three-dimensional trajectory matching result using the two-dimensional trajectory tracking result with higher precision, so that the ID mismatch problem of the two-dimensional trajectory is optimized, the matching ambiguity problem that is prone to occur in the stereo matching is solved, the influence caused by the matching ambiguity is reduced, and finally the objective evaluation index of the trajectory matching result is improved.

[0100] The simulation data of matching the three-dimensional group target provided by the application is provided below, and 16 video sequences are used in the experiment, including 6 pairs of training sets (Pair11~Pair16) and 10 pairs of test sets (Pair01~Pair10).

[0101] The application visualizes the motion trajectory according to the matching result, selects the Pair04 image sequence in the test data for visualization, and subjectively evaluates and analyzes the matching result. The two-dimensional motion trajectory of Pair04 is as shown in Figure 10 The three-dimensional motion trajectory matching result of Pair04 is as shown in Figure 8 The results of the reference algorithm, the trajectory segmentation algorithm, the trajectory re-association algorithm and the fusion algorithm are visualized, due to the limitation of the length, the application cannot list all the image sequences, and only the last frame of the sequence can be displayed. In the subjective evaluation process, the image sequence is played continuously, and the several algorithms of the application are subjectively evaluated, and the conclusion is that the matched three-dimensional trajectory corresponds to the two-dimensional trajectory with correct stereo correspondence.

[0102] In addition, the subjective evaluation is mainly responsible for evaluating the correctness of the trajectory matching, and through the analysis of the above visualization results, it can be known that the three-dimensional motion trajectory matching algorithm can guarantee the correctness of the stereo correspondence, so on the basis of ensuring the correctness, the objective evaluation index will be analyzed below. As shown in Table 1, the objective evaluation index corresponding to the experimental result includes the number of trajectories, the average length and the total length of the trajectory.

[0103] Table 1: Objective evaluation index of three-dimensional motion trajectory matching

[0104]

[0105] Firstly, the number of trajectories is analyzed. As shown in Table 1, compared with the benchmark algorithm, the number of trajectories of the trajectory segmentation algorithm increases by 8.02%, because the trajectory segmentation algorithm divides the trajectories with ID mismatch in the two-dimensional trajectories, which will certainly obtain more matching pairs satisfying the threshold in the stereo matching. The increase in the number of trajectories shows that the trajectory segmentation algorithm can improve the performance of the trajectory matching algorithm; compared with the benchmark algorithm, the number of trajectories of the trajectory reassociation algorithm decreases by 17.18%. Since the trajectory reassociation algorithm associates multiple trajectories of the same target, it will certainly lead to a decrease in the number of trajectories obtained in the stereo matching, and the decrease in the number of trajectories shows that the algorithm can improve the performance of the trajectory matching algorithm; the number of trajectories of the fusion algorithm (the method of the application) is between the number of trajectories of the trajectory segmentation algorithm and the number of trajectories of the trajectory reassociation algorithm, which meets the expected performance of the fusion of the two.

[0106] Then, the total length and average length of the trajectories are analyzed. As shown in Table 1, compared with the benchmark algorithm, the total length of the trajectories of the trajectory segmentation algorithm increases by 6.65%, and the average length slightly decreases, because the trajectory segmentation algorithm divides the trajectories, and the slight decrease in the length of the trajectories is normal; compared with the benchmark algorithm, the total length of the trajectories of the trajectory reassociation algorithm increases by 21.62%, and the average length increases by 48.57%, which shows that the trajectory reassociation algorithm greatly improves the completeness of the trajectory matching; compared with the benchmark algorithm, the total length of the trajectories of the fusion algorithm (the method of the application) increases by 26.46%, and the average length of the trajectories increases by 43.15%, but compared with the trajectory reassociation algorithm, the average length of the trajectories slightly decreases, because the segmentation module is fused on the basis of the reassociation, and the slight decrease in the average length of the trajectories meets the expectation, and the total length of the trajectories obtained by the fusion algorithm (the method of the application) is the highest, which shows that among the comparison algorithms and the single modules, the matching performance of the fusion algorithm (the method of the application) is the best.

[0107] In summary, the trajectory segmentation algorithm is improved in terms of the correctness of the matching trajectories, the trajectory reassociation algorithm is improved in terms of the completeness of the matching trajectories, and compared with the benchmark algorithm, the average length and the total length of the trajectories of the fusion algorithm (the method of the application) are effectively improved, which proves that the fusion algorithm (the method of the application) can improve the matching performance from the correctness and the completeness of the trajectories. It can be seen that the total length of the trajectories of the method of the application in the three-dimensional swarm target trajectory matching task increases by 26.46% compared with the benchmark matching algorithm, and the average length of the trajectories increases by 43.15%, that is, the method of the application is better in the three-dimensional swarm target trajectory matching task.

[0108] According to another aspect of the application, a three-dimensional swarm target trajectory matching device is provided, which comprises:

[0109] determining module, configured to acquire a view pair sequence of a same scene group target under two view angles, acquire a basic matrix F corresponding to the view pair sequence, and acquire a view pair sequence of a same scene group target under two view angles Figure 9 a basic matrix F between the trajectory and the view two trajectory;

[0110] constructing module, configured to traverse the view pair sequence, calculate a view Figure 1 a polar distance between a corresponding polar line of the target corresponding to any trajectory i in the trajectory and the target corresponding to any trajectory j in the view two trajectory, and sequentially add the polar distance to a polar distance cost vector costVector ij at the end, to construct a cost vector matrix CostMat;

[0111] matching module, configured to traverse the cost vector matrix CostMat, utilize each polar distance cost vector costVector ij generate an element cost ij in the cost matrix; and utilize the cost matrix through global optimization to match the trajectory and the view two trajectory. Figure 1 Figure 1 Figure 1 if a cost ab corresponding to the a-th trajectory in the trajectory and the b-th trajectory in the view two trajectory is less than a cost threshold, the a-th trajectory and the b-th trajectory are regarded as a pair of matched two-dimensional trajectories, and a plurality of pairs of matched two-dimensional trajectories are finally obtained; a polar line corresponding to any frame of the a-th trajectory in the view two trajectory is calculated, and a distance vector between the a-th trajectory and the b-th trajectory is obtained; a threshold-satisfying step point is found from the distance vector as a pre-segmentation point;

[0112] determining module, configured to traverse the pre-segmentation point set, regard a pre-segmentation point as a true segmentation point if polar distance costs of front and rear segments of the pre-segmentation point satisfy a preset condition; determine a segmentation trajectory corresponding to the a-th trajectory and the b-th trajectory by utilizing two adjacent true segmentation points; finally obtain segmentation trajectories corresponding to each pair of matched two-dimensional matched trajectories, and form a segmentation trajectory set;

[0113] correlation module, configured to re-correlate the segmentation trajectory set, obtain more accurate two-dimensional trajectories, and thus realize more accurate trajectory matching of three-dimensional group targets.

[0114] According to another aspect of the present application, a computer vision device is provided, which comprises a memory and a processor, the memory stores a computer program, and the processor realizes the steps of the above method when executing the computer program.

[0115] According to another aspect of the present application, a computer readable storage medium is provided, which stores a computer program, and the computer program realizes the steps of the above method when executed by a processor.

[0116] ​Those skilled in the art can easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A three-dimensional group target trajectory matching method, characterized in that, include: S1: Collect a sequence of view pairs of the same group of targets in the same scene from two perspectives, and obtain the fundamental matrix F between the view pair trajectories of view one and view two corresponding to the view pair sequence; S2: Traverse the sequence of view pairs, calculate the epipolar distance between the epipolar line corresponding to the target of any trajectory i in the trajectory of view one and the target of any trajectory j in the trajectory of view two, based on the fundamental matrix F, and add it sequentially to the epipolar distance cost vector costVector. ij Finally, construct the cost vector matrix CostMat; S3: Traverse the cost vector matrix CostMat and use the epipolar distance cost vectors costVector ij Generate the element cost in the cost matrix ij ; By using global optimization and the cost matrix, each trajectory in View 1 is matched with each trajectory in View 2. If the cost of the a-th trajectory in View 1 and the b-th trajectory in View 2 are... ab If the cost threshold is less than the cost threshold, the a-th trajectory and the b-th trajectory are considered as a pair of successfully matched two-dimensional trajectories, and multiple pairs of successfully matched two-dimensional trajectories are obtained in the end; the epipolar line corresponding to any frame of the a-th trajectory in the second view trajectory is calculated, and the distance vector between it and the b-th trajectory is obtained; the step point that satisfies the threshold is found from the distance vector as the pre-segmentation point; S4: Traverse the set of pre-segmentation points. If the epipolar distance cost of the segments before and after a pre-segmentation point meets the preset condition, it is regarded as a true segmentation point. Use two adjacent true segmentation points to determine the segmentation trajectory corresponding to the a-th trajectory and the b-th trajectory. Finally, obtain the segmentation trajectory corresponding to each pair of successfully matched two-dimensional matching trajectories to form a set of segmentation trajectories. S5: Reassociate the segmented trajectory set to obtain a more accurate two-dimensional trajectory.

2. The three-dimensional group target trajectory matching method as described in claim 1, characterized in that, S2 includes: Iterate through the sequence of view pairs, calculate the epipolar distance between the epipolar line corresponding to the target on any trajectory i in the trajectory of view one and the target on any trajectory j in the trajectory of view two, based on the fundamental matrix F, and add the distances sequentially to the epipolar distance cost vector costVector. ij end; The number of overlapping frames OFNum, the start frame number OSFNum, the end frame number OEFNum, and the sequence of overlapping frames FNSeq for the trajectory of view one and the trajectory of view two are determined in order to construct the cost vector matrix CostMat.

3. The three-dimensional group target trajectory matching method as described in claim 2, characterized in that, The element in the i-th row and j-th column of the cost vector matrix CostMat is represented as follows: CostMat[i,j]=[costVector ij OFNum OSFNum OEFNum FNSeq]; Where i = 1, 2, ..., TrkNum1; j = 1, 2, ..., TrkNum2, TrkNum1 represents the number of trajectories in the trajectory of view one, and TrkNum2 represents the number of trajectories in the trajectory of view two.

4. The three-dimensional group target trajectory matching method as described in claim 1, characterized in that, The shape of the distance vector in S3 includes: V-shape, monotonically increasing shape, monotonically decreasing shape, and step shape; Wherein, the value of the step point in the step type is greater than the first threshold and the difference between the two sides of the step point is less than the first threshold.

5. The three-dimensional group target trajectory matching method as described in claim 1, characterized in that, S4 includes: Traverse the set of pre-segmentation points and calculate the mean, standard deviation, and variance of the epipolar distance cost corresponding to the segments before and after each pre-segmentation point; if all three are less than their respective second thresholds, then it is regarded as the true segmentation point. The trajectory between two adjacent true dividing points is taken as a pair of successfully matched two-dimensional matching trajectories, and the trajectory of the first and last points is added; wherein, if there are m true dividing points, the number of dividing trajectories in the trajectory set is m-1+2.

6. The three-dimensional group target trajectory matching method as described in claim 1, characterized in that, S5 includes: S51: For each segmented trajectory, calculate the mean and variance of the corresponding cost vector matrix to generate a mean cost matrix and a variance cost matrix; S52: Reassociate the rows and columns of the mean cost matrix and the variance cost matrix respectively, where each row represents a trajectory in view one and each column represents a trajectory in view two.

7. The three-dimensional group target trajectory matching method as described in claim 6, characterized in that, S52 includes: traversing the rows and columns of the mean cost matrix and the variance cost matrix respectively, if simultaneously satisfying: A1. The number of elements whose mean cost and variance cost are both less than the third threshold is greater than 1; A2. The inter-frame difference, number of overlapping frames, and spatial distance between the two segmented trajectories are all less than their respective fourth thresholds; This is then considered as a re-association of the IDs of the two segmented trajectories.

8. The three-dimensional group target trajectory matching method according to any one of claims 1-7, characterized in that, The method further includes: The accuracy of the trajectory set is evaluated, and the evaluation indicators include the number of trajectories, the total length, and the average length. If the accuracy of the trajectory set meets the preset requirements, it means that the preset parameters are set reasonably. If not, the preset parameters are adjusted and steps S1-S5 are executed again.

9. A three-dimensional group target trajectory matching device, characterized in that, include: The determination module is used to collect a sequence of view pairs of targets in the same scene from two perspectives, and to obtain the fundamental matrix F between the view pair sequence and the view one trajectory and the view two trajectory. The construction module is used to traverse the sequence of view pairs, calculate the epipolar distance between the epipolar line corresponding to the target of any trajectory i in the trajectory of view one and the target of any trajectory j in the trajectory of view two based on the fundamental matrix F, and add the distance to the epipolar distance cost vector costVector in sequence. ij Finally, construct the cost vector matrix CostMat; The matching module is used to traverse the cost vector matrix CostMat and use the epipolar distance cost vectors costVector. ij Generate the element cost in the cost matrix ij ; By using global optimization and the cost matrix, each trajectory in View 1 is matched with each trajectory in View 2. If the cost of the a-th trajectory in View 1 and the b-th trajectory in View 2 are... ab If the cost threshold is less than the cost threshold, the a-th trajectory and the b-th trajectory are considered as a pair of successfully matched two-dimensional trajectories, and multiple pairs of successfully matched two-dimensional trajectories are obtained in the end; the epipolar line corresponding to any frame of the a-th trajectory in the second view trajectory is calculated, and the distance vector between it and the b-th trajectory is obtained; the step point that satisfies the threshold is found from the distance vector as the pre-segmentation point; The determination module is used to traverse the set of pre-segmentation points. If the epipolar distance cost of the segments before and after a pre-segmentation point meets the preset condition, it is regarded as a real segmentation point. The segmentation trajectory corresponding to the a-th trajectory and the b-th trajectory is determined by using two adjacent real segmentation points. Finally, the segmentation trajectory corresponding to each pair of successfully matched two-dimensional matching trajectories is obtained to form a set of segmentation trajectories. The association module is used to reassociate the segmented trajectory set to obtain a more accurate two-dimensional trajectory.

10. A computer vision device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images