A heterogeneous multi-target matching method based on local topological feature description and Sinkhorn

Abstract

The application relates to a heterogeneous multi-target matching method based on local topological feature description and Sinkhorn. Firstly, a plurality of line segments are constructed in an N-neighbor connection mode for the heterogeneous target information to be matched in a first to-be-matched graph and a second to-be-matched graph; the mapping relationship is determined by associating the line segment constructed in advance in the first to-be-matched graph with the window position relationship; the line segment end points are used as a group of bases; the coding coefficients are obtained by linearly representing other targets in the neighborhood view through the group of bases to obtain the topological feature description; the mapping relationship is applied to the neighborhood rectangular window of the other to-be-matched graph to extract the topological features; the coding coefficients are used to predict the positions of other targets in the neighborhood rectangle of the second to-be-matched graph; a cost matrix of the predicted target positions and the actual existing target positions in the second to-be-matched graph is constructed; finally, multi-target data correlation is carried out based on Sinkhorn and the cost matrix to complete the matching.

Description

Technical Field

[0001] This invention relates to the field of image processing and multi-source information fusion technology, specifically to a heterogeneous multi-target matching method based on local topological feature description and Sinkhorn. Background Technology

[0002] Multi-sensor fusion detection technology (such as image, radar, and laser sensors) is increasingly being used in fields such as autonomous driving, industrial robots, and security monitoring. Achieving target identity recognition and information matching among multiple targets from different sources is crucial for successful collaborative detection.

[0003] The heterogeneous multi-target information acquired by the multi-source heterogeneous detection platform mainly contains location information, and the image features are not obvious. In particular, when radar or laser information is involved, there are no image features. At this time, the identity recognition and matching work is mainly focused on processing the location information of the point set.

[0004] Traditional feature extraction algorithms used for target identity recognition include SIFT, SURF, and ORB. SIFT utilizes the gradient direction distribution characteristics of feature points in neighboring pixels of an image for feature description; SURF extracts features by calculating the Haar features of feature points in the image; and ORB describes features based on the Brief descriptor, which is calculated using image grayscale. These algorithms all rely on image grayscale or texture information, making them difficult to apply to feature extraction of heterogeneous and heterogeneous information.

[0005] Multi-target matching can be viewed as a data association problem. Based on existing literature and research, commonly used data association algorithms include nearest neighbor-based algorithms, global nearest neighbor algorithms (greedy algorithm, auction algorithm, Hungarian matching), and joint probability distribution (JPDA). Nearest neighbor-based association methods are often applied to sparse target scenarios but are not suitable for dense scenarios. Global nearest neighbor-based methods are mature and widely used, but their matching results are highly dependent on the constructed cost matrix, lacking robustness. Joint probability distribution-based association algorithms are highly adaptable to dense cluster scenarios, but their complexity is high and they cannot achieve one-to-one matching of each target. Summary of the Invention

[0006] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a heterogeneous multi-target matching method based on local topological feature description and Sinkhorn, which solves the problem of heterogeneous multi-target information matching oriented towards point set location information.

[0007] The technical solution of this invention is: a heterogeneous multi-target matching method based on local topological feature description and Sinkhorn, the method comprising the following steps:

[0008] Obtain two views to be matched, designate one of them as the first view to be matched, and the other as the second view to be matched;

[0009] Construct several line segments from the heterogeneous target information to be matched in the first and second views using an N-nearest neighbor connection method;

[0010] The pre-constructed line segments of the first image to be matched are associated with their window positions to determine the mapping relationship. A set of bases is constructed using the endpoints of the line segments. The coding coefficients are obtained by linearly representing other heterogeneous target points to be matched in the view using this set of bases, and the topological feature description of the heterogeneous target points to be matched is obtained.

[0011] The mapping relationship is applied to the second map to extract neighborhood rectangular windows with the same topological features. Based on the principle of local topological feature invariance, the coding coefficients are used to predict the position of the heterogeneous target point to be matched within the neighborhood rectangle of the second map.

[0012] Calculate the cost matrix M based on the predicted target location and the actual target location in the second matching map;

[0013] Based on the cost matrix M, multi-objective data association is performed using the sinkhorn and the cost matrix, and heterogeneous multi-objective matching is completed.

[0014] Preferably, the nearest neighbor line segment formed in the first view to be matched is denoted as x. i' Let y' = i' = 1 to n, and denote the nearest neighbor line segment formed in the second view to be matched as y'. j' j' = 1 to m, where n is the number of nearest neighbor line segments in the first graph to be matched, and m is the number of nearest neighbor line segments in the second graph to be matched. The specific calculation process of the cost matrix M is as follows:

[0015] S1. Select the i'th nearest neighbor line segment in the first view to be matched, traverse the m nearest neighbor line segments in the second view to be matched, repeat steps S2 to S6, and then proceed to step S7.

[0016] S2. Select any vertex of the rectangular window of the first image to be matched as the reference point O of the rectangular window of the first image to be matched. Establish a first polar coordinate system with one endpoint of the i'th line segment in the first view to be matched as the origin and the line segment as the polar axis. Construct a mapping relationship between the i'th line segment in the first view to be matched and the reference point of the rectangular window of the first image to be matched in the first polar coordinate system.

[0017] S3. Using the reference point O of the first rectangular window to be matched as the starting point and the two endpoints A and B of the line segment as the ending points, construct a set of vectors. and As a basis, denoted as the first vector basis, it linearly represents the positions of all other unmatched heterogeneous target points in the first matching image rectangular window, and takes their coefficients as the encoding parameters (α) of the unmatched heterogeneous target points in the first matching image rectangular window. i ,β i ), i = 1 ~ n s To obtain local topological features, n s This represents the number of heterogeneous target points to be matched in the first rectangular window to be matched.

[0018] S4. For the j'th nearest neighbor line segment in the second graph to be matched, take one of the endpoints of the line segment as the origin and the line segment as the polar axis in the second polar coordinate system. According to the principle that the candidate neighborhood rectangles after rotation, translation and scaling transformation coincide with the position of the second graph to be matched's own window rectangle in the second polar coordinate system and the first polar coordinate system, the candidate neighborhood rectangle window of the line segment in the second graph to be matched is obtained by using the mapping relationship in the first view to be matched.

[0019] S5. Based on the principle of local topological feature invariance, construct a set of vectors with vertex O' of the candidate neighborhood rectangle as the vertex and the two endpoints A' and B' of the j'-th nearest neighbor line segment in the second graph to be matched as the endpoints. and As the second vector basis, the encoding parameters (α) of the heterogeneous target points to be matched obtained in step S5 in the rectangular window of the first matching map are used. i ,β i ), i = 1 ~ n s The vector basis is applied to the second vector basis of the second graph to be matched to solve for the position of the predicted target within the candidate neighborhood rectangle;

[0020] S6. Calculate the minimum Euclidean distance between each predicted target and all actual targets within the candidate neighborhood rectangle corresponding to the j'th nearest neighbor line segment in the second matching image, and then add the minimum values ​​to obtain the Euclidean distances corresponding to the i'th nearest neighbor line segment in the first matching view and the j'th nearest neighbor line segment in the second matching view.

[0021] S7. Traverse the n nearest neighbor line segments in the first view to be matched, repeat steps S1 to S7, and calculate the elements of the cost matrix M based on the Euclidean distance values ​​of the i' nearest neighbor line segment in the first view to be matched and the j' nearest neighbor line segment in the second view to be matched, so as to obtain the cost matrix M, the size of the cost matrix M is n×m.

[0022] Preferably, the specific steps for multi-objective data association and heterogeneous multi-objective matching based on the Sinkhorn and cost matrix are as follows:

[0023] S7.1. Initialize the multi-target matching probability matrix P = exp[M / λ] based on the cost matrix M and the regularization parameter λ.-λM Initialize the row vector c = [1 / n, 1 / n…, 1 / n] with length n; initialize the column vector r = [1 / m, 1 / m…, 1 / m] with length m;

[0024] S7.2. Sum the columns of the multi-target matching probability matrix P to obtain the row middle vector P1. Divide the column vector r by the row middle vector P1 and multiply it by the multi-target matching probability matrix P to obtain the updated multi-target matching probability matrix P.

[0025] S7.3. Sum the rows of the multi-target matching probability matrix P to obtain the column middle vector P2. Divide the row vector c by the column middle vector P2 and multiply it by the multi-target matching probability matrix P to obtain the updated multi-target matching probability matrix P.

[0026] S7.4 Determine whether the multi-target matching probability matrix P converges. If it converges, end the process. If it does not converge, repeat S7.2-S7.3.

[0027] Preferably, the initial value of the regularization parameter λ is 100 to 1000.

[0028] Preferably, the method for determining whether the multi-target matching probability matrix P converges is as follows:

[0029] Before and after iterations in steps S7.2-S7.3, determine whether the maximum absolute value of the difference between the elements of matrix P when summing by row is less than the convergence threshold. If it is less than the convergence threshold, the multi-target matching probability matrix P is considered to be converged; otherwise, the multi-target matching probability matrix P is considered not to be converged.

[0030] Preferably, the convergence threshold is less than or equal to 0.01.

[0031] Preferably, the mapping relationship between the i'th line segment in the first view to be matched and the reference point of the rectangular window of the first image to be matched, and the coordinates (θ, ρ) of the reference point of the rectangular window of the first image to be matched in the first polar coordinate system, are as follows:

[0032]

[0033]

[0034] Among them, (x o ,y o The coordinates of the reference point O in the Cartesian coordinate system of the first matching graph are (x a ,y a Let (x) be the coordinates of endpoint a of line segment ab in the Cartesian coordinate system of the first matching graph. b ,y bLet ) be the coordinates of endpoint b of line segment ab in the first matching graph rectangular coordinate system. The first matching graph rectangular coordinate system takes the reference point O of the first matching graph rectangular window as the origin, the long side of the rectangular window as the X-axis, and the short side of the rectangular window as the Y-axis.

[0035] Preferably, the location of the heterogeneous target point C to be matched is represented in the following way:

[0036]

[0037] in, The vector is constructed with the rectangular window reference point as the starting point and the line segment endpoint A as the ending point;

[0038] The vector is constructed with the rectangular window reference point as the starting point and the line segment endpoint B as the ending point;

[0039] The vector is constructed with the rectangular window reference point as the starting point and the line segment endpoint C as the ending point;

[0040] α and β are the encoding parameters of the heterogeneous target points to be matched in the rectangular window of the first matching map.

[0041] Preferably, the position of the predicted target C″ within the candidate neighborhood rectangle is calculated according to the following formula:

[0042]

[0043] Preferably, the elements of the cost matrix are:

[0044]

[0045] Among them, dis i′,j′ Let th be the Euclidean distance between the i'-th nearest neighbor line segment in the first view to be matched and the j'-th nearest neighbor line segment in the second view to be matched. dis The matching threshold for predicting the Euclidean distance between the target location and the actual target location.

[0046] The advantages of this invention compared to the prior art are:

[0047] (1) The present invention first obtains the cost matrix by describing the local topological features, converts the image information into data information, and then combines the cost matrix with the Sinkhorn. By using the data processing method, heterogeneous multi-target matching is realized. This method has high robustness, low computational complexity, and strong application adaptability.

[0048] (2) The present invention constructs several line segments for the heterogeneous target information to be matched using the N nearest neighbor link method. This method can avoid the instability caused by the selection of a single line segment, ensure its robustness, and at the same time avoid a large amount of computation of fully connected line segments.

[0049] (3) The present invention constructs a set of bases with the vertices of the neighborhood rectangle and a set of line segment endpoints, and completes the linear representation and topological encoding of other targets within the neighborhood rectangle window in polar coordinates; this encoding method can adapt to transformations such as rotation, translation and scaling, and has higher robustness.

[0050] (4) The present invention uses the distance between the predicted target location and the actual location as a parameter to calculate the cost matrix, performs optimal transmission transformation on the cost matrix, and uses Sinkhorn to iteratively solve the problem to obtain the best data association result, thereby improving the adaptability to the target density and complexity of the matching scene. Attached Figure Description

[0051] Figure 1 This is a flowchart of topological feature extraction and multi-target matching using Sinkhorn, as described in an embodiment of the present invention.

[0052] Figure 2(a) is a schematic diagram of the first view to be matched in an embodiment of the present invention for line segment selection and a set of base structures;

[0053] Figure 2(b) is a schematic diagram of the second matching view of the embodiment of the present invention for line segment selection and a set of base structures;

[0054] Figure 3(a) is a schematic diagram of the selection of neighborhood rectangles and calculation of cost matrix for the first topological feature construction of the view to be matched in an embodiment of the present invention;

[0055] Figure 3(b) is a schematic diagram of the selection of neighborhood rectangles and calculation of cost matrix for the topological features of the second view to be matched in the embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of topological encoding within a neighborhood rectangle according to an embodiment of the present invention;

[0057] Figure 5 This is the result of multi-target matching based on topological feature extraction and Sinkhorn heterogeneous textureless information in an embodiment of the present invention. Detailed Implementation

[0058] The present invention will be further described below with reference to the embodiments.

[0059] This invention proposes a topological feature based on the neighborhood of feature points to achieve feature description of heterogeneous key points, and combines it with the optimal transport theory Sinkhorn algorithm to achieve multi-target matching from different sources. The process is as follows: Figure 1 As shown.

[0060] This invention discloses a method based on local topological feature description and a method based on Sinkhorn for heterogeneous multi-target matching. The method includes the following steps:

[0061] Obtain two views to be matched, designate one of them as the first view to be matched, and the other as the second view to be matched;

[0062] Construct several line segments from the heterogeneous target information to be matched in the first and second views using an N-nearest neighbor connection method;

[0063] The pre-constructed line segments of the first image to be matched are associated with their window positions to determine the mapping relationship. A set of bases is constructed using the endpoints of the line segments. The coding coefficients are obtained by linearly representing other heterogeneous target points to be matched in the view using this set of bases, and the topological feature description of the heterogeneous target points to be matched is obtained.

[0064] The mapping relationship is applied to the second map to extract neighborhood rectangular windows with the same topological features. Based on the principle of local topological feature invariance, the coding coefficients are used to predict the position of the heterogeneous target point to be matched within the neighborhood rectangle of the second map.

[0065] Calculate the cost matrix M based on the predicted target location and the actual target location in the second matching map;

[0066] Based on the cost matrix M, multi-objective data association is performed using the sinkhorn and the cost matrix, and heterogeneous multi-objective matching is completed.

[0067] Let x be the nearest neighbor line segment formed in the first view to be matched. i' Let y' = i' = 1 to n, and denote the nearest neighbor line segment formed in the second view to be matched as y'. j' j' = 1 to m, where n is the number of nearest neighbor segments in the first graph to be matched, and m is the number of nearest neighbor segments in the second graph to be matched;

[0068] The heterogeneous target information comes from radar and image information. An N-nearest neighbor (NNN) connection method is used to construct several line segments. This method ensures the robustness of the algorithm while reducing computational load compared to a fully connected approach.

[0069] The specific calculation process of the cost matrix M is as follows:

[0070] S1. Select the i'th nearest neighbor line segment in the first view to be matched, traverse the m nearest neighbor line segments in the second view to be matched, repeat steps S2 to S6, and then proceed to step S7.

[0071] S2. Select any vertex of the rectangular window of the first image to be matched as the reference point O of the rectangular window of the first image to be matched. Establish a first polar coordinate system with one endpoint of the i'th line segment in the first view to be matched as the origin and the line segment as the polar axis. Construct a mapping relationship between the i'th line segment in the first view to be matched and the reference point of the rectangular window of the first image to be matched in the first polar coordinate system.

[0072] S3. Using the reference point O of the first rectangular window to be matched as the starting point and the two endpoints A and B of the line segment as the ending points, construct a set of vectors. and As a basis (denoted as the first vector basis), the positions of all other target points to be matched in the first matching image rectangular window are linearly represented, and their coefficients are taken as the encoding parameters (α) of the target points to be matched in the first matching image rectangular window. i ,β i ), i = 1 ~ n s To obtain local topological features, n s This represents the number of heterogeneous target points to be matched in the first rectangular window to be matched.

[0073] S4. For the j'th nearest neighbor line segment in the second graph to be matched, take one of the endpoints of the line segment as the origin and the line segment as the polar axis in the second polar coordinate system. According to the principle that the candidate neighborhood rectangle after rotation, translation and scaling transformation coincides with the position of the second graph to be matched's own window rectangle in the second polar coordinate system and the first polar coordinate system, the candidate neighborhood rectangle window of the line segment in the second graph to be matched is obtained by using the mapping relationship in the first view to be matched in S4.

[0074] S5. Based on the principle of local topological feature invariance, construct a set of vectors with vertex O' of the candidate neighborhood rectangle as the vertex and the two endpoints A' and B' of the j'-th nearest neighbor line segment in the second graph to be matched as the endpoints. and As the second vector basis, the encoded parameters (α) of the acquired heterogeneous target points to be matched in the rectangular window of the first matching map are used. i ,β i ), i = 1 ~ n s The vector basis is applied to the second vector basis of the second graph to be matched to solve for the position of the predicted target within the candidate neighborhood rectangle;

[0075] S6. Calculate the minimum Euclidean distance between each predicted target and all actual targets within the candidate neighborhood rectangle corresponding to the j'th nearest neighbor line segment in the second matching image, and then add the minimum values ​​to obtain the Euclidean distances corresponding to the i'th nearest neighbor line segment in the first matching view and the j'th nearest neighbor line segment in the second matching view.

[0076] S7. Traverse the n nearest neighbor line segments in the first view to be matched, and repeat steps S3 to S8 to obtain the elements of the cost matrix M based on the Euclidean distance values ​​of the i' nearest neighbor line segment in the first view to be matched and the j' nearest neighbor line segment in the second view to be matched. The cost matrix M is n×m in size.

[0077] This invention enables the topological feature description and matching of heterogeneous multi-target information (such as radar and images) lacking image texture features, providing support for subsequent multi-source information fusion.

[0078] The specific steps of this invention are described in detail below:

[0079] 1. Segment initialization and selection

[0080] The heterogeneous target information to be matched is connected pairwise in an N-nearest neighbor manner to form line segments (as shown in Figures 2(a) and 2(b)). The line segment of the first view to be matched is denoted as x. i' ∈X, the line segment of the second view to be matched is denoted as y. j' ∈Y. Compared to directly connecting all targets, constructing line segments using N nearest neighbors can greatly compress the search space. Therefore, choosing the N nearest neighbor method to construct line segments can ensure the robustness of the algorithm while reducing the amount of computation and improving the running speed.

[0081] 2. Constructing the mapping relationship between line segments and the rectangular window of the first image to be matched.

[0082] Construct the line segment x in the first graph to be matched i' The encoding relationship between the four vertices of the rectangle corresponding to its field of view can be expressed as:

[0083] p = f(x) i' (1)

[0084] Where p is the coordinate of the vertex of the rectangular window of the first graph to be matched, and f(·) is the mapping from the line segment to the vertex coordinate.

[0085] Considering rotation, translation, and scaling transformations, the line segment y in the second match... j' It should also have the same mapping relationship with the coordinates q of the vertex of a certain rectangular window it is in, that is:

[0086] q=f(y j' (2)

[0087] The coordinates of q can be easily obtained from equations (1) and (2), thus the candidate neighborhood rectangles corresponding to the first matching graph can be obtained in the second matching graph, as shown in Figure 2(b).

[0088] 3. Location and representation of candidate neighborhood rectangles in the second matching graph

[0089] To eliminate the effects of rotation transformation, this invention uses polar coordinates to obtain the mapping relationship f(·) between line segments and vertices in the rectangular window of the first image to be matched, as shown in the first view to be matched in Figure 3(a), and constructs a polar coordinate system with a as the origin and ab as the polar axis.

[0090] The mapping relationship between the i'th line segment in the first view to be matched and the reference point of the rectangular window of the first image to be matched is represented by the coordinates (θ, ρ) of the reference point of the rectangular window of the first image to be matched in the first polar coordinate system:

[0091]

[0092]

[0093] Among them, (x o ,y o The coordinates of the reference point O in the Cartesian coordinate system of the first matching graph are (x a ,y a Let (x) be the coordinates of endpoint a of line segment ab in the Cartesian coordinate system of the first matching graph. b ,y b Let ) be the coordinates of endpoint b of line segment ab in the first matching graph rectangular coordinate system. The first matching graph rectangular coordinate system takes the reference point O of the first matching graph rectangular window as the origin, the long side of the rectangular window as the X-axis, and the short side of the rectangular window as the Y-axis.

[0094] Similarly, in the second view to be matched in Figure 3(b), a polar coordinate system is constructed with a′ as the origin and a′b′ as the polar axis. The coordinates of the upper left vertex can be represented as (θ′,ρ′). Then, (θ,ρ) and (θ′,ρ′) have the following relationship:

[0095] θ′ =θ (5)

[0096]

[0097] Therefore, the coordinates of o′ are obtained:

[0098]

[0099]

[0100] Equations (7) and (8) can be used to obtain the coordinates of the rectangular points of the matching neighborhood corresponding to the first matching neighborhood in the second matching map, thus completing the selection of candidate neighborhood rectangles.

[0101] 4. Local topological coding based on vector basis within the neighborhood

[0102] The remaining targets are described using a certain line segment as a reference. (See attached image.) Figure 4 As shown, a set of vectors is constructed with the top-left vertex O of the first matching view as the starting point and the two endpoints A and B of the line segment as the ending points. and (It must be ensured that the two vectors are not collinear). Using this set of vectors as a basis, represent all points in the neighborhood linearly:

[0103]

[0104] in, The vector is constructed with the rectangular window reference point as the starting point and the line segment endpoint A as the ending point;

[0105] The vector is constructed with the rectangular window reference point as the starting point and the line segment endpoint B as the ending point;

[0106] The vector is constructed with the rectangular window reference point as the starting point and the line segment endpoint C as the ending point;

[0107] α and β are the encoding parameters of the heterogeneous target points to be matched in the rectangular window of the first matching map.

[0108] Due to local topological invariance, this encoding remains unchanged in both views to be matched.

[0109] 5. Cost Matrix Construction

[0110] Construct the cost matrix as shown in Figure 3(a), and select a, b, and O in the first view to be matched to construct the vector basis. and The point c is then linearly represented (as shown in Equation 9), and its coefficients are taken as the local neighborhood encoding T1(α1, β1). Next, the neighboring points a′ and b′ in the second matching graph are traversed, and line segments a′b′ are constructed. The aforementioned polar coordinate representation is applied to line segments a′b′ to obtain the corresponding candidate box top-left vertex O′. A vector is then constructed using a′, b′, and O′. and By applying the local neighborhood encoding (T1) of the first view to be matched, the position where the target should appear in this candidate neighborhood rectangle can be determined, denoted as c″, as shown in Figure 3(b):

[0111]

[0112] The elements of the cost matrix are:

[0113]

[0114] Among them, dis i′,j′ Let th be the Euclidean distance between the i'-th nearest neighbor line segment in the first view to be matched and the j'-th nearest neighbor line segment in the second view to be matched. dis The matching threshold for predicting the Euclidean distance between the target location and the actual target location.

[0115] That is, if the Euclidean distance between the i'th nearest neighbor line segment in the first view to be matched and the j'th nearest neighbor line segment in the second view to be matched is greater than the matching threshold, then no matching will be performed. This parameter can filter obviously erroneous matching point pairs.

[0116] 6. Multi-target matching based on Sinkhorn

[0117] After obtaining the cost matrix, the problem can be expressed as:

[0118] d M =min∑P i,j M i,j (12)

[0119] Where P i,j For all possible matches:

[0120] U(r,c)={P∈R m*n |P1m=r,P1n=c} (13)

[0121] r is the expected allocation of set one (n x 1), and c is the expected allocation of set two (m x 1).

[0122] In addition, adding a regularization term λ makes the problem description more reasonable, namely:

[0123]

[0124] Also known as the Sinkhorn distance, in this application, it can achieve matching with the minimum cost matrix. The optimal solution of the above equation can be expressed as:

[0125]

[0126] Where γ i and η j The constant to be solved is obtained through iterative calculation. The iterative process is as follows:

[0127] (1) Initialize the multi-target matching probability matrix P = exp[M] based on the cost matrix M and the regularization parameter λ. -λM Initialize the row vector c = [1 / n, 1 / n…, 1 / n] with a length of n; initialize the column vector r = [1 / m, 1 / m…, 1 / m] with a length of m; in a specific embodiment of the present invention, the value of λ ranges from 100 to 1000.

[0128] (2) Sum the columns of the multi-target matching probability matrix P to obtain the row middle vector P1. Divide the column vector r by the row middle vector P1 and multiply it by the multi-target matching probability matrix P to obtain the updated multi-target matching probability matrix P.

[0129] (3) Sum the multi-target matching probability matrix P by row to obtain the column middle vector P2. Divide the row vector c by the column middle vector P2 and multiply it by the multi-target matching probability matrix P to obtain the updated multi-target matching probability matrix P.

[0130] (4) Determine whether the multi-target matching probability matrix P is converged. If it is converged, end the process. If it is not converged, repeat (2)-(3).

[0131] Before and after iterations (2)-(3), determine whether the maximum absolute value of the difference between the elements of matrix P by summing each row is less than the convergence threshold. If it is less than the convergence threshold, the multi-target matching probability matrix P is considered to be converged; otherwise, the multi-target matching probability matrix P is considered not to be converged.

[0132] In a specific embodiment of the present invention, the convergence threshold is less than or equal to 0.01.

[0133] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention by utilizing the methods and techniques disclosed above without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A heterogeneous multi-target matching method based on local topological feature description and Sinkhorn, the method involving image processing and multi-source information fusion, characterized in that... Includes the following steps: Obtain two views to be matched, designate one of them as the first view to be matched, and the other as the second view to be matched; Construct several line segments from the heterogeneous target information to be matched in the first and second views using an N-nearest neighbor connection method; The line segments pre-constructed in the first view to be matched are associated with their window positions to determine the mapping relationship. A set of bases is constructed with the endpoints of the line segments. The coding coefficients are obtained by linearly representing other heterogeneous target points to be matched in the view through this set of bases, and the topological feature description of the heterogeneous target points to be matched is obtained. The mapping relationship is applied to the second view to be matched to extract neighborhood rectangular windows with the same topological features. Based on the principle of local topological feature invariance, the coding coefficients are used to predict the position of the heterogeneous target point to be matched within the neighborhood rectangle of the second view to be matched. Calculate the cost matrix M based on the predicted target location and the actual target location in the second view to be matched; Based on the cost matrix M, multi-objective data association is performed based on the Sinkhorn and the cost matrix, and heterogeneous multi-objective matching is completed. The nearest neighbor line segment formed in the first view to be matched is denoted as . The nearest neighbor line segment formed in the second view to be matched is denoted as , This represents the number of nearest neighbor line segments in the first view to be matched. The specific calculation process of the cost matrix M, which represents the number of nearest neighbor line segments in the second view to be matched, is as follows: S1. Select the first view to be matched from the first view. Root nearest neighbor line segment, traverse the second view to be matched Find the root nearest neighbor segment, repeat steps S2 to S6, and then proceed to step S7; S2. Select any vertex of the first rectangular window of the view to be matched as the reference point O of the first rectangular window of the view to be matched, and establish a reference point O based on the first rectangular window of the view to be matched. A first polar coordinate system is constructed with one endpoint of the line segment as the origin and the line segment as the polar axis. Within this first polar coordinate system, the first view to be matched is constructed. The mapping relationship between the root line segment and the reference point of the first rectangular window of the view to be matched; S3. Using the reference point O of the first rectangular window to be matched as the starting point and the two endpoints A and B of the line segment as the ending points, construct a set of vectors. and As a basis, denoted as the first vector basis, it linearly represents the positions of all other target points to be matched in the first rectangular window of the view to be matched, and takes their coefficients as the encoding parameters of the target points to be matched in the first rectangular window of the view to be matched. To obtain local topological features, This represents the number of heterogeneous target points to be matched in the first rectangular window to be matched. S4, for the second view to be matched... Based on the principle that the candidate neighborhood rectangles, after rotation, translation, and scaling transformations, coincide with the position of the viewport rectangle of the second view to be matched in the second polar coordinate system and the position of the viewport rectangle of the second view to be matched in the first polar coordinate system, the candidate neighborhood rectangle window of the line segment in the second view to be matched is obtained by using the mapping relationship in the first view to be matched. S5. Based on the principle of local topological feature invariance, select vertex O' of the candidate neighborhood rectangle as the vertex, and the second vertex in the second view to be matched... Construct a set of vectors with the two endpoints A' and B' of the nearest neighbor line segment as the endpoints. and As the second vector basis, the encoded parameters of the heterogeneous target points to be matched obtained in step S3 in the first matching view rectangular window are used. The vector basis is applied to the second vector basis in the second view to be matched to solve for the position of the predicted target within the candidate neighborhood rectangle; S6. Calculate the second view to be matched. The minimum Euclidean distance between each predicted target and all actual targets within the candidate neighborhood rectangle corresponding to the root nearest neighbor segment is calculated, and then the minimum values ​​are summed to form the first matching view. The root nearest neighbor segment, the second segment to be matched in the second view The Euclidean distance to the root nearest neighbor segment; traversing the first view to be matched. Root nearest neighbor segment; S7. Repeat steps S1 to S6 to obtain the first view to be matched. The root nearest neighbor segment, the second segment to be matched in the second view The cost matrix M is calculated based on the Euclidean distance values ​​corresponding to the nearest neighbor line segments. The size of the cost matrix M is... ; The specific steps for multi-objective data association and heterogeneous multi-objective matching based on Sinkhorn and cost matrix are as follows: S7.

1. Initialize the multi-target matching probability matrix based on the cost matrix M and the regularization parameter λ. Initialize the row vector c = [1 / n, 1 / n…, 1 / n] with length n; initialize the column vector r = [1 / m, 1 / m…, 1 / m] with length m; S7.

2. Sum the columns of the multi-target matching probability matrix P to obtain the row middle vector P1. Divide the column vector r by the row middle vector P1 and multiply it by the multi-target matching probability matrix P to obtain the updated multi-target matching probability matrix P. S7.

3. Sum the rows of the multi-target matching probability matrix P to obtain the column middle vector P2. Divide the row vector c by the column middle vector P2 and multiply it by the multi-target matching probability matrix P to obtain the updated multi-target matching probability matrix P. S7.4 Determine whether the multi-target matching probability matrix P converges. If it converges, end the process. If it does not converge, repeat S7.2-S7.

3. The first or second view to be matched is an image.

2. The heterogeneous multi-target matching method based on local topological feature description and Sinkhorn according to claim 1, characterized in that... The initial value of the regularization parameter λ is 100~1000.

3. The heterogeneous multi-target matching method based on local topological feature description and Sinkhorn according to claim 1, characterized in that... The method for determining whether the multi-target matching probability matrix P is convergent is as follows: Before and after iterations in steps S7.2-S7.3, determine whether the maximum absolute value of the difference between the elements of matrix P when summing by row is less than the convergence threshold. If it is less than the convergence threshold, the multi-target matching probability matrix P is considered to be converged; otherwise, the multi-target matching probability matrix P is considered not to be converged.

4. The heterogeneous multi-target matching method based on local topological feature description and Sinkhorn according to claim 1, characterized in that... The convergence threshold is less than or equal to 0.

01.

5. The heterogeneous multi-target matching method based on local topological feature description and Sinkhorn according to claim 1, characterized in that... The first view to be matched The mapping relationship between the root line segment and the reference point of the first rectangular window to be matched, and the coordinates of the reference point of the first rectangular window to be matched in the first polar coordinate system. express: in, The coordinates of reference point O in the Cartesian coordinate system of the first matching view. Let 'a' be the coordinates of endpoint 'a' of line segment 'ab' in the Cartesian coordinate system of the first matching view. Let be the coordinates of endpoint b of line segment ab in the first matching view rectangular coordinate system. The first matching view rectangular coordinate system has the reference point O of the first matching view rectangular window as the origin, the long side of the rectangular window as the X-axis, and the short side of the rectangular window as the Y-axis.

6. The heterogeneous multi-target matching method based on local topological feature description and Sinkhorn according to claim 1, characterized in that... The location of the heterogeneous target point C to be matched is represented in the following way: in, Starting from the reference point of the rectangular window, the endpoints of the line segment The vector constructed for the endpoint; Starting from the reference point of the rectangular window, the endpoints of the line segment The vector constructed for the endpoint; Starting from the reference point of the rectangular window, the endpoints of the line segment The vector constructed for the endpoint; and The encoding parameters for the heterogeneous target points to be matched in the first matching view rectangle window.

7. The heterogeneous multi-target matching method based on local topological feature description and Sinkhorn according to claim 1, characterized in that... The elements of the cost matrix are: in, For the first view to be matched, the first The root nearest neighbor segment and the second line segment in the view to be matched The Euclidean distance to the nearest neighbor segment. The matching threshold for predicting the Euclidean distance between the target location and the actual target location.

Images