A multi-target merging method, medium and device in a cross-camera scenario

Abstract

The present invention discloses a method, medium, and device for merging multiple targets in a cross-camera scenario, relating to the field of image analysis technology. Through hierarchical clustering, particularly similar targets under different cameras are first merged into a cluster. The clusters are then merged into an undirected graph using threshold constraints based on feature similarity between clusters. Using graph analysis, the undirected graph is segmented using effective cut points to reduce mismerging caused by transition noise, thereby improving the accuracy of multi-target merging in cross-camera scenarios.

Description

Technical Field

[0001] The present invention relates to the field of image analysis technology, and in particular to a multi-target merging method, medium, and device in a cross-camera scenario. Background Art

[0002] In the field of video image analysis, multi-target merging across cameras is a crucial algorithmic area. In particular, in the field of multi-target tracking in video analysis, it is necessary to classify and merge identical targets captured simultaneously across different cameras, thereby obtaining different angles and postures of the same target for subsequent logical analysis and processing. Under different cameras, factors such as the target's angle, shape, posture, and lighting intensity can lead to significant differences in targets. Therefore, effectively analyzing logical relationships based on the target's position and features, and effectively merging targets, is crucial to the algorithm.

[0003] Currently, when merging targets across cameras, multiple hierarchical clustering is performed, and different threshold conditions are used to constrain the feature distance and spatiotemporal constraints between clusters, so as to merge the same targets and distinguish different targets. However, the threshold conditions have a "one-size-fits-all" phenomenon. Some jump noise images will cause many different targets to be merged into one category, resulting in frequent mismergers.

[0004] Therefore, how to solve the mis-merging phenomenon caused by hopping noise is the key problem to be solved by this technical solution. Summary of the Invention

[0005] In order to solve at least one of the technical problems mentioned in the background technology, the purpose of the present invention is to provide a multi-target merging method, medium and equipment in a cross-camera scenario, which can reduce the mismerging phenomenon caused by jump noise and improve the accuracy of cluster merging.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a multi-target merging method in a cross-camera scenario, comprising:

[0008] S101, acquiring multiple targets in images captured by multiple cameras within a preset time period;

[0009] S102, clustering based on feature similarities between the targets to obtain multiple clusters; each cluster includes a cluster center;

[0010] S103, calculating the feature similarity for the cluster centers of every two arbitrary clusters to obtain a similarity matrix; each row / column of the similarity matrix corresponds to a cluster center, and the value of the element in the i-th row and j-th column of the similarity matrix represents the feature similarity between the i-th cluster center and the j-th cluster center;

[0011] S104, constructing one or more undirected graphs based on the values of the elements in the similarity matrix, wherein the undirected graphs include a plurality of nodes and edges, wherein the nodes of the undirected graphs correspond one-to-one to the cluster centers of the clusters, and when the value of the element is greater than a first preset threshold, an edge exists between the two nodes corresponding to the element; and the sum of the number of nodes of all the undirected graphs is equal to the number of the clusters;

[0012] S105, traversing each undirected graph, calculating the degrees of all cut points in the undirected graph, and selecting cut points with degrees greater than n, where n is greater than or equal to 2;

[0013] S106, based on the feature similarities of the cut edges corresponding to the selected cut points, select the cut points whose feature similarities of the cut edges are less than a second preset threshold as valid cut points;

[0014] S107 , based on the valid cut points, split the undirected graph to obtain multiple subgraphs, and use the nodes corresponding to each subgraph as the merging result.

[0015] Furthermore, after S104, the step further includes:

[0016] For each undirected graph, determine whether the number of nodes of the undirected graph is greater than a third preset threshold. If so, execute S105 to S107. If not, take all nodes corresponding to the undirected graph as the merge result.

[0017] Furthermore, the S107 includes:

[0018] Traversing the undirected graph, without traversing the valid cut points during the traversal process, until every node in the undirected graph has been traversed, thereby obtaining a plurality of subgraphs;

[0019] Based on the degree of the effective cut point in each subgraph, obtaining a first correlation degree between the effective cut point and each subgraph;

[0020] When there is only one maximum value of the first association degree, deleting the valid cut point in the subgraph where the first association degree is not the maximum value;

[0021] When there are p maximum values of the first correlation, the valid cut point is deleted from any p-1 subgraphs with the maximum first correlation, and the valid cut point is deleted from the subgraphs where the first correlation is not the maximum value, wherein ;

[0022] The nodes corresponding to each subgraph are taken as the merging results.

[0023] Furthermore, when there are p maximum values of the first correlation degree, the sum of the edge values of the effective cut point in the corresponding p subgraphs is calculated respectively to obtain a second correlation degree between the effective cut point and the p subgraphs, wherein the edge value is the feature similarity corresponding to the edge;

[0024] If there is only one maximum value of the second correlation degree, delete the valid cut points in the subgraph where the second correlation degree is not the maximum value;

[0025] If there are q maximum values of the second correlation, delete the valid cut point in any q-1 subgraphs with the maximum second correlation, and delete the valid cut point in the subgraphs where the second correlation is not the maximum value, wherein .

[0026] Furthermore, the calculation formula of the feature similarity is:

[0027] ;

[0028] in, Represents the feature similarity between target X and target Y. The features of targets X and Y are both K-dimensional vectors, which are recorded as: , .

[0029] Furthermore, the feature of the cluster center is the average of the features of all targets in the cluster where the cluster center is located, or the feature of the target that is closest to the cluster center.

[0030] Furthermore, in S102, the clustering algorithm adopts a hierarchical clustering algorithm.

[0031] Furthermore, in S105 , the undirected graph is traversed using a depth-first traversal.

[0032] In a second aspect, an embodiment of the present invention further provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-target merging method in any cross-camera scenario as described above.

[0033] In a third aspect, an embodiment of the present invention further provides a terminal device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the multi-target merging method in any cross-camera scenario as described above is implemented.

[0034] Compared with the existing technology, the beneficial effects of the present invention are: through hierarchical clustering merging, particularly similar targets under different cameras are first merged into one cluster; the feature similarity between clusters is used to perform threshold constraints and merge into an undirected graph, and the graph analysis method is used to segment the undirected graph through effective cut points to reduce the mismerging phenomenon caused by jump noise, thereby improving the accuracy of multi-target merging in cross-camera scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flowchart of a multi-target merging method in a cross-camera scenario provided by an embodiment of the present invention;

[0036] Figure 2 A schematic diagram of a merging result when transition noise is not removed provided in an embodiment of the present invention;

[0037] Figure 3 A schematic diagram of a target image corresponding to a first sub-image provided by an embodiment of the present invention;

[0038] Figure 4 A schematic diagram of a target image corresponding to a second sub-image provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The following is a clear and complete description of the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of the present invention.

[0040] To address the aforementioned issues in the prior art, embodiments of the present invention disclose a method, medium, and device for merging multiple targets in a cross-camera scenario. It should be noted that the execution subject of the embodiments of the present invention can be a computer or any electronic device with data processing capabilities. For ease of description, the following detailed description will be based on an electronic device as the execution subject.

[0041] Example 1:

[0042] like Figure 1 As shown, Figure 1 A flowchart of a multi-target merging method in a cross-camera scenario provided by an embodiment of the present invention.

[0043] S101, acquiring multiple targets in images captured by multiple cameras within a preset time period;

[0044] Get the target set detections that appear under multiple cameras within a preset time: { ,..., },in, Indicates the position (x, y) of the target in the video image, the width and height of the target (width, height) and other information, , indicating that there are m targets in total. The preset duration is set according to actual conditions, such as ten minutes, thirty minutes, etc., and is not limited in the embodiment of the present invention.

[0045] The target may be any object to be observed, such as a pedestrian or an animal. In the embodiment of the present invention, the target refers to a pedestrian captured by a camera. The targets in the target set may belong to the same person or different people.

[0046] S102, clustering is performed based on feature similarity between the targets to obtain multiple clusters; each cluster includes a cluster center;

[0047] Each target image has a corresponding feature, which can be a pedestrian re-identification embedded feature based on deep learning, or a feature such as a color histogram. The corresponding embedded feature describing the above target image is Feas: , ,..., },in, It is a 1*K dimensional vector.

[0048] Clustering is performed according to the feature similarity between targets, ensuring that targets with similar features are merged into one cluster; different targets are classified into different clusters, and the clustering results are obtained: clusters: { , ,..., }, (n<=m), there are n clusters in total.

[0049] Remember the target content characteristics , They are: , , corresponding to , The formula for calculating the image content similarity between is:

[0050] ;

[0051] Preferably, the clustering algorithm adopts a hierarchical clustering algorithm.

[0052] The features corresponding to the cluster centers are recorded as:

[0053] { , ,..., },in, Is a 1*K dimensional vector, representing the cluster center characteristics.

[0054] Cluster center characteristics The method of obtaining can be the target feature closest to the cluster center, or all target features in the same cluster { , ,..., The average of the same dimension of} is calculated as:

[0055] ;

[0056] Here, C refers to the number of targets in the same cluster.

[0057] Calculate the pairwise feature distances of all cluster centers in clusters to obtain an n*n similarity matrix M, where ,express , At the same time, corresponding to n cluster data, the merged flag Set to true, .

[0058] S104: construct one or more undirected graphs based on the values of the elements in the similarity matrix. The undirected graphs include a plurality of nodes and edges. The nodes of the undirected graphs correspond one-to-one to the cluster centers of the clusters. When the value of an element is greater than a first preset threshold, an edge exists between the two nodes corresponding to the element. The sum of the number of nodes in all undirected graphs is equal to the number of clusters.

[0059] Through each , and construct one or more undirected graphs G(Vertexes,E) consisting of all cluster centers belonging to the same target. m ,Vertex n ,..,Vertex k} represents the nodes belonging to the same target, and {m,n,..k} is the cluster index set belonging to the target. m,n ,e k,n ,...} represents the valid edge connection set of these nodes, e m,n Represents a node {Vertex m , Vertex n}Similar pairs of the same target have the same value for an edge, and the corresponding edge value e n,m =e m,n By finding all clusters belonging to the same target, we can form an undirected graph G. The process of forming an undirected graph is as follows:

[0060] from (i=0) starts, if the current merge flag If false, it means that the cluster i Already traversed and analyzed, proceed to the next cluster traversal; if not found When cluster is true, it means that all clusters in the current clusters have been processed.

[0061] like True, indicating cluster i No analysis of the relationship with other clusters was performed. i As a node Vertex i Adding Vertexes to the node set; traversal and clustering i All adjacent cluster members that meet the conditions are added as new nodes to the Vertexes collection as follows:

[0062] If mergedFlag j True, search for cluster i and Feature similarity ,like Greater than the first preset threshold thresh, indicating , The corresponding nodes , is a set of similar pairs, so Vertex j Add to the node collection Vertexes; and at the node , Connecting edges, Assigned to ;at the same time, Set to false, preferably, thresh is set to 0.45; it should be noted that, , The values of are equal, so Also assigned to , no need to repeat the calculation.

[0063] Traverse the cluster as above i Adjacent node methods, recursive traversal and clustering j Other nodes whose similarity distance is greater than thresh (s≠i), if the constraints are met, cluster s Appends to Vertexes as a new node.

[0064] If the feature similarity values of all nodes in Vertexes and the remaining nodes in clusters are not greater than thresh, it means that the current cluster i The graph G(Vertexes,E) with as the starting point is constructed.

[0065] If there are still unprocessed clusters, select any one of the remaining clusters as the starting point to start traversal. The same steps as above are used to continue constructing the next undirected graph until all clusters are processed. i , are all included in the undirected graph; at this point, one or more undirected graphs are obtained.

[0066] S105, traversing each undirected graph, calculating the degrees of all cut points in the undirected graph, and selecting cut points with degrees greater than n, where n is greater than or equal to 2;

[0067] Preferably, the traversal algorithm adopts the DFS depth-first traversal algorithm to obtain the cut points and node degrees of G. In this embodiment, n is set to 2. A node degree less than 2 indicates that the relationship between the two nodes is relatively simple and cannot effectively split the cluster, which easily causes more clusters to split. Therefore, in order to effectively improve the algorithm segmentation effect, cut points with a degree greater than 2 are screened out.

[0068] S106, based on the feature similarities of the cut edges corresponding to the selected cut points, select the cut points whose feature similarities of the cut edges are less than a second preset threshold as valid cut points;

[0069] For the found cut point, traverse the cut edges around the cut point. If the cut edge exists, analyze the feature similarity of the cut edge corresponding to the cut point. If one of the cut edges is less than the second preset threshold EdgeThresh, it is determined to be a valid cut point and added to the cut point set cuts:{cut0,...,cut r}, r>0.

[0070] S107, based on the valid cut points, split the undirected graph to obtain multiple subgraphs, and use the nodes corresponding to each subgraph as the merging result;

[0071] According to the effective cut points, G is divided into multiple subgraphs { , ,...}, all nodes belonging to the same subgraph belong to the same target, and the set of nodes contained in each subgraph is the new merge result.

[0072] The above S107 can be further divided into the following steps:

[0073] Traverse the undirected graph without traversing valid cut points during the traversal process until every node in the undirected graph has been traversed, and multiple subgraphs are obtained;

[0074] Based on the degree of the effective cut point in each subgraph, a first correlation degree between the effective cut point and each subgraph is obtained;

[0075] When there is only one maximum value of the first correlation degree, deleting the valid cut point in the subgraph where the first correlation degree is not the maximum value;

[0076] When there are p maximum values of the first correlation, delete valid cut points in any p-1 subgraphs with the largest first correlation, and delete valid cut points in subgraphs where the first correlation is not the maximum value, where: ;

[0077] The nodes corresponding to each subgraph are taken as the merging results.

[0078] On the undirected graph G, perform DFS traversal to find the valid cut points in the cut point set cuts. i As the boundary point of the subgraph, no valid cut point is cut i Perform recursive traversal on adjacent nodes until all nodes in G are traversed and multiple subgraphs are obtained.

[0079] According to the first correlation degree between the effective cut point and multiple subgraphs, the effective cut point is retained in the subgraph with the largest first correlation degree with the effective cut point, and at the same time, the nodes corresponding to the effective cut point are deleted in other subgraphs; wherein the first correlation degree is the degree of the effective cut point in the subgraph.

[0080] When the DFS algorithm traverses a corresponding graph, a valid cut point may be repeatedly traversed as a boundary node in multiple subgraphs, resulting in it being assigned to multiple subgraphs. This embodiment of the present invention calculates and analyzes the degree of the subgraph containing the valid cut point as the first association degree during the DFS traversal of the subgraphs, ultimately determining which subgraph the valid cut point belongs to and deleting the valid cut point from other subgraphs.

[0081] For example, the graph G(V, E) has a valid cut point A. The DFS depth-first traversal algorithm is used to traverse the graph G. When a valid cut point A is encountered, it means that the subgraph boundary node has been reached. The adjacent nodes of point A are no longer recursively traversed, and the subgraph containing A is obtained. ; While traversing, count the subgraph Then continue the above traversal process for the remaining nodes of graph G. When encountering A, no longer recursively traverse the adjacent nodes of A until all the remaining nodes of G are traversed to obtain the subgraph ; During the traversal process, the subgraph is also counted The degree of A in the middle; finally we get two subgraphs 、 , A as a valid cut point will appear in both subgraphs, . The degree of node A is ;like , determine whether A and the graph The first correlation is large, so you need to Delete A from , A and FIG The first correlation is large, so we need to Delete A from , then retain the valid cut point A in any of the subgraphs and delete A in other subgraphs; thus ensuring that the two subgraphs The above-mentioned cutpoint degree voting method is also used in the scenario with multiple subgraphs to determine which subgraph the cutpoint should be optimally assigned to.

[0082] Assuming that three subgraphs are obtained, it means that the undirected graph originally classified as the same target is mismerged. In fact, there are three different targets in the undirected graph. The effective cut points are used for segmentation, which reduces the mismerging caused by the jump noise image.

[0083] like Figure 2 As shown, Figure 2 A schematic diagram of a merging result when transition noise is not removed is provided in an embodiment of the present invention. Figure 2 There are 14 target images in total, which are classified as the same target. However, there are two different pedestrians in the image. Due to the presence of a certain jump noise image, clusters that do not belong to the same target are merged together, which is a mismerging phenomenon.

[0084] Using effective cutpoint pairs Figure 2 After the corresponding undirected graph is split, two subgraphs are obtained. Figure 3 A schematic diagram of a target image corresponding to a first sub-image provided by an embodiment of the present invention; Figure 3 There are 6 pedestrian images in total, all corresponding to the same pedestrian; Figure 4 A schematic diagram of a target image corresponding to a second sub-image provided by an embodiment of the present invention, Figure 4 There are 8 pedestrian images in total, all corresponding to the same pedestrian; obviously, this technical solution can be used to Figure 2 Different pedestrians are effectively separated.

[0085] Through hierarchical clustering merging, targets that are particularly similar under different cameras are first merged into one cluster; the feature similarity between clusters is used to perform threshold constraints and merge into an undirected graph. Using graph analysis methods, the undirected graph is segmented by effective cut points, reducing the phenomenon of false merging caused by jump noise, thereby improving the accuracy of multi-target merging in cross-camera scenarios.

[0086] Example 2:

[0087] The difference from the first embodiment is that in step S107, when there are p maximum values of the first correlation degree,

[0088] Calculate the sum of the edge values of the effective cut points in the corresponding p subgraphs respectively to obtain the second correlation degree between the effective cut points and the p subgraphs, where the edge value is the feature similarity corresponding to the edge;

[0089] If there is only one maximum value of the second correlation, delete the valid cut points in the subgraph where the second correlation is not the maximum value;

[0090] If there are q maximum values of the second correlation, delete the valid cut points in any q-1 subgraphs with the largest second correlation, and delete the valid cut points in the subgraphs where the second correlation is not the maximum value, where: .

[0091] Still taking the example in the above embodiment 1 as an example, when When , calculate the sum of the edge values of the effective cut point A in G1 and G2 respectively, and record them respectively. The sum of the edge values of the midpoint A is like , determine whether A and the graph The second correlation is large, so you need to Delete A from , A and FIG The second correlation is large, so we need to Delete A from , then retain the valid cut point A in any of the subgraphs and delete A in other subgraphs; thus ensuring that the two subgraphs It is completely separated.

[0092] Although the degree of valid cutpoints in the subgraphs is equal, there may be differences in the feature similarity between the valid cutpoints and the connected nodes in the subgraphs, that is, there may be differences in the sum of the edge values. The higher the feature similarity, the closer the cluster represented by the valid cutpoint is to the subgraph. Therefore, the valid cutpoint is assigned to the subgraph with the second highest degree of association. This can further improve the accuracy of cluster merging.

[0093] Example 3:

[0094] The difference from Example 1 is that after constructing the undirected graph in step S104, it is determined whether the number of nodes in the undirected graph is greater than the third preset threshold ThreshCount. In this embodiment, ThreshCount is set to 5. If the number of nodes is greater than 5, S105~S107 are executed; if not, all nodes in the undirected graph are directly used as the merge result.

[0095] When the number of nodes is small, the probability of mismerging is very low, so there is no need to execute S105 to S107, which can reduce the consumption of network resources and improve the efficiency of image analysis.

[0096] Example 4:

[0097] Corresponding to the embodiment of the multi-target merging method in a cross-camera scenario, an embodiment of the present invention also provides a computer storage medium on which a computer program is stored, which, when executed by a processor, implements any of the above-mentioned multi-target merging methods in a cross-camera scenario.

[0098] Embodiment 5:

[0099] Corresponding to the embodiment of the multi-target merging method in a cross-camera scenario, an embodiment of the present invention also provides a terminal device, including a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein when the processor executes the computer program, it implements any of the above-mentioned multi-target merging methods in a cross-camera scenario.

[0100] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations that come within the meaning and range of equivalents of the claims be embraced therein.

Claims

1. A multi-target merging method in a cross-camera scenario, characterized in that: The following steps are involved: S101, acquiring multiple targets in images captured by multiple cameras within a preset time period; S102, clustering based on feature similarities between the targets to obtain multiple clusters; each cluster includes a cluster center; S103, calculating the feature similarity between the cluster centers of any two clusters to obtain a similarity matrix; Each row / column of the similarity matrix corresponds to a cluster center, and the value of the element in the i-th row and j-th column of the similarity matrix represents the feature similarity between the i-th cluster center and the j-th cluster center; S104, constructing one or more undirected graphs based on the values of the elements in the similarity matrix, wherein the undirected graphs include a plurality of nodes and edges, wherein the nodes of the undirected graphs correspond one-to-one to the cluster centers of the clusters, and when the value of the element is greater than a first preset threshold, an edge exists between the two nodes corresponding to the element; and the sum of the number of nodes of all the undirected graphs is equal to the number of the clusters; S105, traversing each undirected graph, calculating the degrees of all cut points in the undirected graph, and selecting cut points with degrees greater than n, where n is greater than or equal to 2; S106, based on the feature similarities of the cut edges corresponding to the selected cut points, select the cut points whose feature similarities of the cut edges are less than a second preset threshold as valid cut points; S107, based on the valid cut points, split the undirected graph to obtain multiple subgraphs, and use the nodes corresponding to each subgraph as the merging result; The S107 includes: Traversing the undirected graph, without traversing the valid cut points during the traversal process, until every node in the undirected graph has been traversed, thereby obtaining a plurality of subgraphs; Based on the degree of the effective cut point in each subgraph, obtaining a first correlation degree between the effective cut point and each subgraph; When there is only one maximum value of the first association degree, deleting the valid cut point in the subgraph where the first association degree is not the maximum value; When there are p maximum values of the first correlation, the valid cut point is deleted from any p-1 subgraphs with the maximum first correlation, and the valid cut point is deleted from the subgraphs where the first correlation is not the maximum value, wherein ; Take the nodes corresponding to each subgraph as the merge result; When there are p maximum values of the first association degree, the sum of the edge values of the effective cut point in the corresponding p subgraphs is calculated respectively to obtain a second association degree between the effective cut point and the p subgraphs, wherein the edge value is the feature similarity corresponding to the edge; If there is only one maximum value of the second correlation degree, delete the valid cut points in the subgraph where the second correlation degree is not the maximum value; If there are q maximum values of the second correlation, delete the valid cut points in any q-1 subgraphs with the maximum second correlation, and delete the valid cut points in the subgraphs where the second correlation is not the maximum value, wherein, .

2. The multi-target merging method in a cross-camera scenario according to claim 1, characterized in that: After S104, the method further includes: For each undirected graph, determine whether the number of nodes of the undirected graph is greater than a third preset threshold. If so, execute S105 to S107. If not, take all nodes corresponding to the undirected graph as the merge result.

3. The multi-target merging method in a cross-camera scenario according to claim 1, characterized in that: The calculation formula of the feature similarity is: ； in, Represents the feature similarity between target X and target Y. The features of targets X and Y are both K-dimensional vectors, which are recorded as: , .

4. The multi-target merging method in a cross-camera scenario according to claim 1, characterized in that: The feature of the cluster center is the average of the features of all targets in the cluster where the cluster center is located, or the feature of the target closest to the cluster center.

5. The multi-target merging method in a cross-camera scenario according to claim 1, characterized in that: In S102, the clustering algorithm adopts a hierarchical clustering algorithm.

6. The multi-target merging method in a cross-camera scenario according to claim 1, characterized in that: In S105 , the undirected graph is traversed using depth-first traversal.

7. A computer storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

8. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images