Infrared and visible light image data association method based on multi-view clustering

Abstract

The invention discloses an infrared and visible light image data association method based on multi-view clustering, graph construction is completed by aiming at obtained data of different data sources, nodes represent data entities, and edges represent similarities or relationships among the entities. By means of a graph convolution network, input multi-view data features and a structure graph are mapped step by step through a series of nonlinear transformation, and features in a feature graph are extracted. After feature representation of data is independently learned from each view angle, the features are fused to obtain global feature representation, and meanwhile, features of similar samples in other different view angles are integrated to learn a common space. In addition, the multi-view clustering is guided and compared by using the pseudo labels, so that the multi-view data is effectively processed, and the accuracy and efficiency of association are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multi-source data association, and more specifically, relates to an infrared and visible light data image association method based on multi-view clustering. Background Art

[0002] With the advancement of remote sensing technology, computer vision, machine learning, and artificial intelligence, the current status of infrared and visible light data correlation technology is showing a rapid development trend, especially the application of deep learning, machine learning, and multimodal fusion methods, which has significantly improved the accuracy and effect of image fusion. In the future, with the optimization of algorithms and the improvement of hardware performance, the correlation of infrared and visible light data will be more efficient and accurate, and will be widely used in remote sensing, security, medicine, autonomous driving and other fields.

[0003] Traditional multi-source data association methods, such as probabilistic data association (PDA) and joint probabilistic data association (JPDA) algorithms, not only need to process a large amount of calculations, but also do not consider the birth and disappearance of targets during the association process. In addition, during the data fusion process, due to the uncertainty and inconsistency of multi-sensor data, traditional data association methods may find it difficult to handle these complexities. How to enable different data to be efficiently and accurately associated in various environments (especially in complex environments with large data volumes, high dimensions, and high noise) is a problem that multi-source data association methods must solve.

[0004] Traditional infrared and visible light data association methods, such as feature point matching, mutual information registration, and region matching, can work effectively in some simple scenarios, but they often present great challenges when dealing with complex situations such as cross-modality, illumination changes, occlusion, and resolution differences. Therefore, it is necessary to study a method that can improve the robustness of data association in different modalities. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for associating infrared and visible light image data based on multi-view clustering to achieve fast and accurate association of data of different modalities.

[0006] To achieve the above-mentioned object of the invention, the infrared and visible light data image association method based on multi-view clustering of the present invention is characterized by comprising the following steps:

[0007] (1) Graph construction: For each viewpoint, a graph is constructed. First, the entities in the image data are detected, and then nodes are used to represent data entities, and edges are used to represent the similarity or relationship between entities to construct the structural graph of the image.

[0008] (2) Multi-view representation learning: The overall goal is to process multi-view data. First, the view-specific representation is obtained by encoding the feature matrix and structure graph of each view. Then, in order to obtain a consistent view representation, intra-view contrast learning is carried out. The positive and negative sample pairs are determined according to the sample category and the loss is calculated. The loss is minimized to make the representation self-consistent with the pseudo-label. At the same time, inter-view contrast learning is carried out. The sample pairs are also determined and the loss is calculated. The loss is minimized to reduce the differences between heterogeneous views. Finally, the two losses are combined to make the representation learning sufficient, and then the view-specific representations are adaptively weighted to obtain a view-consistent representation, which provides a basis for clustering with comprehensive multi-view information.

[0009] (3) Multi-view clustering: The view-consistent representation learning module first encodes the sample features and the structure graph to obtain a fused representation, which provides input for the contrastive clustering module. In the contrastive clustering module, the clustering layer processes the input with a specific parameterized cluster center to obtain a soft cluster label. Since it is difficult to introduce auxiliary distributions in unsupervised learning optimization, a contrastive clustering loss is constructed based on the two, and the cluster center is iteratively optimized to make the cluster assignment discriminative, ultimately achieving automatic clustering association of data with comprehensive multi-view features;

[0010] (4) Update network parameters and complete data association: Comprehensively represent the losses of learning and clustering, construct the final objective function, update the network until convergence, calculate the cluster labels corresponding to the samples, and use the calculated cluster labels to automatically complete the data association task for infrared and visible light images.

[0011] The infrared and visible image data association method based on multi-view clustering can improve the performance of association by taking advantage of the complementarity and consistency of data. Multi-view clustering integrates information from different perspectives and can more comprehensively understand the intrinsic structure of the data. Specifically, the view-consistent representation learning first learns the nonlinear characteristics of the sample by simultaneously encoding the sample features and the structure graph, and then obtains the view-consistent representation through the adaptive fusion layer; contrastive clustering aims to enable the network to obtain cluster labels without post-processing operations. In addition, pseudo-labels (cluster labels) are used to guide network learning, so that they interact and evolve together under a unified optimization framework to learn more cluster-friendly representations.

[0012] The object of the present invention is achieved in this way.

[0013] The infrared and visible light image data association method based on multi-view clustering of the present invention collects data from different data sources, and then constructs a structural diagram of each perspective after preprocessing the data, in which nodes represent data entities and edges represent similarities or relationships between entities. Using deep learning methods, data representation is obtained from the graph, and representations of different perspectives are combined. Pseudo-labels (clustering labels) are used to guide network learning, so that they interact and evolve together under a unified optimization framework, learn a common space, complete multi-view clustering, and further realize automatic association of multi-source data. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flowchart of a specific implementation method of the infrared and visible light image data association method based on multi-view clustering of the present invention;

[0015] Figure 2 It is a schematic diagram of the graph construction in the present invention, demonstrating how to obtain an entity association graph from visible light and infrared image data;

[0016] Figure 3 It is the description of multi-view similarity learning in the present invention. The content in the box represents the current sample, different colors represent different clustering labels, and arrows in different directions represent the similarity judgment results between other samples;

[0017] Figure 4 It is a network framework diagram of a specific implementation method of the infrared and visible light image data association method based on multi-view clustering of the present invention. DETAILED DESCRIPTION

[0018] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0019] Figure 1 It is a flow chart of a specific implementation of the infrared and visible light image data association method based on multi-view clustering of the present invention.

[0020] In this embodiment, if Figure 1 As shown, the infrared and visible light image data association method based on multi-view clustering of the present invention comprises the following steps:

[0021] S1: Graph Construction

[0022] In multi-view clustering, nodes are used to represent data entities, and edges are used to represent the relationships between entities. Nodes and edges can be defined as follows:

[0023] Node definition: various entities are used as nodes of the graph, among which target entities (objects with clear physical meaning and identifiable), region entities (specific regions in the scene are defined as region entities), and feature point entities (key feature points in infrared and visible light images) can be used;

[0024] Edge definition: Connect corresponding nodes according to various defined relationships (spatial relationship, feature relationship, time relationship) to form the edge of the graph.

[0025] Figure 2 A graph construction method is proposed, which uses the target entity as the node and the spatial relationship as the edge to complete the graph construction. First, a detection algorithm (such as Faster R-CNN, YOLOX, etc.) is used to find the target entity in the image data. The target entity is used as a node, and at the same time, certain attribute information (such as lighting, contour, position, etc.) can be given to the node. Then, according to the spatial relationship between entities, that is, entities within a certain distance are connected, and those outside the range are not connected, so as to construct an entity association graph.

[0026] S2: Multi-view representation learning

[0027] 2.1) Single view representation

[0028] Usually, the feature matrix X of the vth view (v) And structure diagram A (v) The representation is heterogeneous, and the learning goal is to extract the view-specific representation by encoding the feature matrix and the structure graph simultaneously. To this end, for each view, the input {X (v) ,A (v)} is fed into a graph convolutional network (GCNs) to compute the view-specific representation M (v) :

[0029]

[0030] in, d r It is the dimension of perspective-specific representation; represents a l-layer multi-view encoder based on graph convolutional networks (GCNs), is the network parameter of GCNs. For the first layer encoding output, is the adjacency matrix with self-connections added, is its degree matrix; I is the identity matrix of A, and σ(·)=max(0,) represents the ReLU activation function.

[0031] By forward propagation, the vth viewing angle {X (v) ,A (v)} (v) Here, encoders of different perspectives share parameters, that is, they follow the same projection rules, in order to better align the potential representations of different perspectives. In the representation space, it is hoped that the similarity between representations of the same category should be as large as possible, and the similarity between representations of different categories should be as small as possible, so as to facilitate the subsequent clustering task.

[0032] 2.2) Viewpoint-consistent representation

[0033] Since the target to be associated is often not a single target and these targets may come from different perspectives, e.g. Figure 3 As shown, we should consider similarity learning within and between perspectives at the same time, bring data with similar labels closer, and push data with different labels away, so as to learn better consistent representations.

[0034] 2.2.1) In-view contrastive learning. For the i-th sample, from the in-view perspective, the samples belonging to the same class as the i-th sample are regarded as the "positive sample pairs" in each modality, and the other samples that do not belong to the same class as the i-th sample are regarded as the "negative sample pairs". Therefore, the in-view loss for:

[0035]

[0036] Among them, u and q represent the perspective of the data, and the set Π i represents all samples belonging to the same class as the i-th sample, which can be obtained based on the pseudo label; |Π i |YesΠ i is the cardinality of ; ρ is the similarity between samples; τ1 is the temperature coefficient. There are two cases for the value:

[0037]

[0038] It is worth noting that only The similarity with other sample representations, without considering This is because when u = q, if j = i, we know that the similarity It makes no sense to constrain it. Therefore, the i-th sample in the denominator is ignored in the formula. By minimizing This helps to make the learned view-specific representations consistent with the pseudo-labels in the latent space.

[0039] 2.2.2) Inter-view contrast learning. Its goal is to narrow the gap between heterogeneous perspectives as much as possible. Specifically, for the i-th sample, from the perspective of inter-perspective, the sample representations of the same class as the i-th sample in other perspectives are regarded as "positive sample pairs", and the samples of other perspectives that do not belong to the same class as the i-th sample are regarded as "negative sample pairs", thereby maximizing the consistency between the representations of the "positive sample pairs" and minimizing the consistency between the representations of the "negative sample pairs", and obtaining the inter-perspective loss for:

[0040]

[0041] Where g≠h means choosing different perspectives and minimizing Learn a common space where the differences between view-specific representations of similar samples are as small as possible.

[0042] Combining the intra-view loss and inter-view loss, we get the multi-view representation learning loss

[0043]

[0044] in, It represents the consistency within the perspective, requiring that the similarity of representations of samples of the same type in the vth perspective be as large as possible. It represents the consistency between views and aims to minimize the differences in representation between different views.

[0045] At this point, the viewpoint-specific representation M can be obtained (1) and M (2) In practical applications, different perspectives are different and their contributions to clustering are different. In order to integrate information from different perspectives and obtain clustering results with consistent perspectives, the representations of different perspectives are aligned with the help of clustering labels, and the consistent perspective representation M is obtained through adaptive weighted fusion:

[0046] M=γ·M (1) +(1-γ)·M (2)

[0047] Where γ is the parameter of the adaptive fusion layer, which is limited to the range of (0,1). and When there is no common cluster label, the two parts of the representation are calculated separately, such as And γ is adaptively updated through the gradient descent algorithm.

[0048] Since there is no clustering pseudo-label to guide the alignment of representations from different perspectives during the initial fusion, the encoders from different perspectives can share parameters, that is, follow the same projection rules, to align the potential representations from different perspectives. In addition, the Hungarian algorithm can be used to calculate the minimum loss and perform matching to preliminarily align different representations.

[0049] S3: Multi-view clustering

[0050] Most existing deep clustering methods require post-processing operations to obtain cluster labels, such as K-Means or spectral clustering. In the present invention, a contrast clustering layer is proposed to directly output cluster labels. The perspective-consistent representation learning module first learns the nonlinear characteristics of the sample by simultaneously encoding the sample features and the structure graph, and then obtains the perspective-consistent representation through the adaptive fusion layer; in addition, pseudo-labels (cluster labels) are used to guide network learning, which enables the two modules to interact and evolve together under a unified optimization framework to learn more cluster-friendly representations. This module is committed to enabling the network to obtain cluster labels without post-processing operations.

[0051] 3.1) Soft clustering labels

[0052] Specifically, the clustering layer consists of cluster centers Parameterization. This module takes the viewpoint-consistent representation M as input and the soft clustering labels As output, i-th row It can be interpreted as the similarity between the ith sample and the centers of different classes. The following formula is used to calculate

[0053]

[0054] where θ i,j =∥m i -μ j ∥ is the i-th sample representation m i and the jth cluster center μ j The distance between here For large pairwise distance ∥m i -μ j ∥ Approaching the inverse square law. It can be determined that sample m i Which class center is most similar to it, thus serving the sample division.

[0055] 3.2) Auxiliary distribution

[0056] For unsupervised learning, it is difficult to optimize without guidance information. An auxiliary distribution is introduced here To promote the clustering module to learn satisfactory cluster centers μ in a self-supervised manner. Auxiliary distribution It can be defined according to the following formula:

[0057]

[0058] in, is the frequency of the jth class center, used to normalize the loss contribution itself to prevent the “larger” class center from distorting the view-consistent representation. r ≥ 1 means adjustment to its rth power.

[0059] To iteratively optimize μ, a contrastive clustering loss is constructed

[0060]

[0061] Where τ2 is the temperature parameter, is the network parameter of GCNs, γ is the adaptive fusion layer parameter, and μ is the cluster center. When optimizing, minimize the cluster allocation and the auxiliary distribution of the i-th sample The difference between The degree of mismatch between the cluster distribution of the sample and other samples, thus making the learned cluster assignment more discriminative.

[0062] S4: Update network parameters

[0063] Combining the loss of representation learning and contrast clustering, the final objective function can be obtained as follows:

[0064]

[0065] Where λ is a trade-off parameter. By continuously optimizing and updating the network parameters until the network converges, the clustering label of the i-th sample is directly calculated using the following formula:

[0066]

[0067] By clustering data from different perspectives, samples with similar comprehensive characteristics (combining features from both infrared and visible light perspectives) can be divided into the same cluster, obtaining the same cluster label and realizing automatic association of data.

[0068] Figure 4 It is a network framework diagram of a specific implementation method of the infrared and visible light image data association method based on multi-view clustering of the present invention.

[0069] The network is mainly composed of a view-consistent representation learning module and a contrastive clustering module. The view-consistent representation learning module first learns the nonlinear characteristics of the sample by simultaneously encoding the sample features and the structure graph, and then obtains the view-consistent representation through an adaptive fusion layer; the contrastive clustering module is committed to enabling the network to obtain clustering labels without post-processing operations. In addition, pseudo labels are used to guide network learning, which enables the two modules to interact and co-evolve under a unified optimization framework to learn more clustering-friendly representations.

[0070] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A method for associating infrared and visible light image data based on multi-view clustering, characterized in that: The following steps are involved: (1) Graph construction: For each viewpoint, a graph is constructed. First, the entities in the image data are detected, and then nodes are used to represent data entities, and edges are used to represent the similarity or relationship between entities to construct the structural graph of the image. (2) Multi-view representation learning: The overall goal is to process multi-view data. First, the view-specific representation is obtained by encoding the feature matrix and structure graph of each view. Then, in order to obtain a consistent view representation, intra-view contrast learning is carried out. The positive and negative sample pairs are determined according to the sample category and the loss is calculated. The loss is minimized to make the representation self-consistent with the pseudo-label. At the same time, inter-view contrast learning is carried out. The sample pairs are also determined and the loss is calculated. The loss is minimized to reduce the differences between heterogeneous views. Finally, the two losses are combined to make the representation learning sufficient, and then the view-specific representations are adaptively weighted to obtain a view-consistent representation, which provides a basis for clustering with comprehensive multi-view information. (3) Multi-view clustering: The view-consistent representation learning module first encodes the sample features and the structure graph to obtain a view-consistent representation, which provides input for the contrastive clustering module. In the contrastive clustering module, the clustering layer processes the input with a specific parameterized cluster center to obtain a soft cluster label. Since it is difficult to introduce auxiliary distributions in unsupervised learning optimization, a contrastive clustering loss is constructed based on the two, and the cluster center is iteratively optimized to make the cluster assignment discriminative, ultimately achieving automatic clustering association of data with comprehensive multi-view features; (4) Update network parameters and complete data association: Comprehensively represent the losses of learning and clustering, construct the final objective function, update the network until convergence, calculate the cluster labels corresponding to the samples, and use the calculated cluster labels to automatically complete the data association task for infrared and visible light images.

2. The multi-source data automatic association method according to claim 1 is characterized in that the graph representation learning method described in step (2) and the clustering method described in step (3) are: 2.1) Single view representation Usually, the feature matrix X of the vth view (v) And structure diagram A (v) The representation is heterogeneous, and the learning goal is to extract the view-specific representation by encoding the feature matrix and the structure graph simultaneously. To this end, for each view, the input {X (v) ,A (v) } is fed into GCNs to compute the view-specific representation M (v) : in, d r It is the dimension of perspective-specific representation; represents a l-layer multi-view encoder based on graph convolutional networks (GCNs), is the network parameter of GCNs. For the first layer encoding output, is the adjacency matrix with self-connections added, is its degree matrix; I is the identity matrix of A, and σ(·)=max(0,) represents the ReLU activation function. By forward propagation, the vth viewing angle {X (v) ,A (v) Characterization of Here, encoders of different perspectives share parameters, that is, follow the same projection rules, in order to better align the potential representations of different perspectives. In the representation space, it is hoped that the similarity between representations of the same category should be as large as possible, and the similarity between representations of different categories should be as small as possible, so as to facilitate the subsequent clustering task. 2.2) Viewpoint-consistent representation Since the target to be associated is often not a single target and these targets may come from different perspectives, as shown in Figure 3, similarity learning within and between perspectives should be considered simultaneously, bringing data with similar labels closer and pushing data with heterogeneous labels away, so as to learn better consistent representations. 2.2.1) Intra-view contrastive learning. For the i-th sample, from the intra-view perspective, the samples belonging to the same class as the i-th sample are regarded as the "positive sample pairs" in each modality, and the other samples that do not belong to the same class as the i-th sample are regarded as the "negative sample pairs". Therefore, the intra-view loss for: in, u and q represent the perspective of the data, and the set Π i represents all samples belonging to the same class as the i-th sample, which can be obtained based on the pseudo label; |Π i |YesΠ i is the cardinality of ; ρ is the similarity between samples; τ1 is the temperature coefficient. There are two cases for the value: It is worth noting that only The similarity with other sample representations, without considering This is because when u = q, if j = i, we know that the similarity It makes no sense to constrain it. Therefore, the i-th sample in the denominator is ignored in the formula. By minimizing This helps to make the learned view-specific representations consistent with the pseudo-labels in the latent space. 2.2.2) Inter-view contrast learning. Its goal is to narrow the gap between heterogeneous perspectives as much as possible. Specifically, for the i-th sample, from the perspective of inter-perspective, the sample representations of the same class as the i-th sample in other perspectives are regarded as "positive sample pairs", and the samples of other perspectives that do not belong to the same class as the i-th sample are regarded as "negative sample pairs", thereby maximizing the consistency between the representations of the "positive sample pairs" and minimizing the consistency between the representations of the "negative sample pairs", and obtaining the inter-perspective loss for: Where g≠h means choosing different perspectives and minimizing Learn a common space where the differences between view-specific representations of similar samples are as small as possible. Combining the intra-view loss and inter-view loss, we get the multi-view representation learning loss in, It represents the consistency within the perspective, requiring that the similarity of representations of samples of the same type in the vth perspective be as large as possible. It represents the consistency between views and aims to minimize the differences in representation between different views. At this point, the view-specific representation M can be obtained (1) and M (2) In practical applications, different perspectives are different and their contributions to clustering are different. In order to integrate information from different perspectives and obtain clustering results with consistent perspectives, the representations of different modalities are aligned with the help of clustering labels, and the consistent perspective representation M is obtained through adaptive weighted fusion: M=γ·M (1) +(1-γ)·M (2) Where γ is the parameter of the adaptive fusion layer, which is limited to the range of (0,1). and When there is no common cluster label, the two parts of the representation are calculated separately, such as And γ is adaptively updated through the gradient descent algorithm. Since there are no clustering pseudo-labels to guide the alignment of representations from different perspectives during the initial fusion, the Hungarian algorithm can be used to calculate the minimum loss and perform matching to preliminarily align different representations. 3.1) Soft clustering labels Specifically, the clustering layer consists of cluster centers Parameterization. This module takes the viewpoint-consistent representation M as input and the soft clustering labels As output, i-th row It can be interpreted as the similarity between the ith sample and the centers of different classes. The following formula is used to calculate where θ i,j =∥m i -μ j ∥ is the i-th sample representation m i and the jth cluster center μ j The distance between here For large pairwise distance ∥m i -μ j ∥ Approaching the inverse square law. It can be determined that sample m i Which class center is most similar to it, thus serving the sample division. 3.2) Auxiliary distribution For unsupervised learning, it is difficult to optimize without guidance information. An auxiliary distribution is introduced here To promote the clustering module to learn satisfactory cluster centers μ in a self-supervised manner. Auxiliary distribution It can be defined according to the following formula: in, is the frequency of the jth class center, used to normalize the loss contribution itself to prevent "larger" class centers from distorting the view-consistent representation. r ≥ 1 means adjustment to its rth power. To iteratively optimize μ, a contrastive clustering loss is constructed Where τ2 is the temperature parameter, is the network parameter of GCNs, γ is the adaptive fusion layer parameter, and μ is the cluster center. When optimizing, minimize the cluster allocation and the auxiliary distribution of the i-th sample The difference between The degree of mismatch between the cluster distribution of the sample and other samples, thus making the learned cluster assignment more discriminative.

Images