A joint learning method and system for structural representation and clustering of multi-source image fusion

Abstract

The invention discloses a structural representation and clustering joint learning method and system for multi-source image fusion. The invention comprises selecting samples from a sample matrix set of multi-source images to construct an initial dictionary matrix set, generating an initial transformation matrix set, a weight vector, a slack variable, a Lagrangian operator, a penalty factor and a number of iterations; calculating the transformation matrix set, the structural representation matrix, the slack variable, the weight vector, the Lagrangian operator, the penalty factor and the structural representation matrix of the th iteration, and if the preset termination condition is met, performing k-means processing on the structural representation matrix of the th iteration to obtain a clustering result; otherwise, adding 1 to the number of iterations and returning to continue iteration. The invention can effectively mine complementary information between multi-source image data, obtain a structural representation with high discriminability, has strong universality and robustness, and has the advantages of high computational efficiency, noise resistance and high recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a structural representation and clustering joint learning method and system for multi-source image fusion. Background Art

[0002] With the rapid development and popularization of image acquisition technology, the amount of image data is growing exponentially. The traditional method of relying on experts to study images has been seriously challenged. Therefore, using advanced machine learning methods to perform high-precision automatic identification of images has become a key issue that needs to be solved urgently.

[0003] Multi-source images such as visible light, infrared, multi-spectral and hyperspectral images can be regarded as projections of the same observed object in different spaces and spectral spaces, which depict the physical and chemical characteristics of the observed object from different aspects, laying the foundation for comprehensive mining of electromagnetic information of ground objects and achieving high-precision image clustering. Multi-source image data provides multi-dimensional and multi-level observation information, which can support, supplement and correct each other and provide accurate representation of the observed object. For example, in the application of using visible light images to identify urban roads, since such images usually only contain three spectral bands in the spectral range of 400 to 700 nanometers, it is impossible to obtain the spectral information of road paving materials such as asphalt, resulting in low road recognition accuracy. On the contrary, hyperspectral images can obtain hundreds of spectral bands in the spectral range of 400 to 2500 nanometers, and have the ability to capture spectral information of different materials such as asphalt and cement. Therefore, obtaining a multi-source image representation that integrates visible light and hyperspectral information will effectively improve the accuracy of road recognition. When face image recognition technology encounters complex lighting conditions such as backlight or weak light, the accuracy of face image recognition based on visible light decreases. Infrared images rely on the thermal radiation information of objects for imaging and are not affected by changes in lighting conditions. Therefore, by obtaining an image representation that fuses visible light and infrared information, the accuracy of face recognition can be significantly improved. Therefore, using certain technical means to mine different types of information in multi-source images and encode highly discriminative image representations is the key to improving the high-precision clustering of multi-source images.

[0004] Existing multi-source image representation methods are mainly divided into splicing-based, dictionary learning-based and structure learning-based methods. The splicing-based method directly splices the multi-source image features, which has the advantages of simple implementation and efficient calculation. However, the image representation obtained by this type of method usually faces problems such as high feature dimension, susceptibility to noise pollution, information redundancy, and completely abandons the potential association between different data sources. The dictionary learning-based method decomposes the image sample into the product of dictionary items and representation coefficients. Under the condition of satisfying specific constraints, the quality of the dictionary items and the quality of the representation coefficients are characterized by the reconstruction error of the image sample. Although this type of method can embed some prior knowledge of multi-source images into the model by setting constraints in the process of obtaining image representation, it has certain flexibility. However, they cannot capture the local structural association information between samples, and cannot model the potential category distribution between samples. Unlike the first two methods, the structure learning-based method obtains image representation by retaining the local structural association between multi-source samples. Existing methods usually use graph structures to achieve this goal, which has high time complexity and high space complexity, and places high demands on computing and storage resources, making it difficult to apply them on a large scale in practice. Summary of the invention

[0005] Technical problem to be solved by the present invention: In view of the above-mentioned problems in the prior art, a method and system for joint learning of structural representation and clustering of multi-source image fusion is provided. The present invention can effectively mine the complementary information between multi-source image data, obtain structural representation with high discriminability, has strong universality and robustness, and has the advantages of high computational efficiency, resistance to noise interference, and high recognition accuracy.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A structural representation and clustering joint learning method for multi-source image fusion, comprising:

[0008] S1, a set of sample matrices from the input multi-source images by random sampling Select samples to construct the initial dictionary matrix set , initialize and generate the initial transformation matrix set composed of random values , a weight vector consisting of a matrix whose elements are equal and sum to 1 , all element values ​​are The slack variables , all element values ​​are The Lagrangian operator composed of the matrix , greater than The penalty factor And the number of iterations ; Based on the sample matrix set and the transformation matrix set Initialize the structure representation matrix ;

[0009] S2, based on the sample matrix set , No. Dictionary matrix collection of iterations and the structural representation matrix , combined with the non-negative constraint on the transformation matrix, calculate the The transformation matrix set of iterations ;

[0010] S3, based on sample matrix collection , No. The transformation matrix set of iterations , No. Dictionary matrix collection of iterations , weight vector , Lagrangian operator and penalty factor , combining low-rank constraints and nearest neighbor constraints to calculate the The structural representation matrix of the iteration ;

[0011] S4, using the sample matrix set , No. The transformation matrix set of iterations and the structural representation matrix , and The weight vector of the iteration , Lagrangian operator and penalty factor Calculate the Slack variables for iterations ;

[0012] S5, using the sample matrix set , No. The transformation matrix set of iterations and the structural representation matrix , calculate the Dictionary matrix collection of iterations ; Using the sample matrix set , No. Dictionary matrix collection of iterations , transformation matrix set and the structural representation matrix , calculate the The weight vector of the iteration According to The Lagrangian operator of the iteration and penalty factor , and The structural representation matrix of the iteration and slack variables Update The Lagrangian operator of the iteration , and according to The penalty factor for the iteration Update The penalty factor for the iteration ;

[0013] S6, judging whether the preset termination condition is met, if the preset termination condition is met, jumping to the next step; otherwise, increasing the number of iterations k by 1, jumping to step S2;

[0014] S7, for The structural representation matrix of the iteration K-means processing is performed to obtain the clustering results.

[0015] Optionally, the sample matrix set of the multi-source image input in step S1 is The function expression is:

[0016] ,

[0017] In the above formula, For the The image sample matrix corresponding to the data source, ~ is the image sample matrix The 1st to nth samples in , is the number of data sources, is the dimension of the sample, is the image sample matrix The number of samples in; the initial dictionary matrix set is constructed in step S1 The function expression is:

[0018] ,

[0019] In the above formula, is the initial dictionary matrix set The dictionary matrix, ~ is the dictionary matrix In dictionary items, Indicates size , is the number of dictionary entries contained in the dictionary matrix, and the dictionary matrix Any i-th dictionary item in A collection of sample matrices from multiple input source images The function expression obtained is:

[0020] ，

[0021] In the above formula, is the image sample matrix Middle samples, is a random variable that conforms to a uniform distribution, is the sample matrix The number of samples included, represents the floor function; the transformation matrix set consisting of random values ​​constructed in step S1 The function expression is:

[0022] ,

[0023] In the above formula, is the initial transformation matrix set The A transformation matrix, ~ is the transformation matrix In transformation terms, Indicates size , is the number of transformation terms contained in the transformation matrix, is the number of data sources, and the transformation matrix Any element value in is between greater than 0 and less than 1; in step S1, based on the sample matrix set and the transformation matrix set Initialize the structure representation matrix The function expression is:

[0024]

[0025] In the above formula, is the structural representation matrix The Line Column element value, is the image sample matrix Middle samples, is the transformation matrix Middle transformation terms, , .

[0026] Optionally, in step S2, the first The transformation matrix set of iterations The function expression is:

[0027] ,

[0028] In the above formula, is the transformation matrix Corresponding to The value of the iteration, , is the number of data sources, the transformation matrix All elements in are greater than or equal to zero to achieve the mapping between the dictionary matrix and the structure representation matrix, and there are:

[0029] ,

[0030] In the above formula, For the The image sample matrix corresponding to the data source, For the Dictionary matrix collection of iterations The dictionary matrix, For the The structural representation matrix of the iteration.

[0031] Optionally, in step S3, the first The structural representation matrix of the iteration The function expression is:

[0032] ,

[0033] ,

[0034] In the above formula, represents the structural representation matrix, are model parameters with values ​​greater than zero, is the trace function of the matrix, whose value is equal to the sum of the values ​​of the diagonal elements of the matrix; For the The indicator matrix for the iteration, For the The structural representation matrix of the iteration The corresponding Laplace matrix is ​​calculated by singular value decomposition; For the The matrix of the iteration The corresponding Laplacian matrix; For the The penalty factor for the iteration; For the The slack variables for the iterations; For the The Lagrangian operator of the iteration, is the square of the norm; constraint Used to ensure that the structure representation matrix All elements in the are greater than or equal to zero, and the constraint Used to ensure that the structure representation matrix Middle List At most there are only parameters non-zero elements, the constraint Used to ensure that the structure representation matrix The sum of the elements in any row is ; and there is The matrix of the iteration The function expression is:

[0035] ,

[0036] In the above formula, Representative behavior and listed as A matrix whose elements are all zero, Representative behavior and listed as A matrix whose elements are all zero, For the The structural representation matrix of the iteration The transposition of k The indicator matrix for iterations:

[0037] ,

[0038] In the above formula, is the indicator matrix, the structure characterization matrix The cluster distribution consistency information contained in is encoded into the indicator matrix through the Laplacian matrix middle, is the identity matrix, the diagonal elements of the identity matrix are 1 and the other tuple values ​​are 0; is the indicator matrix The transpose of For the The matrix of the iteration The corresponding Laplacian matrix.

[0039] Optionally, in step S4, the first Slack variables for iterations The function expression is:

[0040] ,

[0041] In the above formula, represents the slack variable, is the number of data sources, , For the The weight vector of the iteration The element values, For the The image sample matrix corresponding to the data source, For the Dictionary matrix collection of iterations The dictionary matrix, For the The transformation matrix of the iteration , is the square of the norm; For the The penalty factor for the iteration; For the The structural representation matrix of the iteration is For the The Lagrangian operator of the iteration.

[0042] Optionally, in step S5, the first Dictionary matrix collection of iterations The function expression is:

[0043] ,

[0044] In the above formula, For the The dictionary matrix of the iteration, is the number of data sources, , and there are:

[0045] ,

[0046] In the above formula, Represents the action on the dictionary matrix A set of constraints to ensure that the dictionary matrix All element values ​​in are greater than zero, and the sum of the squares of each element in each dictionary item is , For the The image sample matrix corresponding to the data source, For the The transformation matrix of the iteration , For the The structural representation matrix of the iteration is is the square of the norm;

[0047] In step S5, the The weight vector of the iteration The function expression is:

[0048] ,

[0049] In the above formula, is the number of multi-source image data sources, is the weight vector The element value; Indicates that all elements of the weight vector are greater than or equal to zero. Represents the weight vector The sum of the absolute values ​​of all elements in is 1.

[0050] Optionally, in step S5, update The Lagrangian operator of the iteration The function expression is:

[0051] ,

[0052] In the above formula, For the The Lagrangian operator of the iteration, For the The penalty factor for the iteration, For the The structural representation matrix of the iteration is For the The slack variables of the iteration are updated in step S5. Iteration The function expression is:

[0053] ,

[0054] In the above formula, is the step length.

[0055] Optionally, the function expression of the termination condition preset in step S6 is:

[0056] ,

[0057] In the above formula, is the tolerance parameter, For the The structural representation matrix of the iteration is For the The slack variables for the iterations, express The infinite norm of the resulting difference.

[0058] In addition, the present invention also provides a structural representation and clustering joint learning system for multi-source image fusion, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the structural representation and clustering joint learning method for multi-source image fusion.

[0059] In addition, the present invention also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used to be programmed or configured by a microprocessor to execute the structural representation and clustering joint learning method for multi-source image fusion.

[0060] Compared with the prior art, the present invention mainly has the following advantages:

[0061] 1. The present invention constructs a structural representation and clustering model for adaptive fusion of multi-source images. By analyzing the consistency and difference information between multi-source images, a parameter-free automatic weighted multi-source image structural representation learning model is constructed. On the one hand, based on the assumption that multi-source images share the same structural representation, the common information between multi-source images is encoded. On the other hand, non-negative matrix decomposition is used to optimize the reconstruction error of different source images and encode the differences between multi-source images. It has the advantages of high clustering accuracy, strong robustness, good scalability, etc.

[0062] 2. When performing representation learning on multi-source image structural information, the present invention fully considers relevant prior knowledge such as local correlation of samples and label distribution, and models it as low-rank constraints and nearest neighbor constraints, which can significantly improve the discriminative ability of the obtained structural representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is a basic flow chart of a method according to a first embodiment of the present invention. DETAILED DESCRIPTION

[0064] Embodiment 1:

[0065] The application scenario of the method of this embodiment is given a set of multi-source images that contain visible light, infrared and hyperspectral images, which cover six different categories of targets: asphalt roads, vehicles, buildings, cement roads, parking lots, and gas stations. In order to better carry out urban space planning and construction, it is necessary to divide different types of targets into different clusters to clarify their spatial distribution. However, it is difficult to obtain spatial-spectral features that can simultaneously distinguish the above six different targets using only the spatial and spectral information provided by a single visible light, infrared or hyperspectral image, resulting in the clustering results being unable to meet the requirements of actual applications. In order to solve the above technical problems, such as Figure 1 As shown, the structural representation and clustering joint learning method for multi-source image fusion in this embodiment includes:

[0066] S1, a set of sample matrices from the input multi-source images by random sampling Select samples to construct the initial dictionary matrix set , initialize and generate the initial transformation matrix set composed of random values , a weight vector consisting of a matrix whose elements are equal and sum to 1 , all element values ​​are The slack variables , all element values ​​are The Lagrangian operator composed of the matrix , greater than The penalty factor And the number of iterations ; Based on the sample matrix set and the transformation matrix set Initialize the structure representation matrix ;

[0067] S2, based on the sample matrix set , No. Dictionary matrix collection of iterations and the structural representation matrix , combined with the non-negative constraint on the transformation matrix, calculate the The transformation matrix set of iterations ;

[0068] S3, based on sample matrix collection , No. The transformation matrix set of iterations , No. Dictionary matrix collection of iterations , weight vector , Lagrangian operator and penalty factor , combining low-rank constraints and nearest neighbor constraints to calculate the The structural representation matrix of the iteration ;

[0069] S4, using the sample matrix set , No. The transformation matrix set of iterations and the structural representation matrix , and The weight vector of the iteration , Lagrangian operator and penalty factor Calculate the Slack variables for iterations ;

[0070] S5, using the sample matrix set , No. The transformation matrix set of iterations and the structural representation matrix , calculate the Dictionary matrix collection of iterations ; Using the sample matrix set , No. Dictionary matrix collection of iterations , transformation matrix set and the structural representation matrix , calculate the The weight vector of the iteration According to The Lagrangian operator of the iteration and penalty factor , and The structural representation matrix of the iteration and slack variables Update The Lagrangian operator of the iteration , and according to The penalty factor for the iteration Update The penalty factor for the iteration ;

[0071] S6, judging whether the preset termination condition is met, if the preset termination condition is met, jumping to the next step; otherwise, increasing the number of iterations k by 1, jumping to step S2;

[0072] S7, for The structural representation matrix of the iteration K-means processing is performed to obtain the clustering results.

[0073] The structural representation and clustering joint learning method of multi-source image adaptive fusion in this embodiment includes inputting a set of multi-source image sample matrices, using non-negative matrix decomposition technology to jointly minimize the reconstruction error of multi-source image samples, and extracting dictionaries and structural representation matrices associated with different source images; using a non-parametric weight learning mechanism to quantify the impact of image quality of different data sources on structural representation, and improving the robustness of multi-source image joint representation; by acting on the dual constraints of low rank and nearest neighbor on the structural representation matrix, ensuring the consistency of multi-source image representation results in the local manifold structure and the identity with the potential class distribution; performing spectral clustering on the structural representation matrix to obtain the final clustering result. The structural representation and clustering joint learning method of multi-source image adaptive fusion in this embodiment can effectively measure the differences and consistency between multi-source images, and has the advantages of high clustering accuracy, strong robustness, and good scalability.

[0074] In this embodiment, the sample matrix set of the multi-source image input in step S1 is The function expression is:

[0075] ,

[0076] In the above formula, For the The image sample matrix corresponding to the data source, ~ is the image sample matrix The 1st to nth samples in is the image sample matrix Middle samples, For sample The characteristic dimension of , , is the sample matrix The number of samples included, is the number of data sources, is the dimension of the sample, is the image sample matrix In this embodiment, the image sample matrix Middle Samples It corresponds to one of the six targets: asphalt road, vehicle, building, cement road, parking lot, and gas station. This embodiment examines three data sources: visible light, infrared, and hyperspectral images. Therefore, there are .

[0077] In step S1, the initial dictionary matrix set is constructed The function expression is:

[0078] ,

[0079] In the above formula, is the initial dictionary matrix set The dictionary matrix, ~ is the dictionary matrix In dictionary items, Indicates size , is the number of dictionary entries contained in the dictionary matrix, and the dictionary matrix Any i-th dictionary item in A collection of sample matrices from multiple input source images The function expression obtained is:

[0080] ，

[0081] In the above formula, is the image sample matrix Middle samples, is a random variable that conforms to a uniform distribution, is the sample matrix The number of samples included, Represents the floor function; repeat Secondary dictionary item The acquisition process to realize the dictionary matrix Repeat the above dictionary matrix acquisition process for each data source to initialize the dictionary matrix set Since the dictionary matrix Any dictionary item in All are randomly sampled from the sample matrix So the dictionary item The feature of also corresponds to one of the six targets: asphalt road, vehicle, building, cement road, parking lot, and gas station. In practical applications, the number of dictionary items Usually much smaller than the sample size , so this embodiment takes . Representative is the floor function of the independent variable.

[0082] The transformation matrix set composed of random values ​​constructed in step S1 The function expression is:

[0083] ,

[0084] In the above formula, is the initial transformation matrix set The A transformation matrix, ~ is the transformation matrix In transformation terms, Indicates size , is the number of transformation terms contained in the transformation matrix, is the number of data sources, and the transformation matrix Any element value in is between greater than 0 and less than 1; in step S1, based on the sample matrix set and the transformation matrix set Initialize the structure representation matrix The function expression is:

[0085]

[0086] In the above formula, is the structural representation matrix The Line Column element value, is the image sample matrix Middle samples, is the transformation matrix Middle transformation terms, , .

[0087] In this embodiment, using the multi-source image sample matrix to initialize the corresponding dictionary matrix can avoid meaningless dictionary items caused by randomly initializing the dictionary matrix and accelerate the convergence of the algorithm. In addition, this embodiment also includes initializing the following parameters: , , , model parameters , the number of neighbors , the iteration number threshold wait.

[0088] Steps S2 to S6 are steps for solving model variables using an alternating iteration strategy.

[0089] Step S2 includes using the coordinate descent method to solve the transformation matrices corresponding to the three data sources of visible light, infrared and hyperspectral, and obtain the transformation matrix set Specifically, in this embodiment, the calculation step S2 is The transformation matrix set of iterations The function expression is:

[0090] ,

[0091] In the above formula, is the transformation matrix Corresponding to The value of the iteration, , is the number of data sources, the transformation matrix All elements in are greater than or equal to zero to achieve the mapping between the dictionary matrix and the structure representation matrix, and there are:

[0092] ,

[0093] In the above formula, For the The image sample matrix corresponding to the data source, For the Dictionary matrix collection of iterations The dictionary matrix, For the The structural representation matrix of the iteration. Transformation matrix Corresponding to The value of the iteration Middle Line Analytical solution for the elements The function expression is as follows:

[0094] ,

[0095] In the above formula, For the The transformation matrix of the iteration Line List elements, and With and Similar meanings, For the In the iteration The dictionary matrix corresponding to the image data source, Represents a dot product operation. Solve the transformation matrix element by element Analytical solution for each element in to obtain Iteration transformation matrix On this basis, the first Iterate the transformation matrix to calculate the transformation matrix set No. Iterations purpose.

[0096] In step S3, the The structural representation matrix of the iteration When introduced with The corresponding slack variable The non-convex neighbor constraint problem that is difficult to solve effectively is transformed into an optimization problem with an analytical solution, which can quickly capture the structural representation matrix shared by different image data sources. Specifically, in step S3, the first The structural representation matrix of the iteration The function expression is:

[0097] ,

[0098] ,

[0099] In the above formula, represents the structural representation matrix, are model parameters with values ​​greater than zero, is the trace function of the matrix, whose value is equal to the sum of the values ​​of the diagonal elements of the matrix; For the The indicator matrix for the iteration, For the The structural representation matrix of the iteration The corresponding Laplace matrix is ​​calculated by singular value decomposition; For the The matrix of the iteration The corresponding Laplacian matrix; For the The penalty factor for the iteration; For the The slack variables for the iterations; For the The Lagrangian operator of the iteration, is the square of the norm; constraint Used to ensure that the structure representation matrix All elements in the are greater than or equal to zero, and the constraint Used to ensure that the structure representation matrix Middle List At most there are only parameters non-zero elements, the constraint Used to ensure that the structure representation matrix The sum of the elements in any row is ; and there is The matrix of the iteration The function expression is:

[0100] ,

[0101] In the above formula, Representative behavior and listed as A matrix whose elements are all zero, Representative behavior and listed as A matrix whose elements are all zero, For the The structural representation matrix of the iteration The transpose of . express The trace is used to reflect the manifold structure of different targets to be identified in the label space. In addition, the structural representation matrix The following two constraints must be met: First, the matrix All elements in the ; Second, the matrix The number of non-zero elements in any row of , and the sum of the elements is , represents the number of neighbors closest to each target to be identified, and is used to encode the local manifold structure between targets. The structural representation matrix of the iteration When , the following formula is used to update the The structural representation matrix of the iteration Middle Line Column Elements Values:

[0102] ,

[0103] In the above formula, Satisfies the equation:

[0104] ,

[0105] Satisfies the equation:

[0106] ,

[0107] Satisfies the equation:

[0108] ,

[0109] matrix Middle The row element values ​​satisfy the inequality relationship , and Respectively represent The iteration indicates the matrix Row and OK, For the The penalty factor for the iteration, For the The first iteration of the correlation matrix Line row elements, For the The iterative Lagrangian operator Line Row element.

[0110] In this embodiment, the indicator matrix is ​​used in step S3. Measuring the distribution of the target to be clustered in the label space, and promoting the image samples with similar structural representation to be mapped to the same cluster as much as possible. Specifically, step S3 of this embodiment also includes updating the k The indicator matrix for iterations:

[0111] ,

[0112] In the above formula, is the indicator matrix, the structure characterization matrix The cluster distribution consistency information contained in is encoded into the indicator matrix through the Laplacian matrix middle, is the identity matrix, the diagonal elements of the identity matrix are 1 and the other tuple values ​​are 0; is the indicator matrix The transpose of For the The matrix of the iteration The corresponding Laplace matrix has:

[0113] ,

[0114] in, Representative behavior and listed as A matrix whose elements are all zero, Representative behavior and listed as A matrix whose elements are all zero, For the Iterative Structural Representation Matrix The transpose of the matrix Before Submatrix consisting of rows :

[0115] ,

[0116] Remaining Submatrix consisting of rows :

[0117] ,

[0118] in, and Represents the pair matrix The left singular matrix and the right singular matrix obtained by singular value decomposition are and They are all diagonal matrices. is the number of objects to be clustered in the multi-source image. In this embodiment, there are six types of objects to be clustered: asphalt roads, vehicles, buildings, cement roads, parking lots, and gas stations. .

[0119] In this embodiment, the slack variable is used in step S4 to measure the utility of the structural representation information in reconstructing different image samples. Specifically, in step S4, the first Slack variables for iterations The function expression is:

[0120] ,

[0121] In the above formula, represents the slack variable, is the number of data sources, , For the The weight vector of the iteration The element values, For the The image sample matrix corresponding to the data source, For the Dictionary matrix collection of iterations The dictionary matrix, For the The transformation matrix of the iteration , is the square of the norm; For the The penalty factor for the iteration; For the The structural representation matrix of the iteration is For the The Lagrangian operator of the iteration is:

[0122] ,

[0123] In the above formula, .

[0124] In step S5 of this embodiment, the calculation Dictionary matrix collection of iterations The method is to find the optimal dictionary matrix for each image data source to minimize the sample reconstruction error based on the structure representation matrix, thereby retaining the specificity of different image data sources. Specifically, in step S5, the first Dictionary matrix collection of iterations The function expression is:

[0125] ,

[0126] In the above formula, For the The dictionary matrix of the iteration, is the number of data sources, , and there are:

[0127] ,

[0128] In the above formula, Represents the action on the dictionary matrix A set of constraints to ensure that the dictionary matrix All element values ​​in are greater than zero, and the sum of the squares of each element in each dictionary item is , For the The image sample matrix corresponding to the data source, For the The transformation matrix of the iteration , For the The structural representation matrix of the iteration is is the square of the norm. In this embodiment, the classic block coordinate descent method is used to solve the dictionary matrix one by one. In this way, the entire dictionary can be updated. The dictionary matrix corresponding to the image data source For example, the update of the column The optimal solution for the iteration can be obtained by solving the following formula:

[0129] ,

[0130] In the above formula, ,symbol Representative Matrix No. OK, yes The analytical solution is as follows:

[0131] ,

[0132] In the above formula, the function Represents a projection operation that ensures The value is between 0 and 1. Each column of is calculated as above, and the updated According to the above formula, the dictionary matrices corresponding to the three data sources of visible light, infrared and hyperspectral are calculated one by one. No. Iterations, get the dictionary matrix set No. Iterations .

[0133] In this embodiment, step S5 calculates The weight vector of the iteration When considering the influence of unavoidable noise, the sample matrix in each data source cannot fully and accurately reflect the real spatial and spectral characteristics of the target. Therefore, the noise intensity contained in each data source is measured by combining the sample matrix reconstruction error, and the noise influence is further weakened by automatically adjusting the weight matrix. Specifically, in step S5, the calculation of the first The weight vector of the iteration The function expression is:

[0134] ,

[0135] In the above formula, is the number of multi-source image data sources, is the weight vector The element value; Indicates that all elements of the weight vector are greater than or equal to zero. Represents the weight vector The sum of the absolute values ​​of all element values ​​in is 1. In this embodiment, let:

[0136] ,

[0137] Then the vector Middle The elements are calculated as follows:

[0138] ,

[0139] In the above formula, due to the element value is always greater than or equal to zero, so the weight vector All elements of have values ​​greater than or equal to zero, and the sum of the elements is , thereby achieving the purpose of quantifying the noise intensity in different image data sources.

[0140] As an optional implementation, in step S5, the first The Lagrangian operator of the iteration The function expression is:

[0141] ,

[0142] In the above formula, For the The Lagrangian operator of the iteration, For the The penalty factor for the iteration, For the The structural representation matrix of the iteration is For the As an optional implementation, in step S5, the slack variables of the first iteration are updated. Iteration The function expression is:

[0143] ,

[0144] In the above formula, is the step length.

[0145] It should be noted that the termination condition preset in step S6 can be selected as needed, such as the number of iterations, or the error of the variation of the iteration variable meets the requirements. For example, as an optional implementation, the function expression of the termination condition preset in step S6 of this embodiment is:

[0146] ,

[0147] In the above formula, is the tolerance parameter, For the The structural representation matrix of the iteration is For the The slack variables for the iterations, express The infinite norm of the obtained difference. In this embodiment, the k-means processing structure characterization matrix is ​​used The clustering result is obtained. Since this embodiment includes 6 different targets to be clustered, k=6.

[0148] In order to obtain the effect of the structural representation and clustering joint learning method of multi-source image fusion in this embodiment, a clustering experiment and comparative analysis will be conducted on the method of this embodiment below. This experiment selects three image data sets, Caltech101-7, Caltech101-20 and Handwritten, and two comparative methods for comparative experimental analysis. Comparison method 1 and comparison method 2 are both existing well-known clustering methods based on bipartite graph regularization constraints. The difference is that comparison method 1 is a two-stage method. The first stage requires pre-definition of a bipartite graph for each data source, and the second stage performs collaborative optimization of different bipartite graph fusion and clustering. Comparison method 2 is a single-stage method, that is, bipartite graphs and clustering results can be learned directly from multi-source data. Caltech101-7 and Caltech101-20 are image data sets containing six data sources, of which Caltech101-7 contains 7 different targets and Caltech101-20 contains 20 different targets. Handwritten is a handwritten digital image data set containing four data sources, covering a total of 10 different targets from digits 0 to 9. The widely used evaluation index accuracy (ACC) is used to measure clustering performance. The larger the ACC value, the better the clustering result. First, the structural representation of the image dataset is obtained using the method of this embodiment. On this basis, the final clustering result is obtained using the k-nearest neighbor method. Table 1 lists the objective evaluation indicators of the method of this embodiment and the classic comparison method on the Caltech101-7, Caltech101-20 and Handwritten datasets.

[0149] Table 1: Comparison results of clustering accuracy between the method of this embodiment and comparative methods 1 and 2 on three data sets

[0150]

[0151] It can be seen from Table 1 that the clustering accuracy of the structural representation and clustering joint learning method of multi-source image adaptive fusion in this embodiment on the three data sets is better than that of comparison method 1 and comparison method 2, indicating that the structural representation and clustering joint learning of multi-source image adaptive fusion in this embodiment has a significant advantage in clustering accuracy.

[0152] In summary, the structural representation and clustering joint learning method of multi-source image adaptive fusion in this embodiment comprehensively analyzes the differences and consistency information of the target to be identified in the multi-source image data, and from the perspective of matrix decomposition, establishes a multi-source image structure sharing representation learning model, which fully combines the potential cluster distribution information and spatial-spectral information of the target, so as to obtain highly discriminative representation information; at the same time, considering the unavoidable noise problem in different data sources, through a learnable weighting strategy, the reconstruction error of different sample matrices is given a reasonable weight, and the model's anti-noise ability is improved without increasing the computational burden. The advantage of this embodiment is that the spatial, spectral and potential label distribution information of the target is encoded into the same representation matrix at the same time in a simple and efficient way, and is less affected by the quality differences of different data sources. Compared with other high-performance multi-source image representation and clustering methods, the clustering accuracy obtained by the structural representation and clustering joint learning method of multi-source image adaptive fusion in this embodiment is better.

[0153] In addition, this embodiment also provides a structure representation and clustering joint learning system for multi-source image fusion, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the structure representation and clustering joint learning method for multi-source image fusion. This embodiment also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used to be programmed or configured by the microprocessor to execute the structure representation and clustering joint learning method for multi-source image fusion.

[0154] Embodiment 2:

[0155] This embodiment is basically the same as the first embodiment, and the main difference is that the termination condition preset in step S6 is different. The termination condition preset in step S6 of this embodiment is that the number of iterations k is equal to the preset number of iterations threshold K_max. For example, in this embodiment, the number of iterations threshold K_max is set to 500, that is, the iteration is terminated after 500 iterations. At least by selecting a suitable number of iterations threshold K_max, the effect basically the same as that of the first embodiment can be achieved.

[0156] Those skilled in the art should understand that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0157] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A joint learning method of structural representation and clustering for multi-source image fusion, characterized in that: include: S1, a set of sample matrices from the input multi-source images by random sampling Select samples to construct the initial dictionary matrix set , initialize and generate the initial transformation matrix set composed of random values , a weight vector consisting of a matrix whose elements are equal and sum to 1 , all element values ​​are The slack variables , all element values ​​are The Lagrangian operator composed of the matrix , greater than The penalty factor And the number of iterations ; Based on the sample matrix set and the transformation matrix set Initialize the structure representation matrix ; S2, based on the sample matrix set , No. Dictionary matrix collection of iterations and the structural representation matrix , combined with the non-negative constraint on the transformation matrix, calculate the The transformation matrix set of iterations ; S3, based on sample matrix collection , No. The transformation matrix set of iterations , No. Dictionary matrix collection of iterations , weight vector , Lagrangian operator and penalty factor , combining low-rank constraints and nearest neighbor constraints to calculate the The structural representation matrix of the iteration ; S4, using the sample matrix set , No. The transformation matrix set of iterations and the structural representation matrix , and The weight vector of the iteration , Lagrangian operator and penalty factor Calculate the Slack variables for iterations ; S5, using the sample matrix set , No. The transformation matrix set of iterations and the structural representation matrix , calculate the Dictionary matrix collection of iterations ; Using the sample matrix set , No. Dictionary matrix collection of iterations , transformation matrix set and the structural representation matrix , calculate the The weight vector of the iteration According to The Lagrangian operator of the iteration and penalty factor , and The structural representation matrix of the iteration and slack variables Update The Lagrangian operator of the iteration , and according to The penalty factor for the iteration Update The penalty factor for the iteration ; S6, judging whether the preset termination condition is met, if the preset termination condition is met, jumping to the next step; otherwise, increasing the number of iterations k by 1, jumping to step S2; S7, for The structural representation matrix of the iteration K-means processing is performed to obtain the clustering results.

2. The structural representation and clustering joint learning method for multi-source image fusion according to claim 1, characterized in that: The sample matrix set of the multi-source image input in step S1 The function expression is: ， In the above formula, For the The image sample matrix corresponding to the data source, ~ is the image sample matrix The 1st to nth samples in , is the number of data sources, is the dimension of the sample, is the image sample matrix The number of samples in; the initial dictionary matrix set is constructed in step S1 The function expression is: ， In the above formula, is the initial dictionary matrix set The dictionary matrix, ~ is the dictionary matrix In dictionary items, Indicates size , is the number of dictionary entries contained in the dictionary matrix, and the dictionary matrix Any i-th dictionary item in A collection of sample matrices from the input multi-source image The function expression obtained is: ， In the above formula, is the image sample matrix Middle samples, is a random variable that conforms to a uniform distribution, is the sample matrix The number of samples included, represents the floor function; the transformation matrix set consisting of random values ​​constructed in step S1 The function expression is: ， In the above formula, is the initial transformation matrix set The A transformation matrix, ~ is the transformation matrix In transformation terms, Indicates size , is the number of transformation terms contained in the transformation matrix, is the number of data sources, and the transformation matrix Any element value in is between greater than 0 and less than 1; in step S1, based on the sample matrix set and the transformation matrix set Initialize the structure representation matrix The function expression is: In the above formula, is the structural representation matrix The Line Column element value, is the image sample matrix Middle samples, is the transformation matrix Middle transformation terms, , .

3. The structural representation and clustering joint learning method for multi-source image fusion according to claim 1, characterized in that: In step S2, the The transformation matrix set of iterations The function expression is: ， In the above formula, is the transformation matrix Corresponding to The value of the iteration, , is the number of data sources, the transformation matrix All elements in are greater than or equal to zero to achieve the mapping between the dictionary matrix and the structure representation matrix, and there are: ， In the above formula, For the The image sample matrix corresponding to the data source, For the Dictionary matrix collection of iterations The dictionary matrix, For the The structural representation matrix of the iteration.

4. The structural representation and clustering joint learning method for multi-source image fusion according to claim 1, characterized in that: In step S3, the The structural representation matrix of the iteration The function expression is: ， ， In the above formula, represents the structural representation matrix, are model parameters with values ​​greater than zero, is the trace function of the matrix, whose value is equal to the sum of the values ​​of the diagonal elements of the matrix; For the The indicator matrix for the iteration, For the The structural representation matrix of the iteration The corresponding Laplace matrix is ​​calculated by singular value decomposition; For the The matrix of the iteration The corresponding Laplacian matrix; For the The penalty factor for the iteration; For the The slack variables for the iterations; For the The Lagrangian operator of the iteration, is the square of the norm; constraint Used to ensure that the structure representation matrix All elements in the are greater than or equal to zero, and the constraint Used to ensure that the structure representation matrix Middle List At most there are only parameters non-zero elements, the constraint Used to ensure that the structure representation matrix The sum of the elements in any row is ; and there is The matrix of the iteration The function expression is: ， In the above formula, Representative behavior and listed as A matrix whose elements are all zero, Representative behavior and listed as A matrix whose elements are all zero, For the The structural representation matrix of the iteration The transposition of k The indicator matrix for iterations: ， In the above formula, is the indicator matrix, the structure characterization matrix The cluster distribution consistency information contained in is encoded into the indicator matrix through the Laplacian matrix middle, is the identity matrix, the diagonal elements of the identity matrix are 1 and the other tuple values ​​are 0; is the indicator matrix The transpose of For the The matrix of the iteration The corresponding Laplacian matrix.

5. The structural representation and clustering joint learning method for multi-source image fusion according to claim 1, characterized in that: In step S4, the Slack variables for iterations The function expression is: ， In the above formula, represents the slack variable, is the number of data sources, , For the The weight vector of the iteration The element values, For the The image sample matrix corresponding to the data source, For the Dictionary matrix collection of iterations The dictionary matrix, For the The transformation matrix of the iteration , is the square of the norm; For the The penalty factor for the iteration; For the The structural representation matrix of the iteration is For the The Lagrangian operator of the iteration.

6. The structural representation and clustering joint learning method for multi-source image fusion according to claim 1, characterized in that: In step S5, the Dictionary matrix collection of iterations The function expression is: ， In the above formula, For the The dictionary matrix of the iteration, is the number of data sources, , and there are: ， In the above formula, Represents the action on the dictionary matrix A set of constraints to ensure that the dictionary matrix All element values ​​in are greater than zero, and the sum of the squares of each element in each dictionary item is , For the The image sample matrix corresponding to the data source, For the The transformation matrix of the iteration , For the The structural representation matrix of the iteration is is the square of the norm; In step S5, the The weight vector of the iteration The function expression is: ， In the above formula, is the number of multi-source image data sources, is the weight vector The element value; Indicates that all elements of the weight vector are greater than or equal to zero. Represents the weight vector The sum of the absolute values ​​of all elements in is 1.

7. The structural representation and clustering joint learning method for multi-source image fusion according to claim 1, characterized in that: In step S5, update The Lagrangian operator of the iteration The function expression is: ， In the above formula, For the The Lagrangian operator of the iteration, For the The penalty factor for the iteration, For the The structural representation matrix of the iteration is For the The slack variables of the iteration are updated in step S5. Iteration The function expression is: ， In the above formula, is the step length.

8. The structural representation and clustering joint learning method for multi-source image fusion according to claim 1, characterized in that: The function expression of the termination condition preset in step S6 is: ， In the above formula, is the tolerance parameter, For the The structural representation matrix of the iteration is For the The slack variables for the iterations, express The infinite norm of the resulting difference.

9. A structural representation and clustering joint learning system for multi-source image fusion, comprising a microprocessor and a memory connected to each other, characterized in that: The microprocessor is programmed or configured to execute the structure representation and clustering joint learning method for multi-source image fusion as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored therein, characterized in that: The computer program is used to be programmed or configured by a microprocessor to execute the structure representation and clustering joint learning method for multi-source image fusion as described in any one of claims 1 to 8.

Images