A random sampling method for tensor ring decomposition of large-scale hyperspectral data based on cloud computing

Abstract

The present invention discloses a random sampling method for tensor ring decomposition of large-scale hyperspectral data based on cloud computing. The method comprises the following steps: reading large-scale hyperspectral tensor data, converting its format, and dividing it; calculating the number of sampled data J based on a header file, initializing n initial core tensors coreInitialize using a Gaussian distribution, and broadcasting J and coreInitialize to each computing node; performing histogram sampling J times in the computing node to generate a tensor data coordinate idx1, sampling the hyperspectral tensor data based on idx1; sampling the core tensor G based on idx1 to generate a sampling tensor; determining an unsampled core tensor Z based on the computing node, and calculating and updating the core tensor G using the least squares method; and determining an iterative termination condition. After the iteration ends, the calculated core tensor G is stored in a cloud computing platform. The present invention can quickly and accurately perform tensor ring decomposition on large-scale hyperspectral tensor data.

Description

Technical Field

[0001] The present invention belongs to the field of hyperspectral data processing, and in particular to a large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing. Background Art

[0002] Tensor data decomposition is an important research direction in the application of hyperspectral data. Hyperspectral data processing and decomposition often utilize tensor representation, which maximizes the preservation of the spatiotemporal characteristics of the original data. However, due to the multimodal and high-dimensional nature of tensor data, practical tensor data decomposition often faces single-machine bottlenecks, making it difficult for conventional computers to process large-scale hyperspectral tensor data. Tucker decomposition of tensor data is a commonly used method for large-scale hyperspectral tensor data. This method decomposes tensor data into several low-rank core tensors, which better reflect the dimensional characteristics of the tensor data and facilitate further processing. Compared with small-scale matrix data, Tucker decomposition of large-scale hyperspectral tensor data presents the following difficulties: 1) The data volume is large, making it difficult to load into memory for global decomposition; 2) The high dimensionality of the data requires numerous computations during the processing process; 3) The data index is large, often exceeding the maximum file size of a single file system; 4) The decomposition results are non-unique, with diverse results from Tucker decomposition of tensor data making convergence difficult; and 5) The tensor rank distribution is uncertain. Therefore, conventional matrix methods for processing tensor data cannot meet the needs of large-scale hyperspectral tensor data processing. It is necessary to develop distributed large-scale hyperspectral tensor data decomposition technology based on cloud computing, distributed file systems, randomization methods and other technologies based on the characteristics of large-scale hyperspectral tensor data, so as to more effectively process and store large-scale tensor hyperspectral data that are difficult to process in traditional matrices.

[0003] In recent years, with the advancement of computer storage technology and computing power, distributed computing technology has emerged as an effective technique for storing and processing large-scale hyperspectral tensor data. It employs a parallel, block-based approach to decompose computations, breaking through the bottleneck of single-machine computing and enabling computation on larger scales. Furthermore, its computational process is performed simultaneously and homogeneously, utilizing multiple computing nodes and resources, trading memory and computing power for time, thus accelerating computation. To process large-scale, complex tensor data, researchers have adopted distributed computing frameworks such as MapReduce and Graph to accelerate tensor data processing.

[0004] However, the above methods still have problems: 1) In the computing node, it is still subject to the memory and storage resources of the computing node, and the processing of large-scale hyperspectral tensor data still has certain limitations. 2) Although the use of distributed computing technology to calculate the Tucker decomposition of large-scale hyperspectral tensor data is one of the most popular methods for processing large-scale data. However, there is no method that can process data of different scales. Especially for small-scale data, distributed storage and computing have no advantages. 3) The determination of the tensor rank during the decomposition process is a major problem. The traditional tensor rank distribution is used as a priori information of the tensor data, but in actual data calculations, the rank of the tensor data is unknown and needs to be determined during the decomposition process, which has a high computational cost. Summary of the Invention

[0005] The purpose of the present invention is to provide a large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing.

[0006] The technical solution to achieve the purpose of the present invention is: a large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing, comprising the following steps:

[0007] Step 1: Read large-scale hyperspectral tensor data, convert and divide it into different formats; generate tensor data header files, and deploy the data files into blocks on the cloud platform;

[0008] Step 2: Calculate the number of sampled data J according to the header file, initialize n initial core tensors coreInitialize with Gaussian distribution, and broadcast J and coreInitialize to each computing node;

[0009] Step 3, sampling tensor data; in the computing node, the histogram is sampled J times to generate the tensor data coordinate idx1, the idx1 coordinate is sorted as idx2 according to the data storage, and the idx2 order is read and mapped to the idx1 order to generate the sampling tensor

[0010] Step 4: Sample the core tensor G according to idx1 to generate a sampling tensor Determine the unsampled core tensor Z according to the computing node, and calculate and update the core tensor G using the least squares method;

[0011] Step 5: Determine whether the iteration stop condition is met. If so, stop the iteration and store the core tensor G as the final result on the cloud computing platform. If not, broadcast the core tensor G as coreInitialize and jump to step 2 to continue the calculation.

[0012] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing is implemented.

[0013] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the aforementioned large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing.

[0014] Compared with the existing technology, the present invention has the following significant advantages: (1) It realizes tensor ring decomposition of large-scale data based on cloud computing technology, overcomes the bottleneck and resource limitation of a single machine, can process large-scale data that cannot be processed by a single machine, and fills the technical gap; (2) It uses a distributed file system to store large-scale hyperspectral tensor data, and designs a tensor data storage form, processing method and reading logic. It has a certain robustness for tensor data storage of different scales, which greatly helps data acquisition and processing when parallel computing tensor data; (3) In the process of tensor data processing, random sampling calculation is adopted, and the results of multiple random sampling calculations are used as the overall results of tensor data. While reducing the demand for computing resources, it can not only ensure the accuracy of calculation, but also make large-scale data tensor ring calculation more effective under distributed computing, and effectively reduce the time complexity of the algorithm, ensuring that the calculation time does not change exponentially when the order of tensor data increases, thereby greatly improving the computing efficiency.

[0015] The multi-scale hyperspectral image classification method provided by the present invention will be described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of large-scale data tensor ring decomposition based on cloud computing of the present invention. DETAILED DESCRIPTION

[0017] Combine Figure 1 The present invention proposes a random sampling method for tensor ring decomposition of large-scale hyperspectral data based on cloud computing, which can perform fast and accurate tensor ring decomposition of large-scale hyperspectral tensor data. The specific process of this method is as follows:

[0018] Step 1: Read large-scale hyperspectral tensor data, convert its format, and divide it. Generate a tensor data header file and divide the data file into blocks and deploy it to the cloud platform. The details are as follows:

[0019] For n-order large-scale hyperspectral tensor data A, its dimension is I1×I2×…×I n , the tensor rank distribution rank is {r1,r2,…,r n}, the generated tensor data header file format is:

[0020] Number of files for distributed storage of tensor data "file" = tensor data size / 4GB + 1

[0021] The rank of the tensor data "order" = n

[0022] The dimension of each order of tensor data is "dim" = I1, I2, ..., I n

[0023] The rank of each order of the tensor is "rank" = r1, r2, ..., r n or "null"

[0024] The data type of the tensor data is "type" = double, float, int, short, unsigned int

[0025] For tensor data A, the data division method is as follows:

[0026] Let the i-th index coordinate of the element in the tensor data A be dim[i-1]∈[0,I i ), the element coordinates in the tensor data A are (dim[n-1], dim[n-2], …, dim[0]), and the dim[1] and dim[0] coordinates are kept unchanged for slicing, and the element coordinates are converted to (idx, dim[1], dim[0]).

[0027] in,

[0028] Divide the tensor data A into file files. The data range contained in the kth file among the first file-1 files is The data range contained in the file file is

[0029] The naming rules and sampling data upload methods in the distributed system are as follows:

[0030] The header file is named "data name.hdr" and the data file is named "data name_file sequence number". The header file and data file are uploaded to the same directory of the distributed file system.

[0031] When sampling data, first obtain the tensor data information based on the header file to calculate the index and file number of the required data, and then extract the specific elements of the tensor data through the file number and index.

[0032] Calculate the number of samples J according to the header file. J represents the number of elements of the read tensor data calculated based on the tensor rank rank and tensor order n in the header file. Initialize n initial core tensors coreInitialize with Gaussian distribution. Broadcast J and coreInitialize to each computing node. The specific steps are as follows:

[0033] (2.1) According to the three parameters of tensor order, tensor dimension dim, and tensor rank in the header file, if the number of sampling elements J is determined as:

[0034]

[0035] If the tensor rank in the header file is empty, the estimated rank distribution is used. The estimated tensor rank distribution is determined by multiplying the fixed coefficient and the tensor dimension. The coefficient The tensor rank distribution generated by the i-th iteration calculation

[0036] (2.2) Initialize n core tensors coreInitialize through Gaussian distribution, the size of the i-th core tensor is r mod(i+n-1,n) ×J×r i .

[0037] (2.3) Broadcast data to each computing node. The specific broadcast data is as follows:

[0038] broadcast1=(dim,rank,max(rank),sum(rank),prod(dim))

[0039] broadcast2=(slowIdx,x)

[0040] broadcast3=(normCore,(coreInitaialize,samplingRescaling))

[0041] Where slowIdx is the tensor data index flattened J times and then stretched, normCore is the sum of the l2 norms of all core tensors, samplingRescaling is the sampling scaling factor, which is calculated using the factor matrix of the core tensor, and x is the executor number. Different executors calculate and update different core tensors.

[0042] In order to ensure that the data format initialized in each iteration is the same as the data format calculated in RDD, broadcast3 uses Tuple2 structure for data storage and processing.

[0043] Step 3: In the computing node, the histogram is sampled J times to generate the tensor data coordinate idx1, and the idx1 coordinate is sorted into idx2 according to the data partition, and the sampling tensor is generated in the order of idx2 and mapped to the order of idx1. The specific steps are as follows:

[0044] (3.1) In each executor, the complete tensor index is evenly divided to establish k histograms, ensuring that the frequency of sampled elements in each histogram is the same. Sample J times to obtain the tensor element coordinate idx1 to be extracted, and sort the idx1 coordinate into idx2 according to the data storage.

[0045] (3.2) According to the data sampling method described in step 1, extract tensor data according to idx2, and map the sampled data into sampling tensors in the order of idx1

[0046] In this step, not all data is loaded into memory. However, each iteration only requires loading the data into memory once, and the data is extracted in the order it is arranged, trading time for space. When the tensor data size is less than 500m, it is copied to memory once, persisted in memory using the cache() method, and broadcasted, eliminating the need to read the data repeatedly.

[0047] Step 4: Sample the core tensor G according to idx1 to generate a sampling tensor Determine the unsampled core tensor Z based on the computing node, calculate and update the core tensor G using the least squares method, and the specific steps are as follows:

[0048] (4.1) The calculation process is as follows:

[0049] Calculate the core tensor of the sampling

[0050] Where S represents the sampling coordinates that are the same as the corresponding sampling rule in step 3, and x represents the computation node number.

[0051] Compute core tensors

[0052] Here, Z represents the unsampled core tensor.

[0053] Calculate the sampling scaling factor

[0054] in, Represents the core tensor G (x) The left singular value matrix of the i-th column of the SVD decomposition. Represents the core tensor G (x) rank distribution of .

[0055] (4.2) The update process is as follows:

[0056] In each calculation, each executor randomly generates a sampling index matrix as the data in different partitions of RDD, and calculates different values ​​of mod(x,n) of broadcast2 in each executor. In the RDD calculated in this loop, the result of each calculation is result(key,value)=(normvalue,((core,samplingRescaling),x)). Where normvalue is the sum of the two norms of the core tensor core, core and samplingRescaling are the updated core tensor and scaling factor, respectively, and x is the partition number.

[0057] In the reduce phase, the normvalue of each partition is compared with the difference of the previous iteration, and the result with the largest difference is retained as the core tensor and scaling factor updated in this loop iteration.

[0058] Step 5: Determine whether the iteration termination condition is met. If so, stop and store the core tensor G of the calculation of this iteration as the final calculation result on the cloud computing platform. If not, broadcast the core tensor G as coreInitialize and jump to step 2 to continue the calculation. The specific steps are as follows:

[0059] (5.1) Determine the termination condition of the iteration, and the error of the two norms of the core tensor restored to the original tensor data becomes the error test of the sum of the two norms of the core tensor:

[0060]

[0061] in, represents the sum of the core tensor bi-norm after the i-th iteration, Represents the sum of the core tensor's two norms after the i-1th iteration.

[0062] If the tensor rank in the header file is empty, the termination condition of the iteration is:

[0063] ||tensor(core i )-tensor||2 / ||tensor||2<ε

[0064] In this formula, tensor(core i ) represents the tensor data restored based on the obtained core tensor, tensor represents the original tensor data, and ε represents the iteration accuracy, which depends on the accuracy requirements and is generally 0.05 or 0.1.

[0065] If the iteration termination condition is reached, the core tensor G calculated in this iteration is sorted according to the computing node sequence number x, and n files are generated and uploaded to the cloud platform.

[0066] (5.2) If the iteration termination condition is not met, the core tensor G is broadcast as follows:

[0067] First, perform the unpersist operation on broadcast3, and then broadcast the same variable format. The variable format is (normCore, (core, samplingRescaling)).

[0068] Furthermore, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing is implemented.

[0069] And, a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing.

[0070] In summary, the present invention is different from traditional tensor decomposition processing. It uses distributed computing as the main technical means and achieves better decomposition effect by sampling and calculating large-scale hyperspectral tensor data. In the distributed computing process, the originally indivisible iterative problem can be parallelized in a coarse-grained manner to solve the bottleneck problem of a single machine. Under the above-mentioned tensor ring decomposition framework, in the storage and reading of large-scale hyperspectral tensor data, slice storage is used and split into multiple files, so that the entire tensor data does not need to be read when reading the file; a distributed computing method is used to solve the bottleneck problem of a single machine; finally, when iterating and looping to calculate the core tensor, random sampling calculation and the method of estimating the population by samples can ensure that the error of the core tensor obtained after restoration is smaller than the original tensor, and effectively reduce the time complexity of the algorithm and the space complexity of the model solution, effectively alleviate the computational memory requirements, and greatly improve the computational efficiency.

Claims

1. A random sampling method for tensor ring decomposition of large-scale hyperspectral data based on cloud computing, characterized by: The following steps are involved: Step 1: Read large-scale hyperspectral tensor data, convert and divide it into different formats; generate tensor data header files, and deploy the data files into blocks on the cloud platform; Step 2: Calculate the number of samples J based on the header file. J represents the number of elements of the read tensor data calculated based on the tensor rank rank and tensor order n in the header file. Initialize n initial core tensors coreInitialize using Gaussian distribution. Broadcast J and coreInitialize to each computing node. The specific steps are as follows: (2.1) According to the three parameters of tensor order, tensor dimension dim, and tensor rank in the header file, the number of sampling elements J is determined as: If the tensor rank in the header file is empty, the estimated rank distribution is used; the estimated tensor rank distribution is determined by multiplying the fixed coefficient and the tensor dimension. The tensor rank distribution generated by the i-th iteration calculation (2.2) Initialize n core tensors coreInitialize through Gaussian distribution, the size of the i-th core tensor is r mod ( i+n-1,n )×J×r i ; (2.3) Broadcast data to each computing node. The specific broadcast data is as follows: broadcast1=(dim,rank,max(rank),sum(rank),prod(dim)) broadcast2=(slowIdx,x) broadcast3=(normCore,(coreInitaialize,samplingRescaling)) Where slowIdx is the tensor data index flattened J times and stretched, normCore is the sum of the l2 norms of all core tensors, samplingRescaling is the sampling scaling factor, calculated using the factor matrix of the core tensor, and x is the sequence number of the computing task. Different computing tasks calculate and update different core tensors. Step 3, sampling tensor data; sampling the histogram J times in the computing node to generate tensor data coordinate idx1, sorting the idx1 coordinate into idx2 according to the data storage, reading in the order of idx2 and mapping it to the order of idx1 to generate the sampling tensor Step 4: Sample the core tensor G according to idx1 to generate a sampling tensor Determine the unsampled core tensor Z based on the computing node, and calculate and update the core tensor G using the least squares method. The specific steps are as follows: (4.1) The calculation process is as follows: Calculate the core tensor of the sampling Where S represents the sampling coordinates with the same sampling rules as those in step 3, and x represents the computation node number. Compute core tensors Where Z represents the unsampled core tensor; Calculate the sampling scaling factor in, Represents the core tensor G (x) The left singular value matrix of the i-th column of SVD decomposition; Represents the core tensor G (x) rank distribution of ; (4.2) The update process is as follows: In each calculation, each computing task randomly generates a sampling index matrix as the data in different partitions of RDD, and calculates different values ​​of mod(x,n) of broadcast2 in each computing task. In the RDD calculated in this loop, the result of each calculation is result(key,value)=(normvalue,((core,samplingRescaling),x)); where normvalue is the sum of the two norms of the core tensor core, core and samplingRescaling are the updated core tensor and scaling factor respectively, and x is the partition number; In the reduce phase, the difference between the normvalue of each partition and the previous iteration is compared, and the core tensor and scaling factor result with the largest difference is retained as the result of this iteration; Step 5: Determine whether the iteration stop condition is met. If so, stop the iteration and store the core tensor G, the calculation result of this iteration, as the final result on the cloud computing platform. If not, broadcast the core tensor G as coreInitialize and jump to step 2 to continue the calculation.

2. The large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing according to claim 1 is characterized in that: In step 1, the data format of the tensor data header file includes: the data file number "file", the tensor order "order", the dimension of each order of the tensor "dim", the rank of each order of the tensor "rank", and the data type "type".

3. The large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing according to claim 1 is characterized in that: Step 1: Read large-scale hyperspectral tensor data, convert and divide it into different formats; generate tensor data header files, and deploy the data files in blocks to the cloud platform. The details are as follows: (1.1) For n-order large-scale hyperspectral tensor data A, its dimension is I1×I2×…×I n , the tensor rank distribution rank is {r1,r2,…,r n }, the generated tensor data header file format is: Number of files for distributed storage of tensor data "file" = tensor data size / 4GB + 1 The rank of the tensor data "order" = n The dimension of each order of tensor data is "dim" = I1, I2, ..., I n The rank of each order of the tensor "rank" = r1, r2, ..., r n or "null" The data type of the tensor data is "type" = double, float, int, short, unsignedint; (1.2) For tensor data A, the data division method is as follows: Let the i-th index coordinate of the element in the tensor data A be dim[i-1]∈[0,I i ), then the element coordinates in the tensor data A are (dim[n-1], dim[n-2], …, dim[0]), keep the dim[1] and dim[0] coordinates unchanged for slicing, and convert the element coordinates into (idx, dim[1], dim[0]); in, Divide the tensor data A into file files. The data range contained in the kth file among the first file-1 files is The data range contained in the file file is (1.3) The naming rules and sampling methods for uploading data in the distributed system are as follows: The header file is named "dataname.hdr" and the data file is named "dataname_file number". The header file and data file are uploaded to the same directory of the distributed file system; When sampling data, first obtain the tensor data information based on the header file to calculate the index and file number of the required data, and then extract the specific elements of the tensor data through the file number and index.

4. The large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing according to claim 1 is characterized in that: In step 3, the histogram is sampled J times in the computing node to generate the tensor data coordinate idx1, the idx1 coordinate is sorted into idx2 according to the data partition, and the order of idx2 is extracted and mapped to the order of idx1 to generate the sampling tensor The specific steps are as follows: (3.1) In each computational task, evenly split the complete tensor index to create k histograms, ensuring that the frequency of sampled elements in each histogram is the same. Sample J times to obtain the tensor element coordinate idx1 to be extracted, and sort the idx1 coordinate into idx2 according to the data storage; (3.2) Using the data sampling method described in step 1, read the tensor data according to idx2, and map the sampled data into the sampling tensor in the order of idx1 5. The large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing according to claim 1 is characterized in that: Step 5: Determine whether the iteration termination condition is met. If so, stop and store the core tensor G of the calculation of this iteration as the final calculation result on the cloud computing platform. If not, broadcast the core tensor G as coreInitialize and jump to step 2 to continue the calculation. The specific steps are as follows: (5.1) Determine the termination condition of the iteration, and the error of the two norms of the core tensor restored to the original tensor data becomes the error test of the sum of the two norms of the core tensor: in, represents the sum of the core tensor bi-norm after the i-th iteration, represents the sum of the core tensor's two norms after the i-1th iteration; If the tensor rank in the header file is empty, the termination condition of the iteration is: ||tensor(core i )-tensor||2 / ||tensor||2<ε In this formula, tensor(core i ) represents the tensor data restored based on the obtained core tensor, and tensor represents the original tensor data; If the iteration termination condition is reached, the core tensor G of this iteration calculation is sorted according to the computing node sequence number x, and n files are generated and uploaded to the cloud platform; (5.2) If the iteration termination condition is not met, the core tensor G is broadcast as follows: First delete broadcast3, and then broadcast the same variable format. The variable format is (normCore, (core, samplingRescaling)).

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing is implemented as described in any one of claims 1 to 5.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the large-scale hyperspectral data tensor ring decomposition random sampling method based on cloud computing is implemented.

Images