A real-time industrial chain big data distributed processing method and system

Abstract

The present application relates to the technical field of industry chain big data processing, and proposes a real-time industry chain big data distributed processing method and system, comprising the following steps: concatenating the industry chain big data of the same data type in the industry chain big data set as a local combined vector, calculating the local factor matrix corresponding to the local combined vector, iteratively updating the local factor matrix, obtaining the updated local factor matrix, calculating the intermediate matrix, uploading the intermediate matrix to the server, the server calculating the global public factor matrix, randomly distributing the global public factor matrix to each computing node, each computing node distributedly calculating the updated intermediate matrix and local feature tensor, continuously updating the intermediate matrix and local core feature tensor in the computing node, continuously updating the global public factor matrix in the server, obtaining the final intermediate matrix, final local core feature tensor and final global public factor matrix, and realizing distributed processing of the industry chain big data set.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial chain big data processing, and more specifically, to a real-time industrial chain big data distributed processing method and system. Background Art

[0002] In recent years, industrial manufacturing processes have become increasingly complex, with correspondingly increasing levels of automation. Industrial systems generate massive amounts of data in real time at extremely high speeds, constituting large amounts of industrial chain big data characterized by multiple sources, heterogeneity, high dimensionality, and continuous sampling. To promptly capture the information contained in this data on production processes, equipment status, and product quality, enabling producers to make rapid decisions and take action, methods for real-time processing of this data are urgently needed. Furthermore, because this data may contain varying degrees of privacy, safeguarding its privacy is crucial.

[0003] In summary, there is an urgent need for real-time and confidential processing methods for industrial chain big data.

[0004] The prior art proposes a method for distributed processing of order big data, which includes the following steps: order data cleaning, statistics on the amount of data to be processed, distribution of order data to several task nodes, business processing at each task node, and aggregation of task nodes, and receiving results of task nodes; this method distributes the order big data to be processed to several task nodes, and distributes the order big data through the task nodes. Because each task node only needs to process part of the data in the order big data, and each task node is calculated in parallel, it has a faster processing speed than centrally processing order big data in one computing node, and can complete the processing of order big data instantly, that is, it can achieve the effect of real-time processing of order big data; however, because this method directly sends the order data to the task node, the order data can be obtained by stealing the information sent to the task node, so this method does not have the performance of privacy and confidentiality; and compared with the industrial chain big data, the order big data processed by this method has fewer sources, is non-heterogeneous, and has lower dimensions, so the industrial chain big data cannot be processed in real time by this method.

[0005] Industrial chain big data is high-dimensional data. Feature extraction and dimensionality reduction of high-dimensional data is an important task in machine learning. The principle is to construct a new low-dimensional space to eliminate redundant and irrelevant features to effectively reduce the dimensionality of the original data. For the generalized multidimensional data structure of eigenvectors and the feature matrix composed of eigenvectors, research on high-dimensionalization and dimensionality reduction of eigenvectors has been carried out in many fields such as data structure, computer vision, and pattern recognition analysis. In the past few decades, commonly used feature extraction methods include wavelet transform, principal component analysis, kernel principal component analysis, independent component analysis, and linear discriminant analysis. However, these methods for processing and analyzing raw data all require vectorization of the original data, which largely destroys the internal structure of the original data. In addition, the vectorized representation of high-dimensional data will bring about the curse of dimensionality and higher computational complexity. Therefore, these commonly used feature extraction methods are not suitable for feature extraction of industrial chain big data, which is high-dimensional data.

[0006] The prior art proposes a high-dimensional power big data visualization method and system. The method includes: collecting and serializing global data on power distribution and utilization during power grid operation to obtain a high-dimensional power big data set; performing dimensionality reduction processing on the high-dimensional power big data set using a local linear embedding (LLE) algorithm based on manifold learning; and performing data mapping on the reduced power big data before sending it to a visualization platform to achieve visualization of the high-dimensional power big data. The local linear embedding used in this method is a manifold learning algorithm for nonlinear dimensionality reduction, which can achieve dimensionality reduction by preserving the original manifold structure of the data. However, this dimensionality reduction method has difficulty learning the multimodal and heterogeneous characteristics of industrial chain big data. Therefore, it is difficult to use this method to mine potential information in industrial chain big data, making it difficult to accurately analyze industrial chain big data. Moreover, this method directly sends the reduced dimensionality data to the visualization platform, which can easily lead to the leakage of private information contained in the data, making it unsuitable for processing data that may contain private information. Furthermore, this method takes a long time to process big data, making it impossible to perform real-time analysis and processing of industrial chain big data. Summary of the Invention

[0007] In order to overcome the defects of the above-mentioned prior art that the industrial chain big data cannot be accurately analyzed in real time and the privacy of data cannot be protected, the present invention provides a real-time industrial chain big data distributed processing method and system that can accurately analyze the industrial chain big data and protect data privacy.

[0008] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0009] A real-time industrial chain big data distributed processing method, based on a server and several computing nodes, includes the following steps:

[0010] Collect big data from the industrial chain to form a large data set of the industrial chain;

[0011] The industrial chain big data is concentrated, and the industrial chain big data with the same data type is cascaded and spliced ​​into local combination vectors, and the local combination vectors form a local combination vector set;

[0012] Using Tucker approximate decomposition and truncated singular value decomposition, calculate the set of local factor matrices corresponding to the set of local combination vectors;

[0013] Iteratively update the local factor matrix set until the iteration ends, and obtain the updated local factor matrix set;

[0014] Calculate the intermediate matrix set using the updated local factor matrix set;

[0015] The intermediate matrix set is uploaded to the server, and the server uses the intermediate matrix set to calculate the global common factor matrix set;

[0016] The global common factor matrix in the global common factor matrix set is randomly distributed to each computing node, and the updated intermediate matrix set and local feature tensor set corresponding to the global common factor matrix set are distributedly calculated by each computing node;

[0017] The intermediate matrix set and the local core feature tensor set are continuously updated in the computing nodes, and the global common factor matrix set is continuously updated in the server until the preset update stop condition is reached. Then, the updating of the intermediate matrix set, the local core feature tensor set and the global common factor matrix set is stopped to obtain the final intermediate matrix set, the final local core feature tensor set and the final global common factor matrix set, thereby realizing distributed processing of the industrial chain big data set.

[0018] The present invention also proposes a system for distributed processing of real-time industry chain big data for implementing the above-mentioned distributed processing method of real-time industry chain big data. The system includes:

[0019] The acquisition module is used to collect big data of the industrial chain and form a large data set of the industrial chain;

[0020] A cascade splicing module is used to cascade and splice the industrial chain big data with the same data type into local combination vectors, and form a local combination vector set from the local combination vectors;

[0021] A decomposition module is used to calculate a set of local factor matrices corresponding to a set of local combination vectors using Tucker approximate decomposition and truncated singular value decomposition;

[0022] An iteration module is used to iteratively update the local factor matrix set until the iteration ends and obtain an updated local factor matrix set;

[0023] An intermediate matrix calculation module, used to calculate an intermediate matrix set using the updated local factor matrix set;

[0024] A server calculation module is used to upload the intermediate matrix set to the server, and the server uses the intermediate matrix set to calculate the global common factor matrix set;

[0025] A computing node computing module is used to randomly distribute the global common factor matrix in the global common factor matrix set to each computing node, and to distributely compute the updated intermediate matrix set and local feature tensor set corresponding to the global common factor matrix set through each computing node;

[0026] The update module is used to continuously update the intermediate matrix set and the local core feature tensor set in the computing node, and continuously update the global common factor matrix set in the server until the preset update stop condition is reached, and then stop updating the intermediate matrix set, the local core feature tensor set and the global common factor matrix set, and obtain the final intermediate matrix set, the final local core feature tensor set and the final global common factor matrix set, so as to realize the distributed processing of the industrial chain big data set.

[0027] The present invention also proposes a computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor executes the steps of the real-time industrial chain big data distributed processing method proposed by the present invention.

[0028] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0029] The present invention uses Tucker approximate decomposition and truncated singular value decomposition to perform feature extraction and dimensionality reduction processing on the industrial chain big data, and obtains a set of local factor matrices, which can extract more effective features from the industrial chain big data and is conducive to accurate analysis of the industrial chain big data; the present invention also performs distributed processing on the industrial chain big data by randomly distributing the global public factor matrix set to each computing node. All computing nodes can calculate in parallel, which greatly shortens the time to obtain the industrial chain big data processing results and can process the industrial chain big data in real time; and the data exchanged between the server and each computing unit are only the intermediate matrix set and the global public factor matrix set. Even if the interactive data is stolen, it is impossible to reversely infer the industrial chain big data corresponding to the interactive data, thereby realizing privacy and confidentiality processing of the industrial chain big data. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the real-time industry chain big data distributed processing method of Example 1;

[0031] Figure 2 This is a schematic diagram of the performance comparison results using the ORL industrial monitoring system dataset in Example 2;

[0032] Figure 3 Schematic diagram of the performance comparison results of the centralized method and the distributed method in Example 2;

[0033] Figure 4 This is a schematic diagram of the performance comparison results using the Fabrics dataset in Example 2;

[0034] Figure 5 This is a schematic diagram of the performance comparison results using the KNN classifier and the SVM classifier in Example 2;

[0035] Figure 6 This is the overall framework diagram of the system for distributed processing of real-time industrial chain big data in Example 3. DETAILED DESCRIPTION

[0036] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0037] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0038] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0039] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0040] Example 1

[0041] This embodiment proposes a real-time industry chain big data distributed processing method. Figure 1 This is a flow chart of the real-time industry chain big data distributed processing method of this embodiment.

[0042] The real-time industry chain big data distributed processing method proposed in this embodiment is implemented based on a server and several computing nodes and includes the following steps:

[0043] Collect big data from the industrial chain to form a large data set of the industrial chain;

[0044] The industrial chain big data is concentrated, and the industrial chain big data with the same data type is cascaded and spliced ​​into local combination vectors, and the local combination vectors form a local combination vector set;

[0045] Using Tucker approximate decomposition and truncated singular value decomposition, calculate the set of local factor matrices corresponding to the set of local combination vectors;

[0046] Iteratively update the local factor matrix set until the iteration ends, and obtain the updated local factor matrix set;

[0047] Calculate the intermediate matrix set using the updated local factor matrix set;

[0048] The intermediate matrix set is uploaded to the server, and the server uses the intermediate matrix set to calculate the global common factor matrix set;

[0049] The global common factor matrix in the global common factor matrix set is randomly distributed to each computing node, and the updated intermediate matrix set and local feature tensor set corresponding to the global common factor matrix set are distributedly calculated by each computing node;

[0050] The intermediate matrix set and the local core feature tensor set are continuously updated in the computing nodes, and the global common factor matrix set is continuously updated in the server until the preset update stop condition is reached. Then, the updating of the intermediate matrix set, the local core feature tensor set and the global common factor matrix set is stopped to obtain the final intermediate matrix set, the final local core feature tensor set and the final global common factor matrix set, thereby realizing distributed processing of the industrial chain big data set.

[0051] During the specific implementation process, this embodiment uses Tucker approximate decomposition and truncated singular value decomposition to perform feature extraction and dimensionality reduction processing on the industrial chain big data, and obtains a local factor matrix set, which can extract more effective features from the industrial chain big data and is conducive to accurate analysis of the industrial chain big data; the intermediate matrix set is calculated through the local factor matrix set, and the intermediate matrix set is uploaded to the server. Because the intermediate matrix set only contains part of the features of the industrial chain big data, even if the information of the intermediate matrix set is leaked, the corresponding industrial chain big data cannot be reversed through the intermediate matrix set; the server uses the intermediate matrix set to calculate the global public factor matrix set, and distributes the global public factor matrix set to each computing node, thereby achieving the purpose of distributed real-time processing of the industrial chain big data; because the intermediate matrix set only contains part of the features of the industrial chain big data, the corresponding industrial chain big data cannot be reversed through the global public factor matrix set, thereby achieving the purpose of privacy protection processing of the industrial chain big data.

[0052] In an optional embodiment, the steps of cascading and splicing industrial chain big data with the same data type are as follows:

[0053] The industrial chain big data set is represented in the form of tensor; Big data of industry chains The expression is:

[0054]

[0055] Where, Indicates size N-dimensional real field tensor, 、 and Represents the size of the first dimension, second dimension, and Nth dimension of an N-dimensional tensor respectively; Indicates the total number of data types of the industrial chain big data in the industrial chain big data set;

[0056] The industrial chain big data is concentrated, and the industrial chain big data with the same data type is cascaded and spliced ​​into a local combination vector, and the local combination vectors form a local combination vector set;

[0057] Among them, Local combination vector The calculation expression is:

[0058]

[0059]

[0060] Where, Represents a function used to concatenate K industry chain big data into an N+1 dimensional vector.

[0061] As an example, the industrial chain big data set includes industrial chain big data of several data types, such as industrial chain big data of text data type, industrial chain big data of image data type, and industrial chain big data of audio data type, etc. As an example, the data types corresponding to different entity types are also different. For example, the image data corresponding to one entity type is of a different data type from the image data corresponding to another entity type.

[0062] In this optional embodiment, by concatenating industrial chain big data of the same data type into a local combination vector, each intermediate matrix, each local core feature tensor and each global common factor matrix in the final intermediate matrix set, the final local core feature tensor set and the final global common factor matrix set corresponds to industrial chain big data of a data type. Then, when new industrial chain big data is input, the intermediate matrix, local core feature tensor and global common factor matrix corresponding to the data type corresponding to the industrial chain big data can be used to directly process the industrial chain big data.

[0063] In an optional embodiment, the step of calculating the local factor matrix set corresponding to the local combination vector set is:

[0064] Using Tucker approximate decomposition, the n-type expansion matrix corresponding to each local combination vector in the local combination vector set is calculated, and each n-type expansion matrix is ​​divided into a number of feature blocks to form a feature block set;

[0065] Among them, the Local combination vector The corresponding n-type expansion matrix , divided into several feature blocks:

[0066]

[0067]

[0068]

[0069]

[0070] Where, represents the n-type expansion matrix The b-th feature block in ; represents the n-type expansion matrix rank; represents the rank of the matrix;

[0071] Perform truncated singular value decomposition on each feature block in the feature block set to obtain a local factor matrix corresponding to each feature block, forming a local factor matrix set;

[0072] Among them, for the feature block The calculation expression for truncated singular value decomposition is:

[0073]

[0074]

[0075]

[0076]

[0077] Where, Represents feature blocks The corresponding local factor matrix also represents the feature block The corresponding left orthogonal matrix; Represents feature blocks The corresponding diagonal matrix, Represents feature blocks The corresponding right orthogonal matrix, where The * in represents the conjugate transpose symbol.

[0078] As an example, and All are default values.

[0079] In this optional embodiment, the computational complexity mainly lies in obtaining a diagonal matrix by double diagonalization, which requires ( ) floating point operations, take , then the maximum computational cost is , so the computational cost of the computing node is , the computational cost of the server is ,in, , Represents the scale of data that the server needs to process; the communication complexity of this optional embodiment calculated in the server is similar to that of the computing node. The local data is uploaded to the server. After the server processes the local data information, the generated global factor matrix Send to computing units, the total communication cost is In terms of storage space, the intermediate storage space required for the calculation process of this optional embodiment is small, and the storage cost of the decomposition result is , can be ignored.

[0080] In an optional embodiment, the steps of iteratively updating the set of local factor matrices are:

[0081] Construct a local data optimization model, specifically:

[0082]

[0083]

[0084]

[0085] Where, represents the objective function; 、 and Respectively The corresponding 1st, 2nd and cth local factor matrices; represents the Frobenius norm; express The corresponding local combination vector; express The corresponding local core feature tensor;

[0086] Iteratively update the local data optimization model, in the objective function When the minimum is reached or the number of iterations reaches a preset value, the iteration ends and the updated local factor matrix set is obtained.

[0087] In this optional embodiment, the Euclidean distance between any two local core feature tensors is approximately equal to the Euclidean distance between the two industrial chain big data corresponding to any two local core feature tensors; therefore, by iteratively updating the local data optimization model constructed using the local core feature tensors, the updated local factor matrix set obtained can retain the global information of the industrial chain big data set to the maximum extent, thereby ensuring that the global common factor matrix set indirectly calculated using the updated local factor matrix set can retain the global information of the industrial chain big data set to the maximum extent.

[0088] In an optional embodiment, when calculating the intermediate matrix set using the updated local factor matrix set, the nth intermediate matrix in the intermediate matrix set is calculated. The calculation expression is:

[0089]

[0090] .

[0091] In this optional embodiment, an intermediate matrix set is obtained by removing the right orthogonal matrices corresponding to the feature blocks in the n-type expanded matrix set. The global common factor matrix set calculated using the intermediate matrix set is equal to the global information contained in the global common factor matrix set obtained directly using the n-type expanded matrix set. Because the private data of the industrial dataset can be reversed and restored using the n-type expanded matrix set, the n-type expanded matrix set cannot be directly uploaded to the server. Instead, the intermediate matrix set is uploaded to the server. Because the information of the right orthogonal matrices corresponding to the feature blocks is missing from the intermediate matrix set, even if the intermediate matrix set is stolen, the private data of the industrial dataset cannot be reversed and restored using the intermediate matrix set, thereby achieving the purpose of privacy protection of the large industrial chain dataset. The reason why the global common factor matrix set calculated using the intermediate matrix set is equal to the global common factor matrix set obtained directly using the n-type expanded matrix set is as follows:

[0092] Expand the matrix in n-type For example:

[0093]

[0094]

[0095] in, represents the n-type expansion matrix The corresponding intermediate matrix;

[0096] From the n-type expansion matrix From the definition of :

[0097]

[0098]

[0099] Where I represents the identity matrix;

[0100] Similarly, .

[0101] because , so the set of global common factor matrices calculated using this intermediate matrix set is equal to the set of global common factor matrices obtained directly using the n-type expansion matrix set.

[0102] In an optional embodiment, when the server calculates the global common factor matrix set using the intermediate matrix set, it calculates the nth global common factor matrix in the global common factor matrix set. The calculation expression is:

[0103]

[0104]

[0105] Where, represents the truncated singular value decomposition function.

[0106] In an optional embodiment, the steps of distributively calculating the updated intermediate matrix set and the local feature tensor set corresponding to the global common factor matrix set through each computing node are:

[0107] Each computing node uses the global common factor matrix received from the server to calculate the local factor matrix subset corresponding to the global common factor matrix;

[0108] Each computing node calculates the intermediate matrix and local feature core tensor corresponding to the local factor matrix subset obtained by itself. The intermediate matrices calculated by all computing nodes constitute the updated intermediate matrix set, and the local feature core tensors calculated by all computing nodes constitute the local feature core tensor set;

[0109] Among them, the local factor matrix subset corresponding to the nth global common factor matrix The calculation expression is:

[0110]

[0111] Where, 、 and Respectively The first, second and bth updated local factor matrices in ;

[0112] Local factor matrix subset The corresponding intermediate matrix The calculation expression is:

[0113]

[0114] Where, express The diagonal matrix of the corresponding feature block;

[0115] Local factor matrix subset The corresponding local feature core tensor The expression is:

[0116] .

[0117] In an optional embodiment, the iteration stopping condition includes the following three cases:

[0118] 1. The currently updated local combination vector set and the currently updated local factor matrix set make the objective function Reach minimum;

[0119] 2. The computing time of the server or computing node exceeds the preset time;

[0120] 3. The number of updates reaches the preset value.

[0121] In this embodiment, the amount of operation is mainly used to calculate the n-type expansion matrix, wherein the process of calculating the n-type expansion matrix includes bidiagonalization calculation and diagonalization calculation. The amount of operation is mainly used for bidiagonalization calculation, which requires ( ) floating-point operations, Indicates the total number of data for n-type expansion matrix calculation, take , then the maximum computational cost is , express N+1 matrix calculations, for the distributed industry chain big data feature extraction algorithm, take , Indicates the number of computing nodes, express The data are assigned to The computation cost of the computing node is , the server computation cost is , The rank of the n-type expanded matrix that represents the maximal factorization.

[0122] The communication complexity of this embodiment is similar to that of the previous example. The computing node uploads the local data of size to the server. After the server processes the intermediate matrix set, the computational complexity of the generated global factor matrix is , Represents the maximum expansion dimension of the n-type expansion, and distributes the global factor matrix in the global factor matrix set to computing units, the total communication cost is In terms of storage space, the intermediate storage space required for the calculation process of this method is small, and the storage cost of the decomposition result is , can be ignored;

[0123] The computational complexity of traditional decomposition methods, such as triangular decomposition, QR decomposition, and Jordan decomposition, for a matrix calculation is O(n 3 ), where n is the order of the decomposed matrix;

[0124] The calculation scale of a matrix calculation in this embodiment is 1 n+1 , the computational complexity is approximately O(Q n); the computational complexity is lower than the traditional decomposition method, which can effectively improve the calculation rate.

[0125] Example 2

[0126] This embodiment applies the real-time industry chain big data distributed processing method proposed in Example 1, and proposes the following two performance comparison examples:

[0127] (1) In the first performance comparison example, the ORL industrial monitoring system dataset is used for performance comparison. The ORL industrial monitoring system dataset contains 400 images of 40 different entities, with 10 images for each entity. Each image is processed into grayscale, and the resolution of each image is standardized to 64×64 pixels. Since the different entities contained in the ORL industrial monitoring system dataset are of the same entity type, the ORL industrial monitoring system dataset is a single-source dataset.

[0128] To simulate the information richness of industrial environment collection, 40 Gabor filters are used to extract the texture features of each image in the ORL industrial monitoring system dataset, and each image is represented as a 40×64×64 3D vector.

[0129] In the distributed processing method described in Example 1, the number of computing nodes is set to 1, which is the centralized processing method corresponding to the method described in Example 1;

[0130] The ORL industrial monitoring system dataset was processed using the centralized processing method corresponding to the distributed processing method described in Example 1 (referred to as the centralized method), the distributed processing method described in Example 1 (referred to as the distributed method), and the K-NearestNeighbor (KNN) method with a calculation parameter of N=3.

[0131] like Figure 2 FIG. 1 is a schematic diagram showing the performance comparison results using the ORL industrial monitoring system dataset in this embodiment; Figure 2 The accuracy and computational time of the processing results of the ORL industrial monitoring system dataset using the centralized method, distributed method and KNN method are demonstrated respectively.

[0132] Depend on Figure 2 It can be seen that the accuracy of the centralized method is higher than that of the KNN method, and the accuracy of the distributed method is higher than that of the centralized method. In addition, the computational time of the centralized method is lower than that of the KNN method, and the computational time of the distributed method is lower than that of the centralized method.

[0133] This shows that when processing the ORL industrial monitoring system dataset, a single-source dataset, the centralized method is better than the KNN method, and the distributed method is better than the centralized method using a single node. It also shows that the distributed method using several computing nodes can not only improve the calculation speed but also improve the accuracy of the processing results compared to the centralized method using a single node.

[0134] In addition, if Figure 3 , which is a schematic diagram showing the performance comparison results of the centralized method and the distributed method of this embodiment; it shows the relationship between the accuracy of the processing results and the rank of the feature blocks corresponding to the images in the ORL industrial monitoring system dataset when the centralized method and the distributed method with 2 computing nodes are used to process the ORL industrial monitoring system dataset, and the relationship between the accuracy of the processing results and the rank of the feature blocks corresponding to the images in the ORL industrial monitoring system dataset when the ORL industrial monitoring system dataset divided into a training set and a test set is processed respectively using the distributed method;

[0135] Depend on Figure 3 It can be seen that the accuracy of the centralized method is similar to that of the distributed method with 2 computing nodes, which shows that the processing results of the distributed method are accurate and stable and have a wide range of applications. Moreover, when the centralized method and the distributed method with 2 computing nodes are used to process the ORL industrial monitoring system dataset, the accuracy of the processing results increases with the increase of the rank of the feature blocks corresponding to the processed image. The processing results of the distributed method are accurate and the performance is stable and effective.

[0136] Depend on Figure 3It can be seen that the accuracy of training using the training set obtained by dividing the ORL industrial monitoring system dataset can reach up to 0.96, and the accuracy of testing using the test set obtained by dividing the ORL industrial monitoring system dataset can reach up to 0.90, indicating that the processing results of the distributed method are accurate, stable, and have strong application performance.

[0137] (2) In the second performance comparison example, the Fabrics dataset is used for performance comparison. The Fabrics dataset contains 320 images of 8 different entity types, each type has 40 RGB images, and each image is processed into a 400×400×3 vector format. Since the Fabrics dataset contains different entity types, the Fabrics dataset is multi-source data.

[0138] like Figure 4 FIG2 is a schematic diagram showing the performance comparison results using the Fabrics dataset in this embodiment; Figure 4 The following diagram shows the relationship between the accuracy of the Fabrics dataset and the rank R of the feature blocks corresponding to the images in the dataset, using a centralized method and a distributed method with 2 computing nodes. It also shows the relationship between the number of computing nodes and the accuracy when the Fabrics dataset is processed using a distributed method with rank R of 6, 12, or 18.

[0139] Depend on Figure 4 It can be seen that the accuracy of processing the Fabrics dataset using a centralized method and a distributed method with 2 computing nodes is similar, which shows that the distributed method can realize the simultaneous decomposition and processing of multi-source data, and both increase with the increase of rank R, which shows that the distributed method has good stability and a wide range of applications for processing multi-source data.

[0140] Depend on Figure 4 It can be seen that the change in the number of computing nodes of the distributed method has little effect on the accuracy, which further shows that the use of distributed methods to process multi-source data has good stability.

[0141] like Figure 5 FIG2 is a schematic diagram showing the performance comparison results of the KNN classifier and the SVM classifier in this embodiment; Figure 5A schematic diagram shows the verification results of the classification accuracy of feature blocks with different ranks using the KNN classifier (N=3) and the Support Vector Machine (SVM) classifier. It also shows the impact of the change in the number of computing nodes of the distributed method on the accuracy of the classification results when feature blocks with different ranks are input into the KNN classifier and the Support Vector Machine classifier respectively, when the ranks of the feature blocks are R1=R2=16 and R3=1.

[0142] Depend on Figure 5 It can be seen that when feature blocks with different ranks are input into the KNN classifier and the SVM classifier respectively, good accuracy can be obtained. This shows that the accuracy of the feature blocks obtained by using the distributed method is stable, the distributed method has good stability and a wide range of applicability.

[0143] Depend on Figure 5 It can be seen that the change in the number of computing nodes of the distributed method has little effect on the accuracy of the classification results, which shows that the distributed method adopted by the present invention can well extract the global features of data distributed in different computing nodes.

[0144] Example 3

[0145] This embodiment proposes a system for distributed processing of real-time industrial chain big data, which is used to implement a real-time industrial chain big data distributed processing method proposed in Example 1.

[0146] Figure 6 This is the overall framework diagram of the system for distributed processing of real-time industrial chain big data in this embodiment.

[0147] The system for distributed processing of real-time industry chain big data includes:

[0148] The acquisition module is used to collect big data of the industrial chain and form a large data set of the industrial chain;

[0149] A cascade splicing module is used to cascade and splice the industrial chain big data with the same data type into local combination vectors, and form a local combination vector set from the local combination vectors;

[0150] A decomposition module is used to calculate a set of local factor matrices corresponding to a set of local combination vectors using Tucker approximate decomposition and truncated singular value decomposition;

[0151] An iteration module is used to iteratively update the local factor matrix set until the iteration ends and obtain the updated local factor matrix set;

[0152] An intermediate matrix calculation module, used to calculate an intermediate matrix set using the updated local factor matrix set;

[0153] A server calculation module is used to upload the intermediate matrix set to the server, and the server uses the intermediate matrix set to calculate the global common factor matrix set;

[0154] A computing node computing module is used to randomly distribute the global common factor matrix in the global common factor matrix set to each computing node, and to distributely compute the updated intermediate matrix set and local feature tensor set corresponding to the global common factor matrix set through each computing node;

[0155] The update module is used to continuously update the intermediate matrix set and the local core feature tensor set in the computing node, and continuously update the global common factor matrix set in the server until the preset update stop condition is reached, and then stop updating the intermediate matrix set, the local core feature tensor set and the global common factor matrix set, and obtain the final intermediate matrix set, the final local core feature tensor set and the final global common factor matrix set, so as to realize the distributed processing of the industrial chain big data set.

[0156] It can be understood that the system of this embodiment is applied to the method of the above-mentioned embodiment 1, and the optional items in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described again here.

[0157] Example 4

[0158] This embodiment proposes a computer device including a memory and a processor, wherein the memory stores computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, the processor executes the steps of the real-time industrial chain big data distributed processing method proposed in Example 1.

[0159] It can be understood that the computer device of this embodiment is applied to the method of the above-mentioned embodiment 1, and the optional items in the above-mentioned embodiment 1 are also applicable to this embodiment, so they will not be described again here.

[0160] The same or similar reference numerals correspond to the same or similar components;

[0161] The terms used in the drawings to describe positional relationships are for illustrative purposes only and should not be construed as limiting this patent;

[0162] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A real-time industry chain big data distributed processing method, characterized in that: The method is implemented based on a server and several computing nodes and includes the following steps: Collecting industry chain big data to form an industry chain big data set; the industry chain big data includes: pictures, text and audio; The industrial chain big data is concentrated, and the industrial chain big data with the same data type is cascaded and spliced ​​into local combination vectors, and the local combination vectors form a local combination vector set; Using Tucker approximate decomposition and truncated singular value decomposition, calculate the set of local factor matrices corresponding to the set of local combination vectors; Iteratively update the local factor matrix set until the iteration ends, and obtain the updated local factor matrix set; Calculate the intermediate matrix set using the updated local factor matrix set; The intermediate matrix set is uploaded to the server, and the server uses the intermediate matrix set to calculate the global common factor matrix set; The global common factor matrix in the global common factor matrix set is randomly distributed to each computing node, and the updated intermediate matrix set and local feature tensor set corresponding to the global common factor matrix set are distributedly calculated by each computing node; The intermediate matrix set and the local core feature tensor set are continuously updated in the computing nodes, and the global common factor matrix set is continuously updated in the server until the preset update stop condition is reached. Then, the updating of the intermediate matrix set, the local core feature tensor set and the global common factor matrix set is stopped to obtain the final intermediate matrix set, the final local core feature tensor set and the final global common factor matrix set, thereby realizing distributed processing of the industrial chain big data set.

2. The real-time industry chain big data distributed processing method according to claim 1 is characterized in that: The steps for cascading and splicing industrial chain big data with the same data type are as follows: The industrial chain big data set is represented in the form of tensor; Big data of industry chains The expression is: Where, Indicates size N-dimensional real field tensor, 、 and Represents the size of the first dimension, second dimension, and Nth dimension of an N-dimensional tensor respectively; Indicates the total number of data types of the industrial chain big data in the industrial chain big data set; The industrial chain big data is concentrated, and the industrial chain big data with the same data type is cascaded and spliced ​​into a local combination vector, and the local combination vectors form a local combination vector set; Among them, Local combination vector The calculation expression is: Where, Represents a function used to concatenate K industry chain big data into an N+1 dimensional vector.

3. The real-time industry chain big data distributed processing method according to claim 2 is characterized in that: The steps for calculating the set of local factor matrices corresponding to the set of local combination vectors are: Using Tucker approximate decomposition, the n-type expansion matrix corresponding to each local combination vector in the local combination vector set is calculated, and each n-type expansion matrix is ​​divided into a number of feature blocks to form a feature block set; Among them, the Local combination vector The corresponding n-type expansion matrix , divided into several feature blocks: Where, represents the n-type expansion matrix The b-th feature block in ; represents the n-type expansion matrix rank; represents the rank of the matrix; Perform truncated singular value decomposition on each feature block in the feature block set to obtain a local factor matrix corresponding to each feature block, forming a local factor matrix set; Among them, for the feature block The calculation expression for truncated singular value decomposition is: Where, Represents feature blocks The corresponding local factor matrix also represents the feature block The corresponding left orthogonal matrix; Represents feature blocks The corresponding diagonal matrix, Represents feature blocks The corresponding right orthogonal matrix, where The * in represents the conjugate transpose symbol.

4. The real-time industry chain big data distributed processing method according to claim 3 is characterized in that: The steps for iteratively updating the set of local factor matrices are: Construct a local data optimization model, specifically: Where, represents the objective function; 、 and Respectively The corresponding 1st, 2nd and cth local factor matrices; represents the Frobenius norm; express The corresponding local combination vector; express The corresponding local core feature tensor; Iteratively update the local data optimization model, in the objective function When the minimum is reached or the number of iterations reaches a preset value, the iteration ends and the updated local factor matrix set is obtained.

5. The real-time industry chain big data distributed processing method according to claim 4 is characterized in that: When calculating the intermediate matrix set using the updated local factor matrix set, calculate the nth intermediate matrix in the intermediate matrix set The calculation expression is: 。 6. The real-time industry chain big data distributed processing method according to claim 5 is characterized in that: When the server calculates the global common factor matrix set using the intermediate matrix set, it calculates the nth global common factor matrix in the global common factor matrix set. The calculation expression is: Where, represents the truncated singular value decomposition function.

7. The real-time industry chain big data distributed processing method according to claim 6 is characterized in that: The steps of distributing the calculation of the updated intermediate matrix set and the local feature tensor set corresponding to the global common factor matrix set through each computing node are as follows: Each computing node uses the global common factor matrix received from the server to calculate the local factor matrix subset corresponding to the global common factor matrix; Each computing node calculates the intermediate matrix and local feature core tensor corresponding to the local factor matrix subset obtained by itself. The intermediate matrices calculated by all computing nodes constitute the updated intermediate matrix set, and the local feature core tensors calculated by all computing nodes constitute the local feature core tensor set; Among them, the local factor matrix subset corresponding to the nth global common factor matrix The calculation expression is: Where, 、 and Respectively The first, second and bth updated local factor matrices in ; Local factor matrix subset The corresponding intermediate matrix The calculation expression is: Where, express The diagonal matrix of the corresponding feature block; Local factor matrix subset The corresponding local feature core tensor The expression is: 。 8. The real-time industry chain big data distributed processing method according to any one of claims 1 to 7, characterized in that: The iteration stopping conditions include the following three cases:

1. The currently updated local combination vector set and the currently updated local factor matrix set make the objective function Reach minimum; 2. The computing time of the server or computing node exceeds the preset time; 3. The number of updates reaches the preset value.

9. A system for distributed processing of real-time industry chain big data, used to implement the real-time industry chain big data distributed processing method according to any one of claims 1 to 8, characterized in that: include: The acquisition module is used to collect industrial chain big data to form an industrial chain big data set; the industrial chain big data includes: pictures, text and audio; A cascade splicing module is used to cascade and splice the industrial chain big data with the same data type into local combination vectors, and form a local combination vector set from the local combination vectors; A decomposition module is used to calculate a set of local factor matrices corresponding to a set of local combination vectors using Tucker approximate decomposition and truncated singular value decomposition; An iteration module is used to iteratively update the local factor matrix set until the iteration ends and obtain the updated local factor matrix set; An intermediate matrix calculation module, used to calculate an intermediate matrix set using the updated local factor matrix set; A server calculation module is used to upload the intermediate matrix set to the server, and the server uses the intermediate matrix set to calculate the global common factor matrix set; A computing node computing module is used to randomly distribute the global common factor matrix in the global common factor matrix set to each computing node, and to distributely compute the updated intermediate matrix set and local feature tensor set corresponding to the global common factor matrix set through each computing node; The update module is used to continuously update the intermediate matrix set and the local core feature tensor set in the computing node, and continuously update the global common factor matrix set in the server until the preset update stop condition is reached, and then stop updating the intermediate matrix set, the local core feature tensor set and the global common factor matrix set, and obtain the final intermediate matrix set, the final local core feature tensor set and the final global common factor matrix set, so as to realize the distributed processing of the industrial chain big data set.

10. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, wherein: When the computer-readable instructions are executed by the processor, the processor executes the steps of the real-time industry chain big data distributed processing method as described in any one of claims 1 to 8.

Images