A biological organic fertilizer production process data extraction system and processing method

Abstract

The present invention relates to the technical field of data processing, and in particular to a data extraction system and processing method for a biological organic fertilizer production process, comprising: dividing the data time series of each parameter into several categories, obtaining the similarity sequence and contribution relationship between each data and each comparison sequence in the corresponding category, obtaining the comprehensive correlation contribution degree, positive contribution sequence set and negative contribution sequence set of each data, obtaining a difference matrix and a ratio matrix according to an initial reconstruction matrix and a reconstruction matrix of each eigenvector to be judged; and then obtaining the increase degree of each data, judging and obtaining all valid data, calculating the correlation retention degree of all eigenvectors to be judged, obtaining all retained eigenvalues ​​and retained eigenvectors, reconstructing and obtaining a low-rank matrix and a sparse matrix, and compressing and storing the low-rank matrix and the sparse matrix. The present invention improves the compression effect of various data generated in the production process of biological organic fertilizer and reduces storage costs.

Description

Technical Field

[0001] The invention relates to the technical field of data processing, and in particular to a biological organic fertilizer production process data extraction system and a processing method. Background Art

[0002] By analyzing the various data generated in the production process of bio-organic fertilizer, the quality and efficiency of bio-organic fertilizer production can be monitored, and the process and equipment of the production process can be improved. Therefore, it is necessary to store the various data generated in the production process of bio-organic fertilizer; due to the large amount of data, the data needs to be compressed and stored.

[0003] There is a strong correlation between the various data in the production process. The data matrix composed of various data can be decomposed into low-rank matrices and sparse matrices with strong correlation through the robust principal component decomposition algorithm. By storing the low-rank matrices and sparse matrices, the various data can be compressed and stored. The stronger the correlation of the low-rank matrix, the better the compression efficiency of the low-rank matrix.

[0004] The conventional robust principal component decomposition algorithm obtains eigenvectors by performing singular value decomposition on the data matrix, and reconstructs the low-rank matrix through the eigenvectors. The eigenvectors selected when reconstructing the low-rank matrix are based on the total amount of information contained in the eigenvectors. However, the more total amount of information contained does not mean the greater the correlation of the eigenvectors. As a result, the low-rank matrix reconstructed based on the eigenvectors selected based on the total amount of information contained has a weaker correlation, which makes the compression efficiency of low-rank matrices and sparse matrices poor. Summary of the invention

[0005] The invention provides a method and system for processing data in a biological organic fertilizer production process to solve the existing problems.

[0006] The present invention provides a method for processing data in a bio-organic fertilizer production process, the method comprising the following steps:

[0007] Collect the time series data of each parameter in the production process of bio-organic fertilizer;

[0008] According to the similarity of all data time series, all data time series are classified to obtain several categories;

[0009] Record any data time series as the target sequence, record all other data time series except the target sequence in the category of the target sequence as comparison sequences, record any comparison sequence as the target comparison sequence, and record any data in the target sequence as the target data;

[0010] According to the similarity sequence between the target data and the target comparison sequence, the contribution relationship between the target data and the target comparison sequence is obtained; according to the contribution relationship between the target data and all comparison sequences, the comprehensive correlation contribution degree, positive contribution sequence set and negative contribution sequence set of the target data are obtained;

[0011] Decomposing the original data matrix composed of all data time series to obtain all retained eigenvalues ​​and retained eigenvectors and all eigenvalues ​​and eigenvectors to be determined;

[0012] An initial reconstruction matrix is ​​obtained by reconstructing all the retained eigenvalues ​​and retained eigenvectors;

[0013] Record any eigenvalue to be judged as the target eigenvalue to be judged, record the corresponding eigenvector to be judged as the target eigenvector to be judged, reconstruct the target eigenvector to be judged based on all retained eigenvalues ​​and retained eigenvectors and the target eigenvalue to be judged and the target eigenvector to be judged to obtain a reconstruction matrix of the target eigenvector to be judged; obtain a difference matrix and a ratio matrix of the target eigenvector to be judged based on the initial reconstruction matrix and the reconstruction matrix;

[0014] Obtain the reference point of each column according to the difference matrix, and obtain the increase degree of the target data according to the ratio matrix, the positive contribution sequence set and the negative contribution sequence set; record all data with an increase degree greater than 0 as valid data;

[0015] Obtain all retained eigenvectors and corresponding retained eigenvalues ​​according to the increase degree of all valid data and the comprehensive correlation contribution of the reference points in the column where the valid data is located;

[0016] A low-rank matrix and a sparse matrix are reconstructed according to all the retained eigenvalues ​​and retained eigenvectors, and the low-rank matrix and the sparse matrix are compressed and stored.

[0017] Furthermore, the method of classifying all the data time series sequences according to the similarity of all the data time series sequences to obtain several categories includes the following specific steps:

[0018] The data time series of each parameter is taken as a node of the graph structure; the cosine similarity of the data time series corresponding to two different nodes is taken as the edge value between the two nodes; the embedding vector of each node in the graph structure is obtained by the GrapHSAGE method;

[0019] Calculate the cosine similarity of the embedding vectors of every two nodes, divide the data time series sequences corresponding to all nodes whose cosine similarity of the embedding vectors is greater than the first threshold into one category, and the cosine similarity of the embedding vectors of any two nodes in the same category is greater than the first threshold, and divide the data time series sequences corresponding to all nodes into several categories, each category is a set of data time series sequences corresponding to several nodes.

[0020] Furthermore, the step of obtaining the contribution relationship between the target data and the target comparison sequence includes the following specific steps:

[0021] Each integer within a preset range is used as a window length. For any window length, a subsequence with a length equal to the window length is obtained with the target data as the center point, the mean of all data in the subsequence is obtained, and the target data in the target sequence is replaced with the mean; the cosine similarity between the replaced target sequence and the target comparison sequence is recorded as the similarity between the target data of the target sequence and the target comparison sequence under the window length; a sequence composed of the similarities between the target data of the target sequence and the target comparison sequence under all window lengths is recorded as the similarity sequence between the target data of the target sequence and the target comparison sequence;

[0022] The window length is used as the horizontal coordinate, and the similarity between the target data of the target sequence and the target comparison sequence under each window length is used as the vertical coordinate to construct a two-dimensional space. The similarity between the target data of the target sequence and the target comparison sequence under all window lengths is converted into a number of spatial points in the two-dimensional space. Principal component analysis is performed on all spatial points to obtain the principal component directions of the target data of the target sequence and the target comparison sequence; the angle between the principal component direction and the positive direction of the y-axis is determined. If the angle is within the range of [0°, 90°], the target data of the target sequence has a positive contribution relationship with the target comparison sequence; otherwise, the target data of the target sequence has a negative contribution relationship with the target comparison sequence.

[0023] Furthermore, the step of obtaining the comprehensive correlation contribution degree, the positive contribution sequence set and the negative contribution sequence set of the target data includes the following specific steps:

[0024] The cotangent value of the angle between the principal component direction and the positive direction of the y-axis is recorded as the correlation contribution between the target data of the target sequence and the target comparison sequence; the average of the correlation contributions between the target data of the target sequence and all comparison sequences is recorded as the comprehensive correlation contribution of the target data of the target sequence;

[0025] The contribution relationship between the target data of the target sequence and all comparison sequences is obtained, and the set consisting of all comparison sequences that have a positive contribution relationship with the target data of the target sequence is recorded as the positive contribution sequence set of the target data of the target sequence, and the set consisting of all comparison sequences that have a negative contribution relationship with the target data of the target sequence is recorded as the negative contribution sequence set of the target data of the target sequence.

[0026] Furthermore, the obtaining of all retained eigenvalues ​​and retained eigenvectors and all to-be-determined eigenvalues ​​and to-be-determined eigenvectors comprises the following specific steps:

[0027] The data time series sequence of each parameter is used as each row of the original data matrix to obtain the original data matrix; the original data matrix is ​​decomposed by the singular value decomposition algorithm to obtain several eigenvalues ​​and eigenvectors, the largest first K eigenvalues ​​are recorded as retained eigenvalues, and the eigenvectors corresponding to the retained eigenvalues ​​are recorded as retained eigenvectors, which contain most of the information in the original matrix; the eigenvalues ​​except the largest first K eigenvalues ​​are recorded as eigenvalues ​​to be judged, and the eigenvectors corresponding to the eigenvalues ​​to be judged are recorded as eigenvectors to be judged, and K is a preset number.

[0028] Furthermore, the step of obtaining the difference matrix and the ratio matrix of the target feature vector to be determined includes the following specific steps:

[0029] The difference matrix is ​​obtained by subtracting the elements at the same position in the initial reconstruction matrix and the reconstruction matrix; and the ratio matrix is ​​obtained according to the ratio of the elements at each position in the difference matrix to the elements at the same position in the initial reconstruction matrix.

[0030] Furthermore, the step of obtaining the increase degree of target data includes the following specific steps:

[0031] Obtaining a positive ratio sequence of the target data of the target sequence according to a positive contribution sequence set and a ratio matrix of the target data of the target sequence;

[0032] Obtaining a negative ratio sequence of the target data of the target sequence according to a negative contribution sequence set and a ratio matrix of the target data of the target sequence;

[0033] The difference between the sum of all ratios in the positive ratio sequence of the target data of the target sequence and the sum of all ratios in the negative ratio sequence of the target data of the target sequence is recorded as the increase degree of the target data of the target sequence.

[0034] Furthermore, the step of obtaining all retained eigenvectors and corresponding retained eigenvalues ​​includes the following specific steps:

[0035] The calculation formula for the correlation retention degree of the target feature vector to be judged is:

[0036]

[0037] In the formula, p represents the degree of relevance preservation of the target feature vector to be judged, n represents the number of all valid data, and g i represents the comprehensive correlation contribution of the benchmark point in the column where the i-th valid data is located, z i Indicates the increase degree of the i-th valid data;

[0038] The correlation retention degrees of all feature vectors to be determined are calculated, and feature vectors to be determined whose correlation retention degrees are greater than a second threshold are recorded as retained feature vectors, and the corresponding feature values ​​to be determined are recorded as retained feature values.

[0039] Furthermore, the step of obtaining a positive ratio sequence of the target data of the target sequence according to the positive contribution sequence set and the ratio matrix of the target data of the target sequence comprises the following specific steps:

[0040] Record any positive contribution sequence in the set of positive contribution sequences of the target data of the target sequence as the target positive contribution sequence, obtain the data in the target positive contribution sequence at the same position as the target data, and record it as the positive contribution data of the target data of the target sequence; obtain all the positive contribution data of the target data of the target sequence, and record the sequence composed of the ratios corresponding to all the positive contribution data in the ratio sequence as the positive ratio sequence of the target data of the target sequence.

[0041] The present invention also provides a bio-organic fertilizer production process data extraction system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the steps of the bio-organic fertilizer production process data processing method when executing the computer program.

[0042] The beneficial effects of the technical solution of the present invention are as follows: the present invention divides the data time series sequence of each parameter into several categories according to the similarity of the embedded vectors of every two data time series sequences. Compared with the traditional direct classification by cosine similarity, the present invention combines the connection with other data time series sequences, and the obtained classification result is more accurate and more in line with reality, and will not cause a large deviation in the final result due to the small anomaly of a single data time series sequence; by retaining the eigenvectors with the largest eigenvalues, the reconstructed low-rank matrix contains most of the information of the data matrix, ensuring the sparsity of the sparse matrix, while ensuring the compression efficiency of the sparse matrix. The positive ratio sequence and the negative ratio sequence obtain the degree of increase of the data, and the data with a larger increase degree is retained as the retained data, and the data with a larger positive contribution is retained. The comprehensive correlation contribution of the benchmark point of the column where the valid data is located is used as the weight, and the weighted summation is performed on the increase degree of all valid data corresponding to the eigenvector to be judged to obtain the correlation retention degree of the eigenvector to be judged, and the eigenvector carrying more correlation information is retained. The low-rank matrix is ​​reconstructed according to the retained eigenvectors and retained eigenvalues ​​to improve the compression efficiency of the low-rank matrix, thereby improving the compression effect of various data generated in the production process of biological organic fertilizer and reducing storage costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 The present invention is a flow chart of the steps of a method for processing data in a biological organic fertilizer production process. DETAILED DESCRIPTION

[0045] In order to further elaborate the technical means and the effect that the present invention adopts for reaching the predetermined invention purpose, below in conjunction with the accompanying drawings and preferred embodiments, to a kind of biological organic fertilizer production process data processing method proposed according to the present invention, its specific implementation mode, structure, feature and effect thereof, are described in detail as follows. In the following description, different " one embodiment " or " another embodiment " refer to not necessarily the same embodiment. In addition, the specific features, structure or characteristics in one or more embodiments can be combined by any suitable form.

[0046] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0047] The specific scheme of the data processing method for the biological organic fertilizer production process provided by the present invention is described in detail below with reference to the accompanying drawings.

[0048] When compressing and storing the various data generated in the production process of bio-organic fertilizer, taking into account the strong correlation between the various data in the production process, the data matrix composed of the various data can be decomposed into low-rank matrices and sparse matrices with strong correlation through the robust principal component decomposition algorithm. By storing the low-rank matrices and sparse matrices, the compressed storage of the various data can be achieved.

[0049] The stronger the correlation of the decomposed low-rank matrix is, the fewer row vectors the low-rank matrix can be represented by linear combination, so the compression efficiency of the low-rank matrix is ​​better.

[0050] The conventional robust principal component decomposition algorithm obtains all eigenvectors by performing singular value decomposition on the data matrix, and reconstructs the low-rank matrix by selecting some eigenvectors from all the eigenvectors. The eigenvectors selected when reconstructing the low-rank matrix are based on the total amount of information contained in the eigenvectors. The total amount of information includes data distribution, correlation between data, data size and other information. Therefore, the more total amount of information contained does not mean that the correlation of the eigenvectors is greater. As a result, the correlation of the low-rank matrix reconstructed according to the eigenvectors selected based on the total amount of information contained is weaker, which makes the compression efficiency of the low-rank matrix poor.

[0051] In order to improve the compression efficiency of various data generated in the production process of bio-organic fertilizer, for a data matrix composed of various data, this embodiment retains 10 eigenvectors containing the largest total amount of information as retained eigenvectors, so that the reconstructed low-rank matrix contains most of the information of the data matrix, and the sparsity of the sparse matrix is ​​ensured. While ensuring the compression efficiency of the sparse matrix, according to the correlation between the remaining eigenvectors and the retained eigenvectors, the eigenvectors with strong correlation with the retained eigenvectors are selected, and the basic eigenvectors and the eigenvectors with strong correlation with the retained eigenvectors are combined to jointly construct a low-rank matrix to improve the compression efficiency of the low-rank matrix.

[0052] See also Figure 1 , which shows a flow chart of a method for processing data in a biological organic fertilizer production process provided by an embodiment of the present invention, the method comprising the following steps:

[0053] S001. Collect the time series data of each parameter in the production process of bio-organic fertilizer.

[0054] It should be noted that in the production process of bio-organic fertilizer, a large amount of data is usually generated, including the source and composition of raw materials, details of the processing process, fermentation and composting time, temperature, humidity and pH value, etc. These data are of great value to manufacturers and can be used to monitor the quality and efficiency of the production process, as well as to improve the process and equipment of the production line.

[0055] Specifically, data of each parameter at each moment in the production process of bio-organic fertilizer is collected, and the data of each parameter at all moments are combined into a data time series sequence of each parameter; the parameters that need to collect data in the production process of bio-organic fertilizer include: the source and composition of raw materials, the fermentation time, temperature, humidity, pH value, etc. during the processing.

[0056] In this embodiment, the sampling interval is 15 minutes and the total sampling time is 3 days.

[0057] S002. Classify all data time series according to their similarities to obtain several categories.

[0058] Specifically, a graph structure is constructed according to the data time series sequences of all kinds of parameters, specifically: the data time series sequence of each parameter is used as a node of the graph structure; the cosine similarity of the data time series sequences corresponding to two different nodes is used as the edge value between the two nodes; the embedding vector of each node in the graph structure is obtained by the GrapHSAGE method; cosine similarity and GrapHSAGE method are prior arts and will not be described in detail here.

[0059] It should be noted that the embedding vector of each node represents the feature information of the node and the feature information of its neighboring nodes (nodes with edge weights with the node). Since in the graph structure constructed in this embodiment, each node has edge values ​​with all other nodes, the neighboring nodes of each node are actually all other nodes except the node, that is, the embedding vector of each node represents the feature information of the node and the feature information of all other nodes. Therefore, the similarity of the feature information of every two nodes can be used to characterize the similarity of the feature information of every two nodes, and then all nodes can be classified.

[0060] Specifically, the cosine similarity of the embedding vectors of every two nodes is calculated, and the data time series sequences corresponding to all nodes whose cosine similarity of the embedding vectors is greater than the first threshold are divided into one category, and the cosine similarity of the embedding vectors of any two nodes in the same category is greater than the first threshold, and the data time series sequences corresponding to all nodes are divided into several categories, each category is a set of data time series sequences corresponding to several nodes; the similarity of feature information of data time series sequences in the same category is greater, and the similarity of feature information of data time series sequences in different categories is smaller.

[0061] In this embodiment, the first threshold is 0.7. In other embodiments, implementers may set the first threshold according to actual implementation conditions and experience.

[0062] It should be noted that this embodiment divides the data time series sequences of each parameter into several categories according to the similarity of the embedded vectors of every two data time series sequences. Compared with the traditional classification directly by cosine similarity, the classification result obtained by combining the connection with other data time series sequences is more accurate and more in line with reality, and will not cause large deviations in the final result due to minor anomalies in a single data time series sequence.

[0063] S003. Obtain the similarity sequence and contribution relationship between each data in each data time series sequence and each comparison sequence, and then obtain the comprehensive correlation contribution of each data in each data time series sequence as well as the positive contribution sequence set and the negative contribution sequence set.

[0064] It should be noted that different eigenvectors of the original data matrix can represent different data with different degrees of correlation (for the time series of two types of data, different time points contribute to the correlation to different degrees). The degree to which each eigenvector retains the correlation of each data can be obtained by calculation, and the eigenvectors are screened according to the degree of retention.

[0065] After calculating the correlation between any two data, the correlation contribution of the data at each position is calculated. Considering that after smoothing the data, the local detail information is smoothed, and the correlation between the two data tends to increase, the correlation of each data in each data pair (a data pair is any two data in a certain category) is calculated through the smoothing information of each data pair in each category, and then the correlation contribution of each data in each data pair is obtained by combining the correlation of each data in different data pairs. The more information about the data with a larger correlation contribution, the more the feature vector should be retained.

[0066] 1. Obtain the similarity sequence between each data in each data time series and each comparison sequence.

[0067] Specifically, any data time series sequence is recorded as a target sequence, all other data time series sequences except the target sequence in the category where the target sequence belongs are recorded as comparison sequences, any comparison sequence is recorded as a target comparison sequence, and any data in the target sequence is recorded as target data.

[0068] Furthermore, a similarity sequence between target data of a target sequence and a target comparison sequence is obtained, including: taking each integer within a preset range as a window length, for any window length, taking the target data as a center point to obtain a subsequence whose length is equal to the window length, obtaining a mean of all data in the subsequence, and replacing the target data in the target sequence with the mean; recording the cosine similarity between the replaced target sequence and the target comparison sequence as the similarity between the target data of the target sequence and the target comparison sequence under the window length; and recording a sequence composed of the similarities between the target data of the target sequence and the target comparison sequence under all window lengths in order as the similarity sequence between the target data of the target sequence and the target comparison sequence.

[0069] 2. Obtain the comprehensive correlation contribution of each data in each data time series sequence, and according to the contribution relationship between each data in each data time series sequence and each comparison sequence, obtain the positive contribution sequence set and the negative contribution sequence set between each data in each data time series sequence and each comparison sequence.

[0070] It should be noted that if in the process of increasing the window length, the cosine similarity value of the target data of the target sequence corresponding to the smooth curve and the target comparison sequence shows a decreasing trend, which means that the similarity contribution of the target data of the original target sequence is relatively large, and the greater the decreasing trend, the greater the contribution; if in the process of increasing the window length, the cosine similarity value of the target data of the target sequence corresponding to the smooth curve and the target comparison sequence shows an increasing trend, which means that the similarity contribution of the target data of the original target sequence is a negative contribution, then the more a certain feature vector can represent the information of the point, the smaller the corresponding feature vector is retained, and the more priority is given to selecting other feature vectors to construct a low-rank matrix.

[0071] Specifically, obtaining the contribution relationship between the target data of the target sequence and the target comparison sequence includes: taking the window length as the horizontal coordinate, taking the similarity between the target data of the target sequence and the target comparison sequence under each window length as the vertical coordinate, constructing a two-dimensional space, converting the similarity between the target data of the target sequence and the target comparison sequence under all window lengths into a number of spatial points in the two-dimensional space, performing principal component analysis on all spatial points, and obtaining the principal component directions of the target data of the target sequence and the target comparison sequence; judging the angle between the principal component direction and the positive direction of the y-axis, if the angle is within the range of [0°, 90°], the target data of the target sequence and the target comparison sequence have a positive contribution relationship, otherwise, the target data of the target sequence and the target comparison sequence have a negative contribution relationship.

[0072] It should be noted that, according to the principal component analysis algorithm, a total of multiple principal component directions are obtained, each principal component direction vector corresponds to an eigenvalue, and in this embodiment, only the principal component direction with the largest eigenvalue is retained.

[0073] Specifically, the comprehensive correlation contribution of the target data of the target sequence is obtained, including: recording the cotangent value of the angle between the principal component direction and the positive direction of the y-axis as the correlation contribution of the target data of the target sequence and the target comparison sequence; recording the average of the correlation contributions of the target data of the target sequence and all comparison sequences as the comprehensive correlation contribution of the target data of the target sequence.

[0074] Furthermore, a positive contribution sequence set and a negative contribution sequence set of the target data of the target sequence and the target comparison sequence are obtained, including: obtaining the contribution relationship between the target data of the target sequence and all comparison sequences, recording the set consisting of all comparison sequences that have a positive contribution relationship with the target data of the target sequence as the positive contribution sequence set of the target data of the target sequence, and recording the set consisting of all comparison sequences that have a negative contribution relationship with the target data of the target sequence as the negative contribution sequence set of the target data of the target sequence.

[0075] S004. Obtain all retained eigenvalues ​​and retained eigenvectors and all eigenvalues ​​and eigenvectors to be judged; obtain the initial reconstruction matrix, obtain the reconstruction matrix, difference matrix and ratio matrix corresponding to each eigenvector to be judged; obtain the positive ratio sequence and negative ratio sequence of each data, and then obtain the degree of increase of each data, and determine that all valid data are obtained.

[0076] It should be noted that if a certain eigenvector has a greater number of species data corresponding to positive contribution values ​​and a smaller number of species data corresponding to negative contribution values ​​before and after the eigenvector is added, the eigenvector can represent more correlations, that is, the stronger its carrying capacity for correlations is, and the more it needs to be retained.

[0077] 1. Obtain all retained eigenvalues ​​and retained eigenvectors as well as all eigenvalues ​​and eigenvectors to be determined.

[0078] Specifically, the data time series sequence of each parameter is used as each row of the original data matrix to obtain the original data matrix; the original data matrix is ​​decomposed by the singular value decomposition algorithm to obtain several eigenvalues ​​and eigenvectors, the largest first K eigenvalues ​​are recorded as retained eigenvalues, and the eigenvectors corresponding to the retained eigenvalues ​​are recorded as retained eigenvectors, which contain most of the information in the original matrix; the eigenvalues ​​except the largest first K eigenvalues ​​are recorded as eigenvalues ​​to be judged, and the eigenvectors corresponding to the eigenvalues ​​to be judged are recorded as eigenvectors to be judged, and K represents a preset number.

[0079] In this embodiment, the preset number K is 10. In other embodiments, implementers can set the preset number based on actual implementation conditions and experience.

[0080] 2. Obtain an initial reconstruction matrix, obtain a reconstruction matrix corresponding to each eigenvector to be determined, and then obtain a difference matrix and a ratio matrix corresponding to each eigenvector to be determined.

[0081] Specifically, the original data matrix is ​​reconstructed by retaining eigenvalues ​​and retaining eigenvectors to obtain an initial reconstructed matrix.

[0082] Further, any eigenvalue to be judged is recorded as the target eigenvalue to be judged, and the eigenvector to be judged corresponding to the target eigenvalue to be judged is recorded as the target eigenvector to be judged, and the difference matrix and ratio matrix corresponding to the target eigenvector to be judged are obtained, including: reconstructing the original data matrix of the matrix according to all retained eigenvalues ​​and retained eigenvectors and the target eigenvalues ​​to be judged and the eigenvector to be judged to obtain a reconstructed matrix; subtracting the elements at the same position in the initial reconstruction matrix and the reconstruction matrix to obtain a difference matrix; and obtaining a ratio matrix according to the ratio of the element at each position in the difference matrix to the element at the same position in the initial reconstruction matrix.

[0083] Furthermore, the data corresponding to the maximum difference value of each column in the difference matrix is ​​used as the reference point of the column.

[0084] 3. Obtain the positive ratio sequence and negative ratio sequence of each data in each data time series, and then obtain the increase degree of each data in each data time series, and determine whether all valid data are obtained.

[0085] Specifically, any data time series sequence is recorded as a target sequence, all other data time series sequences except the target sequence in the category where the target sequence belongs are recorded as comparison sequences, any comparison sequence is recorded as a target comparison sequence, and any data in the target sequence is recorded as target data.

[0086] Furthermore, based on the positive contribution sequence set and the ratio matrix of the target data of the target sequence, the positive ratio sequence of the target data of the target sequence is obtained, and then the degree of increase of the target data of the target sequence is obtained, including: recording any positive contribution sequence in the positive contribution sequence set of the target data of the target sequence as the target positive contribution sequence, obtaining the data in the target positive contribution sequence at the same position as that in the target data, and recording it as the positive contribution data of the target data of the target sequence; obtaining all the positive contribution data of the target data of the target sequence, and recording the sequence composed of the ratios corresponding to all the positive contribution data in the ratio sequence as the positive ratio sequence of the target data of the target sequence; and so on, based on the negative contribution sequence set and the ratio matrix of the target data of the target sequence, the negative ratio sequence of the target data of the target sequence is obtained; and the difference between the sum of all ratios in the positive ratio sequence of the target data of the target sequence and the sum of all ratios in the negative ratio sequence of the target data of the target sequence is recorded as the degree of increase of the target data of the target sequence.

[0087] Among all the data in all the data time series, the data with an increase degree greater than 0 is recorded as valid data; and all valid data are obtained.

[0088] S005. Calculate the correlation retention degree of all feature vectors to be judged, and obtain all retained eigenvalues ​​and retained eigenvectors.

[0089] It should be noted that different constraints on the decomposition process will lead to different compression effects of the low-rank matrix. Inappropriate constraints may lead to low compression efficiency. Therefore, it is necessary to enhance the correlation of the data in the low-rank matrix to achieve the purpose of increasing the compression effect of the low-rank matrix; the low-rank matrix is ​​reconstructed by retaining different eigenvectors to be judged, and then the degree of correlation retention of different eigenvectors to be judged is calculated, and then all the retained eigenvalues ​​and retained eigenvectors that need to be retained are obtained. Reconstructing by retaining the eigenvalues ​​and retaining the eigenvectors can maximize the correlation of the reconstructed low-rank matrix, and thus it is very important to improve the compression effect.

[0090] Specifically, any eigenvalue to be judged is recorded as the target eigenvalue to be judged, the eigenvector to be judged corresponding to the target eigenvalue to be judged is recorded as the target eigenvector to be judged, and the correlation retention degree of the target eigenvector to be judged is calculated. The calculation formula is:

[0091]

[0092] In the formula, p represents the degree of relevance preservation of the target feature vector to be judged, n represents the number of all valid data, and g i represents the comprehensive correlation contribution of the benchmark point in the column where the i-th valid data is located, z i represents the increase degree of the i-th valid data; the comprehensive correlation contribution of the reference point in the column where each valid data is located is used as the weight of the growth degree, and the correlation retention degree of the target feature vector to be judged is obtained by weighted summing the increase degree of all valid data corresponding to the target feature vector to be judged. The comprehensive correlation contribution g of the reference point in the column where the i-th valid data is located is i The larger the value is, the greater the degree of correlation preservation of the eigenvector corresponding to the i-th valid data; the degree of increase of the i-th valid data z i The larger it is, the greater the increase in the positive contribution of the i-th valid data relative to the negative contribution, and the greater the correlation retention of the corresponding eigenvector.

[0093] Further, the correlation retention degrees of all feature vectors to be judged are calculated, and feature vectors to be judged whose correlation retention degrees are greater than a second threshold are recorded as retained feature vectors, and the corresponding feature values ​​to be judged are recorded as retained feature values.

[0094] In this embodiment, the second threshold is 0.3. In other embodiments, implementers may set the second threshold according to actual implementation conditions and experience.

[0095] S006. Obtain a low-rank matrix and a sparse matrix based on all retained eigenvalues ​​and retained eigenvectors, and compress and store the low-rank matrix and the sparse matrix.

[0096] According to all retained eigenvalues ​​and retained eigenvectors, the matrix is ​​reconstructed into the original data matrix, and the reconstructed matrix is ​​recorded as a low-rank matrix, and the corresponding sparse matrix is ​​obtained at the same time.

[0097] The low-rank matrix and sparse matrix are compressed and stored separately.

[0098] The present invention also provides a bio-organic fertilizer production process data extraction system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any one of the steps of the bio-organic fertilizer production process data processing method when executing the computer program.

[0099] The present invention divides the data time series sequence of each parameter into several categories according to the similarity of the embedded vectors of every two data time series sequences. Compared with the traditional classification directly performed by cosine similarity, the present invention combines the connection with other data time series sequences, and the obtained classification result is more accurate and more in line with reality, and will not cause a large deviation of the final result due to the small abnormality of a single data time series sequence; by retaining some eigenvectors with the largest eigenvalues, the reconstructed low-rank matrix contains most of the information of the data matrix, and the sparsity of the sparse matrix is ​​guaranteed. While ensuring the compression efficiency of the sparse matrix, the increase degree of the data is obtained through the positive ratio sequence and the negative ratio sequence of the data, and the data with a larger increase degree is retained as the retained data, and the data with a larger positive contribution is retained. The comprehensive correlation contribution of the reference point of the column where the valid data is located is used as a weight, and the increase degree of all valid data corresponding to the eigenvector to be judged is weighted and summed to obtain the correlation retention degree of the eigenvector to be judged, and the eigenvector carrying more correlation information is retained. The low-rank matrix is ​​reconstructed according to the retained eigenvector and the retained eigenvalue, and the compression efficiency of the low-rank matrix is ​​improved, thereby improving the compression effect of various data generated in the production process of biological organic fertilizer, and reducing the storage cost.

[0100] It should be noted that the sequence of the above embodiments of the present invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0101] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A method for processing data in a bio-organic fertilizer production process, characterized in that: The method comprises the following steps: Collect the time series data of each parameter in the production process of bio-organic fertilizer; According to the similarity of all data time series, all data time series are classified to obtain several categories; Record any data time series as the target sequence, record all other data time series except the target sequence in the category of the target sequence as comparison sequences, record any comparison sequence as the target comparison sequence, and record any data in the target sequence as the target data; According to the similarity sequence between the target data and the target comparison sequence, the contribution relationship between the target data and the target comparison sequence is obtained; according to the contribution relationship between the target data and all comparison sequences, the comprehensive correlation contribution degree, positive contribution sequence set and negative contribution sequence set of the target data are obtained; Decomposing the original data matrix composed of all data time series to obtain all retained eigenvalues ​​and retained eigenvectors and all eigenvalues ​​and eigenvectors to be determined; An initial reconstruction matrix is ​​obtained by reconstructing all the retained eigenvalues ​​and retained eigenvectors; Record any eigenvalue to be judged as the target eigenvalue to be judged, record the corresponding eigenvector to be judged as the target eigenvector to be judged, reconstruct the target eigenvector to be judged based on all retained eigenvalues ​​and retained eigenvectors and the target eigenvalue to be judged and the target eigenvector to be judged to obtain a reconstruction matrix of the target eigenvector to be judged; obtain a difference matrix and a ratio matrix of the target eigenvector to be judged based on the initial reconstruction matrix and the reconstruction matrix; Obtain the reference point of each column according to the difference matrix, and obtain the increase degree of the target data according to the ratio matrix, the positive contribution sequence set and the negative contribution sequence set; record all data with an increase degree greater than 0 as valid data; Obtain all retained eigenvectors and corresponding retained eigenvalues ​​according to the increase degree of all valid data and the comprehensive correlation contribution of the reference points in the column where the valid data is located; A low-rank matrix and a sparse matrix are reconstructed according to all the retained eigenvalues ​​and retained eigenvectors, and the low-rank matrix and the sparse matrix are compressed and stored.

2. A method for processing biological organic fertilizer production process data according to claim 1, characterized in that, The method of classifying all data time series according to the similarity of all data time series to obtain several categories includes the following specific steps: The data time series of each parameter is taken as a node of the graph structure; the cosine similarity of the data time series corresponding to two different nodes is taken as the edge value between the two nodes; the embedding vector of each node in the graph structure is obtained by the GrapHSAGE method; Calculate the cosine similarity of the embedding vectors of every two nodes, divide the data time series sequences corresponding to all nodes whose cosine similarity of the embedding vectors is greater than the first threshold into one category, and the cosine similarity of the embedding vectors of any two nodes in the same category is greater than the first threshold, and divide the data time series sequences corresponding to all nodes into several categories, each category is a set of data time series sequences corresponding to several nodes.

3. A method for processing biological organic fertilizer production process data according to claim 1, characterized in that, The step of obtaining the contribution relationship between the target data and the target comparison sequence includes the following specific steps: Each integer within a preset range is used as a window length. For any window length, a subsequence with a length equal to the window length is obtained with the target data as the center point, the mean of all data in the subsequence is obtained, and the target data in the target sequence is replaced with the mean; the cosine similarity between the replaced target sequence and the target comparison sequence is recorded as the similarity between the target data of the target sequence and the target comparison sequence under the window length; a sequence composed of the similarities between the target data of the target sequence and the target comparison sequence under all window lengths is recorded as the similarity sequence between the target data of the target sequence and the target comparison sequence; The window length is used as the horizontal coordinate, and the similarity between the target data of the target sequence and the target comparison sequence under each window length is used as the vertical coordinate to construct a two-dimensional space. The similarity between the target data of the target sequence and the target comparison sequence under all window lengths is converted into a number of spatial points in the two-dimensional space. Principal component analysis is performed on all spatial points to obtain the principal component directions of the target data of the target sequence and the target comparison sequence; the angle between the principal component direction and the positive direction of the y-axis is determined. If the angle is within the range of [0°, 90°], the target data of the target sequence has a positive contribution relationship with the target comparison sequence; otherwise, the target data of the target sequence has a negative contribution relationship with the target comparison sequence.

4. A method for processing biological organic fertilizer production process data according to claim 1, characterized in that, The step of obtaining the comprehensive correlation contribution degree, positive contribution sequence set and negative contribution sequence set of the target data includes the following specific steps: The cotangent value of the angle between the principal component direction and the positive direction of the y-axis is recorded as the correlation contribution between the target data of the target sequence and the target comparison sequence; the average of the correlation contributions between the target data of the target sequence and all comparison sequences is recorded as the comprehensive correlation contribution of the target data of the target sequence; The contribution relationship between the target data of the target sequence and all comparison sequences is obtained, and the set consisting of all comparison sequences that have a positive contribution relationship with the target data of the target sequence is recorded as the positive contribution sequence set of the target data of the target sequence, and the set consisting of all comparison sequences that have a negative contribution relationship with the target data of the target sequence is recorded as the negative contribution sequence set of the target data of the target sequence.

5. A method for processing biological organic fertilizer production process data according to claim 1, characterized in that, The specific steps of obtaining all retained eigenvalues ​​and retained eigenvectors and all to-be-determined eigenvalues ​​and to-be-determined eigenvectors are as follows: The data time series sequence of each parameter is used as each row of the original data matrix to obtain the original data matrix; the original data matrix is ​​decomposed by the singular value decomposition algorithm to obtain several eigenvalues ​​and eigenvectors, the largest first K eigenvalues ​​are recorded as retained eigenvalues, and the eigenvectors corresponding to the retained eigenvalues ​​are recorded as retained eigenvectors, which contain most of the information in the original matrix; the eigenvalues ​​except the largest first K eigenvalues ​​are recorded as eigenvalues ​​to be judged, and the eigenvectors corresponding to the eigenvalues ​​to be judged are recorded as eigenvectors to be judged, and K is a preset number.

6. A method for processing biological organic fertilizer production process data according to claim 1, characterized in that: The specific steps of obtaining the difference matrix and the ratio matrix of the target feature vector to be determined are as follows: The difference matrix is ​​obtained by subtracting the elements at the same position in the initial reconstruction matrix and the reconstruction matrix; and the ratio matrix is ​​obtained according to the ratio of the elements at each position in the difference matrix to the elements at the same position in the initial reconstruction matrix.

7. A method for processing biological organic fertilizer production process data according to claim 1, characterized in that, The step of obtaining the increase degree of target data includes the following specific steps: Obtaining a positive ratio sequence of the target data of the target sequence according to a positive contribution sequence set and a ratio matrix of the target data of the target sequence; Obtaining a negative ratio sequence of the target data of the target sequence according to a negative contribution sequence set and a ratio matrix of the target data of the target sequence; The difference between the sum of all ratios in the positive ratio sequence of the target data of the target sequence and the sum of all ratios in the negative ratio sequence of the target data of the target sequence is recorded as the increase degree of the target data of the target sequence.

8. A method for processing biological organic fertilizer production process data according to claim 1, characterized in that, The specific steps of obtaining all retained eigenvectors and corresponding retained eigenvalues ​​are as follows: The calculation formula for the correlation retention degree of the target feature vector to be judged is: In the formula, p represents the degree of relevance preservation of the target feature vector to be judged, n represents the number of all valid data, and g i represents the comprehensive correlation contribution of the benchmark point in the column where the i-th valid data is located, z i Indicates the increase degree of the i-th valid data; The correlation retention degrees of all feature vectors to be determined are calculated, and feature vectors to be determined whose correlation retention degrees are greater than a second threshold are recorded as retained feature vectors, and the corresponding feature values ​​to be determined are recorded as retained feature values.

9. A method for processing biological organic fertilizer production process data according to claim 7, characterized in that: The step of obtaining the positive ratio sequence of the target data of the target sequence according to the positive contribution sequence set and the ratio matrix of the target data of the target sequence comprises the following specific steps: Record any positive contribution sequence in the set of positive contribution sequences of the target data of the target sequence as the target positive contribution sequence, obtain the data in the target positive contribution sequence at the same position as the target data, and record it as the positive contribution data of the target data of the target sequence; obtain all the positive contribution data of the target data of the target sequence, and record the sequence composed of the ratios corresponding to all the positive contribution data in the ratio sequence as the positive ratio sequence of the target data of the target sequence.

10. A biological organic fertilizer production process data extraction system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

Images