Multi-node data synchronization method and device based on target characteristics

Abstract

The invention discloses a multi-node data synchronization method and device based on target characteristics. The method comprises the following steps: acquiring target data information; performing data segmentation processing on the target data information to obtain an original data packet set; the original data packet set comprises original data packets; issuing the original data packet set to each node of a computer network; and each node performs data synchronization processing on the received original data packet to realize classified storage of the data. According to the method, the multi-node and differentiated construction requirements of the target characteristic database are met, unified collection and on-demand distribution of the target characteristic data are realized, and the multi-level cascade and consistency requirements of the data are met.

Description

Technical Field

[0001] The present invention relates to the fields of data processing and computers, and in particular to a multi-node data synchronization method and device based on target characteristics. Background Art

[0002] In computer networks, traditional data synchronization technology is mainly based on storage layer protocols or application layer protocols to achieve data synchronization between multiple nodes. Synchronization based on storage layer protocols can achieve data synchronization between multiple nodes well, but it cannot meet the business needs of each node and can only achieve full synchronization. Traditional data synchronization based on application layer protocols is mostly oriented to structured data and cannot meet the synchronization needs for unstructured data files.

[0003] In order to meet the needs of multi-node and differentiated construction of the target characteristic database, it is an urgent problem to build a multi-node system that meets different capability requirements, realize the unified collection and on-demand distribution of target characteristic data, and meet the multi-level cascading and consistency requirements of data. Summary of the invention

[0004] The present invention mainly solves the problem of meeting the needs of multi-node and differentiated construction of target characteristic database, building a multi-node system that meets different capability requirements, realizing unified collection and on-demand distribution of target characteristic data, and meeting the multi-level cascading and consistency requirements of data. The present invention discloses a multi-node data synchronization method and device based on target characteristics.

[0005] In a first aspect of an embodiment of the present application, a multi-node data synchronization method based on target characteristics is disclosed, comprising:

[0006] S1, obtain target data information;

[0007] S2, performing data segmentation processing on the target data information to obtain an original data packet set; the original data packet set includes the original data packet;

[0008] S3, sending the original data packet set to each node of the computer network;

[0009] S4, each node performs data synchronization processing on the received original data packets to achieve classified storage of the data.

[0010] Each node performs data synchronization processing on the received original data packets to implement classified storage of the data, including:

[0011] S41, each node pre-processes the received original data packet to obtain a pre-processed data packet;

[0012] S42, using preset classification information, classifying and identifying the preprocessed data packet to obtain category information of each target characteristic data of the preprocessed data packet;

[0013] S43, classifying and storing the target characteristic data of the pre-processed data packet based on the category information of the data.

[0014] The step of preprocessing the received original data packet to obtain a preprocessed data packet includes:

[0015] S411, performing data cleaning processing on the original data packet to obtain a first data packet;

[0016] S412, normalizing the first data packet to obtain a second data packet;

[0017] S413: Perform boundary check processing on the second data packet to obtain a pre-processed data packet.

[0018] The performing boundary check processing on the second data packet to obtain a pre-processed data packet includes:

[0019] S4131, preset a corresponding value range for each data attribute of the second data packet; the value range includes an upper limit value of the value range and a lower limit value of the value range;

[0020] S4132, for each data of the second data packet, judging whether the data value is within the value range according to the value range of its data attribute; if the data value is within the value range, not processing the data; if the data value is not within the value range, setting the data value to the upper limit value or the lower limit value of the value range that is closest to the data value;

[0021] S4133, performing fusion processing on all data of the second data packet after completing S4132 to obtain a preprocessed data packet.

[0022] The performing boundary check processing on the second data packet to obtain a pre-processed data packet includes:

[0023] S4131, for each type of data attribute of the second data packet, using the data collection information of the data as an independent variable and the data value of the data as a dependent variable, perform autoregression-sliding average modeling to obtain regression models of the data attributes; using the regression model, perform calculation processing on the independent variable to obtain regression data values; determine whether the absolute value of the difference between the regression data value and the corresponding dependent variable value is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the second data packet; if it is less than or equal to the first regression discrimination threshold, do not process the data;

[0024] S4132, construct a preprocessed data packet using all data of the second data packet after executing S4131.

[0025] The method of using the preset classification information to classify and identify the preprocessed data packet to obtain the category information of each target characteristic data of the preprocessed data packet includes:

[0026] S421, obtaining preset classification information; the classification information is represented as a classification matrix; the row vector of the classification matrix is ​​a vector constructed by typical values ​​of all data attributes of each target characteristic type; the row number of the row vector of the classification matrix is ​​a classification number; the classification number represents the category information of the target characteristic type corresponding to the row vector;

[0027] S422, using each target characteristic data of the preprocessed data packet, respectively construct a corresponding vector to be identified;

[0028] S423, performing cross-correlation operations on each vector to be identified and each row vector of the classification matrix to obtain a corresponding cross-correlation sequence;

[0029] S424, constructing a corresponding correlation matrix using all the mutual correlation sequences corresponding to each vector to be identified;

[0030] S425, decomposing the correlation matrix to obtain a feature matrix;

[0031] S426, extracting the diagonal elements of the characteristic matrix to obtain a characteristic vector; performing linear fitting processing on the elements and element sequence values ​​of the characteristic vector to obtain the best consistent approximation polynomial;

[0032] S427, using the best consistent approximation polynomial, respectively calculate and process the elements of each eigenvector to obtain a corresponding calculated value;

[0033] S428, determining the element of the feature vector corresponding to the minimum calculated value, the sequence number of which in the feature vector is the classification sequence number of the corresponding vector to be identified;

[0034] S429, executing S422 to S428 in sequence for each target characteristic data of the preprocessed data packet to obtain category information of each target characteristic data of the preprocessed data packet.

[0035] The decomposition process is calculated as follows:

[0036] Y=UAV,

[0037] Among them, U is the left decomposition matrix, A is the characteristic matrix, V is the right decomposition matrix, U and V are both orthogonal matrices, A is a diagonal matrix, and Y is the correlation matrix.

[0038] According to a second aspect of an embodiment of the present invention, a multi-node data synchronization device based on target characteristics is disclosed, the device comprising:

[0039] A memory storing executable program code;

[0040] a processor coupled to the memory;

[0041] The processor calls the executable program code stored in the memory to execute the multi-node data synchronization method based on target characteristics.

[0042] According to a third aspect of an embodiment of the present invention, a computer storable medium is disclosed, wherein the computer storable medium stores computer instructions, and when the computer instructions are called by a computer, they are used to execute the target characteristic-based multi-node data synchronization method.

[0043] According to a fourth aspect of an embodiment of the present invention, an information data processing terminal is disclosed. The information data processing terminal is used to implement the multi-node data synchronization method based on target characteristics.

[0044] The beneficial effects of the present invention are:

[0045] 1. The method and device of the present invention can classify different types of target characteristic data, effectively suppress the influence of noise and interference, meet the needs of multi-node and differentiated construction of the target characteristic database, build a multi-node system that meets different capability requirements, realize the unified collection and on-demand distribution of target characteristic data, and meet the multi-level cascading and consistency requirements of data.

[0046] 2. The method of the present invention can ensure data consistency between system nodes at all levels and provide a channel for online sharing and distribution of data between nodes at all levels, so that the original files and formatted information of the target characteristics can be shared, thereby realizing unified access and sharing of the target characteristic data. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart for implementing the method of the present invention. DETAILED DESCRIPTION

[0048] In order to better understand the content of the present invention, two embodiments are given here.

[0049] Figure 1 It is a flow chart for implementing the method of the present invention.

[0050] In order to solve the online synchronization of target characteristic data between different nodes, the present invention proposes a technical method for online synchronization of multi-node data based on target characteristics, which completes the node synchronization of target characteristic original file data and structured information by segmenting, issuing and parsing the target characteristic data, thereby realizing online data sharing.

[0051] Embodiment 1:

[0052] In a first aspect of an embodiment of the present application, a multi-node data synchronization method based on target characteristics is disclosed, comprising:

[0053] S1, obtain target data information;

[0054] S2, performing data segmentation processing on the target data information to obtain an original data packet set; the original data packet set includes the original data packet;

[0055] S3, sending the original data packet set to each node of the computer network;

[0056] S4, each node performs data synchronization processing on the received original data packets to achieve classified storage of the data.

[0057] Each node performs data synchronization processing on the received original data packets to implement classified storage of the data, including:

[0058] S41, preprocessing the received original data packet to obtain a preprocessed data packet;

[0059] S42, using preset classification information, classifying and identifying the preprocessed data packet to obtain category information of each target characteristic data of the preprocessed data packet;

[0060] S43, classifying and storing the target characteristic data of the pre-processed data packet based on the category information of the data.

[0061] The step of preprocessing the received original data packet to obtain a preprocessed data packet includes:

[0062] S411, performing data cleaning processing on the original data packet to obtain a first data packet;

[0063] S412, normalizing the first data packet to obtain a second data packet;

[0064] S413, performing boundary check processing on the second data packet to obtain a pre-processed data packet;

[0065] The performing boundary check processing on the second data packet to obtain a pre-processed data packet includes:

[0066] S4131, preset a corresponding value range for each data attribute of the second data packet; the value range includes an upper limit value of the value range and a lower limit value of the value range;

[0067] S4132, for each data of the second data packet, judging whether the data value is within the value range according to the value range of its data attribute; if the data value is within the value range, not processing the data; if the data value is not within the value range, setting the data value to the upper limit value or the lower limit value of the value range that is closest to the data value;

[0068] S4133, constructing a pre-processed data packet using all data of the second data packet after S4132 is completed;

[0069] The performing boundary check processing on the second data packet to obtain a pre-processed data packet includes:

[0070] S4131, for each type of data attribute of the second data packet, using the data collection information of the data as an independent variable and the data value of the data as a dependent variable, perform autoregression-sliding average modeling to obtain regression models of the data attributes; using the regression model, perform calculation processing on the independent variable to obtain regression data values; determine whether the absolute value of the difference between the regression data value and the corresponding dependent variable value is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, delete the data from the second data packet; if it is less than or equal to the first regression discrimination threshold, do not process the data;

[0071] S4132, construct a preprocessed data packet using all data of the second data packet after executing S4131.

[0072] The method of using the preset classification information to classify and identify the preprocessed data packet to obtain the category information of each target characteristic data of the preprocessed data packet includes:

[0073] S421, obtaining preset classification information; the classification information is represented as a classification matrix; the row vector of the classification matrix is ​​a vector constructed by typical values ​​of all data attributes of each target characteristic type; the row number of the row vector of the classification matrix is ​​a classification number; the classification number represents the category information of the target characteristic type corresponding to the row vector;

[0074] S422, using each target characteristic data of the preprocessed data packet, respectively construct a corresponding vector to be identified;

[0075] S423, performing cross-correlation operations on each vector to be identified and each row vector of the classification matrix to obtain a corresponding cross-correlation sequence;

[0076] S424, constructing a corresponding correlation matrix using all the mutual correlation sequences corresponding to each vector to be identified;

[0077] S425, decomposing the correlation matrix to obtain a feature matrix;

[0078] S426, extracting the diagonal elements of the characteristic matrix to obtain a characteristic vector; performing linear fitting processing on the elements and element sequence values ​​of the characteristic vector to obtain the best consistent approximation polynomial;

[0079] S427, using the best consistent approximation polynomial, respectively calculate and process the elements of each eigenvector to obtain a corresponding calculated value;

[0080] S428, determining the element of the feature vector corresponding to the minimum calculated value, the sequence number of which in the feature vector is the classification sequence number of the corresponding vector to be identified;

[0081] S429, executing S422 to S428 in sequence for each target characteristic data of the preprocessed data packet to obtain category information of each target characteristic data of the preprocessed data packet.

[0082] The decomposition process is calculated as follows:

[0083] Y=UAV,

[0084] Among them, U is the left decomposition matrix, A is the characteristic matrix, V is the right decomposition matrix, U and V are both orthogonal matrices, A is a diagonal matrix, and Y is the correlation matrix;

[0085] The method of using each target characteristic data of the preprocessed data packet to construct a corresponding vector to be identified is to use each data in each target characteristic data as a vector element to construct a corresponding vector to be identified.

[0086] The decomposition process can be implemented by using a matrix singular value decomposition algorithm.

[0087] The feature vector is represented as I a , I a =[λ1,λ2,…,λ N1 ], N1 is the number of elements contained in the eigenvector;

[0088] The linear fitting process uses the characteristic vector element serial number value Ix as a known independent variable and the characteristic vector element value as a known dependent variable, and constructs the curve to be approximated using the known independent variable and the known dependent variable, and uses the function approximation method to perform curve fitting on the curve to be approximated to obtain the best consistent approximation polynomial f(Ix).

[0089] The curve fitting of the curve to be approximated by the function approximation method may be performed by using the best consistent linear approximation method. The best consistent approximation polynomial f(Ix) is expressed as:

[0090] f(Ix)=α P1 (Ix) P1 +α P1-1 (Ix) P1-1 +…+α2(Ix) 2 +α1(Ix)+α0,

[0091] Where P1 is the order of the best consistent approximation polynomial f(Ix), α0, α1, α2, …, α P1 are the coefficients of the optimal consistent approximation polynomial f(Ix);

[0092] The data cleaning process includes filling missing values, smoothing noise data, and smoothing or deleting outliers; the smoothed noise data is obtained by first determining the noise data, and then smoothing the noise data according to the data before and after the noise data; the noise data is a value whose value is less than the detection sensitivity of the sensor of the observed data, or greater than the measurement upper limit of the sensor of the observed data. The outlier point can be determined by the Kalman filter method. The filling value of the missing value can be determined by averaging the measured values ​​within a certain sampling interval before and after the missing value.

[0093] According to a second aspect of an embodiment of the present invention, a multi-node data synchronization device based on target characteristics is disclosed, the device comprising:

[0094] A memory storing executable program code;

[0095] a processor coupled to the memory;

[0096] The processor calls the executable program code stored in the memory to execute the multi-node data synchronization method based on target characteristics.

[0097] According to a third aspect of an embodiment of the present invention, a computer storable medium is disclosed, wherein the computer storable medium stores computer instructions, and when the computer instructions are called by a computer, they are used to execute the target characteristic-based multi-node data synchronization method.

[0098] According to a fourth aspect of the embodiments of the present invention, an information data processing terminal is disclosed. The information data processing terminal is used to implement the multi-node data synchronization method based on target characteristics.

[0099] Embodiment 2:

[0100] This embodiment discloses a multi-level data online synchronization technology method based on target characteristics, including the following steps:

[0101] (1) Data segmentation of characteristic data to generate data packets containing original files and formatted information (trigger-based data synchronization method);

[0102] (2) Call the data delivery module to deliver data (designed based on the Client / Server architecture);

[0103] (3) After receiving the data packet, the lower level parses the data according to the packet identifier to obtain the target and data information;

[0104] (4) Parsing and storing the characteristic data content according to the target characteristic data format;

[0105] (5) The parsed data is classified, stored and managed to complete multi-level synchronization of target characteristic data.

[0106] The data packet obtains the corresponding target and characteristic data identifiers and files according to the target characteristic data type and the data measurement condition range, generates data packet identification information, and realizes the packaging of data identification and original file information.

[0107] Among them, the data sending module calls the sending module to complete the transmission of the data packet according to the lower-level database information and data type, and calls the receiving module at the lower level to complete the reception of the data packet.

[0108] The data packet header contains metadata information of characteristic data, and the characteristic types include basic characteristics, motion characteristics, scattered radiation characteristics, etc.

[0109] Among them, before the data is entered into the database, the standard format of each characteristic data is customized in combination with the measurement conditions of each characteristic data; then the same-name relationship library is established, and different types of parsing modules are called according to the characteristic data information obtained from the packet header, and the original files are parsed according to the format of each type of characteristic data to complete the parsing and storage of the target characteristic data.

[0110] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A multi-node data synchronization method based on target characteristics, characterized in that: include: S1, obtain target data information; S2, performing data segmentation processing on the target data information to obtain an original data packet set; The original data packet set includes original data packets; S3, sending the original data packet set to each node of the computer network; S4, each node performs data synchronization processing on the received original data packets to achieve classified storage of the data.

2. The multi-node data synchronization method based on target characteristics according to claim 1, characterized in that: Each node performs data synchronization processing on the received original data packets to implement classified storage of the data, including: S41, each node pre-processes the received original data packet to obtain a pre-processed data packet; S42, using preset classification information, classifying and identifying the preprocessed data packet to obtain category information of each target characteristic data of the preprocessed data packet; S43, classifying and storing the target characteristic data of the pre-processed data packet based on the category information of the data.

3. The multi-node data synchronization method based on target characteristics as claimed in claim 2, characterized in that: The step of preprocessing the received original data packet to obtain a preprocessed data packet includes: S411, performing data cleaning processing on the original data packet to obtain a first data packet; S412, normalizing the first data packet to obtain a second data packet; S413: Perform boundary check processing on the second data packet to obtain a pre-processed data packet.

4. The multi-node data synchronization method based on target characteristics as claimed in claim 3, characterized in that: The performing boundary check processing on the second data packet to obtain a pre-processed data packet includes: S4131, preset a corresponding value range for each data attribute of the second data packet; the value range includes an upper limit value of the value range and a lower limit value of the value range; S4132, for each data of the second data packet, judging whether the data value is within the value range according to the value range of its data attribute; if the data value is within the value range, not processing the data; if the data value is not within the value range, setting the data value to the upper limit value or the lower limit value of the value range that is closest to the data value; S4133, performing fusion processing on all data of the second data packet after completing S4132 to obtain a preprocessed data packet.

5. The multi-node data synchronization method based on target characteristics as claimed in claim 3, characterized in that: The performing boundary check processing on the second data packet to obtain a pre-processed data packet includes: S41301, for each type of data attribute of the second data packet, using data collection information of the data as an independent variable and data values ​​of the data as a dependent variable, perform autoregression-sliding average modeling to obtain regression models of the data attributes of each type; S41302, using the regression model, calculating and processing the independent variable to obtain a regression data value; determining whether the absolute value of the difference between the regression data value and the corresponding dependent variable value is greater than a set first regression discrimination threshold; if it is greater than the first regression discrimination threshold, deleting the data from the second data packet; if it is less than or equal to the first regression discrimination threshold, retaining the data in the second data packet; S41303, construct a pre-processed data packet using all data of the second data packet after executing S41301.

6. The multi-node data synchronization method based on target characteristics as claimed in claim 3, characterized in that: The method of using the preset classification information to classify and identify the preprocessed data packet to obtain the category information of each target characteristic data of the preprocessed data packet includes: S421, obtaining preset classification information; the classification information is represented as a classification matrix; the row vector of the classification matrix is ​​a vector constructed by typical values ​​of all data attributes of each target characteristic type; the row number of the row vector of the classification matrix is ​​a classification number; the classification number represents the category information of the target characteristic type corresponding to the row vector; S422, using each target characteristic data of the preprocessed data packet, respectively construct a corresponding vector to be identified; S423, performing cross-correlation operations on each vector to be identified and each row vector of the classification matrix to obtain a corresponding cross-correlation sequence; S424, constructing a corresponding correlation matrix using all the mutual correlation sequences corresponding to each vector to be identified; S425, decomposing the correlation matrix to obtain a feature matrix; S426, extracting the diagonal elements of the characteristic matrix to obtain a characteristic vector; performing linear fitting processing on the elements and element sequence values ​​of the characteristic vector to obtain the best consistent approximation polynomial; S427, using the best consistent approximation polynomial, respectively calculate and process the elements of each eigenvector to obtain a corresponding calculated value; S428, determining the element of the feature vector corresponding to the minimum calculated value, the sequence number of which in the feature vector is the classification sequence number of the corresponding vector to be identified; S429, executing S422 to S428 in sequence for each target characteristic data of the preprocessed data packet to obtain category information of each target characteristic data of the preprocessed data packet.

7. The multi-node data synchronization method based on target characteristics as claimed in claim 6, characterized in that: The decomposition process is calculated as follows: Y=UAV, Among them, U is the left decomposition matrix, A is the characteristic matrix, V is the right decomposition matrix, U and V are both orthogonal matrices, A is a diagonal matrix, and Y is the correlation matrix.

8. A multi-node data synchronization device based on target characteristics, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the target characteristic-based multi-node data synchronization method according to any one of claims 1 to 7.

9. A computer storable medium, characterized in that: The computer storable medium stores computer instructions, and when the computer instructions are called by a computer, they are used to execute the target characteristic-based multi-node data synchronization method according to any one of claims 1 to 7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the multi-node data synchronization method based on target characteristics as described in any one of claims 1 to 7.

Images