A big data analysis system and method for computer parallel synchronization

Abstract

The present invention discloses a big data analysis system and method for computer parallel synchronization, which relates to the field of data analysis technology, including a parallel management module, a data analysis module, a database, a cloud platform, and an intelligent terminal. The present invention can perform parallel management of data in the system; can analyze and integrate system data; can store system data and provide a data basis for the system; can perform cloud-based cluster management and calculation of system data; can set requirements for system data and view system data; can analyze and calculate system parallel nodes; can trace system data; can perform parallel management, analysis and calculation of system parallel data, can analyze and integrate data of different task nodes, evaluate the accuracy of data calculation of different task nodes, facilitate rapid positioning of data nodes with miscalculations, and then re-verify and calculate the data of the data nodes.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a big data analysis system and method for computer parallel synchronization. Background Art

[0002] Parallel processing is a computing method that can execute two or more processors simultaneously in a computer system. The processor can work on different aspects of the same program at the same time. The main purpose of parallel processing is to save time in solving large and complex problems. Its basic idea is to use multiple processors to collaboratively solve the same problem, that is, to decompose the problem to be solved into several parts, each of which is calculated in parallel by an independent processor. Parallel computing can be divided into temporal parallelism and spatial parallelism; temporal parallelism refers to pipeline technology, while spatial parallelism refers to the use of multiple processors to perform calculations concurrently.

[0003] The patent (CN114217871A) discloses a multi-computer cluster parallel simulation method, including sending a model file to a file server, the model file includes a simulation task configuration file and a simulation task file; determining a target node computer, and sending a task instruction to the target node computer to assign a simulation task; when the target node computer obtains the model file in the file server according to the task instruction, sending a start instruction to the target node computer to start the simulation task execution tool in the target node computer, and the simulation task execution tool can execute the corresponding simulation task according to the simulation task configuration file; when the target node computer completes the simulation task, the simulation result of the target node computer is obtained, and the simulation result is analyzed and processed. The present invention also discloses an aircraft engine multi-computer cluster system. The multi-computer cluster parallel simulation method provided by the present invention can realize parallel simulation.

[0004] The multi-computer cluster parallel simulation method in the above patent document has poor parallel computing effect in tracing the accuracy of analysis and calculation of different task nodes, and is not convenient for timely and accurate positioning of data nodes with calculation errors, resulting in cumbersome recalculation once calculation errors occur. Summary of the invention

[0005] The object of the present invention is to provide a big data analysis system and method for computer parallel synchronization to solve the problems raised in the above background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: a big data analysis system for computer parallel synchronization, comprising a parallel management module, a data analysis module, a database, a cloud platform, and an intelligent terminal, wherein the parallel management module performs parallel management on the data in the system; the data analysis module performs analysis, integration and processing on the system data; the database stores the system data and provides a data basis for the system; the cloud platform performs cloud-based cluster management and calculation on the system data; the intelligent terminal is used to set requirements for system data and view system data; the parallel management module, the data analysis module, the database, and the intelligent terminal are respectively connected to the cloud platform for communication data, and the parallel management module is connected to the data analysis module for communication data.

[0007] Furthermore, the parallel management module includes a bottom-level computing unit, a middle-level computing unit, and a top-level computing unit. The bottom-level computing unit performs bottom-level computing on parallelized data; the middle-level computing unit performs middle-level computing on parallelized data; and the top-level computing unit performs top-level computing on parallelized data.

[0008] Furthermore, the bottom-level computing unit is a basic data acquisition component, the middle-level computing unit is a data transmission transfer component, and the top-level computing unit is a terminal computer.

[0009] Furthermore, the bottom-level computing unit is a basic computer, the middle-level computing unit is a data transmission transfer component, and the top-level computing unit is a cloud computing processor.

[0010] Furthermore, the data analysis module includes a node analysis unit and a data tracing unit. The node analysis unit performs analysis and calculation processing on the system parallel nodes; the data tracing unit performs tracing processing on the system data.

[0011] The present invention also provides an analysis method for a big data analysis system for computer parallel synchronization, comprising the following steps:

[0012] S1. The system is turned on through the intelligent terminal, and the system starts to run. The parallel management module performs parallel management, analysis and calculation on the system parallel data; and uploads the calculation results and basic calculation data to the cloud platform;

[0013] S2. The cloud platform sends the data to the data analysis module, which analyzes and integrates the data of different task nodes and evaluates the accuracy of the data calculation of different task nodes;

[0014] Upload the above calculation process and calculation results to the cloud platform;

[0015] S3. The cloud platform sends the data to the database and smart terminal, and the user views the system data through the smart terminal.

[0016] Further, in step S1, a bottom-level computing node X, a middle-level computing node Y, and a top-level computing node Z are established. The bottom-level computing node X performs bottom-level parallel synchronous analysis and calculation on the system data at the bottom level of the system; the middle-level computing node Y performs middle-level parallel synchronous analysis and calculation on the system data at the middle level of the system; and the top-level computing node Z performs top-level parallel synchronous analysis and calculation on the system data at the top level of the system.

[0017] Furthermore, in step S2, the basic data in the underlying computing node X is A x ; The middle-level data of the middle-level computing node Y is B y ; The top-level data of the top-level computing node Z is C z ;

[0018] The analysis and calculation of the data in the underlying computing node X is as follows:

[0019] E x Indicates the discreteness of the data in the underlying computing node X. It is used to determine the correlation between the data in the underlying computing node X. It is directly related to the calculation logic in the underlying computing node X and indicates the degree of deviation between the calculation logic of the previous and next data.

[0020] The analysis and calculation of the data in the middle-level computing node Y is as follows:

[0021] F y Indicates the discreteness of the data in the middle-level computing node Y, which is used to judge the correlation between the data in the middle-level computing node Y. It is directly related to the calculation logic in the middle-level computing node Y and indicates the degree of deviation between the calculation logic of the previous and next data.

[0022] The analysis and calculation of the data in the underlying computing node Z is as follows:

[0023] H z Indicates the discreteness of the data in the top-level computing node Z. It is used to determine the correlation between the data in the top-level computing node Z. It is directly related to the calculation logic in the top-level computing node Z and indicates the degree of deviation between the calculation logic of the previous and next data.

[0024] Furthermore, For x The greater the deviation between the calculation logic of the previous and next data, the greater the x The smaller the value;

[0025] For g yThe greater the deviation between the calculation logic of the previous and next data, the greater the y The smaller the value;

[0026] For z The greater the deviation between the calculation logic of the previous and next data, the greater the z The smaller the value.

[0027] Furthermore, K ixyz =e x ×g y ×j z ;

[0028] L m =K 1111 ×K 2111 ×K 2211 ×K 2121 ×K 2112 ×……×K ixyz ×……×K mmmm ; L m For all K ixyz The product of

[0029] When L m <1, the data is abnormal, trace back and check K directly ixyz <1 data, and then directly locate and check e x <1, g y <1 or j z <1, so that the data node with calculation error can be directly located, and then the data of the data node can be re-checked and calculated.

[0030] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0031] 1. The present invention can perform parallel management of data in the system by setting a parallel management module, a data analysis module, a database, a cloud platform, and an intelligent terminal; can analyze and integrate system data; can store system data and provide a data basis for the system; can perform cloud-based cluster management and calculation of system data; can set requirements for system data and view system data; can analyze and calculate system parallel nodes; can trace system data; can perform parallel management, analysis and calculation of system parallel data, can analyze and integrate data of different task nodes, and evaluate the accuracy of data calculation of different task nodes, so as to quickly locate data nodes with calculation errors, and then re-verify and calculate the data of the data node;

[0032] 2. In the present invention, the bottom computing node X performs bottom-level parallel synchronous analysis and calculation on the system data at the bottom level of the system; the middle-level computing node Y performs middle-level parallel synchronous analysis and calculation on the system data at the middle level of the system; the top-level computing node Z performs top-level parallel synchronous analysis and calculation on the system data at the top level of the system; and uploads the calculation results and the basic calculation data to the cloud platform; the data analysis module analyzes and integrates the data of different task nodes, and evaluates the accuracy of the calculation of the data of different task nodes; the discreteness of the data in the bottom computing node X is used to judge the correlation between the data in the bottom computing node X, which is directly related to the bottom computing node The calculation logic in point X indicates the degree of deviation between the calculation logics of the previous and next data. The discreteness of the data in the middle-level calculation node Y is used to determine the correlation between the data in the middle-level calculation node Y, which is directly related to the calculation logic in the middle-level calculation node Y, indicating the degree of deviation between the calculation logics of the previous and next data. The discreteness of the data in the top-level calculation node Z is used to determine the correlation between the data in the top-level calculation node Z, which is directly related to the calculation logic in the top-level calculation node Z, indicating the degree of deviation between the calculation logics of the previous and next data. The data node with the calculation error can be directly located, and then the data of the data node can be re-checked and calculated. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0034] Figure 1 It is a schematic diagram of the module connection of the present invention as a whole;

[0035] In the figure: 1. Parallel management module; 2. Data analysis module; 3. Database; 4. Cloud platform; 5. Intelligent terminal. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] like Figure 1A big data analysis system for computer parallel synchronization is shown, comprising a parallel management module 1, a data analysis module 2, a database 3, a cloud platform 4, and an intelligent terminal 5. The parallel management module 1 performs parallel management on the data in the system; the data analysis module 2 performs analysis, integration, and processing on the system data; the database 3 stores the system data and provides a data basis for the system; the cloud platform 4 performs cloud-based cluster management and calculations on the system data; the intelligent terminal 5 is used to set requirements for system data and view system data; the parallel management module 1, the data analysis module 2, the database 3, and the intelligent terminal 5 are respectively connected to the cloud platform 4 for communication data, and the parallel management module 1 is connected to the data analysis module 2 for communication data.

[0038] Furthermore, the parallel management module 1 includes a bottom-level computing unit, a middle-level computing unit, and a top-level computing unit. The bottom-level computing unit performs bottom-level computing on parallelized data; the middle-level computing unit performs middle-level computing on parallelized data; and the top-level computing unit performs top-level computing on parallelized data.

[0039] Furthermore, the bottom-level computing unit is a basic data acquisition component, the middle-level computing unit is a data transmission transfer component, and the top-level computing unit is a terminal computer.

[0040] Furthermore, the bottom-level computing unit is a basic computer, the middle-level computing unit is a data transmission transfer component, and the top-level computing unit is a cloud computing processor.

[0041] Furthermore, the data analysis module 2 includes a node analysis unit and a data tracing unit. The node analysis unit performs analysis and calculation processing on the system parallel nodes; the data tracing unit performs tracing processing on the system data.

[0042] The present invention also provides an analysis method for a big data analysis system for computer parallel synchronization, comprising the following steps:

[0043] S1. The system is turned on through the intelligent terminal 5, and the system starts to run. The parallel management module 1 performs parallel management, analysis and calculation on the system parallel data, establishes the bottom computing node X, the middle computing node Y, and the top computing node Z. The bottom computing node X performs bottom-level parallel synchronous analysis and calculation on the system data at the bottom of the system; the middle computing node Y performs middle-level parallel synchronous analysis and calculation on the system data at the middle level of the system; the top computing node Z performs top-level parallel synchronous analysis and calculation on the system data at the top level of the system; and uploads the calculation results and the basic computing data to the cloud platform 4;

[0044] S2, the cloud platform 4 sends the data to the data analysis module 2, which analyzes and integrates the data of different task nodes and evaluates the accuracy of the data calculation of different task nodes; the basic data in the underlying computing node X is A x ; The middle-level data of the middle-level computing node Y is B y ; The top-level data of the top-level computing node Z is C z ;

[0045] The analysis and calculation of the data in the underlying computing node X is as follows:

[0046] E x Indicates the discreteness of the data in the underlying computing node X. It is used to determine the correlation between the data in the underlying computing node X. It is directly related to the calculation logic in the underlying computing node X and indicates the degree of deviation between the calculation logic of the previous and next data.

[0047] The analysis and calculation of the data in the middle-level computing node Y is as follows:

[0048] F y Indicates the discreteness of the data in the middle-level computing node Y, which is used to judge the correlation between the data in the middle-level computing node Y. It is directly related to the calculation logic in the middle-level computing node Y and indicates the degree of deviation between the calculation logic of the previous and next data.

[0049] The analysis and calculation of the data in the underlying computing node Z is as follows:

[0050] H z Indicates the discreteness of the data in the top-level computing node Z. It is used to judge the correlation between the data in the top-level computing node Z. It is directly related to the calculation logic in the top-level computing node Z and indicates the degree of deviation between the calculation logic of the previous and next data.

[0051] For x The greater the deviation between the calculation logic of the previous and next data, the greater the x The smaller the value;

[0052] For g y The greater the deviation between the calculation logic of the previous and next data, the greater the y The smaller the value;

[0053] For z The greater the deviation between the calculation logic of the previous and next data, the greater the z The smaller the value;

[0054] K ixyz =ex ×g y ×j z ;

[0055] L m =K 1111 ×K 2111 ×K 2211 ×K 2121 ×K 2112 ×……×K ixyz ×……×K mmmm ; L m For all K ixyz The product of

[0056] When L m <1, the data is abnormal, trace back and check K directly ixyz <1 data, and then directly locate and check e x <1, g y <1 or j z <1, so that the data node with miscalculation can be directly located, and then the data of the data node can be re-checked and calculated;

[0057] Upload the above calculation process and calculation results to the cloud platform 4;

[0058] S3. The cloud platform 4 sends the data to the database 3 and the smart terminal 5, and the user views the system data through the smart terminal 5.

[0059] This implementation specifically solves the problem in the background technology that parallel computing has poor tracing effect on the accuracy of analysis and calculation of different task nodes, making it difficult to timely and accurately locate data nodes with calculation errors, resulting in cumbersome recalculation once calculation errors occur.

[0060] Working principle of the present invention:

[0061] Refer to the instruction manual Figure 1 By setting up a parallel management module 1, a data analysis module 2, a database 3, a cloud platform 4, and an intelligent terminal 5, the data in the system can be managed in parallel; the system data can be analyzed and integrated; the system data can be stored and a data basis can be provided for the system; the system data can be managed and calculated in a cloud cluster; requirements can be set for system data and system data can be viewed; the system parallel nodes can be analyzed and calculated; the system data can be traced; the system parallel data can be managed, analyzed and calculated in parallel, the data of different task nodes can be analyzed and integrated, and the accuracy of the data calculation of different task nodes can be evaluated, so as to quickly locate the data node with calculation errors, and then the data of the data node can be re-verified and calculated;

[0062] The bottom computing node X performs bottom-level parallel synchronous analysis and calculation on the system data at the bottom foundation of the system; the middle computing node Y performs middle-level parallel synchronous analysis and calculation on the system data at the middle level of the system; the top computing node Z performs top-level parallel synchronous analysis and calculation on the system data at the top level of the system; and uploads the calculation results and the basic calculation data to the cloud platform 4; the data analysis module 2 analyzes and integrates the data of different task nodes, and evaluates the accuracy of the data calculation of different task nodes; the discreteness of the data in the bottom computing node X is used to judge the correlation between the data in the bottom computing node X, which is directly related to the calculation logic in the bottom computing node X, indicating the degree of deviation between the calculation logic of the previous and next data; the discreteness of the data in the middle computing node Y is used to judge the correlation between the data in the middle computing node Y, which is directly related to the calculation logic in the middle computing node Y, indicating the degree of deviation between the calculation logic of the previous and next data; the discreteness of the data in the top computing node Z is used to judge the correlation between the data in the top computing node Z, which is directly related to the calculation logic in the top computing node Z, indicating the degree of deviation between the calculation logic of the previous and next data; the data node with calculation error can be directly located, and then the data of the data node can be re-verified and calculated.

[0063] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A big data analysis system for computer parallel synchronization, comprising a parallel management module (1), a data analysis module (2), a database (3), a cloud platform (4), and an intelligent terminal (5), characterized in that: The parallel management module (1) performs parallel management of the data in the system; the data analysis module (2) performs analysis and integration processing on the system data; the database (3) stores the system data and provides a data basis for the system; the cloud platform (4) performs cloud-based cluster management and calculations on the system data; the intelligent terminal (5) is used to set requirements for system data and view system data; the parallel management module (1), the data analysis module (2), the database (3), and the intelligent terminal (5) are respectively connected to the cloud platform (4) for communication data, and the parallel management module (1) is connected to the data analysis module (2) for communication data; an analysis method for a big data analysis system for computer parallel synchronization comprises the following steps: S1. The system is turned on through the intelligent terminal (5), the system starts to operate, and the parallel management module (1) performs parallel management, analysis and calculation on the system parallel data; and uploads the calculation results and basic calculation data to the cloud platform (4); S2, the cloud platform (4) sends the data to the data analysis module (2), and the data analysis module (2) analyzes and integrates the data of different task nodes and evaluates the accuracy of the data calculation of different task nodes; Upload the above calculation process and calculation results to the cloud platform (4); S3, the cloud platform (4) sends the data to the database (3) and the smart terminal (5), and the user views the system data through the smart terminal (5); In step S1, a bottom computing node X, a middle computing node Y, and a top computing node Z are established. The bottom computing node X performs bottom-level parallel synchronous analysis and calculation on system data at the bottom foundation of the system; the middle computing node Y performs middle-level parallel synchronous analysis and calculation on system data at the middle level of the system; and the top computing node Z performs top-level parallel synchronous analysis and calculation on system data at the top level of the system. In step S2, the basic data in the underlying computing node X is A x ; The middle-level data of the middle-level computing node Y is B y ; The top-level data of the top-level computing node Z is C z ; The analysis and calculation of the data in the underlying computing node X is as follows: E x Indicates the discreteness of the data in the underlying computing node X; The analysis and calculation of the data in the middle-level computing node Y is as follows: F y Indicates the discreteness of the data in the middle-level computing node Y; The analysis and calculation of the data in the underlying computing node Z is as follows: H z Indicates the discreteness of the data in the top-level computing node Z; K ixyz =e x ×g y ×j z ; L m =K 1111 ×K 2111 ×K 2211 ×K 2121 ×K 2112 ×......K ixyz ×......×K mmmm ; L m For all K ixyz The product of When L m <1, the data is abnormal, trace back and check K directly ixyz <1 data, and then directly locate and check e x <1, g y <1 or j z <1, so that the data node with calculation error can be directly located, and then the data of the data node can be re-checked and calculated.

2. A big data analysis system for computer parallel synchronization according to claim 1, characterized in that: The parallel management module (1) comprises a bottom-level computing unit, a middle-level computing unit and a top-level computing unit. The bottom-level computing unit performs bottom-level computing on parallelized data; the middle-level computing unit performs middle-level computing on parallelized data; and the top-level computing unit performs top-level computing on parallelized data.

3. A big data analysis system for computer parallel synchronization according to claim 2, characterized in that: The bottom-level computing unit is a basic data acquisition component, the middle-level computing unit is a data transmission transfer component, and the top-level computing unit is a terminal computer.

4. The big data analysis system for computer parallel synchronization according to claim 2, characterized in that: The bottom-level computing unit is a basic computer, the middle-level computing unit is a data transmission transfer component, and the top-level computing unit is a cloud computing processor.

5. The big data analysis system for computer parallel synchronization according to claim 1, characterized in that: The data analysis module (2) comprises a node analysis unit and a data tracing unit. The node analysis unit performs analysis and calculation processing on the system parallel nodes; the data tracing unit performs tracing processing on the system data.

Images