A CitcomS parallel computing method and system based on the integration of Spark and AI

Abstract

The present invention discloses a CitcomS parallel computing method based on the fusion of Spark and AI. First, the master node collects the original data set required for the geological dynamic evolution process to be simulated, and uses the random data expansion method to expand the original data set to obtain the expanded original data set. Then the master node sets a counter i=1 and determines whether i is greater than the size of the original data set collected in step (1). If not, the master node packages the CitcomS source code file and the i-th data in the original data set into a Java application through the Java native interface JNI. The Java application is a cross-language callable application package. The master node divides the application package into fine-grained tasks to obtain multiple computing tasks with different fine-grained sizes. All computing tasks are submitted to the Spark cluster through the spark-submit tool. The present invention can solve the technical problems that the existing CitcomS computing method cannot simulate complex and diverse geological evolution scenarios and cannot accurately reflect the objective real world.

Description

Technical Field

[0001] The present invention belongs to the technical fields of big data computing, high-performance computing, machine learning and geodynamic simulation. More specifically, it relates to a CitcomS parallel computing method and system based on the fusion of Spark and AI (Artificial Intelligence, abbreviated as AI). Background Art

[0002] CitcomS is a parallel, open-source finite element code for solving the Tethys evolution problem. It is designed to simulate thermochemical convection problems in a three-dimensional domain suitable for convection within the mantle. This modeling approach, applicable to mantle convection, can be viewed as a combination of fluid dynamics and thermodynamics. Compared to other available codes, CitcomS is easy to use, which has attracted interest from the Earth science community and has been used in previous studies of mantle convection.

[0003] The core calculation method of CitcomS is based on solving the motion equation and energy equation in thermochemical convection problems. It starts to iteratively calculate a series of thermodynamic problems in thermochemical convection by setting initial boundary conditions.

[0004] However, the existing CitcomS calculation method has the following problems: First, due to the numerous influencing factors and complex causal mechanisms of the Tethys evolution process, and the small amount of raw data input in the actual geological simulation process, it is impossible to simulate complex and diverse geological evolution scenarios and cannot accurately reflect the objective real world; Second, the existing CitcomS calculation is usually calculated using a single-core or limited-core CPU. The core calculation has a large amount of computation and low parallelism, and does not fully utilize the spatial locality information of its grid calculation, failing to achieve correlation analysis of the data and scientific discoveries based on data statistical laws; Third, due to the limitations of parallel scale and computing power, the core physics complex equations have a large amount of computation and there are a large number of redundant calculations, resulting in low computational efficiency. Summary of the Invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a CitcomS parallel computing method and system based on the fusion of Spark and AI, which aims to solve the technical problems of the existing CitcomS calculation method, which is unable to simulate complex and diverse geological evolution scenarios and accurately reflect the objective real world due to the large number of influencing factors and complex causal mechanism of the Tethys evolution process, and the small amount of original data input in the actual geological simulation process; and the technical problems that the existing CitcomS calculation method cannot simulate complex and diverse geological evolution scenarios and cannot accurately reflect the objective real world due to the large amount of computational complexity of its core calculation, low parallelism, and failure to fully utilize the spatial locality information of its grid calculation, resulting in the failure to realize correlation analysis of data and scientific discovery based on data statistical laws; and the technical problems that the core physics complex equations have a large amount of computational complexity, a large number of redundant calculations, and low computing efficiency due to limitations on parallel scale and computing power.

[0006] To achieve the above objectives, according to one aspect of the present invention, a CitcomS parallel computing method based on the fusion of Spark and AI is provided, which is applied in a computing environment including a master node and a Spark cluster. The CitcomS parallel computing method includes the following steps:

[0007] (1) The master node collects the original data set required for the geodynamic evolution process to be simulated, and uses the random data expansion method to expand the original data set to obtain the expanded original data set.

[0008] (2) The master node sets the counter i=1;

[0009] (3) The master node determines whether i is larger than the size of the original data set collected in step (1). If so, it proceeds to step (11); otherwise, it proceeds to step (4);

[0010] (4) The master node packages the CitcomS source code file and the i-th data in the original dataset into a Java application through the Java native interface JNI, which is an application package that can be called across languages.

[0011] (5) The master node divides the application package obtained in step (4) into fine-grained tasks to obtain multiple computing tasks with different fine-grainedness;

[0012] (6) The master node submits all computing tasks obtained in step (5) to the Spark cluster through the spark-submit tool;

[0013] (7) The Spark cluster distributes all computing tasks submitted by the master node to each node in the cluster;

[0014] (8) Each node in the Spark cluster analyzes the received computing tasks to obtain the computing information therein, and inputs the computing information into the trained convolutional neural network to obtain the predicted computing results in the geodynamic evolution process (such as temperature, pressure, position, composition and stress-related viscosity, etc.), and sends the predicted computing results to the master node;

[0015] (9) The master node aggregates the calculation results from all nodes in the Spark cluster to obtain the calculation output results.

[0016] (10) The master node sets i=i+1 and returns to step (3).

[0017] (11) The master node stores all the calculation results obtained in the output file, and the process ends.

[0018] Preferably, the geodynamic process to be simulated in step (1) is a plate tectonic evolution process including mantle convection, seafloor spreading and continental drift, and the original data set is from CitcomS user manual data or Web data.

[0019] Preferably, the process of packaging the CitcomS source code files and data into a Java application through JNI in step (4) includes the following sub-steps:

[0020] (4-1) Obtain CitcomS source code file;

[0021] (4-2) modifying the makefile file in the CitcomS source code file obtained in step (4-1) to obtain a modified makefile file;

[0022] (4-3) Use the modified makefile file in (4-2) to compile the CitcomS source code file to obtain the dynamic library file libCitcomS.a, which can be called by other languages;

[0023] (4-4) Using the Java program interface to call the local C program according to the JNI tool, and using the C program to link the dynamic library file obtained in step (4-3) to obtain a Java source program file and a C source program file respectively;

[0024] (4-5) Compiling the Java source program file and the C source program file obtained in (4-4), and packaging the compiled Java code and the i-th data to obtain an application package that can be used for Spark to submit tasks;

[0025] Preferably, step (4-2) is to modify the CFLAGS parameter in the makefile file in the CitcomS source code file to -g-02-fPIC to obtain a modified makefile file that can be used for compilation.

[0026] Preferably, step (5) includes the following sub-steps:

[0027] (5-1) The master node obtains the data flow of the computing tasks in the application package and constructs a directed acyclic graph (DAG) based on the data flow;

[0028] (5-2) The master node obtains all Map tasks in all computing tasks in the application package based on the DAG obtained in step (5-1), which are multiple fine-grained computing tasks;

[0029] Preferably, the CNN model is trained by the following steps:

[0030] (8-1) Collect the input information and output results of the complex calculation process of geodynamic simulation. All the input information and output results form the original data set for CNN model training. The original data set is divided into training set and test set at an 8:2 ratio.

[0031] (8-2) performing filtering and denoising on the training set obtained in step (8-1) to obtain a processed training set;

[0032] (8-3) performing Min-max normalization on the training set processed in step (8-2) to obtain an input data set, and inputting the input data set as feature values ​​into the CNN convolutional neural network;

[0033] (8-4) Use the back propagation algorithm to update and optimize the weight parameters and bias parameters of each layer in the CNN convolutional neural network to obtain an updated CNN convolutional neural network;

[0034] (8-5) Iteratively training the CNN convolutional neural network updated in step (8-4) until the loss function of the CNN convolutional neural network reaches a minimum;

[0035] (8-6) Use the test set in the data set obtained in step (8-1) to iteratively verify the iteratively trained CNN convolutional neural network until the obtained prediction accuracy reaches the optimal level, thereby obtaining a trained CNN convolutional neural network.

[0036] Preferably, the following formula is used to normalize the input data set in step (8-3):

[0037]

[0038] Where n represents the total number of parameters in the input data set, x i represents the i-th parameter value in the input data set, x j Represents the jth parameter value in the input data set, y i It represents the i-th parameter value obtained after normalization. The value range of i and j is [1, n]. min and max represent the minimum and maximum values ​​respectively.

[0039] Preferably, the initial value of the weight parameter in step (8-4) is a random value output by a truncated normal distribution with a standard deviation of 0.1, and the initial value of the bias parameter is set to 0;

[0040] The back-propagation algorithm uses Adam-Optimizer as the optimizer.

[0041] Preferably, the loss function of CNN in step (8-5) is the mean absolute error MAE, which is equal to:

[0042]

[0043] Where m is the total number of complex calculations corresponding to the input data set, z i and x i They represent the predicted result and actual calculation result value of the i-th complex calculation respectively.

[0044] According to another aspect of the present invention, a CitcomS parallel computing system based on the fusion of Spark and AI is provided, which is applied in a computing environment including a master node and a Spark cluster. The CitcomS parallel computing system includes:

[0045] The first module is set at the main node and is used to collect the original data set required for the geodynamic evolution process to be simulated, and use the random data expansion method to expand the original data set to obtain the expanded original data set.

[0046] The second module is provided at the master node and is used to set the counter i=1;

[0047] The third module is set in the main node and is used to determine whether i is larger than the size of the original data set collected by the first module. If so, it enters the eleventh module, otherwise it enters the fourth module;

[0048] The fourth module is set in the main node and is used to package the CitcomS source code file and the i-th data in the original data set into a Java application through the Java native interface JNI, which is an application package that can be called across languages.

[0049] A fifth module, which is provided at the master node and is used to perform fine-grained task division on the application package obtained by the fourth module to obtain a plurality of computing tasks with different fine-grainedness;

[0050] The sixth module is set on the master node and is used to submit all computing tasks obtained in the fifth module to the Spark cluster through the spark-submit tool;

[0051] The seventh module is set in the Spark cluster and is used to distribute all computing tasks submitted by the master node to each node in the cluster;

[0052] The eighth module is set in each node of the Spark cluster and is used to analyze the received computing tasks to obtain the computing information therein, and input the computing information into the trained convolutional neural network to obtain the predicted computing results in the geodynamic evolution process (such as temperature, pressure, position, composition and stress-related viscosity, etc.), and send the predicted computing results to the master node;

[0053] The ninth module is set on the master node and is used to aggregate the calculation results from all nodes in the Spark cluster to obtain the calculation output results.

[0054] The tenth module is set at the main node, and is used to set i=i+1 and return to the third module.

[0055] The eleventh module is set in the main node and is used to store all the obtained calculation results into the output file, and the process ends.

[0056] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0057] 1. Since the present invention adopts step (1), the ability of CitcomS to simulate complex geological evolution scenarios is greatly expanded by expanding the original data set, making the CitcomS geological evolution simulation process more consistent with the objective real world;

[0058] 2. Since the present invention adopts steps (4) to (7), first, it expands the parallel processing capabilities of CitcomS through Spark. Compared with the fixed number of available computing cores of native CitcomS, the heterogeneous mode based on CPU + accelerator in the Spark cluster can significantly improve the computing performance of the software system. Second, due to the unique advantages of the Spark cluster, in terms of large-scale simulation calculations and distributed storage, the present invention adopts Spark's own large-scale parallel distributed grid storage technology, and ensures the constant and balanced distribution of data through a fine-grained data distribution algorithm. It not only improves the reliability, availability and access efficiency of large-scale geological simulation application systems, but also is easy to expand. Third, due to the good communication mechanism between Spark cluster nodes, CitcomS can also realize the scientific discovery of correlation analysis and statistical laws of data.

[0059] 3. Since the present invention adopts step (8), it replaces some complex physical calculations in geological simulation evolution engineering with AI, which greatly improves the calculation speed and reduces some redundant calculations in the intermediate similar processes, thereby reducing the complexity of model calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a schematic diagram of the application environment of the CitcomS parallel computing method based on the fusion of Spark and AI in the present invention;

[0061] Figure 2 It is a schematic diagram of the training process of the CNN model of the present invention;

[0062] Figure 3 This is a flowchart of the CitcomS parallel computing method based on the fusion of Spark and AI in the present invention. DETAILED DESCRIPTION

[0063] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0064] Aiming at the existing earth evolution simulation tool CitcomS, this paper proposes a method to expand its parallel processing capability by using the big data processing framework Spark to realize the distributed operation of CitcomS calculations. At the same time, in view of the complexity of the calculation of its momentum and energy equations and the partial iterativeness of the calculation results, an AI simulation calculation incremental iterative optimization method is used to save redundant calculations of similar processes, and machine learning fitting is used to replace some intermediate calculation processes to achieve the purpose of accelerating calculations.

[0065] This paper proposes a CitcomS parallel computing method based on the fusion of Spark and AI. This method first expands the original CitcomS computing data set, then applies the Spark big data computing framework to CitcomS to enhance its parallel computing capabilities. Furthermore, based on the Tianhe-3 heterogeneous accelerator architecture, a large-scale numerical simulation parallel computing framework and collaborative parallel computing mode for dynamic processes and geological system evolution are constructed for each computing node. At the same time, a machine learning method, Convolutional Neural Network (CNN), is used to construct a machine learning model for some of the complex dynamic and thermodynamic calculation processes. An input-output mapping relationship is established to replace some of the complex and redundant calculation processes, thereby greatly reducing the amount of computation and improving node computing efficiency.

[0066] like Figure 1 and Figure 3 As shown, the present invention provides a CitcomS parallel computing method based on the fusion of Spark and AI, which is applied in a computing environment including a master node and a Spark cluster. The method includes the following steps:

[0067] (1) The master node collects the original data set required for the geodynamic evolution process to be simulated, and uses the random data expansion method to expand the original data set to obtain the expanded original data set.

[0068] Specifically, the geodynamic processes to be simulated in this step can include plate tectonic evolution processes such as mantle convection, seafloor spreading, and continental drift. The original data set comes from the CitcomS user manual data or Web data.

[0069] The advantage of this step is that after expanding the data set, it solves the problem that due to the large number of influencing factors and complex causal mechanisms of the Tethys evolution process, the original input parameters for different Tethys geothermodynamic and dynamic simulation processes are small and cannot accurately simulate and reflect the objective real world.

[0070] (2) The master node sets the counter i=1;

[0071] (3) The master node determines whether i is larger than the size of the original data set collected in step (1). If so, it proceeds to step (11); otherwise, it proceeds to step (4);

[0072] (4) The master node packages the CitcomS source code file and the i-th data in the original dataset into a Java application through the Java Native Interface (JNI), which is an application package that can be called across languages.

[0073] In this step, the process of packaging the CitcomS source code files and data into a Java application through JNI includes the following sub-steps:

[0074] (4-1) Obtain CitcomS source code file;

[0075] (4-2) modifying the makefile file in the CitcomS source code file obtained in step (4-1) to obtain a modified makefile file;

[0076] Specifically, this step is to modify the CFLAGS parameter in the makefile file in the CitcomS source code file to -g-02-fPIC to obtain a modified makefile file that can be used for compilation;

[0077] (4-3) Use the modified makefile file in (4-2) to compile the CitcomS source code file to obtain the dynamic library file libCitcomS.a, which can be called by other languages;

[0078] (4-4) Using the Java program interface to call the local C program according to the JNI tool, and using the C program to link the dynamic library file obtained in step (4-3) to obtain a Java source program file and a C source program file respectively;

[0079] (4-5) Compiling the Java source program file and the C source program file obtained in (4-4), and packaging the compiled Java code and the i-th data to obtain an application package that can be used for Spark to submit tasks;

[0080] The advantage of step (4-2) to step (4-5) is that JNI enables Spark to perform CitcomS calculations just like calling local jobs, and compared with the fixed number of available cores of the CitcomS source program, Spark cluster computing is more flexible and can automatically allocate computing resources according to the task volume, thereby greatly improving the concurrency and throughput of CitcomS calculations.

[0081] (5) The master node divides the application package obtained in step (4) into fine-grained tasks to obtain multiple computing tasks with different fine-grainedness;

[0082] Specifically, this step includes the following sub-steps:

[0083] (5-1) The master node obtains the data flow of the computing tasks in the application package and constructs a directed acyclic graph (DAG) based on the data flow;

[0084] (5-2) The master node obtains all Map tasks in all computing tasks in the application package based on the DAG obtained in step (5-1), which are multiple fine-grained computing tasks;

[0085] (6) The master node submits all computing tasks obtained in step (5) to the Spark cluster through the spark-submit tool;

[0086] (7) The Spark cluster distributes all computing tasks submitted by the master node to each node in the cluster by executing the command foreachPartition(SparkNode.start(map));

[0087] The advantage of step (7) is that, due to the use of the Spark parallel computing framework, users can flexibly set the number of nodes according to the complexity of the computing job, and when the number of clusters cannot meet the calculation, Spark's distributed storage and computing capabilities can still effectively cope with large-scale simulation calculations, and based on the communication mechanism between Spark cluster nodes, it can realize the scientific discovery of data correlation analysis and data statistical laws;

[0088] (8) Each node in the Spark cluster analyzes the received computing tasks to obtain the computing information therein, and inputs the computing information into the trained convolutional neural network to obtain the predicted computing results in the geodynamic evolution process (such as temperature, pressure, position, composition and stress-related viscosity, etc.), and sends the predicted computing results to the master node;

[0089] The advantage of step (8) is that it uses CNN to construct the input-output mapping relationship for the computing task, which accelerates the complex computing process and eliminates a large number of complex redundant calculations, thereby greatly reducing the computational complexity of each node and improving the computing efficiency.

[0090] (9) The master node aggregates the calculation results from all nodes in the Spark cluster to obtain the calculation output results.

[0091] (10) The master node sets i=i+1 and returns to step (3).

[0092] (11) The master node stores all the calculation results obtained in the output file, and the process ends.

[0093] Specifically, if Figure 2As shown in Figure 1, the CNN model in step (8) above is trained by the following steps:

[0094] (8-1) Collect the input information and output results of the complex calculation process of geodynamic simulation. All the input information and output results form the original data set for CNN model training. The original data set is divided into training set and test set at an 8:2 ratio.

[0095] Specifically, the present invention needs to find the input information of complex calculations such as the Stokes equations and the momentum conservation equations in the geodynamic simulation process, as well as the data set consisting of the output results after these complex calculations.

[0096] In this embodiment, a total of 5094 complex calculation data are collected as a data set, and are divided into a training set and a test set in a ratio of 8:2.

[0097] (8-2) performing filtering and denoising on the training set obtained in step (8-1) to obtain a processed training set;

[0098] Specifically, the filtering and denoising process uses the median filtering method and the mean filtering method in sequence. First, the value of a point in a series of collected original parameters is replaced by the median value of the values ​​of each point in a neighborhood of the point, in order to remove possible mutation points in the collected data. Then, the median filtering data is smoothed using the mean filtering method, in order to remove irrelevant data points in the parameter data set.

[0099] (8-3) performing Min-max normalization on the training set processed in step (8-2) to obtain an input data set, and inputting the input data set as feature values ​​into the CNN convolutional neural network;

[0100] Specifically, the purpose of normalizing the input data set is to make the features of different dimensions have a certain correlation in value, thereby improving the reliability of the prediction results. The normalization formula used is:

[0101]

[0102] Where n represents the total number of parameters in the input data set, x i represents the i-th parameter value in the input data set, x j Represents the jth parameter value in the input data set, y i It represents the i-th parameter value obtained after normalization. The value range of i and j is [1, n]. min and max represent the minimum and maximum values ​​respectively.

[0103] (8-4) Use the back propagation algorithm to update and optimize the weight parameters and bias parameters of each layer in the CNN convolutional neural network to obtain an updated CNN convolutional neural network;

[0104] Specifically, the initial value of the weight parameter is a random value output by a truncated normal distribution with a standard deviation of 0.1, and the initial value of the bias parameter is set to 0;

[0105] Specifically, in order to speed up the decrease of the loss value, the backpropagation algorithm in this step uses Adam-Optimizer as the optimizer. This is because the optimizer has the advantages of momentum and adaptive learning rate.

[0106] (8-5) Iteratively training the CNN convolutional neural network updated in step (8-4) until the loss function of the CNN convolutional neural network reaches a minimum;

[0107] Specifically, the loss function of CNN is the mean absolute error (MAE), which is equal to:

[0108]

[0109] Where m is the total number of complex calculations corresponding to the input data set, z i and x i They represent the predicted result and actual calculation result value of the i-th complex calculation respectively.

[0110] (8-6) Use the test set in the data set obtained in step (8-1) to iteratively verify the iteratively trained CNN convolutional neural network until the obtained prediction accuracy reaches the optimal level, thereby obtaining a trained CNN convolutional neural network.

[0111] In summary, through the above description of the present invention, the main advantages of the present invention include:

[0112] 1. The expansion of the original data set has greatly expanded CitcomS's ability to simulate complex geological evolution scenarios, making the CitcomS geological evolution simulation process more consistent with the objective real world.

[0113] 2. The Spark cluster greatly expands CitcomS's parallel processing capabilities and job throughput, and can more realistically and quickly restore geological evolution processes in large-scale simulation calculations, providing a more efficient technical solution for geodynamic simulation.

[0114] 3. By replacing some complex and redundant dynamic and thermodynamic calculation processes with the machine learning method CNN, the amount of calculation is greatly reduced, the node computing efficiency is improved, and the parallel processing capability is further expanded, thus accelerating the simulation process of the entire geological evolution.

[0115] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A CitcomS parallel computing method based on the fusion of Spark and AI is applied in a computing environment including a master node and a Spark cluster, characterized in that: The CitcomS parallel computing method comprises the following steps: (1) The master node collects the original data set required for the geodynamic evolution process to be simulated, and uses the random data expansion method to expand the original data set to obtain the expanded original data set; (2) The master node sets the counter i=1; (3) The master node determines whether i is larger than the size of the original data set collected in step (1). If so, it proceeds to step (11); otherwise, it proceeds to step (4); (4) The master node packages the CitcomS source code file and the i-th data in the original dataset into a Java application through the Java native interface JNI, which is an application package that can be called across languages; (5) The master node divides the application package obtained in step (4) into fine-grained tasks to obtain multiple computing tasks with different fine-grainedness; (6) The master node submits all computing tasks obtained in step (5) to the Spark cluster through the spark-submit tool; (7) The Spark cluster distributes all computing tasks submitted by the master node to each node in the cluster; (8) Each node in the Spark cluster analyzes the received computing tasks to obtain the computing information therein, and inputs the computing information into the trained convolutional neural network to obtain the predicted computing results in the geodynamic evolution process, and sends the predicted computing results to the master node; (9) The master node aggregates the calculation results from all nodes in the Spark cluster to obtain the calculation output results; (10) The master node sets i=i+1 and returns to step (3); (11) The master node stores all the calculation results obtained in the output file, and the process ends.

2. The CitcomS parallel computing method based on the fusion of Spark and AI according to claim 1 is characterized in that: The geodynamic process to be simulated in step (1) is the plate tectonic evolution process including mantle convection, seafloor spreading and continental drift. The original data set is from the CitcomS user manual data or Web data.

3. The CitcomS parallel computing method based on the fusion of Spark and AI according to claim 1 or 2, characterized in that: The process of packaging the CitcomS source code files and data into a Java application through JNI in step (4) includes the following sub-steps: (4-1) Obtain CitcomS source code file; (4-2) modifying the makefile file in the CitcomS source code file obtained in step (4-1) to obtain a modified makefile file; (4-3) Use the modified makefile file in (4-2) to compile the CitcomS source code file to obtain the dynamic library file libCitcomS.a, which can be called by other languages; (4-4) Using the Java program interface to call the local C program according to the JNI tool, and using the C program to link the dynamic library file obtained in step (4-3) to obtain a Java source program file and a C source program file respectively; (4-5) Compile the Java source program file and the C source program file obtained in (4-4), and package the compiled Java code and the i-th data to obtain an application package that can be used to submit tasks to Spark.

4. The CitcomS parallel computing method based on the fusion of Spark and AI according to claim 3 is characterized in that: Step (4-2) is to modify the CFLAGS parameter in the makefile file in the CitcomS source code file to -g-02-fPIC to obtain a modified makefile file that can be used for compilation.

5. The CitcomS parallel computing method based on the fusion of Spark and AI according to claim 4 is characterized in that: Step (5) includes the following sub-steps: (5-1) The master node obtains the data flow of the computing tasks in the application package and constructs a directed acyclic graph (DAG) based on the data flow; (5-2) The master node obtains all Map tasks in all computing tasks in the application package based on the DAG obtained in step (5-1), which are multiple fine-grained computing tasks.

6. The CitcomS parallel computing method based on the fusion of Spark and AI according to claim 1 is characterized in that: The CNN model is trained through the following steps: (8-1) Collecting input information and output results from the complex computational process of geodynamic simulation. All input information and output results form the original data set for CNN model training. The original data set is divided into training set and test set at an 8:2 ratio. (8-2) performing filtering and denoising on the training set obtained in step (8-1) to obtain a processed training set; (8-3) performing Min-max normalization on the training set processed in step (8-2) to obtain an input data set, and inputting the input data set as feature values ​​into the CNN convolutional neural network; (8-4) Use the back propagation algorithm to update and optimize the weight parameters and bias parameters of each layer in the CNN convolutional neural network to obtain an updated CNN convolutional neural network; (8-5) Iteratively training the CNN convolutional neural network updated in step (8-4) until the loss function of the CNN convolutional neural network reaches a minimum; (8-6) Use the test set in the data set obtained in step (8-1) to iteratively verify the iteratively trained CNN convolutional neural network until the obtained prediction accuracy reaches the optimal level, thereby obtaining a trained CNN convolutional neural network.

7. The CitcomS parallel computing method based on the fusion of Spark and AI according to claim 6 is characterized in that: In step (8-3), the input data set is normalized using the following formula: Where n represents the total number of parameters in the input data set, x i represents the i-th parameter value in the input data set, x j Represents the jth parameter value in the input data set, y i It represents the i-th parameter value obtained after normalization. The value range of i and j is [1, n]. min and max represent the minimum and maximum values ​​respectively.

8. The CitcomS parallel computing method based on the fusion of Spark and AI according to claim 7 is characterized in that: The initial value of the weight parameter in step (8-4) is a random value output by a truncated normal distribution with a standard deviation of 0.1, and the initial value of the bias parameter is set to 0; The back-propagation algorithm uses Adam-Optimizer as the optimizer.

9. The CitcomS parallel computing method based on the fusion of Spark and AI according to claim 8 is characterized in that: The loss function of CNN in step (8-5) is the mean absolute error MAE, which is equal to: Where m is the total number of complex calculations corresponding to the input data set, z i and x i They represent the predicted result and actual calculation result value of the i-th complex calculation respectively.

10. A CitcomS parallel computing system based on the fusion of Spark and AI is applied in a computing environment including a master node and a Spark cluster, characterized in that: The CitcomS parallel computing system includes: The first module is set at the master node and is used to collect the original data set required for the geodynamic evolution process to be simulated, and expand the original data set using the random data expansion method to obtain the expanded original data set; The second module is provided at the master node and is used to set the counter i=1; The third module is set in the main node and is used to determine whether i is larger than the size of the original data set collected by the first module. If so, it enters the eleventh module, otherwise it enters the fourth module; The fourth module is set in the master node and is used to package the CitcomS source code file and the i-th data in the original data set into a Java application through the Java native interface JNI, which is an application package that can be called across languages; A fifth module, which is provided at the master node and is used to perform fine-grained task division on the application package obtained by the fourth module to obtain a plurality of computing tasks with different fine-grainedness; The sixth module is set on the master node and is used to submit all computing tasks obtained in the fifth module to the Spark cluster through the spark-submit tool; The seventh module is set in the Spark cluster and is used to distribute all computing tasks submitted by the master node to each node in the cluster; The eighth module is set in each node of the Spark cluster and is used to analyze the received computing tasks to obtain the computing information therein, and input the computing information into the trained convolutional neural network to obtain the predicted computing results in the geodynamic evolution process, and send the predicted computing results to the master node; The ninth module is set on the master node and is used to aggregate the calculation results from all nodes in the Spark cluster to obtain the calculation output results; a tenth module, which is provided at the master node, and is used to set i=i+1 and return to the third module; The eleventh module is set in the main node and is used to store all the obtained calculation results into the output file, and the process ends.

Images