Multi-technology fusion field application service platform

Abstract

The invention provides a multi-technology fusion field application service platform, which comprises a component model library and a workflow engine system, and is characterized in that the workflow engine system can determine calculation function information of each component model according to a description layer in a model file of each component model, and display the calculation function information in a model combination page; the workflow engine system responds to a combination request submitted by the user on the model combination page, determines each specified target model and a model execution sequence in the combination request, and further performs business flow calculation according to the model execution sequence; and calling the model file of each target model in the component model library in sequence, executing the corresponding calculation task through a tool layer in the model file, and executing a pre-processing task or a post-processing task of the calculation task through a customization layer in the model file to realize automatic scheduling of each calculation task in the business flow. And the business processing efficiency and accuracy are improved.

Description

Technical Field

[0001] This application relates to the technical field of supercomputing or intelligent computing application service platforms, and particularly relates to a domain application service platform integrating multiple technologies. Background Art

[0002] Currently, in many application service fields of supercomputing or intelligent computing, application service platforms have been independently constructed according to their own scenario requirements in each application field.

[0003] However, these application service platforms cannot integrate the computing tools within the platform and automatically control the computing business processes, and users need to manually execute each computing task. Summary of the Invention

[0004] In view of the above defects or deficiencies in the prior art, this application aims to provide a domain application service platform integrating multiple technologies to solve problems such as the inability to integrate computing tools and automatically control business processes in related technologies, and to achieve automatic scheduling of each computing task.

[0005] An embodiment of this application provides a domain application service platform integrating multiple technologies. The domain application service platform includes a component model library and a workflow engine system. The component model library stores model files of each component model. The workflow engine system is used for:

[0006] Based on the description layer in the model files of each component model, determine the computing function information of each component model, and display the computing function information on the model combination page;

[0007] In response to a combination request submitted by the user on the model combination page, determine each target model specified in the combination request and the model execution order;

[0008] According to the model execution order, sequentially call the model files of each target model in the component model library, execute the corresponding computing tasks through the tool layer in the model files, and execute the pre-processing tasks or post-processing tasks of the computing tasks through the customization layer in the model files.

[0009] Optionally, the domain application service platform is connected to a cloud computing platform and a high-performance computing platform. The high-performance computing platform is a supercomputing platform or an intelligent computing platform. The workflow engine system is further used for:

[0010] Determine resource requirement information through the tool layer in the model files, and generate a resource scheduling script according to the resource requirement information through the customization layer in the model files;

[0011] Schedule the computing tasks to the cloud computing platform or the high-performance computing platform for execution based on the resource scheduling script.

[0012] Optionally, the domain application service platform further includes a data conversion middleware and a conversion tool library. The data conversion middleware is used for:

[0013] During the process of the workflow engine system executing each of the computing tasks, in response to detecting that the current computing task has been completed, determining whether format conversion is required between the current computing task and the next computing task;

[0014] If so, taking the data format output by the current computing task as the source data format and the data format input by the next computing task as the target data format;

[0015] Invoking the conversion tool in the conversion tool library to convert the data output by the current computing task from the source data format to the target data format.

[0016] Optionally, the data conversion middleware is further used for:

[0017] Based on the conversion rules corresponding to each conversion tool in the conversion tool library, determining whether there is a conversion tool in the conversion tool library that can convert the source data format to the target data format;

[0018] If not, determining at least one conversion path based on the conversion rules corresponding to each conversion tool, the source data format, and the target data format;

[0019] Invoking the conversion tool in the conversion tool library based on the conversion path.

[0020] Optionally, the data conversion middleware is further used for:

[0021] For each of the conversion paths, determining the conversion accuracy loss and conversion required time of each sub-path in the conversion path, and based on the conversion accuracy loss and conversion required time of each sub-path, determining the total loss and total required time of the conversion path;

[0022] Eliminating the conversion paths with a total loss greater than the preset loss acceptance degree, and determining the final path according to the total loss and total required time of the remaining conversion paths;

[0023] In the conversion tool library, invoking the conversion tools corresponding to each sub-path in the final path.

[0024] Optionally, the data conversion middleware is further used for:

[0025] Based on the input and output ports in the model file corresponding to the current computing task, determining the data format output by the current computing task;

[0026] Determine the data format of the input data for the next computing task based on the input and output ports in the model file corresponding to the next computing task;

[0027] If the data format output by the current computing task is different from the data format of the input data for the next computing task, it is determined that format conversion is required between the current computing task and the next computing task.

[0028] Optionally, the conversion tool is used for:

[0029] Parse the data output by the current computing task to obtain a source memory object;

[0030] Extract target data items from the source memory object and save the target data items as intermediate objects, where the target data items are the data items required for the execution of the next computing task;

[0031] Assemble the intermediate objects according to the next computing task to obtain a target memory object;

[0032] Format the data of the target memory object to obtain data that conforms to the target data format.

[0033] Optionally, the domain application service platform further includes a task monitoring module, and the task monitoring module is used for:

[0034] During the process of the workflow engine system executing each computing task, determine the task status of each computing task based on the status layer in the model file of each target model, and display the task status of each computing task.

[0035] Optionally, the domain application service platform further includes a shared storage system, and the shared storage system connects the cloud computing platform and the high-performance computing platform. The shared storage system is used for:

[0036] In response to a data storage request from the cloud computing platform or the high-performance computing platform, store the data generated by the cloud computing platform or the high-performance computing platform as shared data;

[0037] In response to a data reading request from the cloud computing platform or the high-performance computing platform, provide the shared data for the cloud computing platform or the high-performance computing platform.

[0038] Optionally, the domain application service platform further includes a remote visualization module, and the remote visualization module connects the user terminal, the cloud computing platform and the high-performance computing platform. The remote visualization module is used for:

[0039] In response to receiving a visualization request from a client, determine the target application in the visualization request;

[0040] Start the target application in the cloud computing platform or the high-performance computing platform, optimize the operation data of the target application, and feedback the optimized operation data to the client for display.

[0041] In summary, the present application proposes a domain application service platform integrating multiple technologies. The platform includes a component model library and a workflow engine system. The workflow engine system can determine the computing function information of each component model according to the description layer in the model file of each component model, and display the computing function information on the model combination page, so that users can combine multiple component models according to actual business needs for business process calculation. In response to a combination request submitted by the user on the model combination page, the workflow engine system determines the specified target models and the model execution order among them, and then sequentially calls the model files of each target model in the component model library according to the model execution order, executes the corresponding computing tasks through the tool layer in the model file, and, through the customization layer in the model file, executes the pre-processing task or post-processing task of the computing task, completes the business process calculation, realizes the automatic scheduling of each computing task in the business process, achieves the purpose of automatic control of the business process, improves the business processing efficiency and accuracy, and, the corresponding computing function information can be displayed through the description layer of the component model, and the computing task can be executed through the tool layer of the component model, which can be applied to different application fields, providing higher flexibility for the business process. In addition, by executing the pre-processing task or post-processing task through the customization layer of the component model, the personalized configuration of specific problems can be solved, further improving the flexibility of business process processing. Description of the Drawings

[0042] In order to more clearly illustrate the specific embodiments of the present application or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0043] Figure 1 is a flowchart of a business process processing provided by an embodiment of the present application;

[0044] Figure 2 is a schematic framework diagram of a component model provided by an embodiment of the present application;

[0045] Figure 3 is a schematic data conversion diagram provided by an embodiment of the present application;

[0046] Figure 4It is a schematic diagram of a conversion path provided by an embodiment of the present application;

[0047] Figure 5 It is a schematic diagram of a final path provided by an embodiment of the present application;

[0048] Figure 6 It is a schematic diagram of the process of data conversion provided by an embodiment of the present application;

[0049] Figure 7 It is a schematic diagram of the process of data conversion by a conversion tool provided by an embodiment of the present application. Detailed implementation manners

[0050] The present application will be further described in detail below with reference to the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention, rather than limiting the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the invention are shown in the drawings.

[0051] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0052] As mentioned in the background art, in view of the problems in the prior art, the present application proposes a field application service platform for multi-technology integration. The field application service platform includes a component model library and a workflow engine system, and the component model library stores model files of each component model. Among them, the workflow engine system in the field application service platform can perform business process processing through each component model combined by the user. Figure 1 It is a flowchart of a business process processing provided by an embodiment of the present application. As Figure 1 shown, the workflow engine system is used to perform the following steps:

[0053] S110. Based on the description layer in the model files of each component model, determine the calculation function information of each component model, and display the calculation function information on the model combination page.

[0054] Among them, the component model can be a calculation tool for completing a specific calculation task. Specifically, based on the program operation characteristics modeling of the calculation tool, a component model suitable for calculations in different application fields can be constructed. The component model can reflect the usage process of the calculation tool, including input, execution, and output. Therefore, a component model can be constructed based on this framework.

[0055] In the embodiments of the present application, the operation characteristics of a computing tool can be encapsulated through a four-layer structure and input / output ports with the "input-execution-output" process as the framework, realizing a hierarchical abstraction mapping from problems to calculation examples and the traceability of results. The component model includes a description layer (interaction interface), a tool layer (core operation characteristics), a customization layer (domain adaptation logic), a status layer (execution status monitoring), and input / output ports (data interaction).

[0056] Figure 2 is a schematic diagram of the framework of a component model provided by the embodiments of the present application. As Figure 2 shown, the component model can include a description layer, a tool layer, a customization layer, a status layer, and input / output ports. The input / output ports can include an input data port and an output data port. The input data port is used to input data into the component model, and the output data port is used to output the data generated by the component model when executing a calculation task.

[0057] Among them, the description layer can be used by the front-end interaction tool, mainly providing the calculation function information of the component model (including an introduction to the function and usage method), facilitating users to understand the basic information of the encapsulated component model; the tool layer can describe information such as the resource requirements and interaction modes of the computing tool, including the types of resources used, operation parameters, etc.; the customization layer can describe the specific functions of the computing tool in the process, including different functions of the same computing tool or different computing functions of different computing tools; the status layer can describe the execution status of the computing tool, including various statuses such as running, suspended, failed, and completed. Through the encapsulation of the above levels, the component model can complete the hierarchical abstraction mapping from problems to calculation examples, ensuring that the calculation results generated by the component model can be traced back to the calculation method and the problem.

[0058] In the embodiments of the present application, the tool layer in the component model is the core part of the encapsulation of the computing tool, that is, the general operation characteristics of the encapsulated computing tool, and it has the strongest reusability in the component model. To support the process integration of different feature simulation software on supercomputing resources, the tool layer can be further refined to characterize the operation characteristics of numerical simulation software from multiple aspects such as interaction mode, resource requirements, and execution method to support multi-level flexible configuration.

[0059] In addition, the customization layer in the component model includes a configurable computing environment. Based on this environment, the automatic generation of scheduling scripts can be realized, so as to be able to interface with the actual high-performance computing resource environment. The customization layer can also define customization scripts before and after the computing task, which are used to map the heterogeneous input and output between the two component models before and after. Users can embed the parsing and parameter extraction functions of non-standard and self-made formats in it to realize the process integration of self-made software and other open-source software. They can also define domain-related data fault tolerance methods in the post-customization script. By parsing the output of the computing task, it is judged whether the computing task is successfully executed, thus providing higher flexibility for process integration.

[0060] Exemplarily, the domain application service platform further includes a component encapsulation module, and the component encapsulation module is used for:

[0061] Obtain the component information fed back by the user on the encapsulation form page. The component information includes description layer information, tool layer information, customization layer information, status layer information and input / output information; construct a component model including a description layer, a tool layer, a customization layer, a status layer and input / output ports based on the component information, obtain the model file corresponding to the component model, and store the model file in the component model library.

[0062] Among them, the component encapsulation module can respond to the user's component encapsulation request and display the encapsulation form page for the user. This encapsulation form page can support the user to enter the information of each layer in the component model through the form.

[0063] Specifically, the user can trigger the submit button after completing the entry of the component information. Then, the component encapsulation module can receive the component information fed back by the user on the encapsulation form page. This component information can be the information of each layer entered by the user in the form, including description layer information, tool layer information, customization layer information, status layer information and input / output information.

[0064] Among them, the description layer information can be the basic meta-information of the component model, which is called by interactive tools (such as the model combination page). The description layer information can include the function brief introduction, usage method and related knowledge of the component model. The function brief introduction can describe the core computing capabilities of the component model, such as "finite element structural mechanics analysis" and "molecular dynamics simulation"; the usage method can be understood as the standardized call description of the component model, including the input parameter format, output data type, typical application scenarios, etc.; the related knowledge can describe the calculation methods (such as "finite difference method" and "finite element method") and problem types (such as "flow field calculation" and "electromagnetic simulation") mapped by the component model to ensure the traceability of the results.

[0065] Among them, the tool layer information can describe the general running characteristics of the component model and is the core reusable layer of the component model. The tool layer information can be obtained by abstracting the general running characteristics of computing tools. In addition to the general running characteristics, the tool layer information can also include interaction modes, API (Application Programming Interface) information, GUI (Graphical User Interface) interaction information, resource requirement information, memory information (or storage information), software dependency information, running modes, fault tolerance mechanisms, and output parsing. The interaction mode can include the parameter format in the command line and the execution command template (such as mpirun -n8. / program input.dat); the API information can include the call protocol (such as RESTful or remote procedure call) and the data transfer format (JSON, binary, etc.); the GUI interaction information can include the automated script interface (such as operating the graphical interface through PyAutoGUI); the resource requirement information can be computing power requirement information, including the number of CPU cores, GPU models, the number of GPUs, heterogeneous acceleration requirements, etc.; the memory information can include the minimum running memory, and the storage information can include the specification of the input / output data storage path; the software dependency information can describe the running environment on which the component model depends, including the operating system, compiler, library version, etc., such as "GCC9.3+", "CUDA 11.2"; the running mode can be serial, parallel (MPI / OpenMP), or container (Docker / Singularity); the fault tolerance mechanism can include retry policies, checkpoint recovery (Checkpointing) configurations, etc.; the output parsing can include the standard output parsing rules or error log parsing rules, such as determining the running failure by the keyword "ERROR".

[0066] Among them, the customization layer information can describe the adaptation and process integration logic of the component model in the set domain, be used for the personalized configuration to solve specific problems, and support heterogeneous tool integration and computing environment adaptation. The customization layer information can include environment variable configurations, customized pre-scripts, and customized post-scripts. The environment variable configuration can describe the injected variable parameters (such as "LD_LIBRARY_PATH", "PYTHONPATH"); the customized pre-script can be a script that needs to be executed before the computing task is executed, such as batch extracting atomic structures from the material database to generate input files; the customized post-script can be a script that needs to be executed after the computing task is executed, such as parsing the output and extracting key metrics such as the band gap and storing them in the database.

[0067] Among them, the status layer information can describe the monitoring logic during the execution of the component model, be used to monitor the task execution status, and automatically retry or adjust parameters for failed computing tasks. It can include statuses such as Running, Suspended, Failed, and Completed. Running means that resources have been allocated and the computing task has started; Suspended means that it is temporarily paused due to insufficient resources or user intervention (breakpoint information can be recorded); Failed means that the computing task has failed, and error logs and failure reasons (such as out-of-memory, incorrect input parameters) can be recorded; Completed means that the computing task has ended normally, and the output file path, execution time consumption, etc. can be marked.

[0068] Among them, the input and output port information can describe the data interaction format of the component model, including the definition of the input port and the output port. The definition of the input port includes the definition of the input data format (such as JSON Schema, XML DTD, etc.), parameter types (scalar, array, file path), and the mandatory rule (or optional rule) of the data. The definition of the output port includes the output data declaration (such as "result file path", "statistical index JSON"), metadata (such as unit, precision, timestamp), etc. The data conversion between the input port and the output port can also be implemented through the custom pre-script or custom post-script in the custom layer.

[0069] Specifically, the component encapsulation module can construct the description layer according to the description layer information, construct the tool layer according to the tool layer information, construct the custom layer according to the custom layer information, construct the status layer according to the status layer information, and construct the input and output ports according to the input and output information to obtain a component model, and store the model file corresponding to the component model in the component model library.

[0070] In this way, users can encapsulate component models according to business requirements, and then can combine the custom component models with other open-source models to implement business process processing. The encapsulated component model can shield complex internal software features and operating environments, and can also cover a wide range of software tools, avoiding repeated development and utilization of the same software. Different computing processes can be formed through flexible combination. The component model can support the automated construction of efficient numerical simulation workflows, and achieve the "plug and play" and process-based collaboration of different computing tools on high-performance computing resources.

[0071] In the embodiment of this application, the component models encapsulated by users can be stored in the component model library in the form of model files. For each component model stored in the component model library, the workflow engine system can determine the computing function information of each component model through the description layer in each model file, and then display the computing function information of each component model on the model combination page.

[0072] Among them, the model combination page can be a page presented by the workflow engine system to the user for multi-component model combination; the user can understand the calculation function information of each component model on the model combination page, select multiple component models for combination on the model combination page, determine the execution order between each component model, and trigger the submission button after completion of the combination, and then the workflow engine system receives the combination request.

[0073] S120. In response to the combination request submitted by the user on the model combination page, determine each target model and the model execution order specified in the combination request.

[0074] Specifically, if the workflow engine system receives the combination request submitted by the user, it can parse the combination request to determine each component model specified by the user, that is, the target model, and the model execution order between each target model.

[0075] It should be noted that the same component model can exist in each target model, that is, the same component model can be selected by the user multiple times to perform multiple calculation tasks in a complete business process.

[0076] In addition to the way that the user manually combines each component model on the model combination page, the user can also first sort out the business process, clarify the calculation task order of each link, and then construct a directed acyclic graph on the model combination page to describe the business process. The nodes in the directed acyclic graph are component models formed by parsing the calculation tool integration interface, and the edges are the data dependency relationships between the component models. Then, the directed acyclic graph can be submitted, and the workflow engine system determines each target model and the model execution order according to the directed acyclic graph in the combination request. In addition, the user can also set the trigger conditions and execution logics of each calculation task during the process of constructing the directed acyclic graph.

[0077] S130. According to the model execution order, sequentially call the model files of each target model in the component model library, execute the corresponding calculation tasks through the tool layer in the model file, and execute the pre-processing task or post-processing task of the calculation task through the customization layer in the model file.

[0078] Specifically, after the workflow engine system determines the target model and the model execution order between each target model, it can sequentially call the model files of each target model in the component model library according to the model execution order, so that each target model sequentially executes the corresponding calculation tasks and customization tasks (pre-processing tasks or post-processing tasks) according to the model execution order to complete the calculation of the entire business process.

[0079] Exemplarily, the target model may include Component Model 1, Component Model 2, and Component Model 3. The model execution order may be Component Model 2 - Component Model 3 - Component Model 1. The workflow engine system may first call the model file of Component Model 2 to execute the calculation task and customization task (pre - processing task or post - processing task) of Component Model 2, then call the model file of Component Model 3 to execute the calculation task and customization task of Component Model 3, and then call the model file of Component Model 1 to execute the calculation task and customization task of Component Model 1.

[0080] It should be noted that if there are trigger conditions and execution logics for the target model in the combined request, the workflow engine system can also call the model file of the target model when detecting that the trigger conditions are met.

[0081] Among them, for each target model, after the workflow engine calls the model file of the target model, it can execute the calculation task corresponding to the target model through the tool layer in the model file, and can execute the pre - processing task or post - processing task of this calculation task through the customization layer in the model file.

[0082] Specifically, the pre - processing task can be a task that needs to be executed before the calculation task, and the post - processing task can be a task that needs to be executed after the calculation task. The pre - processing task and the post - processing task can be defined by the user in the customization layer during the encapsulation process of the component model.

[0083] For example, the pre - processing task can be to pre - process the input data, or extract set features from the input data, or sort the input data according to set dimensions to eliminate some invalid data. The post - processing task can be to organize the output data, or judge whether the calculation task is successfully executed through the output data, or select abnormal data or standard data from the output data according to set rules.

[0084] Through the workflow engine system, the automatic control of the calculation business flow can be realized. For the calculation business flows in different fields, users can use the workflow engine system for modeling. For example, in the material calculation business flow, a series of steps such as material modeling, material parameter setting, material multi - scale (or single - scale calculation), and material calculation result output are defined and arranged through the workflow engine system. When the business flow is started, the workflow engine system can automatically schedule each component model to execute the calculation tasks of each link according to the defined process, realize the automatic control of the calculation business flow, and improve the business processing efficiency and accuracy. Through the workflow engine system, a collaborative tool between specific tasks and other modules can be realized. In actual business operations, the execution situation of the workflow engine system can also be monitored and adjusted to ensure the smooth execution of the business process.

[0085] The domain application service platform provided by the embodiments of the present application includes a component model library and a workflow engine system. The workflow engine system can determine the computing function information of each component model according to the description layer in the model file of each component model, and display the computing function information on the model combination page, so that users can combine multiple component models according to actual business needs for business process calculation. In response to the combination request submitted by the user on the model combination page, the workflow engine system determines each specified target model and the model execution order among them, and then, in accordance with the model execution order, sequentially calls the model files of each target model in the component model library, executes the corresponding computing tasks through the tool layer in the model file, and, through the customization layer in the model file, executes the pre-processing task or post-processing task of the computing task, completes the business process calculation, realizes the automatic scheduling of each computing task in the business process, achieves the purpose of business process automation control, improves the business processing efficiency and accuracy, and, the corresponding computing function information can be displayed through the description layer of the component model, and the computing tasks can be executed through the tool layer of the component model, which can be applied to different application fields, providing higher flexibility for the business process. In addition, by executing the pre-processing task or post-processing task through the customization layer of the component model, the personalized configuration of specific problems can be solved, further improving the flexibility of business process processing.

[0086] In the embodiments of the present application, in addition to being able to customize the pre-processing task and post-processing task of the computing task, the customization layer can also be responsible for the automatic generation of scheduling scripts.

[0087] In one example, the domain application service platform is connected to a cloud computing platform and a high-performance computing platform. The high-performance computing platform is a supercomputing platform or an intelligent computing platform. The workflow engine system is further used for:

[0088] Determine the resource requirement information through the tool layer in the model file, and generate a resource scheduling script according to the resource requirement information through the customization layer in the model file; schedule the computing task to the cloud computing platform or the high-performance computing platform for execution based on the resource scheduling script.

[0089] Among them, the resource requirement information may include the number of CPU cores, GPU model, number of GPUs, heterogeneous acceleration requirements, etc. After the workflow engine system calls the model file of the target model, it can first determine the resource requirement information through the tool layer in the model file, and then, through the customization layer in the model file, generate a resource scheduling script according to the resource requirement information.

[0090] Furthermore, the workflow engine system can send the resource scheduling script to the manager of the supercomputing platform or the intelligent computing platform, and the manager schedules it to the cloud computing platform or the high-performance computing platform to execute the computing task.

[0091] Through the above examples, according to the resources required by the target model, the computing tasks corresponding to the target model can be scheduled to other platforms for execution to ensure the execution efficiency and stability of the computing tasks, thereby ensuring that the entire business process can accurately complete the calculation.

[0092] In the embodiments of the present application, considering that in specific domain applications, there is a large amount of data from different data sources, or data with different formats and different structures, that is, multi-source heterogeneous data, which may result in different data transfer formats between different target models. Therefore, a data conversion middleware can also be set in the domain application service platform to implement data conversion between target models through the data conversion middleware.

[0093] In a specific implementation manner, the domain application service platform further includes a data conversion middleware and a conversion tool library. The data conversion middleware is used to perform the following steps:

[0094] Step 21: During the process of the workflow engine system executing each computing task, in response to detecting that the current computing task has been completed, determine whether format conversion is required between the current computing task and the next computing task;

[0095] Step 22: If so, use the data format output by the current computing task as the source data format, and use the data format input by the next computing task as the target data format;

[0096] Step 23: Invoke the conversion tool in the conversion tool library to convert the data output by the current computing task from the source data format to the target data format.

[0097] Among them, the conversion tool library is used to store each conversion tool, and each conversion tool is used to implement the conversion between one data format and another data format.

[0098] Specifically, in Step 21, the data conversion middleware can detect whether the current computing task has been completed during the process of the workflow engine system sequentially executing each computing task. If it is detected that the current computing task has been completed, determine whether format conversion is required between the current computing task and the next computing task.

[0099] For example, the data conversion middleware can determine whether format conversion is required according to the input and output ports corresponding to the current computing task and the next computing task. In one example, the data conversion middleware is further used for:

[0100] Determine the data format of the output of the current computing task based on the input and output ports in the model file corresponding to the current computing task; determine the data format of the input of the next computing task based on the input and output ports in the model file corresponding to the next computing task; if the data format of the output of the current computing task is different from the data format of the input of the next computing task, then determine that format conversion is required between the current computing task and the next computing task.

[0101] Among them, the data conversion middleware can combine the model file of the target model corresponding to the current computing task to determine the data format of its output, and combine the model file of the target model corresponding to the next computing task to determine the data format of its input. Further, if the two data formats are different, it can be determined that format conversion is required before the next computing task is executed.

[0102] Further, in step 22, if format conversion is required, the data conversion middleware can use the data format of the output of the current computing task as the source data format and the data format of the input of the next computing task as the target data format.

[0103] Further, in step 23, the data conversion middleware can call the conversion tool in the conversion tool library to convert the data output by the current computing task from the source data format to the target data format.

[0104] Figure 3 is a schematic diagram of data conversion provided by an embodiment of the present application. As Figure 3 shown, the data conversion middleware can query the conversion rule library according to the conversion requirements between the source data (the data output by the current computing task) and the target data (the data input by the next computing task). The conversion rule library can record the conversion tools between different data formats in the form of a directed graph, and then call the queried conversion tool in the conversion tool library to convert the source data into the target data, that is, convert the data format of the output of the current computing task into the data format of the input of the next computing task. Among them, the conversion tool library can include multiple conversion tools, which are easy to integrate existing conversion tools and extend new conversion tools.

[0105] In the embodiment of the present application, considering that the existing conversion tools in the conversion tool library may not be able to complete the conversion between the source data format and the target data format, therefore, new format conversion functions can also be derived based on the existing conversion tools in the conversion tool library to complete data conversion.

[0106] In one example, the data conversion middleware is further configured to perform the following steps:

[0107] Step 231: Based on the conversion rules corresponding to each conversion tool in the conversion tool library, determine whether there is a conversion tool in the conversion tool library that can convert the source data format to the target data format;

[0108] Step 232: If not, determine at least one conversion path based on the conversion rules corresponding to each conversion tool, the source data format, and the target data format;

[0109] Step 233: Invoke the conversion tools in the conversion tool library based on the conversion path.

[0110] Among them, in Step 231, the data conversion middleware can first determine whether there is a conversion tool in the conversion tool library that can directly convert the source data format to the target data format according to the source data format, the target data format, and the conversion rules corresponding to each conversion tool in the conversion tool library; the conversion rules can describe the data formats before and after the conversion by the conversion tool.

[0111] Further, in Step 232, if not, at least one conversion path can be determined according to the conversion rules corresponding to each conversion tool, the source data format, and the target data format. Among them, the conversion path can describe the format conversion process from the source data format to the target data format.

[0112] Exemplarily, Figure 4 is a schematic diagram of a conversion path provided by an embodiment of the present application. As Figure 4 shown, in the conversion tool library, there is no conversion tool that can directly convert the data format A to the data format B. Therefore, two conversion paths can be determined. One conversion path is: first convert the data format A to the data format C, and then convert from the data format C to the data format D, that is, AC - CD; the other conversion path is: first convert the data format A to the data format B, and then convert from the data format C to the data format D, that is, AB - BD.

[0113] By querying the conversion path, new format conversion can be realized based on the existing conversion tools in the conversion tool library without writing new conversion tools.

[0114] After determining the conversion path, further, in Step 233, multiple conversion tools in the conversion tool library can be invoked according to the conversion path. When the number of conversion paths is multiple, the final conversion path can be selected by combining the cost.

[0115] Optionally, the data conversion middleware is further configured to perform the following steps:

[0116] Step 2331: For each conversion path, determine the conversion precision loss and conversion required time of each sub-path in the conversion path, and based on the conversion precision loss and conversion required time of each sub-path, determine the total loss and total required time of the conversion path;

[0117] Step 2332: Eliminate the conversion paths with a total loss greater than the preset loss acceptance degree, and based on the total loss and total required time of the remaining conversion paths, determine the final path;

[0118] Step 2333: In the conversion tool library, call the conversion tools corresponding to each sub-path in the final path.

[0119] Among them, in Step 2331, for each conversion path, the conversion precision loss and conversion required time of each sub-path in the conversion path can be determined; a sub-path is the conversion rule corresponding to the conversion tool involved in the conversion path, such as Figure 4 AC, CD, AB, BD in

[0120] The conversion precision loss can be the data precision lost during the conversion of data, that is, the conversion distortion rate, and the conversion required time can be the time required to complete the data conversion.

[0121] For each conversion path, after obtaining the conversion precision loss and conversion required time of each sub-path therein, further, the conversion precision loss and conversion required time of all sub-paths can be accumulated to obtain the total loss and total required time of the conversion path.

[0122] Further, in Step 2332, the conversion paths with a total loss greater than the preset loss acceptance degree can be eliminated first to retain the conversion paths with the total loss (i.e., the distortion rate) within the acceptable range (i.e., the preset loss acceptance degree), and based on the total loss and total required time of the remaining conversion paths, determine the cost of the remaining conversion paths, and select the conversion path with the lowest cost as the final path.

[0123] Reference Figure 4 , the cost of sub-path AC is 0.6, the cost of sub-path CD is 0.9, then the cost of conversion path AC-CD is 1.5; the cost of sub-path AB is 1.0, the cost of sub-path BD is 0.8, then the cost of conversion path AB-BD is 1.8.

[0124] After obtaining the costs of each conversion path, the conversion path with the lowest cost can be selected from them as the final path.Figure 5 It is a schematic diagram of a final path provided by an embodiment of the present application. As Figure 5 shown, the cost of the conversion path AC-CD is 0.6 + 0.9 = 1.5. Therefore, it can be used as the final path.

[0125] In addition to selecting the final path according to the cost, a conversion path including an open-source conversion tool can also be preferentially selected as the final path. For example, if there is a conversion path including an open-source conversion tool, it is used as the final path; otherwise, the conversion path with the lowest cost is selected as the final path.

[0126] After determining the final path, further, in step 2333, the conversion tools corresponding to each sub-path in the final path can be called in the conversion tool library to implement data conversion.

[0127] Figure 6 It is a schematic diagram of a data conversion process provided by an embodiment of the present application. As Figure 6 shown, the input of this conversion process may include the file to be converted (i.e., the file output by the current computing task), the source type (i.e., the source data format), the target type (i.e., the target data format), and the preset loss acceptance. Specifically, the conversion tool library can be read first to find out whether there is a conversion path. If not, it is determined that the conversion fails. If there is, the conversion accuracy loss and conversion required time of each sub-path in the conversion path can be calculated, and then it is judged whether the total loss of the conversion path exceeds the preset loss acceptance. If so, it is determined that the conversion fails. If not, the final path is further determined, and then the conversion tool is prepared, the conversion operation is executed, and it is judged whether it runs correctly to completion. If it runs correctly to completion, it is determined that the conversion is successful. If it does not run correctly to completion, it is determined that the conversion fails.

[0128] In the above optional implementation manner, by integrating a data conversion middleware in the domain application service platform, the multi-technology integration in the domain application service platform can be realized. And by calculating the conversion accuracy loss and conversion required time of each sub-path in the conversion path, the cost of the conversion path is determined, and the conversion paths with total losses not meeting the requirements are eliminated. Then, the final conversion path is selected in combination with the cost, so as to meet the conversion function requirements with the existing conversion power, and as much as possible ensure the efficiency and reliability of data conversion.

[0129] In the embodiments of the present application, the conversion tool can specifically implement format conversion through steps such as data parsing, data extraction, and data assembly. After the current computing task is completed, the generated result file can be output through the output data port, and the data type of the result file is the same as that defined in the input / output port. Therefore, the data conversion middleware can infer the data type generated by the current computing task from the definition in the input / output port, and then transfer information such as the data type to the conversion tool for processing.

[0130] In some embodiments, the conversion tool is used to perform the following steps:

[0131] Step 31: Parse the data output by the current computing task to obtain a source memory object;

[0132] Step 32: Extract target data items from the source memory object and save the target data items as intermediate objects, where the target data items are the data items required for the execution of the next computing task;

[0133] Step 33: Assemble the intermediate objects according to the next computing task to obtain a target memory object;

[0134] Step 34: Format the data of the target memory object to obtain data conforming to the target data format.

[0135] Among them, after the data conversion middleware calls the conversion tool, the conversion tool can first parse the data output by the current computing task to obtain a source memory object. The process of data parsing can be understood as the process of reading the result file of the current computing task (including the data it outputs) and constructing the corresponding source memory object according to the source data type. For example, reading the trajectory result file of the lammps material calculation software to form an atomic coordinate array, etc.

[0136] Furthermore, in step 32, target data items can be extracted from the source memory object and saved as intermediate objects. This process can be understood as determining the data items that need to be extracted from the source data of the current computing task according to the information required by the next computing task, and then obtaining the values of the corresponding data items from the source memory object and saving them as a type of intermediate object. For example, calculating the number of element types from the atomic coordinate array and saving it to an integer variable.

[0137] Furthermore, in step 33, the extracted intermediate objects can be used to assemble according to the target data format to form a target memory object. This process can be understood as reorganizing the intermediate objects according to the specifications of the target data format.

[0138] Further, in step 34, after the data assembly is completed, the target memory object can be formatted according to the target data format to form a string stream or a binary data stream. The serialization result can either be written to a file to form a file in the target data format or directly stored in the result database through a database interface.

[0139] As Figure 7 shown, Figure 7 FIG. is a schematic diagram of the process of data conversion by a conversion tool provided in an embodiment of the present application. First, the source file can be parsed according to the source data format to obtain a source memory object. Then, according to the source data format and the target data format, data extraction is performed on the source memory object to obtain an intermediate object, and data assembly is performed on the intermediate object according to the target data format to obtain a target memory object. Furthermore, the target memory object is serialized according to the target data format to obtain a target file.

[0140] Among them, for the file parsing, data extraction, data assembly, and serialization parts, customized development can be carried out respectively according to the format types of the source data format and the target data format, and then assembled using the template pattern according to the illustrated processing process to complete the data parsing and conversion process. After the target data format and its serialization method are determined, the corresponding serialization program can be written to serialize the data into different exchange formats such as JSON, XML, and CSV according to requirements, and then the data is stored in the corresponding database through a database interface adapter. At the same time, it can be integrated into the computing workflow and rely on the workflow management system to run automatically in batches to achieve automatic parsing of the calculation results.

[0141] Through the above implementation methods, the computing tools of different algorithms in different application fields can be abstracted and standardized by analyzing their input and output data formats, computing resource call methods, etc., to form a data conversion middleware, so that the input and output of the computing tasks corresponding to various computing tools can be connected through this data conversion middleware, realizing the effective integration of computing tasks within the platform and effectively supporting the workflow.

[0142] In the embodiment of the present application, in order to monitor the execution process of the computing tasks, a task monitoring module can also be set in the domain application service platform, and this module monitors each computing task through the status layer of each target model.

[0143] In one example, the domain application service platform further includes a task monitoring module, and the task monitoring module is used for:

[0144] During the process of the workflow engine system executing each computing task, based on the status layer in the model file of each target model, determine the task status of each computing task and display the task status of each computing task.

[0145] Specifically, during the process that the workflow engine system sequentially calls the model files of each component model to execute each computing task, the task monitoring module can determine the task status of each computing task according to the status layer in the model files of each target model and display it.

[0146] In addition, the task monitoring module can also implement the status transition of the computing task through the status layer of each target model. For example, it can trigger status updates (such as monitoring the content of the status.log file) by parsing the execution logs of the tool layer or API callbacks.

[0147] By developing the task monitoring module in the domain application service platform, the above example can achieve the integration of multiple technologies in the domain application service platform, monitor each computing task during the execution of the business process, and can timely remind when an exception occurs in the computing task.

[0148] In the embodiment of the present application, a shared storage system can also be set for the domain application service platform, so that the cloud computing platform and the high-performance computing platform can achieve data sharing and indirectly achieve interconnection.

[0149] In one example, the domain application service platform further includes a shared storage system. The shared storage system connects the cloud computing platform and the high-performance computing platform and is used for:

[0150] In response to a data storage request from the cloud computing platform or the high-performance computing platform, storing the data generated by the cloud computing platform or the high-performance computing platform as shared data; in response to a data reading request from the cloud computing platform or the high-performance computing platform, providing the shared data for the cloud computing platform or the high-performance computing platform.

[0151] Specifically, the cloud computing platform or the high-performance computing platform can send a data storage request to the shared storage system, and the shared storage system can store the data generated by the cloud computing platform or the high-performance computing platform as shared data; and, the cloud computing platform or the high-performance computing platform can send a data reading request to the shared storage system, and the shared storage system can provide the shared data for the cloud computing platform or the high-performance computing platform.

[0152] Among them, the shared storage system can specifically be a network-attached storage or a distributed file system, etc. The cloud computing platform or the high-performance computing platform can store data in the shared storage system, the high-performance computing platform can read data from the shared storage system for calculation, and the calculation results can also be stored in the shared storage system for the cloud computing platform to read.

[0153] In addition, high-performance computing, cloud computing, and big data platforms can also be integrated into the domain application service platform. Through the adoption of a shared storage system in the domain application service platform in the above example, the integration of multiple technologies in the domain application service platform can be achieved, cross-hardware platform data transmission and unified resource management can be realized, and the needs of different application fields can be met. The high-performance computing (supercomputing and intelligent computing) platform can provide scientific computing and artificial intelligence training services; big data provides data storage and analysis and processing services; the cloud computing platform provides remote application services. Through the cloud computing platform, users can perform visual interaction access and workflow visualization operations, and directly access high-performance computing and big data resources to achieve the coordinated integration of the three services.

[0154] In an embodiment of the present application, remote visualization services can also be provided through the domain application service platform. By constructing a remote visualization environment and combining application virtualization and data lightweighting technologies, it is possible to meet user experience and rapid data analysis and display under limited bandwidth. For example, first, through application virtualization technology, the running environment of the application program can be separated from the user device, and the user can remotely access the application program running on the server through the virtual environment network. At the same time, data lightweighting technology is adopted to optimize the data to be transmitted and reduce the amount of data. For large-scale computational result data, technologies such as data compression and data simplification are used to reduce the data transmission volume without affecting the key information of the data; then, a remote visualization environment is constructed to display the processed data to the user in an intuitive and visual manner. Through the combination of these technologies, it not only ensures that users can quickly obtain data and perform analysis and display, but also provides a good user experience.

[0155] In one example, the domain application service platform further includes a remote visualization module. The remote visualization module is connected to the user terminal, the cloud computing platform, and the high-performance computing platform. The remote visualization module is used for:

[0156] In response to receiving a visualization request from the user terminal, determine the target application in the visualization request; start the target application in the cloud computing platform or the high-performance computing platform, and perform data optimization on the running data of the target application, and feedback the optimized running data to the user terminal for display.

[0157] Among them, if the remote visualization module receives a visualization request from the user terminal, it can determine the specified target application therein, and then start the target application in the cloud computing platform or the high-performance computing platform, and then perform data optimization on the data generated after the target application runs, such as data compression, feature extraction, etc., and feedback the optimized running data to the user terminal for display.

[0158] In this way, a remote visualization module can be developed in the domain application service platform to achieve the integration of multiple technologies in the application service platform. Specifically, the application program can be deployed on a cloud computing platform or a high-performance computing platform. Users do not need to start the application program locally. Moreover, by adopting data optimization methods, lightweight data such as pictures and videos is generated and presented to users through the remote visualization environment. This can ensure that users can quickly obtain application data under limited bandwidth conditions, thereby guaranteeing the user experience.

[0159] In addition, to further ensure the user experience, performance testing can be carried out on the remote visualization environment. Under different bandwidth conditions, the user experience and data display effects are tested, and the remote visualization environment is optimized according to the test results.

[0160] In the embodiment of this application, the cloud computing platform and the high-performance computing platform can also develop RESTful API interfaces respectively for receiving and processing requests sent by each other; RESTful API is a type of interface design based on the HTTP protocol, with advantages such as lightweight, easy to implement, and strong scalability. For example, the cloud computing platform can submit a computing task to the high-performance computing platform through the API interface, specifying parameters such as computing resource requirements and input data paths; the high-performance computing platform then returns the computing result to the cloud computing platform through the API interface.

[0161] In addition, the domain application service platform can also adopt the message queue technology. The cloud computing platform can encapsulate the computing task into a message and send it to the message queue. The high-performance computing platform can obtain the task message from the message queue, perform computing processing, and send the result message back to the message queue; then the cloud computing platform reads the computing result from the message queue. Commonly used message queue systems include RabbitMQ, Kafka, etc. Through the message queue, the cloud computing and high-performance computing platforms can be decoupled, improving the reliability and scalability of the system.

[0162] In addition, the cloud computing platform and the high-performance computing platform can also develop remote procedure call technologies respectively, that is, using RPC frameworks such as gRPC and Thrift. The cloud computing and high-performance computing platforms can define service interfaces and message formats, providing a more tightly coupled communication method, which is suitable for scenarios with high performance requirements. For example, the cloud computing platform can call the service on the high-performance computing platform through the RPC client, passing parameters and obtaining the result; the RPC framework is responsible for converting the local call into network communication to achieve remote procedure calls.

[0163] The above-mentioned domain application service platform integrating multiple technologies can also solve the problem that the existing domain application service platforms lack a unified basic service environment for the integration of multiple technologies, realize the effective flow of multi-source heterogeneous data within the computing platform, the integration of computing tools, the automatic control of computing business processes, and, realize meeting user experience and rapid data analysis and display in a remote Internet environment, and finally form a convenient application platform service environment serving users.

[0164] Moreover, the domain application service platform provided by the embodiments of the present application can avoid the repeated construction of the application basic environment, significantly improve efficiency and portability, shorten the deployment cycle of the platform basic environment, realize the rapid connection of computing tools and the automatic calculation of business processes, improve computing efficiency, build a real-time visualization engine in a low-bandwidth environment, and support the smooth access and interactive analysis of large-scale data; solve the problems of repeated construction of cross-domain platforms, difficult data sharing, insufficient business process automation, and poor user remote visualization interaction experience, and realize the high portability of the platform and the rapid adaptation ability of the domain.

[0165] It should be noted that the terms used in the present application are only for describing specific embodiments and do not limit the scope of the present application. As shown in the specification and claims of the present application, unless the context clearly indicates otherwise, words such as "a", "an", "one" and / or "the" are not specifically singular and may also include plural. The term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method or device including a series of elements not only includes those elements but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method or device including the element.

[0166] It should also be noted that the orientation or positional relationship indicated by terms such as "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus cannot be understood as a limitation of the present application. Unless otherwise clearly specified and limited, terms such as "installed", "connected", "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood in specific situations.

[0167] In this text, specific examples are used to elaborate on the principles and implementation modes of this application. The description of the above embodiments is only for helping to understand the method and its core idea of this application. The above is only the preferred implementation mode of this application. It should be noted that due to the limitation of literal expression, and objectively there are infinite specific structures. For those of ordinary skill in the art in this technical field, without departing from the principle of this application, several improvements, refinements or changes can be made, or the above technical features can be combined in an appropriate manner; these improvements, refinements, changes or combinations, or directly applying the concept and technical solution of the invention to other occasions without improvement, shall all be regarded as the protection scope of this application.

Claims

1. A domain application service platform integrating multiple technologies, characterized in that, The domain application service platform includes a component model library and a workflow engine system. The component model library stores model files of each component model. The workflow engine system is used for: Based on the description layer in the model files of each component model, determining the calculation function information of each component model and displaying the calculation function information on the model combination page; Responding to a combination request submitted by a user on the model combination page, determining each target model specified in the combination request and the model execution order; According to the model execution order, sequentially calling the model files of each target model in the component model library, executing the corresponding calculation tasks through the tool layer in the model files, and executing the pre-processing tasks or post-processing tasks of the calculation tasks through the customization layer in the model files.

2. The domain application service platform according to claim 1, characterized in that, The domain application service platform is connected to a cloud computing platform and a high-performance computing platform. The high-performance computing platform is a supercomputing platform or an intelligent computing platform. The workflow engine system is further used for: Determining resource requirement information through the tool layer in the model files and generating a resource scheduling script according to the resource requirement information through the customization layer in the model files; Scheduling the calculation tasks to the cloud computing platform or the high-performance computing platform for execution based on the resource scheduling script.

3. The domain application service platform according to claim 1, characterized in that, The domain application service platform further includes a data conversion middleware and a conversion tool library. The data conversion middleware is used for: During the process of the workflow engine system executing each calculation task, in response to detecting that the current calculation task has been completed, determining whether format conversion is required between the current calculation task and the next calculation task; If so, taking the data format output by the current calculation task as the source data format and the data format input by the next calculation task as the target data format; Calling the conversion tool in the conversion tool library to convert the data output by the current calculation task from the source data format to the target data format.

4. The domain application service platform according to claim 3, characterized in that, The data conversion middleware is further used for: Based on the conversion rules corresponding to each conversion tool in the conversion tool library, determining whether there is a conversion tool in the conversion tool library that can convert the source data format to the target data format; If not, determining at least one conversion path based on the conversion rules corresponding to each conversion tool, the source data format, and the target data format; Calling the conversion tool in the conversion tool library based on the conversion path.

5. The domain application service platform according to claim 4, characterized in that, The data conversion middleware is further used for: For each conversion path, determining the conversion accuracy loss and conversion required time of each sub-path in the conversion path, and based on the conversion accuracy loss and conversion required time of each sub-path, determining the total loss and total required time of the conversion path; Eliminating the conversion paths with a total loss greater than the preset loss acceptance degree, and determining the final path according to the total loss and total required time of the remaining conversion paths; In the conversion tool library, calling the conversion tools corresponding to each sub-path in the final path.

6. The domain application service platform according to claim 3, characterized in that, The data conversion middleware is further used for: Determine the data format of the data output by the current computing task based on the input and output ports in the model file corresponding to the current computing task; Determine the data format of the data input by the next computing task based on the input and output ports in the model file corresponding to the next computing task; If the data format of the data output by the current computing task is different from the data format of the data input by the next computing task, determine that format conversion is required between the current computing task and the next computing task.

7. The domain application service platform according to claim 3, characterized in that, The conversion tool is used for: Parse the data output by the current computing task to obtain a source memory object; Extract target data items from the source memory object and save the target data items as intermediate objects, where the target data items are the data items required for the execution of the next computing task; Assemble the intermediate objects according to the next computing task to obtain a target memory object; Format the data of the target memory object to obtain data that conforms to the target data format.

8. The domain application service platform according to claim 1, characterized in that, The domain application service platform further includes a task monitoring module, and the task monitoring module is used for: During the process of the workflow engine system executing each of the computing tasks, determine the task status of each of the computing tasks based on the status layer in the model file of each of the target models, and display the task status of each of the computing tasks.

9. The domain application service platform according to claim 1, characterized in that, The domain application service platform further includes a shared storage system, and the shared storage system connects the cloud computing platform and the high-performance computing platform. The shared storage system is used for: In response to a data storage request from the cloud computing platform or the high-performance computing platform, store the data generated by the cloud computing platform or the high-performance computing platform as shared data; In response to a data reading request from the cloud computing platform or the high-performance computing platform, provide the shared data for the cloud computing platform or the high-performance computing platform.

10. The domain application service platform according to claim 1, characterized in that, The domain application service platform further includes a remote visualization module, and the remote visualization module connects the user terminal, the cloud computing platform and the high-performance computing platform. The remote visualization module is used for: In response to receiving a visualization request from the user terminal, determine the target application in the visualization request; Start the target application in the cloud computing platform or the high-performance computing platform, optimize the operation data of the target application, and feed back the optimized operation data to the user terminal for display.

Images