Model centralized management scheduling method and device oriented to multiple development languages

Abstract

The invention belongs to the technical field of nuclear power operation and maintenance, and discloses a model centralized management scheduling method and device oriented to multiple development languages, and the method comprises the steps: configuring an actuator cluster on each operation platform, configuring calculation environments of different language types for each actuator in each actuator cluster, obtaining a configured model, and submitting the configured model to a scheduling engine, the scheduling engine selects a corresponding operation platform based on a language type calibrated by the model, selects an actuator from an actuator cluster of the selected operation platform based on a load balancing strategy for executing the model, generates a job of the model according to a model operation mode, calls an interface of the corresponding actuator after the job is triggered, and executes the model according to the interface. Creating a model operation task based on the interface of the corresponding executor and executing the task, and managing and monitoring the model operation task by the scheduling engine through the interface of the corresponding executor; the device can realize the method. According to the invention, all operation platforms, actuators thereof and model operation tasks are managed and monitored.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of nuclear power operation support, and particularly relates to a model centralized management and scheduling method and device for multiple development languages. BACKGROUND

[0002] In the field of nuclear power operation support, a large amount of work has been carried out for health detection and fault diagnosis of equipment, and models developed based on various languages have been researched and developed. However, these models are scattered and deployed on respective computing servers, and the deployment modes of the respective models also differ, and a single-node running mode is generally present, and the reliability is not guaranteed, and problems such as low utilization rate of server resources, waste of hardware resources, high maintenance cost, insufficient running reliability, wide range of operation and maintenance personnel, etc. are generally present.

[0003] In the data analysis model development scenario, different projects have different development environments, different development tools, and different tools are divided into multiple sets of environments at the same time, covering development environments, test environments and pre-production environments. Each environment has its own model running environment, different environment models consume different resources, and different development stages also consume different server resources. In the prior art, there is no unified management and unified allocation of server computing resources, and different environments have different degrees of idle and insufficient computing server resources, which leads to low comprehensive utilization rate of server resources, waste of server resources, waste of energy and other problems. SUMMARY

[0004] The purpose of the present application is to provide a model centralized management and scheduling method and device for multiple development languages, which solves the technical problems of nuclear heterogeneous model scheduling management and the problem of unbalanced utilization of server computing resources.

[0005] In order to achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a model centralized management and scheduling method for multiple development languages, comprising:

[0007] S1, configuring an executor cluster on each running platform, and configuring a computing environment of different language types for each executor in the respective executor cluster;

[0008] S2, obtaining the configured model and submitting it to a scheduling engine, the scheduling engine selecting a corresponding running platform based on the language type calibrated by the model, and selecting an executor in the executor cluster of the selected running platform based on a load balancing strategy for execution of the model;

[0009] S3, generating a job of the model according to the model running mode, calling an interface of the corresponding executor after the job is triggered, creating a model running task based on the interface of the corresponding executor and executing the task;

[0010] S4. The scheduling engine manages and monitors the model's running tasks through the interfaces of the corresponding executors.

[0011] As one feasible approach, the configured model is obtained and submitted to the scheduling engine, specifically as follows:

[0012] Select the model running platform, set the model running mode, configure the model input and output, and complete the model deployment configuration;

[0013] The configured model is uploaded to the scheduling engine, which then configures the model based on its configuration file.

[0014] As an feasible approach, an executor can be selected from the executor cluster of the selected running platform for model execution based on a load balancing strategy, specifically as follows:

[0015] When publishing model tasks, the scheduling engine ignores executors on running platforms where the number of running models has reached a preset value, and automatically deploys the models to idle and available executors.

[0016] As an feasible approach, the model is automatically deployed to the corresponding idle and available executor, specifically as follows:

[0017] The model is deployed to an idle and available executor in the corresponding operating platform by means of polling, randomization, or failover.

[0018] As an feasible approach, an executor is selected from the executor cluster of the selected running platform based on the language type of the model for model execution. The model is matched with each executor in the executor cluster according to the language type of the model, and the corresponding executor is selected for model execution.

[0019] As an feasible approach, the model's job is generated based on the model's execution method, specifically as follows:

[0020] When the model runs in a periodic mode, the scheduling engine obtains the model's start time, end time, and frequency, and executes the model periodically according to the frequency within the running time range. Each time the model runs, it obtains the data range to be executed based on the running frequency and the current time, sets the data range to the data range parameter of the input node, and executes the model.

[0021] When the model runs in a non-periodic mode, the scheduling engine obtains the model's data start time, data end time, and data splitting rules to get the data range for each run. Then, the model is executed multiple times, setting the data range for each run to the data range parameter of the input node and executing the model.

[0022] As an implementable manner, S4 comprises:

[0023] The service interface provided by the executor is called to manage and monitor the model running task.

[0024] The management and monitoring include running state, task starting, task stopping, task restarting, task offline, task transfer, parameter configuration, configuration modification, and log management.

[0025] As an implementable manner, when the server running the task is unavailable, all models on the server are transferred to other available servers for re-running.

[0026] As an implementable manner, based on the task transfer instruction of the user, the model task being executed is offline from the current executor and is transferred to other executors for running.

[0027] In a second aspect, the application provides a model centralized management and scheduling device for multiple development languages, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to realize the model centralized management and scheduling method for multiple development languages.

[0028] Compared with the prior art, the model centralized management and scheduling method and device for multiple development languages provided by the application have the following beneficial effects:

[0029] The application sets a scheduling engine, which manages and monitors all running platforms, executors, and model running tasks. When a model needs to run, the scheduling engine receives a model deployment request in an offline state, creates a model running task, and selects a suitable running platform based on a load balancing strategy to issue the model, and then executes the task.

[0030] The application implements the management of starting, stopping, adding, and deleting tasks based on the executor interface, supports the monitoring of the tasks, interacts through the interface provided by the running platform, controls and monitors the execution of the model, and realizes the ordered management and scheduling of multiple heterogeneous models. DETAILED DESCRIPTION

[0031] In order to more clearly illustrate the technical solutions of the application, the following will briefly introduce the drawings needed to be used in the technical description.

[0032] Figure 1 The flowchart of the model centralized management and scheduling method for multiple development languages provided by the application;

[0033] Figure 2 The flowchart of the scheduling engine provided by the application. DETAILED DESCRIPTION

[0034] The following detailed description provides further details on specific implementation methods.

[0035] Example 1

[0036] like Figure 1 As shown, this embodiment provides a centralized management and scheduling method for models oriented towards multiple development languages, specifically including the following steps:

[0037] S1. Configure an executor cluster on each running platform and configure a computing environment of different language types for each executor in each executor cluster;

[0038] S2. Obtain the configured model and submit it to the scheduling engine. The scheduling engine selects the corresponding running platform based on the language type specified by the model, and selects an executor from the executor cluster of the selected running platform for model execution based on the load balancing strategy.

[0039] S3. Generate model jobs according to the model's running mode. After the job is triggered, call the interface of the corresponding executor, create a model running task based on the interface of the corresponding executor, and execute the task.

[0040] S4. The scheduling engine manages and monitors the model's running tasks through the interfaces of the corresponding executors.

[0041] This embodiment sets up a scheduling engine to manage the multi-variable heterogeneous model developed using nuclear power data analysis tools. The scheduling engine manages the runtime platform and its executors, model execution tasks, and monitors the execution of model execution tasks. The specific process is as follows:

[0042] When deploying a model, select the running platform, set the model running mode, configure the inputs and outputs, and submit the model running task to the scheduling engine.

[0043] The scheduling engine selects a suitable executor from the executors in the running platform through load balancing strategies;

[0044] The scheduling engine generates jobs based on periodic or non-periodic methods;

[0045] After the job is triggered, the executor's interface is called to create a model, run the task, and execute it.

[0046] The scheduling engine manages tasks and monitors their execution through the executor's interface;

[0047] The scheduling engine monitors the running status of the executors. If an executor crashes, it triggers a failover, selects a new executor deployment model, and executes it.

[0048] In this embodiment, the scheduling engine receives the model deployment request in the offline state, creates a model running task and selects a suitable running state to issue the model, and periodically or non-periodically executes the task. A plurality of strategies are supported to select the running state server, such as polling, random, failover, etc., and dynamic expansion and contraction of the running state are supported. The start, stop, increase and delete operations of the task are supported, the task monitoring is supported, and the task running log can be viewed. The model execution of the running state computing engine is controlled and monitored through the interface provided by the running state.

[0049] As shown in Figure 2 The function modules of the distributed scheduling engine mainly include task publishing, load balancing, task monitoring and task migration, and further include running platform management, model management, user management and operation log modules.

[0050] Specifically, in S2, the configured model is acquired and submitted to the scheduling engine, specifically:

[0051] The model running platform is selected, the model running mode is set, the model input and output are configured, and the deployment and configuration of the model are completed; the configured model is uploaded to the scheduling engine, and the model is configured based on the configuration file of the model.

[0052] The running platform management module is mainly used for managing the IP, port, supported language type and other information of the executor, and the number of model running can be configured. When the running number reaches the threshold, the service will be automatically forwarded to other idle available executors for running. The executor configuration can support Python, Java, c / c++ language, and the corresponding computing environment is configured on the computing node.

[0053] In the model management module, the user can add model application execution programs developed in different languages such as Python, JAVA and c / c++. If the execution program has a configuration file, it needs to be configured in the configuration management. The application added here will be executed on the corresponding computing node according to the calibrated language type when a new computing task is created. The configuration of the computing node is configured in the running platform management, and each computing node is marked to support which syntax model program to run.

[0054] The task publishing supports manual publishing program, the execution program of the task is selected from the application management, and the program execution parameters are uploaded or configured according to the program type. When publishing the task, the corresponding running platform needs to be selected according to the program type, and the scheduling platform will automatically select an executor from the executor cluster of the running platform to execute the task.

[0055] Specifically, in S2, an executor is selected from the executor cluster of the selected running platform based on the load balancing strategy for model execution, specifically:

[0056] When publishing a model task, the scheduling engine ignores the executors of the running platform whose number of running models has reached the preset value, and automatically deploys the model to an idle and available executor.

[0057] When publishing a task, the scheduling engine provides a load balancing strategy, ignores the server whose number of running models has reached the preset value, and automatically deploys the model to an idle and available server.

[0058] Specifically, the model is automatically deployed to a corresponding idle and available executor, specifically including:

[0059] An idle and available executor in the corresponding running platform is selected by polling, randomly, or failover, and the model is deployed to the executor.

[0060] The scheduling engine supports multiple strategies for selecting a running state server, such as polling, random, failover, etc., and supports dynamic scaling of the running state.

[0061] Specifically, in S2, the language type based on the model calibration is selected from the executor cluster of the selected running platform for model execution, specifically:

[0062] According to the language type calibrated by the model, the model is matched with each executor in the executor cluster, and the corresponding executor is selected for model execution.

[0063] The configuration of the executor is configured in the running platform management module, and each executor is annotated to support which syntax model program to run. The model task is correspondingly issued to the corresponding executor for execution according to the calibrated language type.

[0064] Specifically, in S3, the model job is generated according to the model running mode, and the model running mode includes periodic running and non-periodic running, specifically:

[0065] When the model running mode is periodic running, the scheduling engine obtains the running start time, running end time, and running frequency of the model, and periodically executes the model within the running time range according to the running frequency. Each time the model is executed according to the running frequency and the current time to obtain the data range, and the data range is set to the data range parameter of the input node and the model is executed.

[0066] When the model running mode is non-periodic running, the scheduling engine obtains the data start time, data end time, and data splitting rule of the model, obtains the data range of each run, and then executes the model multiple times, sets the data range of each time to the data range parameter of the input node, and executes the model.

[0067] The scheduling engine supports deploying the model according to the periodic or non-periodic running mode.

[0068] Periodic running mode: the computing scheduling engine executes the model periodically according to the running start time, the running end time and the running frequency within the running time range. Each running needs to obtain the data range to be executed according to the running frequency and the current time, set the data range to the data range parameter of the input node and execute the model. For example, the running start time is 12 o'clock, the running frequency is 1 time every 10 minutes, the first execution time is 12 o'clock, and the data range is the data between 11:50 and 12 o'clock; the second execution time is 12:10, the data range is the data between 12 o'clock and 12:10, and so on.

[0069] Non-periodic running mode: the computing scheduling engine obtains the data range of each running according to the data start time, the data end time and the data splitting rule. Then the model is executed for multiple times, and the data range of each time is set to the data range parameter of the input node and the model is executed.

[0070] Specifically, S4 comprises:

[0071] The service interface provided by the executor is called to manage and monitor the model running task. The management and monitoring includes running state, task starting, task stopping, task restarting, task offline, task transfer, parameter configuration, configuration modification and log management.

[0072] The scheduling engine is also used for task monitoring. By selecting the executor in the running platform, the running task in each executor can be viewed. The running state of the task can be viewed. The task can be started, stopped, restarted, offline and transferred. The transfer means that the task is transferred to other executors for running, and the task in the original executor is offline.

[0073] Specifically, in the task monitoring process, when it is monitored that the executor with running task is unavailable, the scheduling engine transfers all models in the executor to other available executors for running again.

[0074] Task migration includes automatic migration and manual migration. When the running server is down or the like, the system automatically transfers all models under the down server to the executors of other available servers for running again, so as to realize automatic migration of the models.

[0075] The user can also trigger manual migration. In the task monitoring page, the user can manually offline multiple model tasks from the current executor and transfer them to other executors for running.

[0076] User management module, for logging in user account management, including user's add, delete, modify, query and other functions. Operation log module, for recording the online, offline, modification, other important operation and other operation logs of tasks.

[0077] The internal interface of the embodiment includes two parts: an upload file interface and a task publishing interface.

[0078] The upload file interface is used to upload the py file and other configuration files of the model before publishing the task, the request mode is POST, the request type is multipart / form-data, and the request parameters are shown in Table 1.

[0079] Table 1, upload file request parameters

[0080]

[0081] Response data format: {"code": 200, "msg": "Operation succeeded"}

[0082] The task publishing interface is used to publish the model task to the distributed scheduling engine service for running, the request mode is POST, and the request type is application / json. The task publishing request parameter format is as follows:

[0083] {

[0084] "groupId":"", / / service group id

[0085] "appId":"", / / program id, i.e. model id

[0086] "appName":"", / / program name, i.e. model name

[0087] "appType":"1", / / application type (1: python)

[0088] "taskName":"", / / task name

[0089] "description":"", / / task description

[0090] "proCode":"", / / project number

[0091] "username":"" / / create user

[0092] }

[0093] Response data format: {"code": 200, "msg": "Operation succeeded"}

[0094] Embodiment two

[0095] The embodiment provides a multi-development language-oriented model centralized management scheduling device, which comprises a memory and a processor, and the memory is stored with a computer program, and the computer program is executed by the processor to realize the multi-development language-oriented model centralized management scheduling method in the embodiment one.

[0096] The multi-development language-oriented model centralized management scheduling device provided by the embodiment is used for realizing the multi-development language-oriented model centralized management scheduling method, so that the device also has the technical effects of the multi-development language-oriented model centralized management scheduling method, which will not be repeated here.

[0097] The above merely provides a specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application.

Claims

1. A multi-development language oriented model centralized management scheduling method, characterized in that, The method comprises the following steps: S1, configuring an executor cluster on each operation platform, and configuring a computing environment of different language types for each executor in each executor cluster; S2, obtaining the configured model and submitting it to a scheduling engine, the scheduling engine selecting a corresponding operation platform based on the language type of the model, and selecting an executor in the executor cluster of the selected operation platform for execution of the model based on a load balancing strategy; S3, generating a job of the model according to a model operation mode, calling an interface of the corresponding executor after the job is triggered, creating a model operation task based on the interface of the corresponding executor, and executing the task; S4, the scheduling engine managing and monitoring the model operation task through the interface of the corresponding executor.

2. The multi-development language oriented model centralized management scheduling method according to claim 1, characterized in that, In S2, the configured model is obtained and submitted to the scheduling engine, specifically: selecting a model operation platform, setting a model operation mode, and configuring model input and output to complete deployment and configuration of the model; uploading the configured model to the scheduling engine, and configuring the model based on the configuration file of the model.

3. The multi-development language oriented model centralized management scheduling method according to claim 1, characterized in that, In S2, an executor in the executor cluster of the selected operation platform is selected for execution of the model based on a load balancing strategy, specifically: When a model task is published, the scheduling engine ignores the executors of the operation platform whose number of running models has reached a preset value, and automatically deploys the model to an idle and available executor.

4. The multi-development language oriented model centralized management scheduling method according to claim 3, characterized in that, The model is automatically deployed to a corresponding idle and available executor, specifically: an idle and available executor in the corresponding operation platform is selected by polling, randomly or by failover, and the model is deployed to the executor.

5. The method of claim 1, wherein the method further comprises: In S2, an executor in the executor cluster of the selected operation platform is selected for execution of the model based on the language type of the model, the model is matched with each executor in the executor cluster according to the language type of the model, and the corresponding executor is selected for execution of the model.

6. The multi-development language oriented model centralized management scheduling method according to claim 1, characterized in that, In S3, the job of the model is generated according to the model operation mode, specifically: when the model operation mode is periodic operation, the scheduling engine obtains the running start time, running end time and running frequency of the model, and periodically executes the model within the running time range according to the running frequency; each time the model is run, the data range is obtained according to the running frequency and the current time, the data range is set to the data range parameter of the input node, and the model is executed; when the model operation mode is non-periodic operation, the scheduling engine obtains the data start time, data end time and data splitting rule of the model, obtains the data range of each run, and then executes the model multiple times, sets the data range of each time to the data range parameter of the input node, and executes the model.

7. The multi-development language oriented model centralized management scheduling method according to claim 1, characterized in that, S4 includes: calling the service interface provided by the executor to manage and monitor the model operation task; management and monitoring include running state, task start, task stop, task restart, task offline, task transfer, parameter configuration, configuration modification and log management.

8. The multi-development language oriented model centralized management scheduling method according to claim 1, characterized in that, When the server running the task is unavailable, all models on the server are transferred to other available servers for re-running.

9. The multi-development language oriented model centralized management scheduling method according to claim 1, characterized in that, Based on the user's task transfer instruction, the model task being executed is taken offline from the current executor and transferred to other executors for running.

10. A multi-development language oriented model centralized management and scheduling apparatus, characterized by, The application relates to a multi-development language-oriented model centralized management scheduling method, which comprises a memory and a processor, and the memory stores a computer program which is executed by the processor to realize the method. The application relates to a multi-development language-oriented model centralized management scheduling method, which comprises a memory and a processor, and the memory stores a computer program which is executed by the processor to realize the method.

Images