Big data task scheduling system and method and electronic equipment

Abstract

The invention provides a big data task scheduling system and method and electronic device.The system comprises a front-end module, a database, an execution machine pre-installed service module, a mirror image making module and a task configuration module.Basic mirror image information is obtained in advance according to information of a user group to which a user belongs, and execution machine parameter information needed by the user is combined; and generating an execution machine operation environment, binding the big data tasks initiated by the users in the user group to the same execution machine operation environment, and starting the execution machine operation environment. Therefore, when large data task scheduling needs to be carried out subsequently, the corresponding execution machine operation environment can be determined directly based on the user information to which the user belongs. The execution machine environments owned by the users of different user groups are isolated, so that uniform resource management and uniform scheduling of big data tasks are realized, and the problem of resource waste caused by decentralized management is reduced.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a big data task scheduling system, method and electronic device. Background Art

[0002] In the field of big data technology, a big data task platform is a system used to process and manage large-scale data, which can coordinate the execution, scheduling and monitoring of multiple data processing tasks. Among them, data processing tasks refer to data processing requirements initiated by the task demander and need to be executed with the help of the big data task platform, usually including data acquisition, data management, data calculation, data analysis and other tasks.

[0003] In the current big data task platform, big data task scheduling usually includes the following steps: task creation, execution machine binding, and task running. Although the prior art provides some big data task scheduling solutions in the big data task platform, the execution machines in the existing solutions are all managed in a decentralized manner, which is prone to resource waste. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a big data task scheduling system, method and electronic device to reduce resource waste of the big data task platform and lower resource costs.

[0005] In a first aspect, an embodiment of the present application provides a big data task scheduling system, wherein the system includes: a front-end module, a database, an execution machine pre-installed service module, an image production module, and a task configuration module, wherein:

[0006] The front-end module is used to receive an execution machine configuration script, wherein the execution machine configuration script carries user information and execution machine parameter information, and the user information includes user group information to which the user belongs;

[0007] The database is used to store execution machine information, user information, execution machine image information, and execution machine status information, wherein the execution machine image information includes: the execution machine's base image name and the execution machine's base image version;

[0008] The execution machine pre-assembly service module is used to parse the execution machine configuration script, request the database to obtain the basic image information corresponding to the execution machine parameter information, and send a service image production request to the image production module based on the basic image information;

[0009] The image making module is used to receive the service image making request, generate an execution machine operating environment based on the user group information to which the user belongs and the execution machine parameter information, and send a making success indication message to the task configuration module;

[0010] The task configuration module is used to receive the production success indication message, bind the big data tasks initiated by each user in the user group information to which the user belongs to the execution machine running environment, and start the execution machine running environment.

[0011] In some possible embodiments, the system further includes: a task scheduling module, wherein the task scheduling module is used to:

[0012] Receiving a big data task to be processed initiated by a target user, wherein the big data task to be processed carries user information of the target user;

[0013] According to the target user group information to which the target user belongs, a target execution machine operating environment that has a binding relationship with the target user group information is determined, and the to-be-processed big data task is scheduled to the target execution machine operating environment.

[0014] In some possible embodiments, the task scheduling module includes: a virtual hosting submodule, wherein the virtual hosting submodule is used to:

[0015] Determine the idle execution time existing in the target execution machine running environment according to the task running time of each large data task in the target execution machine running environment, wherein no large data task is executed during the idle execution time;

[0016] According to the execution priority of each of the big data tasks, the execution time of the low-priority big data tasks is modified to run during the idle execution time.

[0017] In some possible embodiments, the virtual hosting submodule is further used to:

[0018] Constructing an execution timeline, and determining an execution start time node and an execution end time node of each of the big data tasks on the execution timeline;

[0019] According to the time distribution of each of the big data tasks on the execution timeline, determine whether there is an overlapping interval, and if so, merge the execution time intervals of the big data tasks with overlapping intervals to obtain a merged execution interval;

[0020] According to the execution intervals of the big data tasks without overlapping intervals and the merged execution intervals, they are sorted according to the execution order of the execution timeline, and the idle execution time is determined based on the sorting result.

[0021] In some possible embodiments, the virtual hosting submodule is further used to:

[0022] The sorting results are traversed, and if the start time of the next execution interval is greater than the end time of the current execution interval, the time interval between the next execution interval and the current execution interval is determined as the idle execution time.

[0023] In a second aspect, an embodiment of the present application provides a big data task scheduling method, wherein the big data task scheduling method is applied to any big data task scheduling system described in the first aspect, and the method includes:

[0024] Receive an execution machine configuration script, wherein the execution machine configuration script carries user information and execution machine parameter information, wherein the user information includes user group information to which the user belongs;

[0025] Parse the execution machine configuration script, and request the database to obtain the basic image information corresponding to the execution machine parameter information, wherein the database stores the execution machine information, user information, execution machine image information, and execution machine status information, wherein the execution machine image information includes: the execution machine basic image name, and the execution machine basic image version;

[0026] Generate an execution machine operating environment based on the user group information to which the user belongs and the execution machine parameter information, bind the big data tasks initiated by each user in the user group information to which the user belongs to the execution machine operating environment, and start the execution machine operating environment to run;

[0027] Receiving a big data task to be processed initiated by a target user, wherein the big data task to be processed carries user information of the target user;

[0028] According to the target user group information to which the target user belongs, a target execution machine operating environment that has a binding relationship with the target user group information is determined, and the to-be-processed big data task is scheduled to the target execution machine operating environment.

[0029] In some possible embodiments, the method further includes:

[0030] Determine the idle execution time existing in the target execution machine running environment according to the task running time of each large data task in the target execution machine running environment, wherein no large data task is executed during the idle execution time;

[0031] According to the execution priority of each of the big data tasks, the execution time of the low-priority big data tasks is modified to run during the idle execution time.

[0032] In some possible embodiments, the method further includes:

[0033] Constructing an execution timeline, and determining an execution start time node and an execution end time node of each of the big data tasks on the execution timeline;

[0034] According to the time distribution of each of the big data tasks on the execution timeline, determine whether there is an overlapping interval, and if so, merge the execution time intervals of the big data tasks with overlapping intervals to obtain a merged execution interval;

[0035] Sorting the execution intervals of the big data tasks without overlapping intervals and the merged execution intervals according to the execution order of the execution timeline, and determining the idle execution time based on the sorting result;

[0036] The sorting results are traversed, and if the start time of the next execution interval is greater than the end time of the current execution interval, the time interval between the next execution interval and the current execution interval is determined as the idle execution time.

[0037] In a third aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes:

[0038] Processor; and

[0039] Memory for storing programs,

[0040] Among them, the program includes instructions, and when the instructions are executed by the processor, the processor executes the big data task scheduling method described in the second aspect.

[0041] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the big data task scheduling method described in the second aspect.

[0042] Beneficial effects of this application:

[0043] The present application provides a big data task scheduling system, method and electronic device, wherein the system includes: a front-end module, a database, an execution machine pre-installed service module, an image production module and a task configuration module. Before scheduling the big data task, the basic image information is obtained in advance according to the user group information to which the user belongs, and then an execution machine operating environment is generated according to the basic image information combined with the execution machine parameter information required by the user, and the big data tasks initiated by each user in the user group information to which the user belongs are bound to the same execution machine operating environment, and then the execution machine operating environment is started. By selecting the embodiment of the present application, the big data tasks initiated by each user belonging to the same user group can be bound to the same execution machine operating environment. When the big data task scheduling is required later, the corresponding execution machine operating environment can be directly determined based on the user information to which the user belongs. The execution machine environments owned by users of different user groups are isolated, which helps to realize unified resource management and unified scheduling of big data tasks, thereby reducing the problem of resource waste caused by decentralized management. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0045] Figure 1 A schematic diagram of a system architecture of a big data task scheduling system provided in an embodiment of the present application is shown;

[0046] Figure 2 Another system architecture diagram of the big data task scheduling system provided by an embodiment of the present application is shown;

[0047] Figure 3 A schematic diagram of a process flow of a big data task scheduling method provided in an embodiment of the present application is shown;

[0048] Figure 4 A schematic diagram of a logical structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0050] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0051] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0052] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0053] As described in the background technology, a big data task platform is a system for processing and managing large-scale data, which can coordinate the execution, scheduling and monitoring of multiple data processing tasks. By providing efficient data storage, analysis and processing capabilities, the big data task platform can help users extract business insights from complex data sets (insight refers to a clear and profound understanding of a thing or principle). The big data task platform can usually implement various stages of data processing, such as data collection, data storage, data processing, data analysis and data visualization.

[0054] The big data task platform is designed to optimize the performance of large-scale data operations while improving fault tolerance and scalability to adapt to the growing amount of data. Enterprises or organizations use big data platforms to perform complex data analysis tasks, such as predictive analysis of certain indicators, real-time data processing, machine learning, and big data mining. It can be said that the big data task platform is crucial to generating business insights, optimizing operational efficiency, improving customer experience, and driving innovation.

[0055] The big data task platform supports data of various data types and data from different data sources, including structured data, semi-structured data, and unstructured data. Data sources include social media, IoT devices, and online transaction data.

[0056] Among them, under the big data task platform, the scheduling process of big data tasks initiated by customers is usually as follows:

[0057] Task creation, execution machine binding, and task running.

[0058] Among them, task creation includes: customers operate on the user interface of the big data task platform, including: writing task scripts, setting task metadata, and configuring task scheduling parameters. The task script is mainly used to process and analyze data. Task metadata includes: task description information, task dependencies, expected output, and error handling mechanism. Task scheduling parameters include: task scheduling priority, expected scheduling data size, expected running time, and possible number of concurrently running tasks.

[0059] Execution machine binding includes: binding a physical machine for the task created by the customer, and the physical machine executes the task. The execution machine is the physical machine that executes the data task. The execution machine binding process includes: determining the resources required for the appropriate task operation, including CPU resources, memory resources, and storage resources. Specifically, the corresponding execution machine can be automatically matched according to the determined task resource requirements, or the customer can manually select the execution machine. In the process, based on factors such as cost, efficiency, and execution efficiency, the binding of the execution machine becomes a key link in performance optimization and cost control.

[0060] Although the execution machine can be specified by the customer, in addition to ensuring the resources required for task operation, additional resources must be reserved to prevent insufficient resources during task operation, resulting in low task efficiency. However, even if it is a customer-specific execution machine, the resources of the execution machine need to be reserved during non-task operation to prevent task execution delays or failures caused by insufficient resources during high-load periods. This means that when tasks are not running, a large amount of computing resources are idle, resulting in a large amount of idle resources and waste.

[0061] In addition, the authority of existing execution machines is completely in the hands of customers, and the big data task platform cannot uniformly manage and schedule various execution machines. Since each data task has its own unique execution characteristics and resource requirements, the existing big data task platform makes it impossible to flexibly schedule data tasks in various execution machines, further limiting the possibility of managing the computing resources of the execution machines, and also limiting the implementation of fault recovery and load balancing strategies. For example, if after task A is completed, the resources of the execution machine that executes task A cannot be immediately reallocated to task B with insufficient resources, resulting in the inability of task B to complete execution in time, causing a waste of resources, inefficient resource utilization and output delays, and lack of a flexible cross-task resource scheduling mechanism, which hinders the optimal utilization of resources.

[0062] In view of this, the present application provides a big data task scheduling system, method and electronic device to reduce resource waste and resource costs of a big data task platform. In the first aspect, the present application provides a big data task scheduling system. As a possible implementation method, the big data task scheduling system is integrated into a management device of a big data cluster or integrated into a big data computing engine.

[0063] In some embodiments, the big data task scheduling system 10 can be as follows: Figure 1 As shown, it includes the following parts:

[0064] Front-end module 11, database 12, execution machine pre-installed service module 13, image making module 14, task configuration module 15. Among them:

[0065] The front-end module is used to receive an execution machine configuration script, wherein the execution machine configuration script carries user information and execution machine parameter information, and the user information includes user group information to which the user belongs;

[0066] The database is used to store execution machine information, user information, execution machine image information, and execution machine status information, wherein the execution machine image information includes: the execution machine's base image name and the execution machine's base image version;

[0067] The execution machine pre-assembly service module is used to parse the execution machine configuration script, request the database to obtain the basic image information corresponding to the execution machine parameter information, and send a service image production request to the image production module based on the basic image information;

[0068] The image making module is used to receive the service image making request, generate an execution machine operating environment based on the user group information to which the user belongs and the execution machine parameter information, and send a making success indication message to the task configuration module;

[0069] The task configuration module is used to receive the production success indication message, bind the big data tasks initiated by each user in the user group information to which the user belongs to the execution machine running environment, and start the execution machine running environment.

[0070] When using an embodiment of the present application, before scheduling a big data task, basic image information is obtained in advance based on the user group information to which the user belongs, and then an execution machine operating environment is generated based on the basic image information combined with the execution machine parameter information required by the user, and the big data tasks initiated by each user in the user group information to which the user belongs are bound to the same execution machine operating environment, and then the execution machine operating environment is started. In this way, the big data tasks initiated by each user belonging to the same user group can be bound to the same execution machine operating environment, and when the big data task scheduling is required later, the corresponding execution machine operating environment can be directly determined based on the user information to which the user belongs. The execution machine environments owned by users of different user groups are isolated, which helps to achieve unified resource management and unified scheduling of big data tasks, thereby reducing the problem of resource waste caused by decentralized management.

[0071] The following will describe in detail the big data task scheduling system provided by the embodiment of the present application with reference to specific examples:

[0072] In the embodiment of the present application, the big data task scheduling system is a big data task platform, that is, a system for processing and managing large-scale data, specifically used to coordinate the execution, scheduling and monitoring of a large number of data processing tasks. Among them, the front-end module can be specifically understood as a user interaction module of the big data task scheduling system, which is used to obtain big data tasks submitted by users or instructions input by users. In the embodiment of the present application, a big data task is a task that relies on large-scale data for data processing. The big data tasks submitted by users are usually set by users based on actual needs, and need to rely on large-scale data for data processing.

[0073] As an implementation method, a big data task may be specific data input, data processing conditions, and final data output conditions entered by a user on a user interface of a front-end module. As another implementation method, a big data task may be an execution script submitted by a user, which specifically specifies data input, data processing conditions, and final data output conditions.

[0074] In an embodiment of the present application, before scheduling the big data task input by the user, it is necessary to pre-allocate the execution environment responsible for executing the big data task input by the user. The execution environment is composed of several physical machines, which are divided into different execution environments by physical isolation, and an operating system is deployed in the execution environment to enable the input big data task to perform data processing operations under the deployed operating system. The division of the execution environment can be based on the execution machine configuration script submitted by the user, and the execution machine configuration script includes the execution machine parameter information set by the user.

[0075] As an implementation method, the execution machine parameter information may include: hardware configuration parameters of the execution machine operating environment, software configuration parameters of the execution machine operating environment, and operating system configuration parameters of the execution machine operating environment. Among them, the hardware configuration parameters include: the number and model of the required physical machine devices, the processor model of the physical machine device, the memory model of the physical machine device, the number of CPU cores of the physical machine device, the memory size, and other information. The software configuration parameters of the execution machine operating environment include: the type and version number of the software that executes the big data task. The operating system configuration parameters of the execution machine environment include: the operating system type, the operating system version number, and the like.

[0076] In addition, the execution machine configuration script carries user information, and the user information specifically includes: the user's identity information and the user group information to which the user belongs. Among them, the user's identity information can specifically refer to the user's ID, which can be the user's device ID, or the ID information assigned to the user by the scheduling platform when the user registers to use the big data task scheduling platform. In an embodiment of the present application, tens of thousands of data tasks are scheduled and executed on multiple physical machines. Users can be grouped, and multiple execution machines are specially configured for each group of users to run the data tasks corresponding to the group of users. At this time, there is a one-to-many mapping relationship between users and execution machines. Specifically, user group information is used to distinguish different user groups. Among them, the user group information to which the user belongs can be the user group information obtained by setting a unified user division rule. For example, employees under different business departments belong to different user groups according to the division of the business part. Specifically, the same user group ID can be assigned to users belonging to the same user group, so that the user group information to which the user belongs can be distinguished by different user group IDs.

[0077] In the embodiment of the present application, the database is the underlying data storage module of the entire big data task scheduling system, which is used to store various data required for big data tasks, log records generated during the processing of big data tasks, etc. It can be said that the database is used to store all information of the big data task scheduling system. The information involved in the embodiment of the present application mainly includes: execution machine information, user information, execution machine image information, and execution machine status information. Among them, the execution machine image information includes: the basic image name of the execution machine and the version of the basic image of the execution machine.

[0078] As an implementation method, the execution machine information may specifically be the execution machine parameter information mentioned above. In addition, the execution machine information also includes: binding relationship information generated after the execution machine operating environment is bound to the big data task. The execution machine status information is used to describe the specific operating status of the execution machine, which may be a scheduled operating status or a real-time operating status, or may be an offline execution machine or an online execution machine, wherein the offline execution machine is used to execute big data tasks processed offline, and the online execution machine is used to execute big data tasks processed online.

[0079] In an embodiment of the present application, the image specifically refers to a big data image, which can be any type of big data image. As a preferred implementation, the image can be a Docker image. Among them, the image is an independent and lightweight software package, which contains everything required for the operation of a big data task, including code, operating environment, and dependencies. In an embodiment of the present application, the base image can be a general Docker image, which predefines the basic projects required for the operation of big data tasks, such as various basic data processing tools, big data computing engines, and distributed computing frameworks. The base image version is usually low, so that the image can be modified later according to the execution machine parameter information set in the configuration script input by the user, so as to achieve the effect of upgrading the corresponding Docker image.

[0080] In the embodiment of the present application, the execution machine pre-assembly service module is a software function module developed by the developer in advance according to the execution machine pre-assembly logic. The core is used to modify the basic Docker image obtained from the database into a high-level Docker image that meets the actual needs of the user according to the execution machine parameter information set by the user according to the actual needs. Specifically, as an implementation method, the execution machine pre-assembly service module obtains the execution machine configuration script passed in by the user from the front-end module, and parses and obtains the user information and execution machine parameter information carried in the configuration script, and writes the user information and execution machine parameter information into the database to maintain the data integrity of the entire big data task scheduling system.

[0081] Then, according to the execution machine parameter information, a data acquisition request is sent to the database to request the basic image information corresponding to the execution machine parameter information, which may be specifically to query the basic image name and the basic image version. The execution machine pre-assembly service module is used as a data transfer, and the acquired basic image name and basic image version are sent to the image production module, and a service image production request is sent to the image production module to request the image production module to generate the corresponding execution machine running environment based on the basic image name, basic image version and execution machine parameter information.

[0082] Specifically, after receiving the service image production request, the image production module will parse the user information carried in the service image production request, and then perform a security check based on the user information to determine whether the user is a user in the whitelist. If so, the corresponding physical machine device is further allocated to the execution machine parameter information based on the hardware configuration parameters and software configuration parameters specified in the execution machine parameter information, and then the corresponding operating system, application software, etc. are generated according to the software configuration parameters in the execution machine parameters based on the allocated physical machine device. As a specific implementation method, the user information and the corresponding basic image information are mapped throughout the process, and the internal PAAS (Platform as a Service, service platform) platform is called by executing the DockerFile method to produce the execution machine environment. After the PAAS production is completed, a production success indication message will be sent to remind the execution machine environment that the production is completed.

[0083] At this time, the task configuration module receives the successful production indication message, and can clearly determine that the execution machine operating environment that meets the user's needs has been successfully produced, and then binds the big data tasks initiated by each user in the user group information to which the user information in the configuration script belongs to the execution machine operating environment, and starts the execution machine operating environment. Specifically, after receiving the service image production request, the execution machine pre-assembly service module can update the execution machine operating environment of the user group to which the user belongs in the database, wait for the information to be updated, and then initiate the execution machine environment startup instruction to successfully produce the execution machine operating environment and start the execution machine operating environment.

[0084] As an implementation method, the existing big data tasks belonging to the same user group can be rebound to the execution machine running environment. In this way, multiple execution machine running environments of users in the same user group can be abstracted and unified into the same execution machine running environment to facilitate unified task scheduling.

[0085] As an example, the big data task platform can automatically trigger the scheduled data task at the time node specified by the task scheduling time, and distribute the task script and necessary configuration information corresponding to the data task to the corresponding execution machine. After receiving the task script and configuration information, the execution machine starts the data task, loads the required data, performs calculations, and finally outputs the calculation results. During the entire task execution process, the big data task platform will monitor the execution progress of the data task and record the execution process in the corresponding log information so that subsequent customers can track the task status.

[0086] The following is a detailed description of specific task scheduling:

[0087] In some embodiments, Figure 2As shown, the big data task scheduling system also includes: a task scheduling module 16, which is used to:

[0088] Receiving a big data task to be processed initiated by a target user, wherein the big data task to be processed carries user information of the target user;

[0089] According to the target user group information to which the target user belongs, a target execution machine operating environment that has a binding relationship with the target user group information is determined, and the to-be-processed big data task is scheduled to the target execution machine operating environment.

[0090] In the embodiment of the present application, the core function of the task scheduling module is to bind the big data tasks initiated by the user to the target execution machine operating environment that has a binding relationship with the user to which the user belongs, according to the user group information to which the user belongs, so as to realize logical isolation and operational isolation between the big data tasks of different user groups, and effectively achieve unified scheduling and unified management of big data tasks of different user groups.

[0091] As an implementation method, the task scheduling module 16 further includes a virtual hosting submodule 161, wherein the virtual hosting submodule 161 is a functional module that obtains user hosting authorization in advance and can perform big data task scheduling on behalf of the user. Specifically, the virtual hosting submodule is specifically used to:

[0092] Determine the idle execution time existing in the target execution machine running environment according to the task running time of each large data task in the target execution machine running environment, wherein no large data task is executed during the idle execution time;

[0093] According to the execution priority of each of the big data tasks, the execution time of the low-priority big data tasks is modified to run during the idle execution time.

[0094] In an embodiment of the present application, the execution priority of a big data task can be configured by the user, wherein the execution priority of an online big data task is higher than that of an offline big data task, and the priority of a big data task of a core user is higher than that of a big data task of an ordinary user. In an embodiment of the present application, a big data task will be configured with a corresponding virtual hosting machine for scheduling and execution, and the virtual hosting machine is a virtual hosting sub-module. The task scheduling module will plan a common time interval for executing big data tasks in a unified virtual environment, and the remaining time zone is the idle time of the environment, i.e., the idle execution time. In this way, the virtual hosting sub-module can modify the execution time of a low-priority big data task to run on the idle execution time, thereby achieving the effect of "peak shaving and valley filling", and ensuring the data processing of high-priority big data tasks with sufficient resources.

[0095] In some embodiments, the virtual hosting submodule is further used to:

[0096] Constructing an execution timeline, and determining an execution start time node and an execution end time node of each of the big data tasks on the execution timeline;

[0097] According to the time distribution of each of the big data tasks on the execution timeline, determine whether there is an overlapping interval, and if so, merge the execution time intervals of the big data tasks with overlapping intervals to obtain a merged execution interval;

[0098] According to the execution intervals of the big data tasks without overlapping intervals and the merged execution intervals, they are sorted according to the execution order of the execution timeline, and the idle execution time is determined based on the sorting result.

[0099] In an embodiment of the present application, the virtual hosting submodule can build an execution timeline by calling the Cron algorithm, specifically by using the Cron-utils library. Parse the Cron expression of each big data task, obtain the execution start time node and the execution end time node of each big data task, and calculate the next running time of the big data task. Specifically, the user can enter the execution start time and execution end time of the big data task in the front-end module, and the Cron algorithm will automatically define the specific time nodes of each big data task on the execution timeline according to the execution time entered by the user, thereby defining the specific execution time of the task.

[0100] Then, for each big data task, the running time interval of the big data task can be determined according to the execution time node input by the user, or the user may not input the execution time, and the big data scheduling system may allocate the running time for each big data task. It may be a fixed running time, such as 30 minutes for each task, or the running time of big data tasks of different priorities may be different, with high-priority big data tasks having a long running time and low-priority big data tasks having a short running time.

[0101] After the execution timeline is constructed, the start and end times of all tasks on the timeline can be represented as a time object, such as a ZoneDataTime object. Each task will occupy an interval on the timeline. At this time, in order to more easily find the idle execution time, all execution intervals can be sorted according to the start time to obtain a sorting result on the start time. In this way, the specific execution distribution of each big data task and the specific location of the idle execution time can be clearly known at a glance, so that low-priority big data tasks can be scheduled according to the idle execution time later.

[0102] As a preferred implementation, automatic sorting can be performed using the TreeMap algorithm in Java, and specifically, the sorting result can be automatically generated in a manner that the start time is used as the Key and the end time is used as the Value.

[0103] In some embodiments, the virtual hosting submodule is further used to:

[0104] The sorting results are traversed, and if the start time of the next execution interval is greater than the end time of the current execution interval, the time interval between the next execution interval and the current execution interval is determined as the idle execution time.

[0105] By traversing the Key-Value data generated by the TreeMap, the execution time of each task can be determined. By checking whether there is overlap in the execution time intervals, it can be determined whether there is overlap in the big data tasks in the same execution machine running environment. If there is overlap, the overlapping time intervals can be merged, and the earliest execution start time node is taken as the execution start time node of the merged time interval, and the latest execution end time node is taken as the execution end time node of the merged time interval. In this way, a merged time interval can be obtained, and the merged time interval is regarded as an execution interval, and then compared with other execution intervals.

[0106] If it is determined that the start time of the next execution interval is greater than the end time of the current execution interval, specifically, the start time of the next execution interval is later than the end time of the current execution interval, then it can be determined that there is idle execution time between the current execution interval and the next execution interval, and new big data tasks can be scheduled for execution during this idle execution time. In order to ensure that data resources are not over-preempted, low-priority big data tasks are preferentially scheduled to run during this idle execution time, which can realize automatic scheduling of big data tasks and effectively ensure the reasonable and sufficient application of big data computing resources.

[0107] As an implementation method, for each idle execution time, the start time and end time of the idle execution time can be recorded and further printed out to the user console, or stored in a data list or other data structure for subsequent data task scheduling based on the idle execution time.

[0108] Based on the big data task scheduling system described in the first aspect, in the second aspect, the present application provides a big data task scheduling method, which is applied to any electronic device with a big data task scheduling function, including but not limited to a personal mobile terminal, a computer or a server, etc. Figure 3 As shown, the method comprises the following steps:

[0109] S31, receiving an execution machine configuration script, wherein the execution machine configuration script carries user information and execution machine parameter information, wherein the user information includes user group information to which the user belongs;

[0110] S32, parsing the execution machine configuration script, and requesting the database to obtain the basic image information corresponding to the execution machine parameter information, wherein the database stores the execution machine information, user information, execution machine image information, and execution machine status information, wherein the execution machine image information includes: the execution machine's basic image name, and the execution machine's basic image version;

[0111] S33, generating an execution machine operating environment based on the user group information to which the user belongs and the execution machine parameter information, binding the big data tasks initiated by each user in the user group information to which the user belongs to the execution machine operating environment, and starting the execution machine operating environment to run;

[0112] S34, receiving a big data task to be processed initiated by a target user, wherein the big data task to be processed carries user information of the target user;

[0113] S35. Determine a target execution machine operating environment that has a binding relationship with the target user group information according to the target user group information, and schedule the to-be-processed big data task to the target execution machine operating environment.

[0114] In some possible embodiments, the method further includes:

[0115] Determine the idle execution time existing in the target execution machine running environment according to the task running time of each large data task in the target execution machine running environment, wherein no large data task is executed during the idle execution time;

[0116] According to the execution priority of each of the big data tasks, the execution time of the low-priority big data tasks is modified to run during the idle execution time.

[0117] In some possible embodiments, the method further includes:

[0118] Constructing an execution timeline, and determining an execution start time node and an execution end time node of each of the big data tasks on the execution timeline;

[0119] According to the time distribution of each of the big data tasks on the execution timeline, determine whether there is an overlapping interval, and if so, merge the execution time intervals of the big data tasks with overlapping intervals to obtain a merged execution interval;

[0120] According to the execution intervals of the big data tasks without overlapping intervals and the merged execution intervals, they are sorted according to the execution order of the execution timeline, and the idle execution time is determined based on the sorting result.

[0121] In some embodiments, the method further comprises:

[0122] The sorting results are traversed, and if the start time of the next execution interval is greater than the end time of the current execution interval, the time interval between the next execution interval and the current execution interval is determined as the idle execution time.

[0123] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0124] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0125] In a third aspect, the exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present application when executed by the at least one processor.

[0126] The exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0127] The exemplary embodiments of the present application further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to execute the method according to the embodiment of the present application.

[0128] refer to Figure 4, the structural block diagram of the electronic device 400 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0129] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for the operation of the electronic device 400 can also be stored. The computing unit 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0130] A plurality of components in the electronic device 400 are connected to the I / O interface 405, including: an input unit 406, an output unit 407, a storage unit 408, and a communication unit 409. The input unit 406 may be any type of device capable of inputting information to the electronic device 400, and the input unit 406 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 407 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 408 may include, but is not limited to, a disk, an optical disk. The communication unit 409 allows the electronic device 400 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0131] The computing unit 401 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 401 performs the various methods and processes described above. For example, in some embodiments, the aforementioned big data task scheduling method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 400 via the ROM 402 and / or the communication unit 409. In some embodiments, the computing unit 401 may be configured to perform the aforementioned big data task scheduling method by any other appropriate means (e.g., by means of firmware).

[0132] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0133] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0134] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0135] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0136] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0137] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A big data task scheduling system, characterized in that: The system includes: a front-end module, a database, an execution machine pre-installed service module, an image making module, and a task configuration module, wherein: The front-end module is used to receive an execution machine configuration script, wherein the execution machine configuration script carries user information and execution machine parameter information, and the user information includes user group information to which the user belongs; The database is used to store execution machine information, user information, execution machine image information, and execution machine status information, wherein the execution machine image information includes: the execution machine's base image name and the execution machine's base image version; The execution machine pre-assembly service module is used to parse the execution machine configuration script, request the database to obtain the basic image information corresponding to the execution machine parameter information, and send a service image production request to the image production module based on the basic image information; The image making module is used to receive the service image making request, generate an execution machine operating environment based on the user group information to which the user belongs and the execution machine parameter information, and send a making success indication message to the task configuration module; The task configuration module is used to receive the production success indication message, bind the big data tasks initiated by each user in the user group information to which the user belongs to the execution machine running environment, and start the execution machine running environment.

2. The system according to claim 1, characterized in that The system further comprises: a task scheduling module, wherein the task scheduling module is used to: Receiving a big data task to be processed initiated by a target user, wherein the big data task to be processed carries user information of the target user; According to the target user group information to which the target user belongs, a target execution machine operating environment that has a binding relationship with the target user group information is determined, and the to-be-processed big data task is scheduled to the target execution machine operating environment.

3. The system according to claim 2, characterized in that The task scheduling module includes: a virtual hosting submodule, wherein the virtual hosting submodule is used to: Determine the idle execution time existing in the target execution machine running environment according to the task running time of each large data task in the target execution machine running environment, wherein no large data task is executed during the idle execution time; According to the execution priority of each of the big data tasks, the execution time of the low-priority big data tasks is modified to run during the idle execution time.

4. The system according to claim 3, characterized in that The virtual hosting submodule is also used for: Constructing an execution timeline, and determining an execution start time node and an execution end time node of each of the big data tasks on the execution timeline; According to the time distribution of each of the big data tasks on the execution timeline, determine whether there is an overlapping interval, and if so, merge the execution time intervals of the big data tasks with overlapping intervals to obtain a merged execution interval; According to the execution intervals of the big data tasks without overlapping intervals and the merged execution intervals, they are sorted according to the execution order of the execution timeline, and the idle execution time is determined based on the sorting result.

5. The system according to claim 4, characterized in that The virtual hosting submodule is also used for: The sorting results are traversed, and if the start time of the next execution interval is greater than the end time of the current execution interval, the time interval between the next execution interval and the current execution interval is determined as the idle execution time.

6. A big data task scheduling method, characterized in that: The big data task scheduling method is applied to the big data task scheduling system as claimed in any one of claims 1 to 5, and the method comprises: Receive an execution machine configuration script, wherein the execution machine configuration script carries user information and execution machine parameter information, wherein the user information includes user group information to which the user belongs; Parse the execution machine configuration script, and request the database to obtain the basic image information corresponding to the execution machine parameter information, wherein the database stores the execution machine information, user information, execution machine image information, and execution machine status information, wherein the execution machine image information includes: the execution machine basic image name, and the execution machine basic image version; Generate an execution machine operating environment based on the user group information to which the user belongs and the execution machine parameter information, bind the big data tasks initiated by each user in the user group information to which the user belongs to the execution machine operating environment, and start the execution machine operating environment to run; Receiving a big data task to be processed initiated by a target user, wherein the big data task to be processed carries user information of the target user; According to the target user group information to which the target user belongs, a target execution machine operating environment that has a binding relationship with the target user group information is determined, and the to-be-processed big data task is scheduled to the target execution machine operating environment.

7. The method according to claim 6, characterized in that The method further comprises: Determine the idle execution time existing in the target execution machine running environment according to the task running time of each large data task in the target execution machine running environment, wherein no large data task is executed during the idle execution time; According to the execution priority of each of the big data tasks, the execution time of the low-priority big data tasks is modified to run during the idle execution time.

8. The method according to claim 7, characterized in that The method further comprises: Constructing an execution timeline, and determining an execution start time node and an execution end time node of each of the big data tasks on the execution timeline; According to the time distribution of each of the big data tasks on the execution timeline, determine whether there is an overlapping interval, and if so, merge the execution time intervals of the big data tasks with overlapping intervals to obtain a merged execution interval; Sorting the execution intervals of the big data tasks without overlapping intervals and the merged execution intervals according to the execution order of the execution timeline, and determining the idle execution time based on the sorting result; The sorting results are traversed, and if the start time of the next execution interval is greater than the end time of the current execution interval, the time interval between the next execution interval and the current execution interval is determined as the idle execution time.

9. An electronic device, characterized in that: The electronic device comprises: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 6 to 8.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 6-8.

Images