Data processing task execution method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data processing task execution method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining business data in a historical time period from a data warehouse; the business data in the data warehouse is obtained by reading a plurality of databases in parallel, and the plurality of databases are respectively used for storing the business data generated by the corresponding resource interaction mechanisms; determining a task theme associated with the business data, and querying a corresponding data processing task based on the task theme; the task theme is used for pre-creating a corresponding data processing task based on an analysis demand of the associated business data; and based on the idle resources of the servers in the server cluster, scheduling the data processing task to the server with the largest idle resources for execution, and outputting an analysis result for the service data. By adopting the method, the analysis efficiency of the business data generated by the resource interaction mechanism can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for executing a data processing task. Background Art

[0002] In the retail industry, especially chain retail, enterprises are facing increasingly fierce market competition and rapid changes in consumer demand. Therefore, it is particularly important to analyze business data in a timely and accurate manner and adjust operational strategies. To this end, enterprises need to regularly perform a series of data processing tasks (for example, issue important report data on time every day) to analyze their business performance.

[0003] However, these data processing tasks involve a large number of stores. With the rapid expansion of stores, business data (including sales records, inventory information, customer behavior, etc.) has shown a sharp growth trend. Business data is not only diverse in type, but also has the characteristics of high frequency and real-time. The current general big data platform cannot meet the needs of retail business data analysis. Summary of the invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for executing data processing tasks that can improve the efficiency of analyzing business data generated by resource interaction organizations in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for executing a data processing task, comprising:

[0006] Acquire business data within a historical time period from a data warehouse; the business data in the data warehouse is obtained based on parallel reading of multiple databases, and the multiple databases are respectively used to store business data generated by corresponding resource interaction mechanisms;

[0007] Determine a task subject associated with the business data, and query a corresponding data processing task based on the task subject; the task subject is used to pre-create a corresponding data processing task based on the analysis requirements of the associated business data;

[0008] Based on the idle resources of the servers in the server cluster, the data processing task is scheduled to the server with the largest idle resources for execution, and the analysis result for the business data is output.

[0009] In one embodiment, before acquiring the business data in the historical time period from the data warehouse, the method includes:

[0010] The business data generated by multiple resource interaction agencies are stored in corresponding databases respectively;

[0011] A corresponding data collection process is created for each of the databases; the data collection process is used to establish a connection with the corresponding database, collect and send the business data in the database to the data warehouse;

[0012] The business data in the database is segmented by using a single database and table division method to obtain multiple data tables corresponding to the database;

[0013] The data collection process is split into multiple threads for execution; the multiple threads are respectively used to collect and send the business data in the multiple data tables to the data warehouse.

[0014] In one embodiment, splitting the data acquisition process into multiple threads for execution includes:

[0015] Get the single-table concurrent number corresponding to the data table;

[0016] Determine the data filtering condition corresponding to the data table based on the concurrent number of the single table;

[0017] The data collection process is split into multiple threads, and based on the data filtering condition, the thread is split into multiple sub-threads for execution.

[0018] In one embodiment, based on the idle resources of the servers in the server cluster, the data processing task is scheduled to the server with the largest idle resources for execution, and the analysis result for the business data is output, which includes:

[0019] Obtaining execution priorities of the plurality of data processing tasks;

[0020] The plurality of data processing tasks are sent to the server cluster in descending order of execution priority.

[0021] In one embodiment, the server cluster includes a first server and a second server, the first server is used to provide services for corresponding tenants, multiple first servers are isolated from each other, and the second server is used to provide public services; based on the idle resources of the servers in the server cluster, the data processing task is scheduled to the server with the largest idle resources for execution, and the analysis results for the business data are output, including:

[0022] Based on the idle resources of the first server corresponding to the current tenant, the data processing task is scheduled to the first server with the largest idle resources for execution, and the analysis result of the business data is output;

[0023] The method further comprises:

[0024] When the idle resources of the first server are full, the data processing task is scheduled to the second server for execution, and the analysis result of the business data is output.

[0025] In one embodiment, based on the idle resources of the servers in the server cluster, the data processing task is scheduled to the server with the largest idle resources for execution, and the analysis result for the business data is output, which includes:

[0026] Replacing the library information corresponding to the data processing task so that the data processing task is switched to connect to the development environment for execution;

[0027] Execute the data processing task, and evaluate the execution result of the data processing task in the development environment;

[0028] When the execution result meets the preset online condition, the library information corresponding to the data processing task is restored.

[0029] In a second aspect, the present application further provides a device for executing a data processing task, comprising:

[0030] A data acquisition module, used to acquire business data within a historical time period from a data warehouse; the business data in the data warehouse is obtained based on parallel reading of multiple databases, and the multiple databases are respectively used to store business data generated by corresponding resource interaction mechanisms;

[0031] A task query module, used to determine the task subject associated with the business data, and based on the task subject, query the corresponding data processing task; the task subject is used to pre-create the corresponding data processing task based on the analysis requirements of the associated business data;

[0032] The task scheduling module is used to schedule the data processing task to the server with the largest idle resources for execution based on the idle resources of the servers in the server cluster, and output the analysis results for the business data.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any of the above-described methods when executing the computer program.

[0034] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any of the methods described above when executed by a processor.

[0035] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the steps of any of the methods described above when executed by a processor.

[0036] The above-mentioned data processing task execution method, device, computer equipment, computer-readable storage medium and computer program product, by adopting a parallel reading method to pre-store the business data generated by each resource interaction organization (such as retail stores) in a data warehouse, can ensure the real-time and accuracy of business data; by querying data processing tasks according to pre-defined task topics, it is possible to quickly locate and query the corresponding data processing tasks and clarify the data processing goals; by scheduling tasks based on the idle resources of the server, it is possible to improve the overall resource utilization of the server cluster and reduce the delay in task execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0038] FIG1 is an application environment diagram of a method for executing a data processing task in one embodiment;

[0039] FIG2 is a schematic flow chart of a method for executing a data processing task in one embodiment;

[0040] FIG3 is a data flow diagram showing how business data generated by multiple resource interaction mechanisms are stored in corresponding databases in one embodiment;

[0041] FIG4 is a data flow diagram of scheduling a data processing task to be executed on a server with the largest idle resources in one embodiment;

[0042] FIG5 is a schematic flow chart of a method for executing a data processing task in another embodiment;

[0043] 6 is a data flow diagram showing how business data generated by multiple resource interaction mechanisms are stored in corresponding databases respectively in another embodiment;

[0044] FIG7 is a schematic diagram of a flow chart of step S2014 in one embodiment;

[0045] 8 is a data flow diagram showing how business data generated by multiple resource interaction mechanisms are stored in corresponding databases respectively in yet another embodiment;

[0046] FIG9 is a schematic flow chart of a method for executing a data processing task in yet another embodiment;

[0047] FIG10 is a schematic flow chart of a method for executing a data processing task in yet another embodiment;

[0048] FIG11 is a data flow diagram showing another embodiment of scheduling a data processing task to be executed on a server with the largest idle resources;

[0049] FIG12 is a block diagram of a device for executing data processing tasks in one embodiment;

[0050] FIG. 13 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The execution method of the data processing task provided in the embodiment of the present application can be applied in the application environment as shown in Figure 1. Among them, the business data in the data warehouse 104 is obtained based on parallel reading of multiple databases 102, and the multiple databases 102 are respectively used to store the business data generated by the corresponding resource interaction mechanism. The server 106 is used to obtain the business data within the historical time period from the data warehouse 104; determine the task subject associated with the business data, and query the corresponding data processing task based on the task subject; based on the idle resources of the servers in the server cluster, schedule the data processing task to the server with the largest idle resources for execution, and output the analysis results for the business data. Among them, the database 102 can be integrated locally, or it can be placed on the cloud or other network servers. The server 106 can be an independent physical server, or it can be a server cluster or distributed system composed of multiple physical servers, or it can be a cloud server that provides cloud computing services.

[0053] In one embodiment, as shown in FIG2 , a method for executing a data processing task is provided. This embodiment uses the method applied to the server 106 in FIG1 as an example for illustration. It is understandable that the method can also be applied to a terminal, and can also be applied to a system including a server and a terminal, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0054] Step S202, obtaining business data within a historical time period from a data warehouse; the business data in the data warehouse is obtained based on parallel reading of multiple databases, and the multiple databases are respectively used to store business data generated by corresponding resource interaction mechanisms.

[0055] The data warehouse may be used to store business data generated by multiple interrelated resource interaction organizations. Multiple interrelated resource interaction organizations may be chain stores (including offline stores and online stores). Specifically, the resource interaction organization may be a retail store. Business data may include, but is not limited to, personnel-related data, goods-related data, and venue-related data.

[0056] Exemplarily, the method of reading multiple databases in parallel may include: storing the business data generated by multiple resource interaction mechanisms in corresponding databases respectively; creating a corresponding data collection process for each database; executing multiple data collection processes in parallel, respectively collecting and sending the business data in the corresponding databases to the data warehouse. For each database, a corresponding data buffer can be set to achieve reading and writing at the same time. In a possible implementation method, please refer to Figure 3, the business data generated by the resource interaction mechanism can be stored in a single table manner, each database is responsible for storing the business data of a resource interaction mechanism, and one table is used in each database to store the business data. In this way, the impact of the maximum number of database connections on the data collection efficiency can be reduced.

[0057] Optionally, business data can be stored in the form of files in hdfs (Hadoop Distributed File System) partitions (including daily partitions, monthly partitions, etc.), and the file names are named according to rules such as institution numbers. In scenarios where rerunning or supplementing is required, only business data of abnormal institutions is collected to accurately and efficiently implement data flushing.

[0058] Step S204, determine the task subject associated with the business data, and query the corresponding data processing task based on the task subject; the task subject is used to pre-create the corresponding data processing task based on the analysis requirements of the associated business data.

[0059] Among them, the task theme can be obtained based on the subject domain stratification of data processing tasks around personnel, goods, and scenarios. Specifically, the task theme may include customers, goods, inventory, traffic, sales, finance, marketing, after-sales, fulfillment, supply chain, etc. In one possible implementation, multiple task themes can be used to be displayed on a preset page, and the preset page can be used to create or find corresponding data processing tasks in response to a selection operation for a task theme. It can be understood that the same type of business data can be associated with one or more task themes, and each task theme can correspond to one or more data processing tasks.

[0060] Exemplarily, the manner of determining the task topic associated with the business data may include: analyzing the content (eg, keywords) of the business data, using a pre-trained classification model, and the like.

[0061] Step S206, based on the idle resources of the servers in the server cluster, the data processing task is scheduled to the server with the largest idle resources for execution, and the analysis result for the business data is output.

[0062] For example, referring to Figure 4, the current data processing task can be sent to the server with the least number of tasks to be executed according to the number of tasks currently being executed by each server in the server cluster. Optionally, the idle resource situation of each server can be evaluated in combination with the number of tasks to be executed and resource utilization (e.g., CPU occupancy, memory usage, disk I / O, etc.).

[0063] Optionally, you can set current limiting to read business data into the data warehouse at the maximum speed while protecting the high concurrency of the database.

[0064] In the above-mentioned method for executing data processing tasks, the business data generated by each resource interaction organization (such as retail stores) is pre-stored in the data warehouse by adopting a parallel reading method, so as to ensure the real-time and accuracy of the business data; by querying the data processing tasks according to the pre-defined task subjects, the corresponding data processing tasks can be quickly located and queried, and the data processing goals can be clarified; by scheduling tasks in combination with the idle resources of the server, the overall resource utilization of the server cluster can be improved, and the delay in task execution can be reduced.

[0065] In an exemplary embodiment, as shown in FIG5 , the method for executing the above data processing task may further include:

[0066] Step S2011, storing the business data generated by multiple resource interaction mechanisms in corresponding databases respectively.

[0067] Step S2012, creating a corresponding data collection process for each database; the data collection process is used to establish a connection with the corresponding database, collect and send the business data in the database to the data warehouse.

[0068] Step S2013, using a single database and table division method to divide the business data in the database, to obtain multiple data tables corresponding to the database.

[0069] For example, one or more databases with a large amount of single-table data in multiple databases may be divided into single-table partitions to perform data segmentation. Specifically, the partitions may be performed based on time, data volume, or business module.

[0070] Step S2014, splitting the data collection process into multiple threads for execution; the multiple threads are used to collect and send the business data in multiple data tables to the data warehouse.

[0071] For example, referring to Fig. 6, in each data collection process, the task can be further divided into multiple threads to process the data in each data table in parallel. Each thread is responsible for collecting data from different data tables.

[0072] In this embodiment, through parallel processing and table partitioning strategies, it is possible to avoid the performance bottleneck caused by excessive data volume in a single table, and improve the stability and response speed of task execution.

[0073] Further, as shown in FIG. 7 , the above step S2014 may include:

[0074] Step A1, obtain the single-table concurrent quantity corresponding to the data table.

[0075] For example, if the data volume of a data table is large and the database in which it is located also has a performance bottleneck, a single-table multi-concurrency method can be used to collect data from the data table. The number of single-table concurrencies corresponding to the data table can be determined based on the size of the data table (for example, the number of rows, the amount of data, etc.) and the structural complexity of the data table.

[0076] Step A2: Determine the data filtering condition corresponding to the data table based on the concurrent number of the single table.

[0077] Step A3: split the data collection process into multiple threads, and split the thread into multiple sub-threads for execution based on the data filtering condition.

[0078] Exemplarily, the data filtering conditions of the data table can be determined based on the number of concurrencies (for example, partitioning by time range, partitioning by data primary key, etc.), and a corresponding query statement is generated for each thread so that each thread processes a subset of the data table (as shown in FIG. 8 ).

[0079] In this embodiment, the speed of data collection can be further improved by using a single table with multiple concurrent processes.

[0080] In an exemplary embodiment, as shown in FIG9 , the method for executing the above data processing task may further include:

[0081] Step S2051, obtaining the execution priorities of multiple data processing tasks.

[0082] Step S2052: Send multiple data processing tasks to the server cluster in order of execution priority from high to low.

[0083] For example, a mechanism for allocating execution priorities can be pre-established to automatically evaluate and assign a numerical value to each data processing task. For example, if a data processing task directly affects a key business decision or operational decision, a higher execution priority can be assigned to the data processing task. The execution priority of a data processing task can also be evaluated based on the deadline, resource requirements, etc. of the data processing task.

[0084] In this embodiment, priority management is used to ensure that important tasks are executed first, thereby improving overall processing efficiency and avoiding obstruction of key businesses.

[0085] In an exemplary embodiment, the server cluster may include a first server and a second server, the first server is used to provide services for corresponding tenants, multiple first servers are isolated from each other, and the second server is used to provide public services. As shown in FIG. 10, the above step S206 may include:

[0086] Step S2061, based on the idle resources of the first server corresponding to the current tenant, the data processing task is scheduled to the first server with the largest idle resources for execution, and the analysis result of the business data is output.

[0087] The execution method of the above data processing task may also include:

[0088] Step S207, when the idle resources of the first server are full, the data processing task is scheduled to the second server for execution, and the analysis result of the business data is output.

[0089] Exemplarily, please refer to Figure 11. The first server can be used to provide services specifically for specific tenants (for example, this can be achieved by adding tags to the server) to ensure data isolation and security. Each tenant can have one or more first servers. The second server can be used to provide shared resources or services for multiple tenants. In one possible implementation, an identity and access management mechanism can be used to define and manage the permissions of users and tenants to ensure that only authorized users can access the first server corresponding to the tenant. A task scheduling mechanism can be used. When a tenant initiates a request, the corresponding first server is preferentially scheduled to execute the task; if the first server has insufficient idle resources, the request is scheduled to the second server for processing to ensure that all requests are responded to.

[0090] In this embodiment, the security of tenant data is ensured through data isolation and access management mechanisms, reducing the risk of data leakage due to resource preemption; under high load conditions, flexible scheduling to public servers can ensure that data processing tasks are processed in a timely manner.

[0091] In an exemplary embodiment, the method for executing the above data processing task may further include:

[0092] Step S2053, replacing the library information corresponding to the data processing task so that the data processing task is switched to connect to the development environment for execution.

[0093] Step S2054, execute the data processing task, and evaluate the execution result of the data processing task in the development environment.

[0094] Step S2055, when the execution result meets the preset online condition, the library information corresponding to the data processing task is restored.

[0095] Exemplarily, in response to test instance requirements, first during data collection, the data warehouse information is replaced (which may include replacing the hive library with the development library and the hdfs path with the development path), and then during data processing, the data warehouse information is replaced (which may include replacing the hive library with the development library and the hdfs path with the development path), and the job is put online after the data information is verified.

[0096] In this embodiment, by testing and verifying the data processing tasks in the development environment, the risk of unqualified tasks being put online in the production environment can be effectively reduced.

[0097] To summarize, in the above-mentioned method for executing data processing tasks, the business data generated by each resource interaction organization (such as retail stores) is pre-stored in the data warehouse by adopting a parallel reading method, so as to ensure the real-time and accuracy of the business data; by querying the data processing tasks according to the pre-defined task subjects, the corresponding data processing tasks can be quickly located and queried, and the data processing goals can be clarified; by scheduling tasks in combination with the idle resources of the server, the overall resource utilization of the server cluster can be improved, and the delay in task execution can be reduced.

[0098] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0099] Based on the same inventive concept, the embodiment of the present application also provides a data processing task execution device for implementing the data processing task execution method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the execution device for one or more data processing tasks provided below can refer to the limitations of the execution method for data processing tasks above, and will not be repeated here.

[0100] In an exemplary embodiment, as shown in FIG. 12 , a data processing task execution device 300 is provided, comprising: a data acquisition module 301, a task query module 302 and a task scheduling module 303, wherein:

[0101] The data acquisition module 301 is used to acquire business data within a historical time period from a data warehouse; the business data in the data warehouse is obtained based on parallel reading of multiple databases, and the multiple databases are respectively used to store business data generated by corresponding resource interaction mechanisms.

[0102] The task query module 302 is used to determine the task subject associated with the business data and query the corresponding data processing task based on the task subject; the task subject is used to pre-create the corresponding data processing task based on the analysis requirements of the associated business data.

[0103] The task scheduling module 303 is used to schedule the data processing task to the server with the largest idle resources for execution based on the idle resources of the servers in the server cluster, and output the analysis results for the business data.

[0104] In an exemplary embodiment, the execution device 300 of the data processing task further includes:

[0105] A data collection module is used to store the business data generated by multiple resource interaction agencies in corresponding databases;

[0106] Create a corresponding data collection process for each database; the data collection process is used to establish a connection with the corresponding database, collect and send the business data in the database to the data warehouse;

[0107] Use the single-database-table splitting method to split the business data in the database to obtain multiple data tables corresponding to the database;

[0108] The data collection process is split into multiple threads for execution; the multiple threads are used to collect and send business data from multiple data tables to the data warehouse.

[0109] In an exemplary embodiment, the data acquisition module is further used to:

[0110] Get the single-table concurrent number corresponding to the data table;

[0111] Determine the data filtering conditions corresponding to the data table based on the number of concurrent connections of a single table;

[0112] The data collection process is split into multiple threads, and the threads are split into multiple sub-threads for execution based on data filtering conditions.

[0113] In an exemplary embodiment, the task scheduling module 303 is further used to:

[0114] Get the execution priority of multiple data processing tasks;

[0115] Multiple data processing tasks are sent to the server cluster in descending order of execution priority.

[0116] In an exemplary embodiment, the server cluster includes a first server and a second server, the first server is used to provide services for the corresponding tenant, the multiple first servers are isolated from each other, and the second server is used to provide public services; the task scheduling module 303 is also used to:

[0117] Based on the idle resources of the first server corresponding to the current tenant, the data processing task is scheduled to the first server with the largest idle resources for execution, and the analysis results for the business data are output;

[0118] When the idle resources of the first server are full, the data processing task is dispatched to the second server for execution, and the analysis results of the business data are output.

[0119] In an exemplary embodiment, the execution device 300 of the data processing task further includes:

[0120] The task testing module is used to replace the library information corresponding to the data processing task so that the data processing task can be switched to connect to the development environment for execution;

[0121] Execute data processing tasks and evaluate the results of data processing tasks in the development environment;

[0122] When the execution result meets the preset online conditions, the library information corresponding to the data processing task is restored.

[0123] Each module in the execution device of the above data processing task can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0124] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG13. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for executing a data processing task is implemented.

[0125] Those skilled in the art will understand that the structure shown in FIG. 13 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0126] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0127] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0128] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0129] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0130] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0131] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0132] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for executing a data processing task, characterized in that: The method comprises: Acquire business data within a historical time period from a data warehouse; the business data in the data warehouse is obtained based on parallel reading of multiple databases, and the multiple databases are respectively used to store business data generated by corresponding resource interaction mechanisms; Determine a task subject associated with the business data, and query a corresponding data processing task based on the task subject; the task subject is used to pre-create a corresponding data processing task based on the analysis requirements of the associated business data; Based on the idle resources of the servers in the server cluster, the data processing task is scheduled to the server with the largest idle resources for execution, and the analysis result for the business data is output.

2. The method according to claim 1, characterized in that The process of acquiring the business data in the historical time period from the data warehouse includes: The business data generated by multiple resource interaction agencies are stored in corresponding databases respectively; A corresponding data collection process is created for each of the databases; the data collection process is used to establish a connection with the corresponding database, collect and send the business data in the database to the data warehouse; The business data in the database is segmented by using a single database and table division method to obtain multiple data tables corresponding to the database; The data collection process is split into multiple threads for execution; the multiple threads are respectively used to collect and send the business data in the multiple data tables to the data warehouse.

3. The method according to claim 2, characterized in that The step of splitting the data collection process into multiple threads for execution includes: Get the single-table concurrent number corresponding to the data table; Determine the data filtering condition corresponding to the data table based on the concurrent number of the single table; The data collection process is split into multiple threads, and based on the data filtering condition, the thread is split into multiple sub-threads for execution.

4. The method according to claim 1, characterized in that: Based on the idle resources of the servers in the server cluster, the data processing task is scheduled to the server with the largest idle resources for execution, and the analysis result for the business data is output, which includes: Obtaining execution priorities of the plurality of data processing tasks; The plurality of data processing tasks are sent to the server cluster in descending order of execution priority.

5. The method according to claim 1, characterized in that The server cluster includes a first server and a second server, the first server is used to provide services for corresponding tenants, a plurality of the first servers are isolated from each other, and the second server is used to provide public services; based on the idle resources of the servers in the server cluster, the data processing task is scheduled to the server with the largest idle resources for execution, and the analysis results for the business data are output, including: Based on the idle resources of the first server corresponding to the current tenant, the data processing task is scheduled to the first server with the largest idle resources for execution, and the analysis result of the business data is output; The method further comprises: When the idle resources of the first server are full, the data processing task is scheduled to the second server for execution, and the analysis result of the business data is output.

6. The method according to claim 1, characterized in that Based on the idle resources of the servers in the server cluster, the data processing task is scheduled to the server with the largest idle resources for execution, and the analysis result for the business data is output, which includes: Replacing the library information corresponding to the data processing task so that the data processing task is switched to connect to the development environment for execution; Execute the data processing task, and evaluate the execution result of the data processing task in the development environment; When the execution result meets the preset online condition, the library information corresponding to the data processing task is restored.

7. A data processing task execution device, characterized in that: The device comprises: A data acquisition module, used to acquire business data within a historical time period from a data warehouse; the business data in the data warehouse is obtained based on parallel reading of multiple databases, and the multiple databases are respectively used to store business data generated by corresponding resource interaction mechanisms; A task query module, used to determine the task subject associated with the business data, and based on the task subject, query the corresponding data processing task; the task subject is used to pre-create the corresponding data processing task based on the analysis requirements of the associated business data; The task scheduling module is used to schedule the data processing task to the server with the largest idle resources for execution based on the idle resources of the servers in the server cluster, and output the analysis results for the business data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images