Data processing scheduling method, system, equipment, medium and product

Abstract

The invention discloses a data processing scheduling method, system and device, a medium and a product. The method comprises the following steps: in response to at least one timing data processing task established by a user, executing each timing data processing task; the data processing result corresponding to each timing data processing task is updated to a cache, and the data processing result corresponding to each timing data processing task is deleted from the cache in a timing mode; in response to a data query request sent by a user, if a data processing result corresponding to the data query request exists in the cache, outputting the data processing result corresponding to the data query request in the cache; and if the data processing result corresponding to the data query request does not exist in the cache, executing a data processing task corresponding to the data query request through the online interface to obtain a data processing result corresponding to the data processing task, and outputting the data processing result corresponding to the data processing task. According to the technical scheme, the data processing efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing scheduling method, system, device, medium and product. Background Art

[0002] In today's big data era, everything is connected, and the business data generated by banks is increasing.

[0003] Data processing and scheduling have become an indispensable part of daily banking work.

[0004] Traditional data processing scheduling methods or systems have the problem of low data processing efficiency. Summary of the Invention

[0005] The present disclosure provides a data processing scheduling method, system, device, medium and product, which improve data processing efficiency.

[0006] According to one aspect of the present disclosure, a data processing scheduling method is provided, comprising:

[0007] In response to at least one scheduled data processing task established by a user, executing each scheduled data processing task to obtain data processing results corresponding to each scheduled data processing task;

[0008] updating the data processing results corresponding to each scheduled data processing task into a cache, and regularly deleting the data processing results corresponding to each scheduled data processing task from the cache;

[0009] In response to a data query request sent by a user, determining whether there is a data processing result corresponding to the data query request in the cache;

[0010] If the data processing result corresponding to the data query request exists in the cache, then the data processing result corresponding to the data query request in the cache is output;

[0011] If the data processing result corresponding to the data query request does not exist in the cache, the data processing task corresponding to the data query request is executed through the online interface to obtain the data processing result corresponding to the data processing task, and the data processing result corresponding to the data processing task is output.

[0012] According to another aspect of the present disclosure, a data processing scheduling system is provided, comprising:

[0013] a data management and processing module, configured to respond to at least one scheduled data processing task established by a user and send each scheduled data processing task to the data processing scheduling module;

[0014] A data processing scheduling module is used to execute each scheduled data processing task and obtain the data processing results corresponding to each scheduled data processing task;

[0015] The data aging processing module is used to update the data processing results corresponding to each of the scheduled data processing tasks to the cache, and regularly delete the data processing results corresponding to each of the scheduled data processing tasks from the cache; in response to a data query request sent by a user, determine whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, output the data processing result corresponding to the data query request in the cache; if there is no data processing result corresponding to the data query request in the cache, execute the data processing task corresponding to the data query request through the online interface, obtain the data processing result corresponding to the data processing task, and output the data processing result corresponding to the data processing task.

[0016] According to another aspect of the present disclosure, an electronic device is provided, comprising:

[0017] at least one processor;

[0018] and a memory communicatively coupled to the at least one processor;

[0019] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data processing scheduling method described in any embodiment of the present disclosure.

[0020] According to another aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing scheduling method described in any embodiment of the present disclosure when executed.

[0021] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the data processing scheduling method as described in any one of the embodiments of the present disclosure.

[0022] The technical solution of the embodiment of the present disclosure is to execute each scheduled data processing task in response to at least one scheduled data processing task established by the user, obtain the data processing results corresponding to each scheduled data processing task; update the data processing results corresponding to each scheduled data processing task to the cache, and regularly delete the data processing results corresponding to each scheduled data processing task from the cache; respond to the data query request sent by the user, determine whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, output the data processing result corresponding to the data query request in the cache; if there is no data processing result corresponding to the data query request in the cache, execute the data processing task corresponding to the data query request through the online interface, obtain the data processing result corresponding to the data processing task, and output the data processing result corresponding to the data processing task. The above technical solution, by regularly deleting the data processing results corresponding to each scheduled data processing task from the cache, realizes the deletion of data that exceeds the retention period, ensuring the timeliness and correctness of the data in the cache. By removing the data processing results corresponding to the data query request from the cache, the response speed of the user's data query request is improved, thereby improving the data processing efficiency. The data processing tasks corresponding to the data query request are executed through the online interface, which realizes the acquisition of data in the case of data expiration and ensures the output of data corresponding to the data query request.

[0023] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 is a flow chart of a data processing scheduling method provided according to an embodiment of the present disclosure;

[0026] Figure 2 is a flowchart of another data processing scheduling method provided according to an embodiment of the present disclosure;

[0027] Figure 3 is a flowchart of another data processing scheduling method provided according to an embodiment of the present disclosure;

[0028] Figure 4is a structural diagram of a data processing scheduling system provided according to an embodiment of the present disclosure;

[0029] Figure 5 is a structural diagram of another data processing scheduling system provided according to an embodiment of the present disclosure;

[0030] Figure 6 It is a structural diagram of an electronic device that implements the data processing scheduling method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the embodiments described are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present disclosure.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. The acquisition, storage, use, processing, etc. of data in the technical solution of the present disclosure comply with the relevant provisions of national laws and regulations.

[0033] The following is a further detailed description of a data processing scheduling method, system, device, medium, and product provided by the embodiments of the present disclosure.

[0034] Figure 1 This is a flow chart of a data processing and scheduling method provided by an embodiment of the present disclosure. This embodiment is applicable to the case of processing and scheduling business data from multiple data sources of a bank. The method can be executed by a data processing and scheduling system. The data processing and scheduling system can be implemented in the form of hardware and / or software. The data processing and scheduling system can be configured in electronic devices such as terminals and servers. Figure 1 As shown, the method includes:

[0035] S110 . In response to at least one scheduled data processing task established by a user, execute each scheduled data processing task to obtain a data processing result corresponding to each scheduled data processing task.

[0036] Scheduled data processing tasks are tasks that automatically process banking business data at preset time points. The execution time of each scheduled data processing task can be the same or different, and there is no limitation here. Banking business data may include, but is not limited to, heterogeneous data such as fund product sales data, bank deposit data, and bank loan data. Data processing can include processing methods such as banking business data cleansing, banking business data calculation, and multivariate data fusion. The data processing result is the result of executing the data processing task.

[0037] For example, the scheduled data processing task may be to obtain yesterday's total fund product sales across all channels at 00:00 every day; the data processing result may be obtained by adding up yesterday's fund product sales across all channels.

[0038] S120: Update the data processing results corresponding to each scheduled data processing task into a cache, and regularly delete the data processing results corresponding to each scheduled data processing task from the cache.

[0039] It's important to note that updating the data processing results corresponding to each scheduled data processing task to the cache allows for rapid response to user data query requests and the return of corresponding data processing results, thereby improving data processing efficiency. By periodically deleting the data processing results corresponding to each scheduled data processing task from the cache, data that has exceeded its retention period is deleted, ensuring the timeliness and accuracy of the data in the cache.

[0040] S130. In response to a data query request sent by a user, determine whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, output the data processing result corresponding to the data query request in the cache; if there is no data processing result corresponding to the data query request in the cache, execute the data processing task corresponding to the data query request through the online interface, obtain the data processing result corresponding to the data processing task, and output the data processing result corresponding to the data processing task.

[0041] Specifically, the data query request can be searched in the cache. If a data processing result matching the data query request is found, it indicates that the data processing result corresponding to the data query request exists in the cache; if no data processing result matching the data query request is found, it indicates that the data processing result corresponding to the data query request does not exist in the cache.

[0042] For example, a user can input a fund product sales query request in the operation interface of the terminal device. If the total fund product sales volume corresponding to the fund product sales query request exists in the cache, the total fund product sales volume in the cache will be output; if the total fund product sales volume does not exist in the cache, the data processing task corresponding to the fund product sales query request will be executed through the online interface to calculate the total fund product sales volume and output the total fund product sales volume.

[0043] The technical solution of the embodiment of the present disclosure is to execute each scheduled data processing task in response to at least one scheduled data processing task established by the user, obtain the data processing results corresponding to each scheduled data processing task; update the data processing results corresponding to each scheduled data processing task to the cache, and regularly delete the data processing results corresponding to each scheduled data processing task from the cache; respond to the data query request sent by the user, determine whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, output the data processing result corresponding to the data query request in the cache; if there is no data processing result corresponding to the data query request in the cache, execute the data processing task corresponding to the data query request through the online interface, obtain the data processing result corresponding to the data processing task, and output the data processing result corresponding to the data processing task. The above technical solution, by regularly deleting the data processing results corresponding to each scheduled data processing task from the cache, realizes the deletion of data that exceeds the retention period, ensuring the timeliness and correctness of the data in the cache. By removing the data processing results corresponding to the data query request from the cache, the response speed of the user's data query request is improved, thereby improving the data processing efficiency. The data processing tasks corresponding to the data query request are executed through the online interface, which realizes the acquisition of data in the case of data expiration and ensures the output of data corresponding to the data query request.

[0044] Figure 2 This is a flow chart of another data processing scheduling method provided by an embodiment of the present disclosure. The method of this embodiment can be combined with the various optional solutions of the data processing scheduling method provided in the above embodiments. Based on the above embodiments, this embodiment further refines "execution of each scheduled data processing task."

[0045] like Figure 2 As shown, the method includes:

[0046] S210. In response to at least one scheduled data processing task established by the user, determine the execution time of each scheduled data processing task; if the execution times of the various scheduled data processing tasks are different, execute the various scheduled data processing tasks in sequence according to the execution times of the various scheduled data processing tasks; if there are two or more scheduled data processing tasks with the same execution time, determine the task scheduling priority of the two or more scheduled data processing tasks with the same execution time, and execute the two or more scheduled data processing tasks with the same execution time in sequence according to the task scheduling priority.

[0047] The execution time refers to the time at which the task is executed and can be pre-set, for example, 00:00 daily. The task scheduling priority refers to the priority of task execution. For example, if the first scheduled data processing task and the second scheduled data processing task have the same execution time and the first scheduled data processing task has a higher priority than the second scheduled data processing task, the first scheduled data processing task will be executed before the second scheduled data processing task.

[0048] For example, if the execution times of the scheduled data processing tasks are different, this indicates that there will be no conflict in the execution of the scheduled data processing tasks, and the scheduled data processing tasks can be executed sequentially in chronological order according to their execution times. If there are two or more scheduled data processing tasks with the same execution time, this indicates that there are scheduled data processing tasks with conflicting execution times. The task scheduling priority of the scheduled data processing tasks with conflicting execution times can be obtained, and then the two or more scheduled data processing tasks with the same execution time can be executed sequentially according to the task scheduling priority, thereby achieving ordered execution of the scheduled data processing tasks with the same execution time.

[0049] It should be noted that the present disclosure implements the sorted execution of scheduled data processing tasks with the same execution time by introducing task scheduling priorities, which can avoid task execution congestion and thus improve data scheduling efficiency.

[0050] S220: Update the data processing results corresponding to each scheduled data processing task into a cache, and regularly delete the data processing results corresponding to each scheduled data processing task from the cache.

[0051] S230. In response to a data query request sent by a user, determine whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, output the data processing result corresponding to the data query request in the cache; if there is no data processing result corresponding to the data query request in the cache, execute the data processing task corresponding to the data query request through the online interface, obtain the data processing result corresponding to the data processing task, and output the data processing result corresponding to the data processing task.

[0052] The technical solution of the embodiment of the present disclosure realizes the ordered execution of scheduled data processing tasks with the same execution time by introducing task scheduling priority, which can avoid task execution congestion and thus improve data scheduling efficiency.

[0053] Figure 3 This is a flow chart of another data processing scheduling method provided by an embodiment of the present disclosure. The method of this embodiment can be combined with the various optional solutions in the data processing scheduling method provided in the above embodiments. Based on the above embodiments, this embodiment adds a step of updating the task scheduling priority.

[0054] like Figure 3 As shown, the method includes:

[0055] S310 . In response to at least one scheduled data processing task established by a user, execute each scheduled data processing task to obtain a data processing result corresponding to each scheduled data processing task.

[0056] S320: Update the data processing results corresponding to each scheduled data processing task into a cache, and regularly delete the data processing results corresponding to each scheduled data processing task from the cache.

[0057] S330. Obtain a user query log in response to a data query request sent by a user, wherein the user query log includes a data access frequency, a data access duration, a data importance parameter, and a data proportion parameter of the query data; determine a current task scheduling priority of the query data based on the data access frequency, the data access duration, the data importance parameter, and the data proportion parameter of the query data; and update a historical task scheduling priority of the query data based on the current task scheduling priority of the query data.

[0058] Among them, query data refers to the data to be queried by the data query request, for example, the data query request can be a fund product sales query request, and the query data is fund product sales data. The data access frequency refers to the number of times the query data is accessed per unit time. The data access duration refers to the duration of each access to the query data. The data importance parameter refers to the importance of each type of data in the query data. The query data may include key data, general data, and non-important data, and is not specifically limited here. The importance can be a score or an indicator, for example, the importance of key data can be 10, and the importance of general data can be 5. The classification coefficient represents the proportion of data of different data importance in the query data, for example, the classification coefficient of key data can be 0.6, the classification coefficient of general data can be 0.3, and the classification coefficient of non-important data can be 0.1.

[0059] Specifically, the data access frequency, data access duration, data importance parameter and data proportion parameter of the query data can be obtained from the user query log, and then the current task scheduling priority of the query data can be calculated based on the data access frequency, data access duration, data importance parameter and data proportion parameter of the query data. Then, based on the current task scheduling priority of the query data, the historical task scheduling priority of the query data is updated, thereby realizing the triggering update of the task scheduling priority and ensuring the accuracy of the task scheduling priority.

[0060] Based on the above embodiment, optionally, the data importance parameter includes the importance of the first type of data in the query data, the importance of the second type of data in the query data, and the importance of the third type of data in the query data; the data proportion parameter includes the proportion of the first type of data in the query data, the proportion of the second type of data in the query data, and the proportion of the third type of data in the query data; the importance of the first type of data is greater than the importance of the second type of data, and the importance of the second type of data is greater than the importance of the third type of data; accordingly, the current task scheduling priority of the query data is determined based on the data access frequency, data access duration, data importance parameter and data proportion parameter of the query data, including: determining the current task scheduling priority of the query data based on the data access frequency, the data access duration, the importance of the first type of data in the query data, the importance of the second type of data in the query data, the importance of the third type of data in the query data, the proportion of the first type of data in the query data, the proportion of the second type of data in the query data, and the proportion of the third type of data in the query data.

[0061] For example, the data access frequency of the query data can be 10 times per minute, the data access duration of the query data can be 2 seconds, the importance of the first type of data represents the importance of critical data and can be 10, the importance of the second type of data represents the importance of general data and can be 5, and the importance of the third type of data represents the importance of non-important data and can be 1. The classification coefficient of critical data can be 0.6, the classification coefficient of general data can be 0.3, and the classification coefficient of non-important data can be 0.1. Furthermore, the current task scheduling priority of the query data can be calculated based on the above data.

[0062] The formula for determining the current task scheduling priority of query data can be:

[0063]

[0064] Where η represents the current task scheduling priority of the query data, f represents the data access frequency of the query data, t represents the data access time of the query data, and D i Indicates the importance of the i-th type of data in the query data, C i Indicates the proportion of the i-th type of data in the query data.

[0065] S340. In response to a data query request sent by a user, determine whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, output the data processing result corresponding to the data query request in the cache; if there is no data processing result corresponding to the data query request in the cache, execute the data processing task corresponding to the data query request through the online interface, obtain the data processing result corresponding to the data processing task, and output the data processing result corresponding to the data processing task.

[0066] The technical solution of the embodiment of the present disclosure obtains the current task scheduling priority of the query data by calculation, and then updates the historical task scheduling priority of the query data according to the current task scheduling priority of the query data, thereby realizing the triggering update of the task scheduling priority and ensuring the accuracy of the task scheduling priority.

[0067] Figure 4 This is a structural diagram of a data processing and scheduling system provided by an embodiment of the present disclosure. Figure 4 As shown, the system includes:

[0068] The data management and processing module 410 is configured to respond to at least one scheduled data processing task established by a user and send each scheduled data processing task to the data processing scheduling module;

[0069] The data processing scheduling module 420 is used to execute each scheduled data processing task and obtain the data processing results corresponding to each scheduled data processing task;

[0070] The data aging processing module 430 is used to update the data processing results corresponding to each of the scheduled data processing tasks to the cache, and regularly delete the data processing results corresponding to each of the scheduled data processing tasks from the cache; in response to a data query request sent by a user, determine whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, output the data processing result corresponding to the data query request in the cache; if there is no data processing result corresponding to the data query request in the cache, execute the data processing task corresponding to the data query request through the online interface, obtain the data processing result corresponding to the data processing task, and output the data processing result corresponding to the data processing task.

[0071] Among them, the data management and processing module 410 can centrally and uniformly manage data from different data sources, and can establish scheduled data processing tasks based on data processing requirements to provide support for the subsequent data processing scheduling module 420 and data aging processing module 430.

[0072] In order to improve the automation level of the entire data processing scheduling system, the present disclosure introduces a data processing scheduling module 420. The data processing scheduling module 420 can regularly execute the scheduled data processing tasks to be processed according to a preset timing scheduling strategy.

[0073] In some embodiments, the data processing scheduling module 420 may use the execution time and task scheduling priority set by the user to schedule tasks. For example, scheduled data processing tasks with the same execution time may be sorted according to the task scheduling priority.

[0074] The data aging processing module 430 can perform aging processing on the data processing results in the cache. For example, it can regularly update or clean up data processing results for different types of data processing results, that is, it can regularly delete or archive and update data that exceeds the retention period to ensure the timeliness and accuracy of the data.

[0075] In some embodiments, the data processing scheduling system also includes a data online processing module, which can receive and execute the data processing tasks sent by the data aging processing module 430, obtain the data processing results corresponding to the data processing tasks, and return the data processing results corresponding to the data processing tasks to the data aging processing module 430.

[0076] The technical solution of the embodiment of the present disclosure is to execute each scheduled data processing task in response to at least one scheduled data processing task established by the user, obtain the data processing results corresponding to each scheduled data processing task; update the data processing results corresponding to each scheduled data processing task to the cache, and regularly delete the data processing results corresponding to each scheduled data processing task from the cache; respond to the data query request sent by the user, determine whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, output the data processing result corresponding to the data query request in the cache; if there is no data processing result corresponding to the data query request in the cache, execute the data processing task corresponding to the data query request through the online interface, obtain the data processing result corresponding to the data processing task, and output the data processing result corresponding to the data processing task. The above technical solution, by regularly deleting the data processing results corresponding to each scheduled data processing task from the cache, realizes the deletion of data that exceeds the retention period, ensuring the timeliness and correctness of the data in the cache. By removing the data processing results corresponding to the data query request from the cache, the response speed of the user's data query request is improved, thereby improving the data processing efficiency. The data processing tasks corresponding to the data query request are executed through the online interface, which realizes the acquisition of data in the case of data expiration and ensures the output of data corresponding to the data query request.

[0077] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the data processing scheduling module may further be specifically configured to:

[0078] Determine the execution time of each scheduled data processing task;

[0079] If the execution time of each scheduled data processing task is different, then each scheduled data processing task is executed in sequence according to the execution time of each scheduled data processing task;

[0080] If there are two or more scheduled data processing tasks with the same execution time, the task scheduling priorities of the two or more scheduled data processing tasks with the same execution time are determined, and the two or more scheduled data processing tasks with the same execution time are executed in sequence according to the task scheduling priorities.

[0081] On the basis of any optional technical solution in the embodiments of the present disclosure, optionally, a data heat evaluation module is used to obtain a user query log in response to a data query request sent by a user, wherein the user query log includes a data access frequency, data access duration, data importance parameter and data proportion parameter of the query data; determine the current task scheduling priority of the query data based on the data access frequency, data access duration, data importance parameter and data proportion parameter of the query data; and update the historical task scheduling priority of the query data based on the current task scheduling priority of the query data.

[0082] In the present disclosure, the data heat evaluation module is used to count the query heat of the query data and characterize the query heat of the query data as a task scheduling priority for use by the data processing scheduling module 420.

[0083] Based on any optional technical solution in the embodiments of the present disclosure, optionally, the data importance parameter includes the importance of the first type of data in the query data, the importance of the second type of data in the query data, and the importance of the third type of data in the query data;

[0084] The data proportion parameter includes the proportion of the first type of data in the query data, the proportion of the second type of data in the query data, and the proportion of the third type of data in the query data;

[0085] The importance of the first type of data is greater than the importance of the second type of data, and the importance of the second type of data is greater than the importance of the third type of data;

[0086] Accordingly, the data heat evaluation module is also used to:

[0087] Determine the current task scheduling priority of the query data based on the data access frequency, the data access duration, the importance of the first type of data in the query data, the importance of the second type of data in the query data, the importance of the third type of data in the query data, the proportion of the first type of data in the query data, the proportion of the second type of data in the query data, and the proportion of the third type of data in the query data.

[0088] Based on any optional technical solution in the embodiments of the present disclosure, optionally, a formula for determining the current task scheduling priority of the query data is:

[0089]

[0090] Where η represents the current task scheduling priority of the query data, f represents the data access frequency of the query data, t represents the data access time of the query data, and D i Indicates the importance of the i-th type of data in the query data, C i Indicates the proportion of the i-th type of data in the query data.

[0091] For example, Figure 5 This is a schematic diagram of the structure of a data processing and scheduling system provided by an embodiment of the present disclosure. The data processing and scheduling system includes a data management and processing module, a data processing and scheduling module, a data aging processing module, a data online processing module, and a data heat evaluation module.

[0092] The data management and processing module is used to respond to at least one scheduled data processing task established by the user and send each scheduled data processing task to the data processing scheduling module.

[0093] The data processing scheduling module is used to execute each scheduled data processing task and obtain the data processing results corresponding to each scheduled data processing task. The data aging processing module is used to update the data processing results corresponding to each scheduled data processing task to the cache, and regularly delete the data processing results corresponding to each scheduled data processing task from the cache; in response to the data query request sent by the user terminal, it is determined whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, the data processing result corresponding to the data query request in the cache is output; if there is no data processing result corresponding to the data query request in the cache, the data online processing module is called through the online interface to execute the data processing task corresponding to the data query request, and the data processing result corresponding to the data processing task is obtained. The data online processing module feeds back the data processing result corresponding to the data processing task to the data aging processing module.

[0094] The data heat evaluation module is used to respond to a data query request sent through a user terminal and obtain a user query log, which includes the data access frequency, data access duration, data importance parameter and data proportion parameter of the query data; determine the current task scheduling priority of the query data based on the data access frequency, data access duration, data importance parameter and data proportion parameter of the query data; and update the historical task scheduling priority of the query data based on the current task scheduling priority of the query data.

[0095] The present disclosure comprehensively considers three factors: data aging, query popularity and timed scheduling, and optimizes data processing scheduling, which can not only improve data processing efficiency but also ensure the stability and reliability of the data processing scheduling system.

[0096] The data processing scheduling system provided by the embodiments of the present disclosure can execute the data processing scheduling method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0097] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0098] like Figure 6 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 and a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. Various programs and data required for the operation of the electronic device 10 can also be stored in the RAM 13. The processor 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An I / O interface 15 is also connected to the bus 14.

[0099] Multiple components in the electronic device 10 are connected to the I / O interface 15, including an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0100] The processor 11 may be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a data processing scheduling method, which includes:

[0101] In response to at least one scheduled data processing task established by a user, executing each scheduled data processing task to obtain data processing results corresponding to each scheduled data processing task;

[0102] updating the data processing results corresponding to each scheduled data processing task into a cache, and regularly deleting the data processing results corresponding to each scheduled data processing task from the cache;

[0103] In response to a data query request sent by a user, determine whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, output the data processing result corresponding to the data query request in the cache; if there is no data processing result corresponding to the data query request in the cache, execute the data processing task corresponding to the data query request through the online interface, obtain the data processing result corresponding to the data processing task, and output the data processing result corresponding to the data processing task.

[0104] In some embodiments, the data processing scheduling method can be implemented as a computer program that is tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data processing scheduling method described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the data processing scheduling method in any other appropriate manner (e.g., by means of firmware).

[0105] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] Computer programs for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when the computer program is executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer program may be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0107] In the context of the present disclosure, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0109] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0110] A computing system may include clients and servers. The clients and servers are typically remote from each other and typically interact via a communication network. This client-server relationship arises through computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host. This server is a hosting product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosting and VPS services.

[0111] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions of this disclosure can be achieved, and this document is not limited here.

[0112] An embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements the data processing scheduling method provided in any embodiment of the present disclosure.

[0113] The computer program product, during implementation, may be written in one or more programming languages, or a combination thereof, for performing the operations of the present disclosure and may include computer program code written in object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0114] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication unit 19, or installed from the storage unit 18, or installed from the ROM 12. When the computer program is executed by the processor 11, the above-mentioned functions defined in the method of the embodiment of the present invention are performed.

[0115] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A data processing scheduling method, characterized in that: include: In response to at least one scheduled data processing task established by a user, executing each scheduled data processing task to obtain data processing results corresponding to each scheduled data processing task; updating the data processing results corresponding to each scheduled data processing task into a cache, and regularly deleting the data processing results corresponding to each scheduled data processing task from the cache; In response to a data query request sent by a user, determining whether there is a data processing result corresponding to the data query request in the cache; If the data processing result corresponding to the data query request exists in the cache, then the data processing result corresponding to the data query request in the cache is output; If the data processing result corresponding to the data query request does not exist in the cache, the data processing task corresponding to the data query request is executed through the online interface to obtain the data processing result corresponding to the data processing task, and the data processing result corresponding to the data processing task is output.

2. The method according to claim 1, characterized in that The execution of each scheduled data processing task includes: Determine the execution time of each scheduled data processing task; If the execution time of each scheduled data processing task is different, then each scheduled data processing task is executed in sequence according to the execution time of each scheduled data processing task; If there are two or more scheduled data processing tasks with the same execution time, the task scheduling priorities of the two or more scheduled data processing tasks with the same execution time are determined, and the two or more scheduled data processing tasks with the same execution time are executed in sequence according to the task scheduling priorities.

3. The method according to claim 1, characterized in that After responding to the data query request sent by the user, the method further includes: Obtaining a user query log, wherein the user query log includes a data access frequency, a data access duration, a data importance parameter, and a data proportion parameter of the query data; Determine the current task scheduling priority of the query data based on the data access frequency, data access duration, data importance parameter and data proportion parameter of the query data; Based on the current task scheduling priority of the query data, the historical task scheduling priority of the query data is updated.

4. The method according to claim 3, characterized in that The data importance parameters include the importance of the first type of data in the query data, the importance of the second type of data in the query data, and the importance of the third type of data in the query data; The data proportion parameter includes the proportion of the first type of data in the query data, the proportion of the second type of data in the query data, and the proportion of the third type of data in the query data; The importance of the first type of data is greater than the importance of the second type of data, and the importance of the second type of data is greater than the importance of the third type of data; Accordingly, determining the current task scheduling priority of the query data based on the data access frequency, data access duration, data importance parameter, and data proportion parameter of the query data includes: Determine the current task scheduling priority of the query data based on the data access frequency, the data access duration, the importance of the first type of data in the query data, the importance of the second type of data in the query data, the importance of the third type of data in the query data, the proportion of the first type of data in the query data, the proportion of the second type of data in the query data, and the proportion of the third type of data in the query data.

5. The method according to claim 4, characterized in that The formula for determining the current task scheduling priority of the query data is: Where η represents the current task scheduling priority of the query data, f represents the data access frequency of the query data, t represents the data access time of the query data, and D i Indicates the importance of the i-th type of data in the query data, C i Indicates the proportion of the i-th type of data in the query data.

6. A data processing scheduling system, characterized in that: include: a data management and processing module, configured to respond to at least one scheduled data processing task established by a user and send each scheduled data processing task to the data processing scheduling module; A data processing scheduling module is used to execute each scheduled data processing task and obtain the data processing results corresponding to each scheduled data processing task; The data aging processing module is used to update the data processing results corresponding to each of the scheduled data processing tasks to the cache, and regularly delete the data processing results corresponding to each of the scheduled data processing tasks from the cache; in response to a data query request sent by a user, determine whether there is a data processing result corresponding to the data query request in the cache; if there is a data processing result corresponding to the data query request in the cache, output the data processing result corresponding to the data query request in the cache; if there is no data processing result corresponding to the data query request in the cache, execute the data processing task corresponding to the data query request through the online interface, obtain the data processing result corresponding to the data processing task, and output the data processing result corresponding to the data processing task.

7. The system according to claim 6, characterized in that The system also includes: A data heat evaluation module is used to respond to a data query request sent by a user and obtain a user query log, wherein the user query log includes a data access frequency, a data access duration, a data importance parameter, and a data proportion parameter of the query data; determine the current task scheduling priority of the query data based on the data access frequency, the data access duration, the data importance parameter, and the data proportion parameter of the query data; and update the historical task scheduling priority of the query data based on the current task scheduling priority of the query data.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the data processing scheduling method according to any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the data processing scheduling method according to any one of claims 1 to 5 when executed.

10. A computer program product, characterized in that The computer program product comprises a computer program, which, when executed by a processor, implements the data processing scheduling method according to any one of claims 1 to 5.

Images