Data processing method and device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BMW BRILLIANCE AUTOMOTIVE
- Filing Date
- 2024-11-21
- Publication Date
- 2026-05-22
AI Technical Summary
Traditional complex data processing relies on real-time manual interaction, resulting in low task execution efficiency and high labor costs, leading to high enterprise operating costs.
By defining the target task parameters of data processing tasks, formulating task priority strategies, and automatically allocating task execution priorities and queues, intelligent task scheduling is achieved.
It improves task execution efficiency, reduces manual intervention, enhances the accuracy and flexibility of task scheduling, and reduces operating costs.
Smart Images
Figure CN122072931A_ABST
Abstract
Description
Technical Field
[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a data processing method and apparatus. Background Technology
[0002] Traditional complex data processing often relies on real-time manual interaction, resulting in low task execution efficiency. The waiting time during this period consumes a lot of human resources, and enterprises rely on a large number of server system resources during operation, thus generating high operating costs.
[0003] With the rapid development of information technology, enterprises are facing more and more business challenges. In order to cope with these challenges, enterprises need to build an efficient, stable and intelligent data processing task scheduling system to efficiently handle various data processing tasks in the business. Summary of the Invention
[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.
[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:
[0006] A data processing task is determined, wherein the data processing task carries task parameters;
[0007] In cases where the data processing task does not carry the task execution priority, a task priority formulation strategy is determined based on the target task parameters of the data processing task.
[0008] A strategy is formulated based on the task priorities to determine the task execution priority of the data processing task;
[0009] Determine the task execution queue corresponding to the task execution priority, and execute the data processing task according to the task execution queue and the task parameters.
[0010] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:
[0011] The task acquisition module is used to determine the data processing task;
[0012] The priority specification module is used to determine the task priority formulation strategy of the data processing task based on the target task parameters of the data processing task when it is determined that there is no task execution priority of the data processing task in the data processing task.
[0013] The priority determination module is used to formulate a strategy based on the task priority and determine the task execution priority of the data processing task;
[0014] The task execution module is used to determine the task execution queue corresponding to the task execution priority, and execute the data processing task according to the task execution queue and the task parameters.
[0015] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:
[0016] Memory and processor;
[0017] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.
[0018] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the data processing method described above.
[0019] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.
[0020] This specification provides a data processing method in one embodiment. By determining a data processing task and judging whether the data processing task carries a task execution priority, if the data processing task does not carry a task execution priority, a task priority strategy can be determined based on the target task parameters of the data processing task. This strategy determines the task execution priority of the data processing task. For example, if the target task parameter is the task data volume, the task execution priority is quickly and accurately determined by comparing the task data volume with a preset data volume. The data processing task is then assigned to the task execution queue corresponding to the task execution priority and executed. This achieves automated and intelligent task scheduling, reduces manual intervention, and improves task execution efficiency. Furthermore, determining the task execution priority and the corresponding task execution queue based on the task priority strategy improves the accuracy and flexibility of task scheduling. Attached Figure Description
[0021] Figure 1 This is a schematic diagram illustrating a data processing method provided in one embodiment of this specification.
[0022] Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;
[0023] Figure 3 This is a schematic diagram of a user interface provided in one embodiment of this specification;
[0024] Figure 4 This is a system architecture diagram of a data processing system provided in one embodiment of this specification;
[0025] Figure 5 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification;
[0026] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation
[0027] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.
[0028] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0029] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."
[0030] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0031] This specification provides a data processing method, and also relates to a data processing apparatus, a computing device, and a computer-readable storage medium, which will be described in detail in the following embodiments.
[0032] See Figure 1 , Figure 1 A schematic diagram of a data processing method according to an embodiment of this specification is shown.
[0033] Specifically, the text processing method is implemented using the terminal device 102 and the server 104. The terminal device 102 is used to send task processing requests to the server 104, such as task creation requests or task editing requests.
[0034] The data processing system utilizes resources on server 104 to complete task execution. The data processing system includes multiple independent service modules, such as a task management module, a resource management module, and a scheduling engine module. The task management module receives task processing requests through a task processing interface. If the task processing request is a task creation request, it generates the data processing task based on the task creation parameters carried in the request. Alternatively, if the task processing request is a task editing request, it adjusts the initial data processing task based on the task editing parameters carried in the task editing request to generate the data processing task. The resource management module performs resource requirement analysis for the data processing task, determines the required resources for executing the data processing task, and allocates resources accordingly. The system allocates the required resources corresponding to the data processing task and reclaims the required resources after the data processing task is completed. The scheduling engine module is used to determine the data processing task, wherein the data processing task carries task parameters. If the data processing task does not carry the task execution priority, the system determines the task priority formulation strategy of the data processing task based on the target task parameters of the data processing task, determines the task execution priority of the data processing task based on the task priority formulation strategy, determines the task execution queue corresponding to the task execution priority, executes the data processing task based on the task execution queue and the task parameters, obtains the task processing result, and returns the task processing result to the end device 102.
[0035] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on the software development kit (SDK) of the corresponding service provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.
[0036] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. For example, it could be a server providing communication services to multiple clients, a server supporting backend training of models used on clients, or a server processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.
[0037] This specification provides a data processing method in one embodiment. By determining a data processing task and judging whether the data processing task carries a task execution priority, if the data processing task does not carry a task execution priority, a task priority strategy can be determined based on the target task parameters of the data processing task. This strategy determines the task execution priority of the data processing task. For example, if the target task parameter is the task data volume, the task execution priority is quickly and accurately determined by comparing the task data volume with a preset data volume. The data processing task is then assigned to the task execution queue corresponding to the task execution priority and executed. This achieves automated and intelligent task scheduling, reduces manual intervention, and improves task execution efficiency. Furthermore, determining the task execution priority and the corresponding task execution queue based on the task priority strategy improves the accuracy and flexibility of task scheduling.
[0038] See Figure 2 , Figure 2 A flowchart of a data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.
[0039] Specifically, this data processing method is applied to a data processing system, and the data processing system is used to implement the data processing method.
[0040] Step 202: Determine the data processing task, wherein the data processing task carries task parameters;
[0041] The application scenarios and data processing tasks differ depending on the data processing method. For example, when the data processing method is applied to a logistics management scenario, the data processing task can be the export of logistics forms; when the data processing method is applied to a financial management scenario, the data processing task can be the import of invoice data.
[0042] Task parameters include, but are not limited to, task name, task execution time, task type, etc. When a user sends a task processing request through the terminal device, if the task execution priority of the data processing task is specified, the task parameters may also include the task execution priority.
[0043] In one or more embodiments of this specification, a task processing request is received through a task processing interface, and a data processing task is generated based on the task processing request. Specific implementation methods are described below:
[0044] Before determining the data processing task, the method further includes:
[0045] Receive task processing requests through the task processing interface;
[0046] If the task processing request is a task creation request, the data processing task is generated according to the task creation parameters carried in the task creation request; or
[0047] If the task processing request is a task editing request, the initial data processing task is adjusted according to the task editing parameters carried in the task editing request to generate the data processing task.
[0048] The task processing interface can be understood as an application programming interface (API) that allows external systems or users to send requests to the data processing system. Through this task processing interface, users can submit task creation requests or task creation requests through a client and receive responses about the task status.
[0049] A task processing request can be understood as a request sent by a user or system through a task processing interface, with the purpose of starting a new data processing task, modifying an existing initial data processing task, or deleting a data processing task. Task processing requests include, but are not limited to, task creation requests, task editing requests, and task deletion requests.
[0050] A task creation request is used to ask the system to create a new data processing task. This task creation request contains the necessary task creation parameters, such as task name, task type, and processing logic. A task editing request is used to ask the system to modify or update an existing data processing task (i.e., the initial data processing task). This task editing request contains the necessary task editing parameters, including the parameters or attributes that need to be changed in the existing data processing task, as well as the corresponding new parameters or attributes.
[0051] Specifically, users submit task creation requests through the task processing interface. When creating a data processing task, users can set task execution parameters, i.e., task creation parameters, which include parameters such as task execution time and task type. Therefore, the task creation request contains task creation parameters. The data processing system reads the task creation parameters in the task creation request and generates a new data processing task based on these parameters.
[0052] Alternatively, if a user wants to edit an existing data processing task, the user needs to submit a task editing request through the task processing interface. This task editing request contains task editing parameters. The data processing system reads the task editing parameters in the task editing request and adjusts the existing data processing task according to these parameters.
[0053] In practical applications, users can also submit task deletion requests through the task processing interface to delete existing data processing tasks.
[0054] The data processing method provided in the embodiments of this specification receives task processing requests through a task processing interface. When the task processing request can be a task creation request or a task editing request, the data processing system can flexibly create or edit tasks according to different request types and corresponding parameters, thereby meeting the diverse needs of users.
[0055] In one or more embodiments of this specification, when a data processing task is being processed, it is necessary to uniformly analyze and process the resources occupied by the task, so as to more intelligently allocate and utilize resources through real-time and reasonable use and recycling. Specific implementation methods are described below:
[0056] After determining the data processing task, the process also includes:
[0057] Perform a resource requirement analysis on the data processing task to determine the required resources for executing the data processing task;
[0058] Allocate the required resources to the data processing task accordingly;
[0059] After executing the data processing task according to the task execution queue and the task parameters, the process further includes:
[0060] Once the data processing task is completed, the required resources are reclaimed.
[0061] Specifically, resource requirements analysis can be understood as a process of assessing the resources required for a data processing task; required resources can be understood as the physical or virtual resources required to successfully execute a data processing task, including computing resources, storage resources, hardware resources, etc., including but not limited to CPU (Central Processing Unit), memory, storage space, network bandwidth, etc.
[0062] A task execution queue can be understood as a way to organize tasks to be processed, typically ordered according to a first-in, first-out (FIFO) principle. It helps manage multiple data processing tasks that are executed in parallel or sequentially, ensuring that each task receives appropriate resources and executes in a certain order. Resource reclamation can be understood as the process of releasing previously allocated resources after a data processing task is completed. Reclaiming resources allows them to be reused by other tasks, improving the system's resource utilization.
[0063] Specifically, a resource requirement analysis is performed on the data processing task to determine the amount of resources required. For example, if a data processing task requires 4 CPU cores and 50GB (gigabyte) of temporary disk space, the data processing system allocates 4 CPU cores and 50GB of disk space to the task based on the results of the resource requirement analysis. The data processing task enters the task execution queue and waits for its turn to use the allocated resources to begin execution. Once the data processing task is completed, the data processing system will reclaim the resources previously allocated to it, making these resources available for use by other data processing tasks.
[0064] The data processing method provided in the embodiments of this specification, through the above process, can analyze and obtain the required resources for data processing tasks, enabling data processing tasks to be executed efficiently based on the required resources, and the data processing system can dynamically manage its resources to ensure that resources are fully utilized.
[0065] Step 204: If it is determined that the data processing task does not carry the task execution priority, determine the task priority formulation strategy of the data processing task according to the target task parameters of the data processing task.
[0066] Specifically, when creating a data processing task, users can specify the task execution priority or choose not to set a corresponding task execution priority. However, if the data processing task does not have a task execution priority, a corresponding task execution priority needs to be assigned to it.
[0067] To ensure the accuracy of the assigned task execution priority, a task priority strategy needs to be determined based on the target task parameters of the data processing task. This strategy enables the data processing task to determine its corresponding task execution priority.
[0068] In one or more embodiments of this specification, when the data processing task carries a task execution priority, the task execution queue corresponding to the task execution priority can be determined directly based on the carried task execution priority. Specific implementation methods are as follows:
[0069] After determining the data processing task, the process also includes:
[0070] In determining the data processing task, if the task execution priority of the data processing task is included, the task execution queue corresponding to the task execution priority is determined.
[0071] The data processing task is executed according to the task execution queue and the task parameters.
[0072] Specifically, when a user creates a data processing task and sets a task execution priority, the data processing task carries that task execution priority. Therefore, there is no need to assign a corresponding task execution priority to the data processing task. Instead, the task execution queue corresponding to the task execution priority can be determined directly based on the carried task execution priority. Then, the data processing task can be executed according to the task execution queue and task parameters.
[0073] Of course, in practical applications, to avoid situations where the task execution priority set by the user does not match the data processing task, for example, if the task execution priority of a data processing task is determined to be low based on the task priority strategy, but the user sets the task execution priority to high, the task execution priority set by the user can be adjusted.
[0074] In one or more embodiments of this specification, the target task parameter can be understood as the task data volume of the data processing task, thereby determining the task priority of the data processing task and formulating a strategy based on the task data volume of the data processing task. Specific implementation methods are as follows:
[0075] The step of determining the task priority and formulating a strategy based on the target task parameters of the data processing task includes:
[0076] Based on the amount of data in the data processing task, a task priority strategy is formulated to determine the data processing task.
[0077] The task priority setting strategy can be understood as the strategy for determining the execution priority of data processing tasks.
[0078] Specifically, based on the amount of data in a data processing task, a strategy can be developed to determine the execution priority of the task according to the amount of data. In practice, data processing tasks with different amounts of data usually require different computing resources and storage space. Therefore, data processing tasks can be classified according to the size of the data, and the corresponding execution priority can be determined for each category of data processing task.
[0079] In practical applications, this applies when the task priority setting strategy is to prioritize the data processing task based on the amount of data in the data processing task.
[0080] The step of formulating a strategy based on the task priority to determine the task execution priority of the data processing task includes:
[0081] The task data volume of the data processing task is compared with the preset data volume, and the task execution priority of the data processing task is determined based on the comparison result.
[0082] The task execution priority can be understood as a value or level assigned to the data processing task to determine the priority level of the data processing task execution; the preset data volume can include multiple, such as the first preset data volume of 100,000 and the second preset data volume of 300,000.
[0083] When the data processing task has a data volume of 200,000, comparing the data volume with the first and second preset data volumes reveals that the data volume is greater than the first preset data volume but less than the second preset data volume. Therefore, the execution priority of this data processing task is medium. For data processing tasks with a data volume less than the first preset data volume (i.e., less than 100,000), the execution priority is low. For data processing tasks with a data volume greater than the second preset data volume (i.e., more than 300,000), the execution priority is high. The specific implementation can be set according to the actual situation, and will not be elaborated here.
[0084] In one or more embodiments of this specification, the target task parameter can also be understood as the task execution time of the data processing task. Based on the task execution time, the urgency of the data processing task can be determined, thereby enabling the determination of task priority strategies based on the task execution time. Specific implementation methods are described below:
[0085] The target task parameter is the task execution time of the data processing task; the step of determining the task priority of the data processing task and formulating a strategy based on the target task parameter includes:
[0086] Based on the task execution time of the data processing task, a task priority strategy is formulated to determine the task priority of the data processing task.
[0087] Specifically, based on the task execution time of the data processing task, a strategy can be formulated to determine the task execution priority based on the task execution time. In practice, the task execution time can usually represent the urgency of processing the data processing task. For example, data processing tasks whose execution time is close to the current time can be considered more urgent to be completed. Therefore, the task priority strategy can be formulated based on the task execution time.
[0088] In practical applications, when the task priority setting strategy is to prioritize the data processing task based on its execution time; the task priority setting strategy is to prioritize the data processing task based on its execution time.
[0089] The step of formulating a strategy based on the task priority to determine the task execution priority of the data processing task includes:
[0090] The task execution time of the data processing task is compared with the current time, and the task execution priority of the data processing task is determined based on the comparison result.
[0091] Specifically, when creating a data processing task, the task execution time is usually set. If the task priority of the data processing task is determined based on the task execution time, the difference between the task execution time and the current time can be compared. Based on the comparison result, a task execution priority can be assigned to the data processing task.
[0092] For example, for data processing task A and data processing task B, if the comparison between the execution time of data processing task A and the current time is less than half an hour, then the execution priority of data processing task A can be determined to be high; if the comparison between the execution time of data processing task B and the current time is greater than 6 hours, then the execution priority of data processing task B can be determined to be low.
[0093] In practical applications, the target task parameter can also be the required resource quantity. Specifically, based on the above embodiment, when the required resources for the data processing task are determined, a task priority strategy can be formulated based on the required resources. Thus, the task execution priority of the data processing task is determined based on the task priority strategy and the required resources. In other words, the target task parameter can be set according to actual needs and is not limited here, thereby determining different task priority strategies, and the task execution priority of the data processing task is determined based on the corresponding task priority strategy.
[0094] The data processing method provided in the embodiments of this specification, through the flexible setting of target task parameters, can determine the task priority of data processing tasks and formulate strategies according to different dimensions. Thus, by formulating strategies based on the task priority of data processing tasks, the task execution priority of data processing tasks can be accurately determined, thereby realizing intelligent determination of task execution priority.
[0095] Step 206: Develop a strategy based on the task priorities to determine the task execution priority of the data processing task;
[0096] In one or more embodiments of this specification, to enable users to view information such as the task execution priority and execution status of data processing tasks, a visual user interface is used to display this information to the user. Specific implementation methods are described below:
[0097] After determining the task execution queue corresponding to the task execution priority, the method further includes:
[0098] The task execution queue and the task parameters are displayed to the user through a visual user interface.
[0099] Specifically, task parameters include, but are not limited to, task name, task type, task status, and task execution start time; see [link / reference]. Figure 3 , Figure 3 A schematic diagram of a user interface provided in an embodiment of this specification is shown.
[0100] Figure 3The task parameters shown include task name, type (i.e., task type), status (i.e., task status), error message, start time (i.e., task execution start time), and end time. The task queue (i.e., task execution queue) corresponding to each data processing task is also shown, and the user can perform operations on each data processing task, such as downloading.
[0101] In practical applications, the user interface can be bound to the user account, so that the data processing tasks displayed on the user interface are all data processing tasks created by the user corresponding to that user account, thus ensuring access control for the data processing tasks.
[0102] Step 208: Determine the task execution queue corresponding to the task execution priority, and execute the data processing task according to the task execution queue and the task parameters.
[0103] Specifically, each task execution priority corresponds to a task execution queue. For example, if the task execution priorities are divided into high, medium and low, the task execution queues include task execution queue 1 corresponding to high task execution priority, task execution queue 2 corresponding to medium task execution priority, and task execution queue 3 corresponding to low task execution priority.
[0104] In specific implementation, task execution queue 3 can correspond to 3 parallel threads, task execution queue 2 can correspond to 2 parallel threads, and task execution queue 1 can correspond to 1 thread. The corresponding task execution queue is determined according to the task execution priority of the data processing task, so that when the data processing task is executed according to the task parameters, the thread pool corresponding to the task execution queue is selected to implement the execution of the data processing task.
[0105] In one or more embodiments of this specification, the task execution queue follows a first-in, first-out (FIFO) strategy. When a data processing task is at the head of the task execution queue, the data processing task is executed according to the task parameters. Specific implementation methods are described below:
[0106] The step of executing the data processing task according to the task execution queue and the task parameters includes:
[0107] Add the data processing task to the task execution queue;
[0108] According to the first-in-first-out (FIFO) strategy of the task execution queue, if it is determined that the data processing task is located at the head of the task execution queue, the data processing task is executed according to the task parameters.
[0109] Specifically, once the task execution queue corresponding to the data processing task is determined, the enqueue operation of the queue can be used to add the data processing task to the tail of the task execution queue. The tasks to be processed are executed sequentially according to the order of the task execution queue. When the data processing task is at the head of the task execution queue, it is a task to be processed. Therefore, the data processing task is taken out from the head of the queue and executed according to the task parameters, that is, according to the task execution time, task type and other parameters of the data processing task. For example, if data processing task A is a data export task for data A at time t, then data A will be exported at time t.
[0110] The data processing method provided in the embodiments of this specification can avoid the queue blocking problem caused by all data processing tasks corresponding to one queue when each data processing task is added to a different task execution queue according to the task execution priority. It can ensure that the data processing tasks are not blocked and realize the rapid flow of data processing tasks.
[0111] In one or more embodiments of this specification, the data processing system can also monitor the execution data during the execution of data processing tasks, and promptly send an anomaly alarm message if the execution data is determined to be abnormal. Specific implementation methods are described below:
[0112] After executing the data processing task according to the task execution queue and the task parameters, the process further includes:
[0113] Determine the execution data during the execution of the data processing task;
[0114] If the execution data is determined to be abnormal, an abnormal alarm message is generated.
[0115] Specifically, during the execution of a data processing task, there are multiple execution data, such as status update data and resource usage data for the data processing task.
[0116] When a data processing task begins execution, the execution data is set as the status update data at the start of execution. When the data processing task is completed, the status update data at the start of execution is changed to the status update data at completion. In other words, the execution status of the data processing task can be determined through this status update data. For example, if historical data indicates that the data processing task will change from the status update data at the start of execution to the status update data at completion within one hour, then if the status update data of the data processing task does not change within one hour, the status update data at the start of execution is determined to be abnormal data. At this time, an abnormal alarm message can be generated to indicate the abnormal status update of the data processing task.
[0117] When a data processing task is being executed, the resource usage data during the execution process can be monitored. If the resource usage data exceeds a preset threshold (which can be set according to the actual situation), the resource usage data can be identified as abnormal data, and an abnormal alarm message can be generated to indicate the abnormal resource usage of the data processing task.
[0118] The data processing method provided in the embodiments of this specification monitors the running status of data processing tasks in real time, including task execution status and resource usage. When anomalies are detected, it can generate anomaly alarm information in a timely manner, thereby achieving efficient anomaly handling.
[0119] In one or more embodiments of this specification, various security measures can be taken to ensure the secure and stable operation of the data processing system and secure access to task data files, including file classification and separate storage, access control, etc. Specific implementation methods are described below:
[0120] After executing the data processing task according to the task execution queue and the task parameters, the process further includes:
[0121] Identify the executable files generated during the execution of the data processing task;
[0122] The executable files are categorized and stored, and / or access control is implemented for the executable files.
[0123] Specifically, during the execution of data processing tasks, various types of execution files are generated, such as log files, intermediate processing result files, and temporary files. In order to better manage and utilize these files, they need to be classified and stored, or appropriate access controls need to be implemented.
[0124] In practice, the executable files generated during the data processing task and their file types are determined. Storage is then split according to file type; for example, intermediate result files can be categorized and stored according to the task identifier or data processing stage, and temporary files can be placed in a dedicated temporary folder for easy cleanup. Access control can be implemented for executable files, such as permission settings, assigning different file access permissions to different users or roles (setting higher file access permissions for administrators than for ordinary users), and encrypted storage; executable files can be stored using encryption.
[0125] In practical applications, data processing systems can also periodically perform file backup snapshots to ensure the security of executable files.
[0126] The data processing method provided in the embodiments of this specification ensures the safe and stable operation of the data processing system and secure access to the executable files through various security measures such as classifying and storing executable files and access control.
[0127] The data processing method provided in the embodiments of this specification reduces manual intervention, improves task execution efficiency, and reduces error rate through automated task scheduling. It can achieve intelligent task scheduling, and can assign corresponding task execution priorities and task execution queues to data processing tasks, thereby improving scheduling accuracy and flexibility.
[0128] See Figure 4 , Figure 4 A system architecture diagram of a data processing system provided in one embodiment of this specification is shown.
[0129] This specification also provides a data processing system for implementing the above-described data processing method. The following describes the data processing method in detail, taking the system architecture diagram of the data processing system and the report export task as an example.
[0130] Specifically, a microservice architecture is adopted, which breaks down the data processing system into multiple independent service modules. This microservice architecture is beneficial to the scalability and maintainability of the data processing system. The data processing system includes a task management module, a resource management module, a scheduling engine module, a monitoring and alarm module, a security module, as well as a business service cluster, a database cluster, and distributed file storage.
[0131] A business service cluster means that different services A, B, and C can all use this data processing system to execute data processing tasks through a gateway.
[0132] The task management module is responsible for tasks such as creation, editing, and deletion. For example, users can submit report export tasks through the task processing interface (i.e., task API) and set task parameters, such as execution time and execution priority. The task management module can also track data processing tasks, including their status, start time, and end time. This data can be displayed to users through a visual user interface.
[0133] The resource management module is responsible for the unified analysis and processing of resources used during the report export task, including computing resources, storage resources, and hardware resources (such as memory and CPU for computation). For example, in the above embodiment, the resource requirement analysis of the report export task is performed to determine the amount of resources required for the report export task. The resource management module can also intelligently complete the allocation and utilization of resources by using and recycling resources in real time (recycling can be done after the data processing task is completed or when the resource utilization rate reaches a certain threshold), thereby improving resource utilization.
[0134] The scheduling engine module is responsible for intelligently allocating data processing tasks to appropriate task execution queues based on factors such as task type, task execution priority, and required resource status. Specifically, when a user creates a report export task, if the user specifies a task execution priority, the scheduling engine module can add the report export task to the task execution queue corresponding to the task execution priority based on a scheduling strategy. This scheduling strategy can be understood as a strategy that adds data processing tasks to task execution queues corresponding to task execution priorities. Alternatively, if the user does not set a task execution priority when creating the report export task, the task execution priority can be determined based on the amount of data in the report export task. For specific implementation details, please refer to the above embodiment, which will not be repeated here. In other words, the module allocates appropriate task execution queues to report export tasks based on their task execution priorities and can adjust (i.e., priority replacement) unreasonable task execution priorities specified by the user.
[0135] In practical applications, as data processing tasks are executed and completed, the data processing system will continuously update the status of the task execution queue, remove completed tasks, and may reorder or reallocate resources for other data processing tasks in the task execution queue as needed.
[0136] The monitoring and alarm module is responsible for real-time monitoring of the data processing system's operational status, including task execution and resource usage. When an anomaly is detected, the monitoring and alarm module can promptly generate alarm information and notify relevant personnel for timely handling via email notification. In practical applications, the monitoring and alarm module can provide functions such as storage space reminders, execution log output, task execution tracking, and email notification.
[0137] The security module employs various security measures to ensure the safe and stable operation of the data processing system and secure access to task data files (i.e., the executable files in the above embodiments). These measures include classifying and separating files generated during task flow for storage, access control, and regular file backup snapshots to ensure the security of task data files.
[0138] In practical applications, when performing report export tasks, reports can be exported using different export methods. For example, exporting can be done using the common SQLID (SQL query identifier), or using custom pagination export based on service units, temporary table pagination export, etc. When exporting to Excel format tables, preset templates can be used to quickly obtain standard format export files.
[0139] After the report export task is completed, the data processing system can process the task execution results, such as storing the task execution results in the database cluster's database. Distributed file storage allows multiple nodes to access and operate on the same file or data simultaneously, realizing data sharing and improving data availability and reliability.
[0140] The data processing method provided in the embodiments of this specification applies intelligent technology to the field of task scheduling, realizing intelligent task scheduling of the data processing system. This data processing system can efficiently transfer data processing tasks, improve user experience, and help enterprises solve the problem of low human resource utilization in the task processing process. By analyzing and classifying the enterprise's data processing task resources, resources can be rationally allocated, resource utilization can be improved, and resource costs can be reduced. By constructing an efficient, stable, and intelligent data processing system, it helps to improve the enterprise's operational efficiency and service quality, thereby enhancing the enterprise's market competitiveness.
[0141] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 5 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 5 As shown, the device includes:
[0142] Task acquisition module 502 is used to determine the data processing task;
[0143] The priority setting module 504 is used to determine the task priority setting strategy of the data processing task based on the target task parameters of the data processing task when it is determined that there is no task execution priority of the data processing task in the data processing task.
[0144] The priority determination module 506 is used to formulate a strategy based on the task priority and determine the task execution priority of the data processing task;
[0145] The task execution module 508 is used to determine the task execution queue corresponding to the task execution priority, and execute the data processing task according to the task execution queue and the task parameters.
[0146] Optionally, the task execution module 508 is further configured to:
[0147] In determining the data processing task, if the task execution priority of the data processing task is included, the task execution queue corresponding to the task execution priority is determined.
[0148] The data processing task is executed according to the task execution queue and the task parameters.
[0149] Optionally, the task execution module 508 is further configured to:
[0150] Add the data processing task to the task execution queue;
[0151] According to the strategy of the task execution queue, if it is determined that the data processing task is located at the head of the task execution queue, the data processing task is executed according to the task parameters.
[0152] The device further includes:
[0153] The task management module is used to receive task processing requests through a task processing interface; if the task processing request is a task creation request, it generates the data processing task according to the task creation parameters carried in the task creation request; or if the task processing request is a task editing request, it adjusts the initial data processing task according to the task editing parameters carried in the task editing request to generate the data processing task.
[0154] The device further includes:
[0155] The monitoring and alarm module is used to determine the execution data during the execution of the data processing task; and to generate abnormal alarm information when the execution data is determined to be abnormal data.
[0156] Optionally, the priority determination module 504 is further configured to:
[0157] Based on the amount of data in the data processing task, a task priority strategy is formulated to determine the data processing task.
[0158] Optionally, the priority determination module 506 is further configured to:
[0159] The task data volume of the data processing task is compared with the preset data volume, and the task execution priority of the data processing task is determined based on the comparison result.
[0160] Optionally, the priority determination module 504 is further configured to:
[0161] Based on the task execution time of the data processing task, a task priority strategy is formulated to determine the task priority of the data processing task.
[0162] Optionally, the priority determination module 506 is further configured to:
[0163] The task execution time of the data processing task is compared with the current time, and the task execution priority of the data processing task is determined based on the comparison result.
[0164] The device further includes:
[0165] The resource management module is used to perform resource requirement analysis for the data processing task, determine the required resources for executing the data processing task, allocate the corresponding required resources to the data processing task, and reclaim the required resources after the data processing task is completed.
[0166] Optionally, the task management module is further configured to:
[0167] The task execution queue and the task parameters are displayed to the user through a visual user interface.
[0168] The device further includes:
[0169] A security control module is used to identify the execution files generated during the execution of the data processing task; classify and store the execution files and / or control access to the execution files.
[0170] The data processing apparatus provided in one embodiment of this specification determines a data processing task and whether the data processing task carries a task execution priority. If the data processing task does not carry a task execution priority, a task priority formulation strategy can be determined based on the target task parameters of the data processing task. This strategy determines the task execution priority of the data processing task. For example, if the target task parameter is the amount of task data, the task execution priority is quickly and accurately determined by comparing the amount of task data with a preset amount of data. The data processing task is then assigned to the task execution queue corresponding to the task execution priority and executed. This achieves automated and intelligent task scheduling, reduces manual intervention, and improves task execution efficiency. Furthermore, determining the task execution priority and the corresponding task execution queue based on the task priority formulation strategy improves the accuracy and flexibility of task scheduling.
[0171] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.
[0172] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.
[0173] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.
[0174] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0175] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.
[0176] The processor 620 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-described data processing method.
[0177] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computing device embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.
[0178] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.
[0179] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiments are basically similar to the data processing method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the data processing method embodiments.
[0180] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data processing method.
[0181] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.
[0182] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0183] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0184] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.
[0185] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0186] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.
Claims
1. A data processing method, characterized in that, include: A data processing task is determined, wherein the data processing task carries task parameters; In cases where the data processing task does not carry the task execution priority, a task priority formulation strategy is determined based on the target task parameters of the data processing task. A strategy is formulated based on the task priorities to determine the task execution priority of the data processing task; Determine the task execution queue corresponding to the task execution priority, and execute the data processing task according to the task execution queue and the task parameters.
2. The data processing method according to claim 1, characterized in that, After determining the data processing task, the process also includes: In determining the data processing task, if the task execution priority of the data processing task is included, the task execution queue corresponding to the task execution priority is determined. The data processing task is executed according to the task execution queue and the task parameters.
3. The data processing method according to claim 1 or 2, characterized in that, The step of executing the data processing task according to the task execution queue and the task parameters includes: Add the data processing task to the task execution queue; According to the strategy of the task execution queue, if it is determined that the data processing task is located at the head of the task execution queue, the data processing task is executed according to the task parameters.
4. The data processing method according to claim 1, characterized in that, Before determining the data processing task, the method further includes: Receive task processing requests through the task processing interface; If the task processing request is a task creation request, the data processing task is generated according to the task creation parameters carried in the task creation request; or If the task processing request is a task editing request, the initial data processing task is adjusted according to the task editing parameters carried in the task editing request to generate the data processing task.
5. The data processing method according to claim 1, characterized in that, After executing the data processing task according to the task execution queue and the task parameters, the process further includes: Determine the execution data during the execution of the data processing task; If the execution data is determined to be abnormal, an abnormal alarm message is generated.
6. The data processing method according to claim 1, characterized in that, The target task parameter is the amount of task data for the data processing task. The step of determining the task priority and formulating a strategy based on the target task parameters of the data processing task includes: Based on the amount of data in the data processing task, a task priority strategy is formulated to determine the data processing task.
7. The data processing method according to claim 1 or 2, characterized in that, The task priority setting strategy is a strategy for setting task priorities based on the amount of data in the data processing task. The step of formulating a strategy based on the task priority to determine the task execution priority of the data processing task includes: The task data volume of the data processing task is compared with the preset data volume, and the task execution priority of the data processing task is determined based on the comparison result.
8. The data processing method according to claim 1, characterized in that, The target task parameter is the task execution time of the data processing task; The step of determining the task priority and formulating a strategy based on the target task parameters of the data processing task includes: Based on the task execution time of the data processing task, a task priority strategy is formulated to determine the task priority of the data processing task.
9. The data processing method according to claim 1 or 2, characterized in that, The task prioritization strategy is a strategy for prioritizing data processing tasks based on their execution time. The step of formulating a strategy based on the task priority to determine the task execution priority of the data processing task includes: The task execution time of the data processing task is compared with the current time, and the task execution priority of the data processing task is determined based on the comparison result.
10. The data processing method according to claim 1, characterized in that, After determining the data processing task, the process also includes: Perform a resource requirement analysis on the data processing task to determine the required resources for executing the data processing task; Allocate the required resources to the data processing task accordingly; After executing the data processing task according to the task execution queue and the task parameters, the process further includes: Once the data processing task is completed, the required resources are reclaimed.
11. The data processing method according to claim 1, characterized in that, After determining the task execution queue corresponding to the task execution priority, the method further includes: The task execution queue and the task parameters are displayed to the user through a visual user interface.
12. The data processing method according to claim 1, characterized in that, After executing the data processing task according to the task execution queue and the task parameters, the process further includes: Identify the executable files generated during the execution of the data processing task; The executable files are categorized and stored, and / or access control is implemented for the executable files.
13. A data processing apparatus, characterized in that, include: The task acquisition module is used to determine the data processing task; The priority setting module is used to determine the task priority setting strategy for the data processing task based on the target task parameters of the data processing task when it is determined that there is no task execution priority of the data processing task in the data processing task. The priority determination module is used to formulate a strategy based on the task priority and determine the task execution priority of the data processing task; The task execution module is used to determine the task execution queue corresponding to the task execution priority, and execute the data processing task according to the task execution queue and the task parameters.
14. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1-12.
15. A computer-readable storage medium, characterized in that, It stores a computer program / instruction that, when executed by a processor, implements the steps of the data processing method according to any one of claims 1-12.
16. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1-12.