Business data processing method and electronic equipment

Abstract

The invention discloses a business data processing method and electronic equipment, and relates to the technical field of computers, the method comprises the following steps: in a current business data processing round, obtaining multiple pieces of business data corresponding to a timed task from a business database; according to the priorities corresponding to the multiple pieces of service data, dividing the service data to obtain at least one service data set; according to the priority corresponding to the service data in the first service data set, the number of threads in the server allocated to the first service data set is determined, and the first service data set is any one of the at least one service data set; and processing the service data in the first service data set according to the number of the threads allocated to the first service data set. According to the application, the number of threads is dynamically determined through the priority of the business data, the server thread resources are reasonably utilized, and the business data processing efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a business data processing method and an electronic device. BACKGROUND

[0002] In an information-based business system, a timing task is a key to automatically process business data, and through preset execution periods or trigger conditions, the timing task processes business data based on preset processing logic to realize automatic operation of a business process and guarantee continuous and stable business. However, as the business scale expands, the data scale processed by the timing task also expands dramatically, which leads to low processing efficiency of the timing task and difficulty in meeting the needs of current business scenarios. SUMMARY

[0003] The present application provides a business data processing method and an electronic device to at least solve the problem of low processing efficiency of a timing task.

[0004] The present application provides a business data processing method, which comprises the following steps. In a current business data processing round, a plurality of business data corresponding to a timing task is acquired from a business database; Each business data is divided according to a priority corresponding to each business data, and at least one business data set is obtained; The number of threads in a server allocated to the first business data set is determined according to the priority of the business data in the first business data set, wherein the first business data set is any one of the at least one business data set; The business data in the first business data set is processed according to the number of threads allocated to the first business data set.

[0005] The present application also provides a business data processing device, which comprises the following modules. An acquisition module is configured to acquire, in a current business data processing round, a plurality of business data corresponding to a timing task from a business database; A division module is configured to divide each business data according to a priority corresponding to each business data, and obtain at least one business data set; A determination module is configured to determine the number of threads in a server allocated to the first business data set according to the priority of the business data in the first business data set, wherein the first business data set is any one of the at least one business data set; A processing module is configured to process the business data in the first business data set according to the number of threads allocated to the first business data set.

[0006] The application further provides an electronic device, comprising a memory for storing a computer program, and a processor for executing the computer program to implement the steps of any of the business data processing methods.

[0007] The application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of any of the business data processing methods.

[0008] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of any of the business data processing methods.

[0009] According to the application, the business data is divided into a plurality of business data sets according to the priority of the obtained business data, and the number of threads allocated to the business data in the business data set is dynamically determined based on the priority of the business data, for example, more threads are allocated to the business data with high priority. On the one hand, the problem of uneven allocation of resources in the server, such as processing low-priority business data with a large number of threads, or not enough threads to cause high-priority business data to be unable to be processed in time, is avoided, the reasonable allocation of server resources is realized, and the waste of resources in the server is reduced. On the other hand, the corresponding number of threads are matched for the business data with different priorities for parallel processing, which can fully exert the concurrent advantage of multi-threading, speed up the overall processing speed of the business data, and improve the processing efficiency of the business data. In addition, the business data set containing business data with different priorities is processed by an independent thread pool, which realizes the isolation between business data with different priorities and avoids the influence of abnormal business data processing on the processing of other priority business data. For example, when a certain priority data (especially low-priority business data) appears processing abnormality (such as logical error, execution blockage), the influence range is limited within its own thread pool, and it will not affect the processing of other priority data, effectively guaranteeing the stability and continuity of high-priority core business data processing. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the application, the drawings needed in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0011] Figure 1 A flowchart of a business data processing method provided by an embodiment of the application; Figure 2 A flowchart of another business data processing method provided by an embodiment of the application; Figure 3A structural schematic diagram of a service data processing apparatus provided by an embodiment of the present application is shown in FIG. 1. Figure 4 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 1. DETAILED DESCRIPTION

[0012] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, any other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0013] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such a process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0014] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0015] First, the application scenarios of the embodiments of the present application are exemplarily introduced.

[0016] In a distributed cluster environment, cluster monitoring and timing tasks are two core functions to ensure the stable operation of the distributed environment. Among them, the cluster monitoring is responsible for tracking the physical state (such as node survival, hardware load) and performance state (such as CPU utilization, memory occupancy, network throughput) of the entire cluster in real time, and effectively avoids the avalanche effect by discovering node downtime, service crashes, resource exhaustion and other abnormalities in time. At the same time, based on the long-term accumulated monitoring data, the system bottleneck can be accurately analyzed, the resource allocation strategy can be optimized, and the overall stability can be improved. The timing task undertakes the periodic business processing responsibility, such as performing log cleaning, database index optimization, model cache refreshing, data processing and other operation and maintenance tasks regularly. Its core goal is to avoid resource contention during peak periods by staggered execution, reduce the risk of accidental deletion, missed execution and other risks caused by human operation, and ultimately realize the full utilization of cluster resources without affecting user experience to complete the supplementary correction of business data.

[0017] Taking an artificial intelligence platform as an example, the artificial intelligence platform provides core services such as reasoning calculation, model training, and algorithm deployment. In a complex distributed cluster environment such as the artificial intelligence platform, cluster resources are managed and scheduled by relying on Kubernetes (k8s), and scheduled tasks are deployed by using Docker containerization technology. The scheduled tasks can be used to process business data, file system backup, and the like. Taking processing of business data as an example, business data (such as occupied disk size, CPU use time, and the like) generated in the process of the platform running is continuously stored in a business database, and the scheduled task extracts and processes the business data according to a preset period, so as to generate information such as a bill corresponding to the business data.

[0018] However, in the case of a large amount of business data, the scheduled task cannot complete the processing of the business data within the preset time, causing the scheduled task to fail to be completed successfully. In the related art, when the scheduled task fails to complete the processing of the business data within the preset time, the execution time of the scheduled task is increased, or the scheduled task is started again in additional time (for example, idle time such as early morning) to complete the scheduled task. However, increasing the execution time of the scheduled task affects the execution of subsequent other tasks, and starting the scheduled task again in the additional time causes the business data corresponding to the scheduled task to be unable to be processed in a timely manner, and the processing result (such as a bill) corresponding to the business data cannot be obtained in a timely manner, thereby affecting the timeliness of the processing of the business data and the utilization efficiency of platform resources, and it is difficult to meet the service demand of high response speed and high stability.

[0019] Therefore, the embodiments of the present application provide a business data processing method to improve the processing efficiency of business data and reasonably allocate thread resources.

[0020] It should be noted that the execution subject of the business data processing method provided by the embodiments of the present application can be a business data processing apparatus, which can be implemented by software, hardware, or a combination of software and hardware to become part or all of an electronic device, where the electronic device can be a server or a terminal, where the server in the embodiments of the present application can be a server or a server cluster composed of multiple servers, and the terminal in the embodiments of the present application can be a smart phone, a personal computer, a tablet computer, a wearable device, a smart robot, and other smart hardware devices. In the following method embodiments, the execution subject is taken as an example of an electronic device.

[0021] According to the embodiments of the present application, a business data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0022] In the embodiment, a service data processing method is provided, which can be applied to the electronic device as described above, such as a server. Figure 1 The service data processing method is shown in FIG. 1, which includes the following steps. Figure 1 S101, in a current service data processing round, obtaining a plurality of service data corresponding to a timing task from a service database.

[0023] Specifically, the timing task refers to a task that is automatically executed according to a preset time or condition (such as log cleaning, data backup, data statistics, etc.). For example, the timing task can be divided into multiple levels according to the execution frequency, such as a first level, a second level, a third level, and a fourth level. Among them, the first level has a short interval, and is usually executed once every 5 minutes or 15 minutes. The second level is executed by hour, such as once every 1 hour or 2 hours. The third level is executed by day, usually at about 2 o'clock in the morning (usually at this time, the business is the least, and the idle resources of the cluster are more). The fourth level is executed every week.

[0024] The current service processing round refers to a service data processing process that is executed once according to a preset period or condition after the timing task is triggered. For example, the bill calculation based on service data at 10 o'clock every day.

[0025] The service data refers to various types of service data required by the timing task. For example, taking the calculation of bills as an example of the timing task, the service data in the service database can be the information of the computing resources used based on the task request. Taking the task request as an example of model training request, the service data contained in the service database can be the time of the task request, the time length of the model training, the CPU resources, bandwidth resources, and memory resources used in the training process. The types of service data can be divided according to the types of task requests corresponding to the service data. For example, the types of service data can be the service data corresponding to the model training request, the service data corresponding to the inference request, the service data corresponding to the algorithm request, and the service data corresponding to the script editing request.

[0026] In one possible implementation, after the timing task is triggered, the server connects the service database and executes an SQL query statement to obtain a plurality of service data corresponding to the timing task.

[0027] S102, according to the priority corresponding to each service data, the service data is divided to obtain at least one service data set.

[0028] ​The priority of the service data is determined according to the importance or urgency of the service data. For example, the priority of the service data corresponding to the reasoning request is higher than the priority of the service data corresponding to the model training request.

[0029] For example, a priority label is set for each service data in advance, and a data classification algorithm is used to classify the service data of high priority, the service data of medium priority and the service data of low priority into different service data sets respectively. In this way, since the urgency and importance of different service data are different, differentiating the service data into different service data sets can realize differentiated processing, avoid the processing of service data of high priority being blocked by service data of low priority, realize the ordered processing of service data, and improve the processing speed of core service data.

[0030] S103, determining the number of threads in the server allocated for the first service data set according to the priority of the service data in the first service data set.

[0031] The first service data set is any one of the at least one service data set.

[0032] In a possible implementation, the number of threads in the server allocated for the first service data set can be determined based on a preset mapping relationship between priority and the number of threads, and the priority of the service data in the first service data set. For example, in the case that the service data in the first service data set is of high priority, 4 threads are allocated for the first service data set; in the case that the service data in the first service data set is of medium priority, 3 threads are allocated for the first service data set; and in the case that the service data in the first service data set is of low priority, 1 thread is allocated for the first service data set.

[0033] This is because, considering that service data of high priority needs more resources to ensure processing efficiency and avoid delay caused by insufficient threads, service data of low priority can reduce thread occupation to avoid waste of server resources, thereby realizing accurate allocation of thread resources and improving server resource utilization.

[0034] In another possible implementation, in addition to the priority of the service data in the first service data set, the number of threads allocated for the first service data set can also be dynamically adjusted in combination with the number of service data in the first service data set. For example, in the case that the data amount of the first service data set exceeds a preset data amount threshold, the number of threads is increased.

[0035] S104, processing the service data in the first service data set according to the number of threads allocated for the first service data set.

[0036] Specifically, the threads in the server are allocated for the first service data set according to the number of threads allocated for the first service data set, and the service data in the first service data set is processed by the threads in the server allocated for the first service data set.

[0037] In the embodiment of the present application, the plurality of service data sets are obtained by dividing the plurality of service data according to the priority of the service data, and the number of threads allocated for the service data set is dynamically determined based on the priority of the service data in the service data set, for example, more threads are allocated for the service data with high priority. On the one hand, it avoids the problem of uneven allocation of resources in the server, such as using a large number of threads to process low-priority service data, or not enough threads to cause high-priority service data to be unable to be processed in time, etc., so as to realize the reasonable allocation of server resources, reduce the waste of resources in the server, and shorten the processing time of high-priority service data from minutes to seconds. On the other hand, the corresponding number of threads are matched for the service data with different priorities for parallel processing, which can fully exert the concurrent advantage of multi-thread, speed up the overall processing speed of service data, and improve the processing efficiency of service data. In addition, the service data set containing service data with different priorities is processed by independent thread pool, which realizes the isolation between service data with different priorities, and avoids the influence of abnormal service data on the processing of service data with other priorities. For example, when a certain priority data (especially low-priority service data) appears processing abnormality (such as logic error, execution blockage), its influence range is limited within its own thread pool, and it will not affect the processing of other priority data, effectively guaranteeing the stability and continuity of high-priority core service data processing.

[0038] In some embodiments, on the basis of the foregoing embodiments, before processing the service data in the first service data set according to the number of threads allocated for the first service data set, the method provided in the embodiment of the present application further includes the following contents: Firstly, a preset data table is constructed.

[0039] Specifically, the preset data table is used to store data related to the timing task, for example, the service data corresponding to the timing task, and the processing result obtained after the server processes the service data by using the threads, etc.

[0040] Then, the plurality of service data corresponding to the timing task are stored in the preset data table.

[0041] Finally, the service data in the first service data set is obtained from the preset data table.

[0042] It can be understood that the preset data table is a temporary data table, and the server will delete the preset data table after the timing task is executed. For example, the preset data table is used to store the business data corresponding to the timing task, and the preset data table is deleted when the execution duration corresponding to the timing task is reached. For another example, the preset data table is used to store not only the business data corresponding to the timing task, but also the processing result corresponding to the business data. When the execution duration corresponding to the timing task is reached, the processing result corresponding to the business data is stored from the preset data table to the preset database (such as a billing database), and then the preset data table is deleted.

[0043] In this way, the plurality of business data used to execute the timing task is stored in the preset data table, so that the number of times of reading different tables in the business data is reduced, and the access pressure on the business database is reduced.

[0044] In a possible implementation, the first set of business data includes business data of at least one preset task that has completed execution and business data of at least one preset task that has not completed execution.

[0045] Specifically, the business data of the preset task that has completed execution refers to business data generated after the preset task (such as a model training task or an inference task) has completed execution, including the task execution time and all computing resources (such as CPU and memory) used in the task execution process. The business data of the preset task that has not completed execution refers to business data generated by the preset task that is being executed (such as in a model training process), including the task execution duration and the computing resources used by the task.

[0046] For example, in the business database, the business data has an execution state identifier corresponding to the task corresponding to the business data, which is used to indicate the completion state of the task corresponding to the business data. For example, the high-priority business data set contains not only the business data corresponding to the model A that has completed training (such as all computing resources used in the entire training process), but also the business data corresponding to the model B that is being trained (such as the computing resources already used). In this way, in the current business data processing round, the execution state of the task corresponding to the business data in the set is determined, which provides a basis for processing a plurality of business data. For example, the business data of the preset task that has completed execution is preferentially processed to ensure that the core result data is preferentially processed.

[0047] In a possible case, on the basis of the foregoing embodiments, in the S104, the business data in the first set of business data is processed according to the number of threads allocated for the first set of business data, and specifically includes the following steps: a1, according to the threads allocated for the first set of business data, preferentially processing each business data of the preset task that has completed execution.

[0048] Optionally, in the a1 above, the service data of the at least one completed preset task is processed by: First, according to the number of service data of the completed preset task in the first service data set and the number of threads allocated for the first service data set, the number of service data of the completed preset task processed by each thread is determined respectively.

[0049] For example, the ratio between the number of service data of the completed preset task in the first service data set and the number of threads allocated for the first service data set is taken as the number of service data of the completed preset task processed by each thread respectively.

[0050] For example, the total number of service data of the completed state in the first service data set is 50, and the number of threads allocated for the set is 10. The number of service data processed by each thread is determined by division operation (50 ÷ 10 = 5). If there is a remainder, the remaining service data is randomly allocated to the above-mentioned threads. In this way, the workload of each thread is relatively balanced, avoiding the idle or overload of some threads due to unbalanced load, improving the utilization rate of thread resources, and shortening the overall processing time.

[0051] Then, according to the number of service data of the completed preset task processed by the first thread, the first thread is allocated with corresponding service data of the completed preset task.

[0052] Among them, the first thread is any one of the threads allocated to the first service data set.

[0053] For example, 10 task queues are created for 10 threads, and 50 service data of the completed preset task are sequentially put into the task queue according to the principle of round robin allocation (i.e. the first service data is allocated to thread 1, the second service data is allocated to thread 2, and so on). In this way, through the explicit allocation mechanism, the thread contention for data is avoided, for example, multiple threads read the same service data at the same time, the binding of thread and data is realized, the data conflict is reduced, and the reliability of data processing process is guaranteed.

[0054] Finally, through the first thread, the service data of the completed preset task corresponding to the first thread is processed.

[0055] For example, through the first thread, based on the service data of the completed preset task and the preset billing rules corresponding to the service data, the bill corresponding to the service data is determined.

[0056] For example, the billing rule of the model training task can be “cost of Graphics Processing Unit (GPU) usage per hour x training duration + data storage cost”. For another example, the billing rule of the inference task can be “cost of inference request per time x total number of inference requests”.

[0057] a2, after all the completed execution preset task business data is processed, processing each unfinished execution preset task business data.

[0058] For example, 4 threads are allocated for high-priority business data, and all the completed execution preset task business data in the high-priority business data is processed first by the 4 threads, such as calculating the bill by using the completed execution preset task business data. After all the completed execution preset task business data is processed, the 4 threads are used to process the unfinished execution preset task business data in the high-priority business data.

[0059] In this way, the completed execution preset task business data is processed preferentially, avoiding the competition for resources with the unfinished execution preset task business data, improving the resource utilization, and ensuring the reliable processing of the completed execution preset task business data.

[0060] Optionally, in the above a2, each unfinished execution preset task business data is processed in the following way: First, each unfinished execution preset task business data is marked to obtain marked business data.

[0061] Then, the marked business data is stored in the business database.

[0062] Of course, when the execution duration corresponding to the timing task is reached, the unprocessed business data can also be marked, and the marked unprocessed business data is stored in the business database. In this way, when the next timing task is started, that is, the next business data processing round, the server preferentially processes the marked unprocessed business data and the marked unfinished execution preset task business data.

[0063] Figure 2 For another flowchart of a business data processing method. In Figure 2When the timing task starts, first, the plurality of service data corresponding to the timing task is read from the service database and stored in the preset data table. Second, according to the priority of each service data, the plurality of service data is divided into a plurality of service data sets, and according to the priority of the service data in the service data set, the number of threads allocated to the service data set is determined. Then, according to the number of threads allocated to the service data set, the service data of the preset task that has completed execution in the service data set is preferentially processed, and the service data of the preset task that has not completed execution is processed. Next, when the execution duration corresponding to the timing task is reached, it is judged whether the service data in the service data set has been processed. When there is unprocessed service data in the service data set, the processing result corresponding to the processed service data is read from the preset data table and stored in the preset database (such as a billing database), and the unprocessed service data is marked and stored in the service database. When there is no unprocessed service data in the service data set, the processing result corresponding to the processed service data is read from the preset data table and stored in the preset database (such as a billing database). Finally, the preset data table and the allocated threads are recycled.

[0064] Optionally, the method provided by the embodiment of the application further includes the following content: In the next service data processing round, when the acquired service data is the marked service data, the marked service data is preferentially processed.

[0065] For example, in the next service data processing round, the server first acquires a plurality of service data from the service database, and when there is marked service data (such as marked unprocessed service data or marked service data of a preset task that has not completed execution) in the plurality of service data, the marked service data is preferentially processed. After the processing of the marked service data is completed, the service data in the first service data set is processed according to the number of threads allocated to the first service data set. In this way, in each service data processing round, the marked service data is preferentially processed, so that the marked service data is not accumulated in the subsequent rounds and is not processed for a long time, and the integrity and reliability of service data processing are improved.

[0066] Optionally, in the above a1, after the thread allocated to the first service data set is used to preferentially process each service data of a preset task that has completed execution, the method provided by the embodiment of the application further includes the following content: After the first service data in the first service data set is processed, the processing result corresponding to the first service data is stored in the preset data table.

[0067] The first service data is any one of at least one service data of a preset task that has completed execution.

[0068] Taking the computing bill of the timing task as an example, the processing result corresponding to the first service data can be a bill based on the first service data.

[0069] Optionally, the method provided in the embodiments of the present application further includes the following content: First, when the execution duration corresponding to the timing task is reached, the processing result corresponding to the processed service data is read from the preset data table.

[0070] Then, the read processing result corresponding to the processed service data is stored in the preset database.

[0071] In the related art, each time a service data is read from the service database, the service data is processed, and the processing result corresponding to the service data is written into the preset database, such as a billing database, and then the next service data is read from the service database. Wherein, each time the preset database is written, the preset database is locked, and the table is unlocked after the writing is completed. In the current service data processing round, a plurality of service data need to be processed to obtain the processing result corresponding to each service data, therefore, multiple reading of the service database and multiple writing of the preset database occur, resulting in frequent locking of the preset database and increasing the memory garbage collection frequency. In order to avoid frequent locking, reduce the locking behavior of the preset database, and reduce the memory garbage collection frequency, in the embodiments of the present application, after the first service data is processed, the processing result corresponding to the first service data is stored in the preset data table, and when the execution duration corresponding to the timing task is reached, the processing result corresponding to all the processed service data is obtained from the preset data table and stored in the preset database. In this way, the preset data table is used as a temporary storage carrier, and the processing result corresponding to a single service data is not directly written into the preset database, that is, the locking mechanism of the preset database is not triggered, and by writing all the processing results in batches into the preset database, the original writing of the database once for processing one data is simplified to batch processing and writing once, so as to reduce the locking frequency and the memory garbage collection frequency.

[0072] In some embodiments, the method provided in the embodiments of the present application further includes the following steps: c1, in the process of processing the service data in the first service data set by each thread, the processing speed corresponding to each thread is obtained.

[0073] Specifically, the processing speed of the thread refers to the number of service data processed per unit time by the thread in the process of processing the service data in the first service data set. For example, the threads for processing high-priority service data are thread 1 and thread 2, wherein thread 1 can process 5 service data per second, and thread 2 can process 3 service data per second.

[0074] For example, the number of service data processed by the thread in a preset time period is obtained, and a ratio between the number and the preset time period is taken as the processing speed of the thread.

[0075] c2, according to the processing speed of each thread corresponding to the first set of service data, determine the thread processing speed corresponding to the first set of service data.

[0076] Specifically, the thread processing speed corresponding to the first set of service data is the overall speed obtained by aggregating the processing speeds of all threads processing the first set of service data, which reflects the processing efficiency of each thread for the first set of service data. For example, the first set of service data has 3 threads, and the processing speeds are 5 pieces per second, 3 pieces per second, and 4 pieces per second. The thread processing speed corresponding to the first set of service data can be the average of the processing speeds of the three threads, i.e. 4 pieces per second.

[0077] c3, when the thread processing speed corresponding to the first set of service data is not within the preset speed range corresponding to the first set of service data, the number of threads allocated for the first set of service data is adjusted according to the thread processing speed corresponding to the first set of service data and the preset speed range.

[0078] Specifically, the preset speed range can be determined according to the data characteristics in the first set of service data, such as data complexity, priority, etc., which is not limited in the embodiments of the present application. For example, the preset speed range of the high-priority service data set is [5 / second, 10 / second], and the preset speed range of the medium-priority service data set is [3 / second, 6 / second].

[0079] In one possible implementation, the number of threads allocated for the first set of service data is adjusted according to the thread processing speed corresponding to the first set of service data and the preset speed range, and the specific steps include: First, according to the thread processing speed and the preset speed range, the speed difference value corresponding to the first set of service data is determined.

[0080] Specifically, the speed difference value refers to the difference between the thread processing speed corresponding to the first set of service data and the boundary value of the preset speed range, which is used to quantify the degree of deviation of the processing speed from the reasonable range.

[0081] Optionally, when the thread processing speed is less than the lower limit value in the preset speed range, the speed difference value is the difference between the lower limit value in the preset speed range and the thread processing speed. When the thread processing speed is greater than the upper limit value in the preset speed range, the speed difference value is the difference between the thread processing speed and the upper limit value in the preset speed range.

[0082] For example, when the preset speed range is [3-6 threads per second], the thread processing speed is 2.5 threads per second, the speed difference is the difference between the lower limit of the preset speed range and the thread processing speed, i.e. 3-2.5=0.5 threads per second; and the thread processing speed is 7 threads per second, the speed difference is the difference between the thread processing speed and the upper limit of the preset speed range, i.e. 7-6=1 thread per second.

[0083] Then, the number of threads allocated for the first service data set is adjusted according to the speed difference and the preset speed range.

[0084] Optionally, when the thread processing speed is less than the lower limit of the preset speed range, the number of threads added for the first service data set is determined according to the speed difference, the lower limit of the preset speed range, and the number of threads currently allocated for the first service data set.

[0085] For example, the number of threads added for the first service data set is determined by the following formula:

[0086] wherein, the number of threads added is, the number of threads currently allocated for the first service data set is, the speed difference is, the lower limit of the preset speed range is. For example, when the thread processing speed is 2.5 threads per second, the preset speed range is [3 threads per second, 6 threads per second], the speed difference is 0.5 threads per second, the lower limit of the preset speed range is 3 threads per second, and the number of threads currently allocated for the first service data set is 5, the number of threads added is 1, i.e. the number of threads allocated for the first service data set after adjustment is 6.

[0087] Optionally, when the thread processing speed is greater than the upper limit of the preset speed range, the number of threads reduced for the first service data set is determined according to the speed difference, the upper limit of the preset speed range, and the number of threads currently allocated for the first service data set.

[0088] For example, the number of threads reduced for the first service data set is determined by the following formula:

[0089] wherein, the number of threads reduced is, the number of threads currently allocated for the first service data set is, the speed difference is, is an upper limit value in the preset speed range. For example, when the thread processing speed is 7 per second, the preset speed range is [3 per second, 6 per second], the speed difference value is 1 per second, the upper limit value in the preset speed range is 6 per second, and the number of threads currently allocated to the first set of service data is 8, the number of threads to be reduced is 1, that is, the number of threads allocated to the first set of service data after adjustment is 7.

[0090] In the embodiments of the present application, the speed difference value reflects the degree of mismatch between thread resources and processing demand. By adjusting the number of threads allocated to the first set of service data in combination with the preset speed range, fine adjustment of the number of threads can be achieved, and the adaptability between threads and service data can be enhanced. That is, when the complexity of service data increases to cause a decrease in processing speed, the number of threads is increased through speed difference value adjustment; when the data is simplified to cause excessively high speed, the number of threads is reduced to release resources, so that the allocation of threads always adapts to the dynamic changes of services, ensuring that the processing speed meets the preset speed range as soon as possible (such as increasing the number of threads to improve the speed when it is lower than the lower limit), and avoiding resource waste (such as reducing the number of threads to release resources when it is higher than the upper limit), thereby improving the accuracy and efficiency of adjustment.

[0091] In some embodiments, the method provided by the embodiments of the present application further includes the following content: When all service data in the first set of service data are processed, and there is service data that has not been processed in the plurality of service data, the service data that has not been processed is processed by using the threads allocated to the first set of service data.

[0092] For example, according to the priority corresponding to each unprocessed service data, the unprocessed service data is sorted, and the unprocessed service data is processed by using the threads allocated to the first set of service data according to the sorting order of the unprocessed service data.

[0093] In the embodiments of the present application, the threads allocated to the first set of service data are in an idle state after completing their own processing tasks, and the unprocessed part of other service data still needs thread resources. If the threads are directly released, resource waste will occur. Therefore, reusing idle threads can improve the processing efficiency of service data, shorten the processing time of the timing task, and improve the resource utilization rate.

[0094] For example, the timing task is executed at 10:00 every day, and the service data from 9:45 to 10:00 in the time period is obtained. It is assumed that, in a certain service data processing round, 13250 service data are obtained, including 3800 high-priority service data, 8200 medium-priority service data, and 1250 low-priority service data. The server allocates 4 threads to process the high-priority service data, 3 threads to process the medium-priority service data, and 1 thread to process the low-priority service data.

[0095] For example, the thread processing high-priority service data processes service data of the preset task that has been completed first, and stores the corresponding processing result in the preset data table, and then processes service data of the preset task that has not been completed.

[0096] In the case that the processing speed of each thread for each service data is the same, for the thread processing high-priority service data, each thread processes 950 service data, for the thread processing low-priority service data, each thread processes 1250 service data, and for the thread processing medium-priority service data, each thread processes 2734 service data. Therefore, after the high-priority service data is processed, the thread allocated for the high-priority service data continues to process medium-priority service data. At this time, the thread processing medium-priority service data is 7, and each thread still processes about 765 service data on average. The thread processing low-priority service data is left with only 300 service data. Thus, it can be predicted that the low-priority data will be executed faster, and when the low-priority service data is processed, medium-priority service data will continue to be processed. At this time, there are still 3250 medium-priority service data that have not been processed, and 8 threads process all the data, and each thread processes about 407 service data on average.

[0097] The above mainly describes the scheme provided by the embodiments of the present application from the perspective of a method.

[0098] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software and a necessary general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better embodiment.

[0099] In the embodiments of the present application, a service data processing device is also provided, which is used to implement the above embodiments and preferred embodiments, and has been described above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, implementation of hardware, or a combination of software and hardware, is also possible and is contemplated.

[0100] The embodiments provide a service data processing device, as shown in Figure 3 The device comprises: The obtaining module 301 is configured to obtain a plurality of service data corresponding to a timing task from a service database in a current service data processing round. The dividing module 302 is configured to divide each service data according to a priority corresponding to the service data, to obtain at least one service data set. The determining module 303 is configured to determine the number of threads in the server allocated for the first service data set according to the priority of the service data in the first service data set, wherein the first service data set is any one of the at least one service data set. The processing module 304 is configured to process the service data in the first service data set according to the number of threads allocated for the first service data set.

[0101] In a possible implementation, the apparatus further includes a constructing module and a storage module. The constructing module is configured to construct a preset data table. The storage module is configured to store the plurality of service data corresponding to the timing task in the preset data table. The obtaining module 301 is further configured to obtain the service data in the first service data set from the preset data table.

[0102] In a possible implementation, the first service data set includes service data of at least one completed preset task and service data of at least one uncompleted preset task. The processing module 304 is specifically configured to preferentially process the service data of each completed preset task according to the threads allocated for the first service data set. After all the service data of the completed preset tasks are processed, the service data of each uncompleted preset task is processed.

[0103] In a possible implementation, the storage module is further configured to store the processing result corresponding to the first service data in the first service data set in the preset data table after the first service data is processed, wherein the first service data is any one of the service data of the at least one completed preset task.

[0104] In a possible implementation, the processing module 304 is specifically configured to mark the service data of each uncompleted preset task to obtain marked service data. The marked service data is stored in the service database.

[0105] In a possible implementation, the obtaining module 301 is further configured to read the processing result corresponding to the processed service data from the preset data table when the execution duration corresponding to the timing task is reached; and the storage module is further configured to store the read processing result corresponding to the processed service data in the preset database.

[0106] In a possible implementation, the processing module 304 is further configured to, in the next service data processing round, when the obtained service data is the marked service data, preferentially process the marked service data.

[0107] In a possible implementation, the obtaining module 301 is further configured to, in the process of processing the service data in the first service data set by the threads, obtain the processing speeds of the threads respectively corresponding to the threads. The determining module 303 is further configured to determine the thread processing speed corresponding to the first service data set according to the processing speeds of the threads respectively corresponding to the threads, and adjust the number of threads allocated for the first service data set according to the thread processing speed corresponding to the first service data set and the preset speed range when the thread processing speed corresponding to the first service data set is not within the preset speed range corresponding to the first service data set.

[0108] In a possible implementation, the determining module 303 is specifically configured to determine the speed difference value corresponding to the first service data set according to the thread processing speed and the preset speed range. The number of threads allocated for the first service data set is adjusted according to the speed difference value and the preset speed range.

[0109] Through the apparatus provided by the embodiments of the present application, the plurality of service data sets are obtained by dividing the plurality of service data according to the priorities of the service data, and the number of threads allocated for each service data set is dynamically determined based on the priority of the service data in the service data set, for example, more threads are allocated for the service data with high priority. On the one hand, the problem of uneven allocation of resources in the server, such as processing low-priority service data with a large number of threads or insufficient threads leading to the inability to process high-priority service data in time, is avoided, the reasonable allocation of server resources is achieved, and the waste of resources in the server is reduced. On the other hand, the corresponding number of threads is matched for the service data with different priorities for parallel processing, which can fully exert the concurrent advantage of multi-threading, accelerate the overall processing speed of the service data, and improve the processing efficiency of the service data. In addition, the service data sets containing service data with different priorities are processed by independent thread pools, which realizes the isolation between service data with different priorities and avoids the influence of processing abnormal service data on the processing of service data with other priorities. For example, when a certain priority data (especially low-priority service data) appears processing abnormality (such as logic error, execution blocking), the influence range is limited within its own thread pool, and it will not affect the processing of other priority data, effectively guaranteeing the stability and continuity of the processing of high-priority core service data.

[0110] The features of the embodiments corresponding to the service data processing apparatus can be referred to the related descriptions of the embodiments corresponding to the service data processing method, which will not be repeated here.

[0111] An embodiment of the present application further provides an electronic device, such as Figure 4 As shown, the electronic device comprises a memory 10 and a processor 20, the memory 10 stores a computer program, and the processor 20 is configured to execute the computer program to perform the steps in any of the above business data processing method embodiments.

[0112] An embodiment of the present application further provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above business data processing method embodiments when executed.

[0113] In an example embodiment, the above computer readable storage medium can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0114] An embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above business data processing method embodiments.

[0115] An embodiment of the present application further provides another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above business data processing method embodiments.

[0116] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0117] The above describes in detail the service data processing method and the electronic device provided by the application. The principles and implementation manners of the application are described by using specific examples, and the above description of the embodiments is only applicable to help understand the method of the application and the core idea thereof. It should be pointed out that, for those skilled in the art, without departing from the principles of the application, the application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the claims of the application.

Claims

1. A business data processing method, characterized in that: Applied to a server, the method includes: In the current business data processing round, multiple business data corresponding to the scheduled tasks are obtained from the business database; Dividing each of the business data according to the priorities corresponding to the plurality of business data to obtain at least one business data set; determining, according to the priority corresponding to the business data in a first business data set, the number of threads in the server allocated to the first business data set, wherein the first business data set is any one of the at least one business data set; The business data in the first business data set is processed according to the number of threads allocated to the first business data set.

2. The method according to claim 1, characterized in that Before processing the business data in the first business data set according to the number of threads allocated to the first business data set, the method further includes: Build a preset data table; storing a plurality of business data corresponding to the scheduled tasks in the preset data table; The business data in the first business data set is obtained from the preset data table.

3. The method according to claim 2, characterized in that The first business data set includes business data of at least one preset task that has been completed, and business data of at least one preset task that has not been completed; The processing of the business data in the first business data set according to the number of threads allocated to the first business data set includes: Prioritizing processing of each of the completed business data sets of the preset task according to the thread allocated to the first business data set; After the business data of all the preset tasks that have been completed are processed, the business data of each of the preset tasks that have not been completed are processed.

4. The method according to claim 3, characterized in that After preferentially processing the business data of each of the completed preset tasks according to the thread allocated to the first business data set, the method further includes: After the first business data in the first business data set is processed, the processing result corresponding to the first business data is stored in the preset data table, wherein the first business data is any one of the business data of at least one preset task that has been completed.

5. The method according to claim 3 or 4, characterized in that Processing the business data of each of the unfinished preset tasks includes: Marking the business data of each of the unfinished preset tasks to obtain marked business data; The marked business data is stored in the business database.

6. The method according to claim 4, characterized in that The method further comprises: When the execution time corresponding to the scheduled task is reached, the processing result corresponding to the processed business data is read from the preset data table; The processing results corresponding to the read processed business data are stored in the preset database.

7. The method according to claim 5, characterized in that The method further comprises: In the next business data processing round, when the acquired business data is the marked business data, the marked business data is processed first.

8. The method according to any one of claims 1 to 4 and 6, characterized in that The method further comprises: In the process of each of the threads processing the business data in the first business data set, obtaining a processing speed corresponding to each of the threads; Determining a thread processing speed corresponding to the first business data set according to processing speeds corresponding to the threads; When the thread processing speed corresponding to the first business data set is not within the preset speed range corresponding to the first business data set, the number of threads allocated to the first business data set is adjusted according to the thread processing speed corresponding to the first business data set and the preset speed range.

9. The method according to claim 8, characterized in that When the thread processing speed corresponding to the first business data set is not within a preset speed range corresponding to the first business data set, adjusting the number of threads allocated to the first business data set according to the thread processing speed corresponding to the first business data set and the preset speed range includes: Determining a speed difference corresponding to the first service data set according to the thread processing speed and the preset speed range; The number of threads allocated to the first service data set is adjusted according to the speed difference and the preset speed range.

10. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the business data processing method according to any one of claims 1 to 9 when executing the computer program.

Images