Data processing method and device, equipment and storage medium

Abstract

The invention relates to a data processing method and device, equipment and a storage medium, and the method comprises the steps: exporting data through employing a concurrent task queue, dynamically monitoring the monitoring data of each concurrent task in the previous period in a process of executing data export in a database, determining the distribution resource information corresponding to each concurrent task in the next period according to the monitoring data, and obtaining the distributed resource information of each concurrent task in the next period; the target thread count is determined according to the resource allocation information, and the configuration file of the concurrent task is further dynamically adjusted according to the target thread count, so that the database controls the concurrent task to execute the data export operation in the next cycle by using the configuration file; according to the method, real-time monitoring and dynamic adjustment of performance parameter configuration of each working thread and reasonable allocation and maximum utilization of resources in the data exporting process are realized, so that the resource idle rate is reduced, and the data exporting performance is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of distributed databases, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] With the continuous development of distributed databases, the performance requirements for concurrent export of large amounts of data from distributed databases are becoming increasingly higher. The existing data export method implements performance monitoring of distributed database resources. When the performance conditions are met, data is exported based on fixed performance parameter configurations set based on performance monitoring.

[0003] However, the existing data export method requires manual analysis of logs and setting fixed tuning performance parameters for re-export. It cannot achieve real-time monitoring of each working thread during the data export process and dynamically adjust the performance parameter configuration, which leads to problems such as resource waste and poor data export performance. Summary of the invention

[0004] The present application provides a data processing method, apparatus, device and storage medium to solve the technical problem that the existing data export method requires manual analysis of logs and setting of fixed tuning performance parameters for re-export, and cannot achieve real-time monitoring of each working thread during the data export process and dynamically adjust the performance parameter configuration, thereby leading to waste of resources and poor data export performance.

[0005] In a first aspect, the present application provides a data processing method, the method comprising:

[0006] During the process of executing the concurrent task queue in the database, for any concurrent task in the concurrent task queue, obtaining monitoring data of the concurrent task queue during the execution of the concurrent task queue in the current cycle;

[0007] Determine the allocation resource information corresponding to the concurrent tasks in the next cycle according to the monitoring data;

[0008] Obtaining the target number of threads corresponding to the concurrent task according to the allocated resource information;

[0009] The configuration file of the concurrent task is adjusted according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform the data export operation in the next cycle.

[0010] In a possible implementation, the method further includes:

[0011] Obtaining a data export instruction, parsing the data export instruction to obtain field description information;

[0012] Obtaining a database statement according to the field description information, and building a concurrent task queue according to the database statement;

[0013] The concurrent task queue is submitted to a database so that the database executes the concurrent tasks in the concurrent task queue.

[0014] In a possible implementation manner, determining the allocation resource information corresponding to the concurrent tasks in the next period according to the monitoring data includes:

[0015] Parsing the monitoring data to obtain blocking information and allocation resource information corresponding to the concurrent tasks;

[0016] Determine whether the current concurrent task is blocked according to the blocking information;

[0017] When it is determined that the current concurrent task is blocked, the allocation resource information corresponding to the current concurrent task is obtained.

[0018] In a possible implementation manner, determining whether the current concurrent task is blocked according to the blocking information includes:

[0019] When the blocking information is greater than the blocking threshold, determining that the current concurrent task is blocked;

[0020] When the blocking information is less than or equal to the blocking threshold, it is determined that there is no blocking for the current concurrent task.

[0021] In a possible implementation manner, obtaining the target number of threads corresponding to the concurrent task according to the allocated resource information includes:

[0022] Parsing the allocated resource information to obtain a processing rate, a remaining data volume, a remaining time, and a current number of threads;

[0023] Calculating an expected rate according to the remaining data amount and the minimum remaining time, wherein the minimum remaining time indicates a minimum value of the remaining time required for the concurrent tasks to be completed;

[0024] The target number of threads is calculated according to the expected rate, the processing rate and the current number of threads.

[0025] In a possible implementation, adjusting the configuration file of the concurrent task according to the target number of threads includes:

[0026] Obtaining a configuration file for the concurrent task;

[0027] Parsing performance parameter information corresponding to the configuration file;

[0028] The performance parameter information is adjusted using the target number of threads to perform adjustment on the configuration file of the concurrent task.

[0029] In a possible implementation, the adjusting the performance parameter information by using the target number of threads to adjust the configuration file of the concurrent task includes:

[0030] Parsing the allocated resource information to obtain the remaining time of any concurrent task in the concurrent task queue;

[0031] Arrange the remaining time in descending order, wherein the larger the remaining time, the higher the priority;

[0032] Arrange any concurrent task in the concurrent task queue in descending order according to the remaining time to obtain priority information;

[0033] The target number of threads is used as the performance parameter to adjust the configuration file of the concurrent task according to the priority information.

[0034] In a second aspect, the present application provides a data processing device, the device comprising:

[0035] A data execution module is used to obtain monitoring data of the concurrent task queue during the execution of the concurrent task queue in the current cycle for any concurrent task in the concurrent task queue during the execution of the concurrent task queue in the database;

[0036] A data monitoring module, used to determine the allocation resource information corresponding to the concurrent tasks in the next cycle according to the monitoring data;

[0037] A thread scheduling module, used to obtain the target number of threads corresponding to the concurrent task according to the allocated resource information;

[0038] The data export module is used to adjust the configuration file of the concurrent task according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform data export operations in the next cycle.

[0039] In a possible implementation, the device further includes:

[0040] An instruction issuing unit, used for obtaining a data export instruction and parsing the data export instruction to obtain field description information;

[0041] A task construction unit, used to obtain a database statement according to the field description information, and to construct a concurrent task queue according to the database statement;

[0042] The task submission unit is used to submit the concurrent task queue to the database so that the database executes the concurrent tasks in the concurrent task queue.

[0043] In a possible implementation, the data monitoring module includes:

[0044] A data parsing unit, used for parsing the monitoring data to obtain blocking information and allocation resource information corresponding to the concurrent tasks;

[0045] A blocking judgment unit, used to determine whether the current concurrent task is blocked according to the blocking information;

[0046] The resource acquisition unit is used to acquire the allocation resource information corresponding to the current concurrent task when it is determined that the current concurrent task is blocked.

[0047] In a possible implementation manner, the blocking determination unit is specifically configured to:

[0048] When the blocking information is greater than the blocking threshold, determining that the current concurrent task is blocked;

[0049] When the blocking information is less than or equal to the blocking threshold, it is determined that there is no blocking for the current concurrent task.

[0050] In a possible implementation, the thread scheduling module is specifically used to:

[0051] Parsing the allocated resource information to obtain a processing rate, a remaining data volume, a remaining time, and a current number of threads;

[0052] Calculating an expected rate according to the remaining data amount and the minimum remaining time, wherein the minimum remaining time indicates a minimum value of the remaining time required for the concurrent tasks to be completed;

[0053] The target number of threads is calculated according to the expected rate, the processing rate and the current number of threads.

[0054] In a possible implementation, the data export module includes:

[0055] A file acquisition unit, used to acquire the configuration file of the concurrent task;

[0056] A file parsing unit, used to parse the performance parameter information corresponding to the configuration file;

[0057] The file adjustment unit is used to adjust the performance parameter information using the target number of threads to perform adjustment on the configuration file of the concurrent task.

[0058] In a possible implementation manner, the file adjustment unit is specifically used to:

[0059] Parsing the allocated resource information to obtain the remaining time of any concurrent task in the concurrent task queue;

[0060] Arrange the remaining time in descending order, wherein the larger the remaining time, the higher the priority;

[0061] Arrange any concurrent task in the concurrent task queue in descending order according to the remaining time to obtain priority information;

[0062] The target number of threads is used as the performance parameter to adjust the configuration file of the concurrent task according to the priority information.

[0063] In a third aspect, the present application provides a data processing terminal / device / server, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to:

[0064] During the process of executing the concurrent task queue in the database, for any concurrent task in the concurrent task queue, obtaining monitoring data of the concurrent task queue during the execution of the concurrent task queue in the current cycle;

[0065] Determine the allocation resource information corresponding to the concurrent tasks in the next cycle according to the monitoring data;

[0066] Obtaining the target number of threads corresponding to the concurrent task according to the allocated resource information;

[0067] The configuration file of the concurrent task is adjusted according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform the data export operation in the next cycle.

[0068] In a fourth aspect, the present application provides a storage medium storing one or more programs, wherein the one or more programs can be executed by one or more processors to implement the data processing method described in any one of the first aspects.

[0069] The above-mentioned technical scheme provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application uses a concurrent task queue to export data, and dynamically monitors the monitoring data of each concurrent task in the previous cycle during the data export process executed by the database, determines the allocation resource information corresponding to each concurrent task in the next cycle according to the monitoring data, determines the target number of threads according to the allocation resource information, and further dynamically adjusts the configuration file of the concurrent task according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform data export operations in the next cycle. This method realizes the reasonable allocation and maximum utilization of resources, reduces the resource idle rate, and effectively improves the data export performance by real-time monitoring and dynamically adjusting the performance parameter configuration of each working thread during the data export process; in addition, the monitoring data and the reallocated target number of threads are displayed at preset time intervals, so as to realize the dynamic display of real-time monitoring data during the data export process, thereby helping users to understand and analyze the progress of the current data export task in real time and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0071] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0072] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0073] Figure 1 A flowchart of an embodiment of a data processing method provided in an embodiment of the present application;

[0074] Figure 2 A flowchart of another data processing method provided in an embodiment of the present application;

[0075] Figure 3 A flow chart of another data processing method embodiment provided in the present application;

[0076] Figure 4 A flow chart of another data processing method embodiment provided in the present application;

[0077] Figure 5A block diagram of a data processing device provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of the structure of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0080] The disclosure below provides many different embodiments or examples to realize the different structures of the present application. In order to simplify the disclosure of the present application, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present application. In addition, the present application can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0081] In order to solve the technical problem that the data export method in the prior art requires manual analysis of logs and setting fixed tuning performance parameters for re-export, and cannot realize real-time monitoring of each working thread in the data export process and dynamically adjust the performance parameter configuration, which leads to waste of resources and poor data export performance, the present application provides a data processing method, device, equipment and storage medium, which can realize real-time monitoring and dynamic adjustment of the performance parameter configuration of each working thread in the data export process, thereby realizing the reasonable allocation and maximum utilization of resources, reducing resource idle rate, and effectively improving data export performance.

[0082] An embodiment of the present application provides a data export performance tuning device, which mainly includes a command input module, a main control module, a parser, an executor, an assembler, a monitoring module, etc.

[0083] The command input module is used to input the export command. The main control module is used to control the overall process of data export.

[0084] The parser is used to perform syntax analysis, legality check, character set conversion, etc. on the input export command. Among them, the legality check refers to checking the format of the input export command to determine whether the input export command conforms to the correct database statement syntax format. If it conforms to the correct database statement syntax format, it means that the legality check has passed. If it does not conform to the correct database statement syntax format, it means that the legality check has failed. If the legality check fails, the main control module prompts the user that the legality check of the currently input export command has failed. If the legality check passes, character set conversion is performed.

[0085] Character set conversion refers to converting the characters in the input export command into a character set with the same encoding format, for example, converting all characters into a character set with UTF-8 encoding format or converting all characters into a character set with GBK encoding format. In addition, it can also be converted into character sets with other encoding formats, which is not limited in the embodiments of the present application.

[0086] Syntax parsing refers to parsing the input export command to obtain field description information, assembling database statements according to the field description information, building a concurrent task queue according to the assembled database statements and submitting it to the database for execution.

[0087] The executor is used to execute the concurrent task queue submitted for data export. The assembler is used to merge the data files exported by each task in the concurrent task queue into one data file.

[0088] The monitoring module is used to monitor the blocking conditions and resource information of each queue during the execution of concurrent task queues by the database, and dynamically calculate and adjust the performance parameters of the configuration file during the data export process according to the monitoring data, and then dynamically adjust the thread resource scheduling information according to the configuration file, thereby reducing the resource idle rate and improving the data export performance. The monitoring module is described in detail in the form of an embodiment.

[0089] Figure 1 The flowchart of an embodiment of a data processing method provided in the embodiment of the present application is shown in FIG. 1 . In one embodiment, the method is executed by a computer program or device installed with the data processing method, such as Figure 1 As shown, the following steps are included:

[0090] Step 101: During the process of executing a concurrent task queue in a database, for any concurrent task in the concurrent task queue, monitoring data of the concurrent task queue during the execution of the current cycle is obtained.

[0091] The concurrent task queue in the database is a thread-safe queue that supports multiple threads or multiple tasks to perform insertion and deletion operations concurrently. The concurrent task queue in the embodiment of the present application contains multiple concurrent tasks for performing data export operations.

[0092] Specifically, a concurrent task queue can be constructed and submitted to the database for execution in the following manner: obtain data export instructions, parse the data export instructions to obtain field description information, assemble database statements according to the field description information, construct a concurrent task queue according to the database statements, and submit it to the database for execution.

[0093] In the monitoring process of executing concurrent task queues to the database, multiple monitoring cycles can be included. Specifically, the monitoring duration of each monitoring cycle can be set in advance, for example, the monitoring duration of a monitoring cycle is set to 2s. In addition, the monitoring cycle can also be automatically set according to the blocking situation of the current concurrent task queue, to which the present application embodiment is not limited.

[0094] In one embodiment, during the process of executing a concurrent task queue in a database, for any concurrent task in the concurrent task queue, a specific implementation method for obtaining monitoring data of the concurrent task queue during the execution of the current cycle is as follows: during the process of executing a concurrent task queue in a database, for any concurrent task in the concurrent task queue, the monitoring data corresponding to the current concurrent task in the current cycle is obtained by automatically reading logs or database prompt information.

[0095] For example, during the execution of the database, the logs in the execution of the database are automatically read by the method of reading the logs, and the monitoring data in the execution of the database is obtained by analyzing the logs. In addition, the monitoring data can also be obtained by other methods, which are not limited in the embodiments of the present application.

[0096] Step 102: Determine the resource allocation information corresponding to the concurrent tasks of the next period according to the monitoring data.

[0097] Monitoring data refers to real-time monitoring and management of data processing to ensure the quality, accuracy and security of data. The monitoring data in the embodiment of the present application may include but is not limited to the number of threads occupied by each concurrent task during the database execution process, the estimated completion time, the execution rate and the blocking situation in the concurrent task queue.

[0098] In one embodiment, based on the monitoring data of the current cycle obtained in step 101, the specific implementation method of determining the allocation resource information corresponding to the concurrent tasks of the next cycle according to the monitoring data can be: according to the blocking situation in the obtained monitoring data and the number of currently occupied threads corresponding to each concurrent task, when there are currently idle thread resources, the allocation resource information corresponding to the next cycle of each concurrent task is obtained. In addition, the allocation resource information corresponding to the concurrent tasks of the next cycle can also be obtained by other methods, which are not limited in the embodiments of the present application.

[0099] Step 103: Obtain the target number of threads corresponding to the concurrent tasks according to the allocated resource information.

[0100] The resource allocation information includes but is not limited to the estimated remaining time, the number of currently occupied threads, blocking information, execution rate, etc.

[0101] In one embodiment, the specific implementation method of obtaining the target number of threads corresponding to the concurrent task according to the allocated resource information is: the target number of threads corresponding to the concurrent task is calculated by the allocated resource information. For example, the target number of threads for the next cycle can be obtained by subtracting the number of currently occupied threads from the estimated remaining time in the allocated resource information, and the target number of threads can also be calculated by other methods. In addition, the target number of threads can also be obtained by other methods, which are not limited in the embodiments of the present application.

[0102] Step 104: Adjust the configuration file of the concurrent task according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform the data export operation in the next cycle.

[0103] The configuration file is the basis for the database to allocate thread resource information during the execution of concurrent tasks. For example, the configuration file records the thread resources corresponding to each concurrent task. When the thread resources required by the concurrent tasks change, it is necessary to modify the performance parameters of the thread resources in the configuration file so that the database can allocate thread resources according to the relevant performance parameters in the configuration file.

[0104] In one embodiment, the configuration file of the concurrent task is adjusted according to the target number of threads so that the database uses the configuration file to control the concurrent task to perform data export operations in the next cycle. The specific implementation method is: obtain the configuration file corresponding to the pre-set concurrent task, find the performance parameter information corresponding to the thread resources, and modify the performance parameter information according to the target number of threads, so that the database controls the concurrent task to perform data export operations in the next cycle according to the modified configuration file.

[0105] For example, the performance parameter of the thread resource corresponding to the current concurrent task in the preset configuration file is 5, and the target number of threads of the current concurrent task is 2. Then the specific operation of adjusting the configuration file is to modify the performance parameter in the above configuration file to 2.

[0106] The method provided by the embodiment of the present application utilizes a concurrent task queue to export data, and during the process of executing data export in the database, dynamically monitors the monitoring data of each concurrent task in the previous cycle, determines the allocation resource information corresponding to each concurrent task in the next cycle based on the monitoring data, determines the target number of threads based on the allocation resource information, and further dynamically adjusts the configuration file of the concurrent task based on the target number of threads, so that the database uses the configuration file to control the concurrent task to perform data export operations in the next cycle. This method realizes the reasonable allocation and maximum utilization of resources, reduces the resource idle rate, and effectively improves the data export performance by real-time monitoring and dynamically adjusting the performance parameter configuration of each working thread during the data export process.

[0107] Figure 2 A flowchart of another data processing method provided in an embodiment of the present application is provided. Figure 2 The embodiment shown in Figure 1 Based on the flowchart shown in the figure, it mainly describes how to determine the allocation resource information corresponding to the concurrent tasks of the next cycle according to the monitoring data, such as Figure 2 As shown, the following steps are included:

[0108] Step 201: During the process of executing a concurrent task queue in a database, for any concurrent task in the concurrent task queue, monitoring data of the concurrent task queue during the execution of the current cycle is obtained.

[0109] For a detailed description of step 201, see Figure 1 Description of the relevant steps in the illustrated embodiment.

[0110] Step 202: parse the monitoring data to obtain blocking information and allocation resource information corresponding to concurrent tasks.

[0111] The monitoring data may include blocking information and resource allocation information, wherein the blocking information may be a specific value, which is used to indicate whether the current concurrent task is blocked. For example, it may be specified that when the blocking information is 1, it indicates that the current concurrent task is blocked, and when the blocking information is 0, it indicates that the current concurrent task is not blocked. In addition, it is also possible to determine whether the current task is blocked by setting a blocking threshold, which is not limited in the embodiments of the present application.

[0112] The allocated resource information may include the remaining data volume, remaining time, minimum remaining time, processing rate, current number of threads, etc., which is described in detail in the relevant embodiments below.

[0113] In one embodiment, the specific implementation method of parsing the monitoring data to obtain the blocking information and the allocation resource information corresponding to the concurrent tasks is: obtaining the monitoring file, extracting the monitoring data in the monitoring file by performing text recognition and extraction, digital extraction, etc. on the monitoring file, and obtaining the blocking information and the allocation resource information corresponding to the concurrent tasks. In addition, the monitoring data can also be parsed by other methods, which are not limited in the embodiments of the present application.

[0114] Step 203: Determine whether the current concurrent task is blocked according to the blocking information.

[0115] In one embodiment, the specific implementation method of determining whether the current concurrent task is blocked according to the blocking information is: when the blocking information is greater than the blocking threshold, it is determined that the current concurrent task is blocked; when the blocking information is less than or equal to the blocking threshold, it is determined that the current concurrent task is not blocked.

[0116] For example, assuming that the preset blocking threshold is 1, the blocking information is 0, and 0 is less than 1, then it means that there is no blocking in the current concurrent task; assuming that the preset blocking threshold is 1, the blocking information is 2, and 2 is greater than 1, it means that there is blocking in the current concurrent task.

[0117] Step 204: When it is determined that the current concurrent task is blocked, the allocated resource information corresponding to the current concurrent task is obtained.

[0118] In one embodiment, when it is determined that the current concurrent task is blocked, the allocated resource information corresponding to the current concurrent task parsed from the monitoring data is obtained.

[0119] Step 205: Obtain the target number of threads corresponding to the concurrent tasks according to the allocated resource information.

[0120] Step 206: Adjust the configuration file of the concurrent task according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform the data export operation in the next cycle.

[0121] For detailed description of steps 205 to 206, see Figure 1 Description of the relevant steps in the illustrated embodiment.

[0122] pass Figure 2In the description of the embodiment shown, it is determined whether the current concurrent task is blocked based on the blocking information. If it is determined that the current concurrent task is blocked, the allocated resource information of the current concurrent task is obtained. If the current concurrent task is not blocked, it is indicated that the current concurrent task does not need to allocate resources. In this way, the accuracy of the obtained allocated resource information is improved, and it is no longer necessary to extract the required allocated resource information from the obtained resource information, further improving the efficiency of dynamically calculating the target number of threads.

[0123] Figure 3 A flowchart of another data processing method embodiment provided in the present application is provided. Figure 3 The embodiment shown in Figure 1 Based on the process shown in the figure, this paper mainly describes how to obtain the target number of threads corresponding to the concurrent tasks according to the allocated resource information, see Figure 3 , including the following steps:

[0124] Step 301: During the process of executing a concurrent task queue in a database, for any concurrent task in the concurrent task queue, monitoring data of the concurrent task queue during the execution of the current cycle is obtained.

[0125] Step 302: Determine the resource allocation information corresponding to the concurrent tasks of the next period according to the monitoring data.

[0126] For detailed description of steps 301 to 302, see Figure 1 Description of the relevant steps in the process shown.

[0127] Step 303: parse the allocated resource information to obtain the processing rate, the remaining data volume, the remaining time, and the current number of threads.

[0128] The processing rate refers to the processing rate at different stages during the execution of the concurrent task queue, including but not limited to the execution rate in the execution stage, the parsing rate in the parsing stage, the assembly rate in the assembly stage, etc.; the remaining data volume refers to the remaining data volume that has not yet been executed in the concurrent task; the remaining time refers to the remaining time required for the current concurrent task to be completed; the current number of threads refers to the number of threads currently occupied by the concurrent task; the minimum remaining time refers to the minimum value of the remaining time required for each concurrent task in the current concurrent task queue to be completed.

[0129] In one embodiment, the processing rate, the amount of remaining data, the remaining time, the current number of threads, etc. are extracted from the obtained allocated resource information.

[0130] Step 304: Calculate the expected rate according to the remaining data amount and the minimum remaining time, wherein the minimum remaining time indicates the minimum value of the remaining time required for the concurrent tasks to be completed.

[0131] In one embodiment, the expected rate calculated based on the remaining data amount and the minimum remaining time can be calculated by the following formula (1):

[0132]

[0133] In the above formula (1), T represents the minimum remaining time, which refers to the minimum value of the remaining time required for each concurrent task in the current concurrent task queue to complete execution; S represents the remaining data volume, which refers to the remaining data volume that has not been executed in the concurrent task.

[0134] For example, if the remaining data amount is 10 and the minimum remaining time is 4, the expected rate is 2.5.

[0135] The expected rate calculated in this way is more accurate, which helps to use the expected rate to calculate the target number of threads later.

[0136] Step 305: Calculate the target number of threads according to the expected rate, the processing rate, and the current number of threads.

[0137] In one embodiment, the target number of threads calculated according to the expected rate, the processing rate and the current number of threads can be calculated by the following formula (II):

[0138]

[0139] In the above formula (2), N represents the expected rate, which is calculated by the above formula (1); H represents the processing rate, which refers to the processing rate at different stages during the execution of the concurrent task queue, including but not limited to the execution rate in the execution stage, the parsing rate in the parsing stage, the assembly rate in the assembly stage, etc.; a represents the current number of threads, which refers to the number of threads currently occupied by the concurrent task; m represents the target number of threads.

[0140] Step 306: Adjust the configuration file of the concurrent task according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform the data export operation in the next cycle.

[0141] For a detailed description of step 306, see Figure 1 Description of the relevant steps in the process shown.

[0142] pass Figure 3 In the description of the embodiment shown, the target number of threads corresponding to the current concurrent task in the next cycle is dynamically calculated using the allocated resource information, so as to dynamically adjust the configuration file according to the target number of threads. By dynamically calculating the target number of threads that need to be reallocated for the concurrent tasks that are blocked in the next monitoring cycle according to the real-time monitoring data, the reasonable allocation and maximum utilization of resources are achieved.

[0143] Figure 4 A flowchart of another data processing method embodiment provided in the present application is shown below. Figure 4 The embodiment shown in Figure 1 The process shown mainly describes how to adjust the configuration file of concurrent tasks according to the target number of threads, including the following steps:

[0144] Step 401: During the process of executing a concurrent task queue in a database, for any concurrent task in the concurrent task queue, monitoring data of the concurrent task queue during the execution of the current cycle is obtained.

[0145] Step 402: Determine the resource allocation information corresponding to the concurrent tasks of the next period according to the monitoring data.

[0146] Step 403: Obtain the target number of threads corresponding to the concurrent tasks according to the allocated resource information.

[0147] For detailed description of steps 401 to 403, see the above Figure 1 Description of the relevant steps of the process shown.

[0148] Step 404: Obtain the configuration file of the concurrent task, and parse the performance parameter information corresponding to the configuration file.

[0149] The preset configuration file records relevant performance parameter information such as the number of threads and database connection information required during the execution of concurrent tasks.

[0150] In one embodiment, the performance parameter information in the configuration file is extracted by identifying and reading the configuration file.

[0151] Step 405: parse the resource allocation information to obtain the remaining time of any concurrent task in the concurrent task queue, and arrange the remaining time in descending order, wherein the larger the remaining time, the higher the priority.

[0152] Based on the relevant description of the above steps, the resource allocation information includes remaining time information, wherein the remaining time refers to the remaining time required for the current concurrent task to be executed and completed.

[0153] In one embodiment, the resource allocation information is parsed to obtain the remaining time of any concurrent task in the concurrent task queue, and the remaining time is arranged in order from large to small, wherein the larger the remaining time, the higher the priority.

[0154] For example, if the current concurrent task queue contains 5 concurrent tasks, and the remaining times of concurrent tasks 1 to 5 are 6, 8, 7, 1, and 9 respectively, then the order of arranging these remaining times from large to small is 9, 8, 7, 6, and 1, where the larger the remaining time, the higher the priority. This means that concurrent task 5 with a remaining time of 9 has the highest priority, and concurrent task 4 with a remaining time of 1 has the lowest priority.

[0155] Step 406: Arrange any concurrent tasks in the concurrent task queue in descending order of remaining time to obtain priority information.

[0156] In one embodiment, priority information is obtained by arranging any concurrent task in the concurrent task queue in descending order according to the remaining time. For example, the priority information obtained based on the order in which the remaining time corresponding to the five concurrent tasks in the above step 405 is arranged from large to small is concurrent task 5, concurrent task 2, concurrent task 3, concurrent task 1, and concurrent task 4. Based on this order, it means that concurrent task 5 has the highest priority and concurrent task 4 has the lowest priority.

[0157] Step 407: Use the target number of threads as a performance parameter to adjust the configuration file of the concurrent task according to the priority information, so that the database uses the configuration file to control the concurrent task to perform the data export operation in the next cycle.

[0158] In one embodiment, the performance parameters in the configuration file are adjusted preferentially for concurrent tasks with high priority in order of priority, so that the database preferentially allocates target numbers of threads to concurrent tasks with high priority according to the configuration file, and then performs data export operations in the next monitoring cycle according to the corresponding target number of threads.

[0159] For example, if a concurrent task queue contains 3 concurrent tasks, and the estimated remaining time for each concurrent task to complete is 10s, 3s, and 5s respectively, and the calculated target number of threads that need to be reallocated for each concurrent task to complete is 3, 5, and 6 respectively, then the concurrent task with an estimated remaining time of 10s is given priority to be allocated a target number of 3 threads, and then the concurrent task with an estimated remaining time of 5s is allocated a target number of 6 threads, and finally the concurrent task with an estimated remaining time of 3s is allocated a target number of 5 threads.

[0160] pass Figure 4 The description of the illustrated embodiment utilizes the target number of threads to dynamically modify the relevant performance parameter information in the configuration file so that the database uses the configuration file to allocate the number of threads for the next cycle to concurrent tasks, thereby dynamically adjusting performance parameters, reducing resource idle rate, and improving data export performance.

[0161] Figure 5A data processing device is provided in an embodiment of the present application. Figure 5 As shown, the device comprises:

[0162] The data execution module 51 is used for acquiring monitoring data of the concurrent task queue during the execution of the concurrent task queue in the current cycle for any concurrent task in the concurrent task queue during the execution of the concurrent task queue in the database;

[0163] A data monitoring module 52, used to determine the allocation resource information corresponding to the concurrent tasks in the next cycle according to the monitoring data;

[0164] A thread scheduling module 53, configured to obtain a target number of threads corresponding to the concurrent task according to the allocated resource information;

[0165] The data export module 54 is used to adjust the configuration file of the concurrent task according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform data export operations in the next cycle.

[0166] In a possible implementation, the device further includes:

[0167] An instruction issuing unit, used for obtaining a data export instruction and parsing the data export instruction to obtain field description information;

[0168] A task construction unit, used to obtain a database statement according to the field description information, and to construct a concurrent task queue according to the database statement;

[0169] The task submission unit is used to submit the concurrent task queue to the database so that the database executes the concurrent tasks in the concurrent task queue.

[0170] In a possible implementation, the data monitoring module 52 includes:

[0171] A data parsing unit, used for parsing the monitoring data to obtain blocking information and allocation resource information corresponding to the concurrent tasks;

[0172] A blocking judgment unit, used to determine whether the current concurrent task is blocked according to the blocking information;

[0173] The resource acquisition unit is used to acquire the allocation resource information corresponding to the current concurrent task when it is determined that the current concurrent task is blocked.

[0174] In a possible implementation manner, the blocking determination unit is specifically configured to:

[0175] When the blocking information is greater than the blocking threshold, determining that the current concurrent task is blocked;

[0176] When the blocking information is less than or equal to the blocking threshold, it is determined that there is no blocking for the current concurrent task.

[0177] In a possible implementation, the thread scheduling module 53 is specifically used to:

[0178] Parsing the allocated resource information to obtain a processing rate, a remaining data volume, a remaining time, and a current number of threads;

[0179] Calculating an expected rate according to the remaining data amount and the minimum remaining time, wherein the minimum remaining time indicates a minimum value of the remaining time required for the concurrent tasks to be completed;

[0180] The target number of threads is calculated according to the expected rate, the processing rate and the current number of threads.

[0181] In a possible implementation, the data export module 54 includes:

[0182] A file acquisition unit, used to acquire the configuration file of the concurrent task;

[0183] A file parsing unit, used to parse the performance parameter information corresponding to the configuration file;

[0184] The file adjustment unit is used to adjust the performance parameter information using the target number of threads to perform adjustment on the configuration file of the concurrent task.

[0185] In a possible implementation manner, the file adjustment unit is specifically used to:

[0186] Parsing the allocated resource information to obtain the remaining time of any concurrent task in the concurrent task queue;

[0187] Arrange the remaining time in descending order, wherein the larger the remaining time, the higher the priority;

[0188] Arrange any concurrent task in the concurrent task queue in descending order according to the remaining time to obtain priority information;

[0189] The target number of threads is used as the performance parameter to adjust the configuration file of the concurrent task according to the priority information.

[0190] like Figure 6 As shown, an embodiment of the present application provides a device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0191] Memory 113, used for storing computer programs;

[0192] In one embodiment of the present application, the processor 111 is used to implement the data processing method provided by any one of the aforementioned method embodiments when executing the program stored in the memory 113, including:

[0193] During the process of executing the concurrent task queue in the database, for any concurrent task in the concurrent task queue, obtaining monitoring data of the concurrent task queue during the execution of the concurrent task queue in the current cycle;

[0194] Determine the allocation resource information corresponding to the concurrent tasks in the next cycle according to the monitoring data;

[0195] Obtaining the target number of threads corresponding to the concurrent task according to the allocated resource information;

[0196] The configuration file of the concurrent task is adjusted according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform the data export operation in the next cycle.

[0197] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the data processing method provided in any of the aforementioned method embodiments are implemented.

[0198] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0200] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0201] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A data processing method, characterized in that: The method comprises: During the process of executing the concurrent task queue in the database, for any concurrent task in the concurrent task queue, obtaining monitoring data of the concurrent task queue during the execution of the concurrent task queue in the current cycle; Determine the allocation resource information corresponding to the concurrent tasks in the next cycle according to the monitoring data; Obtaining the target number of threads corresponding to the concurrent task according to the allocated resource information; The configuration file of the concurrent task is adjusted according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform the data export operation in the next cycle.

2. The method according to claim 1, characterized in that: The method further comprises: Obtaining a data export instruction, parsing the data export instruction to obtain field description information; Obtaining a database statement according to the field description information, and building a concurrent task queue according to the database statement; The concurrent task queue is submitted to a database so that the database executes the concurrent tasks in the concurrent task queue.

3. The method according to claim 1, characterized in that: The determining, according to the monitoring data, the allocation resource information corresponding to the concurrent tasks in the next cycle includes: Parsing the monitoring data to obtain blocking information and allocation resource information corresponding to the concurrent tasks; Determine whether the current concurrent task is blocked according to the blocking information; When it is determined that the current concurrent task is blocked, the allocation resource information corresponding to the current concurrent task is obtained.

4. The method according to claim 3, characterized in that The determining whether the current concurrent task is blocked according to the blocking information includes: When the blocking information is greater than the blocking threshold, determining that the current concurrent task is blocked; When the blocking information is less than or equal to the blocking threshold, it is determined that there is no blocking for the current concurrent task.

5. The method according to claim 1, characterized in that The obtaining the target number of threads corresponding to the concurrent task according to the allocated resource information includes: Parsing the allocated resource information to obtain a processing rate, a remaining data volume, a remaining time, and a current number of threads; Calculating an expected rate according to the remaining data amount and the minimum remaining time, wherein the minimum remaining time indicates a minimum value of the remaining time required for the concurrent tasks to be completed; The target number of threads is calculated according to the expected rate, the processing rate and the current number of threads.

6. The method according to claim 1, characterized in that The adjusting the configuration file of the concurrent task according to the target number of threads includes: Obtaining a configuration file for the concurrent task; Parsing performance parameter information corresponding to the configuration file; The performance parameter information is adjusted using the target number of threads to perform adjustment on the configuration file of the concurrent task.

7. The method according to claim 6, characterized in that The step of adjusting the performance parameter information by using the target number of threads to adjust the configuration file of the concurrent task includes: Parsing the allocated resource information to obtain the remaining time of any concurrent task in the concurrent task queue; Arrange the remaining time in descending order, wherein the larger the remaining time, the higher the priority; Arrange any concurrent task in the concurrent task queue in descending order according to the remaining time to obtain priority information; The target number of threads is used as the performance parameter to adjust the configuration file of the concurrent task according to the priority information.

8. A data processing device, characterized in that: The device comprises: A data execution module is used to obtain monitoring data of the concurrent task queue during the execution of the concurrent task queue in the current cycle for any concurrent task in the concurrent task queue during the execution of the concurrent task queue in the database; A data monitoring module, used to determine the allocation resource information corresponding to the concurrent tasks in the next cycle according to the monitoring data; A thread scheduling module, used to obtain the target number of threads corresponding to the concurrent task according to the allocated resource information; The data export module is used to adjust the configuration file of the concurrent task according to the target number of threads, so that the database uses the configuration file to control the concurrent task to perform data export operations in the next cycle.

9. A device, characterized in that: include: A processor and a memory, wherein the processor is used to execute a data processing program stored in the memory to implement the data processing method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the data processing method according to any one of claims 1 to 7.

Images