Data batch processing method and device and storage medium
By calculating the amount of data accumulated in the message queue of each data processing task, flexibly allocating computing resources and adjusting the batch processing size, the problems of resource waste and latency in different data processing tasks are solved, and more efficient data processing is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG DAHUA TECH CO LTD
- Filing Date
- 2025-12-30
- Publication Date
- 2026-05-01
AI Technical Summary
In big data scenarios, using the same batch processing method to process different data can lead to congestion and accumulation of large and high-volume data, while small and low-volume data may experience delays or resource waste. Existing technologies cannot make reasonable use of computing resources, resulting in low data processing efficiency.
By calculating the amount of data accumulated in the message queue of each data processing task, allocating corresponding computing resources and setting the batch data extraction quantity of processing threads, the allocation of computing resources can be flexibly adjusted according to the amount of data accumulated, ensuring that tasks with a large amount of accumulated data receive more resources for processing and avoiding resource waste.
It improves resource utilization and data processing efficiency, avoids data congestion and loss, and optimizes data processing speed.
Smart Images
Figure CN121957883A_ABST
Abstract
Description
Data batch processing methods, equipment and storage media Technical Field
[0001] This application relates to the field of big data technology, and in particular to a method, device and storage medium for batch data processing. Background Technology
[0002] Big data scenarios involve concurrent batch processing of data. For example, data may need to be structured and organized in batches before it can be stored in a database. This process involves a large number of concurrent processes to handle different data in different ways.
[0003] However, since different data have different sizes and volumes, if the same batch processing method is used, large data with high volume will be congested and accumulated, while small data with low volume will experience delays or waste of resources. Summary of the Invention
[0004] To address the aforementioned technical problems, this application provides at least one data batch processing method, apparatus, and storage medium.
[0005] The first aspect of this application provides a data batch processing method, which creates multiple data processing tasks. Each data processing task consists of a message queue and a processing thread. The message queue is used to cache data, and the processing thread is used to extract data from the message queue for processing. The method includes: calculating the amount of data accumulated in the message queue of each data processing task to obtain the task accumulation amount corresponding to each data processing task; allocating the total allocatable computing resources to each data processing task based on the task accumulation amount corresponding to each data processing task to obtain the computing resource amount corresponding to each data processing task; wherein, the task accumulation amount and the computing resource amount are positively correlated; setting the number of data to be extracted in each batch by the processing thread of each data processing task based on the computing resource amount corresponding to each data processing task to obtain the batch size; and controlling each processing thread to extract data from the message queue for processing based on the batch size corresponding to each processing thread.
[0006] In one embodiment, the amount of data accumulated in the message queue of each data processing task is calculated to obtain the task accumulation amount corresponding to each data processing task. This includes: calculating the average size of the data in the message queue to obtain the average data size; counting the total number of data contained in the message queue to obtain the total amount of accumulated data; and multiplying the average data size and the total amount of accumulated data to obtain the task accumulation amount.
[0007] In one embodiment, the computing resources include memory resources; based on the task backlog corresponding to each data processing task, the total allocatable computing resources are allocated to each data processing task to obtain the computing resource amount corresponding to each data processing task, including: obtaining the total allocatable memory resources; based on the task backlog corresponding to each data processing task, the total allocatable memory resources are allocated to each data processing task to obtain the available memory size corresponding to each data processing task.
[0008] In one embodiment, based on the task backlog corresponding to each data processing task, the total allocatable memory resources are allocated to each data processing task to obtain the available memory size corresponding to each data processing task. This includes: calculating the sum of the task backlog of each data processing task to obtain the total backlog; calculating the ratio between the task backlog of each data processing task and the total backlog to obtain the backlog percentage corresponding to each data processing task; and multiplying the backlog percentage corresponding to each data processing task by the total allocatable memory resources to obtain the available memory size corresponding to each data processing task.
[0009] In one embodiment, based on the computing resources corresponding to each data processing task, the number of data to be extracted in each batch by the processing thread of each data processing task is set to obtain the batch size. This includes: calculating the average size of the data in the message queue to obtain the average data size; and calculating the number of data copies during the statistical data processing process. Based on the available memory size, average data size, and number of data copies of the data processing task, the number of data to be extracted in each batch by the processing thread of the data processing task is calculated to obtain the batch size.
[0010] In one embodiment, the method further includes: detecting whether the adjustment triggering condition is met; if so, readjusting the batch size corresponding to each data processing task; wherein the adjustment triggering condition includes one or more of the following conditions: Condition 1: the time interval since the last adjustment reaches a preset time interval threshold; Condition 2: counting the number of data processing tasks with a task backlog greater than a preset backlog threshold to obtain the number of overloaded tasks, and the number of overloaded tasks is greater than a preset task number threshold; Condition 3: calculating the change in the total available computing resources relative to the total available computing resources in the last adjustment to obtain the total resource change, and the total resource change is greater than a preset change threshold; Condition 4: there is a new data processing task.
[0011] In one embodiment, the method further includes: detecting the amount of computing resources used; if the amount of computing resources used is greater than a preset maximum usage threshold, obtaining the task priority corresponding to each data processing task, selecting data processing tasks whose task priorities meet preset conditions as degraded tasks, and reducing the batch size corresponding to the degraded tasks.
[0012] In one embodiment, obtaining the task priority corresponding to each data processing task includes: detecting the task urgency, and / or data accumulation rate, and / or task accumulation amount of the data processing task; setting the task priority corresponding to the data processing task based on the task urgency, and / or data accumulation rate, and / or task accumulation amount of the data processing task; wherein, the task urgency, data accumulation rate, and task accumulation amount are all positively correlated with the task priority.
[0013] A second aspect of this application provides a data batch processing apparatus, comprising: a backlog calculation module for calculating the amount of data backlogged in the message queue of each data processing task, thereby obtaining the task backlog for each data processing task; a resource allocation module for allocating the total available computing resources to each data processing task based on the task backlog, thereby obtaining the computing resource amount for each data processing task; wherein the task backlog and the computing resource amount are positively correlated; a batch size setting module for setting the number of data items extracted by each processing thread of each data processing task in each batch based on the computing resource amount for each data processing task, thereby obtaining the batch size; and a data processing module for controlling each processing thread to extract data from the message queue for processing based on the batch size corresponding to each processing thread.
[0014] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute program instructions stored in the memory to implement the above-described data batch processing method.
[0015] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described data batch processing method.
[0016] The above scheme calculates the amount of data accumulated in the message queue of each data processing task to obtain the task backlog for each task. Based on the task backlog for each task, the total available computing resources are allocated to each task to obtain the computing resource amount for each task. Based on the computing resource amount for each task, the batch size is determined by setting the number of data points that each processing thread extracts in each batch. Based on the batch size for each thread, the scheme controls the extraction of data from the message queue for processing. This allows for flexible adjustment of computing resource allocation based on the actual data backlog in the message queue of each task. This enables tasks with large backlogs to select more data in each batch and process it using sufficient computing resources. Larger batch sizes result in faster data processing, preventing congestion and data loss in the message queues. Conversely, tasks with small backlogs are protected from excessive computing resource allocation, thus increasing the amount of data processed simultaneously with limited computing resources, improving resource utilization and data processing efficiency.
[0017] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description
[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.
[0019] Figure 1 is a schematic diagram of batch processing of multi-source data in the prior art; Figure 2 is a flowchart of a data batch processing method illustrated in an exemplary embodiment of this application; Figure 3 is a schematic diagram of data storage illustrated in an exemplary embodiment of this application; Figure 4 is a flowchart of a data batch processing method illustrated in another exemplary embodiment of this application; Figure 5 is a block diagram of a data batch processing apparatus illustrated in an exemplary embodiment of this application; Figure 6 is a structural schematic diagram of an electronic device illustrated in an exemplary embodiment of this application; Figure 7 is a structural schematic diagram of a computer-readable storage medium illustrated in an exemplary embodiment of this application. Detailed Implementation
[0020] The embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0021] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.
[0022] In this document, the term "and / or" is merely a description of the association information of related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.
[0023] In scenarios involving multi-source data (such as different data types or different data sources), different data processing tasks are generally used to process different data. However, since different data have different data sizes and data volumes, if the same batch processing method is used, it will lead to congestion and accumulation of large data with high volume, while small data with low volume will experience delays or waste of resources.
[0024] For example, please refer to Figure 1, which is a schematic diagram of batch processing of multi-source data in the prior art. As shown in Figure 1, taking image data and text record data as examples, the image data and text record data are batch-structured and stored in the database. In actual scenarios, the number of image data received per unit time (e.g., per second) is much higher than the number of text record data received, and a single image data is larger than a single text record data. If 1000 data entries are processed at a time, a large amount of image data will accumulate in the message queue or even be lost; while the number of text record data generated per unit time is far less than 1000, so the processing system needs to wait for a long time for data generation, resulting in a long delay in the database before the latest text record data can be found.
[0025] Therefore, in the existing technology, all data processing tasks use the same batch processing method, which cannot make reasonable use of computing resources and results in low data processing efficiency.
[0026] To address the aforementioned technical problems, this application provides at least one data batch processing method, apparatus, and storage medium.
[0027] The data batch processing method provided in the embodiments of this application will be described below.
[0028] The data batch processing method provided in this application can be executed by any electronic device with computing resources. For example, the execution subject of each step can be a terminal (such as a desktop computer, smartphone, tablet computer, laptop computer, etc.), or a server (such as an independent physical server, server cluster, cloud server, etc.), or the terminal and server can interact and cooperate to execute the method. That is, some steps of the method are executed by the terminal and other steps are executed by the server. This application does not limit this.
[0029] Please refer to Figure 2, which is a flowchart of a data batch processing method illustrated in an exemplary embodiment of this application. As shown in Figure 2, the data batch processing method includes at least steps S210 to S240, which are described in detail below: Step S210: Calculate the amount of data accumulated in the message queue of each data processing task to obtain the task accumulation amount corresponding to each data processing task.
[0030] Multiple data processing tasks are created. Each data processing task consists of a message queue and a processing thread. The message queue is used to cache data, and the processing thread is used to extract data from the message queue for processing.
[0031] Taking the data entry scenario as an example, please refer to Figure 3. Figure 3 is a schematic diagram of data entry shown in an exemplary embodiment of this application. As shown in Figure 3, it includes message reception and message processing. When multiple types of data need to be stored in the database at the same time, multiple concurrent data processing tasks will be created. The data processing task consists of a message queue and a processing thread. The message queue stores the data sent by the data source. The processing thread is used to extract n data from the message queue in batches for structured processing and store the structured data in the database.
[0032] It should be noted that the data processing task in Figure 3, consisting of a message queue and a processing thread, is only an illustrative example. In actual application scenarios, a data processing task may contain multiple message queues or multiple processing threads.
[0033] In addition to the processing thread used for structuring data as exemplified above, in other application scenarios, the processing thread can also perform other data processing, such as data filtering, data export, and data merging. This application does not limit the specific implementation of the processing thread algorithm.
[0034] Calculate the amount of data accumulated in the message queue of the data processing task to obtain the task backlog corresponding to that data processing task.
[0035] For example, the total number of data items in the message queue can be counted to obtain the task backlog.
[0036] For example, calculate the total size of all data in the message queue to obtain the task backlog.
[0037] Step S220: Based on the task backlog corresponding to each data processing task, allocate the total allocable computing resources to each data processing task to obtain the computing resource amount corresponding to each data processing task; wherein, the task backlog and the computing resource amount are positively correlated.
[0038] Based on the task backlog corresponding to each data processing task, the total available computing resources are allocated to each data processing task.
[0039] Specifically, the amount of tasks backlog and the amount of computing resources are positively correlated; that is, the larger the amount of tasks backlog, the larger the amount of computing resources allocated; and the smaller the amount of tasks backlog, the smaller the amount of computing resources allocated.
[0040] The computing resources are the computer resources required for the processing thread to execute the corresponding computing task, such as memory and central processing unit (CPU). The specific resources are determined according to the algorithm of the processing thread, and this application does not limit them.
[0041] Step S230: Based on the computing resources corresponding to each data processing task, set the number of data to be extracted in each batch by the processing thread of each data processing task to obtain the batch size.
[0042] After obtaining the amount of computing resources allocated to each data processing task, the number of data to be extracted by each processing thread in each batch is set according to the allocated computing resources, thus obtaining the batch size.
[0043] For example, you can calculate the amount of computing resources required by a processing thread to process a single piece of data within a historical time period. Then, based on the allocated computing resources and the amount of computing resources required to process a single piece of data, you can calculate the number of data that the processing thread can extract in batches each time, and thus obtain the batch size.
[0044] For example, a mapping function is set for each data processing task. This mapping function represents the mapping relationship between the amount of computing resources and the batch size. Substituting the allocated computing resources into this mapping function yields the batch size. The mapping function can be preset based on experience or obtained by statistically analyzing data processing over a historical period; there is no inherent limitation on either.
[0045] Step S240: Based on the batch size corresponding to each processing thread, control each processing thread to extract data from the message queue for processing.
[0046] Each processing thread extracts data from the corresponding message queue for processing based on the batch size.
[0047] The data batch processing method of this application calculates the amount of data accumulated in the message queue of each data processing task to obtain the task accumulation amount corresponding to each data processing task; based on the task accumulation amount corresponding to each data processing task, the total allocable computing resources are allocated to each data processing task to obtain the computing resource amount corresponding to each data processing task; based on the computing resource amount corresponding to each data processing task, the number of data to be extracted in each batch by the processing thread of each data processing task is set to obtain the batch size; based on the batch size corresponding to each processing thread, the data extraction and processing of each processing thread from the message queue is controlled. The allocation of computing resources can be flexibly adjusted according to the actual data accumulation in the message queue of each data processing task, so that data processing tasks with large amounts of accumulated data can select more data in each batch and then be processed in batches with sufficient computing resources. That is, the larger the batch size, the faster the data processing speed, avoiding data congestion and accumulation in the message queue and data loss. In addition, it avoids the waste of resources caused by allocating too many computing resources to data processing tasks with small amounts of accumulated data. Thus, the amount of data processed simultaneously can be increased with limited computing resources, improving resource utilization and data processing efficiency.
[0048] The following describes some embodiments of this application in detail.
[0049] In some implementations, the method further includes: detecting whether the adjustment trigger condition is met; if so, executing a data batch processing method to readjust the batch size corresponding to each data processing task.
[0050] The adjustment trigger conditions include one or more of the following conditions: Condition 1: The time interval since the last adjustment reaches a preset time interval threshold; Condition 2: The number of data processing tasks with a backlog greater than a preset backlog threshold is counted to obtain the number of overloaded tasks, and the number of overloaded tasks is greater than a preset task number threshold; Condition 3: The change in the total available computing resources relative to the total available computing resources in the last adjustment is calculated to obtain the total resource change, and the total resource change is greater than a preset change threshold; Condition 4: There are newly added data processing tasks.
[0051] Specifically, the preset time interval threshold in condition one can be pre-set based on experience or flexibly calculated based on the traffic fluctuation of each data processing task; the preset task number threshold in condition two can be pre-set based on experience or flexibly calculated based on the total number of data processing tasks; and the preset change threshold in condition three can be pre-set based on experience or flexibly calculated based on the total amount of computing resources.
[0052] For example, the calculation of the preset time interval threshold is explained in detail as follows: Record the number of data received by the data processing task per unit time within a historical time period. Based on the recorded results, calculate the traffic fluctuation degree of the data processing task. The greater the traffic fluctuation degree, the greater the difference in the number of data received by the data processing task at different times. The smaller the data traffic fluctuation degree, the smaller the difference in the number of data received by the data processing task at different times. Then, based on the traffic fluctuation degree of each data processing task, set the preset time interval threshold.
[0053] For example, determine the average or maximum value corresponding to the traffic fluctuation level of each data processing task, and set a preset time interval threshold based on the average or maximum value. The larger the average or maximum value, the smaller the preset time interval threshold, and vice versa.
[0054] When the traffic fluctuation of each data processing task is small, it indicates that the data reception of each data processing task is relatively stable. In this case, it is less likely to trigger the adjustment of the batch size to save computing resources. When the traffic fluctuation of each data processing task is large, it indicates that the data reception of each data processing task is unstable. In this case, it is more likely to trigger the adjustment of the batch size to adapt to the current situation.
[0055] For example, if the time interval since the last adjustment is detected to be more than 5 minutes, a data batch processing method is executed; and / or, if the number of overloaded tasks is detected to be greater than 3, a data batch processing method is executed; and / or, if the total resource change is detected to be greater than 20 megabytes (Mb), a data batch processing method is executed; and / or, if a new data processing task is detected, a data batch processing method is executed 5 minutes after the new data processing task has been running.
[0056] It is understandable that multiple conditions can be used individually or in combination. When used in combination, it can mean that all conditions in the combination must be satisfied simultaneously, or it can mean that any one condition in the combination must be satisfied.
[0057] In some implementations, step S210 calculates the amount of data accumulated in the message queue of each data processing task to obtain the task accumulation amount corresponding to each data processing task, including the following steps S211 to S213.
[0058] Step S211: Calculate the average size of the data in the message queue to obtain the average data size.
[0059] For example, record the size of each piece of data received by the message queue within a preset time period, and calculate the average size of the data based on the size of each piece of data recorded.
[0060] Step S212: Count the total number of data in the message queue to obtain the total amount of accumulated data.
[0061] Step S213: Calculate the task backlog by multiplying the average data size and the total backlog of data.
[0062] Determine the total number of data items in the current message queue to obtain the total backlog of data. Then, multiply the average data size and the total backlog of data to calculate the task backlog.
[0063] Of course, the total amount of data can be used directly as the task backlog, or the maximum or median value of the data received within a preset time period can be used as the statistical record, and the maximum or median value can be multiplied by the total amount of backlog data to obtain the task backlog. This application does not limit the specific calculation method of the task backlog.
[0064] In some implementations, computing resources include memory resources; in step S220, based on the task backlog corresponding to each data processing task, the total allocatable computing resources are allocated to each data processing task to obtain the computing resource amount corresponding to each data processing task, including the following steps S221 to S222.
[0065] Step S221: Obtain the total allocatable memory resources.
[0066] The total allocatable memory resources can be preset based on experience, or it can be calculated by detecting the current memory usage, determining the amount of memory resources occupied by tasks other than data processing tasks, and then calculating the total allocatable memory resources for all data processing tasks based on the amount of memory resources occupied by other tasks.
[0067] Step S222: Based on the task backlog corresponding to each data processing task, allocate the total allocatable memory resources to each data processing task to obtain the available memory size corresponding to each data processing task.
[0068] Based on the task backlog corresponding to each data processing task, the total memory resources are allocated to each data processing task.
[0069] Specifically, the larger the value of the task backlog for each data processing task, the larger the available memory size allocated to that data processing task; conversely, the smaller the value of the task backlog for each data processing task, the smaller the available memory size allocated to that data processing task.
[0070] For example, based on the task backlog corresponding to each data processing task, the total memory resources are allocated to each data processing task to obtain the available memory size corresponding to each data processing task. This includes: calculating the sum of the task backlog of each data processing task to obtain the total backlog; calculating the ratio between the task backlog of each data processing task and the total backlog to obtain the backlog percentage corresponding to each data processing task; and multiplying the backlog percentage corresponding to each data processing task by the total allocable memory resources to obtain the available memory size corresponding to each data processing task.
[0071] For example, the available memory size for a data processing task can be calculated using the following formula 1: (Formula 1) Where, Let i be the available memory size corresponding to the i-th data processing task. Let be the average data size corresponding to the i-th data processing task. Let i be the total amount of accumulated data corresponding to the i-th data processing task. This represents the total accumulation. Total allocatable memory resources.
[0072] In some implementations, in conjunction with the above embodiments, step S230 sets the number of data extracted by the processing thread of each data processing task in each batch based on the amount of computing resources corresponding to each data processing task, thereby obtaining the batch size, including the following steps S231 to S232.
[0073] Step S231: Calculate the average size of the data in the message queue to obtain the average data size; and the number of data copies during the statistical data processing.
[0074] Data processing typically requires data copying, which consumes memory resources. Therefore, the number of data copies during statistical data processing is a factor to consider.
[0075] Step S232: Based on the available memory size, average data size, and number of data copies of the data processing task, calculate the number of data extracted by the processing thread of the data processing task in each batch, and obtain the batch size.
[0076] For example, for a single data processing task, assuming the average size of each data item is msgSize, the number of data items extracted by the processing thread in each batch is msgCount, and the number of data copies during processing is copyCount, then the memory used by the processing thread when the data processing task runs, useMemory, can be calculated by the following formula 2: useMemory = msgSize × msgCount × copyCount (Formula 2). According to Formula 2, given that the available memory size, average data size, and number of data copies are known, the number of data items extracted by the processing thread in each batch of the data processing task can be calculated, and the batch size can be obtained.
[0077] Of course, the above embodiments are illustrated using memory resources as an example. For other types of computing resources, different methods may be used to calculate the batch size. The specific calculation method for the batch size can be adjusted according to the type of computing resource, and this application does not limit it in this regard.
[0078] In some implementations, the method further includes: detecting the amount of computing resources used; if the amount of computing resources used is greater than a preset maximum usage threshold, obtaining the task priority corresponding to each data processing task, selecting data processing tasks whose task priorities meet preset conditions as degraded tasks, and reducing the batch size corresponding to the degraded tasks.
[0079] The maximum usage threshold can be preset based on experience or calculated based on the total available computing resources, such as a maximum usage threshold of 95% of the total available computing resources.
[0080] Check whether the amount of computing resources used exceeds the preset maximum usage threshold.
[0081] For example, taking memory resources as an example, based on the above formula 2, the memory used by each data processing task when running the processing thread (useMemory) can be calculated, and then summed to obtain the total memory used by all data processing tasks. Then, it can be checked whether the total memory used is greater than the maximum usage threshold.
[0082] Of course, in addition to calculating the amount of computing resources used by each data processing task in the above embodiments, the actual usage of computing resources (which may be shared by each data processing task and other tasks) can also be read directly to obtain the amount of computing resources used and to detect whether the total amount of memory used is greater than the maximum usage threshold. This application does not limit this.
[0083] If the amount of computing resources used exceeds the preset maximum usage threshold, then obtain the task priority corresponding to each data processing task.
[0084] The priority of data processing tasks can be preset based on experience or set flexibly according to the actual situation.
[0085] For example, the task priority for each data processing task is obtained, including: detecting the urgency of the data processing task, and / or the data accumulation rate, and / or the task accumulation amount; setting the task priority for the data processing task based on the urgency of the data processing task, and / or the data accumulation rate, and / or the task accumulation amount; wherein, the urgency of the task, the data accumulation rate, and the task accumulation amount are all positively correlated with the task priority.
[0086] Specifically, the higher the urgency of the task, and / or the higher the data accumulation rate, and / or the greater the amount of tasks in the backlog, the higher the task priority; the lower the urgency of the task, and / or the lower the data accumulation rate, and / or the smaller the amount of tasks in the backlog, the lower the task priority.
[0087] The methods for selecting downgraded tasks can include: selecting a preset number of data processing tasks with the lowest priority to obtain downgraded tasks; or selecting data processing tasks with a priority lower than a preset priority to obtain downgraded tasks.
[0088] The preset quantity and preset priority can be determined based on the difference between the amount of computing resources used and the maximum usage threshold. The larger the difference, the larger the preset quantity and the higher the preset priority. The smaller the difference, the smaller the preset quantity and the lower the preset priority.
[0089] Selected data processing tasks are designated as degradation tasks, and their corresponding batch sizes are reduced. This can be done by reducing the batch size of each degradation task by the same amount, or by determining the reduction based on the task priority of each degradation task; lower priority tasks result in greater batch size reductions.
[0090] The above embodiments can prioritize reducing the batch size of data processing tasks with low urgency, low data accumulation rate, and / or small task accumulation volume, while ensuring the execution of data processing tasks with high urgency, high data accumulation rate, and / or large task accumulation volume. This ensures the data processing efficiency of high-priority data processing tasks without overloading computing resources.
[0091] Of course, in addition to reducing the batch size of selected data processing tasks, you can also reduce the batch size of all data processing tasks directly to quickly reduce the load on computing resources.
[0092] In some implementations, newly added data processing tasks are first run at the minimum batch size (e.g., the number of data extracted in each batch is 100), and then the batch size of the data processing tasks is readjusted after the adjustment trigger conditions are met.
[0093] To illustrate with a specific application scenario, in this scenario, the batch size is calculated every 5 minutes, and the allocable computing resources are memory resources. Please refer to Figure 4, which is a flowchart of a data batch processing method shown in another exemplary embodiment of this application. As shown in Figure 4, the method includes: Step S401: Obtain each data processing task; Step S402: Each data processing task runs for 5 minutes; Step S403: Calculate the task backlog and total backlog of each data processing task during the running period; Step S404: Allocate the total allocable memory resources to each data processing task according to the task backlog and total backlog to obtain the available memory size; Step S405: Set the batch size corresponding to each data processing task according to the available memory size corresponding to each data processing task.
[0094] The above embodiments can increase the reliability of the data processing system, avoid program crashes caused by exhaustion of computing resources, and are simple to implement, only requiring dynamic calculation of batch size, without intruding on the data processing logic, and avoiding the introduction of unnecessary exceptions.
[0095] Figure 5 is a block diagram of a data batch processing apparatus according to an exemplary embodiment of this application. As shown in Figure 5, the exemplary data batch processing apparatus 500 includes: a backlog calculation module 510, used to calculate the amount of data backlogged in the message queue of each data processing task, to obtain the task backlog corresponding to each data processing task; a resource allocation module 520, used to allocate the total allocable computing resources to each data processing task based on the task backlog corresponding to each data processing task, to obtain the computing resource amount corresponding to each data processing task; wherein, the task backlog and the computing resource amount are positively correlated; a batch size setting module 530, used to set the number of data to be extracted by each processing thread of each data processing task in each batch based on the computing resource amount corresponding to each data processing task, to obtain the batch size; and a data processing module 540, used to control each processing thread to extract data from the message queue for processing based on the batch size corresponding to each processing thread.
[0096] It should be noted that the data batch processing device and the data batch processing method provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the data batch processing device provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.
[0097] Please refer to Figure 6, which is a schematic diagram of the structure of an electronic device illustrated in an exemplary embodiment of this application. The electronic device 600 includes a memory 601 and a processor 602. The processor 602 executes program instructions stored in the memory 601 to implement the steps in any of the above-described data batch processing method embodiments. In a specific implementation scenario, the electronic device 600 may include, but is not limited to, a microcomputer or a server. Furthermore, the electronic device 600 may also include mobile devices such as laptops and tablets, without limitation.
[0098] Specifically, processor 602 controls itself and memory 601 to implement the steps in any of the above-described data batch processing method embodiments. Processor 602 can also be referred to as a Central Processing Unit (CPU). Processor 602 may be an integrated circuit chip with signal processing capabilities. Processor 602 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 602 can be implemented using integrated circuit chips.
[0099] Please refer to Figure 7, which is a schematic diagram illustrating the structure of a computer-readable storage medium according to an exemplary embodiment of this application. The computer-readable storage medium 700 stores program instructions 710 that can be executed by a processor. The program instructions 710 are used to implement the steps in any of the above-described data batch processing method embodiments.
[0100] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0101] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.
[0102] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.
[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A method for batch data processing, characterized in that, Multiple data processing tasks are created, each consisting of a message queue and a processing thread. The message queue is used to cache data, and the processing thread is used to extract data from the message queue for processing. The method includes: calculating the amount of data accumulated in the message queue of each data processing task to obtain the task backlog for each data processing task; allocating the total available computing resources to each data processing task based on the task backlog for each data processing task to obtain the computing resource amount for each data processing task; wherein the task backlog and the computing resource amount are positively correlated; setting the number of data items extracted by the processing thread of each data processing task in each batch based on the computing resource amount for each data processing task to obtain the batch size; and controlling each processing thread to extract data from the message queue for processing based on the batch size for each processing thread.
2. The method according to claim 1, characterized in that, The step of calculating the amount of data accumulated in the message queue of each data processing task to obtain the task backlog amount corresponding to each data processing task includes: calculating the average size of the data in the message queue to obtain the average data size; counting the total number of data contained in the message queue to obtain the total backlog data; and multiplying the average data size and the total backlog data to obtain the task backlog amount.
3. The method according to claim 1, characterized in that, The computing resources include memory resources; the step of allocating the total allocable computing resources to each data processing task based on the task backlog corresponding to each data processing task, to obtain the computing resource amount corresponding to each data processing task, includes: obtaining the total allocable memory resources; allocating the total allocable memory resources to each data processing task based on the task backlog corresponding to each data processing task, to obtain the available memory size corresponding to each data processing task.
4. The method according to claim 3, characterized in that, The step of allocating the total available memory resources to each data processing task based on the task backlog corresponding to each data processing task, to obtain the available memory size corresponding to each data processing task, includes: calculating the sum of the task backlog of each data processing task to obtain the total backlog; calculating the ratio between the task backlog of each data processing task and the total backlog to obtain the backlog percentage corresponding to each data processing task; and multiplying the backlog percentage corresponding to each data processing task by the total available memory resources to obtain the available memory size corresponding to each data processing task.
5. The method according to claim 3, characterized in that, The step of setting the batch size based on the computing resources corresponding to each data processing task, and setting the number of data to be extracted by the processing thread of each data processing task in each batch, includes: calculating the average size of the data in the message queue to obtain the average data size; and calculating the number of data copies during the statistical data processing process; and calculating the number of data to be extracted by the processing thread of each data processing task in each batch based on the available memory size, average data size, and number of data copies of the data processing task to obtain the batch size.
6. The method according to claim 1, characterized in that, The method further includes: detecting whether the adjustment triggering condition is met; if so, readjusting the batch size corresponding to each data processing task; wherein the adjustment triggering condition includes one or more of the following conditions: Condition 1: the time interval since the last adjustment reaches a preset time interval threshold; Condition 2: counting the number of data processing tasks with a task backlog greater than a preset backlog threshold to obtain the number of overloaded tasks, wherein the number of overloaded tasks is greater than a preset task number threshold; Condition 3: calculating the change in the total available computing resources relative to the total available computing resources in the last adjustment to obtain the total resource change, wherein the total resource change is greater than a preset change threshold; Condition 4: the existence of a new data processing task.
7. The method according to claim 1, characterized in that, The method further includes: detecting the amount of computing resources used; if the amount of computing resources used is greater than a preset maximum usage threshold, obtaining the task priority corresponding to each data processing task, selecting data processing tasks whose task priorities meet preset conditions as degraded tasks, and reducing the batch size corresponding to the degraded tasks.
8. The method according to claim 7, characterized in that, The step of obtaining the task priority corresponding to each data processing task includes: detecting the task urgency, and / or data accumulation rate, and / or task accumulation amount of the data processing task; setting the task priority corresponding to the data processing task based on the task urgency, and / or data accumulation rate, and / or task accumulation amount of the data processing task; wherein the task urgency, the data accumulation rate, and the task accumulation amount are all positively correlated with the task priority.
9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the steps of the method as described in any one of claims 1-8.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that can be executed by a processor to implement the steps of the method as described in any one of claims 1-8.