Data processing method and device

Abstract

The invention discloses a data processing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: for acquired to-be-processed data of a plurality of channels, judging a size relationship between a sum of the to-be-processed data volume and a thread pool capacity threshold value, if the sum of the to-be-processed data volume is greater than the thread pool capacity threshold value, calculating a weight value of each channel, determining an initial data processing amount of each channel, and processing the data in combination with the thread pool capacity threshold value. According to the embodiment, the efficiency of processing multiple channel data by the thread pool can be improved, the threads of all channels are reasonably distributed, and the data processing speed is increased.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for data processing. Background Art

[0002] In the field of Internet system development, in order to improve the efficiency of data processing, the thread pool technology is introduced to process the data of multiple tasks through concurrent thread tasks. Concurrent threads can greatly improve the efficiency of data processing. However, the data processing tasks of the existing thread pool are executed by scanning the database. When a channel has no data to process, it is necessary to scan the entire data table, which is inefficient; when a channel has a large number of data, it cannot be processed within a certain period of time, and the data execution of other channels will be blocked and delayed. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a method and device for data processing, which can improve the processing efficiency of a thread pool for multi-channel data and reasonably allocate thread pool resources.

[0004] To achieve the above object, according to one aspect of an embodiment of the present invention, a data processing method is provided, comprising:

[0005] Obtain the data to be processed from multiple channels, and determine the relationship between the total amount of the data to be processed from multiple channels and the thread pool capacity threshold;

[0006] In response to the amount of data to be processed being greater than a thread pool capacity threshold, calculating a weight value of each channel based on the amount of data to be processed of the multiple channels;

[0007] Based on the weight value of each channel and the total amount of data to be processed, the initial data processing amount of each channel is calculated;

[0008] Based on the initial data processing volume of each channel and the thread pool capacity threshold, the pending data of multiple channels are processed.

[0009] Optionally, the method further comprises:

[0010] In response to the data volume of the to-be-processed data being less than or equal to the thread pool capacity threshold, the to-be-processed data is concurrently processed by the thread pool.

[0011] Optionally, the weight value of each channel is calculated based on the data volume of the to-be-processed data of the multiple channels, including:

[0012] For each channel, the ratio of the amount of data to be processed in the channel to the total amount of data to be processed in multiple channels is calculated, and the calculated result is used as the weight value of the channel.

[0013] Optionally, based on the initial data processing volume of each channel and the thread pool capacity threshold, the to-be-processed data of multiple channels are processed, including:

[0014] Calculate the sum of initial data processing volumes of multiple channels based on the initial data processing volume of each channel;

[0015] Determine the relationship between the total amount of initial data processing and the thread pool capacity threshold to obtain a first determination result;

[0016] Based on the data processing strategy corresponding to the first judgment result, the to-be-processed data of multiple channels are processed concurrently through the thread pool.

[0017] Optionally, based on a data processing strategy corresponding to the judgment result, concurrent processing of the to-be-processed data of multiple channels is performed through a thread pool, including:

[0018] In response to the sum of the initial data processing volumes being less than the thread pool capacity threshold, determining the relationship between the sum of the time consumed to calculate the initial data processing volumes of the multiple channels and a preset time threshold, and the relationship between the initial data processing volumes of the multiple channels and the data volumes of the to-be-processed data of the multiple channels, to obtain a second determination result;

[0019] In response to the second judgment result that the sum of the time consumed for calculating the initial data processing volume of the multiple channels is less than the preset time threshold, and the initial data processing volume of the multiple channels is less than the data volume of the to-be-processed data of the multiple channels, based on the initial data processing volume of the multiple channels, the thread pool capacity threshold and the first preset threshold condition, calculating the increment of the initial data processing volume of at least one channel;

[0020] Initial data processing volume based on incremental updates of multiple channels;

[0021] Based on the updated initial data processing volume of multiple channels, the pending data of multiple channels are processed concurrently through the thread pool.

[0022] Optionally, based on the initial data processing volumes of the multiple channels, the thread pool capacity threshold and the first preset threshold condition, calculating the increment of the initial data processing volume of at least one channel includes:

[0023] In response to the initial data processing volume of the multiple channels and the thread pool capacity threshold satisfying the first preset threshold condition, the initial data volume of each channel is increased one by one according to the descending order of the weight value of each channel and the first preset data volume, until the initial data volume of the multiple channels after the increase is satisfied to be equal to the thread pool capacity threshold, thereby obtaining an increment of the initial data processing volume of at least one channel;

[0024] In response to the initial data processing volume of multiple channels and the thread pool capacity threshold not satisfying the first preset threshold condition, the increment of each channel is calculated based on the weight value of each channel and the first difference value, and the first difference value is the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels.

[0025] Optionally, the method further comprises:

[0026] In response to the sum of the initial data processing volumes being greater than the thread pool capacity threshold, calculating a deletion amount of the initial data processing volume of at least one channel based on the initial data processing volumes of the plurality of channels, the thread pool capacity threshold, and a second preset threshold condition;

[0027] Update initial data processing volume for multiple channels based on deletion volume;

[0028] Based on the updated initial data processing volume of multiple channels, the pending data of multiple channels are processed concurrently through the thread pool.

[0029] Optionally, based on the initial data processing volumes of the multiple channels, the thread pool capacity threshold and the second preset threshold condition, calculating the deletion amount of the initial data processing volume of at least one channel includes:

[0030] In response to the initial data processing volume of the multiple channels and the thread pool capacity threshold satisfying the second preset threshold condition, the initial data volume of each channel is deleted one by one according to the ascending order of the weight value of each channel and the second preset data volume, until the initial data volume of the multiple channels after deletion is equal to the second difference, and the deletion amount of the initial data processing volume of at least one channel is obtained, and the second difference is the difference between the initial data volume of the multiple channels and the thread pool capacity threshold;

[0031] In response to the initial data processing volume of the plurality of channels and the thread pool capacity threshold not satisfying the second preset threshold condition, the deletion volume of each channel is calculated based on the weight value of each channel and the second difference value.

[0032] Optionally, the method further comprises:

[0033] In response to the total amount of initial data processing being equal to a thread pool capacity threshold, the to-be-processed data of multiple channels are processed concurrently through the thread pool.

[0034] According to another aspect of an embodiment of the present invention, there is provided a data processing device, including:

[0035] An acquisition module is used to acquire data to be processed from multiple channels and determine the relationship between the total amount of the multiple data to be processed and the thread pool capacity threshold;

[0036] A calculation module, configured to calculate a weight value of each channel based on the data volume of the data to be processed of the multiple channels in response to the data volume of the data to be processed being greater than the thread pool capacity threshold; and calculate an initial data processing volume of each channel based on the weight value of each channel and the sum of the data volumes of the multiple data to be processed;

[0037] The processing module is used to process the to-be-processed data of multiple channels based on the initial data processing volume of each channel and the thread pool capacity threshold.

[0038] According to another aspect of an embodiment of the present invention, there is provided an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method provided by an embodiment of the present invention.

[0039] According to another aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored. When the program is executed by a processor, the data processing method provided by an embodiment of the present invention is implemented.

[0040] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the data processing method provided by an embodiment of the present invention is implemented.

[0041] One embodiment of the above invention has the following advantages or beneficial effects: it is able to calculate the weight value of each channel based on the acquired data to be processed of different channels and the thread pool capacity threshold, and further calculate the initial data processing volume of each channel. Based on the initial data processing volume, the data to be processed of each channel is processed by the thread pool, thereby improving the rationality of resource allocation of thread pools of different channels and improving the efficiency of data processing by the thread pool. In particular, when determining the initial data processing volume of each channel, it can be performed in different scenarios, including the calculation of the increment and deletion of the initial data processing volume, which can further improve the accuracy of resource allocation of thread pools of different channels and improve data processing efficiency.

[0042] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0044] Figure 1 is a schematic diagram of main steps of a method for data processing according to an embodiment of the present invention;

[0045] Figure 2 is a flow chart of the data processing method;

[0046] Figure 3 Processing logic diagram for data in less than scenario;

[0047] Figure 4 Processing logic diagram for data in larger than scenario;

[0048] Figure 5 is a schematic diagram of main modules of a data processing device according to an embodiment of the present invention;

[0049] Figure 6 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0050] Figure 7 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0052] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution disclosed in the present invention are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0053] It should be noted that in the technical solution of the present invention, the collection, use, storage, sharing and transfer of user personal information involved are in compliance with the provisions of relevant laws and regulations, and it is necessary to inform the user and obtain the user's consent or authorization. When applicable, the user's personal information is de-identified and / or anonymized and / or encrypted.

[0054] Figure 1 is a schematic diagram of the main flow of the data processing method according to an embodiment of the present invention, such as Figure 1 As shown, including:

[0055] Step S101, obtaining data to be processed from multiple channels, and determining the relationship between the total amount of data to be processed from multiple channels and a thread pool capacity threshold.

[0056] The pending data of multiple channels are obtained from the redis (remote dictionary service) library, and the amount of data to be processed of each channel is calculated in total, and the total is compared with the capacity threshold of the thread pool that can process data to obtain a comparison result.

[0057] Optionally, the data to be processed is first written into a preset database, such as a clue buffer pool, and stored in the form of a clue buffer pool table. The data to be processed is obtained by monitoring the binlog (a binary log) log of the data table and stored in the redis library. Among them, in the redis library, the storage format of the data to be processed is: the key is a fixed prefix + channel id, and the value is the id (identity card identification number) list of the data to be processed in the channel. The fixed prefix can be set according to the actual situation, and there is no restriction on this. For example, it can be time, system identification, etc., or it can be a random or preset string, etc.

[0058] Step S102 , in response to the data volume of the to-be-processed data being greater than the thread pool capacity threshold, a weight value of each channel is calculated based on the data volume of the to-be-processed data of the multiple channels.

[0059] If the comparison result is that the amount of data to be processed is greater than the thread pool capacity threshold, it is necessary to calculate the weight value of each channel to allocate the number of threads for processing data in each channel. Specifically, the weight value of each channel is calculated according to the amount of data to be processed in each channel, for example, according to the data amount range of the data amount of the data to be processed in each channel, the preset weight value corresponding to the data amount range is determined as the weight value of the channel.

[0060] Optionally, when calculating the weight values ​​of different channels, it is also necessary to combine the preset weights and historical weights for calculation, and the weight values ​​of different channels are obtained by adding the three weights and then normalizing them.

[0061] Step S103, calculating the initial data processing volume of each channel based on the weight value of each channel and the total data volume of multiple data to be processed.

[0062] According to the calculated weight value of each channel, the value of the processable data amount allocated to each channel is determined based on the sum of the amount of data to be processed. Specifically, the initial data processing amount of each channel, that is, the preliminarily determined processable data amount of each channel, can be obtained by calculating the product of the weight value of each channel and the sum of the amount of data to be processed.

[0063] Step S104, based on the initial data processing volume of each channel and the thread pool capacity threshold, the to-be-processed data of multiple channels are processed.

[0064] According to the calculated initial data processing volume of each channel and based on the thread pool capacity threshold, the initial data volume of each channel is put into the thread pool for data processing. The sum of the data of each channel put into the thread pool does not exceed the thread pool capacity threshold, for example, the amount of data put into each channel is determined according to the weight value of each channel.

[0065] According to the data processing method provided in an embodiment of the present invention, the weight value of each channel can be calculated based on the acquired data to be processed of different channels and the thread pool capacity threshold, and the initial data processing volume of each channel can be further calculated. Based on the initial data processing volume, the data to be processed of each channel is processed by the thread pool, thereby improving the rationality of resource allocation of thread pools of different channels and improving the efficiency of thread pools in processing data.

[0066] Optionally, the method further includes: in response to the data volume of the to-be-processed data being less than or equal to the thread pool capacity threshold, concurrently processing the to-be-processed data through the thread pool. When processing the to-be-processed data of each channel, it is necessary to select different processing methods, including allocation of executable data volume, based on the size relationship between the total data volume of the to-be-processed data of all channels and the thread pool capacity threshold. When judging the size relationship between the data volume of the to-be-processed data and the thread pool capacity threshold, if the relationship is less than, it means that the thread pool can accommodate all the to-be-processed data and perform corresponding processing.

[0067] Optionally, the weight value of each channel is calculated based on the amount of data to be processed of multiple channels, including: for each channel, calculating the ratio of the amount of data to be processed of the channel to the total amount of data to be processed of multiple channels, and using the calculated result as the weight value of the channel. When calculating the weight value of each channel, it can be determined based on the ratio of the amount of data to be processed of each channel to the total amount of data to be processed. For example, if the amount of data to be processed of channel 1 is 400 and the total amount of data to be processed is 1000, then the weight value of channel 1 is 0.4.

[0068] Optionally, based on the initial data processing volume of each channel and the thread pool capacity threshold, the data to be processed of multiple channels are processed, including: calculating the sum of the initial data processing volumes of multiple channels based on the initial data processing volume of each channel; judging the size relationship between the sum of the initial data processing volumes and the thread pool capacity threshold to obtain a first judgment result; and based on the data processing strategy corresponding to the first judgment result, concurrently processing the data to be processed of multiple channels through the thread pool. After calculating the initial data processing volume of each channel, it is necessary to further judge the size relationship between the sum of the initial data processing volume and the thread pool capacity threshold, that is, to judge whether the thread pool can accommodate all the data of the initial data processing volume in the current state to obtain the first judgment result. For different size relationships, corresponding different data processing strategies are adopted to realize the concurrent processing of the data to be processed of different channels. Among them, there are preset data processing strategies corresponding to different size relationships, for example, when the sum of the initial data processing volume is less than the thread pool capacity threshold, it corresponds to the first data processing strategy; when the sum of the initial data processing volume is equal to the thread pool capacity threshold, it corresponds to the second data processing strategy; when the initial data processing volume is greater than the thread pool capacity threshold, it corresponds to the third data processing strategy.

[0069] Optionally, based on a data processing strategy corresponding to the judgment result, concurrent processing of the data to be processed of multiple channels is performed through a thread pool, including: in response to the sum of the initial data processing volumes being less than a thread pool capacity threshold, determining the relationship between the sum of the time consumed to calculate the initial data processing volumes of multiple channels and a preset time threshold, and the relationship between the initial data processing volumes of multiple channels and the data volume of the data to be processed of the multiple channels, to obtain a second judgment result; in response to the second judgment result being that the sum of the time consumed to calculate the initial data processing volumes of multiple channels is less than a preset time threshold, and the initial data processing volumes of multiple channels are less than the data volume of the data to be processed of multiple channels, calculating an increment of the initial data processing volume of at least one channel based on the initial data processing volumes of multiple channels, the thread pool capacity threshold and a first preset threshold condition; updating the initial data processing volumes of multiple channels based on the increment; and based on the updated initial data processing volumes of multiple channels, concurrently processing the data to be processed of multiple channels through a thread pool.

[0070] When the sum of the initial data processing volumes is less than the thread pool capacity threshold, the corresponding data processing strategy is adopted. Specifically, firstly, the relationship between the sum of the consumption time required to process the initial data processing volumes of multiple channels and the preset time threshold is determined, wherein the consumption time is the execution time of the data; the initial data is the executable data of the channel. And further determine the relationship between the initial data processing volume of each channel and the data volume of the data to be processed of each channel, and obtain the second judgment result of the two judgment operations. Among them, for any channel, the initial data processing volume can be calculated based on the product of the weight value and the sum of the data volume to be processed, so the value of the initial data volume of the channel must not be greater than the value of the data volume of the data to be processed of the channel. When it is determined that the sum of the consumption time is less than the preset time threshold, and the initial data processing volume is less than the data volume of the data to be processed of each channel, subsequent calculation operations will be performed; otherwise, the thread pool directly processes the data to be processed in parallel through multiple threads. The calculation operation includes: sorting according to the size of the weight value calculated by each channel, and calculating the initial data processing volume that can be increased for each channel in order from large to small. It is determined whether the sum of the thread pool capacity threshold and the initial data processing volume of each channel meets the first preset threshold condition, that is, is less than the first preset threshold, and the increment of the initial data processing volume of each channel is calculated according to the determination result. The obtained increment is added to the initial data processing volume of the corresponding channel for updating, and the updated initial data processing volume of each channel is obtained. Based on the updated data, the data of each channel is concurrently processed through the thread pool.

[0071] Optionally, based on the initial data processing volume of multiple channels, the thread pool capacity threshold and a first preset threshold condition, the increment of the initial data processing volume of at least one channel is calculated, including: in response to the initial data processing volume of multiple channels and the thread pool capacity threshold satisfying the first preset threshold condition, the initial data volume of each channel is increased one by one according to the descending order of the weight value of each channel and the first preset data volume, until the initial data volume of the multiple channels after the increase is equal to the thread pool capacity threshold, so as to obtain the increment of the initial data processing volume of at least one channel; in response to the initial data processing volume of multiple channels and the thread pool capacity threshold not satisfying the first preset threshold condition, the increment of each channel is calculated based on the weight value of each channel and a first difference, the first difference being the difference between the thread pool capacity threshold and the initial data processing volume of the multiple channels.

[0072] When calculating the increment of the initial data processing volume of a channel, it is necessary to make a judgment based on the first preset threshold condition. Specifically, it is necessary to judge whether the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels is less than the first preset threshold. If it is less than the first preset threshold, the initial data processing volume of each channel can be increased by traversal. When the initial data volume of the channel is less than the first preset threshold and increases one by one, it is necessary to first judge whether the first preset number is greater than the above-mentioned difference (the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels). If it is less than the difference, it means that the increment to be increased is small. Then, in order from large to small weights, loop through each channel and add 1 to the initial data processing volume of each channel in turn until the sum of the increments of the initial data processing volume of each channel is equal to the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels. Among them, the first preset number is the total number of channels. Update according to the increment of the initial data processing volume increased by each channel. If it is not less than the above difference, it means that the increments that need to be increased are large. Then, in descending order of weight, loop through the channels and add 1 to the initial data processing volume of each channel until the sum of the increments of the initial data processing volume of each channel is equal to the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels. If the remaining increment of the initial data processing volume that can be increased does not satisfy the increment of 1 for all channels, then add 1 to each channel one by one in descending order of weight. Update the initial data processing volume of each channel according to the final increment.

[0073] If it does not meet the requirement of being less than the first preset threshold, the increment of each channel is calculated according to the weight value of each channel. Specifically, the product of the weight of each channel and the first difference is used as the increment of each channel, and the initial data processing volume of each channel is updated based on the calculated increment. The first difference is the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels.

[0074] Optionally, the method also includes: in response to the total initial data processing volume being greater than a thread pool capacity threshold, calculating a deletion amount of the initial data processing volume of at least one channel based on the initial data processing volumes of multiple channels, the thread pool capacity threshold and a second preset threshold condition; updating the initial data processing volumes of multiple channels based on the deletion amount; and concurrently processing the to-be-processed data of multiple channels through the thread pool based on the updated initial data processing volumes of multiple channels.

[0075] When the sum of the initial data processing volume is greater than the thread pool capacity threshold, it is necessary to reduce the initial data processing volume of each channel. Specifically, the channels are sorted in order from small to large according to the weight value, so as to calculate according to the sorting result, and then it is judged whether the sum of the initial data processing volume of each channel meets the second preset threshold condition, that is, it is judged whether the difference between the sum of the initial data processing volume of each channel and the thread pool capacity threshold is less than the second preset threshold, and the corresponding calculation method is adopted according to the judgment result. If it is less than, it means that the deletion amount of the initial data processing volume to be deleted is small, and the initial data processing volume of each channel is reduced by 1 by traversing the channels; if it is not less than, it means that the deletion amount is large, and the deletion amount of each channel is calculated according to the weight value of each channel and the difference between the sum of the initial data processing volume of each channel and the thread pool capacity threshold. Finally, the initial data processing volume of each channel is updated according to the obtained deletion amount of each channel, and concurrent processing is performed through the thread pool according to the updated initial data processing volume of each channel.

[0076] Optionally, based on the initial data processing volume of multiple channels, the thread pool capacity threshold and the second preset threshold condition, the deletion amount of the initial data processing volume of at least one channel is calculated, including: in response to the initial data processing volume and the thread pool capacity threshold of multiple channels satisfying the second preset threshold condition, deleting the initial data volume of each channel one by one in ascending order of the weight value of each channel and the second preset data volume until the initial data volume of the multiple channels after deletion is equal to the second difference, thereby obtaining the deletion amount of the initial data processing volume of at least one channel, and the second difference is the difference between the initial data volume of the multiple channels and the thread pool capacity threshold; in response to the initial data processing volume and the thread pool capacity threshold of multiple channels not satisfying the second preset threshold condition, calculating the deletion amount of each channel based on the weight value of each channel and the second difference.

[0077] When calculating the deletion amount, if the second preset threshold condition is met, it is first necessary to determine whether the second preset data amount is less than the difference between the initial data amount of multiple channels and the thread pool capacity threshold. Among them, the second preset data amount is the total number of channels. If the judgment result is less than, then loop through multiple channels, and each time reduce the initial data processing amount of all channels by 1 as the deletion amount, until the deletion amount of the initial data processing amount of multiple channels is equal to the difference between the initial data amount of multiple channels and the thread pool capacity threshold. If the judgment result is not less than, then traverse each channel one by one in the order of weight value from small to large and reduce the initial data processing amount by 1, until the deletion amount of the initial data processing amount of multiple channels is equal to the difference between the initial data amount of multiple channels and the thread pool capacity threshold.

[0078] Optionally, if the judgment result is less than, the remaining deletion amount is insufficient to satisfy the initial data processing amount of all channels minus 1, then the initial data processing amount of each channel is reduced by 1 in order of weight value from small to large.

[0079] Optionally, the method further includes: in response to the sum of the initial data processing volume being equal to the thread pool capacity threshold, concurrently processing the to-be-processed data of multiple channels through the thread pool. If it is determined that the sum of the initial data processing volume is equal to the thread pool capacity threshold, it means that the thread pool can process data with better performance, and the to-be-processed data of multiple channels is directly processed concurrently through the thread pool.

[0080] According to the data processing method provided by the embodiment of the present invention, the weight value of each channel can be calculated based on the acquired data to be processed of different channels and the thread pool capacity threshold, and the initial data processing volume of each channel can be further calculated. Based on the initial data processing volume, the data to be processed of each channel is processed by the thread pool, so as to improve the rationality of resource allocation of thread pools of different channels and improve the efficiency of data processing by the thread pool. Among them, when determining the initial data processing volume of each channel, it can be carried out in different scenarios, including the calculation of the increment and deletion of the initial data processing volume, which can further improve the accuracy of resource allocation of thread pools of different channels and improve data processing efficiency.

[0081] like Figure 2 The flowchart of the data processing method is shown. The amount of data to be processed of the channel at the current moment is obtained, and the channels with the amount of data to be processed of 0 are not considered. Determine whether the total amount of data to be processed of all channels is not greater than the thread pool queue threshold. If so, all the data of all channels are executed through the thread pool, and the thread pool is further enabled to execute the data to be processed in parallel. If not, the channel is used as the dimension for calculation, including calculating the weight of each channel and the amount of executable data. Then determine the relationship between the total amount of executable data of all channels and the thread pool queue threshold, and execute the corresponding processing logic according to the judgment result, that is, if it is less than the scene, execute the processing logic less than the scene, if it is greater than the scene, execute the processing logic greater than the scene, and if it is equal to the scene, directly enable the thread pool to execute the data to be processed in parallel.

[0082] like Figure 3The figure shows a logical diagram of data processing in the scenario of less than. In this scenario, it is determined whether the execution time is greater than the preset time threshold. If not, it is determined whether the initial data processing volume of multiple channels is equal to the amount of data to be processed by the channels. If less than, the channels with larger weights are sorted in descending order according to the weights, and the number of executable channels that can be added is prioritized, that is, the increment of the initial data processing volume. It is determined whether the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels is less than 10, where the initial data processing volume of multiple channels is the sum of the initial data processing volume of each channel, and 10 is a preset threshold that can be set according to actual conditions and is not restricted. If less than 10, it is determined whether the total number of executable channels is not less than the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels. If so, the number of executable channels is increased by 1 one by one according to the descending order of the weight value until the total number of channels added by 1 reaches the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels, and the total number of executable channels of all channels is recalculated. If not, loop through the channels, add 1 to the number of executables for all channels until the total number of 1s reaches the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels, and recalculate the total number of executables for all channels. If it is not less than 10, parallel calculation is performed based on the channel dimension, including the specific calculation of the number of executables for each channel: weight * (thread pool capacity threshold - initial data processing volume of multiple channels), recalculate the total number of executables for all channels, and perform parallel processing through the thread pool. After recalculation, the process of determining whether the total number of executables for all channels is equal to the thread pool capacity threshold can be performed again.

[0083] like Figure 4The figure shows the data processing logic diagram under the scenario of greater than. In this scenario, the executable quantity that can be deleted from the channels with smaller weights is prioritized according to the ascending order of weight. It is determined whether the difference between the initial data processing volume of multiple channels and the thread pool capacity threshold is less than 10, where the initial data processing volume of multiple channels is the sum of the initial data processing volume of each channel, and 10 is a preset threshold value, which can be set according to actual conditions and is not restricted. If it is less than 10, it is determined whether the total number of executable channels is not less than the difference between the initial data processing volume of multiple channels and the thread pool capacity threshold. If it is not less than, the executable quantity of each channel is reduced by 1 according to the descending order of weight value until the total number of reduced 1 reaches the difference between the initial data processing volume of multiple channels and the thread pool capacity threshold, and the total number of executables of all channels is recalculated. If it is less than, the channels are traversed in a loop, and the executable quantity of all channels is reduced by 1 until the total number of reduced 1 reaches the difference between the initial data processing volume of multiple channels and the thread pool capacity threshold, and the total number of executables of all channels is recalculated. If it is not less than 10, parallel calculation is performed based on the channel dimension, including the specific calculation of the number of executables for each channel: weight * (initial data processing volume of multiple channels - thread pool capacity threshold), recalculation of the total number of executables for all channels, and parallel processing through the thread pool. After recalculation, the process of determining whether the total number of executables for all channels is equal to the thread pool capacity threshold can be performed again. Among them, after calculating the number of executables for each channel, it is also necessary to determine whether the number of channels that can be deleted is equal to the number of executables for the channel. If it is equal, the number of executables for the channel is set to 1 to ensure that the channel has data to execute and is not empty; if it is less than, the number of executables for the channel is equal to the difference between the original number of executables and the number of deletables.

[0084] Figure 5 is a schematic diagram of a data processing device 500 provided in an embodiment of the present invention, such as Figure 5 As shown, including:

[0085] The acquisition module 501 is used to acquire the to-be-processed data of multiple channels, and determine the relationship between the sum of the data volumes of the multiple to-be-processed data and the thread pool capacity threshold;

[0086] The calculation module 502 is used to calculate the weight value of each channel based on the data volume of the data to be processed of the multiple channels in response to the data volume of the data to be processed being greater than the thread pool capacity threshold; calculate the initial data processing volume of each channel based on the weight value of each channel and the sum of the data volume of the multiple data to be processed;

[0087] The processing module 503 is used to process the to-be-processed data of multiple channels based on the initial data processing volume of each channel and the thread pool capacity threshold.

[0088] The data processing device provided in an embodiment of the present invention can calculate the weight value of each channel according to the acquired to-be-processed data of different channels and the thread pool capacity threshold, and further calculate the initial data processing volume of each channel. Based on the initial data processing volume, the to-be-processed data of each channel is processed by the thread pool, thereby improving the rationality of resource allocation of the thread pools of different channels and improving the efficiency of data processing by the thread pool.

[0089] Optionally, the device further includes: a processing module 503, further configured to concurrently process the data to be processed through the thread pool in response to the data volume of the data to be processed being less than or equal to a thread pool capacity threshold.

[0090] Optionally, the calculation module 502 is further configured to:

[0091] For each channel, the ratio of the amount of data to be processed in the channel to the total amount of data to be processed in multiple channels is calculated, and the calculated result is used as the weight value of the channel.

[0092] Optionally, the processing module 503 is further used to: calculate the sum of the initial data processing amounts of the multiple channels based on the initial data processing amount of each channel;

[0093] Determine the relationship between the total amount of initial data processing and the thread pool capacity threshold to obtain a first determination result;

[0094] Based on the data processing strategy corresponding to the first judgment result, the to-be-processed data of multiple channels are processed concurrently through the thread pool.

[0095] Optionally, the processing module 503 is further configured to:

[0096] In response to the sum of the initial data processing volumes being less than the thread pool capacity threshold, determining the relationship between the sum of the time consumed to calculate the initial data processing volumes of the multiple channels and a preset time threshold, and the relationship between the initial data processing volumes of the multiple channels and the data volumes of the to-be-processed data of the multiple channels, to obtain a second determination result;

[0097] In response to the second judgment result that the sum of the time consumed for calculating the initial data processing volume of the multiple channels is less than the preset time threshold, and the initial data processing volume of the multiple channels is less than the data volume of the to-be-processed data of the multiple channels, based on the initial data processing volume of the multiple channels, the thread pool capacity threshold and the first preset threshold condition, calculating the increment of the initial data processing volume of at least one channel;

[0098] Initial data processing volume based on incremental updates of multiple channels;

[0099] Based on the updated initial data processing volume of multiple channels, the pending data of multiple channels are processed concurrently through the thread pool.

[0100] Optionally, the calculation module 502 is further configured to:

[0101] In response to the initial data processing volume of the multiple channels and the thread pool capacity threshold satisfying the first preset threshold condition, the initial data volume of each channel is increased one by one according to the descending order of the weight value of each channel and the first preset data volume, until the initial data volume of the multiple channels after the increase is satisfied to be equal to the thread pool capacity threshold, thereby obtaining an increment of the initial data processing volume of at least one channel;

[0102] In response to the initial data processing volume of multiple channels and the thread pool capacity threshold not satisfying the first preset threshold condition, the increment of each channel is calculated based on the weight value of each channel and the first difference value, and the first difference value is the difference between the thread pool capacity threshold and the initial data processing volume of multiple channels.

[0103] Optionally, the apparatus further comprises: a calculation module 502, further configured to calculate a deletion amount of the initial data processing amount of at least one channel based on the initial data processing amounts of the plurality of channels, the thread pool capacity threshold and a second preset threshold condition in response to the sum of the initial data processing amounts being greater than the thread pool capacity threshold;

[0104] An updating module 504, configured to update the initial data processing volume of the plurality of channels based on the deletion volume;

[0105] The processing module 503 is further used to concurrently process the to-be-processed data of the multiple channels through the thread pool based on the updated initial data processing volume of the multiple channels.

[0106] Optionally, the calculation module 502 is further configured to:

[0107] In response to the initial data processing volume of the multiple channels and the thread pool capacity threshold satisfying the second preset threshold condition, the initial data volume of each channel is deleted one by one according to the ascending order of the weight value of each channel and the second preset data volume, until the initial data volume of the multiple channels after deletion is equal to the second difference, and the deletion amount of the initial data processing volume of at least one channel is obtained, and the second difference is the difference between the initial data volume of the multiple channels and the thread pool capacity threshold;

[0108] In response to the initial data processing volume of the plurality of channels and the thread pool capacity threshold not satisfying the second preset threshold condition, the deletion volume of each channel is calculated based on the weight value of each channel and the second difference value.

[0109] Optionally, the device further includes: a processing module 503, further configured to concurrently process to-be-processed data of multiple channels through the thread pool in response to the total amount of initial data processing being equal to a thread pool capacity threshold.

[0110] According to the data processing device provided by the embodiment of the present invention, it is possible to calculate the weight value of each channel based on the acquired data to be processed of different channels and the thread pool capacity threshold, and further calculate the initial data processing volume of each channel. Based on the initial data processing volume, the data to be processed of each channel is processed by the thread pool, thereby improving the rationality of resource allocation of thread pools of different channels and improving the efficiency of data processing by the thread pool. In particular, when determining the initial data processing volume of each channel, it can be performed in different scenarios, including the calculation of the increment and deletion of the initial data processing volume, which can further improve the accuracy of resource allocation of thread pools of different channels and improve data processing efficiency.

[0111] Figure 6 An exemplary system architecture 600 is shown to which the data processing method or data processing apparatus according to the embodiment of the present invention can be applied.

[0112] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, 603, network 604 and server 605. Network 604 is used to provide a medium for communication links between terminal devices 601, 602, 603 and server 605. Network 604 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0113] Users can use terminal devices 601, 602, and 603 to interact with server 605 through network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0114] The terminal devices 601 , 602 , and 603 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0115] The server 605 may be a server that provides various services, such as a background management server (only an example) that provides support for shopping websites browsed by users using the terminal devices 601, 602, and 603. The background management server may analyze and process the received processing request and other data, and feed back the processing result (such as the processing result - only an example) to the terminal device.

[0116] It should be noted that the data processing method provided in the embodiment of the present invention is generally executed by the server 605 , and accordingly, the data processing device is generally set in the server 605 .

[0117] It should be understood that Figure 6The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0118] Reference below Figure 7 , which shows a schematic diagram of the structure of a computer system 700 of a terminal device or a server suitable for implementing an embodiment of the present invention. Figure 7 The terminal device or server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0119] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the system 700 are also stored. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0120] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage section 708 as needed.

[0121] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the central processing unit (CPU) 701, the above-mentioned functions defined in the system of the present invention are executed.

[0122] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0123] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0124] The units or modules involved in the embodiments of the present invention may be implemented in software or hardware. The units or modules described may also be arranged in a processor. For example, they may be described as: a processor includes an acquisition module, a calculation module, and a processing module. The names of these modules do not, in some cases, constitute a limitation on the unit or module itself. For example, the acquisition module may also be described as "a module for acquiring data to be processed from multiple channels and determining the relationship between the sum of the data volumes of the multiple data to be processed and the thread pool capacity threshold."

[0125] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0126] Obtain the data to be processed from multiple channels, and determine the relationship between the total amount of the data to be processed from multiple channels and the thread pool capacity threshold;

[0127] In response to the amount of data to be processed being greater than a thread pool capacity threshold, calculating a weight value of each channel based on the amount of data to be processed of the multiple channels;

[0128] Based on the weight value of each channel and the total amount of data to be processed, the initial data processing amount of each channel is calculated;

[0129] Based on the initial data processing volume of each channel and the thread pool capacity threshold, the pending data of multiple channels are processed.

[0130] According to the technical solution of the embodiment of the present invention, the weight value of each channel can be calculated based on the acquired data to be processed of different channels and the thread pool capacity threshold, and the initial data processing volume of each channel can be further calculated. Based on the initial data processing volume, the data to be processed of each channel is processed by the thread pool, so as to improve the rationality of resource allocation of thread pools of different channels and improve the efficiency of data processing by the thread pool. Among them, when determining the initial data processing volume of each channel, it can be carried out in different scenarios, including the calculation of the increment and deletion of the initial data processing volume, which can further improve the accuracy of resource allocation of thread pools of different channels and improve data processing efficiency.

[0131] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for data processing, characterized in that: include: Obtaining data to be processed from multiple channels, and determining a relationship between a total amount of the data to be processed from the multiple channels and a thread pool capacity threshold; In response to the amount of the data to be processed being greater than the thread pool capacity threshold, calculating a weight value of each channel based on the amount of the data to be processed of the multiple channels; Calculating the initial data processing volume of each channel based on the weight value of each channel and the total data volume of the plurality of data to be processed; Based on the initial data processing volume of each channel and the thread pool capacity threshold, the to-be-processed data of the multiple channels are processed.

2. The method according to claim 1, characterized in that The method further comprises: In response to the data volume of the to-be-processed data being less than or equal to the thread pool capacity threshold, the to-be-processed data is concurrently processed by the thread pool.

3. The method according to claim 1 or 2, characterized in that: The calculating the weight value of each channel based on the amount of data to be processed of the multiple channels includes: For each channel, the ratio of the amount of data to be processed in the channel to the total amount of data to be processed in the multiple channels is calculated, and the calculated result is used as the weight value of the channel.

4. The method according to claim 1, characterized in that: The processing of the to-be-processed data of the multiple channels based on the initial data processing volume of each channel and the thread pool capacity threshold comprises: Calculating the sum of initial data processing volumes of multiple channels based on the initial data processing volume of each channel; Determine the relationship between the sum of the initial data processing amount and the thread pool capacity threshold to obtain a first determination result; Based on the data processing strategy corresponding to the first judgment result, the to-be-processed data of the multiple channels are processed concurrently through the thread pool.

5. The method according to claim 4, characterized in that The concurrently processing the to-be-processed data of the multiple channels through the thread pool based on the data processing strategy corresponding to the judgment result includes: In response to the sum of the initial data processing volumes being less than the thread pool capacity threshold, determining a magnitude relationship between a sum of the time consumed for calculating the initial data processing volumes of the multiple channels and a preset time threshold, and a magnitude relationship between the initial data processing volumes of the multiple channels and a data volume of the to-be-processed data of the multiple channels, to obtain a second determination result; In response to the second judgment result that the sum of the time consumed for calculating the initial data processing amounts of the multiple channels is less than the preset time threshold, and the initial data processing amounts of the multiple channels are less than the data amount of the to-be-processed data of the multiple channels, based on the initial data processing amounts of the multiple channels, the thread pool capacity threshold and the first preset threshold condition, calculating the increment of the initial data processing amount of at least one channel; updating the initial data processing volume of the plurality of channels based on the increment; Based on the updated initial data processing volume of the multiple channels, the to-be-processed data of the multiple channels are concurrently processed through the thread pool.

6. The method according to claim 5, characterized in that The step of calculating the increment of the initial data processing volume of at least one channel based on the initial data processing volumes of the multiple channels, the thread pool capacity threshold, and the first preset threshold condition comprises: In response to the initial data processing volumes of the multiple channels and the thread pool capacity threshold satisfying the first preset threshold condition, the initial data volume of each channel is increased one by one according to the descending order of the weight value of each channel and the first preset data volume, until the initial data volume of the multiple channels after the increase is satisfied to be equal to the thread pool capacity threshold, thereby obtaining an increment of the initial data processing volume of at least one channel; In response to the initial data processing volume of the multiple channels and the thread pool capacity threshold not satisfying the first preset threshold condition, an increment of each channel is calculated based on a weight value of each channel and a first difference value, wherein the first difference value is a difference between the thread pool capacity threshold value and the initial data processing volume of the multiple channels.

7. The method according to any one of claims 4 to 6, characterized in that: The method further comprises: In response to the sum of the initial data processing volumes being greater than the thread pool capacity threshold, calculating a deletion amount of the initial data processing volume of at least one channel based on the initial data processing volumes of the multiple channels, the thread pool capacity threshold and a second preset threshold condition; updating the initial data processing amounts of the plurality of channels based on the deletion amounts; Based on the updated initial data processing volume of the multiple channels, the to-be-processed data of the multiple channels are concurrently processed through the thread pool.

8. The method according to claim 7, characterized in that The calculating the deletion amount of the initial data processing amount of at least one channel based on the initial data processing amount of the multiple channels, the thread pool capacity threshold and the second preset threshold condition comprises: In response to the initial data processing volumes of the multiple channels and the thread pool capacity threshold satisfying a second preset threshold condition, the initial data volume of each channel is deleted one by one according to the ascending order of the weight value of each channel and the second preset data volume, until the initial data volume of the multiple channels after deletion is equal to a second difference, thereby obtaining a deleted amount of the initial data processing volume of at least one channel, wherein the second difference is the difference between the initial data volume of the multiple channels and the thread pool capacity threshold; In response to the initial data processing volume of the multiple channels and the thread pool capacity threshold not satisfying a second preset threshold condition, a deletion volume of each channel is calculated based on a weight value of each channel and the second difference value.

9. The method according to any one of claims 4 to 6, characterized in that: The method further comprises: In response to the sum of the initial data processing volumes being equal to the thread pool capacity threshold, the to-be-processed data of the multiple channels are processed concurrently through the thread pool.

10. A data processing device, characterized in that: include: An acquisition module is used to acquire data to be processed from multiple channels, and determine the relationship between the total amount of the multiple data to be processed and the thread pool capacity threshold; a calculation module, configured to calculate a weight value of each channel based on the data volume of the data to be processed of the multiple channels in response to the data volume of the data to be processed being greater than the thread pool capacity threshold; Calculating the initial data processing volume of each channel based on the weight value of each channel and the total data volume of the plurality of data to be processed; The processing module is used to process the to-be-processed data of the multiple channels based on the initial data processing volume of each channel and the thread pool capacity threshold.

11. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

12. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images