Data processing method and device in multi-thread environment

Abstract

The embodiment of the invention discloses a data processing method and device in a multi-thread environment, relates to the technical field of big data, and can improve the utilization rate of threadresources and reduce the situations of low load and idle threads. The method comprises the steps that the number of requested lines and the size of sub-packets are acquired, the number of threads is determined by means of the number of requested lines and the size of the sub-packets, the number of requested lines represents the total number of data lines in requested data, and the size of the sub-packets represents the maximum number of request lines processed by each thread; the request data is grouped to obtain at least two task groups; and thread resources are allocated to each task group according to the determined thread number. The method and the device are suitable for processing mass requests in a multi-thread environment.

Description

technical field [0001] The invention relates to the field of big data technology, in particular to a data processing method and device in a multi-thread environment. Background technique [0002] Multi-threading processes a batch of real-time or asynchronous business data in a balanced manner, and allocates a relatively equal amount of business data to each thread in a balanced manner. Wherein, these service data may be service data without mutual dependence or competition, or service data with resource contention. [0003] In the current application solution, when a request has many rows of data, in order to improve the processing speed, multi-thread processing is adopted. If the thread pool threads are sufficient, the program will obtain a thread for each row of data in the thread pool. [0004] However, as the website traffic increases, when multiple or a large number of requests come at the same time, the threads of the machine will be exhausted. Due to the excessive th...

AI Technical Summary

Problems solved by technology

[0004] However, as the website traffic increases, when multiple or a large number of requests come at the same time, the threads of the machine will be exhausted. Due to the excessive thread allocation of the operating system, frequent context switching, and unbalanced task allocation of threads will further lead to request processing. become slow

Moreover, due to the unbalanced tasks assigned by the threads, some threads process a large number of data rows and take a long time, while some threads process a small number of rows and take a short time, but in the end, they have to wait until all threads are processed before returning. Thread resources cannot be fully utilized

Images