Scheduling method for writing into single file under high concurrency condition

Abstract

The invention relates to a scheduling method for writing into a single file under a high concurrency condition. Aiming at the problem of slow writing to a single file in a high-concurrency multi-thread condition, the method comprises the following steps of: creating a plurality of thread buffer areas An by a plurality of threads, and creating public buffer areas B and C; and setting a threshold value of the multi-thread buffer area An according to the concurrency, and judging whether the multi-thread buffer area An reaches the threshold value or not. And if the set threshold value is reached, writing the data into the buffer area B and emptying the buffer area An to receive new data. Through the method, the problem that writing into a single file is slow under the high concurrency condition is solved. According to the method and the device, the waiting time of writing files under the high-concurrency and multi-thread condition can be shortened, and the efficiency of writing into a single file under the multi-thread and high-concurrency condition can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a scheduling optimization method for writing to a single file under high concurrency conditions. Background Art

[0002] With the rapid development of technologies such as network technology, big data technology, and the Internet of Things, the level of intelligence has been continuously improved. Network applications have spread throughout people's daily lives, becoming an essential part and providing great convenience for people's lives. For example, various delivery software enables people to enjoy a variety of products without leaving home; various travel software meets people's travel needs at any time, ensuring the convenience and safety of travel. The explosive growth of network applications has made high concurrency a normal phenomenon.

[0003] High concurrency means that the system receives a large number of requests within a unit of time. Since the same file does not allow multiple threads to write simultaneously, when multiple threads write data to a single file, a locked state for managing resource access is often adopted to achieve multi-threaded reading or exclusive write access. When an embedded system receives data from other terminals or sensing devices, in the case of high concurrency, multiple threads writing data to a single file often leads to multiple threads competing for the permission to write to the file, resulting in waiting for each other to complete writing the data. The long-term waiting blockage can cause insufficient system resources under high concurrency, longer data processing time, data loss, or memory leakage and other situations. Summary of the Invention

[0004] High concurrency (High Concurrency) means that the system receives a large number of requests within a unit of time. Multithreading (multithreading) refers to the technology of implementing concurrent execution of multiple threads from software or hardware. A computer with multithreading capabilities can execute more than one thread at the same time due to hardware support, thereby improving the overall processing performance. When a large number of requests need to be processed simultaneously, using multithreading can improve the concurrency of the system. When multiple threads write data to a single file, a locked state for managing resource access is often adopted to achieve multi-threaded reading or exclusive write access. In the case of high concurrency, multiple threads writing data to a single file often leads to multiple threads competing for the permission to write to the file, resulting in waiting for each other to complete writing the data. The long-term waiting blockage can cause insufficient system resources under high concurrency, longer data processing time, data loss, or memory leakage and other situations.

[0005] In view of the problem that the existing high-concurrency multi-threads writing data to a single file will cause blocking waiting, the present invention will adopt that each thread independently owns a buffer area, and a certain storage space is reserved in the memory space, and this storage space is used to buffer the input data. To reduce the number of times the thread locks the file, and coupled with the fact that the computer's operation on the buffer is much faster than the operation on the disk, so each thread independently using a buffer can greatly improve the running speed of the computer.

[0006] The technical solution adopted by the present invention is as follows: a scheduling method for writing to a single file under high-concurrency conditions, including the following steps:

[0007] Create a public buffer B, a public buffer C, and multiple local variable thread buffers A respectively;

[0008] When the number of data in a certain thread buffer A reaches its corresponding threshold, lock the current thread buffer A and the public buffer B, and write the data in the current thread buffer A to the public buffer B in the form of a queue. After writing is completed, empty the data in the current thread buffer A, and unlock the thread buffer A and the public buffer B;

[0009] When the number of data in the public buffer B reaches the threshold, lock the public buffer B and the public buffer C, write the content in the B buffer to the public buffer C, empty the data in the public buffer B after writing is completed, and unlock the public buffer B and the public buffer C;

[0010] Then write from the public buffer C to the file.

[0011] The multiple local variable thread buffers A correspond to multiple threads one by one; the data received by each thread is stored in the corresponding thread buffer A in the form of a queue.

[0012] The threshold is configured through an XML configuration file, which is used to set or modify the threshold.

[0013] The configuration file contains the set number of threads n, and the thresholds of the data reception amounts of each local variable thread buffer A, the public buffer B, and the public buffer C.

[0014] For the current thread buffer A, when new data arrives, add it to the end of the queue, and there is no need to reorder or reorganize the entire queue.

[0015] The data is at least one of industrial control data and sensor data.

[0016] The present invention has the following beneficial effects and advantages:

[0017] 1. The present invention can reduce the waiting time for writing files with each other in the case of high concurrency and multi-threading.

[0018] 2. The present invention can reduce the response time.

[0019] 3. The present invention can improve the efficiency of writing to a single file in the case of high concurrency and multi-threading. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of a scheduling optimization method for writing to a single file under high concurrency conditions. DETAILED DESCRIPTION OF THE INVENTION

[0021] The present invention will be further described in detail below with reference to the accompanying drawings.

[0022] The present invention includes creating multiple thread buffers A for multiple threads, and creating common buffers B and C. Set the threshold of the multi-thread buffer A according to the concurrency volume, and determine whether the multi-thread buffer A reaches the threshold. When the set threshold is reached, write the data into the buffer B and clear the buffer A to receive new data. For an embedded system, receiving data from multiple data sources, such as industrial control data, feedback data from multiple sensors, etc., this method can be used to reduce the problem of slow writing to a single file under high concurrency conditions.

[0023] As Figure 1 shown, the present invention allocates an independent buffer for each thread to receive the high-concurrency data volume. An optimization method for improving the speed of writing to a file includes the following steps:

[0024] (1) Create an XML configuration file, which contains the set number of threads n, and the thresholds of the data volumes received by the buffer areas Ai, B, and C. The thresholds included in the configuration file can be modified. When the data volume is small, the number of threads and the thresholds can be set smaller to avoid data loss. When the data volume is large, the number of threads and the thresholds can be set higher to avoid resource contention and thread blocking.

[0025] (2) Read the thresholds set in the configuration file, and create local variable thread buffer areas Ai according to the number of threads n, where i = 1...n; when creating a new thread, create a corresponding thread buffer area Ai and a buffer lock lock_Ai, and the thread buffer area Ai and the buffer lock lock_Ai correspond to the thread one by one. Create a common buffer B, a common buffer C, and corresponding buffer locks lock_B and lock_C.

[0026] (3) Each thread Ai corresponds to a sensor, and uses sockets for network communication to receive the high-concurrency data sent by each sensor.

[0027] (4) Lock the lock_Ai corresponding to buffer Ai to ensure that only the current thread can access and operate on the thread buffer. Determine whether the current thread buffer has reached the set threshold. If not, store the data received by thread Ai in the corresponding thread buffer in the form of a queue (Array) to ensure the orderliness and efficient processing of the data. When a new data item arrives, simply add it to the end of the queue without having to reorder or reorganize the entire queue. Unlock the buffer lock lock_Ai after storage is complete.

[0028] (5) When the number of thread buffers Ai reaches the set threshold, lock the buffer locks lock_Ai and lock_B, and write the content in buffer Ai to the common buffer B in the form of a queue. After writing is complete, clear the content in buffer Ai and unlock the buffer locks lock_Ai and lock_B.

[0029] (6) When the number in buffer B reaches the set threshold, lock the buffer locks lock_B and lock_C, write the content in buffer B to C, clear the content in buffer B after writing is complete, and unlock the buffer locks lock_B and lock_C.

[0030] (7) Then lock the buffer lock lock_C, and write from the common buffer C to the file. Unlock the buffer lock lock_C after writing is complete.

[0031] Through the above scheduling optimization method, each thread is assigned an independent buffer, which can receive a high-concurrency data volume in parallel, improving the speed of writing to the file. At the same time, the buffer and lock methods are used for data storage and writing, reducing competition between threads and improving concurrency efficiency. In addition, by setting the threshold, the data volume of the buffer and the common buffer can be controlled to ensure the stability and reliability of the system.

Claims

1. Scheduling method for writing to a single file under high concurrency conditions, characterized in that, It includes the following steps: Create a public buffer B, a public buffer C, and multiple local variable thread buffers A respectively; When the data quantity in a certain thread buffer A reaches its corresponding threshold, lock the current thread buffer A and the public buffer B, and write the data in the current thread buffer A into the public buffer B in the form of a queue. After the writing is completed, empty the data in the current thread buffer A, and unlock the thread buffer A and the public buffer B; When the data quantity in the public buffer B reaches the threshold, lock the public buffer B and the public buffer C, write the content in the B buffer into the public buffer C. After the writing is completed, empty the data in the public buffer B, and unlock the public buffer B and the public buffer C; Then write from the public buffer C to the file.

2. The scheduling method for writing to a single file under high concurrency conditions according to claim 1, characterized in that, The multiple local variable thread buffers A correspond to multiple threads one by one; the data received by each thread is stored in the corresponding thread buffer A in the form of a queue.

3. The scheduling method for writing to a single file under high concurrency conditions according to claim 1, characterized in that, The threshold is configured through an XML configuration file for setting or modifying the threshold.

4. The scheduling method for writing to a single file under high concurrency conditions according to claim 1, characterized in that, The configuration file contains the set number of threads n and the thresholds for the data quantities received by each local variable thread buffer A, the public buffer B, and the public buffer C.

5. The scheduling method for writing to a single file under high concurrency conditions according to claim 1, characterized in that, For the current thread buffer A, when new data arrives, add it to the end of the queue without reordering or reorganizing the entire queue.

6. The scheduling method for writing to a single file under high concurrency conditions according to claim 1, characterized in that, The data is at least one of industrial control data and sensor data.

Images