A method for providing unique type of data based on distributed

Abstract

The present invention provides a distributed, unique data supply method, belonging to the field of software technology. This method utilizes a high-performance queue to implement distributed file reading, employing a ring process for file dispatch, reading, and rotating data consumption. The present invention utilizes multi-level caching and distributed processing to effectively improve reading efficiency. The present invention pre-reads data batches in accordance with a specified configuration, enabling the execution machine to directly access data without waiting, thus improving overall efficiency, addressing the drawbacks of centralized data reading, and avoiding memory overflow issues caused by large file reading.

Description

Technical Field

[0001] The present invention belongs to the field of software technology, and in particular relates to a method for providing unique type data based on distributed data. Background Art

[0002] Because parameter file reading requires uniqueness in distributed data supply systems, the traditional approach involves using a central processing program to read the data within the file and distribute it. However, disk read / write bottlenecks on individual servers significantly impact data supply efficiency. Technological advancements and the widespread adoption of solid-state drives have significantly improved disk read / write efficiency. However, existing technologies still cannot meet the demands of greater data consumption. Furthermore, when reading large files, reading the entire file at once inevitably results in memory overflows. Summary of the Invention

[0003] In order to solve the above problems, the present invention discloses a method for providing unique type data based on distributed data.

[0004] To achieve the above object, the technical solution of the present invention is as follows:

[0005] The method for providing data based on a distributed unique type of data includes the following steps:

[0006] Step 1: Upload parameter file;

[0007] Step 2: Write the code in the run script, specify the corresponding parameters, and start the distributed executor. The executor can pre-cache a certain number of data packets. When the number of pre-cached data packets on the executor is less than the specified number during data consumption, the executor automatically requests the next batch of data from the data control end.

[0008] Step 3: Start the data control end and the data production end. The data control end is responsible for overall control of parameter file reading and distributes the next parameter file in sequence when a data production end is idle. It requests data from the designated data production end, caches it, and returns a certain batch of data to the execution machine. The data production end reads data based on the file distributed by the data control end and splits the read data into multiple batches of queued data, waiting for the data control end to call and obtain it.

[0009] Step 4: Upon receiving a request from the execution machine, the data control end distributes the files in the corresponding parameter directory to the corresponding data production end in sequence. Each data production end simultaneously reads the data from the files into the high-performance queue of its own process and encapsulates the data to obtain several data packets. When all the data from the data production end has been transmitted to the data control end, the data control end distributes the next parameter file to the data production end. The data production end continues to read the data and stores it in the cache queue, encapsulating the data to obtain several data packets.

[0010] Step 5: The data control end sequentially requests a specified batch of data packets from each data production end, caches them on the data control end, and distributes the data to each execution machine. When each execution machine has cached a sufficient amount of batch data, the data control end stops sending data to the execution machine.

[0011] Step 6: The executor starts running, and the number of data packets in its cache decreases. The executor requests the next batch of data from the corresponding data control end. The data control end directly finds the next batch of data packets from its own cache queue and returns it to the executor. When the data cached by the data control end is less than the specified number of batches, the data from the data production end is triggered to be transmitted to the data control end.

[0012] Step 7: When the data control end recognizes that the parameter file has been read to the last file and there is no data returned to the execution machine from the control end, a data reading completion signal is transmitted to the execution machine, and the execution machine stops requesting data;

[0013] Step 8: The data control end stops providing new data. As the execution machine executes, the batch data executed and cached are gradually exhausted, and the execution machine thread is stopped.

[0014] Furthermore, in Step 1, the size of the uploaded parameter file is limited. When the uploaded parameter file exceeds the limited size, the parameter file is first split into multiple files within the limited size for uploading.

[0015] Furthermore, in Step 1, the reading order of the uploaded parameter files is pre-set.

[0016] Furthermore, in Step 2, each execution machine has a unique identifier.

[0017] Furthermore, in Step 4, the data encapsulation process is as follows: presetting the upper limit of the amount of data that can be contained in a single data packet, and dividing the data and encapsulating it into data packets accordingly, so that the amount of data contained in each data packet does not exceed the upper limit.

[0018] The beneficial effects of the present invention are:

[0019] 1. Combine high-performance queues to implement distributed file reading, and use a ring process for file dispatch, reading, and round-robin data consumption.

[0020] 2. Use multi-level cache and distributed processing to effectively improve reading efficiency.

[0021] 3. Pre-read data batches in advance according to the specified configuration, so that the execution machine can directly obtain data without waiting, improving overall efficiency.

[0022] 4. Solve the drawbacks of central data reading and avoid the memory overflow problem caused by reading large files. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of the main process of the distributed unique type data supply method provided by the present invention, displayed in the form of data pull. DETAILED DESCRIPTION

[0024] The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0025] The present invention provides a method for providing unique data based on distributed data. The main process is as follows: Figure 1 As shown, the following steps are included:

[0026] Step 1: Upload the parameter file. When uploading a parameter file, if it exceeds the specified size (for example, 100MB), you will need to split the file into multiple files of less than 100MB each before uploading. The order in which the files are read must be set, and the file will be read sequentially. The purpose of setting a size limit is to queue the entire file to prevent memory overflow.

[0027] Step 2: The user writes code that uses unique data types in the running script, uses the corresponding parameters (pre-uploaded in step 1), and starts the distributed execution machine. Each execution machine has a unique identifier and will use unique identification data, such as an ID number for account opening operations. Since there are many execution machines started, different ID number information is required at the same time. A powerful data server is needed to provide data support. We used the method of pre-caching and requesting the next batch of data when the number of batches is less than the specified number. The configured batch number of data packets is pre-cached on the execution machine, for example, the specified batch number is 3 batches (assuming 10,000 rows per batch). When there are less than 3 batches of data packets during the data consumption process, the next batch of data will be automatically requested from the data control end. In addition to pre-caching and requesting data from the data control end, the execution machine also has other functions, such as assembling request parameters based on the acquired parameters, calling the interface of the system under test to perform performance stress testing, generating stress testing related report data, etc., but because it is not related to the present invention, it will not be repeated here.

[0028] Step 3: Start the data control and data production terminals. The data control terminal is responsible for overall control of data file reading and can obtain real-time status from the data production terminals. When a data production terminal is idle, it distributes the next parameter file, requests data from the designated data production terminal, and returns a batch of data to the executor. The data production terminal is responsible for reading data based on the files distributed by the data control terminal and splitting the read data into multiple batches of queued data, waiting for the data control terminal to call and retrieve it.

[0029] Step 4: Upon receiving a request from the execution machine, the data control end distributes the data file and caches some data, waiting for the execution machine to call for it. The specific steps are as follows: The data control end assigns the files within a parameter directory to the corresponding data production end one by one. If three data production ends need to be started, file 1 is first assigned to data production end 1, file 2 to data production end 2, and file 3 to data production end 3. Each data production end simultaneously reads the file data into the high-performance queue of its own process. The data is then packaged into a data packet in batches of 10,000 lines. If the current file contains 100,000 lines, it is assembled into 10 data packets for transmission. If all the data from data production end 1 has been transmitted to the data control end, data production end 1 obtains the next parameter file, file 4, continues reading, and stores it in the cache queue. Similarly, each other data production end continues to obtain the next parameter file and read data. Each data producer has a unique number. In this example, there are three data producers, numbered 1, 2, and 3. The producers are used in this order. That is, the data controller distributes parameter files in the order 1-2-3 and reads data in the order 1-2-3. This creates a circular reading and consumption pattern for the data producers. One purpose of this pattern is to ensure that the execution machine maintains a rough order when consuming parameter data.

[0030] Step 5: The data control terminal first requests a specified batch of data packets from data producer 1 (the number of data packet batches cached by the data control terminal should be pre-set) and caches them there. The data control terminal then distributes the data to each executor. Once each executor has cached three batches of data, the data control terminal stops sending data to the executors. After data producer 1 has finished requesting data, the data control terminal then requests a specified batch of data packets from producer 2, and so on, until it reaches the last data producer.

[0031] Step 6: As the executor begins running, the number of cached data packets on the executor decreases to less than three. The executor requests the next batch of data from the corresponding data controller. The data controller directly finds the next batch of data packets from its own cache queue and returns them. When the data controller's cached data falls short of the specified number of batches, it repeatedly triggers production operations on the data producer, causing the data producer to rotate in a continuous loop.

[0032] Step 7: When the data control terminal recognizes that the parameter file has been read to the last file, the circular reading process stops. This signal is then transmitted to the executor when no more data is returned to the executor. This way, the executor will no longer request data from the data control terminal.

[0033] Step 8: As the data control end provides no new data, the batch data cached by the execution machine will gradually be exhausted and the execution machine thread will be stopped.

[0034] It should be noted that the above content merely illustrates the technical idea of the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. A method for providing data based on a unique type of distributed data, characterized in that: The steps include: Step 1: Upload parameter file; Step 2: Write the code in the run script, specify the corresponding parameters, and start the distributed executor. The executor can pre-cache a certain number of data packets. When the number of pre-cached data packets on the executor is less than the specified number during data consumption, the executor automatically requests the next batch of data from the data control end. Step 3: Start the data control end and the data production end. The data control end is responsible for overall control of parameter file reading and distributes the next parameter file in sequence when a data production end is idle. It requests data from the designated data production end, caches it, and returns a certain batch of data to the execution machine. The data production end reads data based on the file distributed by the data control end and splits the read data into multiple batches of queued data, waiting for the data control end to call and obtain it. Step 4: Upon receiving a request from the execution machine, the data control end distributes the files in the corresponding parameter directory to the corresponding data production end in sequence. Each data production end simultaneously reads the data from the files into the high-performance queue of its own process and encapsulates the data to obtain several data packets. When all the data from the data production end has been transmitted to the data control end, the data control end distributes the next parameter file to the data production end. The data production end continues to read the data and stores it in the cache queue, encapsulating the data to obtain several data packets. Step 5: The data control end sequentially requests a specified batch of data packets from each data production end, caches them on the data control end, and distributes the data to each execution machine. When each execution machine has cached a sufficient amount of batch data, the data control end stops sending data to the execution machine. Step 6: The executor starts running, and the number of cached data packets in the executor decreases. The executor requests the next batch of data from the corresponding data control end. The data control end directly finds the next batch of data packets from its own cache queue and returns them to the executor. When the data cached by the data control end is less than the specified number of batches, it triggers the data transmission from the data production end to the data control end. Step 7: When the data control end recognizes that the parameter file has been read to the last file and there is no data returned to the executor from the control end, it transmits a data reading completion signal to the executor, stopping the executor's data request. Step 8: The data control end stops providing new data. As the execution machine executes, the batch data executed and cached are gradually exhausted, and the execution machine thread is stopped.

2. The method for providing unique type data based on distributed data according to claim 1, characterized in that: In Step 1, the size of the uploaded parameter file is limited. When the uploaded parameter file exceeds the limited size, the parameter file is first split into multiple files within the limited size for uploading.

3. The method for providing unique type data based on distributed data according to claim 1, characterized in that: In Step 1, the reading order of the uploaded parameter files is preset.

4. The method for providing unique type data based on distributed data according to claim 1, characterized in that: In Step 2, each execution machine has a unique identifier.

5. The method for providing unique type data based on distributed data according to claim 1, characterized in that: In Step 4, the data encapsulation process is as follows: presetting the upper limit of the amount of data that can be contained in a single data packet, and dividing the data and encapsulating it into data packets accordingly, so that the amount of data contained in each data packet does not exceed the upper limit.

Images