Data exporting method, system and computer program product

By executing data reading, processing and writing steps in parallel, and using data queues for buffering and adjustment, the problem of low resource utilization in the prior art is solved, and an efficient data export process is realized.

CN119066113APending Publication Date: 2024-12-03KE COM (BEIJING) TECHNOLOGY CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411290608.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-13
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

During the existing data export process, serial execution exists between the data reading, processing and writing steps, resulting in low system resource utilization and difficult to meet the needs of efficient export.

Method used

The data reading, processing and writing steps are decoupled, executed in parallel, and the result data is buffered and transmitted through the data queue, and the queue capacity is adjusted to balance resource occupancy and export speed.

Benefits of technology

Improve data export efficiency, make full use of system resources, improve resource utilization, and achieve higher data export performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119066113A_ABST
    Figure CN119066113A_ABST
Patent Text Reader

Abstract

The invention provides a data exporting method, a data exporting system and a computer program product, the method comprises a data reading step, a data processing step and a data writing step, the data reading step is used for reading data from a data source to obtain first data, the data processing step is used for processing the first data to obtain second data, and the data writing step is used for writing the second data into the first data; the data writing step is used for writing second data to a specified position, and at least two of the data reading step, the data processing step and the data writing step can be executed in parallel.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to technical fields such as computer technology, and in particular to a data export method, a data export system, and a computer program product. Background Art

[0002] Data export is a high-frequency scenario in business development. For example, in the process of business development, the required data is exported from the database to the file server. In the relevant technology, the entire data export process includes reading the required data from the database, then processing and processing the read data, and then writing the processing results to the specified location, thereby completing the data export process.

[0003] In the above data export process, the export process of different data to be exported is executed serially. During the export process of the current batch of data to be exported, the next batch of data to be exported needs to wait until the export process of the current batch of data is completed before the next batch of data to be exported can be started. This will result in low system resource utilization and data export efficiency is difficult to meet requirements. Summary of the invention

[0004] In order to solve at least one of the above technical problems, the present disclosure provides a data export method, a data export system, a readable storage medium and a computer program product.

[0005] The first aspect of the present disclosure proposes a data export method, comprising: a data reading step, reading data from a data source to obtain first data; a data processing step, processing the first data to obtain second data; and a data writing step, writing the second data to a specified location; wherein at least two of the data reading step, the data processing step and the data writing step can be executed in parallel.

[0006] According to some embodiments of the present disclosure, the data source, the designated location, and the execution entity of the data processing step are respectively set in different servers.

[0007] According to some embodiments of the present disclosure, at least one of the data reading step and the data processing step corresponds to a data queue, and the data queue is used to store result data generated by executing the corresponding step. The result data serves as the to-be-executed data of the next step in the step execution sequence corresponding to the step in the data queue, and is used to be taken out of the data queue and executed by the corresponding step.

[0008] According to some embodiments of the present disclosure, in the data reading step, after obtaining the first data, the first data is stored in a data queue corresponding to the data reading step.

[0009] According to some embodiments of the present disclosure, in the data processing step, after obtaining the second data, the second data is stored in a data queue corresponding to the data processing step.

[0010] According to some embodiments of the present disclosure, the data transmission mode of the data queue is first-in-first-out.

[0011] According to some embodiments of the present disclosure, the data export method also includes: a queue capacity adjustment step, when the execution status of the data export process meets the first condition, adjusting the queue capacity of part or all of the data queues until the first condition is not met.

[0012] According to some embodiments of the present disclosure, the execution status includes a resource occupancy amount of a data export process, and the first condition includes: the resource occupancy amount is greater than a preset first threshold value.

[0013] According to some embodiments of the present disclosure, any one of the data reading step and the data processing step is used as the first execution step. In the first execution step, if all positions of the data queue corresponding to the first execution step are occupied, then: wait until a vacant position appears in the data queue corresponding to the first execution step, and store the corresponding result data in the data queue corresponding to the first execution step.

[0014] According to some embodiments of the present disclosure, any one of the data reading step, the data processing step and the data writing step is used as the second execution step. If the data to be executed in the second execution step satisfies the corresponding second condition, the identification data corresponding to the second execution step is stored in the data queue corresponding to the second execution step, and the identification data is used to indicate the end of the execution of the corresponding second execution step for this batch of data.

[0015] According to some embodiments of the present disclosure, the second condition corresponding to the data reading step includes: when executing the data reading step, the data read from the data source is empty.

[0016] According to some embodiments of the present disclosure, the second condition corresponding to the data processing step and the data writing step includes: the data to be executed obtained from the data queue corresponding to the previous step in the step execution sequence is the identification data corresponding to the previous step.

[0017] According to some embodiments of the present disclosure, any one of the data reading step, the data processing step and the data writing step is used as the third execution step. If the data stored in the data queue corresponding to the third execution step is the corresponding identification data, the execution of the third execution step is stopped after the identification data is stored.

[0018] A second aspect of the present disclosure proposes a data export system, including: a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the method described in any one of the above embodiments is implemented.

[0019] A third aspect of the present disclosure provides a readable storage medium, wherein the readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the method described in any of the above embodiments.

[0020] A fourth aspect of the present disclosure provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it is used to implement at least the method described in any of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0022] Figure 1 A timing diagram of a data export process in related art is shown.

[0023] Figure 2 A schematic diagram of an application scenario of a data export method according to some embodiments of the present disclosure is shown.

[0024] Figure 3 A schematic diagram of the interactive process of the data export process in some embodiments of the present disclosure is shown.

[0025] Figure 4 Shows the Figure 3 A timing diagram of the data export process through the interactive process.

[0026] Figure 5 A flowchart of a data export method according to some embodiments of the present disclosure is shown.

[0027] Figure 6 A flowchart schematically shows the data reading steps of some embodiments of the present disclosure.

[0028] Figure 7 A flowchart schematically shows the data processing steps of some embodiments of the present disclosure.

[0029] Figure 8 A schematic flow chart of data writing steps according to some embodiments of the present disclosure is shown.

[0030] Fig. 9 A schematic flow chart showing data writing steps according to other embodiments of the present disclosure is shown.

[0031] Fig.10 A schematic diagram of a data export system using a hardware implementation of a processing system of some embodiments is shown. DETAILED DESCRIPTION

[0032] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0033] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0034] Unless otherwise specified, the exemplary embodiments / embodiments shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / embodiments can be combined, separated, interchanged and / or rearranged without departing from the technical concept of the present disclosure.

[0035] The terms used herein are for the purpose of describing specific embodiments, rather than being restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but it is not excluded that there are or add one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or provided values ​​that will be recognized by those of ordinary skill in the art.

[0036] In the process of data export, the overall process may involve three different parts: data source, application and file server. In the related art, the process of data export includes the following three parts: data reading, data processing and data writing. Data reading means that the application reads data from the data source, data processing means that the application processes the read data, and data writing means that the application writes the data processing results to the file server.

[0037] Figure 1 FIG. 1 is a schematic diagram showing a timing diagram of a data export process in a related art. Figure 1 , P1 represents the data reading process, D1 is the data obtained by reading data from time t1. P2 represents the data processing process, D1' is the data obtained after processing data D1 from time t2. P3 represents the data writing process, and data D1' is written to the file server from time t3. Figure 1 As can be seen from the timing sequence, the data export process of reading, processing and writing data for the next batch of data D2 can only be started after the current batch of data D1 has completed the three stages of data reading, data processing and data writing in sequence. And when any of the processes P1, P2 and P3 are executed, the remaining two processes will be in a waiting state, and resources cannot be effectively used for data processing. The writing of data D1' starts at time t3, and the reading of the next batch of data D2 can only be started after the writing of data D1' is completed at time t4.

[0038] exist Figure 1 In the data export process shown, the steps of data export are executed serially. Each step needs to process all the current data before the next step can be started. Before the current data is completely processed (processed by all steps), other steps can only be in a waiting state. This processing logic will lead to low resource utilization efficiency and waste of resources.

[0039] Figure 2 A schematic diagram of an application scenario of the data export method of some embodiments of the present disclosure is shown. Figure 2 In this application scenario, a first device 10, a second device 20 and a third device 30 may be included. The data source is deployed in the first device 10, the application is deployed in the second device 20, and the file system is deployed in the third device 30. The first device 10, the second device 20 and the third device 30 can communicate and transmit data in a wired or wireless manner. Figure 2The shapes and structures of the first device 10, the second device 20 and the third device 30 shown in the figure should not be understood as limiting the scope of protection of the present disclosure. In the present disclosure, the first device 10, the second device 20 and the third device 30 can be devices such as servers, laptops or desktops.

[0040] Figure 3 A schematic diagram of the interactive process of the data export process in some embodiments of the present disclosure is shown. Figure 4 Shows the Figure 3 A timing diagram of the data export process through the interactive process. Figure 3 and Figure 4 The detailed description is for a clearer understanding of the technical solution of the present disclosure and should not be considered as a limitation of the protection scope of the present disclosure. In the process of implementing the technical idea of ​​the present disclosure, one or some steps may be omitted, or other alternatives may be adopted.

[0041] See also Figure 3 and Figure 4 , the database DB is configured in the first device as a data source, the application SW, the data queue Q1 and the data queue Q2 are configured in the second device, and the file server FS is configured in the third device. The database DB stores the data to be exported, and the application SW is used to read and process the data in the database DB, and write the processed data to the file server FS to complete the data export. The data queue Q1 and the data queue Q2 are used as buffer queues to store the intermediate results of different steps in the process of data export. Among them, the queue Q1 is used to store the results of data reading, and the queue Q2 is used to store the results of data processing.

[0042] Data export is usually in batches, and the export of a batch of data may involve multiple data reading, data processing, and data writing. This disclosure decouples the data export process into three steps, namely, the data reading step (corresponding to Figure 4 P1 in), data processing steps (corresponding to Figure 4 P2 in the data writing step (corresponding to Figure 4 After decoupling, these three steps can be executed independently and in parallel.

[0043] The second device 20 deployed with the application SW is an important part of the core process control logic of data export. For each batch of exported data, the complete data export process is that the application SW reads the data from the database DB and stores it at the end of the data queue Q1, then takes out a data from the head of the queue Q1 and reads it into the application memory for conversion processing, stores the processed data at the end of the queue Q2, then takes out a data from the head of the queue Q2 and writes it to the temporary file on the disk, and finally uploads the disk file to the file server FS, thereby completing the process of exporting the data from the database DB to the file server FS. It can be understood that after the data is taken out of the queue, there will be one less data in the queue, and an empty position will be added to the queue; after the data is stored in the queue, there will be one more data in the queue, and an empty position will be reduced in the queue. It can be understood that the above-mentioned "one data" can represent a single data, or a group of data formed by multiple data.

[0044] For detailed data export process, please refer to Figure 3 , when starting to export a batch of data, the batch of data contains a data D1~Da. t1 is the start time of exporting the batch of data. At t1, both the data queue Q1 and the data queue Q2 are empty queues. The application SW starts to execute the data reading step at t1, reads the data D1 from the database DB, and completes the operation of storing the data D1 at the end of the data queue Q1 at t2. Among them, the read data can be called the first data, and the data D1 is the first data. At this time, only the data D1 is stored in the queue Q1. After the data D1 is stored at the end of the queue Q1, it becomes the data at the head of the queue Q1. Since the data D1 has not been stored in the queue Q1 at t1, the queue Q1 and the queue Q2 do not contain data. The application SW fails to obtain data from the queue Q1 to perform the data processing step, and fails to obtain data from the queue Q2 to perform the data writing step.

[0045] At time t2, the application SW executes the data reading step and the data processing step in parallel, reads data D2 from the database DB and stores data D2 at the end of the data queue Q1, and at the same time takes out the data D1 currently at the head of the queue from the queue Q1 and processes it to obtain data D1', and stores data D1' into the data queue Q2. The data obtained after processing can be called the second data, and data D1' is the second data. At this time, only data D2 is stored in the queue Q1, and after data D2 is stored at the end of the queue Q1, it becomes the data at the head of the queue Q1. Since data D1' has not been stored in the queue Q2 at time t2, there is no data in the queue Q2, and the application SW fails to obtain data from the queue Q2 to execute the data writing step. It is understandable that Figure 3The order of occurrence of different operations and steps is only an example. For example, the steps of reading data D2 and processing data D1 can be started at the same time, or one step can be executed first and then the other step. When the data reading step and the data processing step are both completed, it reaches time t3.

[0046] At time t3, the application SW executes the data reading step, the data processing step, and the data writing step in parallel, reads the data D3 from the database DB and stores the data D3 at the end of the data queue Q1, and also takes out the data D2 currently at the head of the queue from the queue Q1 and processes it to obtain the data D2', and stores the data D2' into the data queue Q2, and also takes out the data D1' currently at the head of the queue from the queue Q2 and writes it into a temporary file on the disk, and uploads the file to the file server FS. It is understandable that the steps of reading data D3, processing data D2, and writing data D1' can be started at the same time, or some of the steps can be executed first and then the remaining steps are started. When the data reading step, the data processing step, and the data writing step are all completed, it is time t4.

[0047] Starting from time t4, the application SW executes reading data D4, processing data D3 and writing data D2' in parallel, and all three steps enter a repeated loop until the data retrieved from the database DB is empty data when executing the data reading step, indicating that the data D1 to Da in total a contained in the batch of data have all been read, and then the preset first identification data is stored in the tail of the data queue Q1. The first identification data is used to indicate that the data reading step has completed the reading of the batch of data. When the data stored in the queue Q1 is the first identification data, the data reading step terminates the data reading process.

[0048] When the data processing step processes the data in queue Q1, if the data to be processed obtained from Q1 is the first identification data, the preset second identification data is stored in the tail of the data queue Q2. The second identification data is used to indicate that the data processing step has completed the processing of this batch of data. When the data stored in queue Q2 is the second identification data, the data processing step terminates the data processing process.

[0049] The data writing step can also be configured with a corresponding data queue Q3 (not shown in the figure). When the data writing step processes the data in the queue Q2, if the data to be written obtained from Q2 is the second identification data, the preset third identification data is stored in the tail of the data queue Q3. The third identification data is used to indicate that the data writing step has completed the writing of this batch of data. When the data stored in the queue Q3 is the third identification data, the data writing step terminates the data writing process. At this time, a total of a data from D1 to Da have been exported to the file server FS.

[0050] Different types of steps may take different amounts of time to execute. Figure 3 The execution time required for different steps in is only an example. If the execution speed of the data reading step is faster than the data processing step, data will gradually accumulate in the data queue Q1 until the queue capacity of the queue Q1 is filled with data, and the data reading step will be suspended until there is free space in the queue Q1 to store new data. If the execution speed of the data reading step is slower than the data processing step, the amount of data stored in the data queue Q1 is basically zero. Whenever new data is stored in the queue Q1, it will be immediately taken out for processing. When the queue Q1 does not contain any data D, the data processing step will be suspended until new data is stored in the queue Q1.

[0051] If the execution speed of the data processing step is faster than the data writing step, data will gradually accumulate in the data queue Q2 until the queue capacity of the queue Q2 is filled with data, and the data processing step will be suspended until there is a free space in the queue Q2 to store new data. If the execution speed of the data processing step is slower than the data writing step, the amount of data stored in the data queue Q2 is basically zero, and whenever new data is stored in the queue Q2, it will be immediately taken out for writing. When the queue Q2 does not contain any data D', the data writing step will be suspended until new data is stored in the queue Q2.

[0052] Figure 5 A schematic diagram showing a flow chart of a data export method according to some embodiments of the present disclosure is shown. Figure 5 The present disclosure provides a data export method M100, which may include a data reading step S110, a data processing step S130, and a data writing step S150.

[0053] S110, reading data from a data source to obtain first data.

[0054] S130, processing the first data to obtain second data.

[0055] S150, writing the second data to a designated location.

[0056] Among them, at least two steps among the data reading step, the data processing step and the data writing step can be executed in parallel.

[0057] The data source may be a database, and the data stored in the data source may be business data, financial data, or other types of data, which is not limited in the present disclosure. The application is responsible for executing the above-mentioned data reading step S110, data processing step S130, and data writing step S150, thereby realizing data export. The application reads the first data from the data source, and then processes the first data (for example, performs data conversion or other processing) to obtain the second data, and writes the second data to a specified location. The specified location may be located in a file server.

[0058] During the data export process, the application can execute the above data reading step S110, data processing step S130 and data writing step S150 in parallel, for example, while reading data from the data source, the previously read data is also processed, and the previously processed data is also written. The same step will be executed in a loop until all data are exported. For example, every time a data is read from the data source, the next data can be read from the data source immediately; every time the processing of a read data is completed, the next read data can be processed immediately in sequence; every time a processed data is written to the file server, the next processed data can be written immediately in sequence.

[0059] According to the data export method provided by the present invention, the data export process is decoupled into a data reading step, a data processing step and a data writing step, and these three steps can be executed in parallel and independently. There is no mandatory fixed execution order between different types of steps. For each step, as long as there is data to be executed, the corresponding step can be executed on the data to be executed. Different steps can perform different operations on different data at the same time, that is, the execution of different steps is independent and does not interfere with each other. Compared with only the same step being able to operate on a single data at the same time, resource utilization is improved, the export process can make full use of the existing resources of the device, and has higher data export efficiency.

[0060] The data source, the above-mentioned designated location and the execution subject of the data processing step (i.e., the application program) can be respectively set in different servers. The data source can be configured in the first server. The application program can be configured in the second server, and the second server is connected to the first server in communication. The designated location as the written location is located in the file server, and the file server is also connected to the second server in communication.

[0061] For the second server configured with the application program, since the data export process needs to occupy certain system resources, and the faster the data export speed is, the more resources are usually occupied. In order to balance the data export speed and resource occupation, the data export speed and resource occupation are adjusted, and at least one of the data reading step S110 and the data processing step S130 can correspond to a data queue. The data queue is used to store the result data formed by executing the corresponding step. The result data is used as the to-be-executed data of the next step in the step execution sequence corresponding to the data queue, and is used to be taken out from the data queue and executed by the corresponding step.

[0062] The data queue corresponding to the data reading step S110 may be called queue Q1, and queue Q1 is used to store the first data read in step S110, and the first data is the result data formed by executing step S110. The data queue corresponding to the data processing step S130 may be called queue Q2, and queue Q2 is used to store the second data obtained after processing in step S130, and the second data is the result data formed by executing step S130.

[0063] The execution order of the steps is relative to the same data. For the same data, it is necessary to first read the data according to step S110, then process the data according to step S130, and finally write the data according to step S150. There is no fixed execution order for different data, because different types of steps use data queues as buffers for result data, the execution speeds of different steps may be different, and the queue capacities of different data queues may be different. Therefore, when the 100th data is read according to step S110, step S130 may be processing the 70th data, and step S150 may be writing the 50th processed data.

[0064] The data queue can buffer the input and output between each step in the data export process. The data queue is used to realize data interaction between steps. In this way, the data queue can be used to change the progress and execution speed of the overall data export process, which is conducive to balancing and adjusting the data export speed and resource usage.

[0065] It can be understood that the data writing step S150 may also correspond to a data queue, and the data queue may be used for data buffering between the step S150 and the step for further processing the written file.

[0066] In the data reading step, after obtaining the first data, the first data can be stored in the data queue corresponding to the data reading step. In the data processing step, after obtaining the second data, the second data can be stored in the data queue corresponding to the data processing step.

[0067] If a data export process needs to export a data in the database, the application will execute the data reading step S110 on the a data in sequence until all the a data are read. When the a-th data and the data before the a-th data are read through the data reading step S110, that is, there is executable data for the data reading step S110 to read the data, the data reading step S110 will store the read data into the data queue Q1. When the a-th data and the data before the a-th data are processed through the data processing step S110, that is, the queue Q1 contains executable first data for data processing, the data processing step S130 will store the second data obtained after processing into the data queue Q2.

[0068] The data transmission mode of the data queue can be first-in-first-out, that is, both queue Q1 and queue Q2 can be first-in-first-out queues. First-in-first-out means that the order in which data is stored in the data queue is the order in which data is taken out of the data queue. The data stored first will be taken out first, and the data will no longer exist in the queue after being taken out.

[0069] In order to further balance the speed and resource usage of data export and more reasonably adjust the balance between the two, the data export method may further include a queue capacity adjustment step. The queue capacity adjustment step may include: when the execution state of the data export process meets the first condition, adjusting the queue capacity of part or all of the data queues until the first condition is not met. The execution state may include the resource usage of the data export process. The first condition may include: the resource usage is greater than a preset first threshold.

[0070] During the data export process, if the memory occupied by the data export is too high (greater than the first threshold), the queue capacity of the data queue Q1 and the data queue Q2 can be adjusted. The adjustment method is to reduce the queue capacity of the data queue Q1 and the data queue Q2, so that the amount of data that the data queue can store is reduced, thereby reducing the memory occupation and achieving the purpose of reducing resource occupation. Compared with adjusting the single data reading amount of the data reading step by modifying the paging parameters, and adjusting the single data writing amount of the data writing step by modifying the memory resident data volume parameter, the paging parameters cannot be dynamically modified in real time, and the memory resident data volume parameters cannot be modified after initialization, while the data queue can be dynamically modified in real time, so that the control methods for resources and performance in different steps in the data export process can be unified, and the parameters can be dynamically adjustable and take effect in real time. It can be understood that if you want to reduce the execution speed of data export, you can also adjust the data queue in the above manner.

[0071] Take any one of the data reading step and the data processing step as the first execution step. In the first execution step, if all the positions in the data queue corresponding to the first execution step are occupied, you can: wait until a vacant position appears in the data queue corresponding to the first execution step, and store the corresponding result data in the data queue corresponding to the first execution step.

[0072] The data queues Q1 and Q2 each have a maximum queue capacity allowed, that is, a maximum data storage capacity. Since the execution speed of different steps may be different, the queue capacity of the data queue Q1 of the data reading step S110 or the data queue Q2 of the data processing step S130 may be filled. For example, when the data processing amount per unit time of the data processing step S130 is less than the data processing amount per unit time of the data reading step S110, the data storage capacity in the queue Q1 will gradually increase until it reaches the maximum data storage capacity allowed by the queue Q1. At this time, the operation of the data reading step S110 to store data in the queue Q1 will be blocked, and it is necessary to wait for a free position to appear in the queue Q1, that is, wait for the data processing step S130 to take out a data from the queue Q1, and then read the data from the database and store it in the queue Q1. Or the data reading step S110 can continue to read a data from the database and wait when the queue Q1 is full, and when a free position appears in the queue Q1, the read data is directly stored in the queue Q1.

[0073] For another example, when the amount of data processed per unit time in the data writing step S150 is less than the amount of data processed per unit time in the data processing step S130, the amount of data storage in the queue Q2 will gradually increase until it reaches the maximum amount of data storage allowed by the queue Q2. At this time, the operation of storing data in the queue Q2 in the data processing step S130 will be blocked, and it is necessary to wait for an empty position to appear in the queue Q2, that is, wait for the data writing step S150 to take out a data from the queue Q2, and then take out the data from the queue Q1 for processing and store it in the queue Q2. Alternatively, the data processing step S130 can continue to take out data from the queue Q1 for processing when the queue Q2 is full, and start waiting after the processing is completed. When an empty position appears in the queue Q2, the processed data is directly stored in the queue Q2.

[0074] Taking any one of the data reading step, the data processing step and the data writing step as the second execution step, if the data to be executed of the second execution step meets the corresponding second condition, the identification data corresponding to the second execution step can be stored in the data queue corresponding to the second execution step. Wherein, the identification data is used to indicate the end of the execution of the corresponding second execution step for this batch of data. The second condition corresponding to the data reading step may include: the data read from the data source when executing the data reading step is empty. The second condition corresponding to the data processing step and the data writing step may include: the data to be executed obtained from the data queue corresponding to the previous step in the step execution sequence is the identification data corresponding to the previous step. In addition, taking any one of the data reading step, the data processing step and the data writing step as the third execution step, if the data stored in the data queue corresponding to the third execution step is the corresponding identification data, the execution of the third execution step can be stopped after the identification data is stored. Continuing with the example of a data export process that requires exporting a data in the database, after the ath data is read through the data reading step S110, the application will continue to read data from the database, but at this time all a data have been read, so the application will read empty data. At this time, the application will recognize that the read data is empty, and store the preset first identification data in the data queue Q1. After the first identification data is stored, the application terminates the execution of the data reading step S110, thereby stopping the data reading step S110 from continuing to read data from the database through the first identification data.

[0075] After the ath data is processed through the data processing step S130, the application will continue to take data out of the queue Q1, but at this time all a data have been processed. At this time, the application will recognize that the queue Q1 does not contain data to be processed, and directly store the preset second identification data into the data queue Q2. After the second identification data is stored, the application terminates the execution of the data processing step S130, thereby stopping the data processing step S130 from continuing to take data out of the queue Q1 through the second identification data.

[0076] The data writing step S150 may have a corresponding data queue, represented by Q3. The data queue Q3 may be used for data acquisition when the written data is further processed in subsequent steps. After the writing of the ath data is completed through the data writing step S150, the application will continue to take out data from the queue Q2, but at this time all the a data have been written. At this time, the application will recognize that the queue Q2 does not contain the data to be written, and directly store the preset third identification data into the data queue Q3. After completing the storage of the third identification data, the application terminates the execution of the data writing step S150, thereby stopping the data writing step S150 from continuing to take out data from the queue Q2 through the third identification data.

[0077] Data export is performed through the connection of data queues, which solves the problems of inconsistent resource and performance control methods and non-real-time control effects, and realizes unified and real-time control of resources and performance.

[0078] Figure 6 A schematic diagram showing a flow chart of data reading steps in some embodiments of the present disclosure is shown. Figure 6 When the application executes the data reading step S110, it first reads the data from the database, and then determines whether the reading of all the data in this batch is completed based on the reading result. If the read data is empty, it means that the reading of all the data is completed, and the preset first identification data is used as the data to be stored; if the read data is not empty, it means that the reading of all the data is not completed, and the read data is used as the data to be stored. After determining the data to be stored, it is determined whether there is an empty position in the data queue Q1 used for temporarily storing the result data of step S110. If there is an empty position, the data to be stored is stored at the end of the queue; if there is no empty position, it means that the queue Q1 is full, and the application starts to wait until an empty position appears in the queue Q1, and then stores the data to be stored at the end of the queue. Then it is determined whether the data currently stored in the queue Q1 is the first identification data. If it is not the first identification data, the application starts to read the next data from the database, thereby continuing to read the data; if it is the first identification data, the application terminates the execution of the data reading step S110.

[0079] Figure 7 A schematic diagram showing a flow chart of data processing steps in some embodiments of the present disclosure. Figure 7When the application executes the data processing step S130, it first extracts data from the head position of the queue Q1, and then determines whether the queue Q1 contains data to be processed based on the extraction result. If the extracted data is empty, it means that the queue Q1 does not contain data to be processed, then the application starts to wait, and can continue to extract data from the queue Q1 until the extracted data is not empty; if the extracted data is not empty, the data to be processed is obtained. Then it is determined whether the data to be processed is the first identification data. If it is the first identification data, the application uses the second identification data as the data to be stored; if it is not the first identification data, the application processes and converts the obtained data to be processed, and uses the converted data as the data to be stored. After determining the data to be stored, it is determined whether there is an empty position in the data queue Q2 used for temporarily storing the result data of step S130. If there is an empty position, the data to be stored is stored at the end of the queue; if there is no empty position, it means that the queue Q2 is full, then the application starts to wait until there is an empty position in the queue Q2, and then stores the data to be stored at the end of the queue. Then, it is determined whether the data currently stored in the queue Q2 is the second identification data. If it is not the second identification data, the application continues to fetch the next data from the queue Q1, thereby continuing the data processing; if it is the second identification data, the application terminates the execution of the data processing step S130.

[0080] Figure 8 A schematic diagram showing a flow chart of data writing steps in some embodiments of the present disclosure is shown. Figure 8 When the application executes the data writing step S150, it first extracts data from the head position of queue Q2, and then determines whether queue Q2 contains the data to be written based on the extraction result. If the extracted data is empty, it means that queue Q2 does not contain the data to be written, then the application starts waiting, and can continue to extract data from queue Q2 until the extracted data is not empty; if the extracted data is not empty, the data to be written is obtained. Then it is determined whether the data to be written is the second identification data. If it is the second identification data, the application terminates the execution of the data processing step S130; if it is not the second identification data, the application writes the data to be written into a temporary file on the disk and uploads it to the file server.

[0081] Fig. 9 Schematic diagram showing the flow of data writing steps in some other embodiments of the present disclosure. Fig. 9 , this embodiment and Figure 8The difference of the illustrated implementation is that when the application determines whether the data to be written is the second identification data, if it is the second identification data, then the judgment of whether the step needs to be terminated is not performed temporarily at this time, but the preset third identification data is used as the data to be stored; if it is not the second identification data, after the data to be written is written to the temporary disk file, the data to be written is used as the data to be stored. Then the application determines whether there is an empty position in the data queue Q3 for temporarily storing the result data of step S150. If there is an empty position, the data to be stored is stored at the end of the queue; if there is no empty position, it means that the queue Q3 is full, and the application starts to wait until there is an empty position in the queue Q3, and then stores the data to be stored at the end of the queue. Then it is determined whether the data currently stored in the queue Q3 is the third identification data. If it is not the third identification data, the application continues to perform the next data extraction operation on the queue Q2, thereby continuing to write data; if it is the third identification data, the application terminates the execution of the data writing step S150.

[0082] Fig.10 A schematic diagram of a data export system using a hardware implementation of a processing system in some embodiments is shown. Fig.10 The data export system 1000 provided in the present disclosure includes a processor 1200 and a memory 1300. The processor 1200 may be a central processing unit (CPU). The processor 1200 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0083] The memory 1300 can be used as a non-transitory computer-readable storage medium, and can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions of the computer program in the embodiment of the present disclosure. The processor 1200 executes various functional applications and data processing of the data export system by running the non-transitory software programs, instructions and modules stored in the memory 1300, thereby realizing data export.

[0084] The memory 1300 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created by the processor 1200, such as read data, processed data, temporary file data, queue stored data, etc. In addition, the memory 1300 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1300 may optionally include a memory remotely arranged relative to the processor 1200, and these remote memories may be connected to the processor 1200 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0085] The memory 1300 stores a computer program. When the computer program is executed by the processor 1200, the processor 1200 may perform the following steps to implement the data export method.

[0086] S110, reading data from a data source to obtain first data. Step S110 is a data reading step, which can be implemented by the data reading module 1002.

[0087] S130, processing the first data to obtain second data. Step S130 is a data processing step, which can be implemented by the data processing module 1004.

[0088] S150 , writing the second data to the designated location. Step S150 is a data writing step, which can be implemented by the data writing module 1006 .

[0089] At least two of the data reading step S110 , the data processing step S130 , and the data writing step S150 can be performed in parallel.

[0090] The data source, the above-mentioned designated location and the execution subject of the data processing step (ie, the application program) may be respectively set in different servers.

[0091] At least one of the data reading step S110 and the data processing step S130 may correspond to a data queue. The data queue is used to store the result data formed by executing the corresponding step. The result data is used as the to-be-executed data of the next step in the step execution sequence of the step corresponding to the data queue, and is used to be taken out from the data queue and executed by the corresponding step. The data transmission mode of the data queue can be first-in-first-out.

[0092] In the data reading step, after obtaining the first data, the first data can be stored in the data queue corresponding to the data reading step. In the data processing step, after obtaining the second data, the second data can be stored in the data queue corresponding to the data processing step.

[0093] The data export method may further include a queue capacity adjustment step. The queue capacity adjustment step may include: when the execution state of the data export process satisfies a first condition, adjusting the queue capacity of some or all of the data queues until the first condition is not satisfied. The execution state may include the resource usage of the data export process. The first condition may include: the resource usage is greater than a preset first threshold.

[0094] Take any one of the data reading step and the data processing step as the first execution step. In the first execution step, if all the positions in the data queue corresponding to the first execution step are occupied, you can: wait until a vacant position appears in the data queue corresponding to the first execution step, and store the corresponding result data in the data queue corresponding to the first execution step.

[0095] Taking any one of the data reading step, the data processing step and the data writing step as the second execution step, if the data to be executed of the second execution step meets the corresponding second condition, the identification data corresponding to the second execution step can be stored in the data queue corresponding to the second execution step. Wherein, the identification data is used to indicate the end of the execution of the corresponding second execution step for this batch of data. The second condition corresponding to the data reading step may include: the data read from the data source when executing the data reading step is empty. The second condition corresponding to the data processing step and the data writing step may include: the data to be executed obtained from the data queue corresponding to the previous step in the step execution sequence is the identification data corresponding to the previous step. In addition, taking any one of the data reading step, the data processing step and the data writing step as the third execution step, if the data stored in the data queue corresponding to the third execution step is the corresponding identification data, the execution of the third execution step can be stopped after the identification data is stored.

[0096] The system 1000 may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart may be executed by the corresponding module, and the system 1000 may include one or more modules in these modules. The module may be one or more hardware modules specially configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0097] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0098] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0099] The present disclosure also provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the method of any of the above embodiments.

[0100] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transport a program for use with or in conjunction with an instruction execution system, apparatus or device. More specific examples (a non-exhaustive list) of readable storage media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM).

[0101] The present disclosure also provides a computer program product, and the method of the present disclosure can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device, a user device, a core network device, an OAM or other programmable device.

[0102] The computer program or instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0103] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0104] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0105] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0106] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0107] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, unless they are contradictory.

[0108] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0109] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A data export method, characterized in that: include: A data reading step, reading data from a data source to obtain first data; a data processing step of processing the first data to obtain second data; as well as A data writing step, writing the second data to a specified location; At least two of the data reading step, the data processing step and the data writing step can be executed in parallel.

2. The data export method according to claim 1, characterized in that: The data source, the designated location and the execution subject of the data processing step are respectively arranged in different servers.

3. The data export method according to claim 1, characterized in that: At least one of the data reading step and the data processing step corresponds to a data queue, and the data queue is used to store result data generated by executing the corresponding step. The result data serves as the to-be-executed data of the next step in the step execution sequence of the step corresponding to the data queue, and is used to be taken out of the data queue and executed by the corresponding step.

4. The data export method according to claim 3, characterized in that: In the data reading step, after obtaining the first data, the first data is stored in a data queue corresponding to the data reading step.

5. The data export method according to claim 3, characterized in that: In the data processing step, after obtaining the second data, the second data is stored in a data queue corresponding to the data processing step.

6. The data export method according to any one of claims 3 to 5, characterized in that: The data transmission mode of the data queue is first-in first-out.

7. The data export method according to claim 3, characterized in that: The data export method further includes: The queue capacity adjustment step is to adjust the queue capacity of part or all of the data queues when the execution status of the data export process meets the first condition until the first condition is not met.

8. The data export method according to claim 3, characterized in that: Take any one of the data reading step and the data processing step as the first execution step. In the first execution step, if all the positions of the data queue corresponding to the first execution step are occupied, then: wait until a vacant position appears in the data queue corresponding to the first execution step, and store the corresponding result data in the data queue corresponding to the first execution step.

9. A data export system, characterized in that: include: A processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, it is used to implement the method according to any one of claims 1 to 8.

10. A computer program product, characterized in that The computer program product comprises a computer program, which is used to implement at least the method according to any one of claims 1 to 8 when executed by a processor.

Citation Information

Patent Citations

  • Code reviewer recommendation method based on collaborative filtering

    CN107807978A

  • Solid state disk read-write request parallel processing method

    CN110515859A

  • Data writing method, system and device and computer readable storage medium

    CN111835651A

  • High-concurrency data storage method and system

    CN114489480A

  • Concurrent service processing method and device, medium and program product

    CN115185657A