A data writing method, device, equipment and readable storage medium
By receiving configuration files from Flink and writing data according to the partition fields, the problem of writing data to partitions in distributed file systems is solved, achieving efficient data writing and resource conservation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN ONETHING TECH CO LTD
- Filing Date
- 2022-07-25
- Publication Date
- 2026-06-02
AI Technical Summary
In distributed file systems, data writing presents challenges such as the excessive number of partitions to be written to and the opening of too many handles, which are difficult to resolve effectively with existing technologies.
Flink receives configuration files, reads task information, obtains data, and writes the data to the corresponding time partition of the file system according to the partition field. It delays the writing of data to historical partitions, controls the time and size thresholds of data writing, and avoids excessive handles.
It enables efficient matching of data in the distributed file system, avoids the problem of too many handles caused by too much delayed data, and improves write speed and data consistency.
Smart Images
Figure CN115221116B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of storage technology, and in particular to a data writing method, apparatus, device, and readable storage medium. Background Technology
[0002] Writing data to the file system is typically done using Flink's built-in StreamingFileSink. However, problems still exist when adapting to distributed file systems, such as how data is specifically written to partitions within the distributed file system and the issue of opening too many handles.
[0003] In conclusion, how to effectively solve problems such as data writing is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0004] The purpose of this application is to provide a data writing method, apparatus, device, and readable storage medium that can effectively solve the data writing problem.
[0005] To solve the above-mentioned technical problems, this application provides the following technical solution:
[0006] A data writing method, comprising:
[0007] Flink receives a configuration file and reads task information from the configuration file;
[0008] Based on the task information, obtain the data corresponding to the task;
[0009] Read the partition field from the data, and write the data to the corresponding time partition in the file system according to the partition field;
[0010] If the data is delayed, then the data is written to the history partition of the file system.
[0011] Preferably, writing the data to the corresponding time partition in the file system includes:
[0012] If the size of the data reaches the write size threshold, then the step of writing the data to the corresponding time partition in the file system is executed;
[0013] If the data size does not reach the write size threshold, and no data is written to the file system within a timeout period, then the step of writing the data to the corresponding time partition in the file system is executed.
[0014] Preferably, the timeout for not writing data to the file system includes:
[0015] If the clock data generated by the timer is obtained, it is determined that no data has been written to the file system within the timeout period.
[0016] Preferably, the receiving configuration file includes:
[0017] Receive the configuration folder; the task folder includes the configuration files corresponding to at least two tasks.
[0018] Preferably, writing the data to the corresponding time partition in the file system includes:
[0019] Retrieve the field size and field type corresponding to the data from the database;
[0020] The data is written to the corresponding time partition in the file system according to the field size and the field type.
[0021] Preferably, it further includes:
[0022] Read the temporarily stored delayed data from the historical partition, and read the partition field of the delayed data;
[0023] Based on the partition field, the delayed data is written to the corresponding time partition in the file system.
[0024] Preferably, if the data is delayed in transmission, then writing the data to the history partition of the file system includes:
[0025] If the data is not retrieved within a specified time after it is generated, the data will be written to the historical partition once it is retrieved.
[0026] Preferably, writing the data to the corresponding time partition in the file system includes:
[0027] The data is written in parallel to a temporary directory in the file system;
[0028] After the data writing is complete, the data is transferred to the partition directory.
[0029] A data writing device, used in Flink, includes:
[0030] The information acquisition module is used to receive the configuration file and read task information from the configuration file;
[0031] The data retrieval module is used to retrieve the data corresponding to the task based on the task information.
[0032] The partition writing module is used to read the partition field from the data and write the data to the corresponding time partition in the file system according to the partition field.
[0033] The merge write module is used to write the data to the historical partition of the file system if the data is delayed in transmission.
[0034] A merge-write module is an electronic device comprising:
[0035] Memory, used to store computer programs;
[0036] A processor is used to implement the steps of the above-described data writing method when executing the computer program.
[0037] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described data writing method.
[0038] Using the method provided in the embodiments of this application, Flink receives a configuration file and reads task information from the configuration file; obtains the data corresponding to the task based on the task information; reads the partition field from the data and writes the data to the corresponding time partition in the file system based on the partition field; if the data is delayed, it is written to the history partition of the file system.
[0039] In this application, task information is obtained through a configuration file, and corresponding data is retrieved based on the task information. A partition field can be read from the data. Based on this partition field, the data can be written to the corresponding time partition in the file system. This solves the problem of how to write data to each partition in the file system with matching characteristics. Furthermore, if data is delayed in transmission, it is directly written to the historical partition of the file system, avoiding cluster problems caused by too many open file handles on the machine due to excessive delayed data being written to normal partitions.
[0040] Accordingly, embodiments of this application also provide a data writing device, apparatus, and readable storage medium corresponding to the above-described data writing method, which have the aforementioned technical effects, and will not be elaborated further here. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is a flowchart illustrating the implementation of a data writing method in an embodiment of this application.
[0043] Figure 2 This is a schematic diagram of a data processing framework in an embodiment of this application;
[0044] Figure 3 This is a schematic diagram of the structure of a data writing device according to an embodiment of this application;
[0045] Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;
[0046] Figure 5 This is a schematic diagram of the specific structure of an electronic device in an embodiment of this application. Detailed Implementation
[0047] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0048] Please refer to Figure 1 , Figure 1 This is a flowchart of a data writing method according to an embodiment of this application. This method can be specifically applied in Flink (a distributed streaming data engine). Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner. Flink's pipelined runtime system can execute batch processing and streaming processing programs. In addition, Flink's runtime itself also supports the execution of iterative algorithms.
[0049] Flink can be specifically applied to, for example... Figure 2 In the data processing framework shown, it acts as a data consumer.
[0050] Kafka is a high-throughput distributed publish-subscribe messaging system that can handle all action streams of data from consumers on a website.
[0051] For example, after a Flume (a highly available, reliable, distributed system for collecting, aggregating, and transmitting massive amounts of logs, providing the ability to perform simple data processing and write to various data receivers) data producers send data to Kafka, Kafka acts as a middleware buffer, and then Flink consumes the data and persists it to the Hadoop Distributed File System (HDFS). Alternatively, the consumed data can be persisted to distributed databases (such as ClickHouse), Kudu, and TiDB, and automated access and persistence can be achieved by specifying the persistence name.
[0052] Right now, Figure 2 The flow of data in China is roughly as follows:
[0053] Flume collects file logs written to disk by the machine and sends the file log messages to Kafka, acting as a message producer for Kafka.
[0054] As a message middleware, Kafka primarily acts as a data buffer to prevent data loss caused by sudden large amounts of data being written and consumption not being timely.
[0055] Flink acts as a consumer of Kafka, actively pulling data from Kafka and writing it to the backend storage component HDFS.
[0056] The following section uses the example of writing data to a distributed file system to illustrate the data writing method in detail. This method includes the following steps:
[0057] S101: Flink receives the configuration file and reads the task information from the configuration file.
[0058] Configure the change items for each task in the configuration file. Here, "task" specifically refers to the data writing task, and "change items" specify which table the data corresponds to, the specific read / write location, and other task information. Each configuration file can correspond to a specific task.
[0059] The configuration file can be received using Flink's built-in utility classes. Once the configuration file is obtained, task information can be read from it.
[0060] S102. Obtain the data corresponding to the task based on the task information.
[0061] Based on the task information, the data corresponding to the task can be pulled from Kafka.
[0062] S103. Read the partition field from the data and write the data to the corresponding time partition in the file system according to the partition field.
[0063] The partition field can be a partition identifier, such as 5min, 60min, 24h, or if different partitions have their own numbers, the partition field can correspond to the partition number, such as 01, 02, 03, etc.
[0064] The partitions can be pre-defined based on time. Each partition can correspond to a different time period; therefore, in this application, the partitions are referred to as time partitions. For example, a partition can correspond to a 5-minute partition, a 60-minute partition, or a 24-hour partition.
[0065] The data contains a partition field, which determines which partition in the file system the current data should be written to. Specifically, in Flink's custom hdfs-sink (distributed file system gateway node), data can be placed into different time partitions according to the time specified in the partition field.
[0066] S104. If the data is delayed, then write the data to the historical partition of the file system.
[0067] To avoid opening too many handles, a history partition can be set up. In the event of delayed data, the corresponding data can be written directly to the history partition.
[0068] It is important to note that this historical partition can be identical in hardware and software architecture to any other partition within the time partition. Since the historical partition stores latency data corresponding to different time partitions, it can also be considered a special time partition with mixed time signatures.
[0069] When dealing with large amounts of data, data transmission requires queuing, which may result in some data being sent with delays. In this context, this is referred to as delayed data. In other words, data whose generation time differs significantly from its consumption time (this can be determined by setting a threshold) can be considered delayed data.
[0070] One way to determine if data is delayed is to include a timestamp (or partition time) and an expiration time in the data. Based on this timestamp and expiration time, it can be determined whether the data is late. For example, if data A's partition time corresponds to 7:00-8:00 and its expiration time is 2 hours, then if the data is retrieved after 10:00, it has already expired, thus confirming that the data is expired and can be directly written to the historical partition. Of course, determining whether data was sent late, or whether the data is delayed, can also be based on the time difference between the data's generation time and the time it was retrieved. This application only needs to determine whether the data is delayed; the specific implementation method for determining whether the data is delayed is not limited.
[0071] It should be noted that for delayed data, after obtaining the data in step S102, step S104 must be executed first, and then step S103 must be executed; for data that arrives normally, after obtaining the data in step S102, step S103 can be executed directly.
[0072] Using the method provided in the embodiments of this application, Flink receives a configuration file and reads task information from the configuration file; obtains the data corresponding to the task based on the task information; reads the partition field from the data and writes the data to the corresponding time partition in the file system based on the partition field; if the data is delayed, it is written to the history partition of the file system.
[0073] In this application, task information is obtained through a configuration file, and corresponding data is retrieved based on the task information. A partition field can be read from the data. Based on this partition field, the data can be written to the corresponding time partition in the file system. This solves the problem of how to write data to each partition in the file system with matching characteristics. Furthermore, if data is delayed in transmission, it is directly written to the historical partition of the file system, avoiding cluster problems caused by too many open file handles on the machine due to excessive delayed data being written to normal partitions.
[0074] It should be noted that, based on the above embodiments, the embodiments of this application also provide corresponding improvement schemes. In the preferred / improved embodiments, the same or corresponding steps as in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other; however, these will not be elaborated upon in the preferred / improved embodiments herein.
[0075] In a specific embodiment of this application, step S104, writing data to the corresponding time partition in the file system, specifically includes:
[0076] If the data size reaches the write size threshold, then the step of writing the data to the corresponding time partition in the file system will be executed.
[0077] If the data size has not reached the write size threshold, and no data is written to the file system within a timeout period, then the step of writing the data to the corresponding time partition in the file system will be executed.
[0078] In other words, when writing data to the file system, data can be written to the file system in batches. That is, once the data size reaches the write size threshold, the data can be directly written to the corresponding time partition in the file system.
[0079] Specifically, to avoid cluster problems caused by too many open handles, data can be written to the corresponding time partition in the file system if the data size has not reached the write size threshold but a timeout occurs before the data is written to the file system. This timeout can be determined by setting a time threshold or by maintaining a periodic clock. For example, it could be determined by the time since the last data write being greater than a preset threshold, or by the data generation time (which can be determined by adding a timestamp to the data to determine the generation time) being greater than a preset threshold.
[0080] In one specific embodiment of this application, a time-driven data stream (referred to herein as clock data) can be set to push the timer so that data can be properly flushed to the backend storage when the data has not reached the threshold within a batch (corresponding to the write size threshold) or interval-time (write interval time). This ensures that no further data may arrive, but data loss is prevented. In other words, if the timed clock data is obtained, it is determined that no data has been written to the file system within the specified timeout period.
[0081] For example, Flink generates clock data at fixed intervals (e.g., 5 seconds). The purpose of this time-driven data is to allow the system to obtain the time, and when the time exceeds a threshold, the data is directly written to storage. Of course, this clock data can also be generated by devices such as Kafka and sent to Flink.
[0082] In one specific embodiment of this application, multiple tasks can share the cluster's CPU, thereby saving resources. Specifically, in the implementation step S101 of receiving configuration files, the following steps are included: receiving a configuration folder; the task folder includes configuration files corresponding to at least two tasks respectively.
[0083] Specifically, changes to each task are configured through configuration files. Furthermore, Flink's built-in utility classes can submit a folder as a single task to share the machine's CPU, thus saving resources. In other words, this utility class can submit multiple configuration files within a folder at once, allowing Flink to process multiple tasks with multiple configuration files as a single task in parallel.
[0084] In one specific embodiment of this application, Avro format data can be supported. Specifically, step S102, based on task information, obtains data corresponding to the task, including: obtaining Avro format data from the Kafka cluster corresponding to the task information, and performing deserialization processing on the Avro format data to obtain the data.
[0085] Avro format is a row-based storage format used in Hadoop and is widely used as a serialization platform. Data in Avro format with locale is typically read as a whole for further processing by downstream systems (in which case the row-based format is more efficient); downstream systems can easily retrieve table schemas from Avro files (without storing the schema separately in external meta storage); any changes to the source schema are easily handled (schema evolution).
[0086] Data stored within a Kafka cluster is serialized data, therefore, deserialization is required after the data is retrieved. The specific implementation of deserialization can be found in relevant implementation schemes, and will not be elaborated upon here.
[0087] In one specific embodiment of this application, step S103, writing data to the corresponding time partition in the file system, includes:
[0088] Step 1: Retrieve the field size and field type of the data from the database;
[0089] Step 2: Write the data to the corresponding time partition in the file system according to the field size and field type.
[0090] For ease of description, the two steps above will be explained together below.
[0091] Before writing, the metadata of the HDFS tables resides in a MySQL database. Therefore, the mapping between fields and data types can be automatically performed by accessing the MySQL metadata, based on the field size and type. The HDFS data format is obtained through MySQL, allowing for direct retrieval of the corresponding file format during writing to HDFS. In other words, the field size and type are retrieved from the database. Based on this field size and type, the corresponding data can be written.
[0092] In one specific embodiment of this application, if the data is delayed in step S104, the data is written to the historical partition of the file system. Specifically, this includes: if the data is not retrieved within a specified time after its generation, the data is written to the historical partition after it is retrieved. Specifically, time information can be read from the data; the time information can be used to determine whether the data is delayed; if so, the data is written to the historical partition of the file system.
[0093] The time information can be specifically the partition time and expiration time. That is, by reading the time partition from the data, and based on the partition time and expiration time, it can be determined whether the data is late. If the data is late, to avoid opening too many file handles, the data can be written to the file system's history partition.
[0094] Furthermore, after writing data to the historical partition of the file system, for easier data management, the data in that historical partition can be read out and stored in the corresponding time partition. Specific implementations include:
[0095] Step 1: Read the temporarily stored delayed data from the historical partition and read the partition field of the delayed data.
[0096] Step 2: Write the delayed data to the corresponding time partition in the file system according to the partition field.
[0097] Periodically, or when the number of historical partitions reaches a threshold, temporary data (referred to as delayed data in this article for clarity) can be read from historical partitions in a first-in, first-out order. The partition field within this delayed data can then be used to determine its normal time partition. For details on how to write delayed data to the corresponding time partition in the file system based on the partition field, please refer to the description above; it will not be repeated here.
[0098] In one specific embodiment of this application, step S103, writing data to the corresponding time partition in the file system, includes:
[0099] Step 1: Write the data in parallel to a temporary directory in the file system;
[0100] Step 2: After writing the data, transfer the data to the partition directory.
[0101] A staging directory can be pre-defined. When data is written, it can first be written to a staging directory, similar to Hadoop's native MapReduce program. When the data flushing is complete or the task is canceled, a method is called to write the temporary files in the staging directory to the final partition directory. In other words, storing the data to be written in a temporary folder beforehand and changing the directory after the writing is complete can effectively prevent incomplete data from being displayed prematurely.
[0102] To facilitate those skilled in the art to better understand and implement the data writing methods provided in the above method embodiments, the key contents of each method are described in detail below.
[0103] In Flink's custom HDFS sink, data is distributed into different partitions based on the time specified in the field. Assuming the partition time is 5 minutes and the expiration time is 2 hours, data within the last 2 hours will be placed in the rolling 5-minute partition; data outside the last 2 hours will be placed in the history partition's file path. Later, dynamic partitioning queries will retrieve data from each partition to fill in the gaps.
[0104] The expiration time can help prevent opening too many files.
[0105] Please refer to Figure 2 The process of data from generation to consumption is as follows:
[0106] Flume collects data from log files on the business / server side or from UDP (User Datagram Protocol) protocol, and acts as a producer to package the data into Avro format and send it to Kafka.
[0107] Kafka acts as a buffer for the entire system, storing data in the Kafka broker.
[0108] Flink acts as a consumer, pulling data from Kafka, including data in Avro format, and then writing the data to different storage backends through a custom sink.
[0109] Writing data to HDFS typically involves using Flink's built-in StreamingFileSink or Flume's hdfs-sink. However, this approach suffers from slow write speeds for large datasets and the risk of data loss. A custom implementation of Parquet-sink, using the data writing method described in the above examples, offers significantly faster write speeds compared to the native ParquetWriter.
[0110] The Parquet-sink step includes:
[0111] First, configuration files are used to specify changes for each task. Then, using Flink's built-in utility classes, a folder can be submitted as a single task to share the machine's CPU, thus saving resources.
[0112] The system reads data from the corresponding Kafka cluster through the configuration file, retrieves avro, string, or other data types, and then deserializes the data.
[0113] Set up a time-driven data stream to ensure that data is not lost if it has not reached the threshold within a batch or interval-time period. This pushes the time to ensure that the data is properly flushed to the backend storage and avoids data being written to disk if the threshold is not reached for an extended period.
[0114] Before writing, the metadata of the HDFS table exists in MySQL. Therefore, the mapping of fields and types to be written can be automatically done by accessing the metadata of MySQL based on the size and type of the fields.
[0115] Based on the specified partition field in the data, the data is divided into corresponding partitions, such as 5-minute partitions, 60-minute partitions, and 24-hour partitions. Data is also divided into partitions based on its actual time. Furthermore, a history partition is set; data arriving within the set time delay is written to the normal partition, while data arriving much later is written to the history partition. This avoids cluster problems caused by too many open file handles on the machine if too much delayed data is written to the normal partition.
[0116] For the writing part, it is first written to a staging directory, similar to the native MapReduce program in Hadoop. When the data flushing is completed or the task is canceled, a method is called to write the temporary files in the temporary staging directory to the formal partition directory, which can effectively prevent the data from being displayed in advance.
[0117] Accordingly, the Clickhouse-sink steps are as follows:
[0118] Similarly, tasks and timelines are set up using configuration files to drive data flow.
[0119] The table's metadata is retrieved by accessing the tables in ClickHouse. The schema is returned to the program, which then inserts data into ClickHouse via the JDBC protocol.
[0120] Data is written to the ClickHouse database using ClickHouse's native JDBC write functionality. Tests have shown that using a third-party library results in a speed 3 million transactions per minute slower than using the native library under load testing.
[0121] In other words, this application can customize the pipeline from Flink to each sink data terminal, and also customizes various functions required by the platform, and has the following advantages:
[0122] 1. Due to Flink's asynchronous distributed snapshot checkpointing, data consistency is guaranteed, resulting in a much faster write speed than Flume, and no data loss occurs.
[0123] 2. Limit the size of the file written by each thread by controlling the write time and the size of the file being written.
[0124] 3. After automatically obtaining various metadata of the table, automatically establish field and type mapping relationships and write the data to the file.
[0125] 4. It will differentiate data partitions to prevent cluster problems caused by too many file handles opened by the program.
[0126] Corresponding to the above method embodiments, this application also provides a data writing device, and the data writing device described below can be referred to in correspondence with the data writing method described above.
[0127] See Figure 3 As shown, this device is used in Flink and includes the following modules:
[0128] The information acquisition module 101 is used to receive the configuration file and read task information from the configuration file;
[0129] The data retrieval module 102 is used to retrieve the data corresponding to the task based on the task information;
[0130] The partition writing module 103 is used to read the partition field from the data and write the data to the corresponding time partition in the file system according to the partition field.
[0131] The merge write module 104 is used to write the data to the historical partition of the file system if the data is delayed.
[0132] Using the apparatus provided in the embodiments of this application, Flink receives a configuration file and reads task information from the configuration file; obtains data corresponding to the task based on the task information; reads the partition field from the data and writes the data to the corresponding time partition in the file system based on the partition field; if the data is delayed, it is written to the history partition of the file system.
[0133] In this application, task information is obtained through a configuration file, and corresponding data is retrieved based on the task information. A partition field can be read from the data. Based on this partition field, the data can be written to the corresponding time partition in the file system. This solves the problem of how to write data to each partition in the file system with matching characteristics. Furthermore, if data is delayed in transmission, it is directly written to the historical partition of the file system, avoiding cluster problems caused by too many open file handles on the machine due to excessive delayed data being written to normal partitions.
[0134] In one specific embodiment of this application, the partition writing module 103 is specifically used to execute the step of writing the data to the corresponding time partition in the file system when the size of the data reaches the write size threshold.
[0135] If the data size has not reached the write size threshold, and no data is written to the file system within a timeout period, then the step of writing the data to the corresponding time partition in the file system will be executed.
[0136] In one specific embodiment of this application, the timeout for not writing data to the file system includes:
[0137] If the clock data generated by the timer is obtained, it is determined that no data has been written to the file system within the timeout period.
[0138] In one specific embodiment of this application, the information acquisition module 101 is used to receive a configuration folder; the task folder includes configuration files corresponding to at least two tasks.
[0139] In one specific embodiment of this application, the partition writing module 103 is specifically used to obtain the field size and field type corresponding to the data from the database;
[0140] Data is written to the corresponding time partition in the file system according to the field size and field type.
[0141] In one specific embodiment of this application, it further includes:
[0142] The data regression processing module is used to read temporarily stored delayed data from historical partitions and read the partition field of the delayed data;
[0143] Based on the partition field, the delayed data is written to the corresponding time partition in the file system.
[0144] In one specific embodiment of this application, the merge writing module 104 is specifically used to write the data to the historical partition after the data is obtained if the data is not obtained within a specified time after the data is generated.
[0145] In one specific embodiment of this application, the partition writing module 103 is specifically used to write data in parallel to a temporary directory in the file system;
[0146] After the data writing is complete, the data will be transferred to the partition directory.
[0147] Corresponding to the above method embodiments, this application also provides an electronic device. The electronic device described below and the data writing method described above can be referred to in correspondence.
[0148] See Figure 4 As shown, the electronic device includes:
[0149] Memory 332 is used to store computer programs;
[0150] The processor 322 is used to implement the steps of the data writing method in the above method embodiment when executing a computer program.
[0151] For details, please refer to Figure 5 , Figure 5This is a schematic diagram of a specific structure of an electronic device provided in this embodiment. The electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer application programs 342 or data 344. The memory 332 can be temporary or persistent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data processing device. Furthermore, the central processing unit 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the electronic device 301.
[0152] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.
[0153] The steps in the data writing method described above can be implemented by the structure of an electronic device.
[0154] Corresponding to the above method embodiments, this application also provides a readable storage medium. The readable storage medium described below and the data writing method described above can be referred to in correspondence.
[0155] A readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the data writing method described in the above method embodiments.
[0156] Specifically, the readable storage medium can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other readable storage medium capable of storing program code.
[0157] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
Claims
1. A data writing method, characterized in that, include: Flink receives a configuration file and reads task information from the configuration file; Based on the task information, obtain the data corresponding to the task; Read the partition field from the data, and write the data to the corresponding time partition in the file system according to the partition field; If the data is delayed, then the data is written to the history partition of the file system; the history partition stores the delayed data corresponding to different time partitions; When the amount of delayed data in a historical partition reaches a threshold, the temporarily stored delayed data is read from the historical partition in a first-in-first-out order, and the partition field of the delayed data is read. Based on the partition field, the delayed data is written to the corresponding time partition in the file system.
2. The data writing method according to claim 1, characterized in that, Writing the data to the corresponding time partition in the file system includes: If the size of the data reaches the write size threshold, then the step of writing the data to the corresponding time partition in the file system is executed; If the data size does not reach the write size threshold, and no data is written to the file system within a timeout period, then the step of writing the data to the corresponding time partition in the file system is executed.
3. The data writing method according to claim 2, characterized in that, Data not written to the file system within the specified timeout period includes: If the clock data generated by the timer is obtained, it is determined that no data has been written to the file system within the timeout period.
4. The data writing method according to claim 1, characterized in that, The received configuration file includes: Receive the configuration folder; the configuration folder includes the configuration files corresponding to at least two tasks.
5. The data writing method according to claim 1, characterized in that, Writing the data to the corresponding time partition in the file system includes: Retrieve the field size and field type corresponding to the data from the database; The data is written to the corresponding time partition in the file system according to the field size and the field type.
6. The data writing method according to claim 1, characterized in that, If the data is delayed, then the data is written to the historical partition of the file system, including: If the data is not retrieved within a specified time after it is generated, the data will be written to the historical partition once it is retrieved.
7. The data writing method according to any one of claims 1 to 6, characterized in that, Writing the data to the corresponding time partition in the file system includes: The data is written in parallel to a temporary directory in the file system; After the data writing is complete, the data is transferred to the partition directory.
8. A data writing device, characterized in that, Applied to Flink, including: The information acquisition module is used to receive the configuration file and read task information from the configuration file; The data retrieval module is used to retrieve the data corresponding to the task based on the task information. The partition writing module is used to read the partition field from the data and write the data to the corresponding time partition in the file system according to the partition field. The merge write module is used to write the data to the historical partition of the file system if the data is delayed; the historical partition stores the delayed data corresponding to different time partitions. The data regression processing module is used to periodically or when the amount of delayed data in the historical partition reaches a threshold, read the temporarily stored delayed data from the historical partition in a first-in-first-out order, and read the partition field of the delayed data; according to the partition field, write the delayed data into the corresponding time partition in the file system.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data writing method as described in any one of claims 1 to 7 when executing the computer program.
10. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data writing method as described in any one of claims 1 to 7.