Data lake-oriented data read-write method, device and equipment, medium and vehicle

Abstract

The invention relates to the technical field of data reading and writing, and discloses a data lake-oriented data reading and writing method, device and equipment, a medium and a vehicle, and the method comprises the steps: writing an obtained original message into a message queue in real time based on a log identifier and offset information of a current batch, and writing the original message into a data file of a data lake at the same time; double writing based on the message queue and the data file is adopted to complete data lake writing operation, so that the global consistency of the data is ensured. When the data is read, the data files are read in batches, and the message queue is read in real time to obtain the target original message stored in the data lake, so that the integrity and comprehensiveness of information acquisition can be ensured, and the real-time performance of data reading can also be ensured. According to the invention, double writing of data in the data lake is realized based on the message queue and the data file, and the data reading efficiency of the data lake is improved.

Description

Technical Field

[0001] The present invention relates to the field of data reading and writing technology, and in particular to a data lake-oriented data reading and writing method, device, equipment, medium and vehicle. Background Art

[0002] Data Lake (Iceberg) is a large data storage and processing system used to store and process data of various types and formats. It can handle massive amounts of data and provide high reliability and high scalability.

[0003] At present, when the existing technology processes related data through the data lake, it often matches the data file in the data lake file according to the corresponding relationship between the data file and the index, and locks the target data file. However, the above method only speeds up data retrieval, and the visibility of data lake data is mainly determined by the time of submitting snapshot files. In the existing technology, the data lake cannot submit snapshots at the second level. If the snapshot submission at the second level is forced, a large number of small files will be generated, which will affect the read and write efficiency of the table. Summary of the invention

[0004] In view of this, the present invention provides a data reading and writing method, device, equipment, medium and vehicle for a data lake to solve the problems in the prior art that snapshot files are submitted too slowly resulting in low data visibility, and submitted too quickly and easily resulting in a large number of small files affecting data reading and writing efficiency.

[0005] In a first aspect, the present invention provides a data lake-oriented data reading and writing method, the method comprising:

[0006] Record the log identifier of the current batch when data is written and the offset information of the message queue;

[0007] The original message is written into the message queue in real time based on the log identifier and offset information, and the original message is written into the data file of the data lake. When the preset data submission cycle is detected, a snapshot file is generated based on the offset information, log identifier and data file.

[0008] When reading data, according to the offset information, log identifier and data file corresponding to the snapshot file in the data lake, the data file is batch read to obtain the first storage data and the message queue is read in real time to obtain the second storage data;

[0009] According to the first stored data and the second stored data, a target original message stored in the data lake is obtained.

[0010] Beneficial effects: Based on the log identifier and offset information of the current batch, the original message obtained is written into the message queue in real time, and the original message is written into the data file of the data lake at the same time. The dual writing based on the message queue and the data file is adopted to complete the data lake writing operation, which is conducive to ensuring the global consistency of the data. When reading data, the data files are read in batches and the message queue is read in real time to obtain the target original message stored in the data lake, which can ensure the integrity and comprehensiveness of information acquisition and the real-time nature of data reading. The present invention realizes dual writing of data lake data based on message queues and data files, and improves the efficiency of data lake data reading.

[0011] In an optional implementation, according to the offset information, log identifier, and data file corresponding to the snapshot file in the data lake, batch reading of the data file is performed to obtain the first storage data, and real-time reading of the message queue is performed to obtain the second storage data, including:

[0012] Based on the log identifier corresponding to the snapshot file in the data lake, the data file, and the target log identifier when reading data, the corresponding target data file is batch read to obtain the first storage data stored in the target data file;

[0013] The current offset of the message queue is obtained according to the offset information, and the current offset is set as the start reading position of the message queue. The message queue is read in real time based on the start reading position to obtain the second storage data most recently stored in the message queue.

[0014] Beneficial effects: In the embodiments of the present invention, batch or full reading and writing of external data is completed based on data files, which does not generate a large number of small files and improves reading and writing efficiency. Real-time reading and writing of external data is completed based on message queues, so that data visibility can be achieved in seconds. Real-time reading and writing of the data lake is based on the current offset in the snapshot file, and data is read starting from the current offset to obtain complete data.

[0015] In an optional implementation, writing the acquired original message into the message queue in real time based on the log identifier and the offset information includes:

[0016] Message encoding is performed according to the log identifier and the obtained original message to obtain message data, and the message data is written into the message queue in real time;

[0017] When the arrival of the preset state synchronization cycle is detected, the initial offset and current offset of the message queue are obtained according to the offset information, and the state is encoded according to the initial offset, current offset and log identifier to obtain the state data, and the state data is written to the message queue in real time.

[0018] Beneficial effect: In an embodiment of the present invention, the original message is encoded and written into the message queue in real time. At the same time, when a preset state synchronization cycle arrives, the initial offset when the original message begins to be stored in the message queue, the current offset when the state synchronization cycle arrives, and the log identifier are state-encoded and stored in the message queue, so as to record the storage location and identifier of the message data in the message queue.

[0019] In an optional implementation, message encoding is performed according to the log identifier and the obtained original message to obtain message data, including:

[0020] Generate a message object based on the log identifier and the acquired original message; wherein the message object includes a first identifier attribute corresponding to the log identifier and a message attribute corresponding to the original message;

[0021] Serialize the message object into a byte array to get the message data.

[0022] Beneficial effect: In an embodiment of the present invention, when data is written in real time, a message object is formed according to the log identifier and the original message, thereby encoding new message data, which facilitates the message queue to store the original message in real time and subsequently read the message queue in real time.

[0023] In an optional implementation, status encoding is performed according to the initial offset, the current offset, and the log identifier to obtain status data, including:

[0024] Generate a state object based on the initial offset, the current offset and the log identifier; wherein the state object includes a start position attribute corresponding to the initial offset, an end position attribute corresponding to the current offset and a second identifier attribute corresponding to the log identifier;

[0025] Serialize the state object into a byte array to get the state data.

[0026] Beneficial effect: In an embodiment of the present invention, when the set state synchronization cycle arrives, the recorded initial offset, current offset and log identifier are encoded to generate state data so as to record the offset information and log identifier of the message data in the message queue.

[0027] In an optional implementation, obtaining a target original message stored in a data lake according to the first stored data and the second stored data includes:

[0028] Based on the second storage data, obtaining the latest message data and the latest status data;

[0029] Based on the latest status data, verify the validity of the latest message data;

[0030] When it is detected that the validity verification of the latest message data has passed, the target original message stored in the data lake is obtained according to the first stored data and the latest message data.

[0031] Beneficial effect: The present invention reads the latest status data and the latest message data in the message queue in real time. After the validity of the latest message data is verified, the original message stored in the data lake is completely read according to the first storage data read in batches in the snapshot file and the latest message data read in real time in the message queue. This ensures the real-time and integrity of data reading, and does not generate too many small files due to frequent submission of snapshot files, which will not affect the reading efficiency of the data lake.

[0032] In an optional implementation, based on the latest status data, the validity of the latest message data is verified, including:

[0033] Decode the latest message data to obtain the first log identifier and the latest offset;

[0034] Decoding the latest state data to obtain a second log identifier, a latest initial offset, and a latest current offset, and determining an offset range based on the latest initial offset and the latest current offset;

[0035] When it is detected that the first log identifier is consistent with the second log identifier and the latest offset is not within the offset range, it is determined that the validity verification of the latest message data has passed.

[0036] Beneficial effect: In the embodiment of the present invention, the validity of the latest message data in the message queue is verified by the latest status data. If the first log identifier and the second log identifier are not equal, it means that the message read in the message queue is inconsistent with the current read batch and needs to be discarded; if the latest offset of the latest message data already exists in the offset range of the status data record, it means that these data can be obtained by batch reading snapshot files and need to be discarded to avoid reading data duplication. The present invention can avoid reading duplicate invalid data and improve the accuracy and effectiveness of data reading.

[0037] In an optional embodiment, the method further includes:

[0038] Based on the first log identifier, the latest initial offset and the latest current offset, the log identifier and offset information of the current record are updated.

[0039] Beneficial effect: By updating the log identifier and offset information of the current record, the accuracy of data reading and writing is improved when data is read and written according to the log identifier and offset information.

[0040] In a second aspect, the present invention provides a data reading and writing device for a data lake, the device comprising:

[0041] The first processing module is used to record the log identifier of the current batch and the offset information of the message queue when data is written;

[0042] The second processing module is used to write the obtained original message into the message queue in real time based on the log identifier and offset information, and write the original message into the data file of the data lake at the same time, and generate a snapshot file according to the offset information, log identifier and data file when the preset data submission cycle is detected;

[0043] The third processing module is used to batch read the data files to obtain the first storage data and read the message queue in real time to obtain the second storage data according to the offset information, log identifier and data file corresponding to the snapshot file in the data lake when reading the data;

[0044] The fourth processing module is used to obtain the target original message stored in the data lake according to the first stored data and the second stored data.

[0045] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data reading and writing method for a data lake according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0046] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data reading and writing method for a data lake according to the first aspect or any corresponding embodiment thereof.

[0047] In a fifth aspect, the present invention provides a vehicle for executing the data reading and writing method for a data lake according to the first aspect or any corresponding embodiment thereof.

[0048] The beneficial effects of the present invention are:

[0049] Based on the log identifier and offset information of the current batch, the original message obtained is written into the message queue in real time, and the original message is written into the data file of the data lake at the same time. The dual writing based on the message queue and the data file is adopted to complete the data lake writing operation, which is conducive to ensuring the global consistency of the data. When reading data, the data files are read in batches and the message queue is read in real time to obtain the target original message stored in the data lake, which can ensure the integrity and comprehensiveness of information acquisition and the real-time nature of data reading. The present invention realizes dual writing of data lake data based on message queues and data files, and improves the efficiency of data lake data reading. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 is a flowchart of a data lake-oriented data reading and writing method according to an embodiment of the present invention;

[0052] Figure 2 is a flowchart of another data lake-oriented data reading and writing method according to an embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of a data lake writing process according to an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of a snapshot file submission process according to an embodiment of the present invention;

[0055] Figure 5 is a schematic diagram of a data lake reading process according to an embodiment of the present invention;

[0056] Figure 6 is a schematic diagram of a snapshot file batch reading process according to an embodiment of the present invention;

[0057] Figure 7 is a schematic diagram of a message data validity verification process according to an embodiment of the present invention;

[0058] Fig. 8A is a schematic diagram of data reading and writing efficiency according to an embodiment of the present invention;

[0059] Figure 8B is a schematic diagram of another data reading and writing efficiency according to an embodiment of the present invention;

[0060] Fig. 9 is a structural block diagram of a data reading and writing device for a data lake according to an embodiment of the present invention;

[0061] Fig.10 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0063] When the related technology processes data based on the data lake (Iceberg), it adds index information to the metadata file of the data table, and then establishes the correspondence between the data file and the index in Iceberg. According to the correspondence, the target data is indexed and matched to determine the target data file for data processing. However, the above method only speeds up data retrieval. Although the task writes data to Iceberg in real time, the visibility of the data is mainly determined by the time of submitting the snapshot. For the visibility of data at the second level, Iceberg cannot submit snapshots at this level. If the snapshot submission at the second level is forced, a large number of small files will be generated, which will affect the read and write efficiency of the table.

[0064] At present, the data lake's support for stream and batch integration is not particularly perfect, and the low latency requirements for real-time computing are not sufficiently supported, only supporting the minute level. In real-time statistical analysis of data, the data large screen system, instant analysis operation products, cockpit car real-time diagnosis system, scene arrangement, etc. all require relevant data to be received and processed in real time. However, the current data lake is only at the minute level, and the data visibility is poor, and the data calculation required for the second level above cannot be met. In addition, submitting snapshot files too quickly can easily lead to too many small files affecting read and write efficiency.

[0065] Therefore, an embodiment of the present invention provides a data reading and writing method for a data lake, which uses data dual writing based on message queues and data files to complete the reading and writing operations of data to Iceberg. Real-time reading and writing of external data is completed by message queues, so that data visibility can be achieved in seconds. Batch or full reading and writing of external data is completed based on data files, which will not generate a large number of small files, avoid affecting the reading and writing efficiency, and achieve global consistency of data through data dual writing.

[0066] According to an embodiment of the present invention, an embodiment of a data reading and writing method for a data lake is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0067] In this embodiment, a data lake-oriented data reading and writing method is provided, which can be used for a computer device or electronic device for data reading and writing, such as a computer, a tablet computer, etc. Figure 1 is a flow chart of a data lake-oriented data reading and writing method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0068] Step S101, recording the log identifier of the current batch and the offset information of the message queue when data is written.

[0069] Specifically, when data is written, metadata such as log identifiers and offset information are generated and recorded. The log identifier of each write batch can be generated periodically, and the offset information of the message queue, such as the initial offset and the current offset, is generated. The initial offset refers to the initial offset value before the current batch of data begins to be stored in the message queue, and the current offset refers to the actual offset of the message queue at the current moment.

[0070] It should be noted that the offset is a digital identifier in the message queue, which is used to indicate the position of a message in the message queue, and is usually an integer that increases from 0.

[0071] Step S102, based on the log identifier and offset information, the obtained original message is written into the message queue in real time, and the original message is written into the data file of the data lake. When the preset data submission cycle is detected, a snapshot file is generated according to the offset information, log identifier and data file.

[0072] Specifically, the received original message is written into the message queue and the data file at the same time. When writing into the message queue, the metadata such as the log identifier and offset information corresponding to the current batch needs to be written into the message queue synchronously with the original message to facilitate the subsequent reading operation of the original message in the message queue.

[0073] Specifically, the data submission cycle, that is, the snapshot file submission cycle, can be set and detected. When the data submission cycle arrives, the snapshot data is written into the snapshot file according to the offset, log identifier and all the information of the data file recorded at the current moment. It should be noted that the submission of the snapshot file is carried out according to the preset data submission cycle, which means that the snapshot file can only store a part of the original message. When the data submission cycle has not arrived, the latest original message will only be stored in the message queue. This part of the latest original message will only be triggered when the next data submission cycle arrives, along with the submission of the data file, and stored in the snapshot file.

[0074] In the embodiment of the present invention, in order to support real-time tasks with a delay of seconds, a dual-write Catalog based on a message queue and a data file is used to complete the data writing operation to Iceberg, and the global consistency of the data is achieved through the dual writing of the Catalog. It should be noted that the Catalog is composed of the metadata of a database instance, including basic tables, synonyms, indexes, users, etc., which is equivalent to the index of the database. The Catalog is usually used to describe the process of mapping the data written to the device to the database instance and mapping the database instance to the reading device.

[0075] Step S103, when reading data, according to the offset information, log identifier and data file corresponding to the snapshot file in the data lake, the data file is batch read to obtain the first storage data and the message queue is read in real time to obtain the second storage data.

[0076] Specifically, when reading data from the data lake, batch reading can be performed based on the data file to obtain the first storage data stored in the data file. In addition, the message queue can be read in real time according to the offset information of the snapshot file to obtain the second storage data most recently stored in the message queue.

[0077] In an embodiment of the present invention, batch or full reading and writing of external data is completed based on the Catalog operation of the data file, and a large number of small files will not be generated. Real-time reading and writing of external data is completed by Catalog operations based on message queues, so as to achieve second-level visibility of data. It should be noted that the real-time reading and writing of the data lake is based on the current offset in the snapshot file, and data is read from the current offset. In other words, the second storage data read by the real-time reading and writing does not exist in the data file corresponding to the snapshot file, and the purpose of obtaining complete data is achieved through real-time reading.

[0078] Step S104: Obtain the target original message stored in the data lake according to the first stored data and the second stored data.

[0079] The data reading and writing method for the data lake provided in this embodiment writes the acquired original message into the message queue in real time based on the log identifier and offset information of the current batch, and writes the original message into the data file of the data lake at the same time, and adopts dual writing based on the message queue and the data file to complete the data lake writing operation, which is conducive to ensuring the global consistency of the data. When reading data, the data file is read in batches and the message queue is read in real time to obtain the target original message stored in the data lake, which can ensure the integrity and comprehensiveness of information acquisition and the real-time nature of data reading. The present invention realizes dual writing of data lake data based on the message queue and the data file, and improves the efficiency of data lake data reading.

[0080] In this embodiment, a data lake-oriented data reading and writing method is provided, which can be used for a computer device or electronic device for data reading and writing, such as a computer, a tablet computer, etc. Figure 2 is a flow chart of a data lake-oriented data reading and writing method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0081] Step S201, record the log identifier of the current batch and the offset information of the message queue when the data is written. For details, please refer to Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0082] Step S202, based on the log identifier and offset information, the obtained original message is written into the message queue in real time, and the original message is written into the data file of the data lake. When the preset data submission cycle is detected, a snapshot file is generated according to the offset information, log identifier and data file.

[0083] Specifically, the above step S202 includes:

[0084] Step S2021, message encoding is performed according to the log identifier and the obtained original message to obtain message data, and the message data is written into the message queue in real time.

[0085] Specifically, based on the log identifier and the acquired original message, a message object is generated, wherein the message object includes a first identifier attribute corresponding to the log identifier and a message attribute corresponding to the original message. Then, the message object is serialized into a byte array to obtain message data.

[0086] In some optional embodiments, such as Figure 3 As shown, the log identifier generator and current offset recorder in the metadata synchronizer can be used to generate log identifiers, record the initial offset of the message queue, and the current offset and other metadata. The message object has two attributes: message and first identifier. The message encoder can be used to put the received original message into the message attribute of the message object, and the message encoder can be used to put the received log identifier into the first identifier attribute of the message object. Then the message encoder serializes the message object into a binary byte array to obtain the message data. When the message writer receives the message data generated by the message encoder, it sends it to the message queue in real time.

[0087] In an embodiment of the present invention, when data is written in real time, a message object is formed according to the log identifier and the original message, thereby encoding new message data to facilitate the message queue to store the original message in real time and subsequently read the message queue in real time.

[0088] Step S2022, when it is detected that the preset state synchronization cycle has arrived, the initial offset and current offset of the message queue are obtained according to the offset information, the state is encoded according to the initial offset, the current offset and the log identifier, the state data is obtained, and the state data is written into the message queue in real time.

[0089] Specifically, based on the initial offset, the current offset and the log identifier, a state object is generated, wherein the state object includes a start position attribute corresponding to the initial offset, an end position attribute corresponding to the current offset and a second identifier attribute corresponding to the log identifier. Then, the state object is serialized into a byte array to obtain state data.

[0090] In some optional embodiments, see again Figure 3 , the metadata synchronizer detects that the set state synchronization period has arrived and triggers state data synchronization. The state encoder collects the initial offset of the message queue generated by the metadata synchronizer, the current offset of the message queue calculated by the current offset recorder, and the log identifier of the current batch recorded by the log identifier generator. Then, these three pieces of information are used to form a state object and encode it into a new state.

[0091] In some optional implementations, the state object consists of three attributes: start position, end position, and second identifier. The state encoder puts the current offset of the message queue into the end position attribute, puts the initial offset of the message queue generated by the metadata synchronizer into the start position attribute, and puts the log identifier of the current batch recorded by the log identifier generator into the second identifier attribute. The state encoder serializes the formed state object into a binary byte array to obtain state data. The state writer sends the encoded state data to the message queue.

[0092] In the embodiment of the present invention, when the set state synchronization period arrives, the recorded initial offset, current offset and log identifier are encoded to generate state data so as to record the offset information and log identifier of the message data in the message queue.

[0093] In an embodiment of the present invention, the original message is encoded and written into the message queue in real time. At the same time, when a preset state synchronization cycle arrives, the initial offset when the original message begins to be stored in the message queue, the current offset when the state synchronization cycle arrives, and the log identifier are state-encoded and stored in the message queue, so as to record the storage location and identifier of the message data in the message queue.

[0094] Step S2023, write the original message into the data file of the data lake, and when it is detected that the preset data submission cycle has arrived, generate a snapshot file according to the offset information, log identifier and data file.

[0095] Specifically, see again Figure 3 , the original message received by the message encoder will be sent to the data writer at the same time, and the data writer will write the original message into the corresponding data file. Figure 4 As shown, when the metadata synchronizer detects that the data submission cycle has arrived, it triggers the data submitter to submit a snapshot. The data submitter needs to collect the offset information of the message queue recorded by the current offset recorder, the log identifier of the current batch generated by the log identifier generator, and the information of all Iceberg data files written by the data writer to form snapshot data and write it into the snapshot file.

[0096] In some optional implementations, the set data submission cycle can be consistent with the state synchronization cycle. The metadata synchronizer triggers the data submitter every time the state data synchronization is completed to submit the data file information currently written in real time. In this way, the corresponding message between the start position and the end position of the state data is consistent with the message recorded in the snapshot file submitted at the corresponding time point, ensuring the standardization of data storage.

[0097] Step S203, when reading data, according to the offset information, log identifier and data file corresponding to the snapshot file in the data lake, the data file is batch read to obtain the first storage data and the message queue is read in real time to obtain the second storage data.

[0098] Specifically, the above step S203 includes:

[0099] Step S2031: Based on the log identifier corresponding to the snapshot file in the data lake, the data file, and the target log identifier when reading data, the corresponding target data file is batch read to obtain the first storage data stored in the target data file.

[0100] Specifically, Figure 5 As shown, the data reader quickly locks the data file of the data to be read through the snapshot file. Figure 6 As shown, when using the data lake to read data, you can quickly find the corresponding snapshot metadata list file based on the snapshot ID (for example, S1), and then locate the specific snapshot file based on the query conditions and the snapshot metadata list file. Then, combined with the query conditions such as the target log identifier, you can locate the required target data file, thereby obtaining the message stored in the target data file. Figure 5 , the data reader reads the data file and sends the data to the subsequent operators to participate in the calculation.

[0101] Step S2032, obtaining the current offset of the message queue according to the offset information, and setting the current offset as the start reading position of the message queue, reading the message queue in real time based on the start reading position, and obtaining the second storage data most recently stored in the message queue.

[0102] Specifically, see again Figure 5 When the data reader reads the snapshot file, it will also retrieve the latest recorded log identifier and the current offset of the message queue. The message reader continues to read data from the message queue that is not in the data file by setting the current offset as the start reading position. For the latest message data recorded in the message queue, the metadata synchronizer has not yet started to submit the snapshot, and this part of the data will not appear in the data file. To read all the data, the data in the message queue must also be read out. When the message reader reads the data in the message queue, it needs to specify a start reading position as the start position for reading messages. From this position, messages in the message queue are read backwards. The current offset in the snapshot file is used as the start position required for reading the message queue this time.

[0103] In the embodiment of the present invention, batch or full reading and writing of external data is completed based on data files, which does not generate a large number of small files and improves reading and writing efficiency. Real-time reading and writing of external data is completed based on message queues, so that data visibility can be achieved in seconds. Real-time reading and writing of the data lake is based on the current offset in the snapshot file, and data is read starting from the current offset to obtain complete data.

[0104] Step S204: Obtain the target original message stored in the data lake according to the first stored data and the second stored data.

[0105] Specifically, the above step S204 includes:

[0106] Step S2041, based on the second stored data, obtain the latest message data and the latest status data.

[0107] Specifically, see again Figure 5 The message reader reads the data in the message queue. If it is state data, it is handed over to the state decoder. If it is message data, it is handed over to the message decoder, which is decoded by the state decoder and the message decoder respectively.

[0108] Step S2042: verify the validity of the latest message data based on the latest status data.

[0109] Specifically, the latest message data is decoded to obtain a first log identifier and a latest offset. The latest status data is decoded to obtain a second log identifier, a latest initial offset, and a latest current offset, and an offset range is determined based on the latest initial offset and the latest current offset. When it is detected that the first log identifier is consistent with the second log identifier and the latest offset is not within the offset range, it is determined that the validity verification of the latest message data has passed.

[0110] In some optional embodiments, see again Figure 5 , the state decoder deserializes the state data into a state object, and obtains the offset range and the second log identifier recorded in the state data. For message data, the message decoder deserializes the message data into a message object, and decodes to obtain the first log identifier and the latest offset corresponding to the latest message.

[0111] like Figure 7 As shown, the data determiner combines with the log identifier determiner to detect whether the first log identifier (i.e., the message record identifier) ​​and the second log identifier (i.e., the status record identifier) ​​are consistent, and combines with the offset determiner to detect whether the offset corresponding to the latest message data is within the offset range of the latest status data record. If the message record identifier and the status record identifier are not equal, it means that the message read in the message queue is inconsistent with the current read batch and needs to be discarded; if the offset of the latest message data already exists in the offset range of the status data record, it means that these data may have been stored in the snapshot file and submitted, and can be obtained by batch reading the snapshot file. In order to avoid duplication of reading data, it needs to be discarded. Only when the message record identifier and the status record identifier are equal, and the offset of the latest message data does not exist in the offset range of the status data record, it means that the latest message data is data that does not exist in the snapshot file obtained by real-time reading of the message queue, then it is judged that the data is available and the data is sent to the downstream.

[0112] In an embodiment of the present invention, the validity of the latest message data in the message queue is verified by the latest status data. If the first log identifier and the second log identifier are not equal, it means that the message read in the message queue is inconsistent with the current read batch and needs to be discarded; if the latest offset of the latest message data already exists in the offset range of the status data record, it means that these data can be obtained by batch reading snapshot files and need to be discarded to avoid duplication of reading data. The present invention can avoid reading duplicate invalid data and improve the accuracy and effectiveness of data reading.

[0113] In some optional implementations, the currently recorded log identifier and offset information may also be updated based on the first log identifier, the latest initial offset, and the latest current offset. This helps improve the accuracy of data reading and writing when data is read and written based on the log identifier and offset information.

[0114] Step S2043: When it is detected that the validity verification of the latest message data is passed, the target original message stored in the data lake is obtained according to the first stored data and the latest message data.

[0115] Specifically, based on the first storage data obtained by batch reading of the snapshot file, a portion of the original messages stored in the data lake can be obtained in batches, and then combined with the latest message data stored in the message queue in real time, the latest original message that does not exist in the snapshot file can be obtained. Based on the original message corresponding to the first storage data and the latest original message in the latest message data, the target original message of the current read batch can be obtained.

[0116] The present invention reads the latest status data and the latest message data in the message queue in real time. After the validity of the latest message data is verified, the original message stored in the data lake is completely read according to the first storage data read in batches in the snapshot file and the latest message data read in real time in the message queue. This ensures the real-time and integrity of data reading, and does not generate too many small files due to frequent submission of snapshot files, which does not affect the reading efficiency of the data lake.

[0117] The data lake-oriented data reading and writing method of the present invention is described in detail below with reference to a specific application example. This application example consists of two parts: real-time reading and writing and batch reading and writing. The real-time reading and writing part is responsible for real-time reading and writing of data, and the batch reading and writing part is responsible for periodically synchronizing data to the Iceberg table and updating the data snapshot identifier of the real-time reading and writing part.

[0118] See again Figure 3 The message encoder of the real-time read-write part is responsible for real-time encoding of messages and metadata, the message writer is responsible for writing the encoded message data into the message queue in real time, and the metadata synchronizer is responsible for periodically synchronizing metadata such as log identifiers and offsets to the message queue. The data writer of the batch read-write part is responsible for writing data to Iceberg's data files in real time, and the data submitter is responsible for submitting the writing results of the data writer to Iceberg.

[0119] See again Figure 5 The message reader of the real-time read / write part is responsible for reading the data in the message queue in real time, the message decoder is responsible for decoding the message data in real time, and the data judger is responsible for judging whether the decoded data is usable data. The data reader of the batch read / write part is responsible for reading the file data of Iceberg.

[0120] See again Figure 3 When the data lake catalog starts working, the metadata synchronizer generates a batch log identifier for the message, and will periodically generate new batch log identifiers for the message in the future. The message encoder encodes the generated message and adds the log identifier of the current message to form the message data to be sent to the message writer, and the message writer sends the message data to the message queue.

[0121] See again Figure 3 Before the metadata synchronizer generates a new log identifier each time, it needs to record the offset of the current message in the message queue so far, and form a meta message together with the current log identifier. The message encoder encodes the meta message to obtain status data for the status writer, and the status writer sends the status data to the message queue to generate a new data identifier.

[0122] See again Figure 5 The message decoder decodes the stored data in the message queue, and the decoded message is handed over to the data judge. The data judge determines whether the message is status data or message data. The status data records the log identifier and the offset of the message in the status data. The message data compares the offset and log identifier of this data with the log identifier and offset range of the most recent status data recorded to determine whether the message data is the required data.

[0123] See again Figure 3 The data writer writes the original message sent to the message encoder into the data file in real time. The metadata synchronizer triggers the data submitter every time it completes the synchronization of the state data, submitting the data file information currently being written in real time.

[0124] See again Figure 5 The data reader of the batch read-write part reads the file data of the current latest snapshot, and gives the latest log identifier and offset to the message reader of the real-time read-write part. The message reader of the real-time read-write part completes the data reading by completing the data that does not exist in the snapshot file and is in the message queue at that time.

[0125] Combine the following Figures 3 to 8B The real-time and batch writing processes of data are introduced in detail.

[0126] Step S11, the metadata synchronizer periodically generates a log identifier and an offset of the current message queue.

[0127] Step S12: the message encoder converts the received original message and log identifier into a message object, and then encodes and generates new message data.

[0128] Step S13: The message writer receives the message data generated by the message encoder and sends it to the message queue in real time.

[0129] Step S14: the original message received by the message encoder will be sent to the data writer at the same time, and the data writer will write the original message into the corresponding data file.

[0130] Step S15, the metadata synchronizer timing cycle arrives, triggering the state data synchronization. The state encoder collects the initial offset of the message queue generated by the metadata synchronizer, the current offset of the message queue calculated by the current offset recorder, and the log identifier of the current batch recorded by the log identifier generator, and then uses these three information to form a state object and encode it into new state data.

[0131] Step S16, the status writer sends the encoded status data to the message queue.

[0132] Step S17, the metadata synchronizer timing cycle arrives, triggering the data submitter to submit a snapshot. The data submitter needs to collect the offset information of the message queue recorded by the current offset recorder, the log identifier of the current batch generated by the log identifier generator, and the information of all Iceberg data files written by the data writer, form snapshot data and write it into the snapshot file.

[0133] Combine the following Figures 3 to 8B The real-time and batch reading processes of data are introduced in detail.

[0134] Step S21 : the data reader quickly locks the data file of the data to be read through the snapshot file.

[0135] See again Figure 6 As shown, when using the data lake to read data, you can quickly find the corresponding snapshot metadata list file based on the snapshot ID (for example, S1), and then locate the specific snapshot file based on the query conditions and the snapshot metadata list file. Combined with the query conditions such as the target log identifier, you can locate the required target data file and obtain the message stored in the target data file.

[0136] Step S22: The data reader reads the data file and sends the data to the following operator for calculation.

[0137] Step S23: When the data reader reads the snapshot file, it also retrieves the recorded log identifier and the current offset of the message.

[0138] Step S24: The message reader continues to read data that is not in the snapshot file in the message queue by setting the current offset as the offset for starting to read.

[0139] Step S25, the message reader reads the stored data in the message queue, and if it is state data, it is handed over to the state decoder, and if it is message data, it is handed over to the message decoder for decoding.

[0140] Step S26, according to the log identifier and current offset recorded in step S23, the log identifier in the state decoder and the offset of the message queue are initialized first.

[0141] Step S27: If the data in step S25 is state data, the state decoder decodes the state data and updates the current log identifier and the offset of the message queue.

[0142] Step S28: If the data in step S25 is message data, the message decoder decodes and obtains the original message, log identifier, and message offset.

[0143] Step S29: The data determiner determines whether the decoded data is available. If the data is available, the data is sent to the downstream, otherwise it is discarded.

[0144] The present invention can achieve data reading and writing within 1 second through the coordination of real-time reading and writing and batch reading and writing, and at the same time, there is no need to generate snapshots too frequently, thereby reducing the generation of small files. In combination with the small file merging function of Iceberg, small files can be merged regularly to ensure the stability of the number of small files.

[0145] like Fig. 8A As shown in the figure, when the Duration is 3 minutes and 46 seconds, the number of processed data is 2698657; Figure 8B As shown, when the duration is 3 minutes and 57 seconds, 2831516 data items are processed, which means that 132859 more data items are processed in 10 seconds. The present invention optimizes the reading and writing of the data lake based on the dual-write Catalog, which can accurately read and write data to the second level, improve the real-time data reading efficiency of the Iceberg table by nearly 100 times, and improve the writing efficiency of the Iceberg table by 60 times.

[0146] The present invention can not only support data calculation needs of more than one minute but also complete real-time calculation within 1 second, effectively supporting the real-time calculation of vehicle and application (map, navigation, etc.) activity status on the data large screen; it can well meet the impromptu analysis of operations and products, and can grasp the effect of activity delivery in real time; it can well support the real-time diagnosis of vehicles by the cockpit vehicle real-time diagnosis system, so that vehicle or application failures can be discovered in time within 1 second, and corresponding fault analysis can be performed; it can provide real-time scene usage analysis for scene arrangement, and timely understand the user's usage of the scene and feedback on the scene.

[0147] In this embodiment, a data reading and writing device for a data lake is also provided, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated here. As used below, the term "module" can implement a combination of software and / or hardware for a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0148] This embodiment provides a data reading and writing device for a data lake, such as Fig. 9 As shown, including:

[0149] The first processing module 901 is used to record the log identifier of the current batch and the offset information of the message queue when data is written;

[0150] The second processing module 902 is used to write the obtained original message into the message queue in real time based on the log identifier and the offset information, and write the original message into the data file of the data lake, and generate a snapshot file according to the offset information, the log identifier and the data file when the preset data submission cycle is detected.

[0151] The third processing module 903 is used to read the data files in batches to obtain the first storage data and read the message queue in real time to obtain the second storage data according to the offset information, log identifier and data file corresponding to the snapshot file in the data lake when reading the data;

[0152] The fourth processing module 904 is used to obtain the target original message stored in the data lake according to the first stored data and the second stored data.

[0153] In some optional implementations, the second processing module 902 is further configured to:

[0154] Message encoding is performed according to the log identifier and the obtained original message to obtain message data, and the message data is written into the message queue in real time;

[0155] When the arrival of the preset state synchronization cycle is detected, the initial offset and current offset of the message queue are obtained according to the offset information, and the state is encoded according to the initial offset, current offset and log identifier to obtain the state data, and the state data is written to the message queue in real time.

[0156] In some optional implementations, the second processing module 902 is further configured to:

[0157] Generate a message object based on the log identifier and the acquired original message; wherein the message object includes a first identifier attribute corresponding to the log identifier and a message attribute corresponding to the original message;

[0158] Serialize the message object into a byte array to get the message data.

[0159] In some optional implementations, the second processing module 902 is further configured to:

[0160] Generate a state object based on the initial offset, the current offset and the log identifier; wherein the state object includes a start position attribute corresponding to the initial offset, an end position attribute corresponding to the current offset and a second identifier attribute corresponding to the log identifier;

[0161] Serialize the state object into a byte array to get the state data.

[0162] In some optional implementations, the third processing module 903 is further configured to:

[0163] Based on the log identifier corresponding to the snapshot file in the data lake, the data file, and the target log identifier when reading data, the corresponding target data file is batch read to obtain the first storage data stored in the target data file;

[0164] The current offset of the message queue is obtained according to the offset information, and the current offset is set as the start reading position of the message queue. The message queue is read in real time based on the start reading position to obtain the second storage data most recently stored in the message queue.

[0165] In some optional implementations, the fourth processing module 904 is further configured to:

[0166] Based on the second storage data, obtain the latest message data and the latest status data;

[0167] Based on the latest status data, verify the validity of the latest message data;

[0168] When it is detected that the validity verification of the latest message data has passed, the target original message stored in the data lake is obtained according to the first stored data and the latest message data.

[0169] In some optional implementations, the fourth processing module 904 is further configured to:

[0170] Decode the latest message data to obtain the first log identifier and the latest offset;

[0171] Decoding the latest state data to obtain a second log identifier, a latest initial offset, and a latest current offset, and determining an offset range based on the latest initial offset and the latest current offset;

[0172] When it is detected that the first log identifier is consistent with the second log identifier and the latest offset is not within the offset range, it is determined that the validity verification of the latest message data has passed.

[0173] In some optional embodiments, the device is also used for:

[0174] Based on the first log identifier, the latest initial offset and the latest current offset, the log identifier and offset information of the current record are updated.

[0175] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0176] The data reading and writing device for the data lake in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0177] The embodiment of the present invention also provides a computer device having the above Fig. 9 The data reading and writing device for the data lake is shown. In addition, the computer device can be installed in the vehicle to read and write data based on the data lake, providing data support for functions such as real-time diagnosis of vehicle faults.

[0178] See also Fig.10 , Fig.10 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig.10 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.10 A processor 10 is taken as an example.

[0179] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0180] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0181] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0182] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0183] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.10 The example of connecting through bus is taken in the following.

[0184] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0185] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0186] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data reading and writing method for a data lake, characterized in that: The method comprises: Record the log identifier of the current batch when data is written and the offset information of the message queue; The original message obtained is written into the message queue in real time based on the log identifier and the offset information, and the original message is written into the data file of the data lake at the same time, and when the arrival of the preset data submission cycle is detected, a snapshot file is generated according to the offset information, the log identifier and the data file; When reading data, according to the offset information, log identifier and data file corresponding to the snapshot file in the data lake, the data file is batch read to obtain the first storage data and the message queue is read in real time to obtain the second storage data; According to the first stored data and the second stored data, a target original message stored in the data lake is obtained.

2. The method according to claim 1, characterized in that The method of batch reading the data files to obtain first storage data and reading the message queue in real time to obtain second storage data according to the offset information, log identifier, and data files corresponding to the snapshot files in the data lake includes: Based on the log identifier corresponding to the snapshot file in the data lake, the data file, and the target log identifier when reading data, the corresponding target data file is batch read to obtain the first storage data stored in the target data file; The current offset of the message queue is obtained according to the offset information, and the current offset is set as the start reading position of the message queue. The message queue is read in real time based on the start reading position to obtain the second storage data most recently stored in the message queue.

3. The method according to claim 2, characterized in that The step of writing the acquired original message into the message queue in real time based on the log identifier and the offset information includes: Message encoding is performed according to the log identifier and the acquired original message to obtain message data, and the message data is written into the message queue in real time; When it is detected that the preset state synchronization cycle has arrived, the initial offset and current offset of the message queue are obtained according to the offset information, state encoding is performed according to the initial offset, current offset and log identifier to obtain state data, and the state data is written into the message queue in real time.

4. The method according to claim 3, characterized in that The message encoding is performed according to the log identifier and the obtained original message to obtain the message data, including: Generate a message object based on the log identifier and the acquired original message; wherein the message object includes a first identifier attribute corresponding to the log identifier and a message attribute corresponding to the original message; The message object is serialized into a byte array to obtain message data.

5. The method according to claim 3, characterized in that: The state encoding is performed according to the initial offset, the current offset and the log identifier to obtain the state data, including: Generate a state object based on the initial offset, the current offset and the log identifier; wherein the state object includes a start position attribute corresponding to the initial offset, an end position attribute corresponding to the current offset and a second identifier attribute corresponding to the log identifier; The state object is serialized into a byte array to obtain state data.

6. The method according to any one of claims 3 to 5, characterized in that: The step of obtaining the target original message stored in the data lake according to the first stored data and the second stored data includes: Based on the second stored data, obtaining the latest message data and the latest status data; Based on the latest status data, verify the validity of the latest message data; When it is detected that the validity verification of the latest message data has passed, the target original message stored in the data lake is obtained according to the first stored data and the latest message data.

7. The method according to claim 6, characterized in that The verifying the validity of the latest message data based on the latest status data includes: Decoding the latest message data to obtain a first log identifier and a latest offset; Decoding the latest status data to obtain a second log identifier, a latest initial offset, and a latest current offset, and determining an offset range based on the latest initial offset and the latest current offset; When it is detected that the first log identifier is consistent with the second log identifier and the latest offset is not within the offset range, it is determined that the validity verification of the latest message data has passed.

8. The method according to any one of claims 6, characterized in that The method further comprises: Based on the first log identifier, the latest initial offset and the latest current offset, the log identifier and offset information of the current record are updated.

9. A data reading and writing device for a data lake, characterized in that: The device comprises: The first processing module is used to record the log identifier of the current batch and the offset information of the message queue when data is written; The second processing module is used to write the obtained original message into the message queue in real time based on the log identifier and the offset information, and write the original message into the data file of the data lake at the same time, and generate a snapshot file according to the offset information, the log identifier and the data file when the preset data submission cycle is detected to have arrived; A third processing module is used to, when reading data, batch read the data files to obtain first storage data and read the message queue in real time to obtain second storage data according to the offset information, log identifier and data file corresponding to the snapshot file in the data lake; The fourth processing module is used to obtain the target original message stored in the data lake according to the first stored data and the second stored data.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data reading and writing method for a data lake according to any one of claims 1 to 8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data reading and writing method for a data lake according to any one of claims 1 to 8.

12. A vehicle, characterized in that: The vehicle is used to execute the data reading and writing method for a data lake according to any one of claims 1 to 8.

Images