Data storage method and system
By serializing and diversion of source data, and using target metadata analysis and conversion, the problem of low efficiency of metadata changes and multiple tables entering the lake in the data into the lake system is solved, and the accuracy and efficiency of data storage are achieved.
Patent Information
- Application Number
- PCT/CN2024/090132
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-25
- Filing Date
- 2024-04-26
- Publication Date
- 2024-11-28
AI Technical Summary
The existing data entry system cannot support metadata changes when data is stored, resulting in the inability to ensure that data is written to the target storage location in its original state during data synchronization. The overhead of multiple table entry into the lake is high and the efficiency is low.
By reading source data for serialization, diversion into initial operation data stream and defining data stream, the initial operation data stream is analyzed and converted using target metadata to generate a sequence of stored information to ensure that data is written to the target data storage in the correct format. space.
It realizes metadata change support when data storage, improves the accuracy and efficiency of data writing, and ensures that data is stored in the target location in a state that meets the needs.
Smart Images

Figure CN2024090132_28112024_PF_FP_ABST
Abstract
Description
Data storage method and system
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 25, 2023, with application number 202310606285.2 and invention name “Data Storage Method and System”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The embodiments of this specification relate to the field of data storage technology, and more particularly to a data storage method and system. Background Art
[0003] With the development of Internet technology, various services will generate a large amount of data during operation. In order to support data persistence, data ingestion technology is usually combined to store data of different structures. Data ingestion refers to the process of storing data of various types and formats in a data lake. Data lakes usually use distributed storage and computing technologies, can store massive amounts of data, and support data acquisition from different data sources. In other words, data ingestion does not require pre-defined data models and can accommodate structured, semi-structured, and unstructured data, thus being able to adapt to various data analysis and application needs. In related technologies, when performing data ingestion processing, most of them will adopt batch processing or streaming processing; however, due to the relatively complex upstream interfaces in the data ingestion system, the overhead of multiple tables ingesting the lake is large, and most data ingestion systems do not pay attention to data change operations, which leads to the failure to support metadata changes during data synchronization and the inability to ensure that the data can be written to the target storage location in its original state. Therefore, an effective solution is urgently needed to solve the above problems.
[0004] Summary of the Invention
[0005] In view of this, embodiments of this specification provide a data storage method. One or more embodiments of this specification also relate to a data storage device, a data storage system, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in related technologies.
[0006] According to a first aspect of an embodiment of this specification, a data storage method is provided, comprising: reading source data in response to a data storage request, and serializing the source data; splitting the serialized source data into an initial operation data stream and a definition data stream, and determining target metadata corresponding to the initial operation data stream based on the definition data stream, wherein the initial operation data stream is constructed based on original data, and the definition data stream is constructed based on operation data that changes the original data; parsing data contained in the initial operation data stream using the target metadata, and determining a target operation data stream based on the parsing result; determining that the initial operation data stream is parsed into a parsed information sequence corresponding to the target operation data stream, and generating a storage information sequence based on the parsed information sequence; and storing the data contained in the target operation data stream in a target data storage space corresponding to the data operation request according to the storage information sequence.
[0007] According to a second aspect of an embodiment of this specification, a data storage device is provided, comprising: a reading module configured to read source data in response to a data storage request and serialize the source data; a determining module configured to split the serialized source data into an initial operation data stream and a definition data stream, and determine target metadata corresponding to the initial operation data stream according to the definition data stream, wherein the initial operation data stream is constructed based on original data, and the definition data stream is constructed based on operation data that changes the original data; a parsing module configured to parse data contained in the initial operation data stream using the target metadata, and determine a target operation data stream according to the parsing result; a generating module configured to determine that the initial operation data stream is parsed into a parsed information sequence corresponding to the target operation data stream, and generate a storage information sequence according to the parsed information sequence; and a storing module configured to store the data contained in the target operation data stream in a target data storage space corresponding to the data operation request according to the storage information sequence.
[0008] According to a third aspect of an embodiment of this specification, a data storage system is provided, comprising: a server and a target data storage end; the server is used to store data storage executable instructions, the target data storage end is used to store data, and the steps of the data storage method are implemented when the storage executable instructions are executed by the server.
[0009] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to implement any step of the above-mentioned data storage method when executing the computer-executable instructions.
[0010] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned data storage method are implemented.
[0011] According to a sixth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data storage method. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] FIG1 is a schematic diagram of a data storage method provided by one embodiment of this specification;
[0013] FIG2 is a flow chart of a data storage method provided by one embodiment of this specification;
[0014] FIG3 is a schematic diagram of a data storage link in a data storage method provided by one embodiment of this specification;
[0015] FIG4 is a schematic diagram of data splitting in a data storage method provided by one embodiment of this specification;
[0016] FIG5 is a schematic diagram of data parsing in a data storage method provided by one embodiment of this specification;
[0017] FIG6 is a schematic diagram of data writing in a data storage method provided by one embodiment of this specification;
[0018] FIG7 is a flowchart of a data storage method according to an embodiment of the present disclosure;
[0019] FIG8 is a schematic structural diagram of a data storage device provided by one embodiment of this specification;
[0020] FIG9 is a structural block diagram of a computing device provided by one embodiment of this specification. DETAILED DESCRIPTION
[0021] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.
[0022] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.
[0023] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0024] First, the terms involved in one or more embodiments of this specification are explained.
[0025] Data lake: A system for storing data of various types and formats. Data lakes are typically highly scalable, flexible, and cost-effective. Unlike traditional data warehouses, data lakes do not require predefined data schemas and can accommodate structured, semi-structured, and unstructured data, as well as data from diverse data sources such as sensors, logs, and social media.
[0026] Restructuring: This refers to the process of modifying or adjusting the structure of data stored in a data lake. Data lakes are typically used to store a variety of data types, including structured, unstructured, and semi-structured data. Restructuring can include adding, deleting, modifying, or reorganizing the structure of data stored in a data lake.
[0027] Watermark mechanism: The Watermark mechanism in stream computing is a time management method used to handle out-of-order events in infinite data streams. A watermark is a pluggable time marker that represents the maximum timestamp of events that have been processed in the data stream (or the upper limit of the event time). When processing data streams, the stream processing engine will use the watermark to determine which events have arrived or have been delayed to a certain extent, so as to determine whether to trigger the corresponding computing operations or delay waiting. The main function of the watermark mechanism is to ensure the correctness and real-time performance of data processing, and can effectively solve the problems caused by out-of-order events, such as repeated calculations, lost calculations, etc. The watermark mechanism usually needs to be adjusted and optimized according to the specific service scenarios and data characteristics, such as setting the appropriate window size, tolerated delay time, etc.
[0028] Flink is an open-source stream processing framework. Its core is a distributed streaming data flow engine written in Java and Scala. Flink executes arbitrary streaming programs in a data-parallel, pipelined manner. Flink's pipeline runtime system can execute both batch and stream processing programs. Furthermore, the Flink runtime itself supports iterative algorithms.
[0029] Spark: A fast, general-purpose computing engine designed for large-scale data processing. Implemented in Scala, it uses Scala as its application framework. Spark enables in-memory distributed datasets and, in addition to providing interactive queries, optimizes iterative workloads. It can be used to build large-scale, low-latency data analytics applications.
[0030] In this specification, a data storage method is provided. This specification also relates to a data storage device, a data storage system, a computing device, a computer-readable storage medium and a computer program, which are described in detail one by one in the following embodiments.
[0031] Referring to the schematic diagram shown in Figure 1, the data storage method provided in this embodiment can solve the problem that metadata cannot be changed when storing source data, that is, when the original data stored at the source cannot be changed according to the change requirements corresponding to the metadata, after receiving the data storage request, the source data can be read from the source, and the source data can be deserialized and then diverted to obtain the initial operation data stream and the definition data stream; at this time, in order to be able to realize the metadata change trigger during data storage, the data in the data stream can be processed in time, the target metadata corresponding to the initial operation data stream can be determined according to the definition data stream, and then the target metadata can be used to iteratively parse the data contained in the initial operation data stream, so as to obtain the target operation data stream corresponding to the target data storage space according to the parsing result. On this basis, in order to ensure that data can be written to the target data storage space quickly and accurately, it is possible to first determine that the initial operation data stream is parsed into the parsing information sequence corresponding to the target operation data stream, so as to generate a storage information sequence for storing data based on the parsing information sequence; finally, the data contained in the target operation data stream can be stored in the target data storage space corresponding to the data operation request according to the storage information sequence; during the data storage process, the target metadata can be determined according to the defined data stream, and iterative parsing and storage processing can be performed based on this to achieve accuracy and efficiency in data storage, so as to support data change requirements during data storage, so that the data finally stored in the target data storage space is the target data that the data storage party needs to store.
[0032] FIG2 shows a flow chart of a data storage method provided according to an embodiment of the present specification, which specifically includes the following steps.
[0033] Step S202 : reading source data in response to a data storage request, and performing serialization processing on the source data.
[0034] The data storage method provided in this embodiment is used to read data from a relational database and write it to the target data storage space corresponding to the data lake system. It is used to realize dynamic changes in data structure when the data in the relational database is stored in the target data storage space corresponding to the data lake system without relying on external components, thereby ensuring data correctness and ensuring that data can be efficiently entered into the lake.
[0035] Specifically, a data storage request refers to a request triggered when writing data from a source end to a target data storage space. This request can be triggered based on a data storage task. For example, a data storage task is set to store data once every set time. When the set time is reached, a data storage request can be triggered. In addition, a data storage request can also be triggered by an operator, and this embodiment does not impose any restrictions on this. Accordingly, source data refers to data read from a source end, which can be subsequently written to a target data storage space in a streaming manner in response to a data storage request. Accordingly, serialization processing refers to converting source data into an operation that can be used for subsequent diversion and storage.
[0036] Based on this, after receiving a data storage request, the source data can be read from the source end first, and then the source data can be serialized and split in response to the data storage request to obtain the initial operation data stream and the definition data stream; at this time, in order to be able to process the data in the data stream in a timely manner after the metadata change is triggered during data storage, the target metadata corresponding to the initial operation data stream can be determined according to the definition data stream, and then the data contained in the initial operation data stream can be parsed using the target metadata, so as to obtain the target operation data stream corresponding to the target data storage space according to the parsing result. On this basis, in order to ensure that the data can be written to the target data storage space quickly and accurately, the initial operation data stream can be parsed into the parsing information sequence corresponding to the target operation data stream, so as to generate a storage information sequence for storing data according to the parsing information sequence; finally, the data contained in the target operation data stream can be stored in the target data storage space corresponding to the data operation request according to the storage information sequence; in the data storage process, the target metadata can be determined according to the definition data stream, and parsed and stored based on this, so as to achieve accuracy and efficiency of data storage, so as to support the data change requirements during data storage, so that the data finally stored in the target data storage space is the target data that the data storage party needs to store.
[0037] In practical applications, in order to complete storage processing in different data storage scenarios through the data storage method provided in this embodiment, processing can be performed on data tables in different relational databases. For example, referring to the schematic diagram shown in (a) of Figure 3, for single-table synchronization logic, the Source (source data reading operator) operator can first concurrently read the data source, including full data reading and incremental data reading; then the Deserialize (deserialize operator) operator deserializes and splits the data read from the data source to obtain different sub-stream DML and DDL; at this time, the TypeConverter (format conversion operator) operator is used to convert the data format from the intermediate format of the stream computing engine to the format supported by the target big data system based on the dynamic schema; the Watermark Assigner operator is then used to check each record of the data stream and detect schema changes at the same time, so as to generate a watermark based on the detection results and pass it to the downstream; finally, the StreamWrite (stream writing operator) operator can be used to stream the data into the target big data system based on the dynamic schema.
[0038] On the other hand, referring to the schematic diagram shown in (b) of Figure 3, the multi-table synchronization logic is similar to the single-table synchronization logic described above. The difference is that when deserializing and diverting data through the Deserialize operator, multiple data writing tasks will be generated accordingly. The subsequent processing logic of each data writing task is the same as described above, so that multiple tables can complete data synchronization processing concurrently. In addition, referring to the schematic diagram shown in (c) of Figure 3, for the single-table synchronization physical scenario, different Source operators are actually used to complete source data reading at the initial stage, and then different Deserialize operators are used to complete deserialization and diversion. Thereafter, different TypeConverter operators are used to achieve format conversion. When watermarking, it can be achieved through the same Watermark Assigner operator. Finally, different StreamWrite operators are used to stream data into the target big data system based on the dynamic schema.
[0039] In one embodiment, when reading source data, considering that different source terminals correspond to different data access rules, in order to achieve docking with different source terminals and efficiently complete the operation of storing data in the target data storage space, data can be read in a concurrent manner; in this embodiment, the specific implementation is as follows:
[0040] Receive a data storage request submitted for a source data storage space; concurrently read full data or incremental data in the source data storage space according to the data storage request; and use the full data or the incremental data as the source data.
[0041] In one embodiment, the source data storage space specifically refers to the source end when importing data into the target data storage space; correspondingly, the full data specifically refers to all data read from the source data storage space; correspondingly, the incremental data specifically refers to only reading part of the data that has not yet been entered into the lake.
[0042] Based on this, after receiving the data storage request submitted to the source data storage space, it means that the data stored on the source end needs to be migrated to the target data storage space. In order to ensure data storage efficiency, the full data or incremental data can be read concurrently in the source data storage space according to the data storage request; the full data or incremental data can be used as the source data.
[0043] In actual applications, concurrent reading of full data or incremental data from the source data end can be achieved through the Source (source data reading operator), and this operator is equipped with a streaming and batch integrated interface to achieve concurrent reading of data and complete data storage operations.
[0044] In summary, by triggering the data storage mechanism by concurrently reading data, the high-concurrency data processing capability of the data storage system can be improved to ensure that data can be written to the target data storage space accurately and efficiently.
[0045] In addition, in order to achieve higher applicability of the above data storage process and avoid changes to the original data lake system, a blocking data writing method can be used to complete it. In this embodiment, the specific implementation method is as follows:
[0046] In response to the data storage request, the data reading link corresponding to the source data storage space is adjusted to a reading state; after executing the step of using the full data or the incremental data as the source data, it also includes: adjusting the data reading link corresponding to the source data storage space to a blocking state.
[0047] Specifically, the data read link refers to a link for accessing the data storage space, and the link corresponds to multiple states, such as a read state and a blocked state. The read state is when the data storage space can be accessed, and the blocked state is when the access interface is closed.
[0048] Based on this, in order to avoid frequent data changes and consume more computing resources, the data reading link corresponding to the source data storage space can be adjusted to a reading state in response to a data storage request; and after using the full data or incremental data as the source data, the data reading link corresponding to the source data storage space can be adjusted to a blocking state to achieve dynamic adjustment of the data reading link state to meet data storage needs.
[0049] In actual applications, after reading the structure change DDL from the source, the link can be blocked to stop sending subsequent data to the downstream. After the downstream write is completed, the external scheduling component persists the state, identifies the latest structure of the library table, updates the link topology, and starts the task from the snapshot. This allows the data storage operation to be completed by only modifying the topology of the upstream data source reading operator and leaving the downstream writing operator unchanged. In other words, in scenarios where real-time data processing is not required, blocking the link can be used to implement data storage operations.
[0050] In summary, by adjusting the data reading link status to achieve data reading and blocking, it is possible to avoid introducing excessive data change requirements, thereby affecting data storage efficiency.
[0051] Step S204: split the serialized source data into an initial operation data stream and a definition data stream, and determine the target metadata corresponding to the initial operation data stream based on the definition data stream, wherein the initial operation data stream is constructed based on the original data, and the definition data stream is constructed based on the operation data that changes the original data.
[0052] Specifically, after the source data is read and serialized as described above, considering that the data read from the source end contains not only the original data, but may also contain the operation data used when changing the data structure, and the operation data is the basis for changing the data structure, after the source data is read and serialized, the original data and the operation data need to be separated to ensure that when the source data is streamed to the target data storage space, the operation data can be used to change the data structure, thereby supporting the data lake system to adjust the final write structure of the data according to the data change requirements when streaming data to meet the storage requirements of the storage demand party to which the data storage request belongs. Therefore, after serialization, the serialized source data can also be separated into the initial operation data stream and the definition data stream, and then the target metadata corresponding to the initial operation data stream is determined according to the definition data stream, so that when the data structure is changed subsequently, it can be completed in combination with the target metadata determined according to the definition data stream.
[0053] Based on this, the initial operation data stream specifically refers to the data stream composed of data corresponding to DML (Data Manipulation Language). Correspondingly, the definition data stream specifically refers to the data stream composed of data corresponding to DDL (Data Definition Language). The definition data stream is used for data modifications, while the initial operation data stream is used for streaming storage to the target data storage space. In other words, the initial operation data stream is constructed based on the original data in the source data, while the definition data stream is constructed based on the operational data that is modified from the original data, to enable subsequent data entry into the lake. Correspondingly, the target metadata specifically refers to the metadata used to parse the data contained in the initial operation data stream. It is used to convert the data in the initial operation data stream into a format supported by the target data storage space, and this conversion can be completed dynamically. That is, the target metadata can be understood as a schema, which is used to record the declaration of elements and attributes and the definition of complex and simple data types. When streaming data in the initial data stream, structural changes and storage can be performed according to the target metadata, thereby ensuring that the stored data meets the user's storage requirements.
[0054] In one embodiment, when performing data serialization processing, in order to achieve efficient data storage, the initial data format can be converted into an intermediate data format; that is, according to a preset format conversion strategy, the source data in the initial data format is converted into intermediate data in the intermediate data format; and when performing data diversion on this basis, label allocation can be used. In this embodiment, the specific implementation method is as follows:
[0055] Determine the data attribute information corresponding to the intermediate sub-data contained in the intermediate data, and assign labels to the intermediate sub-data contained in the intermediate data according to the data attribute information; perform diversion processing on the intermediate sub-data contained in the intermediate data according to the label assignment result to obtain the initial operation data flow and the definition data flow.
[0056] Specifically, intermediate sub-data refers to the smallest unit of data contained in the intermediate data; correspondingly, data attribute information refers to the attribute description information corresponding to the intermediate sub-data, which is used to determine the data type and thus assign different labels; for example, data attribute information can be the writing language, data format, data field type, etc. corresponding to the intermediate sub-data; correspondingly, the data type is the data type corresponding to the DDL or the data type corresponding to the DML.
[0057] Based on this, in order to be able to accurately divert the source data in the future, the data attribute information corresponding to the intermediate sub-data contained in the intermediate data can be determined, and then labels can be assigned to the intermediate sub-data contained in the intermediate data according to the data attribute information; so as to assign different labels to different intermediate data, and finally, the intermediate sub-data contained in the intermediate data can be diverted according to the label assignment results to obtain the initial operation data flow and the definition data flow.
[0058] In actual applications, when the stream computing engine is Flink and the data lake is Hudi, the deserialization and diversion of source data can be achieved through the Deserialize operator preset in the system; that is, after reading the source data, the Deserialize operator can be used to convert the read data format (Binlog) into an intermediate format (RowData); DML data and DDL data can be converted into a unified format. At this time, different labels can be added to data in different formats, such as DB1.TB1.DML, DB1.TB1.DDL, and then according to Tags are used for data flow diversion, where DB1 represents the identifier corresponding to the data source, TB1 represents the unit identifier corresponding to the data, and DML and DDL represent the data types. As shown in the schematic diagram in Figure 4 (a), DML data and DDL data are divided into DML sub-flows and DDL sub-flows to facilitate the use of downstream operators. Each record in the DML data flow will only be forwarded to a single downstream operator, while each record in the DDL data flow can be broadcast to all related downstream operators. These related operators refer to operators that rely on the dynamic schema for format conversion, partition allocation, and data writing. All these operators will receive DDL broadcasts.
[0059] In addition, it should be noted that different stream computing engines are limited by the stream computing engines themselves when converting the read source data format into an intermediate format. For example, if the stream computing engine is Spark, then under this stream computing engine, the intermediate format of the data is RDD (resilient distributed dataset). In actual applications, the data conversion corresponding to different stream computing engines can be set according to actual needs, and this embodiment does not impose too many restrictions here.
[0060] In summary, by using labels for diversion processing, it is possible to complete data diversion by identifying labels after label allocation is completed, so as to accurately obtain the initial operation data flow and definition data flow, making it convenient to use the more accurate initial operation data flow and definition data flow for data storage and processing in the future.
[0061] In one embodiment, when determining the target metadata corresponding to the initial operation data stream, the process is actually to update each initial sub-metadata corresponding to the initial operation data stream using the definition data stream. In this embodiment, the specific implementation is as follows:
[0062] Acquire at least one initial sub-metadata corresponding to the initial operation data stream; update the at least one initial sub-metadata using the definition data stream to obtain at least one target sub-metadata; and generate target metadata corresponding to the initial operation data stream based on the at least one target sub-metadata.
[0063] Specifically, the initial sub-metadata refers to the data used when performing structural change processing on different batches of data contained in the initial operation data stream. Since the initial operation data stream is a streaming data structure, different batches of data can be parsed using different metadata according to the structural change requirements. Therefore, it is necessary to first read the initial sub-metadata corresponding to different batches of data from the source data to facilitate the subsequent update of the target sub-metadata according to the structural change requirements.
[0064] Based on this, considering that the data contained in the initial operation data stream is written and written in a streaming manner, different batches of data will correspond to different metadata. Therefore, in order to ensure the accuracy of data parsing in the future, at least one initial sub-metadata corresponding to the initial operation data stream can be obtained from the source data first; and the at least one initial sub-metadata is updated using the defined data stream to obtain at least one target sub-metadata; finally, the target metadata corresponding to the initial operation data stream is generated based on the at least one target sub-metadata. In specific implementation, when the initial sub-metadata is updated using the defined data stream, the initial sub-metadata corresponding to the same batch of data is actually first found, and the operation data associated with the batch of data is determined in the defined data stream, and the initial sub-metadata is updated based on the operation data, so that the target sub-metadata corresponding to the initial sub-metadata can be obtained; and in the update process, the metadata information that needs to be changed in the initial sub-metadata is adjusted using the operation data, so that the target metadata information is obtained to form the target sub-metadata.
[0065] In addition, in order to facilitate the management of metadata, the metadata management terminal can also maintain it separately, so that metadata can be loaded and used in a request manner during the application phase. In this embodiment, the specific implementation is as follows:
[0066] Generate a metadata loading request according to the defined data stream, and send the metadata loading request to the metadata management end corresponding to the target data storage space; receive the initial metadata fed back by the metadata management end in response to the metadata loading request, and determine the target metadata corresponding to the initial operation data stream based on the initial metadata.
[0067] Specifically, a load request is a request for the metadata management end to return the target metadata. The metadata management end, in turn, is a separate end that maintains metadata. It is deployed outside the data ingestion system to avoid metadata loss in the event of a downtime, which could lead to inaccurate data ingestion.
[0068] Based on this, in order to facilitate the management of metadata, it can be completed independently by the metadata management end and read from the metadata management end when needed. That is to say, a metadata loading request can be generated according to the defined data stream, and the metadata loading request can be sent to the metadata management end corresponding to the target data storage space; after receiving the initial metadata feedback from the metadata management end for the metadata loading request, the target metadata corresponding to the initial operation data stream can be determined based on the initial metadata to facilitate the use of downstream services.
[0069] In practical applications, schema management can be performed using the metadata management client. By separating data schemas from data processing tasks, unified management and evolution of data schemas are achieved. In this structure, all metadata can be stored in the metadata management client, and data processing tasks can obtain metadata by querying registration information, enabling schema reuse and management.
[0070] In summary, by managing metadata on the metadata management side, it is possible to ensure that data processing and data storage are maintained separately, thereby saving computing resources consumed during data storage and effectively improving data storage efficiency.
[0071] Step S206: parse the data contained in the initial operation data stream using the target metadata, and determine the target operation data stream according to the parsing result.
[0072] In one embodiment, after the target metadata corresponding to the initial operation data stream is determined as described above, it is considered that when the target metadata parses the data contained in the initial operation data stream, it is actually a process of determining multiple target sub-metadata based on the definition data stream of the upstream broadcast, and performing streaming parsing on data at different stages through multiple parsing cycles. Therefore, by using the target metadata to parse the data contained in the initial operation data stream, different metadata can be applied at different parsing stages to complete data structure changes, thereby obtaining the target operation data stream based on the parsing results, so that downstream processing can complete data storage operations based on the target operation data stream.
[0073] Specifically, parsing refers to multiple parsing processes that parse the data contained in the initial operational data stream in batches using different metadata at different parsing cycles, ensuring that the same batch of data can be parsed using the same metadata, so that downstream processing can obtain data corresponding to the same parsing results for use in the target operational data stream. Accordingly, the target operational data stream specifically refers to the data stream obtained by parsing the data contained in the initial operational data stream using the target metadata, and the data structure corresponding to the data contained in the target operational data stream is different from the data structure corresponding to the data contained in the initial operational data stream.
[0074] In one embodiment, when parsing the data in the initial operation data stream, in order to improve the parsing efficiency, a state data parser can be used. In this embodiment, the specific implementation is as follows:
[0075] A data parser is created for the data contained in the initial operation data stream according to the target metadata; the data contained in the initial operation data stream is parsed using the data parser, and the target operation data stream is generated according to the parsing result; wherein the data parser is used to convert the data into a data format that matches the target data storage space.
[0076] Specifically, a data parser refers to a processor that can perform structural conversion on the data contained in the initial data stream based on metadata, and is used to implement data parsing and processing corresponding to different fields. In other words, different data parsers can be created for data corresponding to different fields in the initial operation data stream based on the target metadata. This allows different data parsers to be used to parse data of different field types during the application phase. It should be noted that the creation of a data parser can be completed at any time before parsing, and this embodiment does not impose any limitations on this.
[0077] Based on this, when performing parsing, you can first create a data parser for the data contained in the initial operation data stream based on the target metadata; at this time, use the data parser to parse the data contained in the initial operation data stream, that is, data of the same field type in the initial operation data stream will be parsed in sequence by the data parser corresponding to the field, so as to generate the target operation data stream corresponding to the initial operation data stream according to the parsing results corresponding to each data parser; and the data parser is used to convert the data into a data format that matches the target data storage space, so that it can be convenient for subsequent processing.
[0078] In summary, by parsing the data through the parser, the data in the initial operation data stream can be parsed into the target operation data stream to ensure that the format of the data after parsing is the same as the data format stored in the target data storage space, thereby ensuring that the data is written more accurately.
[0079] In one embodiment, considering that each parsing process requires different parsers to parse different data, in order to ensure parsing accuracy, the same batch of data is parsed in the same manner before being sent to the downstream node. In this embodiment, the specific implementation is as follows:
[0080] Determine a preset waiting time interval and a triggering time node corresponding to the data parser, and determine a data parsing time node based on the waiting time interval and the triggering time node; read the i-th data in the initial operation data stream, and detect whether the reading time node of the i-th data reaches the data parsing time node, wherein i starts at 1 and i is a positive integer; if not, add the i-th data to the parsing queue corresponding to the data parser, i is incremented by 1, and execute the step of reading the i-th data in the initial operation data stream; if so, use the data parser to parse the data contained in the parsing queue, and generate the target operation data stream according to the parsing result.
[0081] Specifically, the waiting time interval specifically refers to the tolerance time, which is used to achieve that when processing data, each operator can wait for the new metadata to complete the data parsing process, so as to avoid the problem of different metadata parsing of the same batch of data. Correspondingly, the trigger time node specifically refers to the time node when the data parser receives the target metadata. Correspondingly, the parsing time node specifically refers to the time node when the new metadata is used to parse the data, which is obtained by adding the departure time node and the waiting time length. That is to say, the parsing time node is the time node obtained by adding the waiting time interval length to the trigger time node, which is used to achieve the purpose of tolerance. For example, if the waiting time interval is T minutes and the trigger time node is 5:00, the parsing time is the time after 5:00+T minutes.
[0082] Based on this, when performing data parsing, it is actually a streaming processing operation, that is: you can first determine the preset waiting time interval and the trigger time node corresponding to the data parser, and determine the data parsing time node based on the waiting time interval and the trigger time node; thereafter, you can read the i-th data in the initial operation data stream, and detect whether the reading time node of the i-th data reaches the data parsing time node, where i starts from 1 and i is a positive integer; if not, it means that the time has not been reached, and further means that other operators have not synchronized the same metadata as the operator, then the i-th data can be added to the parsing queue corresponding to the data parser, i is incremented by 1, and the step of reading the i-th data in the initial operation data stream is executed; if so, it means that all operators have completed loading the same initial data, then the data parser can be used to parse the data contained in the parsing queue, and generate the target operation data stream based on the parsing result, so as to complete the parsing processing operation of the same batch of data according to the same parser.
[0083] In actual applications, when parsing data, this can be achieved through the TypeConverter (format conversion operator) operator; that is, the RowData of the DML data stream can be serialized into formats such as JSON or Avro. That is, first create a corresponding converter for each field of RowData in the data stream according to the Schema. When data is sent to the TypeConverter operator, you can traverse each field and match the converter corresponding to the field to use the converter to convert the data corresponding to the field into formats such as JSON or Avro. In specific implementation, different fields can correspond to different converters, such as string fields have stringconverter, and datetime fields have datetimeconverter; among them, the field type can be specified as a parameter before the task is deployed.
[0084] In one implementation, when an upstream operator broadcasts a DDL data stream, the TypeConverter operator also receives the broadcast of the DDL substream. Because each DDL (structural change) causes schema changes, it is necessary to determine which schema to use when parsing the current data. Furthermore, because TypeConverter operators can be multi-concurrent, ensuring data write accuracy requires ensuring that all TypeConverter operators at the same time parse the data stream using the same schema.
[0085] In specific implementation, see the diagram in Figure 4(b). Assume that the data to be parsed, rows 1 through 3, is parsed using Schema = [a, b], and rows 5 through 9 are parsed using Schema = [a, b, c]. Each record is in the format XXX-N, where N is the event time, i.e., the time when the DML / DDL was generated at the source. The DML substream operates in multiple concurrent operations. Within a single concurrent operation, the timing guarantees that records are sent to the TypeConverter operator in order for parsing. Currently, there are nine records, including eight DML records and one DDL record. See the diagram in Figure 4(c). Assume that rows 1 through 3 are sent. If the TypeConverter operator parses the data using Schema = [a, b] before DDL-4 reaches it, then rows 1 through 3 will be parsed using Schema = [a, b].
[0086] As shown in Figure 4 (d), when the DDL substream begins broadcasting DDL, the upper TypeConverter operator does not receive the DDL in time due to factors such as network latency, and does not process the two data items ROW-1 and ROW-2. At this point, the upper TypeConverter operator's schema remains unchanged, still [a, b], but the lower TypeConverter operator has received and updated the schema to [a, b, c]. Furthermore, as shown in Figure 4 (e), when the upper TypeConverter operator resumes consumption, it continues to send data downstream according to the schema [a, b]. Meanwhile, the lower TypeConverter operator parses and sends new data according to the schema [a, b, c]. Data sent simultaneously have different schemas, making downstream processing incompatible and affecting data accuracy.
[0087] In view of this, when the TypeConverter operator performs data parsing, it can perform micro-batch processing, that is, define a tolerance T (Tolerance, millisecond-level timestamp). When the DDL arrives, it starts to accumulate multiple batches of data, and uses the processing time as the trigger condition. All cached data is processed after T time. Among them, the role of the tolerance T is: allowing the TypeConverter operator to receive data after the DDL event time after the DDL arrives, which can be understood as allowing the TypeConverter to receive data later than the time when the current DDL arrives at the TypeConverter operator before the DDL arrives; for example, the event time of DDL-4 is 4, and the event time of ROW-6 is 6, allowing ROW-6 to arrive at the upper TypeConverter before DDL-4; in addition, it also allows the time difference between the DDL broadcasts arriving at each TypeConverter operator, and the maximum time difference is T. For example, DDL-4 is broadcast to two TypeConverter operators at the same time, and the arrival time is allowed to have a maximum difference of T milliseconds. That is to say, after the DDL arrives at the TypeConverter operator, it will not immediately perform data parsing processing with the new Schema, but will do so after T time, so as to ensure that all TypeConverter operators parse data with the same Schema at the same time.
[0088] Refer to the schematic diagram shown in (a) of Figure 5. Taking T as the waiting time interval and T1 and T2 as the trigger time nodes corresponding to different data parsers respectively, the above parsing process is described. Specifically, when the TypeConverter A operator triggers the data parsing operation at the processing time A (parsing time node), at T1+T time, the TypeConverter A operator can safely parse the subsequent data with the new Schema because (T1+T)-T2<T, [T2,T1+T] is within the window, and the TypeConverter B operator will not send ROW-1 and ROW-2 parsed with the old Schema; where T is the tolerance, T1 is the time when the TypeConverter A operator receives DDL-4, and T2 is the time when the TypeConverter B operator receives DDL-4. At the same time, ROW-1 and ROW-2 will not be sent within the [T1,T2] window because each Flush (assuming the time is S) will only Flush the data at the time of (0,S-T]. Therefore, ROW-1 and ROW-2 will not be parsed and sent with the old Schema within the [T1,T1+T] window. Instead, ROW-1 and ROW-2 are cached in the queue and sent to the downstream after T2+T time.
[0089] When the TypeConverter operator parses old data using the new schema, as shown in Figure 5 (b), the TypeConverter B operator triggers the parsing operation at time T2+T. At this time, ROW-1 and ROW-2, which were parsed using the old schema = [a, b], exist in the queue. All records in the micro-batch should be parsed using the schema = [a, b, c] and sent downstream. This means that if the TypeConverter B operator contains data based on different schemas during a single parsing process, all data must be unified into the same schema. Since ROW-1 and ROW-2 lack the c field, null values can be used to fill it during normalization.
[0090] In summary, by combining the preset waiting time interval, it can be ensured that each operator waits for the metadata of the data parser to be received. By tolerating the waiting time interval, each operator can use the same data parser to complete the parsing of the data, so that the downstream operator can receive the parsing results corresponding to the same parsing operation.
[0091] Step S208: Determine whether the initial operation data stream is parsed into a parsed information sequence corresponding to the target operation data stream, and generate a storage information sequence according to the parsed information sequence.
[0092] In one embodiment, after the initial operation data stream is parsed into the target operation data stream, in order to enable the data contained in the target operation data stream to be stored in accordance with the requirements corresponding to the data storage request when subsequently written to the target data storage space, it is possible to first determine that the initial operation data stream is parsed into a parsing information sequence corresponding to the target operation data stream, so as to generate a storage information sequence through the parsing information sequence, so that when the data contained in the target operation data stream is finally stored in the target data storage space, the data storage operation can be completed according to the storage information sequence, so as to ensure that the data conforms to the result after the data structure is changed when it is written to the target data storage space.
[0093] The parsing information sequence specifically refers to a sequence of parsing information recorded when the data contained in the initial operation data stream is parsed into data in the target operation data stream. The parsing information contained in the sequence is used to record the conversion relationship when the data structure changes, so that downstream computing nodes can generate storage information sequences based on this information, so that when the data is finally stored, the storage operation can be completed according to the information contained in the storage information sequence. Correspondingly, the storage information sequence specifically refers to a sequence composed of metadata change information, so that when the data contained in the target operation data stream is stored in the target data storage space, the data can be processed into data based on the metadata change results according to the storage information contained in the storage information sequence, so as to ensure that the data meets the user's storage requirements after being stored in the target data storage space.
[0094] In one embodiment, when determining the parsing information sequence, in order to ensure that storage information can be accurately generated according to the parsing information contained in the parsing information sequence during subsequent storage, so that the data can meet the user's storage requirements when written to the data storage space, the parsing information sequence can be generated based on the structural conversion relationship. In this embodiment, the specific implementation method is as follows:
[0095] Determine an initial data structure corresponding to the initial operation data contained in the initial operation data stream, and a target data structure corresponding to the target operation data contained in the target operation data stream; generate a parsing information set based on a conversion relationship from the initial data structure to the target data structure; sort the parsing information contained in the parsing information set according to a data parsing order to obtain the parsing information sequence.
[0096] Specifically, the initial data structure specifically refers to the data structure corresponding to the initial operation data, and the target data structure specifically refers to the data structure corresponding to the target operation data, wherein the initial data structure is different from the target data structure.
[0097] Based on this, in order to ensure the accuracy of the generation of the parsing information sequence, we can first determine the initial data structure corresponding to the initial operation data contained in the initial operation data stream, and the target data structure corresponding to the target operation data contained in the target operation data stream; thereafter, we can generate a parsing information set based on the conversion relationship from the initial data structure to the target data structure; at this time, we can sort the parsing information contained in the parsing information set according to the data parsing order to obtain the parsing information sequence, which is convenient for the subsequent generation of the storage information sequence.
[0098] In summary, by combining the conversion relationship between the initial data structure and the target data structure, a parsing information set is generated, and a parsing information sequence is generated based on this, which can ensure the accuracy of the generation of the parsing information sequence, facilitate subsequent sequential data storage, and ensure the orderliness of the data.
[0099] In one embodiment, in order to support data structure changes when data is written to the data lake, a storage information sequence can be generated based on the parsed information sequence, so that the data in the target operation data stream can be written to the data storage space on demand according to the storage information sequence. In this embodiment, the specific implementation method is as follows:
[0100] Determine metadata version information corresponding to the parsed information contained in the parsed information sequence; generate storage information corresponding to the parsed information in the parsed information sequence according to the metadata version information; and sort the storage information corresponding to the parsed information in the parsed information sequence according to the arrangement order of the parsed information in the parsed information sequence to obtain the storage information sequence.
[0101] Specifically, metadata version information refers to the version description information corresponding to the metadata. It is used to identify different versions of metadata and facilitate the subsequent generation of storage information based on the version information. Correspondingly, sorting specifically refers to ensuring that the stored information can be used in order, ensuring that the data is written to the target data storage space in an orderly and more accurate manner.
[0102] Based on this, in order to obtain accurate storage information and store data in an orderly manner based on this, we can first determine the metadata version information corresponding to the parsed information contained in the parsed information sequence; and generate the storage information corresponding to the parsed information in the parsed information sequence based on the metadata version information; thereafter, sort the storage information corresponding to the parsed information in the parsed information sequence according to the arrangement order of the parsed information in the parsed information sequence, so as to obtain a storage information sequence with higher accuracy.
[0103] In summary, by combining metadata version information to generate a storage information sequence, the storage information contained in the storage information sequence can be related to the metadata change results. Based on this, when storing data, the data can be stored in the target data storage space according to the specified data structure.
[0104] On this basis, since the storage information is the information used for parsing during data storage, it determines the structure in which the data is written to the target data storage space. Therefore, accurate generation is required to ensure the accuracy of data writing. In this process, in order to ensure that the data is in order, the target computing node can be used to complete it. In this embodiment, the specific implementation method is as follows:
[0105] For any of the parsed information, the parsed information and the metadata version information corresponding to the parsed information are sent to the target computing node; the target computing node performs information processing based on the parsed information and the corresponding metadata version information to obtain the storage information corresponding to the target parsed information.
[0106] Specifically, the target computing node refers to a node that performs information processing calculations independently. It can be set outside the data lake system and can be set according to actual needs. This embodiment does not impose any limitations here.
[0107] Based on this, when using a single node to generate storage information, the target resolution information and the target metadata version information corresponding to the target resolution information can be sent to the target computing node; then, the target computing node processes the information based on the target resolution information and the target metadata version information, so that the storage information corresponding to the target resolution information can be obtained based on the processing result of the target computing node. In specific implementation, when the target computing node processes the information based on the resolution information and its corresponding metadata version information, it is actually a process of using the target computing node to convert the resolution information according to the metadata version information. Due to the conversion relationship of the data structure change of the resolution information record set, the metadata version information can be used to convert the resolution information into storage information in the format set before recording the data storage, so that when the data contained in the target operation data stream is stored in the target data storage space, it can be completed according to the storage information, thereby ensuring that the stored data meets the user's storage requirements.
[0108] In addition, in order to improve the efficiency of generating storage information, it can also be completed in combination with a computing node cluster; in this embodiment, the specific implementation method is as follows:
[0109] Determine a computing node cluster and send the metadata version information and the parsing information contained in the parsing information sequence to the computing nodes included in the computing node cluster; receive a storage information set fed back by the computing nodes in the computing node cluster, and generate storage information corresponding to the parsing information in the parsing information sequence based on the storage information set.
[0110] Specifically, a computing node cluster specifically refers to a cluster composed of devices that can respectively receive metadata and process data that needs to be computed.
[0111] Based on this, when storage information is generated through multiple computing nodes, the computing node cluster can be determined, and then the metadata version information and the parsing information contained in the parsing information sequence are sent to the computing nodes included in the computing node cluster; thereafter, each computing node processes it separately, and after the processing is completed, the results of all nodes are obtained, and the storage information set fed back by the computing nodes in the computing node cluster can be received, and the storage information corresponding to the parsing information in the parsing information sequence is generated according to the storage information set.
[0112] In practical applications, assigning watermark information for data storage can be achieved using the WatermarkAssigner operator. Specifically, after an upstream operator receives a DDL signal, determining when to parse the data using the new schema is the basis for determining data structure changes. Downstream operators also need to know which schema version is used to parse the data, allowing downstream operators to distinguish between different schemas when writing data. Therefore, after the data is parsed by the upstream operator and sent downstream, the WatermarkAssigner operator is used to assign a watermark and pass it downstream. The assigned watermark is the same as the schema version and can be either the event time or the processing time of the DDL. The event time is the event that the DDL occurs at the source, while the processing time is the event that the DDL is received and processed by the stream computing engine. For example, if MySQL generates DDL at 10:00 and is processed by the Flink engine at 10:05, the event time = 10:00 and the processing time = 10:05.
[0113] In specific implementations, the Watermark Assigner operator can be used as the entry point for upstream write operators. It must process all data streams to detect schema changes, converting the structure into watermarks and passing them downstream. When detecting schema changes, it can be understood that source records contain schema information. Upstream operators calculate different schema IDs based on different schemas as additional metadata for the records. The Watermark Assigner operator detects changes in the record's schema ID and sends a watermark signal if any changes occur.
[0114] On this basis, whether the Watermark Assigner operator is single-point (single parallelism) or multi-concurrency (multi-parallelism) determines how downstream operators are implemented. In the single-point Watermark Assigner operator scenario, all data from upstream operator A will pass through the single-point Watermark Assigner operator. When a watermark needs to be generated, it will be broadcast to all downstream operators B. As shown in Figure 5 (c), the single-point Watermark Assigner operator processes the output results of the upstream operator. As shown in Figure 5 (d), after ROW-1 is processed by the single-point Watermark Assigner operator, ROW-1 is forwarded to a specific instance of Operator B according to the rules. At the same time, the watermark is broadcast to all instances of Operator B (number of instances = Operator B's operator parallelism). After the downstream operator Operator B receives the watermark signal, it immediately passes the watermark downstream.
[0115] In a scenario with multiple concurrent Watermark Assigner operators, if the parallelism of the multiple concurrent Watermark Assigner operators is equal to the parallelism of upstream operator A, each Watermark Assigner only processes the output of a single operator. As shown in Figure 5(e), Watermark Assigner-1 only processes the input of Operator A-1. As shown in Figure 5(f), when ROW-1 is processed by Watermark Assigner-1, ROW-1 is forwarded to a specific instance of Operator B according to the rules, and the Watermark is broadcast to all instances of Operator B (number of instances = Operator B operator parallelism). After receiving the Watermark signal, the downstream operator Operator B will not immediately pass the Watermark downstream because it has not yet received the Watermark generated by Watermark Assigner-2. In this case, it can wait until Watermark Assigner-2 processes the data of Operator A-2 and the generated Watermark is processed by Operator B before it is passed downstream.
[0116] In summary, by selecting different computing nodes to generate different types of storage information, it can be adapted to different scenarios, thereby improving the flexibility of data storage, so that the generation of storage information can be selected on demand when storing data.
[0117] Step S210 , storing the data included in the target operation data stream into the target data storage space corresponding to the data operation request according to the storage information sequence.
[0118] In one embodiment, after the storage information sequence is generated based on the parsed information sequence, in order to ensure that the data contained in the target operation data stream can be written to the target data storage space as needed, it can be completed in combination with the storage information sequence that records the metadata change results. That is, the data contained in the target operation data stream can be processed in sequence according to the storage information contained in the storage information sequence and then written to the target data storage space, so that the data finally written to the target data storage space is the data that the user needs to write.
[0119] The target data storage space specifically refers to a distributed database corresponding to a target large system for storing data from different source ends in the data lake system. The target data storage space can be selected based on actual needs and is not limited in this embodiment.
[0120] In addition, when processing data that needs to be written to the target data storage space, corresponding containers can be allocated according to different operation types (insert, update, delete), partitions, and file information to write the data to the target data storage space.
[0121] In one embodiment, when storing the data corresponding to the target operation data stream into the target data storage space, in order to ensure that the data can be stored in accordance with the requirements corresponding to the data storage request, it is necessary to perform a structural change on the data in combination with the storage information contained in the storage information sequence, so as to obtain the target data before performing the data writing process; in this embodiment, the specific implementation method is as follows:
[0122] Select the jth storage information in the storage information sequence, and read the initial data corresponding to the jth storage information in the target operation data stream, wherein j starts from 1 and is a positive integer; use the jth storage information to parse the initial data into target data, and call the target operator to store the target data in the target data storage space corresponding to the data operation request; detect whether there is unselected storage information in the storage information sequence; if not, j is incremented by 1, and the step of selecting the jth storage information in the storage information sequence is executed; if so, generate a data storage result and send it to the service user corresponding to the data storage request.
[0123] Based on this, when performing data storage, since the data contained in the target operation data stream is streaming data, it will also be implemented in a streaming storage manner when written to the target data storage space. That is, you can first select the jth storage information in the storage information sequence, and read the initial data corresponding to the jth storage information in the target operation data stream, where j starts at 1 and j is a positive integer; at this time, you can use the jth storage information to parse the initial data into target data, and call the target operator to store the target data in the target data storage space corresponding to the data operation request; at this time, you can check whether there is unselected storage information in the storage information sequence; if not, it means that there is still stored data, then j can be incremented by 1, and the step of selecting the jth storage information in the storage information sequence is executed; if so, all data have been stored, then a data storage result can be generated and sent to the service user corresponding to the data storage request.
[0124] In summary, when storing data, the data can be parsed according to the storage information contained in the storage information sequence, converted into a data structure corresponding to the target data storage space, and stored based on this, which can ensure that the data stored in the data storage space meets the data storage request.
[0125] On this basis, in order to complete the write operation using a stateful operator, the data write process can be implemented by changing the operator state. In this embodiment, the specific implementation method is as follows:
[0126] Call the initial operator and determine the operator state corresponding to the initial operator; when the operator state is a baseline state, update the initial operator according to the j-th storage information to obtain the target operator of the target state; use the target operator of the target state to store the target data in the target data storage space corresponding to the data operation request.
[0127] Based on this, when storing data, in order to ensure that data can be stored in the same way in the same batch, the initial operator can be called first, and the operator state corresponding to the initial operator can be determined; when the operator state is the baseline state, the initial operator can be updated according to the j-th storage information to obtain the target operator of the target state according to the update result; thereafter, the target operator of the target state can be used to store the target data in the target data storage space corresponding to the data operation request.
[0128] In practical applications, when writing data to the target data storage space, the StreamWrite operator can be used. Considering that the StreamWrite operator requires schema information during the data writing process, it also needs to receive DDL broadcasts. Furthermore, considering that the StreamWrite operator may be stateless, if the operator writing to the data lake relies on schema information and is a stateless operator, the operator needs to be adapted into a stateful operator to record different versions of schema information. Therefore, the stateless operator can be converted into a stateful operator, and the operator state can be either recording state (recording state) or suspended state (bpendingSchemas). The recording state means that if the schema changes and the StreamWrite operator receives a DDL broadcast, it needs to cache the record in a queue before receiving the corresponding watermarks from all upstream operators to ensure schema consistency at the same time. The suspended state refers to the schema that will take effect. For a schema, it will only take effect after the StreamWrite operator receives the corresponding watermarks from all upstream operators.
[0129] In specific implementation, as shown in Figure 6 (a), the StreamWrite operator can wait until all upstream watermarks for the new schema (S-2) have arrived. That is, after the watermarks of the three upstream operators (Operator B) have arrived, it can write all cached data for the current schema (S-1), as shown in Figure 6 (b). After the data is successfully written, the current schema can be updated to S-2. Subsequent data will be parsed and written using the new schema until all data has been written to the target big data system.
[0130] In order to solve the problem that metadata cannot be changed during data storage and improve data storage efficiency and accuracy, the data storage method provided in this embodiment can, after receiving a data storage request, read the source data, serialize the source data, and then divert it to obtain an initial operation data stream and a definition data stream. At this time, considering that if metadata changes triggered during data storage are not paid attention to, the data ultimately entering the lake will not meet actual storage requirements, in order to enable timely processing of data in the data stream after metadata changes during data storage are triggered, it is necessary to determine the target metadata corresponding to the initial operation data stream based on the definition data stream composed of the operation data diverted after serialization. Then, the target metadata is used to parse the data contained in the initial operation data stream, and the target operation data stream corresponding to the target data storage space is obtained based on the parsing results. This allows attention to the problem of metadata changes during the data format conversion process, so that the data in the data stream can be processed according to the changed metadata during the data format conversion stage before entering the lake. On this basis, in order to ensure that data can be written to the target data storage space quickly and accurately, it is possible to first determine that the initial operation data stream is parsed into the parsing information sequence corresponding to the target operation data stream, so as to generate a storage information sequence for storing data based on the parsing information sequence; finally, the data contained in the target operation data stream can be stored in the target data storage space corresponding to the data operation request according to the storage information sequence; in the data storage process, the target metadata can be determined according to the defined data stream, and parsed and stored based on this, so that the data format conversion stage can be combined with the impact of metadata changes to form data that ultimately meets the storage requirements when data is stored, thereby ensuring the accuracy of data storage. At the same time, since the source data will be diverted into the defined data stream and the initial operation data stream after the storage task is triggered, the original data and the operation data can be separated to form a structure that focuses on the original data and the operation data respectively, thereby supporting the data change requirements during data storage, and thus making the data finally stored in the target data storage space the target data that the data storage party needs to store.
[0131] The following further illustrates the data storage method provided in this specification using the application of the data storage method in a data lake scenario as an example, in conjunction with Figure 7. Figure 7 shows a flowchart of the processing process of a data storage method provided in one embodiment of this specification, specifically including the following steps.
[0132] Step S702: receiving a data storage request submitted for a source data storage space.
[0133] Step S704 : concurrently reading the full data or incremental data in the source data storage space according to the data storage request, and using the full data or incremental data as the source data.
[0134] Step S706 : Convert the source data in the initial data format into intermediate data in the intermediate data format according to a preset format conversion strategy.
[0135] Step S708 : determining data attribute information corresponding to the intermediate sub-data contained in the intermediate data, and assigning labels to the intermediate sub-data contained in the intermediate data according to the data attribute information.
[0136] Step S710 : performing flow splitting processing on the intermediate sub-data included in the intermediate data according to the label allocation result to obtain an initial operation data flow and a definition data flow.
[0137] Step S712: determining target metadata corresponding to the initial operation data stream according to the defined data stream.
[0138] Specifically, at least one initial sub-metadata corresponding to the initial data stream is determined; the at least one initial sub-metadata is updated using the definition data stream to obtain at least one target sub-metadata; and target metadata corresponding to the initial operation data stream is generated based on the at least one target sub-metadata.
[0139] Step S714: Create a data parser for the data included in the initial operation data stream according to the target metadata.
[0140] Step S716: parse the data contained in the initial operation data stream using a data parser, and generate a target operation data stream according to the parsing result.
[0141] The data parser is used to convert the data into a data format that matches the target data storage space.
[0142] Specifically, determine the preset waiting time interval and the trigger time node corresponding to the data parser, and determine the data parsing time node based on the waiting time interval and the trigger time node; read the i-th data in the initial operation data stream, and detect whether the reading time node of the i-th data reaches the data parsing time node, where i starts from 1 and i is a positive integer; if not, add the i-th data to the parsing queue corresponding to the data parser, i is incremented by 1, and execute the step of reading the i-th data in the initial operation data stream; if so, use the data parser to parse the data contained in the parsing queue, and generate the target operation data stream according to the parsing result.
[0143] Step S718: Determine the initial data structure corresponding to the initial operation data included in the initial operation data stream, and the target data structure corresponding to the target operation data included in the target operation data stream.
[0144] Step S720: Generate a parsing information set according to the conversion relationship from the initial data structure to the target data structure.
[0145] Step S722: sort the parsing information contained in the parsing information set according to the data parsing order to obtain a parsing information sequence.
[0146] Step S724: Determine metadata version information corresponding to the parsed information contained in the parsed information sequence.
[0147] Step S726: Generate storage information corresponding to the parsed information in the parsed information sequence according to the metadata version information.
[0148] Step S728: Sort the storage information corresponding to the parsed information in the parsed information sequence according to the arrangement order of the parsed information in the parsed information sequence to obtain a storage information sequence.
[0149] Specifically, the target resolution information and the target metadata version information corresponding to the target resolution information are sent to the target computing node; the target computing node processes the target resolution information and the target metadata version information to obtain the storage information corresponding to the target resolution information.
[0150] In addition, a computing node cluster is determined, and metadata version information and parsing information contained in the parsing information sequence are sent to the computing nodes included in the computing node cluster; a storage information set fed back by the computing nodes in the computing node cluster is received, and storage information corresponding to the parsing information in the parsing information sequence is generated based on the storage information set.
[0151] Step S730 , storing the data included in the target operation data stream into the target data storage space corresponding to the data operation request according to the storage information sequence.
[0152] Specifically, the jth storage information is selected in the storage information sequence, and the target data corresponding to the jth storage information is read in the target operation data stream, where j starts at 1 and is a positive integer; the initial data is parsed into target data using the jth storage information, and the target operator is called to store the target data in the target data storage space corresponding to the data operation request; it is detected whether there is unselected storage information in the storage information sequence; if not, j is incremented by 1, and the step of selecting the jth storage information in the storage information sequence is executed; if so, a data storage result is generated and sent to the service user corresponding to the data storage request.
[0153] In one embodiment, an initial operator is called and the operator state corresponding to the initial operator is determined; when the operator state is a baseline state, the initial operator is updated according to the j-th storage information to obtain a target operator of a target state; and using the target operator of the target state, the target data is stored in the target data storage space corresponding to the data operation request.
[0154] In order to solve the problem that metadata cannot be changed during data storage and improve data storage efficiency and accuracy, the data storage method provided in this embodiment can, after receiving a data storage request, read the source data, serialize the source data, and then divert it to obtain an initial operation data stream and a definition data stream. At this time, considering that if metadata changes triggered during data storage are not paid attention to, the data ultimately entering the lake will not meet actual storage requirements, in order to enable timely processing of data in the data stream after metadata changes during data storage are triggered, it is necessary to determine the target metadata corresponding to the initial operation data stream based on the definition data stream composed of the operation data diverted after serialization. Then, the target metadata is used to parse the data contained in the initial operation data stream, and the target operation data stream corresponding to the target data storage space is obtained based on the parsing results. This allows attention to the problem of metadata changes during the data format conversion process, so that the data in the data stream can be processed according to the changed metadata during the data format conversion stage before entering the lake. On this basis, in order to ensure that data can be written to the target data storage space quickly and accurately, it is possible to first determine that the initial operation data stream is parsed into the parsing information sequence corresponding to the target operation data stream, so as to generate a storage information sequence for storing data based on the parsing information sequence; finally, the data contained in the target operation data stream can be stored in the target data storage space corresponding to the data operation request according to the storage information sequence; in the data storage process, the target metadata can be determined according to the defined data stream, and parsed and stored based on this, so that the data format conversion stage can be combined with the impact of metadata changes to form data that ultimately meets the storage requirements when data is stored, thereby ensuring the accuracy of data storage. At the same time, since the source data will be diverted into the defined data stream and the initial operation data stream after the storage task is triggered, the original data and the operation data can be separated to form a structure that focuses on the original data and the operation data respectively, thereby supporting the data change requirements during data storage, and thus making the data finally stored in the target data storage space the target data that the data storage party needs to store.
[0155] Corresponding to the above method embodiment, this specification also provides a data storage device embodiment. FIG8 shows a schematic diagram of the structure of a data storage device provided by one embodiment of this specification. As shown in FIG8 , the device includes:
[0156] The reading module 802 is configured to read the source data in response to a data storage request and serialize the source data; the determining module 804 is configured to split the serialized source data into an initial operation data stream and a definition data stream, and determine the target metadata corresponding to the initial operation data stream according to the definition data stream, wherein the initial operation data stream is constructed based on the original data, and the definition data stream is constructed based on the operation data that changes the original data; the parsing module 806 is configured to use the target metadata to parse the data contained in the initial operation data stream, and determine the target operation data stream according to the parsing result; the generating module 808 is configured to determine that the initial operation data stream is parsed into a parsed information sequence corresponding to the target operation data stream, and generate a storage information sequence according to the parsed information sequence; the storing module 810 is configured to store the data contained in the target operation data stream according to the storage information sequence to the target data storage space corresponding to the data operation request.
[0157] In one embodiment, the reading module 802 is further configured to: convert the source data in the initial data format into intermediate data in the intermediate data format according to a preset format conversion strategy; the determination module 804 is further configured to: determine the data attribute information corresponding to the intermediate sub-data contained in the intermediate data, and assign labels to the intermediate sub-data contained in the intermediate data according to the data attribute information; and perform diversion processing on the intermediate sub-data contained in the intermediate data according to the label assignment result to obtain the initial operation data flow and the definition data flow.
[0158] In one embodiment, the determination module 804 is further configured to: obtain at least one initial sub-metadata corresponding to the initial operation data stream; update the at least one initial sub-metadata using the definition data stream to obtain at least one target sub-metadata; and generate target metadata corresponding to the initial operation data stream based on the at least one target sub-metadata.
[0159] In one embodiment, the parsing module 806 is further configured to: create a data parser for the data contained in the initial operation data stream based on the target metadata; use the data parser to parse the data contained in the initial operation data stream, and generate the target operation data stream based on the parsing results; wherein the data parser is used to convert the data into a data format that matches the target data storage space.
[0160] In one embodiment, the parsing module 806 is further configured to: determine a preset waiting time interval and a trigger time node corresponding to the data parser, and determine a data parsing time node based on the waiting time interval and the trigger time node; read the i-th data in the initial operation data stream, and detect whether the reading time node for reading the i-th data reaches the data parsing time node, where i starts at 1 and is a positive integer; if not, add the i-th data to the parsing queue corresponding to the data parser, i is incremented by 1, and execute the step of reading the i-th data in the initial operation data stream; if so, use the data parser to parse the data contained in the parsing queue, and generate the target operation data stream based on the parsing result.
[0161] In one embodiment, the generation module 808 is further configured to: determine the initial data structure corresponding to the initial operation data contained in the initial operation data stream, and the target data structure corresponding to the target operation data contained in the target operation data stream; generate a parsing information set based on the conversion relationship from the initial data structure to the target data structure; sort the parsing information contained in the parsing information set according to the data parsing order to obtain the parsing information sequence.
[0162] In one embodiment, the generation module 808 is further configured to: determine metadata version information corresponding to the parsed information contained in the parsed information sequence; generate storage information corresponding to the parsed information in the parsed information sequence based on the metadata version information; and sort the storage information corresponding to the parsed information in the parsed information sequence according to the arrangement order of the parsed information in the parsed information sequence to obtain the storage information sequence.
[0163] In one embodiment, the generation module 808 is further configured to: for any one of the parsed information, send the parsed information and the metadata version information corresponding to the parsed information to the target computing node; and obtain the storage information corresponding to the target parsed information by performing information processing according to the parsed information and the corresponding metadata version information through the target computing node.
[0164] In one embodiment, the generation module 808 is further configured to: determine a computing node cluster, and send the metadata version information and the parsing information contained in the parsing information sequence to the computing nodes included in the computing node cluster; receive a storage information set fed back by the computing nodes in the computing node cluster, and generate storage information corresponding to the parsing information in the parsing information sequence based on the storage information set.
[0165] In one embodiment, the storage module 810 is further configured to: select the jth storage information in the storage information sequence, and read the initial data corresponding to the jth storage information in the target operation data stream, where j starts from 1 and j is a positive integer; use the jth storage information to parse the initial data into target data, and call the target operator to store the target data in the target data storage space corresponding to the data operation request; detect whether there is unselected storage information in the storage information sequence; if not, j is incremented by 1, and the step of selecting the jth storage information in the storage information sequence is executed; if so, generate a data storage result and send it to the service user corresponding to the data storage request.
[0166] In one embodiment, the storage module 810 is further configured to: call the initial operator and determine the operator state corresponding to the initial operator; when the operator state is a baseline state, update the initial operator according to the j-th storage information to obtain the target operator of the target state; use the target operator of the target state to store the target data in the target data storage space corresponding to the data operation request.
[0167] In one embodiment, the reading module 802 is further configured to: receive a data storage request submitted for a source data storage space; concurrently read full data or incremental data in the source data storage space according to the data storage request; and use the full data or the incremental data as the source data.
[0168] In one embodiment, the reading module 802 is further configured to: adjust the data reading link corresponding to the source data storage space to a reading state in response to the data storage request; after executing the step of using the full data or the incremental data as the source data, it also includes: adjusting the data reading link corresponding to the source data storage space to a blocking state.
[0169] In one embodiment, the determination module 804 is further configured to: generate a metadata loading request based on the defined data stream, and send the metadata loading request to the metadata management end corresponding to the target data storage space; receive the initial metadata fed back by the metadata management end in response to the metadata loading request, and determine the target metadata corresponding to the initial operation data stream based on the initial metadata.
[0170] In order to solve the problem that metadata cannot be changed during data storage and improve data storage efficiency and accuracy, the data storage device provided in this embodiment can read the source data after receiving a data storage request, and serialize the source data before diverting it to obtain an initial operation data stream and a definition data stream. At this time, considering that if metadata changes triggered during data storage are not paid attention to, the data finally entering the lake will not meet the actual storage requirements, therefore, in order to be able to process the data in the data stream in a timely manner after the metadata change during data storage is triggered, it is necessary to determine the target metadata corresponding to the initial operation data stream based on the definition data stream composed of the operation data diverted after serialization, and then use the target metadata to parse the data contained in the initial operation data stream, so as to obtain the target operation data stream corresponding to the target data storage space based on the parsing result, so that the problem of metadata changes can be paid attention to during the data format conversion process, so that the data in the data stream can be processed according to the changed metadata during the data format conversion stage before entering the lake. On this basis, in order to ensure that data can be written to the target data storage space quickly and accurately, it is possible to first determine that the initial operation data stream is parsed into the parsing information sequence corresponding to the target operation data stream, so as to generate a storage information sequence for storing data based on the parsing information sequence; finally, the data contained in the target operation data stream can be stored in the target data storage space corresponding to the data operation request according to the storage information sequence; in the data storage process, the target metadata can be determined according to the defined data stream, and parsed and stored based on this, so that the data format conversion stage can be combined with the impact of metadata changes to form data that ultimately meets the storage requirements when data is stored, thereby ensuring the accuracy of data storage. At the same time, since the source data will be diverted into the defined data stream and the initial operation data stream after the storage task is triggered, the original data and the operation data can be separated to form a structure that focuses on the original data and the operation data respectively, thereby supporting the data change requirements during data storage, and thus making the data finally stored in the target data storage space the target data that the data storage party needs to store.
[0171] The above is a schematic diagram of a data storage device according to this embodiment. It should be noted that the technical solution of the data storage device and the technical solution of the data storage method described above are of the same concept. For details not described in detail in the technical solution of the data storage device, please refer to the description of the technical solution of the data storage method described above.
[0172] Figure 9 shows a block diagram of a computing device 900 according to one embodiment of this specification. Components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.
[0173] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.
[0174] In one embodiment of the present application, the aforementioned components of the computing device 900 and other components not shown in FIG9 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG9 is for illustrative purposes only and does not limit the scope of the present application. Those skilled in the art may add or replace other components as needed.
[0175] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 may also be a mobile or stationary server.
[0176] The processor 920 is configured to execute the following computer-executable instructions, which implement the steps of the above-mentioned data storage method when executed by the processor.
[0177] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of the computing device and the technical solution of the above-mentioned data storage method are of the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above-mentioned data storage method.
[0178] An embodiment of the present specification also provides a data storage system, comprising: a server and a target data storage; the server is used to store data storage executable instructions, the target data storage is used to store data, and the steps of the data storage method are performed when the storage executable instructions are executed by the server.
[0179] The above is a schematic diagram of a data storage system according to this embodiment. It should be noted that the technical solution of the data storage system and the technical solution of the data storage method described above are based on the same concept. For details not described in detail in the technical solution of the data storage system, please refer to the description of the technical solution of the data storage method described above.
[0180] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which implement the steps of the above-mentioned data storage method when executed by a processor.
[0181] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned data storage method are based on the same concept. For details not described in detail in the technical scheme of the storage medium, please refer to the description of the technical scheme of the above-mentioned data storage method.
[0182] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned data storage method.
[0183] The above is an illustrative solution of a computer program of this embodiment. It should be noted that the technical solution of the computer program and the technical solution of the above-mentioned data storage method are of the same concept. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solution of the above-mentioned data storage method.
[0184] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0185] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.
[0186] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all one implementation method, and the actions and modules involved are not necessarily required by the embodiments of this specification.
[0187] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0188] The embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.
Claims
1. A data storage method, comprising: Reading source data in response to a data storage request, and performing serialization processing on the source data; Splitting the serialized source data into an initial operation data stream and a definition data stream, and determining target metadata corresponding to the initial operation data stream according to the definition data stream, wherein the initial operation data stream is constructed based on the original data, and the definition data stream is constructed based on the operation data that changes the original data; Parsing the data contained in the initial operation data stream using the target metadata, and determining a target operation data stream according to the parsing result; Determine that the initial operation data stream is parsed into a parsing information sequence corresponding to the target operation data stream, and generate a storage information sequence according to the parsing information sequence; The data contained in the target operation data stream is stored in the target data storage space corresponding to the data operation request according to the storage information sequence.
2. According to the method of claim 1, the serialization processing of the source data comprises: According to a preset format conversion strategy, the source data in the initial data format is converted into intermediate data in the intermediate data format; The step of dividing the serialized source data into an initial operation data stream and a definition data stream includes: Determining data attribute information respectively corresponding to the intermediate sub-data contained in the intermediate data, and assigning labels to the intermediate sub-data contained in the intermediate data according to the data attribute information; The intermediate sub-data included in the intermediate data are split according to the label allocation result to obtain an initial operation data flow and a definition data flow.
3. The method according to claim 1, wherein determining the target metadata corresponding to the initial operation data stream according to the definition data stream comprises: Acquire at least one initial sub-metadata corresponding to the initial operation data stream; Using the definition data stream to update the at least one initial sub-metadata to obtain at least one target sub-metadata; The target metadata corresponding to the initial operation data stream is generated based on the at least one target sub-metadata.
4. The method according to claim 1, wherein the step of parsing the data contained in the initial operation data stream using the target metadata and determining the target operation data stream according to the parsing result comprises: creating a data parser for the data contained in the initial operational data stream according to the target metadata; Parsing the data contained in the initial operation data stream using the data parser, and generating the target operation data stream according to the parsing result; The data parser is used to convert the data into a data format that matches the target data storage space.
5. The method according to claim 4, wherein the step of parsing the data contained in the initial operation data stream using the data parser and generating the target operation data stream according to the parsing result comprises: Determine a preset waiting time interval and a triggering time node corresponding to the data parser, and determine a data parsing time node according to the waiting time interval and the triggering time node; Reading the i-th data in the initial operation data stream, and detecting whether a reading time node of the i-th data reaches the data parsing time node, wherein i starts from 1 and i is a positive integer; If not, the i-th data is added to the parsing queue corresponding to the data parser, i is incremented by 1, and the step of reading the i-th data in the initial operation data stream is executed; If so, the data parser is used to parse the data contained in the parsing queue, and the target operation data stream is generated according to the parsing result.
6. The method according to claim 1, wherein determining that the initial operation data stream is parsed into a parsed information sequence corresponding to the target operation data stream comprises: Determine an initial data structure corresponding to the initial operation data included in the initial operation data stream, and a target data structure corresponding to the target operation data included in the target operation data stream; Generate a parsing information set according to a conversion relationship between the initial data structure and the target data structure; The parsing information contained in the parsing information set is sorted according to the data parsing order to obtain the parsing information sequence.
7. The method according to claim 1, wherein generating a storage information sequence according to the parsed information sequence comprises: Determining metadata version information corresponding to the parsed information contained in the parsed information sequence; Generate storage information corresponding to the parsed information in the parsed information sequence according to the metadata version information; According to the arrangement order of the parsed information in the parsed information sequence, the storage information corresponding to the parsed information in the parsed information sequence is sorted to obtain the storage information sequence.
8. The method according to claim 7, wherein generating storage information corresponding to the parsed information in the parsed information sequence according to the metadata version information comprises: For any of the parsed information, sending the parsed information and metadata version information corresponding to the parsed information to a target computing node; The target computing node performs information processing according to the parsed information and the corresponding metadata version information to obtain storage information corresponding to the target parsed information.
9. The method according to claim 7, wherein generating storage information corresponding to the parsed information in the parsed information sequence according to the metadata version information comprises: Determine a computing node cluster, and send the metadata version information and the parsing information included in the parsing information sequence to the computing nodes included in the computing node cluster; A storage information set fed back by a computing node in the computing node cluster is received, and storage information corresponding to the parsed information in the parsed information sequence is generated according to the storage information set.
10. The method according to claim 1, wherein storing the data contained in the target operation data stream into the target data storage space corresponding to the data operation request according to the storage information sequence comprises: Selecting the jth storage information in the storage information sequence, and reading the initial data corresponding to the jth storage information in the target operation data stream, wherein j starts from 1 and is a positive integer; parsing the initial data into target data using the j-th storage information, and calling a target operator to store the target data in a target data storage space corresponding to the data operation request; Detecting whether there is unselected storage information in the storage information sequence; If not, j is incremented by 1, and the step of selecting the jth storage information in the storage information sequence is executed; If so, generate a data storage result and send it to the service user corresponding to the data storage request.
11. The method according to claim 10, wherein the calling of the target operator to store the target data in the target data storage space corresponding to the data operation request comprises: Calling an initial operator and determining an operator state corresponding to the initial operator; When the operator state is a reference state, updating the initial operator according to the j-th stored information to obtain a target operator of a target state; The target data is stored in the target data storage space corresponding to the data operation request using the target operator of the target state.
12. The method according to claim 1, wherein determining the target metadata corresponding to the initial operation data stream according to the definition data stream comprises: Generate a metadata loading request according to the defined data stream, and send the metadata loading request to a metadata management terminal corresponding to the target data storage space; The initial metadata fed back by the metadata management end in response to the metadata loading request is received, and target metadata corresponding to the initial operation data stream is determined according to the initial metadata.
13. A data storage system comprising: The server and the target data storage; The server is used to store data storage executable instructions, and the target data storage end is used to store data. When the storage executable instructions are executed by the server, the steps of the method described in any one of claims 1 to 12 are implemented.
14. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.
Citation Information
Patent Citations
Database transaction processing method and device and server
CN111427966A
Method and system for synchronizing heterogeneous data sources based on MongoDB metadata management
CN111737355A
Data management method and system, equipment and medium
CN112084224A
Database interface conversion system
CN112463796A
Metadata synchronization method based on MySQL binlog
CN114691704A
Cited By
Security institution business data distribution method and device based on directed link, and medium
CN122089472A
