Data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a data processing method and device, equipment and a storage medium, and the method comprises the steps: storing a data stream into a data queue according to a line storage format when the data stream is received, synchronizing the data stream in the data queue to subscription equipment of the data stream, and enabling the subscription equipment to process the data stream in real time; and obtaining structure information and storage index information of the data stream in the data queue. When the data queue meets the offline processing condition, generating a data index file matched with the offline processing engine according to the structure information and the storage index information of the data stream in the data queue; the data index file is sent to an offline processing engine, and the offline processing engine is used for reading the data flow from the data queue according to the data index file and performing offline processing on the read data flow. According to the invention, the complexity and maintenance cost of a data processing platform architecture can be reduced, and diversified requirements of users on data processing are met.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] Real-time data processing and massive data processing (i.e., offline data processing) are two important application scenarios in the field of big data. In order to accommodate both scenarios, the data processing platform architecture widely adopted in the industry requires two storage systems to provide data for the streaming processing system and the offline processing system, respectively. It is difficult to keep the behavior and processing logic of the two storage systems consistent, resulting in high overall complexity and maintenance costs, which cannot meet the diverse data processing needs of users. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and storage medium, which reduces the complexity and maintenance cost of the data processing platform architecture and meets users' diverse data processing needs.

[0004] One embodiment of this application provides a data processing method, including:

[0005] When a data stream is received, it is stored in a data queue according to the row storage format. The data stream in the data queue is then synchronized to the subscribing device of the data stream, which is used to process the data stream in real time.

[0006] Obtain the structure information and storage index information of the data stream in the above data queue;

[0007] When the above data queue meets the offline processing conditions, a data index file matching the offline processing engine is generated based on the structure information and storage index information of the data stream in the above data queue.

[0008] The aforementioned data index file is sent to the aforementioned offline processing engine, which reads the data stream from the aforementioned data queue based on the aforementioned data index file and processes the read data stream offline.

[0009] One embodiment of this application provides a data processing apparatus, including:

[0010] The synchronization module is used to store the data stream in a data queue according to the row storage format when the data stream is received, and to synchronize the data stream in the data queue to the subscription device of the data stream. The subscription device is used to process the data stream in real time.

[0011] The acquisition module is used to acquire the structure information and storage index information of the data stream in the above data queue;

[0012] The generation module is used to generate a data index file that matches the offline processing engine based on the structure information and storage index information of the data stream in the data queue when the above data queue meets the offline processing conditions.

[0013] The sending module is used to send the aforementioned data index file to the aforementioned offline processing engine. The aforementioned offline processing engine is used to read the data stream from the aforementioned data queue according to the aforementioned data index file and process the read data stream offline.

[0014] One aspect of this application provides a computer device, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0015] One embodiment of this application provides a computer storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described method.

[0016] One aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0017] This application provides a unified data processing platform architecture for lake-stream data. In this architecture, for any data stream, only one copy needs to be stored in the data queue according to the row-based storage format. The data stream in the data queue is then synchronized to the subscribing devices to achieve real-time processing of the data stream. Based on the data stream's structural information and storage index information, a data index file for the data stream is generated and sent to the offline processing engine. The offline processing engine can then perform offline processing of the data stream based on the data index file. Therefore, this data processing platform architecture only requires one copy of the data stream storage, meaning only one data storage system is needed to achieve both real-time and offline processing of the data stream. This reduces the data storage pressure, complexity, and maintenance costs of the data processing platform architecture, while meeting users' diverse needs for real-time performance, consistency, and flexibility in data processing. Attached Figure Description

[0018] Figure 1 This is a comparative diagram of the existing data processing platform architecture and the data processing platform architecture in this application;

[0019] Figure 2 This is a schematic diagram of a data processing platform architecture provided in this application;

[0020] Figure 3 This is a schematic diagram illustrating the interaction scenario between the client and server in a data processing platform architecture provided in this application;

[0021] Figure 4 This is a flowchart illustrating a data processing method provided in this application;

[0022] Figure 5 This is a schematic diagram illustrating a scenario where data flow in a data queue is migrated to a database instance, as provided in this application.

[0023] Figure 6 This is a flowchart illustrating another data processing method provided in this application;

[0024] Figure 7 This is a schematic diagram illustrating a scenario for generating a data index file, as provided in this application.

[0025] Figure 8 This is a schematic diagram illustrating a scenario for querying incremental data streams provided in this application;

[0026] Figure 9 This is a schematic diagram illustrating a scenario where the same directory is generated, as provided in this application.

[0027] Figure 10 This is a schematic diagram illustrating a scenario for implementing data stream analysis provided in this application;

[0028] Figure 11 This is a schematic diagram of the structure of a data processing device provided in this application;

[0029] Figure 12 This is a schematic diagram of the structure of a computer device provided in this application. Detailed Implementation

[0030] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0031] First, the relevant terms used in this application will be explained.

[0032] A data stream is an ordered sequence of bytes with a start and an end. A data stream corresponds to a data topic, which reflects the type, purpose, or field of the data stream. Types can include video, audio, and text, purposes can include data mining, data analysis, and data visualization, and fields can include finance, gaming, and medicine.

[0033] A data stream can include one or more data items, each corresponding to a field. The field reflects the meaning or data type of the data. For example, in a game scenario, a data stream could include game video data and game audio data. The field corresponding to game video data could be "video," and the field corresponding to game audio data could be "audio." In a product transaction scenario, a data stream could include basic product attribute information, transaction information, and production information. Basic attribute information could include product name, product purpose, and product identifier. The field corresponding to the product name is "name," the field corresponding to the product purpose is "purpose," and the field corresponding to the product identifier is "identifier." Transaction information could include transaction date, transaction amount, and transacting parties. The field corresponding to the transaction date could be "date," the field corresponding to the transaction amount could be "amount," and the field corresponding to the transacting parties could be "name" or "personal name."

[0034] Row-based storage format: This is a data storage method that organizes data in a data stream row by row. In row-based storage format, an entire row of data is treated as a single data stream and stored in the data queue of the data processing platform architecture. Row-based storage format has the following advantages: 1. It enables parallel storage of data streams, resulting in high storage efficiency. 2. Data consistency and transactionality are easier to achieve. Row-based storage format aligns better with the design principles of relational databases, emphasizing data consistency and transactionality, making it suitable for scenarios requiring high data consistency and transactionality. 3. High read efficiency. Because the data stream in row-based storage format is stored contiguously, reading a specific row only requires locating that row's position for reading, resulting in high efficiency.

[0035] Columnar storage format: This format stores data streams in columns, meaning each column of data in a table is considered a data stream. Columnar storage format offers the following advantages: 1. Simplified insertion and update operations: Because it stores data column-wise, inserting and updating data only requires operating on the columns that need modification, resulting in simpler and more efficient operations. 2. Data compression and query optimization advantages: Since the data type and volume of a column are uniform in columnar storage, more effective data compression is possible. Furthermore, query optimization can be performed on each column, improving query efficiency.

[0036] Data queue: also known as message queue, is a container that stores data streams during data transmission. The data stream in the data queue can be stored in row-oriented format.

[0037] The data queue in the data processing platform architecture can include at least one data table. Each data table corresponds to a table topic, which identifies the type, purpose, or domain of the data streams within the table. A table topic can include one or more topic partitions, which are physical divisions of the table topic. Each table topic can be divided into one or more topic partitions, and each topic partition is a physical subset of the table topic. A topic partition corresponds to one or more ledgers, which are the smallest units used to store data streams. A data table corresponds to one or more table partitions, which can be based on fields in the data table. For example, using months in the data table, partition p1 can contain the data streams for January, and partition p2 can contain the data streams for February. In other words, the data streams within a table partition are stored in one or more storage units.

[0038] Real-time processing refers to the process of processing data immediately after it is generated. The advantage of real-time processing is that it can analyze and process data streams in real time, thereby better responding to emergencies and abnormal situations. Application scenarios of real-time processing include real-time data analysis (such as real-time incremental rate, real-time access rate), real-time data visualization, and real-time data mining.

[0039] Offline processing: This refers to the process of batch processing large datasets while they are being collected. The advantage of offline processing lies in its ability to comprehensively analyze and process large datasets, leading to a better understanding of their characteristics and patterns. Applications of offline processing include data mining, data analysis (such as model training), and data visualization.

[0040] Data processing platform architecture: refers to the architecture used to store and process data streams. For example, ... Figure 1 As shown, Figure 1 This is a comparative diagram of the existing data processing platform architecture and the data processing platform architecture of this application. Figure 1 The existing data processing platform architecture includes two storage systems: a distributed messaging system and a data warehouse system. The distributed messaging system stores structured and semi-structured data streams collected in real-time from data sources and provides data streams to the real-time processing system (i.e., the streaming processing system). The data warehouse system stores structured and semi-structured data streams collected offline from data sources and provides data streams to the offline processing system. The real-time processing system can process the data streams in real-time, and the offline processing system can process the data streams in real-time to provide data services, such as data mining, data analysis, and data visualization.

[0041] It is evident that the existing data processing platform architecture 11 requires storing each data stream in both the distributed message system and the data warehouse system, increasing the data storage pressure and cost. Furthermore, the architecture requires maintaining two storage systems, further increasing its complexity and maintenance costs, and failing to meet the diverse data processing needs of users. Therefore, as... Figure 1 The data processing platform architecture 12 provided in this application can be used to store data streams collected in real time or offline from data sources. These data streams can refer to structured or semi-structured data. This data processing platform architecture 12 can provide data streams for both real-time and offline processing systems. This data processing platform architecture 12 only needs to store one copy of the data stream; that is, only one data storage system is required to realize both real-time and offline processing of the data stream. This helps reduce the data storage pressure on the data processing platform architecture 12, reduces the complexity and maintenance costs of the data processing platform architecture, and meets the diverse needs of users for real-time performance, consistency, and flexibility in data processing.

[0042] It should be noted that the data processing platform architecture 12 in this application can be built on Apache Pulsar, which inherently possesses data queue functionality. The data streams in the original data queue only support consumption and do not support real-time querying. Building upon this, this application innovatively introduces the concept of metadata records into the data processing platform architecture. Each time a producer sends a batch of data streams to the data processing platform architecture, a metadata record is generated in real time. Each metadata record contains the table statistics (structural information) and storage index information of this batch of data streams, and supports periodic merging with file index information (see description below). Data streams in the data queue can be directly queried through a unified query device, solving the problem that the original data queue does not support querying.

[0043] It should be noted that the data processing platform architecture 12 in this application can be referred to as a lake-stream integrated data processing platform architecture, that is, the real-time data processing system (i.e., streaming processing system) and the offline processing system (data lake processing system) share a data storage system (i.e., data queue).

[0044] To facilitate a clearer understanding of this application, we will first introduce... Figure 2 , Figure 2 This is a schematic diagram of a data processing platform architecture provided in this application. Figure 2 As shown, the data processing platform architecture may include a server 20 and a terminal cluster, which may include multiple terminals. Figure 2As shown, taking a terminal cluster consisting of three terminals as an example, the terminal cluster can specifically include a first terminal, a second terminal, and a third terminal. It can be understood that the first terminal, the second terminal, and the third terminal can all connect to the server via the network so that each terminal can interact with the server through the network connection.

[0045] Understandably, server 20 can be a single physical server, a server cluster or distributed system consisting of at least two physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud knowledge bases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals can specifically refer to in-vehicle terminals, smartphones, tablets, laptops, desktop computers, smart speakers, speakers with screens, smart TVs, smartwatches, etc., but are not limited to these. Each terminal and server 20 can be directly or indirectly connected via wired or wireless communication. Furthermore, the number of terminals and servers can be one or at least two; this application does not impose any restrictions.

[0046] The terminal can have a client installed on the data processing platform architecture. Through this client, the terminal can write data streams to the data processing platform architecture, query, read, and subscribe to data streams within the architecture, and process the read data streams. In other words, the terminal can refer to a subscription device, a query device, etc. The client can refer to a standalone application, a web application, or a mini-program within a host application.

[0047] Server 20 can be deployed with a data processing platform architecture, which can be used to store the written data stream and synchronize the data stream to subscribed devices, etc.

[0048] Specifically, such as Figure 3 As shown, Figure 3 This application provides a schematic diagram illustrating the interaction between the client and server in a data processing platform architecture. Figure 3 In this architecture, the data processing platform includes server 33, client 32, and client 36. Server 33 can be deployed on... Figure 2 In server 20, clients 32 and 36 can be deployed separately on... Figure 2 In the terminal, client 32 is deployed in Figure 2 The first terminal in the process, client 36, is deployed in Figure 2 The second terminal in the example will be used for illustration.

[0049] Client 32 can be used to read data streams from database 31 and write data streams to server 33. Client 36 can be used to read data streams from server 33 and send the read data streams to data processing system 37, which processes the data streams.

[0050] The data processing system 37 may include at least one of a streaming processing system or an offline processing system. The streaming processing system is used to process the data stream in real time, and the offline processing system is used to process the data stream offline.

[0051] Among them, the server-side component 33 is the core of the entire data processing platform architecture. Its role is to compensate for the shortcomings of the data lake (data warehouse system) in streaming scenarios (streaming processing scenarios) and solve problems such as streaming writes, streaming reads, zero-latency queries, and sequentiality in the data lake. It mainly includes middleware 34 and the data storage system 35; the middleware 34 may include a log writer, data migration service, transaction manager, subscription service (ManifestStore), metadata service (Manifest Service), and file service, etc.

[0052] The log editor can be an extension of Apache Pulsar's Log Writer. In addition to persisting the real-time data stream sent by the client 32, it will read the data stream from the client 32 and write it to the data queue of the data storage system 35, thereby achieving persistent storage of the data stream.

[0053] Among them, Apache Pulsar, one of the top-level projects of the Apache Software Foundation, is a distributed messaging system (i.e., a message publish / subscribe platform) that provides high-throughput, low-latency data streaming services and supports multi-tenancy, persistent storage, and stream processing.

[0054] Among them, the data migration service can also be called the multi-tenant service (Offload Service), which is a custom-implemented migration (Pulsar Offload) plugin in the data processing platform architecture. It is responsible for writing the data stream in the data queue to the database instance. The database instance can be a cheaper storage medium suitable for long-term storage. The database instance can include distributed file system (Hadoop Distributed File System, HDFS), simple storage service (S3 Simple Storage Service, S3), or cloud object storage (COS), etc.

[0055] In this system, one storage unit in the data queue corresponds to one migration thread. Each migration thread can group the data stream in the storage unit according to the field identifier corresponding to the data stream in the storage unit, obtain a data group, and migrate the data group to the database instance.

[0056] The transaction manager, which can be an extension of the transaction features of Apache Pulsar, is responsible for coordinating and managing distributed transactions, ensuring that data streams are delivered exactly once, and also supports unified transaction management for both streaming and incremental data streams.

[0057] The subscription service can be used to provide persistence and fault recovery capabilities for incremental data streams. This is achieved through a table topic of a data queue on the server side 33. After the data stream is stored in the data queue, the metadata record of the incremental data stream will be obtained synchronously. The metadata record includes structure information and storage index information. The subscription service can provide persistence capabilities for the incremental data stream through the structure information and storage index information.

[0058] The metadata service comprises a consumer thread, a manifest cache, and a commit thread, which are used to read metadata records from the subscription service, organize data index files, and commit data index files to the offline processing engine, respectively. The consumer thread retrieves the historical storage location corresponding to the last data stream read by the offline processing engine from its latest snapshot file. This historical storage location can refer to the latest consumption point of the offline processing engine. Based on this historical storage location, it reads the metadata records of the incremental data stream from the subscription service, adds the metadata records of the incremental data stream to the manifest cache, and updates the latest consumption point of the offline processing engine based on the storage location of the incremental data stream in the data queue.

[0059] The information cache can be used to group and store the metadata records of the incremental data stream into information groups. When the amount of data in the target information group is greater than the data amount threshold, it is determined that the data queue meets the offline processing conditions, and a data index file is generated based on the metadata records in the target information group.

[0060] The submission thread is used to submit the data index file in the information cache to the offline processing engine. Before submission, it locks the information cache, records the latest consumption point, serializes and compresses the latest consumption point into a string, calls the metadata submission interface, and submits the latest data index file and the latest consumption point to the offline processing engine. After submission, it calls the acknowledgement (ACK) interface of the data processing platform architecture to update the subscription consumption point of the metadata service.

[0061] Among them, the file service supports reading data index files, reading data streams from migration files in storage units and database instances through Pulsar Raw Reader, and providing direct querying of the corresponding data streams for entity code segments; it also supports client short-circuit reads, reducing the IO pressure caused by querying data processing platform architecture by replacing the ability to read data index files with migration files without the client's awareness of the data index files.

[0062] When the storage mode of the target storage unit in the data queue is the automatic optimization mode, after the data stream is stored in the target storage unit, the data stream in the target storage unit can be migrated to the data layer of the data warehouse database 38, and the data index file of the target storage unit is also stored in the metadata layer of the data warehouse 38. This is conducive to realizing batch query for the data stream.

[0063] In summary, the data processing platform architecture in this application only needs to store one copy of the data stream, that is, only one data storage system is needed to realize the implementation and offline processing of the data stream. This helps to reduce the data storage pressure of the data processing platform architecture, reduce the complexity and maintenance cost of the data processing platform architecture, and meet the diverse needs of users for real-time, consistency and flexibility of data processing.

[0064] Please see Figure 4 , Figure 4 This is a flowchart illustrating a data processing method provided in this application. Figure 4 As shown, this method can be derived from... Figure 2 The server in the middle can be used to execute it, or it can be executed by... Figure 2 The terminal and server in the process work together to execute the data processing method. The device used to execute this data processing method in this application can be collectively referred to as a computer device. The method may include the following steps:

[0065] S101. When a data stream is received, the data stream is stored in a data queue according to the row storage format, and the data stream in the data queue is synchronized to the subscription device of the data stream. The subscription device is used to process the data stream in real time.

[0066] In this application, when a data stream is received, the computer device can store the data stream in a row-based format into a data queue. This facilitates parallel storage of the data stream into the data queue, improving storage efficiency. The data stream in the data queue is synchronized to the subscribing devices that subscribe to the corresponding data stream in a first-in, first-out (FIFO) order. This enables real-time processing of the data stream through the subscribing devices, allowing for real-time writing and reading of the data stream, ensuring second-level data visibility.

[0067] In one embodiment, when a data stream is received, the data topic of the data stream can be determined based on its source or the data within it. A target data table matching the data stream topic in the data processing platform architecture can then be identified. The partitioning method corresponding to the table topic can be used to determine the target partition topic for the data stream, and the data stream can be stored in a storage unit under the target partition topic.

[0068] It should be noted that the data stream can be stored in multiple target storage units, which can prevent the data stream from being read from other storage units if a single storage unit fails.

[0069] Different subscription devices can pre-subscribe to the data topics corresponding to the data streams they need. For example, if subscription device 1 subscribes to the game topic, when a data stream belonging to the game topic is written into the message queue, the computer device can synchronize the data stream belonging to the game topic to subscription device 1. In this way, subscription device 1 can process the data stream belonging to the game topic in real time, improving the real-time processing of the data stream.

[0070] S102. Obtain the structure information and storage index information of the data stream in the above data queue.

[0071] In this application, the computer device can obtain the structural information of the data stream in the data queue and determine the storage index information of the data stream in the data queue based on the storage location of the data stream in the data queue.

[0072] Here, "data stream in the data queue" can refer to the incremental data stream within the data queue. An incremental data stream can be the data stream stored in the data queue within a specific time period, which can include the time from when the data queue last met the offline processing conditions to when it meets them again. Alternatively, "data stream in the data queue" can refer to all data streams in the data queue.

[0073] The structural information of the data stream can be referred to as the table statistics of the data stream. The structural information reflects the data attributes of the data stream and may include at least one of the following: partition identifier (spec Id), field identifier (schema Id), partition value (partition Data), and watermark value (watermark).

[0074] Field identifiers are used to identify the fields corresponding to data in a data stream. Partition identifiers reflect the operation permissions of the data stream, including add, delete, modify, and difference operations. Partition values ​​refer to the values ​​corresponding to partition fields in the data stream. Partition fields can be the fields used to partition a data table. For example, if the partition field is "month," and the data stream contains a company's revenue data for October, the partition value would be 10. Watermark values ​​refer to the tamper-proof identifiers of the data stream.

[0075] The storage index information can reflect the storage location, number of data records, and data volume in the data stream's data queue. This storage index information can include at least one of the following: table partition, topic partition index, ledger ID, entry ID, record count, and size in bytes. The record count can refer to the number of data records in the data stream, and the size in bytes can refer to the number of bytes in the data stream. The entry ID can reflect the order in which the data stream is stored in the storage unit; that is, the entry IDs of the data streams in each storage unit are consecutive. The topic partition index can refer to the partition identifier of the partition topic.

[0076] S103. When the above data queue meets the offline processing conditions, a data index file matching the offline processing engine is generated based on the structure information and storage index information of the data stream in the above data queue.

[0077] In this application, since the offline processing engine reads data streams as files, when the waiting data queue meets the offline processing conditions, the computer device can generate a data index file that matches the offline processing engine based on the structure information and storage index information of the data stream in the data queue. This is beneficial for the offline processing engine to read the data stream in the message queue.

[0078] In this context, matching the data index file with the offline processing engine can mean that the data index file is arranged according to the data arrangement format of the offline processing engine, and the structure information and storage index information of the data stream are arranged accordingly. In other words, the offline processing engine can identify each parameter in the data index file and read the data stream from the data queue based on the data index file.

[0079] The data index file refers to a file used to locate data streams in a data queue, reflecting the storage location and data attributes of the data streams within the queue. This data index file enables efficient fusion analysis of both full and incremental data streams. It also supports querying data streams in the data queue using a data lake table approach, resolving the issue of traditional data queues where data streams cannot be queried.

[0080] The offline processing conditions for a data queue can be one or more of the following: 1. The interval between the current time and the last time a data stream was processed offline is greater than a duration threshold; 2. The number of new data streams in the data queue is greater than the first threshold; 3. The number of incremental data streams in a certain storage unit is greater than the second threshold; 4. The number of incremental data streams belonging to a certain field is greater than the third threshold; 5. An offline processing request for a data stream in the data queue is received. The first, second, and third thresholds can be set empirical values.

[0081] In one embodiment, a computer device can generate a filename based on structural information and storage index information, and use the file with that filename as a data index file. For example, the filename may include the partition topic, the cell identifier of the storage unit, the field identifier, and the entity number of a batch of data streams.

[0082] S104. The above-mentioned data index file is sent to the above-mentioned offline processing engine, which is used to read the data stream from the above-mentioned data queue according to the above-mentioned data index file and process the read data stream offline.

[0083] In this application, a computer device can send the data index file to an offline processing engine, which can then read the data stream from a data queue or read the data stream based on the data index file and process the read data stream offline. By implementing offline processing of the data stream through a data queue, there is no need to store a separate copy of the data stream in the data warehouse system, reducing the storage pressure on the data processing platform architecture, lowering its complexity and maintenance costs, and meeting users' diverse needs for real-time performance, consistency, and flexibility in data processing.

[0084] In one instance, after a computer device obtains a data index file, it can store the data index file in an offline storage system. The offline processing engine then reads the data index file from the offline storage system, reads the data stream from the data queue based on the data index file, and performs offline processing on the data stream.

[0085] In one embodiment, the data queue includes K storage units, and the data stream is stored in these K storage units, where K is an integer greater than 1. Each storage unit corresponds to a migration thread. The computer device can use the i-th migration thread to group the data stream in the i-th storage unit according to the field identifier of the data stream in the i-th storage unit, obtaining at least one data group; i is an integer less than or equal to K, and the i-th storage unit corresponds to the i-th migration thread. The computer device can migrate the data stream in the i-th storage unit to the database instance according to the at least one data group, that is, according to the at least one data group, migrate the historical data stream or all data streams in the i-th storage unit to the database instance, and delete the data stream in the i-th storage unit. By migrating the data stream to the database instance, it is beneficial to achieve long-term storage of the data stream, reduce the storage cost of the data stream, and avoid the problem of data stream loss.

[0086] Here, "database instance" can refer to a storage medium suitable for storage durations longer than a duration threshold, i.e., a storage medium suitable for long-term storage; or, it can refer to a storage medium with storage costs lower than a cost threshold. "Database instance" can refer to a distributed file system (Hadoop Distributed File System, HDFS), a simple storage service (S3 Simple Storage Service, S3), or cloud object storage (COS), etc.

[0087] Here, historical data stream can refer to a data stream in the data queue whose storage duration exceeds a duration threshold, or it can refer to a data stream in the data queue that has already been read.

[0088] In one embodiment, the process of migrating the data stream in the i-th storage unit to the database instance according to at least one data group includes: a computer device generating a second entity number segment corresponding to each data group based on the entity number of the data stream in the at least one data group; and generating a migration file corresponding to each data group based on the second entity number segment of the data group and the unit identifier of the i-th storage unit, i.e., one migration file for each data group. Migrating the data stream within each data group in the at least one data group to the corresponding migration file, and migrating the migration files corresponding to each data group to the database instance, facilitates long-term storage and long-cycle querying of the data stream.

[0089] For example, the i-th storage unit includes data streams with field identifiers a1, a2, and a3. The i-th migration thread can group the data stream with field identifier a1 to data group b1, the data stream with field identifier a2 to data group b2, and the data stream with field identifier a3 to data group b3. Assume the entity numbers of the data streams within data group b2 are 1, 2, 3, and 4; the entity numbers of the data streams within data group b2 are 5, 6, 7, and 8; and the entity numbers of the data streams within data group b3 are 9 and 10. The second entity number segment corresponding to data group b1 can be 1-4, the second entity number segment corresponding to data group b2 can be 5-8, and the second entity number segment corresponding to data group b3 can be 9-10. The computer device can use the unit identifier of the i-th storage unit and the second entity number segment of data group b1 as the file name of migration file c1, the unit identifier of the i-th storage unit and the second entity number segment of data group b2 as the file name of migration file c2, and the unit identifier of the i-th storage unit and the second entity number segment of data group b3 as the file name of migration file c3. The data stream within data group b1 is migrated to migration file c1, the data stream within data group b2 is migrated to migration file c2, and the data stream within data group b3 is migrated to migration file c3. Migration files c1, c2, and c3 are then stored in the database instance, and the data stream in the i-th storage unit is deleted.

[0090] In one embodiment, a computer device can receive a first query request from a first query device for a database instance via a file service. The first query request includes a third entity number segment and a unit identifier for a storage unit. Based on the third entity number segment and the unit identifier of the storage unit in the query request, a migration file associated with the first query request is determined from the database instance. Specifically, migration files in the database instance whose filenames include the third entity number and the unit identifier of the storage unit are identified as migration files associated with the first query request. From the determined migration file, data streams whose entity numbers belong to the third entity number segment are queried, and the data streams belonging to the third entity number segment are returned to the first query device. By migrating the data stream to the database instance, long-term queries on the data stream are achieved. By migrating the data stream to the database instance, the problem that the original data queue does not support queries is solved.

[0091] For example, such as Figure 5As shown, computer devices can store structured and semi-structured data streams collected from data sources into a data queue within the data processing platform architecture. The data stream in this queue is then sent to a real-time data warehouse / data service for real-time processing. The computer devices can use migration services to migrate the data stream from the data queue to a more cost-effective database instance suitable for long-term storage. This allows offline processing engines or query devices to query the data stream from the database instance and process it offline. This enables the data processing platform architecture to support long-term queries even for data streams that were originally limited to short-cycle queries.

[0092] This application provides a unified data processing platform architecture for lake-stream data. In this architecture, for any data stream, only one copy needs to be stored in the data queue according to the row-based storage format. The data stream in the data queue is then synchronized to the subscribing devices to achieve real-time processing of the data stream. Based on the data stream's structural information and storage index information, a data index file for the data stream is generated and sent to the offline processing engine. The offline processing engine can then perform offline processing of the data stream based on the data index file. Therefore, this data processing platform architecture only requires one copy of the data stream storage, meaning only one data storage system is needed to achieve both real-time and offline processing of the data stream. This reduces the data storage pressure, complexity, and maintenance costs of the data processing platform architecture, while meeting users' diverse needs for real-time performance, consistency, and flexibility in data processing.

[0093] Please see Figure 6 , Figure 6 This is a flowchart illustrating another data processing method provided in this application. For example... Figure 6 As shown, this method can be derived from... Figure 2 The server in the middle can be used to execute it, or it can be executed by... Figure 2 The terminal and server in the process work together to execute the data processing method. The device used to execute this data processing method in this application can be collectively referred to as a computer device. The method may include the following steps:

[0094] S201. When a data stream is received, the data stream is stored in a data queue according to the row storage format, and the data stream in the data queue is synchronized to the subscription device of the data stream. The subscription device is used to process the data stream in real time.

[0095] S202. Obtain the structure information and storage index information of the data stream in the above data queue.

[0096] In one embodiment, step S102 may include: when the metadata service is in the started state, the computer device can obtain the historical storage location corresponding to the data stream last read by the offline processing engine from the latest snapshot file of the offline processing engine through the metadata service. The computer device can determine the unread data streams in the data queue based on the historical storage location, and identify these unread data streams as incremental data streams stored in the data queue. The computer device can obtain the structural information of the incremental data streams in the data queue, and generate storage index information for the incremental data in the data queue based on the storage location of the incremental data streams in the data queue. By obtaining the structural information and storage index information of the incremental data streams, it is possible to avoid repeatedly obtaining the structural information and storage index information of data streams that have already been read, thus saving resources.

[0097] S203. The structure information and storage index information of the incremental data stream in the above data queue are grouped and stored in M ​​information groups in the information buffer; M is a positive integer greater than 1, the above incremental data stream is the data stream newly stored in the above data queue within a time period, the above time period includes the time when the data queue last met the data reading conditions and the time when the data reading conditions are met this time.

[0098] In this application, the computer device can group and store the structure information and storage index information of incremental data streams in the data queue into M information groups in the information buffer. That is, the structure information and storage index information of incremental data streams with similar or identical characteristics in the data queue are grouped and stored into M information groups in the information buffer, which helps to improve the efficiency and convenience of data stream reading.

[0099] In one embodiment, a storage unit corresponds to at least one information group. Step S203 may include: the computer device may, according to the storage units where the incremental data streams in the data queue are stored, group and store the structure information and storage index information of the incremental data streams into M information groups in the information buffer. The incremental data streams are stored in at least one storage unit of the data queue. That is, storing the structure information and storage index information of data streams stored in the same storage unit into one or more information groups facilitates the subsequent generation of a data index file for the data streams stored in the same storage unit. Based on the data index file, the data streams stored in the same storage unit can be read, thus improving reading efficiency.

[0100] In one embodiment, a computer device can group and store the structure information and storage index information of incremental data streams belonging to the same storage unit and the same table partition into the same information group. This facilitates the one-time reading of incremental data streams belonging to the same storage unit and the same table partition.

[0101] In one embodiment, an information group can correspond to a field identifier, which reflects the field corresponding to the data in the data stream. Step S203 can include: the computer device can group and store the structure information and storage index information of the incremental data stream into the M information groups according to the field identifiers corresponding to the incremental data streams in the data queue. That is, storing the structure information and storage index information of data streams with the same field into the same information group facilitates the subsequent generation of a data index file for data streams with the same field. Based on the data index file, data streams with the same field can be read, which is beneficial for offline processing of data streams with the same field.

[0102] S204. When there is a target information group among the above M information groups whose data volume is greater than the data volume threshold, it is determined that the above data queue meets the offline processing conditions.

[0103] In this application, the computer device can count the amount of data in M ​​information groups. When there is a target information group in the above M information groups whose data amount is greater than the data amount threshold, the number of a certain type of data stream is relatively large, and the computer device can determine that the data queue meets the offline processing conditions.

[0104] In one embodiment, a computer device can count the number of data streams corresponding to the structural information and storage index information stored in M ​​information groups. When there is a target information group in the M information groups whose corresponding number of data streams is greater than the number of data streams threshold, the computer device can determine that the data queue meets the offline processing conditions. The number of data streams threshold here can refer to the second or third number of data streams threshold mentioned above.

[0105] S205. Based on the structural information and storage index information of the incremental data stream in the above target information group, generate a data index file that matches the offline processing engine.

[0106] In this application, the computer device can generate a data index file that matches the offline processing engine based on the structure information and storage index information of the incremental data stream in the target information group. This is beneficial for reading the incremental data stream in the data queue through the data index file, without having to repeatedly read the data stream that has already been read in the data queue, thus saving resources and improving the reading efficiency of the data stream.

[0107] In one embodiment, the storage index information includes the entity number, storage unit identifier, table partition, and partition topic corresponding to the incremental data streams in the target information group; the structure information includes the field identifier of the incremental data streams in the target information group. Step S205 may include: the computer device generating a first entity number segment based on the entity number corresponding to the incremental data streams in the target information group, i.e., the first entity number segment reflects the range to which the entity numbers corresponding to the incremental data streams in the target information group belong. The computer device may arrange the first entity number segment, the field identifier of the incremental data streams in the target information group, the storage unit identifier and partition topic, and the table partition according to the data arrangement format of the offline processing engine to obtain a data index file matching the offline processing engine. The data index file facilitates rapid reading of the incremental data streams in the data queue, improving the data stream reading efficiency.

[0108] For example, such as Figure 7As shown, taking a table topic in the data queue as an example, this table topic includes partition topic 1, partition topic 2, and partition topic 3. Partition topic 1 corresponds to storage unit 1 and storage unit 2, partition topic 2 corresponds to storage unit 3, and partition topic 3 corresponds to storage unit 4. The unit identifiers of storage unit 1, storage unit 2, storage unit 3, and storage unit 4 are L1, L2, L3, and L4, respectively. Storage unit 1 stores the data stream of entity numbers 1-3 in table partition p1, storage unit 2 stores the data stream of entity numbers 4-5 in table partition p2, storage unit 3 stores the data stream of entity numbers 2-6 in table partition p1, and storage unit 4 stores the data stream of entity numbers 1-5 in table partition p2. The computer device can generate a data index file f1 for the data stream with entity numbers 1-3 in table partition p1. The filename of data index file f1 can be P1 / L1_1-3.log. Data streams belonging to table partition p1 and with entity numbers 1-3 in storage unit 1 can be read using data index file f1. Similarly, the computer device can generate a data index file f2 for the data stream with entity numbers 4-5 in table partition p2. The filename of data index file f2 can be P2 / L1_4-5.log. Data streams belonging to table partition p2 and with entity numbers 4-5 in storage unit 1 can be read using data index file f2. Finally, the computer device can generate a data index file f3 for the data stream with entity numbers 2-6 in table partition p1. The filename of data index file f3 can be P1 / L3_2-6.log. Data streams belonging to table partition p1 and with entity numbers 2-6 in storage unit 3 can be read using data index file f3. The computer device can generate a data index file f4 for the data stream in table partition p2 whose entity numbers belong to 1-5. The filename of data index file f4 can be P2 / L4_1-5.log. Through data index file f4, the data stream in storage unit 4 belonging to table partition p2 and whose entity numbers belong to 1-5 can be read. When the storage mode of storage unit 1 is in automatic optimization mode, the computer device can convert the data stream of storage unit 1 into columnar storage format and store it in the data lake service.

[0109] S206. The above-mentioned data index file is sent to the above-mentioned offline processing engine, which is used to read the data stream from the above-mentioned data queue according to the above-mentioned data index file and process the read data stream offline.

[0110] In this application, by sending the aforementioned data index file to the aforementioned offline processing engine, the aforementioned offline processing engine is used to read incremental data streams from the aforementioned data queue according to the aforementioned data index file, and to process the read incremental data streams offline, without needing to read the full data stream each time, thus saving resources.

[0111] In one embodiment, the incremental data stream in the aforementioned data queue is stored in M ​​storage units of the data queue, where M is a positive integer greater than 1. The computer device can determine the target storage unit with an automatic optimization mode from these M storage units. Following a columnar storage format, the incremental data stream in the target storage unit is written to the data lake service; that is, after the incremental data stream in the target storage unit is synchronized to the subscribing devices, the computer device can write the incremental data stream in the target storage unit to the data lake service according to the columnar storage format and delete the incremental data stream in the target storage unit. The computer device can generate a snapshot identifier and index metadata associated with the target storage unit based on the storage location information of the incremental data stream in the target storage unit within the data lake service. The snapshot identifier and index metadata are used to query the incremental data stream corresponding to the target storage unit from the data lake service. By writing the incremental data stream in the target storage unit to the data lake service, it is beneficial to achieve batch querying of the incremental data stream, improve the query efficiency of the incremental data stream, and facilitate offline processing of the incremental data stream. By migrating data streams to data lake services, the capabilities of data queues and data lakes are integrated. Through a unified real-time read / write layer, users can achieve zero-latency queries by directly connecting to online services through the offline processing engine in real-time multidimensional analysis scenarios, improving data stream visibility from minutes to seconds.

[0112] The automatic optimization mode refers to converting the data stream stored in the target storage unit according to the row storage format into the column storage format and rewriting it to the data lake service when the data stream is stored in the target storage unit. In other words, it realizes writing the data stream according to the row storage format and storing the data stream according to the column storage format. This is conducive to improving the writing efficiency of the data stream, and storing the data stream in the column storage format improves the efficiency of batch querying the data stream, eliminates the dual storage redundancy problem in the data processing platform architecture, and optimizes storage efficiency and performance.

[0113] Data lake services can refer to components in a data warehouse system that provide data streams to offline processing systems. The data streams in the data warehouse system are stored in a columnar storage format.

[0114] In one embodiment, the incremental data stream in the target storage unit is stored in S data files within the data lake service, where S is an integer greater than 1. The storage location information includes file attribute information corresponding to each of the S data files. Generating a snapshot identifier and index metadata associated with the target storage unit based on the storage location information of the incremental data stream in the target storage unit within the data lake service includes: a computer device generating file index information corresponding to each of the S data files based on the file attribute information; generating list index information based on the file index information; and generating a snapshot identifier associated with the target storage unit based on the list index information. The list index information is then determined as the index metadata associated with the target storage unit.

[0115] The file attribute information includes the size, storage location, partition information, and column statistics (such as the maximum value, minimum value, and number of null values ​​for each column). The file index information (Manifest File) can be a metadata file used to list the file attribute information of the data file; that is, each line of the file index information includes the file attribute information of one data file.

[0116] The manifest list is a metadata file that lists all file index information for a table snapshot. The metadata file stores a list of file index information, with each row containing one file index. A table snapshot reflects the state of a data table at a specific point in time. Each table snapshot lists the manifest index information for that table at that point. Each table snapshot has a snapshot identifier, and the snapshot identifier associated with the target storage unit can be used to identify the snapshot table of the data files stored in the incremental data stream within the target storage unit.

[0117] In one embodiment, a computer device can receive a second query request from a second query device for an incremental data stream in the data queue; and, through a metadata interface, return a snapshot identifier associated with the target storage unit and a data index file corresponding to the incremental data stream to the second query device based on the second query request. The second query device is used to obtain index metadata associated with the target storage unit based on the snapshot identifier, query the incremental data stream corresponding to the target storage unit from the data lake service based on the index metadata, and query the incremental data stream from the remaining storage units in the data queue based on the data index file. The remaining storage units are the storage units other than the target storage unit among the M storage units. Through the snapshot identifier and the data index file, it supports reading incremental data streams that have been persisted to the data queue of the data processing platform architecture but not submitted to the data lake service, achieving zero-latency querying of the data lake service.

[0118] As data streams are written to and deleted from tables in the data lake service over time, the table snapshots change. Therefore, sending the snapshot identifier to the second query device as the latest snapshot identifier corresponding to the target storage unit helps ensure that the latest snapshot identifier in the data lake is consistent with the latest snapshot identifier received by the second query device.

[0119] It should be noted that the second query request can be an extended query syntax, such as: select * from db.table.rt, where .rt can be used to indicate that the query syntax is an extended query syntax, and select * from db.table.rt is used to indicate that the data lake service and data queue are queried to obtain incremental data streams.

[0120] For example, such as Figure 8As shown, the data lake service includes table 1. The metadata catalog of the data lake service indicates that the current metadata pointer is located at table 1 in database 1. The metadata file of the data lake includes a snapshot of table 1 at a first time and a snapshot of table 1 at a second time. The snapshot identifier of table 1 at the first time is s0, and the snapshot identifier of table 1 at the second time is s1. The second time is later than the first time. The table snapshot at the first time includes list index information list1, which includes file index information file1 and file index information file2. File index information file1 includes file attribute information of data file d1, and file index information file2 includes file attribute information of data file d2. The table snapshot at the second time includes list index information list2, which includes file index information file1, file index information file2, and file index information file3. File index information file3 includes file attribute information of data file d3. Data files d1, d2, and d3 are all stored in the data lake service, while data file d4 is stored in a storage unit in the data queue. The data index file file4 is used to locate the incremental data stream in the storage unit of the data queue.

[0121] Upon receiving a second query request from the second query device for the incremental data stream in the aforementioned data queue, the second query device sends S1 (latest snapshot identifier) ​​and data index file file4 to the second query device via the metadata interface. The second query device can obtain index metadata based on S1, which includes list index information list2. Based on list index information list2, it determines file index information file1, file index information file2, and file index information file3. Based on file index information file1, file index information file2, and file index information file3, it reads the incremental data stream from data file d1, data file d2, and data file d3 respectively. Finally, based on data index file file4, it reads the incremental data stream from the storage unit corresponding to the data queue.

[0122] In one embodiment, this application provides a unified directory, such as Figure 9 As shown, the data processing platform architecture uniformly uses data index files to locate data flow tables, data lake tables, and integrated lake-flow tables. From the user's perspective, only one type of table is perceived, eliminating the need to maintain multiple metadata systems. The metadata records of the data tables corresponding to the data queues are uniformly stored in the data index files. The data processing engine can use the data index files to create, read, and modify (edit) data tables, and the metadata service in the middleware can read data tables through the data index files.

[0123] In one embodiment, this application enables real-time analysis of data streams, meaning the data processing platform architecture has flexible scalability. Figure 10 As shown, it supports rapid integration with Online Analytical Processing (OLAP) engines. For example, the data processing platform architecture can be used as an external data table for querying, storing the query results in the database. Through the full data stream query capability provided by the data processing platform architecture, it can realize operational data storage, detailed data warehouse analysis, and data warehouse aggregation to provide users with various application services.

[0124] This application provides a unified data processing platform architecture for lake-stream data. In this architecture, for any data stream, only one copy needs to be stored in the data queue according to the row-based storage format. The data stream in the data queue is then synchronized to the subscribing devices to achieve real-time processing of the data stream. Based on the data stream's structural information and storage index information, a data index file for the data stream is generated and sent to the offline processing engine. The offline processing engine can then perform offline processing of the data stream based on the data index file. Therefore, this data processing platform architecture only requires one copy of the data stream storage, meaning only one data storage system is needed to achieve both real-time and offline processing of the data stream. This reduces the data storage pressure, complexity, and maintenance costs of the data processing platform architecture, while meeting users' diverse needs for real-time performance, consistency, and flexibility in data processing.

[0125] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a data processing device provided in this application. Figure 11 As shown, the data processing apparatus may include:

[0126] The synchronization module 1101 is used to store the data stream in a data queue according to the row storage format when the data stream is received, and to synchronize the data stream in the data queue to the subscription device of the data stream. The subscription device is used to process the data stream in real time.

[0127] The acquisition module 1102 is used to acquire the structure information and storage index information of the data stream in the above data queue;

[0128] The generation module 1103 is used to generate a data index file that matches the offline processing engine based on the structure information and storage index information of the data stream in the data queue when the above data queue meets the offline processing conditions.

[0129] The sending module 1104 is used to send the data index file to the offline processing engine, and the offline processing engine is used to read the data stream from the data queue according to the data index file and process the read data stream offline.

[0130] Optionally, module 1103 is generated, specifically for:

[0131] The structure information and storage index information of the incremental data stream in the aforementioned data queue are grouped and stored in M ​​information groups in the information buffer; M is a positive integer greater than 1. The aforementioned incremental data stream is the data stream newly stored in the aforementioned data queue within a time period, which includes the time when the data queue last met the data read conditions and the time when the data read conditions are met this time.

[0132] When there is a target information group among the above M information groups whose data volume is greater than the data volume threshold, it is determined that the above data queue meets the offline processing conditions.

[0133] Based on the structural information and storage index information of the incremental data stream in the above target information grouping, a data index file matching the offline processing engine is generated.

[0134] Optionally, module 1103 is generated, specifically for:

[0135] According to the storage units where the incremental data streams in the aforementioned data queue are stored, the structure information and storage index information of the incremental data streams are grouped and stored in M ​​information groups of the information buffer, and the incremental data streams are stored in at least one storage unit of the aforementioned data queue; or,

[0136] Based on the field identifiers corresponding to the incremental data streams in the data queues, the structure information and storage index information of the incremental data streams are grouped and stored into the M information groups.

[0137] Optionally, the above storage index information includes the entity number, storage unit identifier, and partition topic corresponding to the incremental data streams in the above target information group; the above structure information includes the field identifiers of the incremental data streams in the above target information group.

[0138] Optionally, module 1103 is generated, specifically for:

[0139] Based on the entity numbers corresponding to the incremental data streams in the above target information groupings, generate the first entity number segment;

[0140] The field identifiers of the incremental data streams in the first entity number segment, the cell identifiers of the storage units, and the partition topics in the target information group are arranged to obtain a data index file that matches the offline processing engine.

[0141] Optionally, module 1102 is used specifically for:

[0142] When the metadata service is running, the historical storage location corresponding to the data stream last read by the offline processing engine can be obtained from the latest snapshot file of the offline processing engine through the metadata service.

[0143] Based on the aforementioned historical storage locations, determine the incremental data stream stored in the aforementioned data queue;

[0144] Obtain the structural information of the incremental data stream in the data queue, and generate the storage index information of the incremental data in the data queue based on the storage location of the incremental data stream in the data queue.

[0145] Optionally, the data queue includes K storage units, and the data stream is stored in the K storage units, where K is an integer greater than 1, and one storage unit corresponds to one migration thread.

[0146] Optionally, the data processing device may also include a migration module 1105;

[0147] The migration module 1105 is used to group the data stream in the i-th storage unit according to the field identifier of the data stream in the i-th storage unit through the i-th migration thread to obtain at least one data group; i is an integer less than or equal to K, and the i-th storage unit corresponds to the i-th migration thread.

[0148] According to at least one of the above data groups, the data stream in the i-th storage unit is migrated to the database instance.

[0149] Optional, migration module 1105, specifically used for:

[0150] Based on the entity number of the data stream of at least one of the data groups mentioned above, generate a second entity number segment corresponding to each data group.

[0151] Based on the second entity number segment of the data group in at least one of the above data groups and the unit identifier of the i-th storage unit, a migration file corresponding to each data group is generated.

[0152] Migrate the data streams within each data group in at least one of the above data groups to the corresponding migration files, and migrate the migration files corresponding to each data group to the database instance.

[0153] Optionally, the data processing device may also include a query module 1106;

[0154] The query module 1106 is used to receive a first query request from the first query device for the database instance via a file service. The first query request includes a third entity number segment and a unit identifier of the storage unit.

[0155] Based on the third entity number segment and the unit identifier of the storage unit in the above data query request, the migration file associated with the above first query request is determined from the above database instance;

[0156] From the determined migration file, query the data stream whose entity number belongs to the aforementioned third entity number segment, and return the data stream belonging to the aforementioned third entity number segment to the aforementioned first query device.

[0157] Optionally, the incremental data stream in the aforementioned data queue is stored in M ​​storage units of the aforementioned data queue, where M is a positive integer greater than 1. The synchronization module 1101 is used for:

[0158] From the above M storage units, the target storage unit with the automatic optimization mode is determined;

[0159] According to the columnar storage format, the incremental data stream in the target storage unit is written to the data lake service;

[0160] Based on the storage location information of the incremental data stream in the target storage unit in the data lake service, a snapshot identifier and index metadata associated with the target storage unit are generated; the snapshot identifier and the index metadata are used to query the incremental data stream corresponding to the target storage unit from the data lake service.

[0161] Optionally, the incremental data stream in the target storage unit is stored in S data files in the data lake service, where S is an integer greater than 1, and the storage location information includes the file attribute information corresponding to the S data files respectively.

[0162] Synchronization module 1101 is used to generate file index information corresponding to the S data files respectively based on the file attribute information corresponding to the S data files respectively;

[0163] Based on the file index information corresponding to the above S data files, generate list index information, and based on the above list index information, generate snapshot identifiers associated with the above target storage units;

[0164] The above list index information and the above file index information are identified as index metadata associated with the above target storage unit.

[0165] Optionally, the query module 1106 is used to receive a second query request from the second query device for the incremental data stream in the aforementioned data queue;

[0166] Through the metadata interface, based on the second query request, the snapshot identifier associated with the target storage unit and the data index file corresponding to the incremental data stream are returned to the second query device.

[0167] The second query device is used to obtain index metadata associated with the target storage unit based on the snapshot identifier, query the incremental data stream corresponding to the target storage unit from the data lake service based on the index metadata, and query the incremental data stream from the remaining storage units in the data queue based on the data index file. The remaining storage units are the storage units other than the target storage unit among the M storage units.

[0168] This application provides a unified data processing platform architecture for lake-stream data. In this architecture, for any data stream, only one copy needs to be stored in the data queue according to the row-based storage format. The data stream in the data queue is then synchronized to the subscribing devices to achieve real-time processing of the data stream. Based on the data stream's structural information and storage index information, a data index file for the data stream is generated and sent to the offline processing engine. The offline processing engine can then perform offline processing of the data stream based on the data index file. Therefore, this data processing platform architecture only requires one copy of the data stream storage, meaning only one data storage system is needed to achieve both real-time and offline processing of the data stream. This reduces the data storage pressure, complexity, and maintenance costs of the data processing platform architecture, while meeting users' diverse needs for real-time performance, consistency, and flexibility in data processing.

[0169] Please see Figure 12 , Figure 12 This is a schematic diagram of the structure of a computer device provided in this application. Figure 12As shown, the aforementioned computer device 1000 can refer to a server or terminal, including: a processor 1001, a network interface 1004, and a memory 1005. Furthermore, the aforementioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. The communication bus 1002 is used to implement communication between these components. In some embodiments, the user interface 1003 may include a display screen and a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as at least one disk storage device. The memory 1005 may also optionally be at least one storage device located remotely from the aforementioned processor 1001. Figure 12 As shown, the memory 1005, which is a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and computer programs.

[0170] Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0171] When a data stream is received, it is stored in a data queue according to the row storage format. The data stream in the data queue is then synchronized to the subscribing device of the data stream, which is used to process the data stream in real time.

[0172] Obtain the structure information and storage index information of the data stream in the above data queue;

[0173] When the above data queue meets the offline processing conditions, a data index file matching the offline processing engine is generated based on the structure information and storage index information of the data stream in the above data queue.

[0174] The aforementioned data index file is sent to the aforementioned offline processing engine, which reads the data stream from the aforementioned data queue based on the aforementioned data index file and processes the read data stream offline.

[0175] Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0176] The structure information and storage index information of the incremental data stream in the aforementioned data queue are grouped and stored in M ​​information groups in the information buffer; M is a positive integer greater than 1. The aforementioned incremental data stream is the data stream newly stored in the aforementioned data queue within a time period, which includes the time when the data queue last met the data read conditions and the time when the data read conditions are met this time.

[0177] When there is a target information group among the above M information groups whose data volume is greater than the data volume threshold, it is determined that the above data queue meets the offline processing conditions.

[0178] Based on the structural information and storage index information of the incremental data stream in the above target information grouping, a data index file matching the offline processing engine is generated.

[0179] Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0180] According to the storage units where the incremental data streams in the aforementioned data queue are stored, the structure information and storage index information of the incremental data streams are grouped and stored in M ​​information groups of the information buffer, and the incremental data streams are stored in the M storage units of the aforementioned data queue; or,

[0181] Based on the field identifiers corresponding to the incremental data streams in the data queues, the structure information and storage index information of the incremental data streams are grouped and stored into the M information groups.

[0182] Optionally, the above storage index information includes the entity number, storage unit identifier, and partition topic corresponding to the incremental data streams in the above target information group; the above structure information includes the field identifiers of the incremental data streams in the above target information group.

[0183] Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0184] Based on the entity numbers corresponding to the incremental data streams in the above target information groupings, generate the first entity number segment;

[0185] The field identifiers of the incremental data streams in the first entity number segment, the cell identifiers of the storage units, and the partition topics in the target information group are arranged to obtain a data index file that matches the offline processing engine.

[0186] Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0187] When the metadata service is running, the historical storage location corresponding to the data stream last read by the offline processing engine can be obtained from the latest snapshot file of the offline processing engine through the metadata service.

[0188] Based on the aforementioned historical storage locations, determine the incremental data stream stored in the aforementioned data queue;

[0189] Obtain the structural information of the incremental data stream in the data queue, and generate the storage index information of the incremental data in the data queue based on the storage location of the incremental data stream in the data queue.

[0190] Optionally, the data queue includes K storage units, and the data stream is stored in the K storage units, where K is an integer greater than 1. Each storage unit corresponds to one migration thread. Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0191] Using the i-th migration thread, the data stream in the i-th storage unit is grouped according to the field identifier of the data stream in the i-th storage unit to obtain at least one data group; i is an integer less than or equal to K, and the i-th storage unit corresponds to the i-th migration thread.

[0192] According to at least one of the above data groups, the data stream in the i-th storage unit is migrated to the database instance.

[0193] Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0194] Based on the entity number of the data stream of at least one of the data groups mentioned above, generate a second entity number segment corresponding to each data group.

[0195] Based on the second entity number segment of the data group in at least one of the above data groups and the unit identifier of the i-th storage unit, a migration file corresponding to each data group is generated.

[0196] Migrate the data streams within each data group in at least one of the above data groups to the corresponding migration files, and migrate the migration files corresponding to each data group to the database instance.

[0197] Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0198] The file service receives a first query request from the first query device for the database instance, the first query request including a third entity number segment and a unit identifier of the storage unit;

[0199] Based on the third entity number segment and the unit identifier of the storage unit in the above data query request, the migration file associated with the above first query request is determined from the above database instance;

[0200] From the determined migration file, query the data stream whose entity number belongs to the aforementioned third entity number segment, and return the data stream belonging to the aforementioned third entity number segment to the aforementioned first query device.

[0201] Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0202] From the above M storage units, the target storage unit with the automatic optimization mode is determined;

[0203] According to the columnar storage format, the incremental data stream in the target storage unit is written to the data lake service;

[0204] Based on the storage location information of the incremental data stream in the target storage unit in the data lake service, a snapshot identifier and index metadata associated with the target storage unit are generated; the snapshot identifier and the index metadata are used to query the incremental data stream corresponding to the target storage unit from the data lake service.

[0205] Optionally, the incremental data stream in the target storage unit is stored in S data files in the data lake service, where S is an integer greater than 1, and the storage location information includes the file attribute information corresponding to the S data files respectively.

[0206] Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0207] Based on the file attribute information corresponding to the above S data files, generate file index information corresponding to the above S data files respectively;

[0208] Based on the file index information corresponding to the above S data files, generate list index information, and based on the above list index information, generate snapshot identifiers associated with the above target storage units;

[0209] The above list index information and the above file index information are identified as index metadata associated with the above target storage unit.

[0210] Optionally, the processor 1001 can be used to call a computer program stored in the memory 1005 to implement the following steps of this application:

[0211] Receive a second query request from the second query device for the incremental data stream in the aforementioned data queue;

[0212] Through the metadata interface, based on the second query request, the snapshot identifier associated with the target storage unit and the data index file corresponding to the incremental data stream are returned to the second query device.

[0213] The second query device is used to obtain index metadata associated with the target storage unit based on the snapshot identifier, query the incremental data stream corresponding to the target storage unit from the data lake service based on the index metadata, and query the incremental data stream from the remaining storage units in the data queue based on the data index file. The remaining storage units are the storage units other than the target storage unit among the M storage units.

[0214] This application provides a unified data processing platform architecture for lake-stream data. In this architecture, for any data stream, only one copy needs to be stored in the data queue according to the row-based storage format. The data stream in the data queue is then synchronized to the subscribing devices to achieve real-time processing of the data stream. Based on the data stream's structural information and storage index information, a data index file for the data stream is generated and sent to the offline processing engine. The offline processing engine can then perform offline processing of the data stream based on the data index file. Therefore, this data processing platform architecture only requires one copy of the data stream storage, meaning only one data storage system is needed to achieve both real-time and offline processing of the data stream. This reduces the data storage pressure, complexity, and maintenance costs of the data processing platform architecture, while meeting users' diverse needs for real-time performance, consistency, and flexibility in data processing.

[0215] Furthermore, it should be noted that this application also provides a computer-readable storage medium storing a computer program executed by the aforementioned data processing apparatus. This computer program includes program instructions, which, when executed by the processor, enable the execution of the data processing method described in the corresponding embodiments above. Therefore, further details will not be repeated here. Additionally, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer-readable storage medium involved in this application, please refer to the description of the method embodiments of this application.

[0216] As an example, the above program instructions can be deployed and executed on a computer device, or deployed and executed on at least two computer devices in one location, or executed on at least two computer devices distributed in at least two locations and interconnected by a communication network. At least two computer devices distributed in at least two locations and interconnected by a communication network can form a blockchain network.

[0217] The aforementioned computer-readable storage medium may be a data processing apparatus provided in any of the foregoing embodiments or a central storage unit of the aforementioned computer device, such as a hard disk or central storage of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart memory card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device. Furthermore, the computer-readable storage medium may include both the central storage unit and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0218] The terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish content in different media, rather than to describe a specific order. Furthermore, the term "comprising," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules not listed, or may optionally include other step units inherent to these processes, methods, apparatuses, products, or devices.

[0219] In practice, the collection and processing of data in this application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the data subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the data subject.

[0220] This application also provides a computer program product, including a computer program. When executed by a processor, the computer program implements the data processing method and decoding method described in the preceding embodiments, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the embodiments of the computer program product involved in this application, please refer to the description of the method embodiments of this application.

[0221] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0222] The methods and related apparatus provided in this application are described with reference to the method flowcharts and / or structural diagrams provided in this application. Specifically, each block of the method flowchart and / or structural diagram, as well as combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable network-connected device to create a machine, such that the instructions, which execute via the processor of the computer or other programmable network-connected device, generate instructions for implementing the process. Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable network-connected device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 A schematic diagram of one or more processes and / or structures. Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable network-connected device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 A process or multiple processes and / or structures illustrate the steps of the functions specified in one or more boxes.

[0223] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, include: When a data stream is received, it is stored in a data queue according to the row storage format, and the data stream in the data queue is synchronized to the subscription device of the data stream. The subscription device is used to process the data stream in real time. Obtain the structure information and storage index information of the data stream in the data queue; When the data queue meets the offline processing conditions, a data index file matching the offline processing engine is generated based on the structure information and storage index information of the data stream in the data queue. The data index file is sent to the offline processing engine, which reads the data stream from the data queue according to the data index file and processes the read data stream offline.

2. The method according to claim 1, characterized in that, When the data queue meets the offline processing conditions, a data index file matching the offline processing engine is generated based on the structure information and storage index information of the data stream in the data queue, including: The structure information and storage index information of the incremental data stream in the data queue are grouped and stored in M ​​information groups in the information buffer; M is a positive integer greater than 1. The incremental data stream is the data stream newly stored in the data queue within a time period, and the time period includes the time when the data queue last met the data read conditions and the time when the data read conditions are met this time. When there is a target information group among the M information groups whose data volume is greater than the data volume threshold, it is determined that the data queue meets the offline processing conditions. Based on the structural information and storage index information of the incremental data stream in the target information group, a data index file matching the offline processing engine is generated.

3. The method according to claim 2, characterized in that, The step of grouping and storing the structure information and storage index information of the incremental data stream in the data queue into M information groups in the information buffer includes: According to the storage units where the incremental data streams in the data queue are stored, the structure information and storage index information of the incremental data streams are grouped and stored in M ​​information groups of the information buffer, and the incremental data streams are stored in at least one storage unit of the data queue; or, According to the field identifiers corresponding to the incremental data streams in the data queue, the structure information and storage index information of the incremental data streams are grouped and stored into the M information groups.

4. The method according to claim 2, characterized in that, The storage index information includes the entity number, storage unit identifier, partition topic, and table partition corresponding to the incremental data streams in the target information group; the structure information includes the field identifiers of the incremental data streams in the target information group. The step of generating a data index file matching the offline processing engine based on the structure information and storage index information of the incremental data stream in the target information group includes: Based on the entity numbers corresponding to the incremental data streams in the target information group, a first entity number segment is generated; The first entity number segment, table partition, field identifiers of incremental data streams in the target information group, unit identifiers of storage units, and partition topics are arranged to obtain a data index file that matches the offline processing engine.

5. The method according to claim 1, characterized in that, The step of obtaining the structure information and storage index information of the data stream in the data queue includes: When the metadata service is running, the historical storage location corresponding to the data stream last read by the offline processing engine can be obtained from the latest snapshot file of the offline processing engine through the metadata service. Based on the historical storage location, determine the incremental data stream stored in the data queue; Obtain the structure information of the incremental data stream in the data queue, and generate the storage index information of the incremental data in the data queue according to the storage location of the incremental data stream in the data queue.

6. The method according to claim 1, characterized in that, The data queue comprises K storage units, and the data stream is stored in the K storage units, where K is an integer greater than 1. Each storage unit corresponds to one migration thread. The method further includes: Using the i-th migration thread, the data stream in the i-th storage unit is grouped according to the field identifier of the data stream in the i-th storage unit to obtain at least one data group; i is an integer less than or equal to K, and the i-th storage unit corresponds to the i-th migration thread; According to the at least one data group, the data stream in the i-th storage unit is migrated to the database instance.

7. The method according to claim 6, characterized in that, The step of migrating the data stream in the i-th storage unit to the database instance according to the at least one data group includes: Based on the entity number of the data stream of the data group in the at least one data group, generate a second entity number segment corresponding to each data group; Based on the second entity number segment of the data group in the at least one data group and the unit identifier of the i-th storage unit, a migration file corresponding to each data group is generated; The data streams within each data group in the at least one data group are migrated to the corresponding migration files, and the migration files corresponding to each data group are migrated to the database instance.

8. The method according to claim 7, characterized in that, The method further includes: Through the file service, a first query request for a database instance is received from the first query device. The first query request includes a third entity number segment and a unit identifier of the storage unit. Based on the third entity number segment and the unit identifier of the storage unit in the data query request, determine the migration file associated with the first query request from the database instance; From the determined migration file, query the data stream whose entity number belongs to the third entity number segment, and return the data stream belonging to the third entity number segment to the first query device.

9. The method according to claim 1, characterized in that, The incremental data stream in the data queue is stored in M ​​storage units of the data queue, where M is a positive integer greater than 1. The method further includes: From the M storage cells, determine the target storage cell whose storage mode is automatic optimization mode; According to the columnar storage format, the incremental data stream in the target storage unit is written to the data lake service; Based on the storage location information of the incremental data stream in the target storage unit in the data lake service, a snapshot identifier and index metadata associated with the target storage unit are generated; the snapshot identifier and the index metadata are used to query the incremental data stream corresponding to the target storage unit from the data lake service.

10. The method according to claim 9, characterized in that, The incremental data stream in the target storage unit is stored in S data files of the data lake service, where S is an integer greater than 1, and the storage location information includes the file attribute information corresponding to the S data files respectively. The step of generating a snapshot identifier and index metadata associated with the target storage unit based on the storage location information of the incremental data stream in the data lake service includes: Based on the file attribute information corresponding to the S data files, generate file index information corresponding to the S data files respectively; Based on the file index information corresponding to the S data files, generate list index information, and based on the list index information, generate a snapshot identifier associated with the target storage unit; The list index information is determined as index metadata associated with the target storage unit.

11. The method according to claim 9, characterized in that, The method further includes: Receive a second query request from the second query device for the incremental data stream in the data queue; Through the metadata interface, the snapshot identifier associated with the target storage unit and the data index file corresponding to the incremental data stream are returned to the second query device according to the second query request; The second query device is used to obtain index metadata associated with the target storage unit based on the snapshot identifier, query the incremental data stream corresponding to the target storage unit from the data lake service based on the index metadata, and query the incremental data stream from the remaining storage units in the data queue based on the data index file. The remaining storage units are the storage units other than the target storage unit among the M storage units.

12. A data processing apparatus, characterized in that, include: The synchronization module is used to store the data stream in a data queue according to the row storage format when a data stream is received, and to synchronize the data stream in the data queue to the subscription device of the data stream. The subscription device is used to process the data stream in real time. The acquisition module is used to acquire the structure information and storage index information of the data stream in the data queue; The generation module is used to generate a data index file that matches the offline processing engine based on the structure information and storage index information of the data stream in the data queue when the data queue meets the offline processing conditions. The sending module is used to send the data index file to the offline processing engine, and the offline processing engine is used to read the data stream from the data queue according to the data index file and process the read data stream offline.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 11.

14. A computer storage medium, characterized in that, The computer storage medium stores a computer program, which, when executed by a processor, performs the method as described in any one of claims 1 to 11.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 11.

Images