Real-time data flattening processing method and device and medium

Abstract

The invention discloses a real-time data flattening processing method and device and a medium, belongs to the technical field of data processing, and is used for solving the problems that an existing data flattening processing technology is high in data delay, cannot support real-time retrieval, needs manual intervention and is poor in flexibility; and high-concurrency data streams cannot be handled, so that the technical problem of poor data expansibility is solved. The method comprises the following steps: acquiring an original data stream in a data access task based on dynamic rule configuration of a data source layer; performing format configuration processing on the original data stream in a related specific scene to obtain an initial data stream; performing flow table conversion on the theme information in the initial data stream through the attribute table to obtain a theme conversion data stream; performing branch strategy processing on the combined subject conversion data stream to obtain a processing result data stream; performing conversion processing on MAP data related to JOIN data in the processing result data stream to obtain a conversion result data stream; and performing real-time data flattening processing on the conversion result data stream.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a real-time data flattening processing method, device and medium. Background Technology

[0002] With the rapid development of e-commerce, the Internet of Things (IoT), and fintech, real-time processing of multi-source heterogeneous data has become a key technological challenge. For example, e-commerce platforms (such as JD.com and Taobao) need to process product attributes (such as specifications and inventory) in real time to support accurate search and recommendations. IoT devices (such as industrial sensors and smart home devices) generate massive amounts of log data every second, requiring rapid parsing and storage. Financial transactions (such as high-frequency trading and risk control systems) rely on millisecond-level data processing capabilities.

[0003] Currently, the industry commonly uses ETL (Extract-Transform-Load) tools (such as Apache NiFi and Talend) or the data processing functions built into search engines (such as Elasticsearch Ingest Pipeline) for data transformation. However, these solutions have significant shortcomings in terms of real-time performance, flexibility, and scalability.

[0004] When performing data flattening, existing technologies generally adopt the traditional ETL (Extract-Transform-Load) approach, which typically uses offline batch processing. However, this approach cannot meet the needs of real-time data streams. Furthermore, static rule engines cannot adapt to dynamically changing business logic (e.g., frequent updates to e-commerce product attributes). Simultaneously, single-machine processing mode struggles to handle high-concurrency data streams, resulting in high processing latency and low throughput.

[0005] In other words, traditional data flattening solutions do not combine streaming computing and dynamic rule engines, resulting in insufficient real-time performance and flexibility. Furthermore, they lack adaptive optimization mechanisms, making it impossible to dynamically adjust processing strategies based on data characteristics. Rule logic is hard-coded into the program, requiring redeployment for modifications, leading to high operational costs.

[0006] In other words, existing data flattening processing technologies still have the following limitations:

[0007] 1) High latency: Offline batch processing causes data updates to be delayed, making it impossible to support real-time retrieval (such as product search and IoT device monitoring).

[0008] 2) Poor flexibility: Static rules cannot be dynamically adjusted, require manual intervention, and have high operation and maintenance costs.

[0009] 3) Insufficient scalability: The single-machine architecture cannot handle massive amounts of data, affecting system stability. Summary of the Invention

[0010] This application provides a real-time data flattening processing method, device, and medium to solve the following technical problems: existing data flattening processing technologies have high data latency, cannot support real-time retrieval, require manual intervention, and have poor flexibility; they also cannot cope with high-concurrency data streams, resulting in poor data scalability.

[0011] The embodiments of this application adopt the following technical solutions:

[0012] On one hand, embodiments of this application provide a real-time data flattening processing method, including: obtaining the original data stream in the data access task based on dynamic rule configuration of the data source layer; performing format configuration processing on the original data stream for a specific scenario to obtain an initial data stream; performing flow table transformation on the topic information in the initial data stream through an attribute table to obtain a topic transformation data stream; performing branch strategy processing on the merged topic transformation data stream to obtain a processing result data stream; performing transformation processing on the JOIN data related to MAP data in the processing result data stream to obtain a transformation result data stream; and performing real-time data flattening processing on the transformation result data stream.

[0013] This application's embodiments combine a dynamic rule engine with streaming computing to achieve real-time data flattening while supporting flexible adjustments to business logic. Through an adaptive optimization mechanism and a feedback loop, processing efficiency is dynamically improved, reducing operational costs. Furthermore, the distributed architecture design ensures high availability and scalability of the system, making it suitable for scenarios with hundreds of millions of data points.

[0014] In one feasible implementation, before obtaining the raw data stream in the data access task based on the dynamic rule configuration of the data source layer, the method further includes: reading the corresponding dynamic rules through the multi-source data accessed by the data source layer; wherein, the dynamic rules include at least one or more of the following: data processing field mapping, flattened field selection, and conditional branching rules; configuring the dynamic rules to generate the dynamic rule configuration.

[0015] In one feasible implementation, the raw data stream in the data access task is obtained based on the dynamic rule configuration of the data source layer, specifically including: generating task configuration information based on business requirements; combining the task configuration information with the dynamic rule configuration to obtain combined configuration information; generating a topic task for listening to the Kafka distributed publish-subscribe messaging system based on the combined configuration information; and obtaining the raw data stream in the data access task related to the topic task.

[0016] In one feasible implementation, the original data stream is processed with a format configuration relevant to a specific scenario to obtain an initial data stream. Specifically, this includes: performing type analysis on the original data stream to identify the data stream type; wherein, the data stream type includes: flattening operation of JSON data and delete operation of original data; if the data stream type is flattening operation of JSON data, then the JSON format data stream is processed without a schema to obtain an initial data stream in JSON format; if the data stream type is delete operation of original data, then according to whether to retain the delete operation of original data, a special format data is generated accordingly, and an initial data stream under the delete operation is obtained.

[0017] In one feasible implementation, the topic information in the initial data stream is transformed using a flow table to obtain a topic-transformed data stream. Specifically, this includes: identifying and processing the topic information in the initial data stream using the attribute table; merging the identified topic information using a flow table JOIN operation to obtain merged topic information; wherein the flow table JOIN operation associates and processes the real-time data stream and historical data stream in the dynamic dataset in real time; and performing transformation processing on the merged topic information according to a preset ksqlDB defined flow table, based on STREAM data and / or table data, to obtain the topic-transformed data stream; wherein the topic-transformed data stream includes both stream data and table data.

[0018] In one feasible implementation, the merged topic transformation data stream is processed using a branching strategy to obtain a processing result data stream. Specifically, this includes: using SQL operations defined in the flow table of ksqlDB to perform JOIN operations on the relevant flow tables to obtain the merged topic transformation data stream; identifying branch data streams under branch processing; determining the corresponding branching strategy based on the different types of the branch data streams; wherein the branching strategy includes at least: processing logic for e-commerce product attributes and IoT logs; and executing the branching strategy to obtain the processing result data stream.

[0019] In one feasible implementation, the conversion result data stream is subjected to real-time data flattening processing, specifically including: using a preset Flatten converter to perform real-time data flattening processing on the conversion result data stream to obtain a flattened data stream; wherein, the conversion result data stream is a data stream converted based on MAP data.

[0020] In one feasible implementation, after flattening the transformed data stream in real time, the method further includes: sending the flattened data stream in the stream processing layer to the storage layer via a Kafka connector; and performing storage processing on the flattened data stream based on the storage layer.

[0021] Secondly, embodiments of this application also provide a real-time data flattening processing device, the device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to execute a real-time data flattening processing method as described in any of the above embodiments.

[0022] Thirdly, embodiments of this application also provide a non-volatile computer storage medium, which is a non-volatile computer-readable storage medium storing at least one program, each program including instructions, which, when executed by a terminal, cause the terminal to perform a real-time data flattening processing method as described in any of the above embodiments.

[0023] This application provides a real-time data flattening processing method, device, and medium. Compared with the prior art, the embodiments of this application have the following beneficial technical effects:

[0024] 1. Improve data processing efficiency: By optimizing data flow and transformation processes, the time and resource consumption of data processing are reduced.

[0025] 2. Enhanced data accessibility: A flat data structure makes data easier to access and analyze.

[0026] 3. Improve data consistency: Dynamic rule configuration and format processing ensure data consistency and accuracy.

[0027] 4. Enhance system flexibility: It can quickly adapt to different data processing needs and scenario changes.

[0028] 5. Reduced maintenance costs: The simplified data processing flow reduces the complexity and cost of system maintenance.

[0029] 6. Improve data quality: Through effective data processing and transformation, the quality and reliability of the data are improved. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0031] Figure 1 A flowchart of a real-time data flattening processing method provided in this application embodiment;

[0032] Figure 2 A real-time data flattening processing system architecture diagram provided in this application embodiment;

[0033] Figure 3 This is a schematic diagram of the structure of a real-time data flattening processing device provided in an embodiment of this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0035] It should be noted that this application belongs to the field of real-time data processing technology, and specifically proposes an innovative solution for the demand scenario of real-time flattened data processing. Taking the product search function of an e-commerce platform as an example, the system needs to dynamically process product data with heterogeneous attributes. Since the attribute fields involved in different product categories vary significantly, and the business scenario requires support for dynamic attribute expansion, the fixed table structure of traditional relational databases is difficult to meet such requirements. Therefore, existing technologies typically use schema-less databases such as ElasticSearch and Cassandra. The characteristics of these databases allow for dynamic adjustment of indexes or the number of columns in a table, enabling flexible attribute expansion without the need for a predefined strict table structure.

[0036] This application also includes: (1) a dynamic rule configuration module: providing a GUI interface that supports flexible rule definition such as JSON Path and regular expressions. The rules and attribute fields are stored in the database and support hot updates.

[0037] (2) Streaming Data Processing Engine: Built on the Apache Kafka ecosystem, this engine supports real-time pipelines and multiple data source accesses (e.g., databases, files, messages) and real-time streaming data access (e.g., incremental access). Based on its real-time streaming data processing capabilities, it supports field mapping (e.g., attributes.color → attr_color), type conversion (e.g., string to number), expanding nested data structures (e.g., splitting JSON structures into multiple columns), and data filtering (e.g., including data under certain conditions). It also features ksqlDB's multi-stream / multi-table join capabilities: time-window-based stream processing and various SQL processing functions (e.g., scalar functions, table functions).

[0038] (3) Adaptive optimization module: Monitors processing latency and dynamically adjusts the rule execution order (e.g., prioritizes high-frequency fields). Avoids data skew through consistent hashing.

[0039] This application provides a real-time data flattening processing method, such as... Figure 1 As shown, the real-time data flattening processing method specifically includes steps S101-S106:

[0040] S101. Based on the dynamic rule configuration of the data source layer, obtain the raw data stream in the data access task.

[0041] Specifically, it is necessary to first read the corresponding dynamic rules from the multi-source data accessed by the data source layer. Among them, dynamic rules include at least one or more of the following: data processing field mapping, flattening field selection, and conditional branching rules.

[0042] Furthermore, the dynamic rules are configured to generate dynamic rule configurations.

[0043] Furthermore, task configuration information is generated based on business requirements.

[0044] Furthermore, the task configuration information is combined with the dynamic rule configuration to obtain combined configuration information.

[0045] Furthermore, based on the combined configuration information, a topic task for listening to the Kafka distributed publish-subscribe messaging system is generated. The raw data stream from the data access task within the topic task is then obtained.

[0046] In one embodiment, Figure 2 A real-time data flattening processing system architecture diagram is provided for embodiments of this application, such as... Figure 2As shown, it also includes: Data source layer: Message queues such as Kafka access multiple data sources. Rule management layer: Provides a web interface for configuring rules, which are stored in a MySQL + Redis cache. Stream processing layer: Kafka and ksqlDB clusters execute real-time flattening tasks. Storage layer: Outputs to Elasticsearch, HBase, etc.

[0047] As a possible implementation method, such as Figure 2 As shown, dynamic rule configurations can be read first from MySQL or the cache. These configurations include data processing field mappings, flattening field selection, and conditional branching rules. Then, by combining the read rule configurations with the task configuration information, the system can generate a task to listen to a Kafka topic, thereby further obtaining the raw data stream from the data access task.

[0048] S102. Perform format configuration processing on the original data stream for a specific scenario to obtain the initial data stream.

[0049] Specifically, type analysis is performed on the raw data stream to identify its type. This type includes flattening operations on JSON data and delete operations on the raw data.

[0050] If the data stream type is a flattening operation of JSON data, then the JSON formatted data stream is processed without a schema to obtain the initial data stream in JSON format. If the data stream type is a delete operation of the original data, then depending on whether the original data is retained in the delete operation, a special formatted data is generated accordingly, and the initial data stream under the delete operation is obtained.

[0051] In one embodiment, such as Figure 2 As shown, the acquired raw data stream may have format requirements in specific scenarios. For example, when performing a flattening operation on JSON data, the JSON data stream needs to be schema-less. Another example is whether to retain the original delete operation, which requires generating a separate data stream with a special format. Therefore, this data format processing step is necessary.

[0052] S103. The topic information in the initial data stream is transformed into a topic-transformed data stream by using the attribute table.

[0053] Specifically, the subject information in the initial data stream also needs to be identified and processed through the attribute table.

[0054] Furthermore, the identified topic information is merged using the flow table JOIN operation to obtain merged topic information. Specifically, the flow table JOIN operation associates and processes real-time data streams with historical data streams in the dynamic dataset.

[0055] Furthermore, based on the predefined ksqlDB flow table, the merged topic information undergoes transformation processing on STREAM data and / or table data to obtain a topic-transformed data stream. This topic-transformed data stream includes both stream data and table data.

[0056] In one embodiment, such as Figure 2 As shown, after formatting, the flattened data is organized through attribute tables. This involves merging data from multiple topics, which requires a stream table join operation. (Stream table join operations enable real-time data correlation analysis, linking and processing streaming data with historical data in real time. Stream table join operations are mainly used to process dynamic datasets, achieving real-time correlation analysis by connecting real-time data streams with static tables or another stream.) Therefore, in this step, the topic is converted into a STREAM or table using ksqlDB's stream table definition.

[0057] S104. The merged topic transformation data stream is processed using a branching strategy to obtain the processing result data stream.

[0058] Specifically, it is also necessary to define SQL operations in the flow table through ksqlDB to perform relevant flow table JOIN operations on the topic transformation data stream to obtain the merged topic transformation data stream.

[0059] Furthermore, the merged topic transformation data stream is further processed by branching to identify the branch data streams.

[0060] Furthermore, based on the different types of branch data streams, the corresponding branching strategies are determined. These branching strategies include at least the processing logic for e-commerce product attributes and IoT logs. Finally, the branching strategies are executed to obtain the processing result data stream.

[0061] In one implementation, such as Figure 2 As shown, the join operation (JOIN operation is a query method in SQL used to merge data from two or more tables based on the fields of the relationship between the tables. Its core purpose is to generate a result set containing information from multiple tables by horizontally integrating related data) is completed using SQL operations in ksqlDB. Then, the merged subject is transformed into a data stream and processed according to the corresponding strategy based on the branch operation, that is, different processing strategies are adopted, such as using different logic for product attributes in e-commerce and IoT logs.

[0062] S105. Transform the JOIN data related to the MAP data in the processing result data stream to obtain the transformation result data stream.

[0063] Specifically, in e-commerce product attributes, before performing flattened data processing, it is necessary to convert JOIN data into MAP data, which requires the use of three functions: AS_MAP, MAP_KEYS, and JSON_RECORDS.

[0064] S106. Perform real-time data flattening processing on the transformation result data stream.

[0065] Specifically, a pre-defined Flatten converter is needed to flatten the converted data stream in real time, resulting in a flattened data stream. The converted data stream is the data stream after conversion based on MAP data.

[0066] As one feasible implementation, a flattened data stream from the stream processing layer is sent to the storage layer via a Kafka connector. The storage layer then performs storage processing on the flattened data stream. Specifically, the Kafka connector connects to Elasticsearch or other storage in the storage layer.

[0067] In addition, embodiments of this application also provide a real-time data flattening processing device, such as... Figure 3 As shown, the real-time data flattening processing device specifically includes:

[0068] At least one processor 301. And a memory 302 communicatively connected to the at least one processor 301. The memory 302 stores instructions executable by the at least one processor 301, enabling the at least one processor 301 to execute:

[0069] Based on dynamic rule configuration at the data source layer, the raw data stream in the data access task is obtained;

[0070] The raw data stream is processed with a format configuration relevant to a specific scenario to obtain the initial data stream;

[0071] The topic information in the initial data stream is transformed into a topic-transformed data stream by using the attribute table;

[0072] The merged topic transformation data stream is processed using a branching strategy to obtain the processing result data stream;

[0073] The JOIN data in the processing result data stream is transformed and processed to obtain the transformed result data stream;

[0074] The transformed data stream is flattened in real time.

[0075] This application's embodiments combine a dynamic rule engine with streaming computing to achieve real-time data flattening while supporting flexible adjustments to business logic. Through an adaptive optimization mechanism and a feedback loop, processing efficiency is dynamically improved, reducing operational costs. Furthermore, the distributed architecture design ensures high availability and scalability of the system, making it suitable for scenarios with hundreds of millions of data points.

[0076] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the description of the method embodiments.

[0077] The devices and media provided in this application are one-to-one with the methods. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0078] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0083] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0084] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0085] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0086] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of this specification.

Claims

1. A real-time data flattening processing method, characterized in that, The method includes: Based on dynamic rule configuration at the data source layer, the raw data stream in the data access task is obtained; The original data stream is processed with a format configuration relevant to a specific scenario to obtain an initial data stream; The topic information in the initial data stream is transformed into a topic-transformed data stream by using the attribute table. The merged topic transformation data stream is processed using a branching strategy to obtain the processing result data stream; The JOIN data in the processing result data stream is transformed and processed to obtain the transformation result data stream; The transformed data stream is flattened in real time.

2. The real-time data flattening processing method according to claim 1, characterized in that, Before obtaining the raw data stream from the data access task based on the dynamic rule configuration of the data source layer, the method further includes: By accessing multi-source data through the data source layer, the corresponding dynamic rules are read; wherein, the dynamic rules include at least one or more of the following: data processing field mapping, flattening field selection, and conditional branching rules; Configure the dynamic rules to generate the dynamic rule configuration.

3. The real-time data flattening processing method according to claim 2, characterized in that, Based on dynamic rule configuration at the data source layer, the raw data stream in the data access task is obtained, specifically including: Generate task configuration information based on business requirements; The task configuration information and the dynamic rule configuration are combined to obtain combined configuration information; Based on the combined configuration information, a topic task for monitoring the Kafka distributed publish-subscribe messaging system is generated; Obtain the raw data stream from the data access task in the topic task.

4. The real-time data flattening processing method according to claim 1, characterized in that, The original data stream is processed according to a format configuration specific to the scenario to obtain an initial data stream, specifically including: The raw data stream is subjected to type analysis to identify the data stream type; wherein, the data stream type includes: flattening operation of JSON data and delete operation of raw data; If the data stream type is a flattening operation of the JSON data, then the JSON format data stream is processed without a schema to obtain the initial data stream in JSON format; If the data stream type is the delete operation of the original data, then depending on whether the delete operation of the original data is retained, a special format data is generated accordingly, and the initial data stream under the delete operation is obtained.

5. The real-time data flattening processing method according to claim 1, characterized in that, The topic information in the initial data stream is transformed using an attribute table to obtain a topic-transformed data stream, specifically including: The subject information in the initial data stream is identified and processed using the attribute table. The identified topic information is merged through the flow table JOIN operation to obtain merged topic information; wherein, the flow table JOIN operation associates and processes the real-time data stream and historical data stream in the dynamic dataset in real time. According to the preset ksqlDB definition flow table, the merged topic information is transformed under the relevant STREAM data and / or table data to obtain the topic transformation data stream; wherein, the topic transformation data stream includes: stream data and table data.

6. The real-time data flattening processing method according to claim 1, characterized in that, The merged topic transformation data stream is processed using a branching strategy to obtain the processing result data stream, which specifically includes: By defining SQL operations in the flow table using ksqlDB, the topic transformation data stream is processed by JOIN operations on the relevant flow tables to obtain the merged topic transformation data stream; The merged topic transformation data stream is identified under branch processing to determine the branch data stream; Based on the different types of the branch data streams, the corresponding branching strategies are determined; wherein, the branching strategies include at least: the processing logic of e-commerce product attributes and IoT logs; The branching strategy is executed to obtain the processing result data stream.

7. The real-time data flattening processing method according to claim 1, characterized in that, The transformation result data stream undergoes real-time data flattening processing, specifically including: The conversion result data stream is flattened in real time using a preset Flatten converter to obtain a flattened data stream; wherein, the conversion result data stream is a data stream converted based on MAP data.

8. The real-time data flattening processing method according to claim 1, characterized in that, After flattening the transformed data stream in real time, the method further includes: The flattened data stream from the stream processing layer is sent to the storage layer via the Kafka connector; Based on the storage layer, the flattened data stream is stored and processed.

9. A real-time data flattening processing device, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform a real-time data flattening processing method according to any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The storage medium is a non-volatile computer-readable storage medium that stores at least one program, each program including instructions that, when executed by a terminal, cause the terminal to perform a real-time data flattening processing method according to any one of claims 1-8.

Images