Streaming service data processing method and device and electronic equipment
By decomposing streaming business data into multiple business processes and creating message queues and primary key tables, and using materialized view wide tables to achieve real-time association and integration of streaming business data, the problems of insufficient real-time performance and high system complexity in existing technologies are solved, thereby improving the real-time performance of data processing and operational efficiency.
Patent Information
- Application Number
- CN202511759618.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-27
- Publication Date
- 2026-03-03
AI Technical Summary
Existing technologies suffer from insufficient real-time performance, high system complexity, high resource consumption, and lack of intelligent end-to-end monitoring and self-healing capabilities when processing multi-source heterogeneous streaming business data, thus failing to meet the needs for real-time data insight and rapid fault response.
Multi-source heterogeneous streaming business data is decomposed into multiple business processes, and message queues and primary key tables are created for each process. Data is then imported into the primary key table through a data import task. Materialized view wide tables are used to achieve real-time data association and integration, reducing the number of system components and improving operational efficiency.
It enables real-time automatic association and integration of multi-stage, multi-source heterogeneous streaming business data, reduces architectural complexity and operation and maintenance costs, improves the real-time performance and query efficiency of data processing, supports high concurrency and strong real-time requirements, and reduces manual intervention.
Smart Images

Figure CN121597689A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data processing technology, and in particular to a method, apparatus, and electronic device for processing streaming business data. Background Technology
[0002] In today's big data era, the data generated by business systems is characterized by its massive volume, multiple sources, and real-time nature. Especially in fields involving complex business chains, such as telecommunications, finance, and e-commerce, how to integrate, correlate, query, and monitor streaming data scattered across multiple links and systems in real time to form a unified global view, and ensure the stability and efficiency of the data processing process, has become a core technical challenge facing the industry.
[0003] Wide tables, as an effective data integration method, are widely used to solve such problems. Their goal is to "stretch" the data from multiple related tables into a single large table to simplify queries and improve performance.
[0004] However, the existing solutions mainly have the following technical problems: Insufficient real-time performance: Most solutions are non-real-time batch processing modes with high data latency, which cannot meet the business's needs for real-time data insights and rapid fault response.
[0005] High system complexity and high resource consumption: To achieve real-time performance, existing solutions often have complex architectures and numerous components (such as databases, multiple Flink jobs, and message queues), resulting in high development and maintenance costs and a tendency to create performance bottlenecks and waste resources.
[0006] Lack of intelligent end-to-end monitoring and self-healing capabilities: Existing solutions generally focus on data processing itself, but lack systematic monitoring indicators and automatic repair mechanisms after exceeding thresholds for the health status of the data pipeline (such as data delay, loss, and computational anomalies). Anomaly investigation relies heavily on manual labor, resulting in low operation and maintenance efficiency. Summary of the Invention
[0007] The technical problem to be solved by this invention is to provide a method, apparatus, and electronic device for processing streaming business data. This enables real-time, automatic association and integration of multi-stage, multi-source, heterogeneous streaming business data, reducing the number of system components and significantly lowering architectural complexity and maintenance costs.
[0008] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: A method for processing streaming business data includes: Decompose the streaming business chain data, which contains multi-source heterogeneous streaming business data, into multiple business links; Create a message queue and a primary key table for each of the multiple business processes; Import the business process data of each business step into the corresponding message queue; Based on the data import task, the business process data in the message queue is imported into the corresponding primary key table to obtain the target primary key table; Based on the target primary key table and the identifier of the streaming business chain data corresponding to each business segment, the data of each business segment are associated to obtain the materialized view wide table of the streaming business chain data.
[0009] Optionally, the streaming business chain data, which contains multi-source heterogeneous streaming business data, can be decomposed into multiple business links, including: Based on the state of the data sub-streams in the streaming business chain data, the streaming business chain data containing multi-source heterogeneous streaming business data is decomposed into multiple business links.
[0010] Optionally, a message queue and primary key table are created for each of the multiple business processes, including: Based on the number of business processes, create message queues in the message queue cluster that correspond one-to-one with each business process; and create primary key tables in the database that correspond one-to-one with each business process.
[0011] Optionally, based on the data import task, the business process data in the message queue is imported into the corresponding primary key table to obtain the target primary key table, including: Based on the configuration information of the data import task, consume business process data from the message queue; Extract the business fields and values of the consumption process data; Based on the field mapping rules in the configuration information, the business fields are written into the corresponding columns of the primary key table to obtain structured data; Based on the primary keys of the business process identifier and the identifier of the streaming business chain data, the structured data is written into the corresponding primary key table in an insert or update manner to obtain the target primary key table.
[0012] Optionally, based on the target primary key table corresponding to each business segment and the identifier of the streaming business chain data, the data of each business segment are associated to obtain a materialized view wide table of the streaming business chain data, including: Using the identifier of the streaming business chain data as the association key, all target primary key tables are associated with the target primary key tables corresponding to each business link through a left join operation of the database; Based on the core business fields in the target primary key table, a materialized view wide table of streaming business chain data is obtained. The core business fields include: basic information of the streaming business chain, status information of each business link, and timestamp of each business link.
[0013] Optionally, the processing methods for streaming business data also include: Receive new streaming business chain data; Update the materialized view wide table based on the newly added streaming business chain data and its identifier.
[0014] Optionally, the processing methods for streaming business data also include: Receive a query request, wherein the query request carries at least: an identifier of the streaming business chain data; Based on the request parameters, a query operation is performed on the target primary key table corresponding to the target business process in the materialized view wide table to obtain the query results.
[0015] Embodiments of the present invention also provide a streaming business data processing apparatus, comprising: The decomposition module is used to decompose streaming business chain data, which contains multi-source heterogeneous streaming business data, into multiple business links. The processing module is used to create a message queue and a primary key table for each of the multiple business processes; import the business process data of each business process into the corresponding message queue; import the business process data in the message queue into the corresponding primary key table according to the data import task to obtain the target primary key table; and associate the data of each business process with the target primary key table corresponding to each business process and the identifier of the streaming business chain data to obtain the materialized view wide table of the streaming business chain data.
[0016] Embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.
[0017] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the method described above.
[0018] The above-described solution of the present invention has at least the following beneficial effects: The above-described solution of the present invention decomposes streaming business chain data containing multi-source heterogeneous streaming business data into multiple business stages. A processing module is used to create a message queue and a primary key table for each of the multiple business stages. The business stage data of each stage is imported into the corresponding message queue. According to the data import task, the business stage data in the message queue is imported into the corresponding primary key table to obtain a target primary key table. Based on the target primary key table corresponding to each business stage and the identifier of the streaming business chain data, the data of each business stage is associated to obtain a materialized view wide table of the streaming business chain data. This enables real-time and automatic association and integration of multi-stage, multi-source heterogeneous streaming business data, reducing the number of system components and significantly reducing architectural complexity and operation and maintenance costs. Attached Figure Description
[0019] Figure 1 This is a schematic diagram of the process for processing streaming business data according to the present invention; Figure 2 This is a schematic diagram of the structure of the streaming business data processing device of the present invention; Figure 3 This is a system schematic diagram of the streaming business data processing device of the present invention. Detailed Implementation
[0020] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
[0021] like Figure 1 As shown, an embodiment of the present invention proposes a method for processing streaming business data, including: Step S10: Decompose the streaming business chain data containing multi-source heterogeneous streaming business data into multiple business links; for example, decompose the entire process of "signaling reception - business trigger judgment - package number deduction - SMS / flash message delivery - terminal display" of the Yiming card, define the required fields and format standards of the logs of each link, and set a unified unique link ID field for all links; Step S20: Create a message queue and a primary key table for each of the multiple business processes; for example, create Kafka topics according to the decomposed business processes, with one topic for each process. Create a corresponding StarRocks primary key table, setting the primary key to `process ID + unique link ID`, and configure bucketing and replication strategies; each topic corresponds one-to-one with its primary key. Step S30: Import the business process data of each business process into the corresponding message queue; Step S40: According to the data import task, import the business process data in the message queue into the corresponding primary key table to obtain the target primary key table; specifically, create a Routine Load task for each primary key table, configure the Kafka Broker address, topic, consumer group, set field mapping rules, automatic retry, breakpoint resume and data filtering rules to realize real-time data import of each process. Step S50: Based on the target primary key table and the identifier of the streaming business chain data corresponding to each business link, the data of each business link are associated to obtain the materialized view wide table of the streaming business chain data. Here, based on all primary key tables, the data of each link is associated using the "unique link ID" as the association key, and the core fields are filtered to create the materialized view wide table.
[0022] The streaming service data processing method of this invention is applicable to scenarios involving real-time, automatic association, integration, tracking, and monitoring of multi-stage, multi-source, heterogeneous streaming service data with process-oriented or interconnected characteristics. Examples include real-time monitoring, fault diagnosis, and big data analysis of service links in value-added communication services, and it is particularly suitable for services like Yimingbiao, which involve multiple stages, high concurrency, and stringent real-time requirements.
[0023] In this embodiment, the producer (business server) actively generates streaming business chain data during the processing of business logic. Data generated by each business link is collected in real time and sent to the corresponding message queue topic. The data import task consumes business link data from the corresponding message queue topic and imports it into the corresponding primary key table. Using the identifier of the streaming business chain data as the association key, all primary key tables are associated, actually associating the business data of each link, and generating a materialized view wide table of streaming business chain data. This embodiment disperses the data of each business link into the corresponding message queue topic, and uses the data import task to import the corresponding business link data consumed by the message queue topic into the corresponding primary key table in real time, dynamically associating the business link data to generate a materialized view wide table, i.e., a streaming wide table. A streaming wide table is a pre-joined, wide-column data table structure designed specifically for real-time data streams. It integrates the dimension and indicator fields that were originally scattered in multiple tables into a large wide table by dynamically associating and splicing multiple real-time data streams, thereby providing extremely low-latency data services for real-time querying and analysis. The materialized view wide table generated by the method in this embodiment can perform real-time and automatic association and integration of multi-stage, multi-source heterogeneous streaming business data, solving technical problems such as data dispersion, unstable import, slow query, difficult association, and complex operation and maintenance in the prior art. By replacing manual association scripts with configurable materialized views, only the view SQL needs to be updated when adding new business stages, shortening the adaptation cycle from several days to less than 1 hour. Based on multi-stage parallel import and primary key deduplication model, it supports real-time and accurate inbound and storage of billions of data points per day, providing strong data support for big data analysis and achieving significant technical effects.
[0024] In an optional embodiment of the present invention, step S10, decomposing the streaming service chain data containing multi-source heterogeneous streaming service data into multiple service links, includes: Step S11: According to the state of the data sub-streams in the streaming business chain data, decompose the streaming business chain data containing multi-source heterogeneous streaming business data into multiple business links.
[0025] In this embodiment, the data of each business process is standardized into a standard data record format, wherein the data record format includes at least: the identifier of the streaming business chain data, the process identifier, the timestamp, the business entity identifier, the business status, and the key business fields.
[0026] In this embodiment, each step is converted into pre-configured exclusive fields such as the identifier of the streaming business chain data and the business step identifier. The streaming business chain data is a unique identifier that remains unchanged in all steps of the same business chain and is also a unique link identifier. The business identifier is used to distinguish the type of business step to which the data belongs.
[0027] In the streaming business chain process path, the division of stages is guided by identifying core business entities in the business process. These business entities are core conceptual objects in the business domain that have a unique identifier, an independent lifecycle, and can undergo state changes, such as orders, sessions, and accounts. Based on the state transition paths of these business entities, data sub-streams that generate substantial state changes in the business process are identified as candidate business stages.
[0028] Persistent changes refer to modifications to core business entities that are permanently recorded, and subsequent business logic depends on this changed state. Such changes are irreversible, or reversing them requires executing another equivalent business operation.
[0029] The persistent change in the status, quantity, or attributes of the core business entity refers to the change meeting the following conditions simultaneously: Data persistence: The changes are written to a database or other non-volatile storage medium; And at least one of the following conditions must be met: State visibility: The changed state is queryable and perceptible to users or external systems; Downstream dependency: Subsequent business processes use the changed results as input conditions or decision-making basis for execution; Rollback compensation: Undoing the change requires performing an equivalent reverse business operation rather than a simple deletion.
[0030] A data substream is classified as a separate business process only when it causes the persistent change.
[0031] In this embodiment, by meeting preset conditions, the accuracy and efficiency of business process decomposition are improved for different streaming business chain data, and the application scope is wide.
[0032] In one specific embodiment, for ease of understanding, the streaming service link is decomposed and data modeled using YiBai ID as an example. YiBai ID includes two products: incoming call ID and hang-up ID. In the calling scenario, before a user calls, the caller's real-name identity is pushed and displayed to the called user via flash message or rich media to improve the call connection rate. In the called scenario, after a user's call ends (including hang-up, busy tone, and prolonged unanswered calls), a pre-set multimedia message from the called party is pushed to the calling user for brand promotion. Step S11 can be used to decompose YiBai ID into a complete process: signaling reception - service trigger judgment - package number deduction - SMS / flash message delivery - terminal display. Required fields and format standards for the logs of each process are defined, and a unified unique link ID field is set for all processes, i.e., the identifier of the streaming service chain data.
[0033] In a specific embodiment, the complete business process of the Yiming business card is analyzed and broken down into five core stages: signaling reception, service trigger judgment, package deduction, flash message delivery, and terminal display. Each stage defines its own unique fields, including a unique link ID (link_id), calling and called numbers, and a timestamp (in a unified format of yyyy-MM-dd HH:mm:ss), thus standardizing the data model and providing standard field specifications for subsequent message queue topics and primary key table design, ensuring data format consistency.
[0034] The methods of this invention can be applied to fields such as e-commerce / retail, fintech (e.g., transaction risk control pipelines, credit approval processes), logistics and supply chain (e.g., parcel lifecycle monitoring), software development and DevOps (CI / CD pipeline monitoring), or the Internet of Things (e.g., device operation status chains). For e-commerce order placement processes, the methods of this invention can decompose the entire order data into stages: locking inventory, creating an order, payment deduction, inventory reduction, shipping, logistics updates, and receipt confirmation. Definitive fields are defined for each stage, which will not be elaborated here. Link data: A collection of data from all business stages linked by a unique link ID, including timestamps, statuses, and anomaly information for each stage. It visually displays the complete process from initiation to completion of the business, and is used for anomaly tracing and business monitoring.
[0035] In an optional embodiment of the present invention, step S20 involves creating a message queue and a primary key table for each of the plurality of business processes, including: Step S21: Based on the number of multiple business processes, create a message queue in the message queue cluster that corresponds one-to-one with each business process. Step S22: In the database, create a primary key table that corresponds one-to-one with each business process.
[0036] In one specific embodiment, based on the multiple business segments obtained from the above decomposition, a corresponding Kafka topic is created in the Kafka cluster for each business segment, and N partitions and M replicas are configured for each Kafka topic to achieve isolation and parallel transmission of segment data.
[0037] In StarRocks, a corresponding primary key table is created for each business process. The primary key in the primary key table is a combination of the business process identifier and the identifier of the streaming business chain data to ensure data uniqueness. Each primary key table is configured with a bucketing and replication strategy, with N buckets and M replicas, completing the construction of a high-availability data pipeline and structured storage infrastructure. The primary key table is a data table with specified primary key constraints.
[0038] Taking the aforementioned Yiming business chain data as an example, five topics are created in the Kafka cluster for the five stages of Yiming business chain: signaling reception, business trigger judgment, package deduction, flash message delivery, and terminal display. Each of the five topics corresponds to one of the five business stages, and each topic is configured with 5 partitions and 3 replicas. In StarRocks, a primary key table corresponding to each business stage is created. The primary key in the primary key table uses a combination of stage ID and unique link_id.
[0039] Considering the Kafka cluster contains 5 Broker nodes, and to fully leverage its parallel processing capabilities, the number of partitions for each topic is set to 5, ensuring a balanced distribution of data production and consumption across all nodes. Simultaneously, to guarantee high availability during data transmission, the replication factor is set to 3, ensuring no data loss and continuous service availability even if a single Broker node fails. Correspondingly, when creating the primary key table in StarRocks, a 3-replica deployment is adopted, consistent with Kafka's replication strategy, thus providing high availability at the data storage layer. Through the above configuration, a high-availability data pipeline and structured storage foundation are established. StarRocks: a high-performance analytical data warehouse compatible with the MySQL protocol, allowing integration with MySQL clients, and supporting real-time data ingestion, efficient querying, and complex data analysis, suitable for real-time data warehouses, data lake analytics, and other scenarios. Kafka: a high-throughput distributed message queue used for real-time collection and delivery of streaming data. Unique Link ID: a globally unique identifier, used throughout the entire business process, is the core association field forming a complete data link. Primary Key Table: A table type in StarRocks that uniquely identifies recorded data using a primary key. It supports data updates and deduplication. The primary key is set as a combination of stage ID and unique link ID to avoid duplicate data.
[0040] In step S30, the business process data for each business segment is imported into the corresponding message queue, thereby triggering the subsequent streaming data processing flow. In an optional embodiment of the present invention, a data import task is created for each business segment to consume business segment data from the corresponding Kafka topic and import it into the primary key table, thus completing the construction of the target primary key table for each segment. The import cycle of the data import task is: micro-batch data consumption is performed every X seconds, where X is less than 30 seconds; specifically, X can be 2.
[0041] Based on the created topic and primary key table, a data import task is created for each business process's data flow. During creation, the task is configured to consume data from its corresponding Kafka topic and import the data into its corresponding StarRocks primary key table via field mapping. Simultaneously, the batch processing interval for this task is configured to be 2 seconds, and breakpoint resumption and error retry mechanisms are enabled to achieve second-level latency data import for each stage. The data import task can be a StarRocks Routine Load task.
[0042] In one specific embodiment, an independent data import task is created and configured for each Kafka topic. The configuration includes: Kafka Broker address, corresponding topic and consumer group, target primary key table, field mapping rules, failure retry mechanism (automatic retries 3 times), breakpoint resumption, and data filtering rules, enabling real-time data import at each stage. After the task starts, it runs continuously, consuming one piece of data every 2 seconds in micro-batch and writing it to the corresponding StarRocks primary key table, achieving second-level latency data entry at each stage. Specifically, Routine Load: a continuous data import tool provided by StarRocks, can periodically read data from data sources such as Kafka and write it to the target table, supporting breakpoint resumption, automatic retries, and unique tag identification to ensure the integrity and consistency of the imported data.
[0043] The data import task can be a data ingestion tool built into the target database (such as StarRocks' Routine Load) or a standalone stream processing application (such as one built on Flink or Spark Streaming). The goal is to establish and maintain a continuous data stream from the message queue to the database tables.
[0044] In an optional embodiment of the present invention, step S40, according to the data import task, imports the business process data in the message queue into the corresponding primary key table to obtain the target primary key table, including: Step S41: Based on the configuration information of the data import task, consume business process data from the message queue. Create an independent data import task for each primary key table, and configure field mapping, breakpoint resumption, and failure retry mechanism (automatic retries 3 times).
[0045] Step S42: Extract the business fields and values of the consumption business process data.
[0046] Step S43: According to the field mapping rules in the configuration information, write the business fields into the corresponding columns of the primary key table to obtain structured data.
[0047] Step S44: Based on the primary keys of the business process identifier and the identifier of the streaming business chain data, the structured data is written into the corresponding primary key table in an insert or update manner to obtain the target primary key table.
[0048] In this embodiment of the invention, a micro-batch of data is consumed every 2 seconds and written to the corresponding StarRocks primary key table, achieving second-level latency data entry into the database at each stage; a Routine Load task is created for each primary key table, configuring the Kafka Broker address, topic, consumer group, setting field mapping rules, automatic retry, breakpoint resume, and data filtering rules, to achieve real-time data import at each stage.
[0049] After obtaining the target primary key table, the data import task successfully processes a batch of data, submits the consumption offset, and continues to consume subsequent data, thereby achieving second-level delay in data entry at each stage.
[0050] In this embodiment, based on the data import task, the data can be imported into the streaming wide table of the primary key table in a timely manner, reducing the update delay of the link detail data from several minutes in the traditional solution to the second level (≤2 seconds), and the query response time is ≤2 seconds, which meets the real-time requirement of business minute-level fault location.
[0051] In an optional embodiment of the present invention, in step S50, the data of each business link is associated according to the target primary key table corresponding to each business link and the identifier of the streaming business chain data to obtain a materialized view wide table of the streaming business chain data, including: Step S51: Using the identifier of the streaming business chain data as the association key, associate all target primary key tables with the target primary key tables corresponding to each business link through a left join operation of the database.
[0052] Step S52: Based on the core business fields in the target primary key table, obtain the materialized view wide table of the streaming business chain data. The core business fields include: basic information of the streaming business chain, status information of each business link, and timestamp of each business link.
[0053] In this embodiment, multiple target primary key tables storing data from various business processes are associated to form a temporary, logically related result set containing all fields from all associated tables. Core business fields are then selected to define the structure of the materialized view wide table, and its creation is executed, resulting in a materialized view wide table.
[0054] In one specific embodiment, based on all primary key tables, using the unique link ID (link_id) as the association key, the primary key tables of five stages are joined using the LEFT JOIN syntax to associate data from each stage. A materialized view wide table is then created by filtering core fields. The core fields include: basic link information, timestamps for each stage, and status information for each stage. Taking the aforementioned Yiming business card as an example, the basic link information includes: unique link identifier ID (link_id), calling number, and called number; the status information for each stage includes: signaling reception status, service details (service triggering result), and flash message content (package deduction status); the timestamps for each stage include: signaling reception time, service triggering time, and package deduction time.
[0055] Materialized Views: Pre-computed and stored query result sets in StarRocks. They can generate a complete wide table of links based on data from multiple primary key tables through a "unique link ID". They support automatic refresh, reduce multi-table join calculations during queries, and improve response speed.
[0056] In another embodiment of the present invention, the method for processing streaming service data may further include: Step S60: Based on the start timestamp, the historical data of the streaming business chain data that exceeds the preset time is divided into cold data and stored in the cold storage medium. Step S61: Divide historical data that has not exceeded the preset time into hot data and store it on the local disk of the database node.
[0057] This embodiment also sets a lifecycle management strategy for the data in the materialized view wide table: based on the start timestamp of the streaming business chain data, historical data exceeding a specific time window (e.g., 1 month) is partitioned by time unit (e.g., month) and archived to the database's cold storage medium; while hot data within a specific time window (e.g., within 1 month) is retained on the local disk of the database node. The database query engine automatically routes queries for historical data to cold storage, significantly reducing storage costs while ensuring the integrity of the query function.
[0058] In a specific implementation, taking Yiming Business Card as an example, historical data with a signaling reception time (receive_time) exceeding one month in the wide table is archived to StarRocks' cold storage by month, and queries are automatically routed to the cold storage; hot data, which can be data within one month, is kept on the local disk of the BE node to reduce storage costs.
[0059] In an optional embodiment of the present invention, the method for processing streaming service data may further include: Step S70: Receive newly added streaming service chain data; update the materialized view wide table according to the newly added streaming service chain data and its identifier.
[0060] This invention configures an incremental refresh strategy for the materialized view wide table: when data is inserted into any of the target primary key tables or existing data is updated, the database system will automatically trigger an incremental task. This task will only recalculate and refresh the records in the materialized view wide table associated with this batch of updated data, thereby ensuring near real-time performance of the wide table data. The read time period is Y seconds, where Y is less than 5 seconds; specifically, Y can be 1 second.
[0061] This incremental task locates the streaming business chain data identifier corresponding to the changed data and only refreshes the records associated with these identifiers in the materialized view wide table. By setting the delay threshold of this refresh strategy to 1 second, it ensures that the materialized view wide table can be updated in a very short time after the data is updated, generating a near real-time streaming wide table containing complete link information.
[0062] In one specific embodiment, a 1-second incremental refresh strategy is configured for the materialized view wide table after data updates. This means that the primary key table is checked for updates every second. When an INSERT / UPDATE operation is detected in any primary key table, the materialized view only incrementally refreshes the associated link data. In other words, only the associated link data is incrementally calculated and updated, automatically generating a streaming wide table containing complete link information. Query performance is optimized by time partitioning. This achieves second-level updates to the data wide table (latency ≤ 2 seconds), providing a unified data source for efficient queries.
[0063] In an optional embodiment of the present invention, an independent query interface service is developed to query the materialized view wide table, supporting queries based on identifiers of streaming business chain data or queries combining core business entity identifiers with time ranges. The query results are automatically converted to JSON format and support pagination.
[0064] In one specific implementation, a RESTful query interface is developed based on Spring Boot and its parameters are adapted. This query interface is a standalone application service that connects to the StarRocks database via standard JDBC to query the materialized view wide table. An HTTP query interface is also developed, supporting precise queries using unique link IDs and queries combining business numbers and time ranges. Internally, the interface implements parameter validation, dynamic SQL generation, and result formatting. Results are automatically converted to JSON format and pagination is supported. The average response time of the interface is ≤1 second, meeting the real-time query needs of customer service, operations, and other departments. Simultaneously, the wide table can be provided to reporting systems and data dashboards for subsequent data application development.
[0065] In an optional embodiment of the present invention, the method for processing streaming service data further includes: Step S80: Receive a query request, which carries at least the identifier of the streaming business chain data; according to the request parameters, perform a query operation on the target primary key table corresponding to the target business link in the materialized view wide table to obtain the query result.
[0066] In this embodiment, when the request parameter is an identifier (unique identifier) of the streaming business chain data, an equality query is performed; when the request parameter is a business entity identifier and a time range, a range query is performed. The query result is one or more wide table records containing complete information on all relevant business processes. The query result set is automatically converted into JSON format and supports pagination; that is, the result is encapsulated into a paginated JSON format and returned. The root object of the JSON format contains fields such as data (array of records on the current page) and total (total number of records).
[0067] In another embodiment of the present invention, the method for processing streaming service data further includes: Step S801: Cache the detailed data of the streaming business chain queried within a preset time period from the current time.
[0068] Step S802: Receive the query request and parse the request parameters; if the query result corresponding to the request parameters already exists in the cache, return the cached result directly.
[0069] This invention employs a query caching strategy at the query interface layer to improve performance: a caching component (such as Redis) is introduced into the query interface service to cache detailed data records of recently frequently queried streaming business chains, and a cache validity period is set, i.e., a preset time period. This preset time period can be from 20 minutes to 1 hour, preferably 30 minutes. For query requests that hit the cache, the results are directly returned from the cache, thereby reducing the pressure of repeated database queries and further improving the response efficiency of the query interface and system throughput.
[0070] In one specific embodiment, taking Yimingpai as an example, a Redis cache is added to the interface layer to cache the link details queried frequently in the past hour. The cache validity period is 30 minutes, which further reduces the query pressure of StarRocks.
[0071] In an optional embodiment of the present invention, the method for processing streaming service data further includes: Step 90: Monitor the data import, materialized view wide table refresh, and data query processing flow of the streaming business link, and issue alarms and fixes for abnormal processing flows.
[0072] This invention provides a monitoring and anomaly alarm service for the streaming business chain data processing process, enabling full-link monitoring of data import, materialized view wide table refresh, and data query services.
[0073] In this embodiment of the invention, a full-link monitoring system covering data import, data association (materialized view) and data service (query interface) is constructed, defining core indicators such as import delay, failure rate, refresh delay, and query timeout rate; and setting multi-level thresholds for abnormal alarms, and linking automatic repair mechanisms (such as automatically restarting failed tasks, re-consuming abnormal data ranges, and optimizing slow query indexes).
[0074] In a specific embodiment of the present invention, the data import, materialized view wide table refresh, and data query processing flow of the monitoring streaming service link, and the alarm and repair of the anomaly handling process, include: Step S901: Extract three monitoring indicators—data import indicator, materialized view refresh indicator, and query indicator—from the data import process, materialized view refresh process, and data query process, respectively.
[0075] Step S902: Based on each monitoring indicator and the corresponding preset threshold, determine whether the corresponding process is abnormal. If abnormal, trigger an information alarm.
[0076] Step S903: Based on the data import indicators and query indicators and their respective preset thresholds, determine whether each process is abnormal. If abnormal, start automatic repair.
[0077] The data import metrics include: data import latency and failure rate; the materialized view refresh metrics include: materialized view refresh latency; and the query metrics include: monitoring interface response time and its timeout rate.
[0078] This embodiment uses monitoring metrics to monitor data import, data association (materialized views), and data services (query interfaces). Data import metrics include: data import latency for each data import route load task (e.g., current time - data entry time); failure rate (e.g., number of failed records / total number of records). Materialized view refresh metrics include: materialized view refresh latency (e.g., time to complete data update to the wide table - time to complete data update to the primary key table). Query metrics include: interface response time and timeout rate (e.g., percentage of requests with a response time > 2 seconds).
[0079] In an optional embodiment of the present invention, step S902, which involves determining whether each process is abnormal based on each monitoring indicator and a corresponding preset threshold, and triggering an information alarm if an abnormality is found, includes: If the data import delay is greater than the delay threshold, the data import failure rate is greater than the failure threshold, and the materialized view refresh delay is greater than the refresh threshold, then the first information alarm will be triggered; if the query response time is greater than the query threshold and the timeout rate is greater than the timeout threshold, then the second information alarm will be triggered.
[0080] In this embodiment, the delay threshold can be 2 to 5 seconds, preferably 3 seconds; the failure threshold can be 7% to 15%, preferably 10%; the refresh threshold can be 7 to 15 seconds, preferably 10 seconds; the first information alarm can be an SMS and email alarm, and the second information alarm can be an SMS alarm. Those skilled in the art can set the corresponding thresholds and alarm methods according to the actual situation.
[0081] In other words, if the data import delay is greater than 3 seconds, the data import failure rate is greater than 10%, or the materialized view refresh delay is greater than 10 seconds, SMS and email level alerts will be triggered; if the query response time is greater than 2 seconds and the percentage exceeds 10%, an SMS alert will be triggered.
[0082] In an optional embodiment of the present invention, step 903, which involves determining whether each process is abnormal based on the data import indicators and query indicators and their respective preset thresholds, and initiating automatic repair if an abnormality is found, includes: If the data import failure rate is greater than the failure threshold, the data import task will be restarted and the subsequent unprocessed message range will be consumed again. If the timeout rate of the query interface is greater than the timeout threshold, analyze the slow query database and add indexes to speed up the query.
[0083] In other words, multiple thresholds are set for anomaly alerts, and an automatic repair mechanism is implemented (such as automatically restarting failed tasks, re-consuming abnormal data ranges, and optimizing slow query indexes). Data import repair: If the data import failure rate exceeds the failure threshold, the database management interface is automatically invoked to restart the corresponding data import task, specifying that it should start from the last successfully committed consumption offset in Kafka and re-consume the subsequent unprocessed message range to ensure data integrity. Query performance repair: If the query interface timeout rate exceeds the timeout threshold, slow query log analysis is automatically triggered, and query speed is optimized by creating indexes on the query condition fields in the materialized view wide table.
[0084] In one specific implementation, when the failure rate of the Routine Load task exceeds the failure threshold, the monitoring system automatically calls the StarRocks API to restart the task and re-consume the offset range that was not successfully imported from Kafka; when the timeout rate of the interface exceeds the standard, the system automatically analyzes slow query SQL and adds indexes to speed up the query.
[0085] The streaming business data processing method of this invention enables intelligent monitoring and self-healing by configuring monitoring and anomaly alarms for the streaming business chain data processing flow. System anomalies can be detected and automatically repaired within 10 minutes, reducing the manual intervention rate to below 30% and significantly improving system stability and operation and maintenance efficiency. The key indicator compliance rate (such as import delay ≤ 2 seconds) remains above 95%, ensuring the stability of data storage and querying of the Yiming business chain.
[0086] This invention can be applied to the fields of big data processing and intelligent management of communication services. Specifically, it relates to a method and system for real-time storage, automatic association, efficient querying and intelligent monitoring of business link data based on streaming wide table technology. It is applicable to scenarios of full-link data traceability, business monitoring, fault diagnosis and operation and maintenance management of communication value-added services (such as Yiming business card service).
[0087] The core application scenario of a streaming business data processing method according to an embodiment of the present invention, taking Yiming Business Card as an example, includes: Anomaly tracing scenario: When a user reports "not receiving SMS / flash messages", customer service can quickly obtain the link details by entering the "business number and call time combination" on the query page and clicking "query" to determine whether the cause is "signaling not received" or "the package limit has been used up", etc., and complete the problem investigation within 1 minute; Business monitoring scenario: Operations personnel can check whether the business is running normally through the "Yiming Business Alarm Dashboard". The dashboard is based on wide table data to count the abnormal percentage of indicators such as "signaling reception failure", "short / flash message delivery success rate" and "short / flash message delivery timeout" in the past 5 minutes. The business configuration can be adjusted in real time through the alarm indicators on the dashboard. Data analysis scenario: Calculate "SMS / Flash Message Display Success Rate of Each Province" and "SMS / Flash Message Reception Time of Each Province" based on wide table data. During offline analysis, the wide table data is directly called, avoiding multi-table joins and significantly improving analysis efficiency.
[0088] Ultimate improvement in data timeliness and query performance: Through two core technologies, real-time import via Routine Load and incremental refresh of materialized views at the second level, the latency from generation to queryability of link data is reduced from several minutes in traditional solutions to the second level, and the query response time is controlled within 2 seconds, perfectly supporting real-time business traceability scenarios.
[0089] Operations automation and significantly reduced costs: The end-to-end intelligent monitoring and self-healing system frees operations personnel from tedious daily inspections and manual fault recovery, reducing manual intervention by more than 70%. Meanwhile, configurable materialized views replace manual ETL script development, shortening the business change adaptation cycle from days to hours, greatly reducing labor costs.
[0090] System scalability and stability are significantly enhanced: the distributed architecture (Kafka, StarRocks) naturally supports horizontal scaling, easily handling daily processing volumes of billions of data points. The monitoring system's real-time awareness and automated handling of bottlenecks across the entire value chain effectively prevents overall service unavailability caused by localized failures.
[0091] Resource utilization and ROI optimization: The hot and cold data tiering scheme significantly reduces storage costs while maintaining performance. The query caching mechanism effectively reduces redundant calculations, supporting higher concurrent requests with fewer hardware resources and improving ROI.
[0092] like Figure 2 As shown, embodiments of the present invention also provide a streaming service data processing apparatus 20, comprising: The decomposition module 21 is used to decompose streaming business chain data containing multi-source heterogeneous streaming business data into multiple business links.
[0093] Processing module 22 is used to create a message queue and a primary key table for each of the multiple business processes; import the business process data of each business process into the corresponding message queue; import the business process data in the message queue into the corresponding primary key table according to the data import task to obtain the target primary key table; and associate the data of each business process with the target primary key table corresponding to each business process and the identifier of the streaming business chain data to obtain the materialized view wide table of the streaming business chain data.
[0094] Optionally, the streaming business chain data, which contains multi-source heterogeneous streaming business data, can be decomposed into multiple business links, including: Based on the state of the data sub-streams in the streaming business chain data, the streaming business chain data containing multi-source heterogeneous streaming business data is decomposed into multiple business links.
[0095] Optionally, a message queue and primary key table are created for each of the multiple business processes, including: Based on the number of business processes, create message queues in the message queue cluster that correspond one-to-one with each business process; and create primary key tables in the database that correspond one-to-one with each business process.
[0096] Optionally, based on the data import task, the business process data in the message queue is imported into the corresponding primary key table to obtain the target primary key table, including: Based on the configuration information of the data import task, consume business process data from the message queue; Extract the business fields and values of the consumption process data; Based on the field mapping rules in the configuration information, the business fields are written into the corresponding columns of the primary key table to obtain structured data; Based on the primary keys of the business process identifier and the identifier of the streaming business chain data, the structured data is written into the corresponding primary key table in an insert or update manner to obtain the target primary key table.
[0097] Optionally, based on the target primary key table corresponding to each business segment and the identifier of the streaming business chain data, the data of each business segment are associated to obtain a materialized view wide table of the streaming business chain data, including: Using the identifier of the streaming business chain data as the association key, all target primary key tables are associated with the target primary key tables corresponding to each business link through a left join operation of the database; Based on the core business fields in the target primary key table, a materialized view wide table of streaming business chain data is obtained. The core business fields include: basic information of the streaming business chain, status information of each business link, and timestamp of each business link.
[0098] Optionally, the processing module 22 is further used for Receive new streaming business chain data; Update the materialized view wide table based on the newly added streaming business chain data and its identifier.
[0099] This embodiment of the invention configures an incremental refresh strategy for the materialized view wide table: When data is inserted into any of the target primary key tables or existing data is updated, the database system will automatically trigger an incremental task. This task will only recalculate and refresh the records in the materialized view wide table associated with this batch of updated data, thereby ensuring near real-time performance of the wide table data. The read time period is Y seconds, where Y is less than 5 seconds; specifically, Y can be 1 second.
[0100] This task locates the streaming business chain data identifiers corresponding to the changed data and only refreshes the records associated with these identifiers in the materialized view wide table. By setting the delay threshold of this refresh strategy to 1 second, it ensures that the materialized view wide table can be updated in a very short time after the data is updated, generating a near real-time streaming wide table containing complete link information.
[0101] Optionally, the processing module 22 is further configured to receive a query request, wherein the query request carries at least: an identifier of the streaming business chain data; Based on the request parameters, a query operation is performed on the target primary key table corresponding to the target business process in the materialized view wide table to obtain the query results.
[0102] The system provides query services through a developed independent query interface to query materialized view wide tables. It supports queries based on identifiers of streaming business chain data or queries combining core business entity identifiers with time ranges. Query results are automatically converted to JSON format and pagination is supported.
[0103] In this embodiment, when the request parameter is an identifier (unique identifier) of the streaming business chain data, an equality query is performed; when the request parameter is a business entity identifier and a time range, a range query is performed. The query result is one or more wide table records containing complete information on all relevant business processes. The query results in the query result set are automatically converted into a predefined, semi-structured data format and support pagination, such as JSON format.
[0104] The processing module 22 is also used to cache the detailed data of the streaming business chain queried within a preset time period from the current time.
[0105] In addition, it receives query requests and parses the request parameters; if a query result corresponding to the request parameters already exists in the cache, it directly returns the cached result.
[0106] This invention employs a query caching strategy at the query interface layer to improve performance: a caching component (such as Redis) is introduced into the query interface service to cache detailed data records of recently frequently queried streaming business chains, and a cache validity period is set, i.e., a preset time period. This preset time period can be from 20 minutes to 1 hour, preferably 30 minutes. For query requests that hit the cache, the results are directly returned from the cache, thereby reducing the pressure of repeated database queries and further improving the response efficiency of the query interface and system throughput.
[0107] The apparatus of this embodiment further includes: a data partitioning module, used to partition the streaming business chain data according to the start timestamp, dividing historical data that exceeds a preset time into cold data and storing it in a cold storage medium; and dividing historical data that does not exceed the preset time into hot data and storing it in the local disk of the database node.
[0108] The data partitioning module implements a lifecycle management strategy for the data in the materialized view wide table.
[0109] The system in this embodiment of the invention also includes a monitoring and anomaly alarm module, which is used to monitor the data import, materialized view wide table refresh and data query processing flow of the streaming service link, and to alarm and repair the anomaly handling process.
[0110] This invention provides a monitoring and anomaly alarm service for the streaming business chain data processing process, enabling full-link monitoring of data import, materialized view wide table refresh, and data query services.
[0111] The monitoring process includes data import, materialized view wide table refresh, and data query processing of the streaming service link, as well as alarming and repairing the anomaly handling process. Data import metrics, materialized view refresh delay monitoring metrics, and query metrics monitoring metrics are extracted from the data import process, materialized view refresh process, and data query process, respectively.
[0112] Based on each monitoring indicator and the corresponding preset threshold, determine whether the corresponding process is abnormal. If abnormal, trigger an information alarm.
[0113] Based on the data import and query metrics and their respective preset thresholds, determine whether each process is abnormal, and if abnormal, initiate automatic repair.
[0114] The data import metrics include: data import latency and failure rate; the materialized view refresh latency monitoring metrics include: materialized view refresh latency time; and the query metrics include: monitoring interface response time and its timeout rate.
[0115] The process determines whether each process is abnormal based on each monitoring indicator and the corresponding preset threshold. If an abnormality is detected, an information alarm is triggered, including: if the data import delay is greater than the delay threshold, the data import failure rate is greater than the failure threshold, and the materialized view refresh delay is greater than the refresh threshold, then a first information alarm is triggered; if the query response time is greater than the query threshold and the timeout rate is greater than the timeout threshold, then a second information alarm is triggered.
[0116] The process is judged to be abnormal based on the data import indicators and query indicators and their respective preset thresholds. If abnormal, automatic repair is initiated, including: if the data import failure rate is greater than the failure threshold, the data import task is restarted and the subsequent unprocessed message range is re-consumed. If the timeout rate of the query interface is greater than the timeout threshold, analyze the slow query database and add indexes to speed up the query.
[0117] The monitoring and anomaly alerting module has built a full-link monitoring system covering data import, data processing and data services (query interface), defining core indicators such as import delay, failure rate, refresh delay and query timeout rate; and setting multi-level thresholds for anomaly alerts, and linking automatic repair mechanisms (such as automatically restarting failed tasks, re-consuming abnormal data ranges, and optimizing slow query indexes).
[0118] The device in this embodiment of the invention is the same as the method described above. All implementation processes in the method described above are applicable to this device and can achieve the same technical effect. Specifically, a data management platform is constructed that can perform real-time and automatic association and integration of multi-stage and multi-source heterogeneous streaming business data, and has full-link intelligent monitoring and self-healing capabilities. This ensures extremely high data processing timeliness while significantly reducing the system's operational complexity and manual intervention costs.
[0119] The embodiments of this invention construct an intelligent monitoring and self-healing device covering the entire data chain from access and processing to service. This device breaks through the limitations of traditional isolated and passive monitoring. By defining a progressive health index system (import delay > 3s, refresh delay > 10s, query timeout > 2s), it establishes a closed-loop automatic handling link, realizing real-time perception of the data pipeline status under high concurrency and high timeliness requirements. More importantly, it designs an automatic repair mechanism, such as automatically restarting routine load tasks and automatically analyzing slow SQL and adding indexes. It is no longer a simple "problem discovery - alarm - manual handling" but an intelligent closed-loop control system, reducing the manual intervention rate to below 30% and significantly improving the system's reliability and operational efficiency.
[0120] The device in this invention uses a globally unique business link ID as the core association key, horizontally connecting multiple asynchronous stages such as communication message reception, service triggering, package deduction, SMS / flash message delivery and receipt, to construct a unified query cross-stage real-time streaming wide table. It achieves deep integration of streaming data processing and materialized view technology. By configuring an incremental refresh strategy within 1 second, it ensures that the wide table can be updated within a very short latency during continuous data writing, thereby supporting dynamic concatenation and efficient querying of multiple real-time data streams, such as 5 Kafka topics. Compared with traditional solutions that only support batch processing or single-stream processing, this invention effectively solves the bottleneck problem of automatic association and query performance in the real-time generation process of multi-stage, multi-topic streaming data, achieving a second-level latency from data generation to queryability. This streaming wide table construction method significantly improves the efficiency of business problem localization, making real-time link tracing and anomaly diagnosis possible.
[0121] like Figure 3 As shown, the streaming business data processing system to which the above-described method of the present invention is applied includes: The data source layer is used to generate logs for each stage and receive external business-triggered events.
[0122] The data transmission layer is based on a constructed message queue cluster, which is divided into multiple independent message queue topics according to business processes, so as to realize parallel data transmission and temporary storage.
[0123] The data storage and computing layer consists of a constructed database cluster, which stores primary key tables corresponding to each business process. Based on the primary key tables, a materialized view wide table is generated through association operations, and a data import task is run to realize a continuous data flow from business process data in the message queue topic to the database.
[0124] In the data application layer, the deployed query interface service encapsulates the query logic for materialized view wide tables, providing a simplified data query access point to external business systems.
[0125] Data flow: Source layer - Data transmission layer corresponding topic - Data import task of data storage and computing layer - Primary key table - Materialized view wide table - Query interface of data application layer - External business system.
[0126] In a specific implementation, the data source layer includes: a business server (generating logs for each stage) and signaling data from each province. The data transmission layer uses a Kafka cluster, splitting message queue topics according to business stages to achieve high throughput and low latency data storage. For example, the Kafka cluster is configured with 5 Broker nodes, each with 16 cores and 32GB of memory. Each of the 5 topics is allocated 5 partitions and 3 replicas. The data storage and computation layer uses a StarRocks cluster as the core storage, storing primary key tables, materialized views, and data import routine load tasks. Taking Yimingban as an example, the data storage layer includes 5 FE nodes and 9 BE nodes, 5 primary key tables, and 1 wide materialized view table. The data application layer provides a query service that encapsulates the materialized view query logic, supporting multi-condition combined queries. The data application layer is deployed on 3 application servers, using Nginx load balancing, and provides an HTTP interface for use by business systems such as customer service platforms, operation monitoring platforms, and data analysis platforms. Business server logs / signaling from various provinces - Kafka corresponding topics - Routine Load tasks - StarRocks primary key table - materialized view wide table - query interface - business system.
[0127] The system in this embodiment of the invention also includes a monitoring system that monitors the data source layer, data transmission layer, data storage layer, and data application layer, spanning all layers to ensure system health and data quality. It covers full-link indicator monitoring such as Kafka message backlog, data import Routine Load task success rate, materialized view refresh latency, and interface response time, ensuring stable system operation.
[0128] Another embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the streaming business data processing method described above.
[0129] Another embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, performs the steps of the streaming business data processing method described above.
[0130] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.
Claims
1. A method for processing streaming business data, characterized in that, include: Decompose the streaming business chain data, which contains multi-source heterogeneous streaming business data, into multiple business links; Create a message queue and a primary key table for each of the multiple business processes; Import the business process data of each business step into the corresponding message queue; Based on the data import task, the business process data in the message queue is imported into the corresponding primary key table to obtain the target primary key table; Based on the target primary key table and the identifier of the streaming business chain data corresponding to each business segment, the data of each business segment are associated to obtain the materialized view wide table of the streaming business chain data.
2. The method for processing streaming business data according to claim 1, characterized in that, The streaming business chain data, which contains multi-source heterogeneous streaming business data, is decomposed into multiple business links, including: Based on the state of the data sub-streams in the streaming business chain data, the streaming business chain data containing multi-source heterogeneous streaming business data is decomposed into multiple business links.
3. The method for processing streaming business data according to claim 1, characterized in that, For each of the multiple business processes, a message queue and a primary key table are created, including: Based on the number of business processes, create message queues in the message queue cluster that correspond one-to-one with each business process; and create primary key tables in the database that correspond one-to-one with each business process.
4. The method for processing streaming business data according to claim 1, characterized in that, Based on the data import task, the business process data in the message queue is imported into the corresponding primary key table to obtain the target primary key table, including: Based on the configuration information of the data import task, consume business process data from the message queue; Extract the business fields and values of the consumption process data; Based on the field mapping rules in the configuration information, the business fields are written into the corresponding columns of the primary key table to obtain structured data; Based on the primary keys of the business process identifier and the identifier of the streaming business chain data, the structured data is written into the corresponding primary key table in an insert or update manner to obtain the target primary key table.
5. The method for processing streaming business data according to claim 1, characterized in that, Based on the target primary key table corresponding to each business segment and the identifier of the streaming business chain data, the data of each business segment are associated to obtain a materialized view wide table of the streaming business chain data, including: Using the identifier of the streaming business chain data as the association key, all target primary key tables are associated with the target primary key tables corresponding to each business link through a left join operation of the database; Based on the core business fields in the target primary key table, a materialized view wide table of streaming business chain data is obtained. The core business fields include: basic information of the streaming business chain, status information of each business link, and timestamp of each business link.
6. The method for processing streaming business data according to claim 1, characterized in that, Also includes: Receive new streaming business chain data; Update the materialized view wide table based on the newly added streaming business chain data and its identifier.
7. The method for processing streaming business data according to claim 1, characterized in that, Also includes: Receive a query request, wherein the query request carries at least: an identifier of the streaming business chain data; Based on the request parameters, a query operation is performed on the target primary key table corresponding to the target business process in the materialized view wide table to obtain the query results.
8. A streaming business data processing device, characterized in that, include: The decomposition module is used to decompose streaming business chain data, which contains multi-source heterogeneous streaming business data, into multiple business links. The processing module is used to create a message queue and a primary key table for each of the multiple business processes; import the business process data of each business process into the corresponding message queue; import the business process data in the message queue into the corresponding primary key table according to the data import task to obtain the target primary key table; and associate the data of each business process with the target primary key table corresponding to each business process and the identifier of the streaming business chain data to obtain the materialized view wide table of the streaming business chain data.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method as described in any one of claims 1 to 7.