Streaming calculation method and related device

Abstract

The invention provides a streaming computing method and a related device. The method comprises the following steps: writing a high-timeliness data item output by a stream processing layer into a first sub-table; wherein the stream processing layer is used for performing stream-oriented calculation on a data stream to obtain a high-timeliness data item; wherein the first sub-table and the second sub-table respectively comprise a validity field; the field value of the validity field of the high-timeliness data item is a first value representing a valid state; writing a low-timeliness data item output by the batch processing layer into a second sub-table, and setting a validity field in a high-timeliness data item corresponding to the low-timeliness data item as a second value representing an invalid state; wherein a validity field in the low-timeliness data item is set as a first value; and when a query request is received, responding to the query request based on the high-timeliness data item with the validity field being the first value in the first sub-table and / or the low-timeliness data item with the validity field being the first value in the second sub-table. The data consistency can be improved.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and in particular to a streaming computing method and related apparatus. Background Technology

[0002] In related technologies, in the field of data processing that integrates streaming computing and batch processing, a layered processing approach is usually adopted. That is, the streaming processing layer performs calculations on the real-time data stream to generate timely data results, while the batch processing layer performs offline calculations on historical data to generate infrequent data results, and these results are stored in different databases or tables.

[0003] While this processing method can support both real-time and offline data analysis, the independent computation and storage of the stream processing layer and the batch processing layer lead to inconsistencies between the results of the stream processing layer and the offline results of the batch processing layer in the same transaction scenario. Summary of the Invention

[0004] In view of this, one or more embodiments of this application provide a streaming computing method and related apparatus, which can improve data consistency to a certain extent.

[0005] In a first aspect, one or more embodiments of this application propose a streaming computation method, comprising: writing high-time-sensitivity data items output by a streaming processing layer into a first sub-table; wherein the streaming processing layer is used to perform streaming computation on a data stream to obtain the high-time-sensitivity data items; wherein the first sub-table and the second sub-table belong to the same main table, and the first sub-table and the second sub-table respectively include data fields of the main table, and the data fields of the main table include a validity field indicating data validity; the high-time-sensitivity data items include field values ​​corresponding to the data fields of the first sub-table, and the validity field of the high-time-sensitivity data items... The field value is a first value indicating a valid state; the low-timeliness data items output by the batch processing layer are written into the second sub-table, and the validity field of the high-timeliness data item corresponding to the low-timeliness data item is set to a second value indicating an invalid state; wherein, the low-timeliness data item includes field values ​​corresponding to the data fields of the second sub-table, and the validity field of the low-timeliness data item is set to the first value; upon receiving a query request, the query request is responded to based on the high-timeliness data item in the first sub-table with the validity field set to the first value and / or the low-timeliness data item in the second sub-table with the validity field set to the first value.

[0006] Secondly, one or more embodiments of this application propose a streaming computing device, comprising: a first writing module, configured to write high-time-sensitivity data items output by a streaming processing layer into a first sub-table; wherein the streaming processing layer is configured to perform real-time calculations on the data stream to derive the high-time-sensitivity data items; wherein the first sub-table and the second sub-table belong to the same main table, and the first sub-table and the second sub-table respectively include data fields of the main table, and the data fields of the main table include a validity field indicating data validity; the high-time-sensitivity data item includes a field value corresponding to a data field of the first sub-table, and the field value of the validity field of the high-time-sensitivity data item is... The system includes: a first value representing a valid state; a second writing module, configured to write low-time-sensitive data items output by the batch processing layer into a second sub-table, and set the validity field of the high-time-sensitive data item corresponding to the low-time-sensitive data item to a second value representing an invalid state; wherein the low-time-sensitive data item includes field values ​​corresponding to data fields in the second sub-table, and the validity field of the low-time-sensitive data item is set to the first value; and a receiving response module, configured to respond to the query request based on the high-time-sensitive data item in the first sub-table with the validity field set to the first value and / or the low-time-sensitive data item in the second sub-table with the validity field set to the first value upon receiving a query request.

[0007] Secondly, one or more embodiments of this application provide a computer device including a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the method described above.

[0008] Thirdly, one or more embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the method as described above.

[0009] Fourthly, one or more embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method as described above.

[0010] As can be seen from the above embodiments, multiple embodiments of this application achieve unified management of high-timeliness data items output by the stream processing layer and low-timeliness data items output by the batch processing layer by setting a table structure including a first sub-table and a second sub-table in the main table, and introducing a validity field for identifying data validity in the main table. When a high-timeliness data item from the stream processing layer is written to the first sub-table, its validity field is set to a first value indicating a valid state; when a low-timeliness data item from the batch processing layer is written to the second sub-table, the validity field of the corresponding high-timeliness data item is updated to a second value indicating an invalid state, so that the result of the batch processing layer can replace the result of the stream processing layer without causing data conflicts. When a query request is received, the response can be based solely on data items whose validity field has the first value. Thus, the technical solution of this application improves both data consistency and the accuracy of queried data. Attached Figure Description

[0011] Figure 1 This is a scenario example diagram illustrating the interaction between a streaming computing device and an e-commerce system, provided in this manual.

[0012] Figure 2 This is a flowchart illustrating a streaming computing method provided in one embodiment of this specification.

[0013] Figure 3 This is a schematic diagram of the relationship between a stream processing layer, a batch processing layer, and a master table, provided in one embodiment of this specification.

[0014] Figure 4 This is a schematic diagram of a stream processing layer and a first sub-table provided in one embodiment of this specification.

[0015] Figure 5 This is a block diagram of a streaming computing device provided in one embodiment of this specification.

[0016] Figure 6 This is a schematic diagram of a computer device provided in one embodiment of this specification. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments.

[0018] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0019] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0020] In related technologies, the data processing field that integrates streaming computing and batch processing typically employs a layered design. The streaming layer is responsible for computing real-time or near-real-time data, generating real-time or near-real-time results with high timeliness, while the batch processing layer performs offline computing on historical data, generating summary results with lower timeliness. These two types of results are stored in different tables or databases to support the parallel execution of real-time or near-real-time analysis and offline analysis. However, because the streaming and batch processing layers are independent in terms of computational logic and storage structure, when real-time tasks experience delays or restarts, or when batch processing tasks generate overlapping results, inconsistencies in the computational results for the same data item across different layers can occur, leading to data conflicts or incorrect query results.

[0021] A unified storage structure can be used to integrate streaming and batch data, reducing data fragmentation and synchronization costs. By managing both types of data uniformly, the complexities of multi-table concatenation and multi-database data alignment can be avoided, thereby improving the real-time performance of data access. However, inconsistencies between streaming and batch data still exist, which can lead to incorrect query results.

[0022] In several embodiments provided in this application, the streaming computing method can be applied to electronic devices with certain computing power and network access capabilities. These electronic devices can be desktop computers, laptops, tablets, smartphones, or servers. Specifically, the electronic device includes a processor, memory, and a network access module for network communication. The server can be an electronic device with strong data processing capabilities; of course, a server can also refer to a server cluster formed by multiple electronic devices, or a quantum server built using a quantum computer.

[0023] Please see Figure 1 One embodiment of this application provides an example application scenario of a streaming computing method. The streaming computing method can be applied to streaming computing devices. Streaming computing devices can achieve unified storage and query response for high-timeliness and low-timeliness data through collaborative computation of real-time data streams and offline log data, thereby improving the consistency and accuracy of data analysis. In this scenario example, taking an e-commerce system as an example, specifically, using order data generated by users on the e-commerce platform as initial data, the streaming computing device, through the collaborative operation of the stream processing layer and the batch processing layer, achieves real-time status updates and periodic statistical analysis of orders.

[0024] For example, when a user places an order for goods in an e-commerce system, the system generates a real-time order event containing fields such as order number, user identifier, order time, product ID, and order amount. The stream processing layer performs streaming computation on this real-time order event, forming corresponding high-time-sensitivity data items and writing them to the first sub-table. Simultaneously, the validity field of this high-time-sensitivity data item is set to the first value indicating a valid status. Subsequently, as the order process progresses, real-time events such as payment completion, shipment, and receipt will be generated. The stream processing layer can merge and write the field value groups obtained from different stream data according to the identifier field, allowing the high-time-sensitivity data items corresponding to the same identifier field in the first sub-table to gradually complete the order lifecycle information. For user-side query requests, an appropriate physical storage structure can be selected based on the type of query request.

[0025] Meanwhile, within a preset offline scheduling period, the batch processing layer performs verification, deduplication, and aggregation based on the log data of the e-commerce system. It generates low-time-sensitivity data items for the same initial order data and writes them to the second sub-table. It also updates the validity field of the high-time-sensitivity data item corresponding to the low-time-sensitivity data item to a second value indicating an invalid state, so as to improve the consistency of data and the accuracy of results when responding to query requests.

[0026] For example, a data table storing data primarily used to respond to queries requesting source data can have a row-based physical storage structure as its main table. The streaming computing device can first quickly filter out high-time-sensitivity data items with values ​​of the first specified value from the first sub-table based on validity fields, and retrieve low-time-sensitivity data items with values ​​of the first specified value from the second sub-table; then, it returns at least some field values ​​according to the requested field range, thus completing the real-time display of order details in milliseconds. Row-based tables offer relatively fast response times for queries requiring rapid response and with relatively small data volumes. In some examples, for data tables primarily used to respond to queries requesting data analysis, the main table can be a column-based table. In scenarios such as business analysis, trend statistics, or report generation, the streaming computing device filters out high-time-sensitivity data items with values ​​of the first specified value from the first sub-table based on validity fields, and filters out low-time-sensitivity data items with values ​​of the first specified value from the second sub-table based on validity fields, and performs data analysis processing such as grouping, aggregation, sorting, or interval statistics on at least some of their field values ​​to obtain the required analytical data.

[0027] In this example scenario, the main table contains an identifier field and a generation date field. The identifier field identifies the data relationships between different sub-tables for the same order data, such as the order number. The generation date field records the generation dates of high-sensitivity and low-sensitivity data items, allowing the streaming computing device to partition data and manage clustered indexes based on the time dimension. For example, the streaming computing device can automatically create a clustered index for daily generated order data to improve query efficiency. Simultaneously, the streaming computing device sets a bitmap index on the validity field of both the first and second sub-tables to quickly filter currently valid data items, accelerating query performance in large-scale data scenarios.

[0028] Furthermore, after the batch processing layer generates new low-time-sensitivity data items, the streaming computing device automatically cleans up high-time-sensitivity data items marked as invalid in the first sub-table at specified time intervals, thereby maintaining the controllability of table data size and avoiding redundant accumulation. For multi-dimensional data with the same order number, such as order creation information and payment information, the streaming processing layer merges the first field value group (such as basic order information) output from the first stream data with the second field value group (such as payment status information) output from the second stream data, allowing high-time-sensitivity data items to be dynamically updated under the same identifier field, gradually forming complete order lifecycle data. For example, when a user completes payment, the streaming processing layer can capture the payment event in real time, merge the second field value group (payment status, payment time, etc.) with the first field value group of the original order creation event, thereby forming a high-time-sensitivity data item containing the complete transaction status in the first sub-table. At the same time, in the next offline task, the batch processing layer will read the same order data based on log data, comprehensively verify and correct information such as order status and amount statistics, and generate low-time-sensitivity data items.

[0029] Please see Figure 2 and Figure 3 One embodiment of this application provides a streaming computing method. The streaming computing method can be applied to a streaming computing device, which can be applied to the aforementioned electronic device possessing certain computing power and network access capabilities. Of course, in some embodiments, the streaming computing device can also be software running on an electronic device. The streaming computing method may include the following steps.

[0030] Step S110: Write the high-timeliness data items output by the stream processing layer into the first sub-table; wherein the stream processing layer is used to perform streaming computation on the data stream to obtain the high-timeliness data items; wherein the first sub-table and the second sub-table belong to the same main table, and the first sub-table and the second sub-table respectively include the data fields of the main table, and the data fields of the main table include a validity field indicating data validity; the high-timeliness data items include field values ​​corresponding to the data fields of the first sub-table, and the field value of the validity field of the high-timeliness data items is a first value indicating a valid state.

[0031] Step S120: Write the low-time-sensitive data items output by the batch processing layer into the second sub-table, and set the validity field of the high-time-sensitive data items corresponding to the low-time-sensitive data items to a second value indicating an invalid state; wherein, the low-time-sensitive data items include field values ​​corresponding to the data fields of the second sub-table, and the validity field of the low-time-sensitive data items is set to a first value.

[0032] Step S130: Upon receiving a query request, respond to the query request based on the high-time-sensitivity data items in the first sub-table whose validity field has a first value and / or the low-time-sensitivity data items in the second sub-table whose validity field has a first value.

[0033] In this embodiment, the stream processing layer performs streaming computation on real-time data streams to generate high-time-sensitivity data items. High-time-sensitivity data items characterize real-time or near-real-time transaction states and are suitable for scenarios with high timeliness requirements. The streaming computation device processes the collected real-time or near-real-time data through predefined data stream processing logic to obtain the field values ​​of the high-time-sensitivity data items. The high-time-sensitivity data items are written to a first sub-table, which is part of the main table and shares the main table's data field structure with the second sub-table. The main table is used to uniformly manage data items from different sources, and its data fields may include, but are not limited to: a transaction identifier field, a generation time field, a statistical field, and a validity field indicating data validity. The validity field indicates the availability status of the data item in a query scenario. When the validity field value is the first value, the data item is valid; when the validity field value is the second value, the data item is invalid. When a high-time-sensitivity data item is written to the first sub-table, the streaming computation device can automatically set the value of its validity field to the first value indicating a valid state, so that it can be selected during a query.

[0034] In this embodiment, the batch processing layer performs offline calculations on historical or full data to generate low-time-sensitivity data items. These low-time-sensitivity data items typically contain data results that have undergone periodic statistical analysis, summarization, or correction, used to supplement data that may have been missed or whose calculation accuracy was insufficient during the stream processing stage. After the batch processing layer completes its calculations, the stream processing device writes the low-time-sensitivity data items into a second sub-table corresponding to the main table. The data fields of the second sub-table are consistent with those of the first sub-table, and when the low-time-sensitivity data item is written, its validity field value is set to the first value indicating a valid state. Simultaneously, to ensure consistency between the stream processing results and the batch processing results, the stream processing device can locate the corresponding high-time-sensitivity data item based on the identifier field in the low-time-sensitivity data item and update the validity field of that high-time-sensitivity data item to the second value indicating an invalid state. Through this update operation, the low-time-sensitivity data items output by the batch processing layer become effective in the query results, while the high-time-sensitivity data items output by the stream processing layer automatically become invalid, thus achieving replacement and unified management of the same data item within the main table.

[0035] In this implementation, the master table serves as a unified storage structure, storing the data content of the first and second sub-tables and maintaining the logical relationship between the two types of data items. In some implementations, the master table can be implemented using a hybrid row-column storage mechanism, for example, storing real-time or near-real-time data partitions and offline data partitions separately under the same logical table to achieve a performance balance between high-concurrency writes and batch queries. The validity field in the master table is used to quickly filter invalid data, ensuring that only currently valid data items are selected for calculation or display during the query process.

[0036] In this implementation, upon receiving a query request, the system can automatically filter data items that meet the validity criteria from the main table based on the type of the query request. If the query request involves real-time transaction metrics or the display of the latest data, the streaming computing device can respond based on high-timeliness data items in the first sub-table whose validity field has a first value. If the query request involves statistical analysis, trend calculation, or historical summarization, the streaming computing device can respond based on low-timeliness data items in the second sub-table whose validity field has a first value. If the query scenario requires the fusion of real-time and historical results, the streaming computing device can jointly read and aggregate the valid data from the first and second sub-tables during the same query process, thereby ensuring the completeness and accuracy of the query results.

[0037] In this implementation, unified storage and state management of output data from both the stream processing layer and the batch processing layer are achieved under the same master table structure, effectively alleviating the data fragmentation problem caused by the separation of real-time and offline tables in traditional technologies. After the batch processing layer generates results, the validity fields of highly time-sensitive data items are automatically updated, ensuring that historical statistical results accurately replace real-time calculation results without conflicts or redundancy. Therefore, this implementation improves data consistency and the accuracy of query results while achieving unified data management.

[0038] In some implementations, the main table includes an identifier field; the identifier field has the same value in the corresponding high-time-sensitivity data items and low-time-sensitivity data items.

[0039] In this implementation, the master table may include an identifier field to identify the source of data items and the transaction object. The identifier field uniquely identifies data records of the same transaction object in both the stream processing layer and the batch processing layer, enabling the streaming computing device to establish a one-to-one correspondence between different data items. The identifier field can be a single field or a composite primary key composed of multiple fields, such as a combination of order number, user identifier, and timestamp. In specific implementations, the value of the identifier field remains consistent across high-sensitivity and low-sensitivity data items, thereby ensuring that the two types of data items logically correspond to the same source data object.

[0040] During data writing, when the stream processing layer generates high-time-sensitivity data items and writes them to the first sub-table, the streaming computing device assigns a corresponding identifier field value to the high-time-sensitivity data item to indicate the transaction object to which the data item belongs. When the batch processing layer generates low-time-sensitivity data items, the streaming computing device can write them to the second sub-table based on the same identifier field value to ensure consistency between the low-time-sensitivity data item and the corresponding high-time-sensitivity data item in the identifier field. Through this consistent mapping relationship, the streaming computing device can accurately locate the corresponding high-time-sensitivity data item after the batch processing layer writes data and perform an update operation on the validity field, setting the validity field of the original high-time-sensitivity data item to a second value indicating an invalid state.

[0041] In query or data maintenance scenarios, the identifier field is also used to assist the system in data tracking and updating within the main table. Streaming computing devices can quickly retrieve the corresponding data items of a transaction object in the first and second sub-tables based on the identifier field value, thereby achieving cross-sub-table data matching.

[0042] In some implementations, the main table corresponding to a query request for source data is a row-based table; wherein, the query request for source data is used to request at least some field values ​​of high-timeliness data items in the first sub-table and to request at least some field values ​​of low-timeliness data items in the second sub-table; the main table corresponding to a query request for data analysis is a column-based table; wherein, the query request for data analysis is used to request the acquisition of analysis data; wherein, the query request for data analysis is used to request data analysis processing based on at least some field values ​​of high-timeliness data items in the first sub-table and at least some field values ​​of low-timeliness data items in the second sub-table to obtain analysis data.

[0043] In this embodiment, to further improve the response performance of the streaming computing device to different types of query requests, the physical storage structure of the corresponding main table can be matched according to the type of query request. Specifically, for query requests that request source data, the main table adopts a row-based storage structure; for query requests that request data analysis, the main table adopts a column-based storage structure. By matching different physical storage structures to different characteristics of query requests, real-time or near-real-time queries and offline analysis can both achieve better response speeds within the same data system.

[0044] In some implementations, a query request requesting source data is used to request at least some field values ​​of high-timeliness data items in a first sub-table and at least some field values ​​of low-timeliness data items in a second sub-table. For example, in a real-time order monitoring scenario of an e-commerce system, a query request requesting source data can be used to obtain currently generated or updated order records in real-time or near real-time. When responding to such a request, the streaming computing device can first filter high-timeliness and low-timeliness data items based on the validity field, where the field value represents a first value indicating the validity status. Subsequently, the streaming computing device can extract at least some field values ​​from the corresponding high-timeliness and low-timeliness data items according to the set of fields specified in the parameters of the query request and return them to the requester. Since row-based storage tables are stored contiguously in rows in the physical storage structure, which is beneficial for fast writing and low-latency random reading, the streaming computing device can complete the response to the query request in a very short time (e.g., milliseconds), thereby meeting the data usage requirements of high-concurrency access and near-real-time data display.

[0045] In some implementations, a query request for data analysis is used to request analytical data. This analytical data is obtained by processing at least some field values ​​of high-timeliness data items in a first sub-table and at least some field values ​​of low-timeliness data items in a second sub-table. For example, in sales trend analysis or user behavior statistics scenarios in e-commerce systems, a query request for data analysis can be used to aggregate order amounts, calculate product sales volume, or generate transaction reports by time or region. When processing such requests, the streaming computing device also filters high-timeliness and low-timeliness data items based on the validity field, selecting those with the first value. The streaming computing device then performs analysis operations such as aggregation, sorting, grouping, or association to generate statistical results that conform to the logic of the transaction. Because column-oriented tables store and compress data column by column, they can significantly improve data reading efficiency and parallel computing efficiency when reading large numbers of field values ​​and performing multi-dimensional aggregation calculations. Therefore, they can balance calculation accuracy and execution efficiency when performing analysis operations such as aggregation, sorting, grouping, or association.

[0046] In some implementations, the data fields of the main table include a generation date field; the validity fields of the first sub-table and the second sub-table are collectively set with bitmap indexes, and the generation date fields of the first sub-table and the second sub-table are respectively set with clustered indexes; wherein, the field value of the generation date field is used to characterize the generation date of the initial data corresponding to the high-timeliness data item or the low-timeliness data item.

[0047] In this embodiment, the streaming computing device further includes a generation date field in the main table's data structure design to identify the generation time of the source data. The generation date field characterizes the generation date of the initial data corresponding to high-timeliness or low-timeliness data items, and its value is typically recorded in a timestamp or standard date format to reflect the temporal sequence attribute of the data generation. By introducing the generation date field, the streaming computing device can distinguish data generated in different time periods at the storage level, thereby achieving efficient time range limiting during querying, aggregation, and cleaning. Taking an e-commerce system as an example, the initial data can be actual transaction data generated by the platform, such as order data, payment records, or refund data. Each order data item has a definite generation date, which is stored as the field value of the corresponding data item's generation date field.

[0048] In this embodiment, to improve data retrieval and filtering efficiency, the streaming computing device establishes different types of index structures for key fields in the first and second sub-tables, respectively. Specifically, a bitmap index is set up for the validity fields in both types of sub-tables. A bitmap index is an index structure that represents the value state of a field in a bit vector manner, suitable for fields with a limited value range but a large data volume. Since the field value of a validity field only includes a first value indicating a valid state and a second value indicating an invalid state, using a bitmap index can quickly identify valid or invalid data items in the main table, significantly reducing the amount of data scanned for invalid data during the query process. When the streaming computing device receives a query request, it can directly filter out high-timeliness or low-timeliness data items with the first value of the validity field through the bitmap index, thereby achieving efficient location of valid data.

[0049] In this embodiment, to further improve time-based query performance, the streaming computing device sets clustered indexes for the generation date field in both the first and second sub-tables. A clustered index is an index structure that organizes physical storage according to the order of field values, ensuring that records with adjacent dates are also stored adjacently, thereby significantly reducing storage access overhead in time-range queries or batch scan scenarios. For example, when analyzing order statistics or sales trends within the past 7 days in an e-commerce system, the streaming computing device can quickly locate high- or low-time-sensitivity data items for the generation date field within a specified time interval using a clustered index, avoiding full table scans and thus accelerating data aggregation and statistical analysis.

[0050] In this implementation, by setting bitmap indexes and clustered indexes on the key fields of the main table, the streaming computing device simultaneously achieves efficient filtering and location of validity status and generation time within the same data architecture. Bitmap indexes are used to quickly filter out currently valid data items during queries, while clustered indexes can be used to efficiently limit the time range within the valid dataset, thereby enabling fast retrieval and ordered access to both high- and low-time-sensitivity data.

[0051] In some implementations, the streaming computing device may delete the highly time-sensitive data item from the first sub-table after a specified interval, provided that the validity field of the highly time-sensitive data item has been modified to a second value.

[0052] In this embodiment, to prevent data accumulation and storage resource waste caused by the prolonged retention of high-sensitivity data items in the main table after they have been replaced by the batch processing layer, the streaming computing device also has a time-interval-based high-sensitivity data item cleanup mechanism. Specifically, when low-sensitivity data items output by the batch processing layer are written to the second sub-table, the streaming computing device modifies the validity field of the corresponding high-sensitivity data item from a first value indicating a valid state to a second value indicating an invalid state. To maintain transaction continuity and data consistency, the streaming computing device does not immediately delete the high-sensitivity data item, but instead sets a configurable time delay parameter to control the retention period of the high-sensitivity data item.

[0053] In this embodiment, when the streaming computing device detects that the validity field of a high-time-sensitivity data item has been modified to a second value, it initiates a delayed deletion timer task. After the delay reaches a specified duration (e.g., 30 minutes, 1 hour, 5 days, or any duration determined by system configuration), the streaming computing device automatically deletes the corresponding high-time-sensitivity data item from the first sub-table. This specified duration can be dynamically set according to transaction characteristics. For example, in real-time or near-real-time monitoring scenarios with high data update frequency, the specified duration can be set shorter to accelerate data cleanup; while in transactional systems sensitive to historical backtracking, it can be set longer to retain short-term traceability and query capabilities.

[0054] Please see Figure 4 In some implementations, the streaming computing device may receive a first group of field values ​​output by the streaming layer based on a first stream of data, and a second group of field values ​​output by the streaming layer based on a second stream of data; the first group of field values ​​and the second group of field values ​​each include at least one field value, and the data field corresponding to the first group of field values ​​and the data field corresponding to the second group of field values ​​are partially different; the field values ​​of the identifier field corresponding to the first group of field values ​​and the second group of field values ​​are written into the first sub-table; wherein, high-timeliness data items include a first group of field values ​​and a second group of field values ​​corresponding to the field values ​​of the same identifier field.

[0055] In this embodiment, to support the flexible fusion and unified storage of multi-source data in the high-timeliness layer, the streaming computing device can receive the computation results of different streaming data from the streaming processing layer and combine them to generate a complete high-timeliness data item. Specifically, the streaming computing device can receive a first field value group output by the streaming processing layer based on the first streaming data, and a second field value group output by the streaming processing layer based on the second streaming data. The first streaming data and the second streaming data can originate from different data streams of the same transaction event. For example, in an e-commerce system, the first streaming data can be a real-time order generation event stream, and the second streaming data can be a corresponding payment status update event stream.

[0056] In this embodiment, the first field value group and the second field value group each include at least one field value to describe the characteristics of high-timeliness data in different dimensions. For example, the first field value group may include field values ​​such as order number, user identifier, and order time, while the second field value group may include field values ​​such as order number, payment status, payment time, and payment method. Due to the layered processing of transaction logic, the data fields corresponding to the first field value group and the second field value group are partially different, thereby enabling the stream processing layer to process real-time stream data of different dimensions in parallel and output results independently, improving data computing throughput and system scalability.

[0057] In this embodiment, after receiving the two types of field value groups, the streaming computing device writes them into the first sub-table according to the field value of the common identifier field in the field value groups, thereby completing the fusion and storage of high-time-sensitive data under the main table structure. The identifier field is used to identify data belonging to the same transaction entity, such as the order number of an e-commerce order, or the consultation number or registration number in an internet medical system. When the streaming computing device detects that two sets of field value groups have the same identifier field value, it will regard them as components of the same high-time-sensitive data item and store them in the first sub-table in a merged manner.

[0058] In some implementations, the high-time-efficiency data and the low-time-efficiency data are generated based on the same initial data; wherein, the initial data is order data from a specified e-commerce system.

[0059] In this embodiment, both the high-timeliness and low-timeliness data in the streaming computing device are generated based on the same initial data. The initial data refers to data generated by an external transaction processing system. The stream processing layer and batch processing layer can process data generated by the same event at different granularities and timeframes, thereby ensuring consistency in the data source between high-timeliness and low-timeliness data items.

[0060] In some implementations, the initial data can be order data from a designated e-commerce system. This order data includes multiple field values ​​such as order number, user identifier, product identifier, order amount, order time, payment method, payment status, delivery information, and logistics information, which can completely reflect the order generation, payment, and fulfillment process. After receiving real-time event stream data such as order creation, payment update, and shipment, the stream processing layer processes the above field values ​​in real-time or near real-time to generate corresponding high-time-sensitivity data items and writes them into the first sub-table to support real-time or near real-time monitoring and rapid response to transaction queries. Furthermore, the designated e-commerce system can correspond to log data of the initial data record. Based on the initial data that can be obtained by reading the log data, the batch processing layer executes offline computing tasks within a specified time period to perform operations such as deduplication, completion, verification, and aggregation on the order data read from the log data to generate low-time-sensitivity data items and writes them into the second sub-table. These low-time-sensitivity data items have higher data completeness and accuracy than the high-time-sensitivity data items and are used for subsequent statistical analysis, trend prediction, or transaction report generation. For example, in the daily sales analysis scenario of an e-commerce system, the stream processing layer can output sales and order volume every minute in real time, while the batch processing layer performs a full aggregation based on the same order data at midnight every day to generate corrected and verified offline analysis results.

[0061] Please see Figure 5 Embodiments of this application also provide a streaming computing device. The streaming computing device may include: a first writing module, a second writing module, and a receiving response module.

[0062] The first writing module is used to write high-time-sensitivity data items output by the stream processing layer into a first sub-table; wherein the stream processing layer is used to perform real-time calculations on the data stream to obtain the high-time-sensitivity data items; wherein the first sub-table and the second sub-table belong to the same main table, and the first sub-table and the second sub-table respectively include data fields of the main table, and the data fields of the main table include a validity field indicating data validity; the high-time-sensitivity data item includes field values ​​corresponding to the data fields of the first sub-table, and the field value of the validity field of the high-time-sensitivity data item is a first value indicating a valid state.

[0063] The second writing module is used to write the low-time-sensitivity data items output by the batch processing layer into the second sub-table, and set the validity field of the high-time-sensitivity data item corresponding to the low-time-sensitivity data item to a second value indicating an invalid state; wherein, the low-time-sensitivity data item includes field values ​​corresponding to the data fields of the second sub-table, and the validity field of the low-time-sensitivity data item is set to a first value.

[0064] The response receiving module is configured to, upon receiving a query request, respond to the query request based on high-time-sensitivity data items in the first sub-table whose validity field has a first value and / or low-time-sensitivity data items in the second sub-table whose validity field has a first value.

[0065] In this embodiment, the functions and effects achieved by the streaming computing device can be explained in comparison with the aforementioned embodiments, and will not be repeated here.

[0066] Please see Figure 6 This application also provides a computer device comprising: a memory and a processor, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the method described above.

[0067] The memory, processor, and communication interface in the computer device can communicate with each other via the system bus and network communication.

[0068] In this embodiment, the functions and effects implemented by the computer device can be explained by referring to the foregoing embodiments, and will not be repeated here.

[0069] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method as described above.

[0070] The functions and effects achieved in this embodiment can be explained by referring to other embodiments, and will not be repeated here.

[0071] This application also provides a computer program product containing instructions, including a computer program / instructions that, when executed by a processor, implement the method as described above.

[0072] The functions and effects achieved in this embodiment can be explained by referring to other embodiments, and will not be repeated here.

[0073] It is understood that the specific examples in this document are only intended to help those skilled in the art better understand the embodiments of this application, and are not intended to limit the scope of the invention.

[0074] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0075] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and the implementation methods in this application are not limited in this respect.

[0076] Unless otherwise stated, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0077] It is understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0078] It is understood that the memory in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Specifically, non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0079] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0080] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the aforementioned method implementations, and will not be repeated here.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0083] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0084] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A streaming computing method, characterized in that, include: The high-timeliness data items output by the stream processing layer are written into a first sub-table; wherein the stream processing layer is used to perform streaming computation on the data stream to obtain the high-timeliness data items; wherein the first sub-table and the second sub-table belong to the same main table, and the first sub-table and the second sub-table respectively include the data fields of the main table, and the data fields of the main table include a validity field indicating data validity; the high-timeliness data items include field values ​​corresponding to the data fields of the first sub-table, and the field value of the validity field of the high-timeliness data items is a first value indicating a valid state; The low-time-sensitivity data items output by the batch processing layer are written into the second sub-table, and the validity field of the high-time-sensitivity data item corresponding to the low-time-sensitivity data item is set to a second value indicating an invalid state; wherein, the low-time-sensitivity data item includes field values ​​corresponding to the data fields of the second sub-table, and the validity field of the low-time-sensitivity data item is set to a first value; Upon receiving a query request, the system responds to the query request based on high-time-sensitivity data items in the first sub-table whose validity field has a first value and / or low-time-sensitivity data items in the second sub-table whose validity field has a first value.

2. The method according to claim 1, characterized in that, The main table includes an identifier field; in the corresponding high-time-sensitivity data items and low-time-sensitivity data items, the field value of the identifier field is the same.

3. The method according to claim 2, characterized in that, The query request corresponding to the request to query source data is a row-stored table; wherein, the query request to query source data is used to request to obtain at least some field values ​​of high-timeliness data items in the first sub-table and to request to obtain at least some field values ​​of low-timeliness data items in the second sub-table. The main table is a columnar table, corresponding to the query request for data analysis. The query request for data analysis is used to request the acquisition of analysis data. Specifically, the query request for data analysis is used to request data analysis processing based on at least some field values ​​of high-timeliness data items in the first sub-table and at least some field values ​​of low-timeliness data items in the second sub-table to obtain analysis data.

4. The method according to claim 1, characterized in that, The main table includes a generation date field; the validity fields of the first and second sub-tables are all set with bitmap indexes, and the generation date fields of the first and second sub-tables are respectively set with clustered indexes; wherein, the field value of the generation date field is used to represent the generation date of the initial data corresponding to the high-timeliness data item or the low-timeliness data item.

5. The method according to claim 1, characterized in that, The method further includes: If the validity field of a high-time-sensitivity data item is modified to a second value, the high-time-sensitivity data item is deleted from the first sub-table after a specified interval.

6. The method according to claim 2, characterized in that, The steps of writing high-time data items output from the stream processing layer into the first sub-table include: The system receives a first group of field values ​​output by the stream processing layer based on a first stream of data, and a second group of field values ​​output by the stream processing layer based on a second stream of data; the first group of field values ​​and the second group of field values ​​each include at least one field value, and the data fields corresponding to the first group of field values ​​and the data fields corresponding to the second group of field values ​​are not the same. Write the field values ​​of the identifier field corresponding to the first field value group and the second field value group into the first sub-table; wherein, the high-timeliness data item includes the first field group and the second field group corresponding to the field value of the same identifier field.

7. The method according to claim 1, characterized in that, The high-timeliness data and the low-timeliness data are generated based on the same initial data; wherein, the initial data is order data from a specified e-commerce system.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, causes the processor to implement the method as described in any one of claims 1 to 7.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 7.

Images