Real-time intelligent analysis processing method, device and equipment for streaming data and medium

Abstract

The application relates to a real-time intelligent analysis processing method, device and equipment for stream data and a medium. The method comprises the following steps: obtaining a query request for stream data, analyzing the logical structure of the query request, and obtaining a corresponding query syntax tree; reading the metadata of a unified storage cluster based on the query syntax tree, and generating a physical execution path; estimating the cost of each physical execution path based on a preset cost rule, and selecting the physical execution path with the minimum cost as a physical execution plan; executing the physical execution plan, obtaining a target data set from the unified storage cluster, analyzing the target data set through a preset machine learning analysis model, and obtaining a deep analysis result set; and generating a final analysis report corresponding to the stream data based on the deep analysis result set. The method can improve resource utilization and simultaneously consider real-time data and historical data for analysis.

Description

Technical Field

[0001] This invention belongs to the field of real-time information processing, and in particular relates to a method, apparatus, equipment and medium for real-time intelligent analysis and processing of streaming data. Background Technology

[0002] With the rapid development of big data and real-time computing technologies, streaming data real-time intelligent analysis and processing technology has emerged. This technology features low latency, high throughput, and scalability, enabling real-time processing and analysis of continuously generated data streams, thus raising the need for complex real-time and historical data correlation analysis. Traditional technologies typically employ a hybrid architecture that separates batch processing and stream processing. A batch processing layer handles all historical data, a speed layer handles real-time incremental data, and a service layer merges the results to respond to queries; alternatively, a single stream processing layer may be used to process all data, but this still relies on replaying historical data. Current traditional architectures suffer from problems such as difficulty in ensuring data consistency, low resource utilization, and insufficient capabilities for real-time and historical data correlation analysis. Summary of the Invention

[0003] Therefore, it is necessary to provide a method, device, equipment, and medium for real-time intelligent analysis and processing of streaming data that can improve resource utilization and correlate real-time and historical data to address the above-mentioned technical problems.

[0004] Firstly, this application provides a method for real-time intelligent analysis and processing of streaming data, including:

[0005] Obtain query requests for streaming data, analyze the logical structure of the query requests, and obtain the corresponding query syntax tree;

[0006] Based on the query syntax tree, the metadata of the unified storage cluster is read to generate the physical execution path; the unified storage cluster is a data storage architecture for storing streaming data that is updated in real time.

[0007] The cost of each physical execution path is estimated based on preset cost rules, and the physical execution path with the lowest cost is selected as the physical execution plan.

[0008] Execute the physical execution plan, obtain the target data set from the unified storage cluster, analyze the target data set through a pre-set machine learning analysis model, and obtain a deep analysis result set; the machine learning analysis model includes at least one of anomaly detection model, classification prediction model, time series prediction model, and clustering model;

[0009] Based on the deep analysis result set, a final analysis report of the corresponding streaming data is generated.

[0010] Furthermore, the unified storage cluster is built using the following methods:

[0011] Acquire real-time incremental data and historical data of streaming data, and standardize the raw data and historical data to obtain a standardized event stream;

[0012] Based on the timestamps of data in the standardized event stream, the corresponding data is divided into the time intervals of the corresponding timestamps to obtain a time tag set;

[0013] Based on preset hierarchical rules, the access frequency of data is mapped to the corresponding popularity tags to obtain a set of popularity tags;

[0014] A comprehensive tag dataset is generated based on the timeliness tag set and the popularity tag set;

[0015] Based on the comprehensive tag dataset, standardized event streams are written into the corresponding data storage layer to obtain a unified storage cluster.

[0016] Furthermore, based on the comprehensive tag dataset, the standardized event stream is written into the corresponding data storage layer to obtain a unified storage cluster, including:

[0017] The comprehensive label dataset is matched with preset combination rules to obtain the data storage address;

[0018] Based on the data storage address, the data in the standardized event stream is stored in different data storage layers to obtain a hierarchical storage cluster; the data storage layer includes at least one of the following: real-time layer, near-line layer, warm storage layer, and cold storage layer;

[0019] Based on the migration strategy engine, data is migrated to different data storage layers to obtain an adaptive storage cluster;

[0020] Based on an adaptive storage cluster, the distribution, index type and statistical information of data at each layer are automatically extracted to obtain a global metadata directory;

[0021] By integrating the global metadata catalog and the adaptive storage cluster, a unified storage cluster is obtained.

[0022] Furthermore, the raw data and historical data are standardized to obtain a standardized event stream, including:

[0023] The original data and historical data are processed using structured parsing to obtain a set of key-value pairs;

[0024] Using a pre-defined lightweight decision tree model, the set of key-value pairs is transformed to a unified unit to obtain a normalized dataset; and the normalized dataset is then aligned to a time reference to obtain a time-aligned event sequence.

[0025] Based on time-aligned event sequences, the real-time access frequency of data is calculated using the following formula:

[0026]

[0027] Among them, F real (k) represents the real-time access frequency of data k in the current window, T is the sliding window size, and t c e is the current timestamp. i For event i, Entity k For the target data, t i The time it takes for the event to be processed;

[0028] Adding real-time access frequency to a time-aligned event sequence yields a standardized event stream.

[0029] Furthermore, by analyzing the logical structure of the query request, the corresponding query syntax tree is obtained, including:

[0030] Add a predefined type label to each character in the query request to obtain a lexical unit stream; the predefined type labels include at least one of keywords, identifiers, constants, operators, and delimiters;

[0031] Based on a predefined context-free grammar, a specific syntax tree is obtained by processing the lexical unit stream through a shift-reduce algorithm.

[0032] The logical expressions of the concrete syntax tree are reorganized to generate an abstract syntax tree;

[0033] Perform context-dependent checks on the abstract syntax tree to obtain the query syntax tree; the context-dependent checks include one of the following: whether the identifier exists, whether the data types are compatible, and whether the scope is valid.

[0034] Furthermore, based on the query syntax tree, the metadata of the unified storage cluster is read to generate the physical execution path, including:

[0035] Based on the metadata of the unified storage cluster, semantic analysis is performed on the query syntax tree to obtain the logical query plan;

[0036] Based on metadata, all feasible physical operator implementation schemes are enumerated for each logical operator in the logical query plan, generating a set of candidate physical execution paths;

[0037] By adding relevant statistical data to the candidate physical execution path set, the physical execution path is obtained.

[0038] Furthermore, the cost of each physical execution path is estimated based on preset cost rules, including:

[0039] Determine the execution dependencies of physical operators in the physical execution path to obtain a path sequence set;

[0040] Based on the preset cost rules, the cost factors of physical operators in the path sequence set are calculated to obtain the cost path sequence set;

[0041] Based on the cost path sequence set, the cost of the physical execution path is calculated using the following formula:

[0042] Total Cost =CPU Cost +I / O Cost +Communication Cost

[0043] Among them, Total Cost The cost of the physical execution path, CPU Cost To calculate the total time consumption of the operation, I / O Cost Communication is the total time consumed for data reading and writing. Cost This represents the total time consumed for data network transmission.

[0044] Secondly, this application also provides a real-time intelligent analysis and processing device for streaming data, comprising:

[0045] The query module is used to obtain query requests for streaming data, analyze the logical structure of the query requests, and obtain the corresponding query syntax tree;

[0046] The path module is used to read the metadata of the unified storage cluster based on the query syntax tree and generate the physical execution path; the unified storage cluster is a data storage architecture for storing streaming data that is updated in real time.

[0047] The cost module is used to estimate the cost of each physical execution path based on preset cost rules, and select the physical execution path with the lowest cost as the physical execution plan;

[0048] The execution module is used to execute the physical execution plan, obtain the target data set from the unified storage cluster, analyze the target data set through a preset machine learning analysis model, and obtain a deep analysis result set; the machine learning analysis model includes at least one of anomaly detection model, classification prediction model, time series prediction model and clustering model;

[0049] The analysis module is used to generate a final analysis report of the corresponding streaming data based on the deep analysis result set.

[0050] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step of the method provided in the first aspect of this application.

[0051] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any step of the method provided in the first aspect of this application.

[0052] The aforementioned real-time intelligent analysis and processing method, apparatus, device, and medium for streaming data obtains query requests for streaming data and analyzes the logical structure of the query requests to obtain a corresponding query syntax tree. Based on the query syntax tree, it reads the metadata of a unified storage cluster and generates physical execution paths. The unified storage cluster is a data storage architecture for storing streaming data that is updated in real time. Based on preset cost rules, it estimates the cost of each physical execution path and selects the physical execution path with the lowest cost as the physical execution plan. It executes the physical execution plan, obtains the target data set from the unified storage cluster, and analyzes the target data set through a preset machine learning analysis model to obtain a deep analysis result set. The machine learning analysis model includes at least one of anomaly detection model, classification prediction model, time series prediction model, and clustering model. Based on the deep analysis result set, it generates a final analysis report for the corresponding streaming data. This technical means considers both real-time and historical data for data analysis, effectively improving resource utilization. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is a schematic diagram of the flow of a real-time intelligent analysis and processing method for streaming data provided in an embodiment of the present invention;

[0055] Figure 2 This is a schematic diagram of the structure of a real-time intelligent analysis and processing device for streaming data provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] In one embodiment, such as Figure 1As shown, a real-time intelligent analysis and processing method for streaming data is provided. This embodiment illustrates the method's application to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and is implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0058] Step 101: Obtain the query request for the streaming data, analyze the logical structure of the query request, and obtain the corresponding query syntax tree.

[0059] The query request is a user-submitted instruction for analyzing streaming data. As input to the overall process, it triggers subsequent analysis flows and can include logical descriptions such as analysis objectives, filtering conditions, and time ranges. The query syntax tree (MSB) is a tree-like data structure representing the logical structure of the query request. It transforms natural language or structured queries into executable computer logic, and its nodes can include operators, data source identifiers, and conditional expressions. The terminal breaks down the query characters into basic units, adds type labels, constructs a concrete syntax tree reflecting the grammatical structure based on predefined syntax rules, optimizes the syntax tree structure, generates an abstract syntax tree, verifies the existence of identifiers and data type compatibility, ensuring executability, and ultimately generates the query syntax tree.

[0060] Step 102: Based on the query syntax tree, read the metadata of the unified storage cluster and generate the physical execution path; the unified storage cluster is a data storage architecture for storing streaming data that is updated in real time.

[0061] Specifically, the unified storage cluster is a real-time updated data storage architecture that integrates real-time and historical streaming data as a data source, supporting efficient queries. The physical execution path is a set of candidate execution schemes for the query, providing alternative plans for subsequent cost optimization. It can include physical operation steps such as data reading order and compute node allocation. The terminal combines the query syntax tree and metadata to generate a logical query plan, listing feasible physical implementation schemes for each logical operator, and adding statistical information such as data volume and index efficiency to the candidate paths, resulting in a physical execution path containing statistical information.

[0062] Step 103: Estimate the cost of each physical execution path based on the preset cost rules, and select the physical execution path with the lowest cost as the physical execution plan.

[0063] Specifically, the physical execution plan is the final implementation scheme of the optimal execution path, used to guide the data acquisition and computation process. The terminal determines the execution order between physical operators, quantifies the total cost of each path through CPU (Central Processing Unit) cost, I / O (in / out) cost, and communication cost, and selects the path with the minimum total cost as the physical execution plan to avoid resource waste, minimize latency, and meet the low latency requirements of streaming data.

[0064] Step 104: Execute the physical execution plan, obtain the target data set from the unified storage cluster, analyze the target data set through the preset machine learning analysis model, and obtain the deep analysis result set; the machine learning analysis model includes at least one of the following: anomaly detection model, classification prediction model, time series prediction model, and clustering model.

[0065] The target dataset is a subset of data to be analyzed extracted from a unified storage cluster, serving as input to the machine learning model. It includes a combination of real-time and historical data that meet the query criteria. The deep analysis result set is the output of the machine learning analysis model, forming the basis for generating the final report. This set may include anomaly markers, predicted classification labels, and clustering information. The terminal extracts the target dataset from the storage cluster according to the physical execution plan, inputs the data into a preset model, and outputs a deep analysis result set containing intelligent analysis conclusions. For example, the preset model includes an anomaly detection model to identify data anomalies; a classification prediction model to output category labels; a time-series prediction model to predict future trends; and a clustering model to group similar data.

[0066] Step 105: Based on the deep analysis result set, generate the final analysis report of the corresponding streaming data.

[0067] The final analysis report presents the user-facing conclusions of the streaming data analysis, designed to support decision-making. The terminal aggregates key information from the in-depth analysis result set, transforms it into a readable format, and generates the final analysis report for the streaming data.

[0068] This embodiment provides a real-time intelligent analysis and processing method, apparatus, device, and medium for streaming data. It acquires a query request and analyzes its logical structure to obtain a query syntax tree. The query request is an analysis request for streaming data. Based on the query syntax tree, it reads metadata from a unified storage cluster to generate physical execution paths. The unified storage cluster is a real-time updated data storage architecture for storing streaming data. It estimates the cost of each physical execution path based on preset cost rules and selects the physical execution path with the lowest cost as the physical execution plan. It executes the physical execution plan, acquires a target data set from the unified storage cluster, and analyzes the target data set using a preset machine learning analysis model to obtain a deep analysis result set. The machine learning analysis model includes at least one of anomaly detection, classification prediction, time series prediction, and clustering models. Based on the deep analysis result set, it generates a final analysis report for the corresponding streaming data. By using the above techniques, it simultaneously considers real-time streaming data and historical batch data for data analysis, effectively improving resource utilization and enhancing the correlation between data.

[0069] In one embodiment, the unified storage cluster is constructed using the following method:

[0070] Step 201: Obtain real-time incremental data and historical data of streaming data, and standardize the original data and historical data to obtain a standardized event stream.

[0071] Real-time incremental data consists of new data generated in real time within the streaming data stream, representing the current data state and used for real-time analysis. It is a continuous data stream containing timestamps. Historical data is older data already stored within the streaming data stream, used to provide a basis for long-term trend analysis; it is an archived batch dataset. Standardized event streams are cleaned, formatted, and time-aligned standardized data sequences, eliminating the heterogeneity of the original data and providing a unified input for subsequent labeling. The terminal parses the original and historical data into a set of key-value pairs, converts the units using a lightweight decision tree model, unifies the timestamps of all data, and counts the data access frequency within a sliding window to obtain the standardized event stream.

[0072] Step 202: Based on the timestamps of the data in the standardized event stream, divide the corresponding data into the time intervals of the corresponding timestamps to obtain the time tag set.

[0073] Specifically, the timeliness interval is a preset time segmentation rule used to define the data freshness level, including the mapping relationship between time range and storage strategy. The timeliness tag set is a set of tags corresponding to the timeliness interval of data points, used to identify the storage timeliness level of data. The terminal reads the data timestamp from the standardized event stream, compares the current time with the data timestamp, assigns timeliness tags to all data points, and obtains the timeliness tag set.

[0074] Step 203: Based on the preset hierarchical rules, the access frequency of the data is mapped to the corresponding popularity tags to obtain a popularity tag set.

[0075] Specifically, the grading rule is a mapping rule between access frequency and popularity level, serving as a classification standard for defining data access popularity. For example, the correspondence between threshold ranges and tags could be that more than 100 accesses per minute constitutes high popularity. The popularity tag set is a set of tags corresponding to the access frequency of data points, used to identify the access popularity level of the data. The terminal extracts access frequencies from the standardized event stream, matches the frequency values ​​with preset rules, and assigns popularity tags to all data points. This results in a popularity tag set that quantifies the data's popularity.

[0076] Step 204: Generate a comprehensive tag dataset based on the timeliness tag set and the popularity tag set.

[0077] The comprehensive tag dataset is a collection of metadata that combines time-sensitive tags and popularity tags, used to guide the allocation of data storage locations. The terminal merges the time-sensitive tags and popularity tags for the same data point to generate a dataset containing comprehensive tags for all data points.

[0078] Step 205: Based on the comprehensive tag dataset, write the standardized event stream into the corresponding data storage layer to obtain a unified storage cluster.

[0079] Specifically, the data storage layer refers to physical storage media categorized by performance and / or cost, used for hierarchical storage of data with different value densities. For example, the data storage layer may include a real-time layer, a near-line layer, a warm storage layer, and a cold storage layer. The unified storage cluster is a unified architecture integrating hierarchical storage and global metadata, serving as the central storage system for streaming data. Terminals match comprehensive tags with preset rules, write data to the corresponding storage layer according to address, and move data between storage layers based on a migration strategy engine. It automatically collects the distribution, index type, and statistical information of data at each layer, constructing a unified storage cluster for hierarchical intelligent storage.

[0080] This embodiment eliminates obstacles to mixed analysis by standardizing and processing real-time and historical data in a unified manner. Popularity tags and time-sensitive tags are automatically stored in a hierarchical manner, and the metadata directory provides decision-making basis for the query optimizer, effectively improving the accuracy and real-time performance of streaming data processing.

[0081] In one embodiment, based on a comprehensive tag dataset, a standardized event stream is written to the corresponding data storage layer to obtain a unified storage cluster, including:

[0082] Step 301: Match the comprehensive label dataset with the preset combination rules to obtain the data storage address.

[0083] The combination rule is a preset tag-to-storage layer mapping strategy, which is the core logic for determining the data storage location. The data storage address is the physical location identifier of the data in the hierarchical storage architecture, used to guide data write operations, and is in the form of a storage layer identifier. The terminal reads the comprehensive tag dataset, matches the tags with the combination rules one by one, and assigns a corresponding data storage address to each data point.

[0084] Step 302: Based on the data storage address, the data in the standardized event stream is stored in different data storage layers to obtain a hierarchical storage cluster; the data storage layer includes at least one of the following: real-time layer, near-line layer, warm storage layer, and cold storage layer.

[0085] Specifically, the data storage layer refers to the physical storage media types categorized by performance and / or cost, used for tiered storage of data with different value densities. For example, the data storage layer may include a real-time layer with millisecond-level access via in-memory database Redis (Remote Dictionary Server); a near-line layer with second-level access via SSD (Solid State Drive) storage like Apache Kafka; a warm storage layer with minute-level access via HDD (Hard Disk Drive) distributed file system; and a cold storage layer with hourly-level access via tape library and / or cloud cold storage. The tiered storage cluster is a primary storage architecture that allocates data by address, used to achieve physically isolated data storage. Terminals read the data storage address, write data from the standardized event stream to the corresponding storage layer, record data location information, forming a preliminary tiered storage cluster, enabling high-speed access to high-frequency data and low-cost storage of low-frequency data.

[0086] Step 303: Based on the migration strategy engine, the data is migrated to different data storage layers to obtain an adaptive storage cluster.

[0087] Specifically, the migration strategy engine is a rule-based decision-making module that monitors data status and triggers migrations to dynamically optimize data storage locations. The adaptive storage cluster is an intelligent storage architecture that supports dynamic adjustment of data locations to respond to changes in data access patterns. The terminal tracks data access frequency, storage load, and other metrics in real time. When the data status meets migration conditions, the engine triggers a migration action, moving the data to a new storage layer and generating an adaptive storage cluster with adaptive capabilities.

[0088] Step 304: Based on the adaptive storage cluster, automatically extract the distribution, index type and statistical information of data at each layer to obtain the global metadata directory.

[0089] The global metadata directory is a central directory describing the distribution and characteristics of data within the storage cluster, providing decision-making support for the query optimizer. For example, the global metadata directory may include data distribution (storage location, partition information); index type (tree index, hash index, etc.); and statistical information (data volume, access frequency, storage cost, etc.). The terminal scans data at each layer of the adaptive storage cluster, automatically collecting these three types of metadata—distribution information, index information, and statistical information—to generate a global metadata directory that supports a global data view.

[0090] Step 305: Integrate the global metadata directory and the adaptive storage cluster to obtain a unified storage cluster.

[0091] The unified storage cluster is an integrated architecture that combines physical storage and metadata management. It serves as the central storage system for streaming data and includes a global metadata directory and an adaptive storage cluster. The terminal binds the adaptive storage cluster to the global metadata directory, establishes an indexing mechanism, and generates the unified storage cluster.

[0092] This embodiment achieves automatic data stratification and dynamic adjustment through tag matching and migration engine. The global metadata catalog guides the query optimizer to quickly locate data and improve query efficiency, adapting to the dynamic characteristics of streaming data, responding to changes in data popularity, and significantly improving the processing speed of streaming data analysis.

[0093] In one embodiment, the raw data and historical data are standardized to obtain a standardized event stream, including:

[0094] Step 401: Perform structured parsing processing on the original data and historical data to obtain a set of key-value pairs.

[0095] The key-value pair set is a structured collection of data, where each data point is stored in key-value format, providing a standard input format for subsequent unit conversions. The terminal extracts field names and values ​​from real-time streaming data, maps historical batch data to key-value pairs by column names, fills in default values ​​for missing fields, discards data with incorrect formats, and obtains the key-value pair set.

[0096] Step 402: Using a preset lightweight decision tree model, the key-value pair set is converted to a unified unit to obtain a normalized dataset; and the normalized dataset is aligned with a time reference to obtain a time-aligned event sequence.

[0097] Specifically, the lightweight decision tree model is a simple, non-trained classification model composed of predefined rules for automated unit conversion. The normalized dataset is a unified set of data used to eliminate the impact of unit differences on the analysis. The time-aligned event sequence is an ordered sequence of all data timestamps aligned to the same baseline, ensuring comparability in the time dimension. For example, the terminal application uses the decision tree to traverse the key-value pair set, matches conversion rules based on field names, performs unit normalization, unifies all timestamps to UTC (Coordinated Universal Time) format, arranges the data in ascending order of timestamps, and generates a time-aligned event sequence with unified units and ordered time.

[0098] Step 403: Based on the time-aligned event sequence, calculate the real-time access frequency of the data using the following formula:

[0099]

[0100] Among them, F reak (k) represents the real-time access frequency of data k in the current window, T is the sliding window size, and t c e is the current timestamp. i For event i, Entity k For the target data, t i This refers to the time it takes for the event to be processed.

[0101] Specifically, an event can be a data read operation, and the target data can be a specific sensor entity. The terminal determines a time window, iterates through all events within the window, and if the event belongs to the target data entity, the count is incremented by 1. The average access frequency is calculated to obtain the real-time access frequency value for each data entity.

[0102] Step 404: Add real-time access frequency to the time-aligned event sequence to obtain a standardized event stream.

[0103] Specifically, the standardized event stream is the final processed data stream, which may contain the original value, timestamp, unit, and access frequency. The terminal adds the access frequency as a new field to the data points, sorts all the enhanced data points by timestamp, forms a continuous event stream, and generates a standardized event stream containing complete metadata.

[0104] This embodiment solves the statistical distortion problem caused by mixed sources of traditional streaming data by eliminating unit differences and time zone differences, calculates access frequency to quantify hot and cold data, supports downstream optimization, and thus improves the targeting and real-time performance of data processing.

[0105] In one embodiment, the logical structure of the query request is analyzed to obtain the corresponding query syntax tree, including:

[0106] Step 501: Add a predefined type label to each character of the query request to obtain a lexical unit stream; the predefined type label includes at least one of keywords, identifiers, constants, operators, and delimiters.

[0107] The lexical unit stream is a structured set of tags categorized from the query character sequence, used to provide standard input for syntax parsing. The terminal reads the query request character by character, matches the query request by type, adds a type tag to each character and / or phrase, and generates this lexical unit stream. For example, type tags can be keywords (predefined commands); identifiers (data object names); constants (fixed values); operators (computation symbols); and delimiters (syntax boundary symbols).

[0108] Step 502: Based on the predefined context-free grammar, the lexical unit stream is processed by the shift-reduce algorithm to obtain the specific syntax tree.

[0109] Specifically, a context-free grammar is a predefined set of grammar rules used to describe the grammatical structure of a query language. The shift-reduce algorithm, a syntax parsing algorithm in compiler theory, uses a stack to process lexical units and is the underlying technique for building a syntax tree. It includes shift and reduce operations: shifting involves pushing lexical units onto the stack; reduction replaces the top symbol of the stack with a grammar rule as a syntax node. The concrete syntax tree (PTB) is a tree-like representation that strictly reflects the grammatical structure of the query surface and is used to verify grammatical correctness. The terminal prepares an empty stack to store the parsing state, reads the lexical unit stream, performs shift and / or reduce operations according to the context-free grammar rules, and outputs a PTB containing complete grammatical details.

[0110] Step 503: Reorganize the logical expressions of the concrete syntax tree to generate the abstract syntax tree.

[0111] Specifically, an abstract syntax tree (AST) is a simplified tree structure that strips away surface syntactic details while retaining the core logic, used to optimize query execution efficiency. The terminal accesses each syntax node, removes delimiters, merges nested structures, refactors logical expressions, and generates an AST that retains only the business logic.

[0112] Step 504: Perform context-dependent checks on the abstract syntax tree to obtain the query syntax tree; the context-dependent checks include one of the following: whether the identifier exists, whether the data types are compatible, and whether the scope is valid.

[0113] Specifically, context-sensitive validation is a semantic validation rule that relies on metadata to ensure that queries are executable in the business scenario. For example, validation types may include identifier existence (to verify the existence of a field or table name), data type compatibility (to check operational compatibility), and scope validity (to confirm that variables are within their scope). The query syntax tree is an executable logical structure validated across the entire cluster. The terminal reads the metadata of the unified storage cluster and performs context-sensitive validation node by node. When an invalid node is detected, an error is triggered, and a semantically valid query syntax tree is generated.

[0114] This embodiment intercepts the risk of query semantic distortion by parsing query requests. The clean query syntax tree significantly simplifies the processing of subsequent steps, enables correlation analysis of real-time and historical data, provides good input for subsequent steps, and effectively improves the correlation of streaming data processing.

[0115] In one embodiment, based on the query syntax tree, the metadata of the unified storage cluster is read to generate a physical execution path, including:

[0116] Step 601: Based on the metadata of the unified storage cluster, perform semantic analysis on the query syntax tree to obtain a logical query plan.

[0117] Specifically, a logical query plan is a high-level, storage-independent sequence of query execution steps used to describe the abstract computational logic of a query. The terminal reads metadata from the unified storage cluster, maps query syntax tree nodes to storage objects, generates logical operation steps according to query semantics, and obtains the logical query plan, ensuring that the same logical plan can adapt to different storage architectures.

[0118] Step 602: Based on metadata, enumerate all feasible physical operator implementation schemes for each logical operator in the logical query plan, and generate a set of candidate physical execution paths.

[0119] The physical operator implementation scheme refers to the specific execution method of the logical operator on the actual hardware, transforming abstract computation into a runnable task. The candidate physical execution path set is the set of all feasible physical implementation schemes for the same logical plan, providing a selection pool for cost optimization. The terminal enumerates feasible schemes based on metadata, generates multiple physical implementation schemes for each logical operator, and combines them to form a complete path, adapting to different resource scenarios and avoiding performance limitations caused by a single implementation scheme.

[0120] Step 603: Append relevant statistical data to the candidate physical execution path set to obtain the physical execution path.

[0121] Among them, the physical execution path is a candidate path bound to statistical data, which can be directly used for cost estimation. The terminal obtains data distribution, index efficiency, and historical statistical data from the unified storage cluster, adds relevant statistics to each path in the candidate set, and generates a physical execution path with quantifiable cost.

[0122] This embodiment solves the problem of rigid static planning in traditional systems by dynamically generating paths through metadata, predicting resource consumption by statistical tags, reducing the error rate of cluster resource allocation, and improving resource utilization efficiency.

[0123] In one embodiment, the cost of each physical execution path is estimated based on a preset cost rule, including:

[0124] Step 701: Determine the execution dependencies of physical operators in the physical execution path to obtain a path sequence set.

[0125] Execution dependencies are the sequential execution constraints between physical operators used to define the topological order of operation sequences. The path sequence set is an ordered set of physical operators that satisfy the dependencies, used to describe a complete executable chain of operations. The terminal identifies the data input / output relationships between operators, marks operations that can be executed in parallel, and generates a set of feasible path sequences that satisfy hardware constraints, preventing execution conflicts caused by incomplete data due to missing dependencies.

[0126] Step 702: Based on the preset cost rules, calculate the cost factors of the physical operators in the path sequence set to obtain the cost path sequence set.

[0127] The cost rule is a predefined cost calculation strategy used to quantify the resource consumption of operators. The cost factor is a detailed cost indicator for a single physical operator, serving as the basic unit for calculating the total cost. The cost path sequence set is a collection of path sequences bound to cost factors, used to support the calculation of the total cost. The terminal calculates the three main costs for each operator according to the cost rule: computation time, disk read / write time, and cross-boundary data transfer time. It then binds cost factors to each operator in the path sequence set, resulting in the cost path sequence set, which provides data support for selecting the minimum cost.

[0128] Step 703: Based on the cost path sequence set, calculate the cost of the physical execution path using the following formula:

[0129] Total Cost =CPU Cost +I / O Cost +Communication Cost

[0130] Among them, Total Cost The cost of the physical execution path, CPU CostTo calculate the total time consumption of the operation, I / O Cost Communication is the total time consumed for data reading and writing. Cost This represents the total time consumed for data network transmission.

[0131] Specifically, the cost of a physical execution path is the sum of the cost factors of all operators in the path sequence, used to compare the overall performance of different paths. The terminal traverses the sequence set and calculates the total cost of each path in the cost path sequence set using a formula.

[0132] This embodiment addresses the limitations of traditional optimizers' single-metric approach through three-dimensional quantization, achieving dynamic adaptability based on cost rules derived from real-time metadata, and improving resource utilization in streaming data analysis and processing.

[0133] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0134] Based on the same inventive concept, this application also provides a streaming data real-time intelligent analysis and processing device for implementing the above-mentioned streaming data real-time intelligent analysis and processing method. The solution provided by this device is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more streaming data real-time intelligent analysis and processing device embodiments provided below can be found in the limitations of the streaming data real-time intelligent analysis and processing method described above, and will not be repeated here.

[0135] In one exemplary embodiment, such as Figure 2 As shown, a real-time intelligent analysis and processing device 800 for streaming data is provided, comprising:

[0136] The query module 801 is used to obtain query requests for streaming data, analyze the logical structure of the query requests, and obtain the corresponding query syntax tree.

[0137] The path module 802 is used to read the metadata of the unified storage cluster based on the query syntax tree and generate the physical execution path; the unified storage cluster is a data storage architecture for storing streaming data that is updated in real time.

[0138] The cost module 803 is used to estimate the cost of each physical execution path based on preset cost rules, and select the physical execution path with the lowest cost as the physical execution plan;

[0139] The execution module 804 is used to execute the physical execution plan, obtain the target data set from the unified storage cluster, analyze the target data set through a preset machine learning analysis model, and obtain a deep analysis result set; the machine learning analysis model includes at least one of anomaly detection model, classification prediction model, time series prediction model and clustering model;

[0140] Analysis module 805 is used to generate a final analysis report of the corresponding streaming data based on the deep analysis result set.

[0141] Furthermore, the path module 802 is also used for:

[0142] Acquire real-time incremental data and historical data of streaming data, and standardize the raw data and historical data to obtain a standardized event stream;

[0143] Based on the timestamps of data in the standardized event stream, the corresponding data is divided into the time intervals of the corresponding timestamps to obtain a time tag set;

[0144] Based on preset hierarchical rules, the access frequency of data is mapped to the corresponding popularity tags to obtain a set of popularity tags;

[0145] A comprehensive tag dataset is generated based on the timeliness tag set and the popularity tag set;

[0146] Based on the comprehensive tag dataset, standardized event streams are written into the corresponding data storage layer to obtain a unified storage cluster.

[0147] Furthermore, the path module 802 is also used for:

[0148] The comprehensive label dataset is matched with preset combination rules to obtain the data storage address;

[0149] Based on the data storage address, the data in the standardized event stream is stored in different data storage layers to obtain a hierarchical storage cluster; the data storage layer includes at least one of the following: real-time layer, near-line layer, warm storage layer, and cold storage layer;

[0150] Based on the migration strategy engine, data is migrated to different data storage layers to obtain an adaptive storage cluster;

[0151] Based on an adaptive storage cluster, the distribution, index type and statistical information of data at each layer are automatically extracted to obtain a global metadata directory;

[0152] By integrating the global metadata catalog and the adaptive storage cluster, a unified storage cluster is obtained.

[0153] Furthermore, the path module 802 is also used for:

[0154] The original data and historical data are processed using structured parsing to obtain a set of key-value pairs;

[0155] Using a pre-defined lightweight decision tree model, the set of key-value pairs is transformed to a unified unit to obtain a normalized dataset; and the normalized dataset is then aligned to a time reference to obtain a time-aligned event sequence.

[0156] Based on time-aligned event sequences, the real-time access frequency of data is calculated using the following formula:

[0157]

[0158] Among them, F real (k) represents the real-time access frequency of data k in the current window, T is the sliding window size, and t c e is the current timestamp. i For event i, Emtity k For the target data, t i The time it takes for the event to be processed;

[0159] Adding real-time access frequency to a time-aligned event sequence yields a standardized event stream.

[0160] Furthermore, query module 801 is also used for:

[0161] Add a predefined type label to each character in the query request to obtain a lexical unit stream; the predefined type labels include at least one of keywords, identifiers, constants, operators, and delimiters;

[0162] Based on a predefined context-free grammar, a specific syntax tree is obtained by processing the lexical unit stream through a shift-reduce algorithm.

[0163] The logical expressions of the concrete syntax tree are reorganized to generate an abstract syntax tree;

[0164] Perform context-dependent checks on the abstract syntax tree to obtain the query syntax tree; the context-dependent checks include one of the following: whether the identifier exists, whether the data types are compatible, and whether the scope is valid.

[0165] Furthermore, the path module 802 is also used for:

[0166] Based on the metadata of the unified storage cluster, semantic analysis is performed on the query syntax tree to obtain the logical query plan;

[0167] Based on metadata, all feasible physical operator implementation schemes are enumerated for each logical operator in the logical query plan, generating a set of candidate physical execution paths;

[0168] By adding relevant statistical data to the candidate physical execution path set, the physical execution path is obtained.

[0169] Furthermore, the cost module is also used for:

[0170] Determine the execution dependencies of physical operators in the physical execution path to obtain a path sequence set;

[0171] Based on the preset cost rules, the cost factors of physical operators in the path sequence set are calculated to obtain the cost path sequence set;

[0172] Based on the cost path sequence set, the cost of the physical execution path is calculated using the following formula:

[0173] Total Cost =CPU Cost +I / O Cost +Communication Cost

[0174] Among them, Total Cost The cost of the physical execution path, CPU Cost To calculate the total time consumption of the operation, I / O Cost Communication is the total time consumed for data reading and writing. Cost This represents the total time consumed for data network transmission.

[0175] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the streaming data real-time intelligent analysis and processing method as described above.

[0176] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0177] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0178] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for real-time intelligent analysis and processing of streaming data, characterized in that, The method includes: Obtain query requests for streaming data, analyze the logical structure of the query requests, and obtain the corresponding query syntax tree; Based on the query syntax tree, the metadata of the unified storage cluster is read to generate a physical execution path; the unified storage cluster is a data storage architecture that is updated in real time to store the streaming data. The cost of each physical execution path is estimated based on preset cost rules, and the physical execution path with the lowest cost is selected as the physical execution plan. The physical execution plan is executed to obtain a target data set from the unified storage cluster. The target data set is then analyzed using a preset machine learning analysis model to obtain a deep analysis result set. The machine learning analysis model includes at least one of anomaly detection model, classification prediction model, time series prediction model, and clustering model. Based on the deep analysis result set, a final analysis report corresponding to the streaming data is generated; The step of analyzing the logical structure of the query request to obtain the corresponding query syntax tree includes: Each character in the query request is added with a predefined type label to obtain a lexical unit stream; the predefined type label includes at least one of keywords, identifiers, constants, operators, and delimiters; Based on a predefined context-free grammar, the lexical unit stream is processed by a shift-reduce algorithm to obtain a specific syntax tree; Reorganize the logical expressions of the specific syntax tree to generate an abstract syntax tree; The abstract syntax tree is subjected to context-dependent checks to obtain the query syntax tree; the context-dependent checks include one of the following: whether the identifier exists, whether the data types are compatible, and whether the scope is valid. The step of reading metadata from the unified storage cluster and generating a physical execution path based on the query syntax tree includes: Based on the metadata of the unified storage cluster, semantic analysis is performed on the query syntax tree to obtain a logical query plan; Based on the metadata, all feasible physical operator implementation schemes are enumerated for each logical operator in the logical query plan to generate a set of candidate physical execution paths; The physical execution paths are obtained by appending relevant statistical data to the candidate physical execution path set. The step of estimating the cost of each physical execution path based on preset cost rules includes: Determine the execution dependencies of the physical operators in the physical execution path to obtain a path sequence set; Based on the preset cost rules, the cost factors of the physical operators in the path sequence set are calculated to obtain the cost path sequence set; Based on the cost path sequence set, the cost of the physical execution path is calculated using the following formula: ； in, The cost of the physical execution path, To calculate the total time consumption of the operation, This represents the total time consumed for data reading and writing. This represents the total time consumed for data network transmission.

2. The method according to claim 1, characterized in that, The unified storage cluster is constructed using the following methods: Acquire real-time incremental data and historical data of the streaming data, and standardize the real-time incremental data and the historical data to obtain a standardized event stream; Based on the timestamps of the data in the standardized event stream, the corresponding data is divided into time intervals corresponding to the timestamps to obtain a time tag set; Based on preset hierarchical rules, the access frequency of the data is mapped to the corresponding popularity tags to obtain a popularity tag set; Based on the timeliness tag set and the popularity tag set, a comprehensive tag dataset is generated; Based on the comprehensive tag dataset, the standardized event stream is written into the corresponding data storage layer to obtain a unified storage cluster.

3. The method according to claim 2, characterized in that, The step of writing the standardized event stream into the corresponding data storage layer based on the comprehensive tag dataset to obtain a unified storage cluster includes: The comprehensive tag dataset is matched with preset combination rules to obtain the data storage address; Based on the data storage address, the data in the standardized event stream is stored in different data storage layers to obtain a hierarchical storage cluster; the data storage layer includes at least one of a real-time layer, a near-line layer, a warm storage layer, and a cold storage layer; Based on the migration strategy engine, the data is migrated to different data storage layers to obtain an adaptive storage cluster; Based on the adaptive storage cluster, the distribution, index type and statistical information of the data in each layer are automatically extracted to obtain a global metadata directory; The unified storage cluster is obtained by integrating the global metadata directory and the adaptive storage cluster.

4. The method according to claim 2, characterized in that, The step of standardizing the real-time incremental data and the historical data to obtain a standardized event stream includes: The real-time incremental data and the historical data are subjected to structured parsing processing to obtain a set of key-value pairs; Using a pre-defined lightweight decision tree model, the set of key-value pairs is transformed to a unified unit to obtain a normalized dataset; and the normalized dataset is then aligned to a time reference to obtain a time-aligned event sequence. Based on the time-aligned event sequence, the real-time access frequency of the data is calculated using the following formula: ； in, Let T be the real-time access frequency of data k in the current window, and T be the size of the sliding window. This is the current timestamp. For event i, For target data, The time it takes for the event to be processed; The real-time access frequency is added to the time-aligned event sequence to obtain the standardized event stream.

5. A real-time intelligent analysis and processing device for streaming data, characterized in that, The device includes: The query module is used to obtain query requests for streaming data, analyze the logical structure of the query requests, and obtain the corresponding query syntax tree; The path module is used to read the metadata of the unified storage cluster based on the query syntax tree and generate a physical execution path; the unified storage cluster is a data storage architecture that is updated in real time to store the streaming data. The cost module is used to estimate the cost of each physical execution path based on preset cost rules, and select the physical execution path with the lowest cost as the physical execution plan; An execution module is used to execute the physical execution plan, obtain a target data set from the unified storage cluster, analyze the target data set through a preset machine learning analysis model, and obtain a deep analysis result set; the machine learning analysis model includes at least one of an anomaly detection model, a classification prediction model, a time series prediction model, and a clustering model; The analysis module is used to generate a final analysis report corresponding to the streaming data based on the deep analysis result set. The query module is further configured to: Each character in the query request is added with a predefined type label to obtain a lexical unit stream; the predefined type label includes at least one of keywords, identifiers, constants, operators, and delimiters; Based on a predefined context-free grammar, the lexical unit stream is processed by a shift-reduce algorithm to obtain a specific syntax tree; Reorganize the logical expressions of the specific syntax tree to generate an abstract syntax tree; The abstract syntax tree is subjected to context-dependent checks to obtain the query syntax tree; the context-dependent checks include one of the following: whether the identifier exists, whether the data types are compatible, and whether the scope is valid. The path module is further configured to: Based on the metadata of the unified storage cluster, semantic analysis is performed on the query syntax tree to obtain a logical query plan; Based on the metadata, all feasible physical operator implementation schemes are enumerated for each logical operator in the logical query plan to generate a set of candidate physical execution paths; The physical execution paths are obtained by appending relevant statistical data to the candidate physical execution path set. The cost module is further configured to: Determine the execution dependencies of the physical operators in the physical execution path to obtain a path sequence set; Based on the preset cost rules, the cost factors of the physical operators in the path sequence set are calculated to obtain the cost path sequence set; Based on the cost path sequence set, the cost of the physical execution path is calculated using the following formula: ； in, The cost of the physical execution path, To calculate the total time consumption of the operation, This represents the total time consumed for data reading and writing. This represents the total time consumed for data network transmission.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images