Real-time data processing method, system and device, storage medium and program product

Abstract

The invention discloses a real-time data processing method, system and device, a storage medium and a program product, and relates to the technical field of real-time data processing, and the method comprises the steps: obtaining and standardizing real-time data; configuration management parameters corresponding to the standard real-time data are configured; standard real-time data can be processed by the DAG and configuration management parameters; and during data operation, abnormal data monitoring and early warning are implemented based on configuration management parameters in combination with a Kafka platform and dimension table data. According to the method, diversified data formats are standardized through a unified interface, and the influence of upstream change on the downstream is reduced. The DAG supports efficient parallel computing, and large-scale data processing is accelerated. The Kafka and dimension table data are matched to manage parameters, so that timely data anomaly detection and early warning are realized, the system stability and safety are improved, the data consistency and processing efficiency are ensured, and the overall data governance capability is enhanced.

Description

Technical Field

[0001] The present application relates to the field of real-time data processing technology, and in particular to a real-time data processing method, system, device, storage medium and program product. Background Art

[0002] In the field of real-time data processing, existing solutions face some significant challenges. First, in order to achieve modularity of the system, it is usually necessary to implement independent application containers for each individual real-time data stream. Although this strategy helps decoupling, it greatly hinders the efficiency of system optimization and transformation. Secondly, relying on Redis (database) as a storage solution for background data and temporary buffering, its highly centralized nature means that any failure of Redis may cause the interruption of the entire service, posing a serious threat to the stability and reliability of the service. In addition, when upstream data changes, data consistency maintenance involving multiple applications becomes an expensive task, increasing operating costs. Finally, the lack of a centralized operation management system makes it difficult to efficiently perform key operations such as application deployment, expansion, and monitoring, further highlighting the problem of management complexity. Summary of the invention

[0003] The main purpose of this application is to provide a real-time data processing method, system, device and storage medium, aiming to solve the technical problem of how to build a real-time data processing ecosystem to improve the efficiency, reliability and management efficiency of real-time data processing while reducing operating costs and risks.

[0004] To achieve the above objectives, the present application proposes a real-time data processing method, the method comprising:

[0005] Acquire real-time data, and standardize the format of the real-time data to acquire standard real-time data;

[0006] Configuring configuration management parameters corresponding to the standard real-time data;

[0007] Processing the standard real-time data through the directed acyclic graph DAG and the configuration management parameters to obtain the processed standard real-time data;

[0008] When the processed standard real-time data is running, based on the configuration management parameters and in combination with the pre-built Kafka and dimension table data, data anomaly monitoring and early warning are performed on the processed standard real-time data.

[0009] In one embodiment, the step of standardizing the format of the real-time data to obtain standard real-time data includes:

[0010] Build a unified data interface;

[0011] Based on the unified data interface, the real-time data is input into a data converter, and the format of the real-time data is standardized by the data converter to obtain standard real-time data.

[0012] In one embodiment, the step of configuring the configuration management parameters corresponding to the standard real-time data includes:

[0013] According to the data experiment function, configure the data verification strategy corresponding to the processed standard real-time data;

[0014] Configuring corresponding monitoring strategies and early warning mechanisms for the processed standard real-time data to monitor and warn of abnormal situations of the processed standard real-time data;

[0015] Based on the data verification strategy, the monitoring strategy and the early warning mechanism, configuration management parameters corresponding to the processed standard real-time data are obtained.

[0016] In one embodiment, the step of processing the standard real-time data through the directed acyclic graph DAG and the configuration management parameters to obtain the processed standard real-time data includes:

[0017] Testing and verifying the standard real-time data through the data verification strategy;

[0018] Convert the tested and verified standard real-time data into individual task units according to DAG;

[0019] Logically process each of the task units to obtain processed standard real-time data.

[0020] In one embodiment, when the processed standard real-time data is running, based on the configuration management parameters, combined with the pre-built Kafka and dimension table data, data anomaly monitoring and early warning of the processed standard real-time data includes:

[0021] When the processed standard real-time data is running, the processed standard real-time data is monitored and analyzed using the pre-built Kafka and dimension table data based on the monitoring strategy;

[0022] Based on the early warning mechanism, an abnormal early warning is performed on the abnormal standard real-time data monitored in the processed standard real-time data.

[0023] In one embodiment, the step of performing abnormal warning on abnormal standard real-time data monitored in the processed standard real-time data based on the warning strategy includes:

[0024] When abnormal standard real-time data is detected in the processed standard real-time data, identifying application behavior of the abnormal standard real-time data;

[0025] Based on the application behavior of the abnormal standard real-time data, obtaining key indicators corresponding to the abnormal standard real-time data;

[0026] Verify whether the key indicators corresponding to the abnormal standard real-time data deviate from the preset warning threshold;

[0027] If so, an alarm signal is triggered, and in response to the alarm signal, the abnormal standard real-time data is displayed to the user.

[0028] In addition, to achieve the above purpose, the present application also proposes a real-time data processing system, the real-time data processing system comprising:

[0029] A conversion module, used to obtain real-time data and standardize the format of the real-time data to obtain standard real-time data;

[0030] A configuration module, used to configure configuration management parameters corresponding to the standard real-time data;

[0031] A processing module, used to process the standard real-time data through the directed acyclic graph DAG and the configuration management parameters to obtain the processed standard real-time data;

[0032] The early warning module is used to monitor and warn the processed standard real-time data for data anomalies based on the configuration management parameters and in combination with pre-built Kafka and dimension table data when the processed standard real-time data is running.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a real-time data processing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the real-time data processing method as described above.

[0034] In addition, to achieve the above objectives, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the real-time data processing method described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the real-time data processing method described above are implemented.

[0036] The embodiment of the present application provides a real-time data processing method, which includes: acquiring and standardizing real-time data; configuring configuration management parameters corresponding to standard real-time data; processing standard real-time data using DAG and configuration management parameters; when data is running, based on configuration management parameters, combined with Kafka platform and dimension table data, implementing abnormal data monitoring and early warning. This method standardizes diverse data formats through a unified interface, reducing the impact of upstream changes on downstream. DAG supports efficient parallel computing and accelerates large-scale data processing. Kafka and dimension table data cooperate with management parameters to achieve timely data anomaly detection and early warning, improve system stability and security, and ensure data consistency and processing efficiency, and enhance overall data governance capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 A flowchart of the first embodiment of the real-time data processing method of the present application is provided;

[0040] Figure 2 A schematic diagram of the application integration configuration provided in Example 1 of the present application;

[0041] Figure 3 A schematic diagram of a data processing flow based on a GAD map provided in Example 1 of the present application;

[0042] Figure 4 A code schematic diagram of the data processing process based on the GAD graph provided in Example 1 of the present application;

[0043] Figure 5 A flowchart of the second embodiment of the real-time data processing method of the present application is provided;

[0044] Figure 6 A schematic diagram of the early warning and alarm process provided in Example 2 of the present application;

[0045] Figure 7 A schematic diagram of the process of blood relationship data management provided in Example 2 of the present application;

[0046] Figure 8A system architecture diagram of the real-time data processing method provided in Example 1 of the present application;

[0047] Fig. 9 A schematic diagram of the microservice architecture design provided in Example 1 of the present application;

[0048] Fig.10 This is a schematic diagram of the module structure of the real-time data processing system according to an embodiment of the present application;

[0049] Fig.11 Schematic diagram of the device structure of the hardware operating environment involved in the real-time data processing method in the embodiment of the present application.

[0050] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0051] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0052] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0053] The main solutions of the embodiments of the present application are: obtaining real-time data, and standardizing the format of the real-time data to obtain standard real-time data; configuring configuration management parameters corresponding to the standard real-time data; processing the standard real-time data through DAG and configuration management parameters to obtain processed standard real-time data; when the processed standard real-time data is running, based on the configuration management parameters, combined with the pre-built Kafka and dimension table data, data anomaly monitoring and early warning are performed on the processed standard real-time data.

[0054] In order to achieve independent processing of each real-time data stream and system decoupling, the existing technology usually adopts an application isolation strategy. However, while this strategy improves modularity, it also brings significant optimization problems, making system improvements and upgrades complicated and time-consuming. In addition, Redis is widely relied on as a storage solution for background data and temporary buffering. This dependence exposes a potential weakness of the system. Once Redis fails, the entire service ecosystem may be paralyzed in an instant, causing serious impact on business continuity and stability. When upstream data changes, maintaining data associations between multiple applications is an expensive task that not only requires a large amount of resource investment, but also may lead to data inconsistency and management chaos, affecting overall efficiency. In addition, the current life cycle management of real-time data processing applications, including online, offline and expansion, lacks a centralized and automated management platform, which increases the complexity of operations and the possibility of errors, hindering the ability to quickly respond to business needs.

[0055] This application provides a solution to build a real-time data processing ecological chain, including a model conversion system, a real-time computing center, a configuration management center, and a monitoring and alarm system. The model conversion system is equivalent to a data converter, which can standardize and diversify the external stream data format, and can also eliminate the impact of upstream data changes on downstream systems; in the real-time computing center, with a unified DAG as the carrier, the data stream is divided into multiple task units for distributed processing at the same time, which improves the flexibility and efficiency of the system and ensures real-time and consistency; a series of configuration parameters applicable to standardized data are set in the configuration management center, such as verification rules and monitoring strategies, which are used for the real-time computing center to process and run real-time data; combined with the Kafka platform and dimension table data, real-time data anomaly detection is performed, and an early warning is issued immediately when a problem is found, thereby improving the stability and security of the system.

[0056] It should be noted that the execution subject of this embodiment can be a computing service device with real-time data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, etc. The following takes a personal computer as an example to illustrate this embodiment and the following embodiments.

[0057] Based on this, the present application embodiment provides a real-time data processing method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the real-time data processing method of the present application.

[0058] In this embodiment, the real-time data processing method includes steps S10 to S40:

[0059] Step S10, acquiring real-time data, and standardizing the format of the real-time data to acquire standard real-time data;

[0060] It should be noted that real-time data refers to data that is collected, processed, and used almost immediately after it is generated. This type of data is characterized by immediacy, and usually requires the entire process from generation to analysis to be completed within a few milliseconds to a few seconds.

[0061] It is understandable that, since the current data-driven environment contains a large amount of real-time data and different data formats, performing step S10 can standardize the diverse external stream data formats and eliminate the impact of upstream data changes on downstream systems.

[0062] In a feasible implementation, step S10 may include steps S11 to S12:

[0063] Step S11, constructing a unified data interface;

[0064] Optionally, in the current data-driven environment, efficient and flexible real-time data access is the key to business continuity and data analysis capabilities. This embodiment provides a model conversion system to address this challenge. In this embodiment, real-time data is received by building a unified data interface as a bridge between the external system and the internal processing logic. The interface not only standardizes the diverse external stream data formats, but also eliminates the impact of upstream data changes on downstream systems. Among them, the unified data interface can support multiple data sources (such as APIs, message queues, databases, etc.).

[0065] Step S12: Based on the unified data interface, the real-time data is input into a data converter, and the format of the real-time data is standardized by the data converter to obtain standard real-time data.

[0066] It should be noted that in this embodiment, the model conversion system plays the role of a data converter.

[0067] Optionally, when real-time data enters the system through a unified data interface and then enters the model conversion system, the model conversion system plays the role of a data converter, performs data cleaning and conversion logic on non-standardized real-time data from different sources, converts non-standardized real-time data into a predefined standard format, and obtains real-time data in a standardized format. Standardized data includes but is not limited to a unified timestamp format, a unified numerical unit, a unified data structure, etc., to ensure that all data meets the processing requirements of the internal system. In addition, the model conversion system has a built-in error handling and recovery mechanism that can continuously and reliably provide data services when fluctuations in the data source are detected.

[0068] Through the above steps and intelligent transformation logic, the system can adapt to changes in upstream data, reduce interruptions caused by changes in data structure, and provide a stable data access layer.

[0069] Step S20, configuring the configuration management parameters corresponding to the standard real-time data;

[0070] Optionally, a configuration management center is set up in this embodiment. The configuration management center not only covers traditional configuration and data management functions, but also introduces data experiment management and comprehensive system monitoring, and provides corresponding configuration management parameters, such as data verification strategy, monitoring strategy and early warning mechanism, for the received real-time data during processing and operation. In addition, the configuration management center goes beyond basic configuration management and incorporates real-time event processing and underlying application management to ensure the accuracy and consistency of system configuration and provide robust support for complex system environments. Please refer to Figure 2 , Figure 2 A schematic diagram of the application integration configuration.

[0071] In a feasible implementation, step S20 may include steps S21 to S23:

[0072] Step S21, configuring a data verification strategy corresponding to the processed standard real-time data according to the data experiment function;

[0073] It should be noted that the data experiment function refers to a mechanism that uses experimental methods to test, verify and optimize different data processing strategies, algorithms or models during data processing and analysis.

[0074] Optionally, define data validation strategies based on business needs and real-time data characteristics, including but not limited to data format check, data integrity check, data range check, etc. And use the data experiment management function to perform A / B testing on different data validation strategies, for example, to compare the accuracy and performance of different validation strategies. Among them, A / B testing is a method of randomly dividing users into two or more groups, using different processing strategies or algorithms for each group, and then comparing the effects of these strategies.

[0075] By introducing advanced data experimentation capabilities, new data strategies can be safely tested and validated to optimize business performance while mitigating potential risks.

[0076] Step S22, configuring corresponding monitoring strategies and early warning mechanisms for the standard real-time data to monitor and warn of abnormal situations of the standard real-time data;

[0077] Optionally, first determine the key indicators that need to be monitored, including but not limited to data delay, data volume, error rate, system resource usage, etc. Set the normal range and warning range for each key indicator. When the indicator exceeds the preset range, trigger the warning. During the monitoring process, use monitoring tools to monitor key indicators in real time and display them visually. At the same time, configure multiple response mechanisms to establish an efficient alarm response process to ensure that abnormal problems are handled in a timely manner.

[0078] Through the above steps, the configuration management center provides customized monitoring configuration, captures system anomalies in real time, ensures business continuity and stability, and warns of potential problems in advance through advanced early warning mechanisms.

[0079] Step S23, determining the configuration management parameters corresponding to the standard real-time data through the data verification strategy, the monitoring strategy and the early warning mechanism.

[0080] Optionally, combined with the results of data validation strategy, monitoring strategy and early warning mechanism, the configuration management parameters suitable for the current data processing process are determined. These parameters include but are not limited to: specific settings of data validation rules, thresholds of monitoring indicators, triggering conditions of alarm mechanisms, key parameters in data processing logic, etc.

[0081] In addition, the system allows these configuration management parameters to be adjusted dynamically during data operation to adapt to changing data characteristics and business needs. For example, the configuration management center can also provide in-depth operation and maintenance insights through detailed report output, helping decision makers understand system behavior and drive data-driven decisions.

[0082] Through the above steps, the configuration management center can not only ensure the consistency and quality of data, but also provide real-time monitoring and early warning, thereby improving the stability and reliability of the system. It helps to quickly discover and solve problems during data processing and ensure the continuity and efficiency of data services.

[0083] Step S30, processing the standard real-time data through the directed acyclic graph DAG and the configuration management parameters to obtain the processed standard real-time data;

[0084] It should be noted that DAG (Direct Acyclic Graph) is a graph data structure, which consists of nodes and directed edges, and does not have any cyclic paths.

[0085] It is worth noting that since real-time data usually has high throughput and a very large amount of data, traditional batch processing methods are difficult to handle. Therefore, executing step S30 can avoid problems caused by single point failures and resource bottlenecks, improve data processing efficiency and system stability, etc.

[0086] In a feasible embodiment, step S30 further includes steps S31 to S33;

[0087] Step S31, converting the standard real-time data into various task units according to DAG;

[0088] Before the standard real-time data is logically processed, the system receives user-defined business rules, such as data cleaning, feature extraction, and aggregation calculation. These business rules are accurately mapped into DAG representations before actual execution. In addition, according to the structure of DAG, the standard real-time data is divided into multiple task units, each of which corresponds to a node in DAG, indicating that at this node, the task unit needs to perform data processing according to the business rules defined by the user. At the same time, each task unit exchanges information through the preset Shuffle mechanism to ensure seamless connection of data flow. Among them, the Shuffle mechanism refers to the process of redistributing real-time data from one task (or node) to another task during the real-time data processing process. This process includes: dividing the real-time data into multiple parts, each part corresponds to a target task, sorting the data in each partition, and transmitting the partition data to the corresponding target task, receiving and merging data from different source tasks in the target task.

[0089] Step S32, testing and verifying each task unit through the data verification strategy;

[0090] Optionally, in the application step S20, the data verification policy set by the configuration management center includes format check, integrity check, range check, etc.

[0091] Apply data validation strategies to each task unit and record the validation results of each task unit, including tasks that passed validation and tasks that failed validation. For task units that failed validation, take appropriate error handling measures, such as retry, log recording, alarm notification, etc.

[0092] Step S33, when the task unit passes the verification, logical processing is performed on the verified task unit to obtain processed standard real-time data.

[0093] Optionally, the preset logic processing is performed on the verified task units, that is, the real-time data in the task units is parsed, aggregated, etc. During the processing, the topological structure of DAG is used to identify task units that can be executed in parallel and run simultaneously on multiple computing nodes to improve processing efficiency.

[0094] Please refer to Figure 3 , Figure 3 The following is a schematic diagram of the data processing flow based on the GAD map provided in this embodiment. Figure 3It can be seen that in this embodiment, the "sourceTable" node is connected to the "idMappingTable" node, and both serve as different data sources. The data in the "sourceTable" node and the "idMappingTable" node are sent to the processing node in the form of Kafka queues. After the node parses and aggregates the real-time data, the processed data is sent to the sinkTable node, which can be used to store the processed data. Please refer to Figure 4 , Figure 4 The code diagram of the data processing process based on the GAD map provided in this embodiment. It should be noted that: Figure 4 The table shows the relevant information of the browse_src table, which is the source of the mobile banking browsing data. The table obtains data from the Kafka topic RDP_ALW54_EBM_MTS_MB001_SZ, using the EBM-Kafka connector, version 1. The connector is configured with information such as the Kafka server address, security protocol, authentication method, and username and password. The data format is EBM-JSON, which contains two fields: APPID and PageNo, which represent the device number and the browsing page code respectively.

[0095] Step S40, when the processed standard real-time data is running, based on the configuration management parameters, combined with the pre-built distributed event stream platform Kafka and dimension table data, data anomaly monitoring and early warning are performed on the processed standard real-time data.

[0096] It should be noted that Kafka is an open source distributed event streaming platform with high throughput and low latency characteristics, which enables efficient capture and processing of real-time data and dynamic monitoring of system operation status.

[0097] Dimension table data refers to the tables used in the data warehouse to describe the detailed information of the data in the fact table. Dimension table data usually contains attributes and descriptive information to provide richer contextual information and ensure accurate analysis and early warning of complex business scenarios.

[0098] In a feasible implementation, step S40 may include steps S41 to S42:

[0099] Step S41, when the processed standard real-time data is running, based on the monitoring strategy, the processed standard real-time data is monitored and analyzed using the pre-built Kafka and dimension table data;

[0100] When the processed standard real-time data starts running, the system will use the already built Kafka distributed event stream platform and dimension table data to continuously monitor and deeply analyze the data according to the pre-defined monitoring strategy.

[0101] Specifically, consumer groups are created in different task units, and each group can subscribe to one or more Kafka topics. Kafka stores processed standard real-time data, and continuously monitors the status of the Kafka cluster based on Kafka's own monitoring tools or third-party monitoring systems (such as Prometheus, Grafana, etc.). In addition, combined with the static information in the dimension table, it can provide richer contextual information for the monitored business scenarios, thereby enhancing the depth and accuracy of data analysis. Kafka establishes a baseline model based on historical data, compares the differences between current data and the baseline in real time, and discovers potential problems.

[0102] The system combines Kafka and dimension table data to monitor key indicators such as CPU, memory, and network to promptly identify potential performance bottlenecks.

[0103] Step S42: Based on the early warning mechanism, an early warning is issued for abnormal standard real-time data monitored in the processed standard real-time data.

[0104] Optionally, the early warning mechanism configured in step S20 is used to warn the monitored abnormal implementation data. Specifically, using a pre-set early warning threshold, when the abnormal implementation data meets the early warning threshold, the system will immediately trigger an alarm. The early warning threshold in this embodiment is a dynamic threshold set according to business needs. Once the key indicator deviates from the normal range, the alarm is immediately triggered.

[0105] Among them, by establishing an efficient alarm response process in advance, we ensure that the problem is handled in a timely manner. The alarm methods include but are not limited to SMS, email, phone or internal communication software notifications. For different serious abnormal situations, different alarm levels are set and responded to with corresponding urgency.

[0106] In addition, this embodiment also formulates preventive measures through historical data and early warning information to reduce the possibility of failure. The historical data includes operation logs, maintenance records, fault reports and other information within a period of time.

[0107] In summary, this embodiment builds an early warning mechanism that integrates Kafka, dimension table data processing, and all-round monitoring to detect and prevent various anomalies in real time, thereby improving the overall system performance.

[0108] The above-mentioned embodiment method includes obtaining real-time data, standardizing the format of real-time data, and obtaining standard real-time data; configuring the corresponding configuration management parameters of standard real-time data; processing the standard real-time data through DAG and configuration management parameters to obtain the processed standard real-time data; when the processed standard real-time data is running, based on the configuration management parameters, combined with the pre-built distributed event stream platform Kafka and dimension table data, the processed standard real-time data is monitored and warned for data anomalies. This method standardizes the diverse external stream data formats by building a unified interface, and also eliminates the impact of upstream data changes on downstream systems; using DAG to organize data processing processes can achieve efficient parallel computing, so that large-scale data sets can be processed quickly; using Kafka and dimension table data in conjunction with configuration management parameters to monitor and warn of data anomalies can detect potential problems in the first place, and take measures to prevent the spread of adverse effects, thereby improving the stability and security of the system.

[0109] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 5 , step S42 also includes steps A11 to A14:

[0110] Step A11, when abnormal standard real-time data is detected in the processed standard real-time data, identifying application behavior of the abnormal standard real-time data;

[0111] It should be noted that application behavior refers to the operating mode exhibited by the system or application under specific conditions, including but not limited to starting, stopping, and adjusting parameters of the device.

[0112] Optionally, first, the application behavior in the abnormal standard real-time data is identified through a machine learning algorithm. Among them, normal application behavior can be defined based on historical data, including but not limited to routine operations such as device startup, shutdown, and parameter adjustment. At the same time, abnormal application behavior can be defined based on historical data, including but not limited to unusual operation modes, frequent restarts, abnormal parameter settings, etc.

[0113] For real-time data that is judged to be abnormal, further analyze its specific behavior patterns. For example, use sequence pattern mining algorithms (such as PrefixSpan, GSP) to identify specific operation sequences. Or build a rule engine based on business rules and expert knowledge to identify specific abnormal patterns.

[0114] Step A12, based on the application behavior of the abnormal standard real-time data, obtaining key indicators corresponding to the abnormal standard real-time data;

[0115] Optionally, based on the identified application behavior, determine which performance indicators are critical. For example, if the device is identified to be frequently restarted, CPU usage, memory usage, temperature, etc. may be critical indicators. And extract the specific values ​​of these critical indicators from the abnormal data.

[0116] In addition, this embodiment also constructs a business dashboard to intuitively display key business indicators and reflect the business health status of real-time data to users in real time.

[0117] Step A13, verifying whether the key indicator corresponding to the abnormal standard real-time data deviates from the preset warning threshold;

[0118] Optionally, first, set a dynamic warning threshold according to business needs, and then compare the extracted key indicator value with the preset warning threshold. For example, collect traffic data in the past month. Calculate the 767th percentile traffic value and use the 767th percentile as the warning threshold. Then, collect and calculate the traffic value of the current month, and compare the traffic value of the current month with the warning threshold. If the traffic value of the current month is greater than the warning threshold, it is determined that the abnormal standard real-time data meets the warning standard. Please refer to Figure 6 , Figure 6 A schematic diagram of the early warning and alarm process provided by the method of this embodiment.

[0119] Step A14: If yes, an alarm signal is triggered, and in response to the alarm signal, the abnormal standard real-time data is displayed to the user.

[0120] Optionally, if an abnormality is detected (i.e., real-time data exceeds the warning threshold), an alarm signal is triggered immediately. Relevant personnel are notified via SMS, email, internal communication platform, etc., and the abnormal data and its context information are displayed on the monitoring interface to facilitate quick problem location.

[0121] In addition, through the association between parent data and child data, when the application behavior of abnormal real-time data and the cause of the abnormality are identified, the established data lineage relationship is used to trace back all related child data starting from the abnormal data. This can usually be achieved through graph database queries or pre-built data lineage maps. Key performance indicators such as temperature, pressure, flow, etc. are extracted from the child data. These key indicators are compared with the preset dynamic warning thresholds to determine whether there are further abnormalities. Combined with the analysis results of the parent data and child data, a comprehensive assessment is conducted to determine the root cause of the problem. If the child data is also found to be abnormal, or the severity of the problem is further confirmed, an alarm signal is immediately triggered. The abnormal data and its contextual information, including the specific situation of the parent data and child data, are displayed on the monitoring interface. Please refer to Figure 7 , Figure 7 This is a flowchart of bloodline data management.

[0122] Through the above-mentioned embodiment method, dynamic warning thresholds can be flexibly set according to business needs, and abnormal situations can be discovered and handled in a timely manner. This can not only improve the response speed of the system, but also reduce false positives and missed positives.

[0123] For example, to help understand the implementation process of the real-time data processing method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 8 , Figure 8 A system architecture diagram of a real-time data processing method is provided, specifically:

[0124] This embodiment provides a real-time data processing system, which is divided into a computing layer, a support layer, a configuration management layer, and a storage layer. This layering strategy aims to implement a refined split of the business field and build a modular architecture that includes a real-time data analysis engine, a model conversion service platform, a data injection system, and a configuration management hub. The core concept is that through such decoupling processing, the mutual dependence between business logic and data processing logic is effectively weakened to achieve the purpose of reducing complexity. We emphasize unified service component encapsulation, aiming to hide the complexity of the underlying implementation, thereby improving the maintainability and scalability of the system and ensuring smooth business processes.

[0125] Please refer to Fig. 9 , Fig. 9 The microservice architecture design diagram of the method of this embodiment is shown in FIG. Fig. 9 It can be seen that the real-time data processing system of this embodiment adopts a microservice architecture, provides services externally through HTTP API, and uses event-driven communication internally. After processing, the data is stored in the data lake, part of the online data is stored in Redis for fast access, and the other part of the offline data is used for batch processing and analysis. The system uses Docker and Kubernetes for containerized management and deployment, and Elasticsearch is used for full-text retrieval and data analysis.

[0126] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the real-time data processing method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0127] This application also provides a real-time data processing system, please refer to Fig.10 , the real-time data processing system comprises:

[0128] The conversion module 10 is used to obtain real-time data and standardize the format of the real-time data to obtain standard real-time data;

[0129] Configuration module 20, used to configure configuration management parameters corresponding to the standard real-time data;

[0130] The processing module 30 is used to process the standard real-time data through the directed acyclic graph DAG and the configuration management parameters to obtain the processed standard real-time data;

[0131] The early warning module 40 is used to monitor and warn the processed standard real-time data for data anomalies when the processed standard real-time data is running, based on the configuration management parameters, combined with the pre-built distributed event stream platform Kafka and dimension table data.

[0132] The real-time data processing system provided by the present application adopts the real-time data processing method in the above embodiment, which can solve the technical problem of how to build a real-time data processing ecological chain to improve the efficiency, reliability and management efficiency of real-time data processing, while reducing operating costs and risks. Compared with the prior art, the beneficial effects of the real-time data processing system provided by the present application are the same as the beneficial effects of the real-time data processing method provided by the above embodiment, and other technical features in the real-time data processing system are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0133] The present application provides a real-time data processing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the real-time data processing method in the above-mentioned embodiment 1.

[0134] Reference below Fig.11 , which shows a schematic diagram of the structure of a real-time data processing device suitable for implementing the embodiment of the present application. The real-time data processing device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.11 The real-time data processing device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0135] like Fig.11As shown, the real-time data processing device may include a processing system 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage system 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the real-time data processing device are also stored. The processing system 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input system 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output system 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage system 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication system 1009. The communication system 1009 can allow the real-time data processing device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a real-time data processing device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0136] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication system, or installed from a storage system 1003, or installed from a ROM 1002. When the computer program is executed by the processing system 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0137] The real-time data processing device provided by the present application adopts the real-time data processing method in the above embodiment, which can solve the technical problem of how to build a real-time data processing ecological chain to improve the efficiency, reliability and management efficiency of real-time data processing, while reducing operating costs and risks. Compared with the prior art, the beneficial effects of the real-time data processing device provided by the present application are the same as the beneficial effects of the real-time data processing method provided by the above embodiment, and the other technical features in the real-time data processing device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0138] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0139] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0140] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the real-time data processing method in the above-mentioned embodiment.

[0141] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0142] The computer-readable storage medium may be included in the real-time data processing device; or may exist independently without being assembled into the real-time data processing device.

[0143] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a real-time data processing device, the real-time data processing device: obtains real-time data, and standardizes the format of the real-time data to obtain standard real-time data; configures configuration management parameters corresponding to the standard real-time data; processes the standard real-time data through a directed acyclic graph DAG and the configuration management parameters to obtain processed standard real-time data; when the processed standard real-time data is running, based on the configuration management parameters, combined with the pre-built distributed event stream platform Kafka and dimension table data, performs data anomaly monitoring and early warning on the processed standard real-time data.

[0144] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0145] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0146] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0147] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned real-time data processing method, and can solve the technical problem of how to build a real-time data processing ecological chain to improve the efficiency, reliability, and management efficiency of real-time data processing, while reducing operating costs and risks. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the real-time data processing method provided by the above-mentioned embodiment, and will not be repeated here.

[0148] The present application also provides a computer program product, including a computer program, which implements the steps of the real-time data processing method as described above when executed by a processor.

[0149] The computer program product provided by this application can solve the technical problem of how to build a real-time data processing ecosystem to improve the efficiency, reliability and management efficiency of real-time data processing while reducing operating costs and risks. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the real-time data processing method provided by the above embodiment, and will not be repeated here.

[0150] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A real-time data processing method, characterized in that: The method comprises: Acquire real-time data, and standardize the format of the real-time data to obtain standard real-time data; Configuring configuration management parameters corresponding to the standard real-time data; Processing the standard real-time data through the directed acyclic graph DAG and the configuration management parameters to obtain the processed standard real-time data; When the processed standard real-time data is running, based on the configuration management parameters, combined with the pre-built distributed event stream platform Kafka and dimension table data, data anomaly monitoring and early warning are performed on the processed standard real-time data.

2. The method according to claim 1, characterized in that The step of standardizing the format of the real-time data to obtain standard real-time data includes: Build a unified data interface; Based on the unified data interface, the real-time data is input into a data converter, and the format of the real-time data is standardized by the data converter to obtain standard real-time data.

3. The method according to claim 1, characterized in that The step of configuring the configuration management parameters corresponding to the standard real-time data includes: According to the data experiment function, configure the data verification strategy corresponding to the processed standard real-time data; Configuring corresponding monitoring strategies and early warning mechanisms for the standard real-time data to monitor and warn of abnormal situations of the standard real-time data; The configuration management parameters corresponding to the standard real-time data are determined through the data verification strategy, the monitoring strategy and the early warning mechanism.

4. The method according to claim 3, characterized in that The step of processing the standard real-time data through the directed acyclic graph DAG and the configuration management parameters to obtain the processed standard real-time data includes: Convert standard real-time data into individual task units according to DAG; Testing and verifying each of the task units through the data verification strategy; When the task unit passes the verification, logic processing is performed on the verified task unit to obtain processed standard real-time data.

5. The method according to claim 3, characterized in that When the processed standard real-time data is running, based on the configuration management parameters, combined with the pre-built distributed event stream platform Kafka and dimension table data, data anomaly monitoring and early warning of the processed standard real-time data include: When the processed standard real-time data is running, the processed standard real-time data is monitored and analyzed using the pre-built Kafka and dimension table data based on the monitoring strategy; Based on the early warning mechanism, early warning and alarm are performed on abnormal standard real-time data monitored in the processed standard real-time data.

6. The method according to claim 5, characterized in that The step of providing early warning and alarm for abnormal standard real-time data monitored in the processed standard real-time data based on the early warning mechanism includes: When abnormal standard real-time data is detected in the processed standard real-time data, identifying application behavior of the abnormal standard real-time data; Based on the application behavior of the abnormal standard real-time data, obtaining key indicators corresponding to the abnormal standard real-time data; Verify whether the key indicators corresponding to the abnormal standard real-time data deviate from the preset warning threshold; If so, an alarm signal is triggered, and in response to the alarm signal, the abnormal standard real-time data is displayed to the user.

7. A real-time data processing system, characterized in that: The system comprises: A conversion module, used to obtain real-time data and standardize the format of the real-time data to obtain standard real-time data; A configuration module, used to configure configuration management parameters corresponding to the standard real-time data; A processing module, used to process the standard real-time data through the directed acyclic graph DAG and the configuration management parameters to obtain the processed standard real-time data; The early warning module is used to monitor and warn the processed standard real-time data for data anomalies when the processed standard real-time data is running, based on the configuration management parameters, combined with the pre-built distributed event stream platform Kafka and dimension table data.

8. A real-time data processing device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the real-time data processing method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the real-time data processing method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the real-time data processing method according to any one of claims 1 to 6 are implemented.

Images