Data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to a data processing method and device, computer equipment, a readable storage medium and a program product. The method comprises the following steps: inputting to-be-written data in a data writing request into a data decompression component, and outputting the decompressed to-be-written data; decoding the decompressed data to be written on the basis of a data serialization protocol to obtain structured time sequence data; obtaining a tetrad of any to-be-written data in the time series data; and concurrently writing all tetrads corresponding to the time sequence data into a memory buffer area of the writer based on the write statement and the collaborative program, and storing the data in the memory buffer area into a column database. According to the method provided by the invention, few memory resources are occupied in the data writing process, and stable operation of the service can be ensured.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, readable storage medium, and program product. Background Technology

[0002] With the rapid adoption of container technology, enterprises are increasingly inclined to containerize their applications to achieve highly scalable and flexible application deployment. However, the high dynamism and complexity of containers lead to increasingly large amounts of data generated by them. This results in significant memory consumption during the process of writing container data to the database, impacting database stability. Summary of the Invention

[0003] Therefore, it is necessary to provide a data processing method, apparatus, computer equipment, readable storage medium, and program product that can reduce the memory resources occupied during data storage, in order to address the above-mentioned technical problems.

[0004] In a first aspect, this application provides a data processing method, the method comprising:

[0005] Receives a data write request sent by the client, inputs the data to be written in the data write request to the data decompression component, and outputs the decompressed data to be written.

[0006] The decompressed data to be written is decoded based on a data serialization protocol to obtain structured time-series data; wherein the data to be written in the time-series data is sorted according to the acquisition time.

[0007] Obtain a quadruple of any data to be written from the time series data; wherein, the quadruple includes the data to be written, the identifier of the indicator object to which the data to be written belongs, and the label and collection time of the data to be written;

[0008] Obtain the write statement and the cooperative procedure. Based on the write statement and the cooperative procedure, concurrently write all the corresponding quadruples of the time series data to the memory buffer of the writer, and store the data in the memory buffer into the columnar database.

[0009] In one embodiment, the writer is connected to the columnar database via a database connection pool; the step of storing the data in the memory buffer into the columnar database includes:

[0010] According to a preset period, the data cached in the memory buffer is written to the columnar database through the database connection pool.

[0011] In one embodiment, the method further includes:

[0012] The system receives a data read request sent by the client, inputs the request data in the data read request into the data decompression component, and outputs the decompressed request data.

[0013] The decompressed request data is decoded based on the data serialization protocol to obtain at least one data query request instruction, and a data query statement is constructed based on the data query request instruction.

[0014] Based on the data query statement, the result data corresponding to the data read request is read from the columnar database, and the result data is encoded based on the data serialization protocol to obtain the encoded result data;

[0015] The encoded result data is input into the data compression component, the compressed result data is output, and the compressed result data is returned to the client.

[0016] In one embodiment, reading the result data corresponding to the data read request from the columnar database based on the data query statement includes:

[0017] For the business attributes of the data queried by the data query statement, determine the data columns in the columnar database that store the business attribute data;

[0018] Based on the data column, the data row containing the result data is filtered out from the columnar database;

[0019] The result data is filtered from the data rows.

[0020] In one embodiment, reading the result data corresponding to the data read request from the columnar database based on the data query statement includes:

[0021] Based on a preset sampling key, the data stored in the columnar database is divided into multiple uniform initial sampling groups;

[0022] The target sampling group is determined from multiple initial sampling groups according to a preset ratio;

[0023] The data in the target sampling group are integrated, and the integrated data is scaled based on the preset ratio;

[0024] The scaled data is determined as the resulting data.

[0025] In one embodiment, the method further includes:

[0026] Multiple preset alarm rules are obtained, and the triggering condition of any one preset alarm rule is determined; wherein, the triggering condition indicates that the indicator data of at least one indicator object in the business container meets the preset condition;

[0027] Real-time acquisition of metric data from the metric objects in the business container;

[0028] For any given preset alarm rule, if the indicator data of all the corresponding indicator objects meet the corresponding preset conditions, an alarm is issued based on the preset alarm rule.

[0029] Secondly, this application also provides a data processing apparatus, the apparatus comprising:

[0030] The receiving module is used to receive data write requests sent by the client, input the data to be written in the data write request to the data decompression component, and output the decompressed data to be written.

[0031] The decoding module is used to decode the decompressed data to be written based on a data serialization protocol to obtain structured time-series data; wherein the data to be written in the time-series data is sorted according to the acquisition time.

[0032] The first acquisition module is used to acquire a quadruple of any data to be written in the time series data; wherein, the quadruple includes the data to be written, the identifier of the indicator object to which the data to be written belongs, and the label and acquisition time of the data to be written;

[0033] The second acquisition module is used to acquire the write statement and the cooperative program, and based on the write statement and the cooperative program, concurrently write all the corresponding quadruples of the time series data to the memory buffer of the writer, and store the data in the memory buffer into the columnar database.

[0034] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the methods in any of the above embodiments.

[0035] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0036] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the steps of the methods in any of the above embodiments.

[0037] The aforementioned data processing method, apparatus, computer equipment, readable storage medium, and program product receive a data write request sent by a client, input the data to be written in the data write request to a data decompression component, and output the decompressed data to be written; decode the decompressed data to be written based on a data serialization protocol to obtain structured time-series data; wherein the data to be written in the time-series data is sorted according to the acquisition time; obtain a quadruple of any data to be written in the time-series data; wherein the quadruple includes the data to be written, the identifier of the indicator object to which the data to be written belongs, and the label and acquisition time of the data to be written; obtain a write statement and a cooperative program, and based on the write statement and the cooperative program, concurrently write all the corresponding quadruples of the time-series data to the memory buffer of the writer, and store the data in the memory buffer in a columnar database. The method provided in this application writes all quadruples to the memory buffer of the writer first, instead of writing them directly to the database. This reduces the number of database writes, lowers database I / O (Input / Output) overhead and connection usage. The cooperative program is more lightweight than operating system threads, consumes very little memory resources during concurrent processing, and can ensure stable service operation. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating a data processing method in one embodiment;

[0040] Figure 2 This is a flowchart illustrating a data reading method in one embodiment;

[0041] Figure 3 This is a structural block diagram of a data processing device in one embodiment;

[0042] Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0045] In one embodiment, such as Figure 1 As shown, a data processing method is provided. This embodiment illustrates the method applied to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0046] S102. Receive the data write request sent by the client, input the data to be written in the data write request to the data decompression component, and output the decompressed data to be written.

[0047] Optionally, the decompression component may include, but is not limited to, the Snappy algorithm, which is an open-source high-speed data compression / decompression algorithm.

[0048] Optionally, in Prometheus application scenarios, the data to be written can be, but is not limited to, metric data of various monitored metric objects. Prometheus is an open-source system monitoring and alerting tool and a core component in the monitoring field of the cloud-native ecosystem. The monitored metric objects can include, but are not limited to, MySQL connection counts, Redis (RemoteDictionary Server) connection counts, and TCP port status. MySQL is an open-source relational database management system. MySQL and Redis connection counts can help track concurrent access to databases and caching services, while TCP port status can observe the stability and load of network connections. Before collecting the data to be written, the defined metric objects are registered to the default registry of Prometheus through a registration function, enabling Prometheus to recognize these metric objects and pull and monitor the metric data through an HTTP interface. The registration function can be, but is not limited to, the prometheus.MustRegister function, which is a core function in the official Prometheus Go client library.

[0049] S104. Decode the decompressed data to be written based on the data serialization protocol to obtain structured time-series data; wherein, the data to be written in the time-series data is sorted according to the acquisition time.

[0050] Optionally, the data serialization protocol may be, but is not limited to, the Protocol Buffers protocol. The Protocol Buffers protocol is a platform-independent, language-independent, and scalable structured data serialization protocol. Its core function is to convert complex structured data into a compact binary format for efficient data storage and network transmission. It is often used as a cross-system and cross-language data exchange format.

[0051] Optionally, the Protocol Buffers protocol first decodes the data to be written into a remote.WriteRequest structure, and then processes the remote.WriteRequest object through the process function to extract the time series. Here, remote.WriteRequest is a core data structure defined in the Prometheus Remote Write Protocol, which is specifically used to carry monitoring sample data pushed by Prometheus to external systems. The process function is a core data processing function whose core function is to process the remote.WriteRequest object obtained after decoding Protobuf into standardized data that can be written to the database later.

[0052] S106. Obtain a quadruple of any data to be written from the time series data; wherein, the quadruple includes the data to be written, the identifier of the indicator object to which the data to be written belongs, and the label and collection time of the data to be written.

[0053] Optionally, by constructing a quadruple of the data to be written, it is possible to quickly verify the integrity and format validity of the data, construct a batch writing template, and confirm the timing sorting rules, providing a unified benchmark for subsequent concurrent writing to the memory buffer and batch writing, and ensuring the efficiency, reliability and orderliness of data writing.

[0054] S108. Obtain the write statement and the cooperative procedure. Based on the write statement and the cooperative procedure, concurrently write all the corresponding quadruples of the time series data to the memory buffer of the writer, and store the data in the memory buffer into the columnar database.

[0055] Optionally, the write statement can be, but is not limited to, SQL (Structured Query Language) statements; cooperative programs specifically refer to "lightweight concurrent execution units," namely Goroutines, whose core function is to quickly process massive four-tuple write tasks through "cooperative concurrency," thereby improving the overall write throughput; ClickHouse is an open-source column-oriented database management system designed specifically for analytical scenarios.

[0056] Optionally, when the write service needs to be shut down, first close the write request channel, then wait for the writer to complete all write operations, and finally control the writer to release resources, close database connections, clean up the worker pool, and release coroutines in a timely manner to ensure that the service is shut down correctly and gracefully. This can avoid resource leaks and unnecessary resource occupation, and ensure that the system can release all occupied resources when shutting down. If the write operation is not completed within 10 seconds, a log will be printed to indicate that the sample is lost.

[0057] In the above data processing method, a data write request is received from the client, and the data to be written in the data write request is input to the data decompression component, outputting the decompressed data to be written; the decompressed data to be written is decoded based on the data serialization protocol to obtain structured time-series data; the data to be written in the time-series data is sorted according to the acquisition time; a quadruple of any data to be written in the time-series data is obtained; the quadruple includes the data to be written, the identifier of the indicator object to which the data to be written belongs, the label of the data to be written, and the acquisition time; a write statement and a cooperative program are obtained, and based on the write statement and the cooperative program, all the corresponding quadruples of the time-series data are concurrently written to the memory buffer of the writer, and the data in the memory buffer is stored in the columnar database. The method provided in this application writes all quadruples to the memory buffer of the writer first, instead of writing directly to the database, which can reduce the number of database writes, reduce database IO (Input / Output) overhead and connection occupation; the cooperative program is lighter than the operating system thread, occupies very little memory resources during concurrent processing, and can ensure stable service operation.

[0058] In some embodiments, the writer is connected to the columnar database via a database connection pool; storing data in the memory buffer into the columnar database includes: writing the cached data in the memory buffer into the columnar database via the database connection pool at a preset period.

[0059] A database connection pool is a pre-created, reusable database connection resource pool with a management mechanism. Its core is to establish a certain number of database connections in advance, manage and reuse these connections in a unified manner, and avoid recreating / closing connections every time data is written, thereby reducing connection overhead, improving write efficiency, and ensuring database stability.

[0060] Optionally, when writing data, the writer can combine multiple write operations into a transaction to ensure the atomicity of the write and data consistency; if a write operation fails, the entire transaction will be rolled back to avoid data inconsistency.

[0061] In this embodiment, sample data is temporarily stored in a memory buffer in the writer instead of being written directly to the columnar database. The data is written to the columnar database in batches periodically, reducing frequent database write operations. This can significantly improve write performance, reduce the overhead of each write operation, and alleviate the load on the columnar database.

[0062] In some embodiments, such as Figure 2 As shown, the method also includes:

[0063] S202. Receive the data read request sent by the client, input the request data in the data read request into the data decompression component, and output the decompressed request data.

[0064] S204. Decode the decompressed request data based on the data serialization protocol to obtain at least one data query request instruction, and construct a data query statement based on the data query request instruction.

[0065] S206. Based on the data query statement, read the corresponding result data from the columnar database, and encode the result data according to the data serialization protocol to obtain the encoded result data.

[0066] S208. Input the encoded result data into the data compression component, output the compressed result data, and return the compressed result data to the client.

[0067] Optionally, after receiving a data read request from the client, the server first decompresses the data read request using the Snappy algorithm and decodes it into a remote.ReadRequest object using the Protocol Buffers protocol. The remote.ReadRequest object contains one or more query instructions. Then, the reader processes the remote.ReadRequest object, constructs a corresponding SQL query statement based on each query instruction, and queries the ClickHouse database for data. The query results are encoded using the Protocol Buffers protocol, compressed using the Snappy algorithm, and then returned to the client as an HTTP response.

[0068] Optionally, during the data writing and reading process, the writer and reader processes can be comprehensively monitored by recording metrics such as the processing time of each request, database write latency, and database query time. By exposing the metric data to Prometheus, service performance can be monitored and optimized in real time, and performance bottlenecks and problems can be identified in a timely manner.

[0069] Optionally, when reading large amounts of data, paginated queries can be added to return the query results in batches, avoiding excessive memory usage caused by loading all data at once.

[0070] In this embodiment, a standardized process of "decompression → decoding → constructing query statement → reading database → encoding → compression → returning" is adopted to achieve high efficiency, accuracy and compatibility of data reading, while reducing network bandwidth consumption and database load, and improving client query response speed.

[0071] In some embodiments, reading the result data corresponding to the data read request from the columnar database based on the data query statement includes: determining the data column in the columnar database that stores the business attribute data based on the business attributes of the data queried by the data query statement; filtering out the data row containing the result data from the columnar database based on the data column; and filtering out the result data from the data row.

[0072] Among them, business attribute data refers to "descriptive data used to identify the business scenario, dimension or feature to which the data belongs. Its core function is to accurately locate "which monitoring data is needed by the client". In essence, it is "filtering dimension / identification information" when querying.

[0073] Optionally, the result data can be read from the database using the PREWHERE clause. The PREWHERE clause is a "pre-filtering clause" exclusive to ClickHouse. Its core function is to perform "lightweight and fast filtering" on the data before executing the main query (WHERE clause), keeping only the data blocks that may meet the conditions, and then performing subsequent field reading and filtering, ultimately reducing IO overhead and improving query speed.

[0074] In this embodiment, the precise positioning logic of "business attribute → corresponding data column → filter data row → extract result" avoids full data scanning, reduces database IO overhead and query time, and ensures the accuracy and relevance of query results.

[0075] In some embodiments, reading the result data corresponding to the data reading request from the columnar database based on the data query statement includes: dividing the data stored in the columnar database into multiple uniform initial sampling groups based on a preset sampling key; determining a target sampling group from the multiple initial sampling groups according to a preset ratio; integrating the data in the target sampling group and scaling the integrated data according to a preset ratio; and determining the scaled data as the result data.

[0076] The sampling key is the core field / expression used to "uniformly group" the full data. It is essentially a "grouping identifier" for the data and must have the characteristics of "uniform distribution and high discrimination" to ensure that the sampling results can approximately reflect the characteristics of the full data and avoid sampling bias.

[0077] Optionally, the data in the target sampling group can be integrated, but is not limited to obtaining the average value of the data in the target sampling group.

[0078] Optionally, the SAMPLE BY clause is ClickHouse's proprietary "data sampling syntax," whose core function is to quickly extract a portion of samples from massive amounts of data based on the sampling key.

[0079] In this embodiment, by uniformly grouping the sampling key, selecting samples proportionally, and integrating and scaling, a near-full result can be quickly obtained with a small amount of sample data, which greatly reduces query I / O and computational overhead, improves the response speed of massive data queries, and balances query efficiency and result accuracy.

[0080] In some embodiments, the method further includes: acquiring a plurality of preset alarm rules and determining the triggering condition of any one preset alarm rule; wherein the triggering condition indicates that the metric data of at least one metric object in the business container meets the preset condition; acquiring the metric data of the metric objects in the business container in real time; and issuing an alarm based on the preset alarm rule when the metric data of all metric objects corresponding to the preset alarm rule meet the corresponding preset condition.

[0081] Optionally, the preset alarm rules can be, but are not limited to, combined alarm rules. Combined alarms can establish a correlation between multiple individual alarms. A combined alarm will only be triggered when a specific set of alarms meets the conditions simultaneously. This helps to avoid confusion and duplication of single alarms.

[0082] Optionally, when operations and maintenance personnel need to set a memory leak alert, they typically monitor container memory usage, designated as monitoring metric A. When memory usage exceeds a certain threshold, it indicates a risk of container memory leaks. However, monitoring only this metric often leads to numerous false alarms. Container CPU usage is another important metric that needs to be monitored. When there is a memory leak, it often causes an abnormal increase in CPU usage, designated as monitoring metric B. Container runtime is also a metric worth paying attention to, as memory leaks often become apparent only after the container has been running for a period of time, designated as monitoring metric C. Therefore, for memory leak alerts, when metrics A, B, and C are all met simultaneously, a combined alert is formed, which significantly improves the monitoring success rate of memory leak alerts.

[0083] In this embodiment, by combining multiple related alarms into one alarm, unnecessary noise can be reduced and alarms with more informational value can be provided. By combining alarms, more complex problem logic can be handled, so that some more complex problems can be more accurately identified through conditional judgment from multiple perspectives.

[0084] Optionally, the metrics objects that need to be monitored in the business container in this application may include, but are not limited to, hardware resource metrics, JVM (Java Virtual Machine) metrics, Tomcat metrics, and container connection information; wherein, Tomcat refers to an open-source Java Web (World Wide Web) server used to deploy and run Java Web applications.

[0085] Optionally, hardware resource metrics may include, but are not limited to, CPU utilization, memory utilization, disk space, and network bandwidth.

[0086] Optionally, JVM metrics may include, but are not limited to, JVM running status, JVM runtime, heap memory, non-heap memory, number of GC (Garbage Collection) events and GC time, peak number of threads, current number of threads, number of daemon threads, and thread status.

[0087] Optionally, Tomcat metrics may include, but are not limited to, thread status, connection count, received traffic data, received request volume, error request volume, and startup time. Thread status is used to monitor the maximum number of Tomcat threads, the number of busy threads, and the current number of threads, so as to dynamically adjust the thread pool configuration. Connection count is used to monitor the number of Tomcat connections to avoid performance degradation or resource exhaustion due to excessive connections. Received traffic data is used to track the traffic received by Tomcat to ensure that the application processes requests normally. Received request volume is used to record the number of requests received by Tomcat to help analyze the load. Error request volume is used to monitor the number of Tomcat error requests to help troubleshoot anomalies. Startup time is used to record the Tomcat startup time to evaluate application startup performance.

[0088] Optionally, container connection information may include, but is not limited to, HTTP connection status, MySQL connection count, Redis connection count, and Redis QPS (Queries Per Second). HTTP connection status is used to monitor the status of HTTP connections, including the number of Listen connections, Established connections, TimeWait connections, FinWait1 connections, FinWait2 connections, and Closing connections. Listen indicates that the server-side port is waiting for client connection requests; monitoring this value can determine whether the service is providing normal connection capabilities. Established indicates that the client and server have completed the TCP three-way handshake, and the connection is in normal communication; this is the core business connection status. TimeWait indicates that the connection has been closed, and the server waits for a period of time to ensure the client receives the FIN packet, avoiding excessive port usage. FinWait1 indicates that the server has sent a FIN packet. The first step in closing the connection is to send a FIN packet (with a termination bit) and wait for the client to return an ACK (Acknowledgment) packet. FinWait2 indicates that the server has received the client's ACK confirmation for the FIN packet and is waiting for the client to send its own FIN packet (the client also needs to close the connection). Closing indicates that the server sends a FIN packet but does not receive an ACK confirmation; instead, it receives the client's FIN packet first. At this point, both parties close the connection simultaneously, entering a transitional state. MySQL connection count is used to monitor the number of connections established between the container and MySQL, preventing the connection pool and database resources from being exhausted. Redis connection count is used to monitor the number of connections established between the container and Redis, ensuring that the application can communicate normally with Redis. RedisQPS is used to monitor Redis QPS peaks, helping to identify potential performance bottlenecks, such as frequent hot queries or write operations. By optimizing related business logic or using caching strategies, the load on Redis can be reduced, improving overall performance.

[0089] In one exemplary embodiment, another data processing method is provided, which includes the following:

[0090] (1) Data writing logic

[0091] Prometheus sends collected metric data to the P2CServer via HTTP API in the form of WriteRequests. Upon receiving a WriteRequest, the P2CServer decompresses the data (using the snappy compression algorithm) and deserializes it to obtain the write request. The P2CServer uses multiple goroutines to concurrently process the write requests, writing the metric data to a memory buffer. The main components of the P2CServer service include an HTTP service handler, a writer, a reader, and a logger. It is responsible for handling Prometheus's write and read requests, as well as starting and stopping the service.

[0092] (2) Buffer write logic

[0093] In the p2cWriter, sample data is temporarily stored in a memory buffer instead of being written directly to the ClickHouse database. Data is periodically written to the ClickHouse database in batches, reducing frequent database write operations. This significantly improves write performance, reduces the overhead of each write operation, and alleviates the load on the ClickHouse database.

[0094] (3) Connection pool and transaction processing logic

[0095] The p2cWriter uses a database connection pool to reuse connections, avoiding the need to create a new connection for each write operation. Managing database connections through a connection pool reduces the overhead of connection creation and destruction, improving write performance. Furthermore, when writing data, the writer combines multiple write operations into a single transaction, ensuring atomicity and data consistency. If a write operation fails, the entire transaction is rolled back, preventing data inconsistencies.

[0096] (4) Performance monitoring and indicator logic

[0097] Using Prometheus's metrics library, we can monitor each stage of data writing in both the P2C Server and the writer. This records metrics such as processing time for each request, database write latency, and database query time. By exposing these metrics to Prometheus, we can monitor and optimize service performance in real time, and promptly identify performance bottlenecks and issues.

[0098] (5) Resource recycling logic

[0099] When shutting down, P2CServer ensures timely resource release, closes database connections, cleans up the worker pool, and releases coroutines, guaranteeing a proper and graceful service shutdown. This avoids resource leaks and unnecessary resource consumption, ensuring that the system releases all occupied resources upon shutdown.

[0100] (6) Optimize read performance logic

[0101] The p2cReader optimizes query statements, using appropriate indexes and optimizer hints to reduce the amount of data filtered and speed up queries. ClickHouse's PREWHERE and SAMPLEBY clauses are used to further optimize query performance. When reading large amounts of data, pagination queries are added to return query results in batches, avoiding excessive memory usage caused by loading all data at once.

[0102] (7) Error handling and logging logic

[0103] Add error handling mechanisms to the code to handle database connection errors, query errors, etc., and return error information to the upper layer for debugging and handling of exceptions. Enhance logging to record service operation information and error messages for easier monitoring and troubleshooting. Use log levels appropriately to ensure that log information is detailed and easy to understand, enabling quick location and resolution of problems.

[0104] (8) Combined alarm logic

[0105] 1) DiscoverTask: When operations personnel configure multi-rule combined alarms or single-rule ordinary alarms through the management interface, the relevant data will be stored in a configuration table in the database. The "DiscoverTask" is a scheduled task that queries the configuration table to check if any new combined alarms have been issued. When the "DiscoverTask" periodically polls the configuration table and finds a combined alarm configuration that exists in the database but is not in the program's memory, it is considered a new alarm rule. The "DiscoverTask" will create a new scheduled task named "Processing Process" based on the configuration ID. Each combined alarm or ordinary alarm has a specific "Processing Process" to perform subsequent processing logic.

[0106] 2) Process Task: A Process Task is responsible for the processing logic of one alarm. Its main functions are to control the reading and updating of alarm rules and the creation of corresponding coroutine subtasks. The Process Task queries the configuration details table for specific alarm rules based on the configuration ID. Each configuration detail contains one or more alarm rules. The Process Task creates a corresponding number of coroutine subtasks based on the number of alarm rules, and performs the specific calculation logic in the subtasks. At the same time, the Process Task queries the configuration details table periodically, and when the configuration is updated, it will be passed to the coroutine subtasks.

[0107] 3) Goroutine Subtasks: Within the "processing process," a corresponding number of goroutine subtasks are created based on the number of alarm rules. Each goroutine subtask is responsible for processing one alarm rule. These goroutine subtasks execute in parallel, which can improve processing efficiency and avoid alarm backlog when dealing with large volumes of data.

[0108] 4) Subtask Processing Logic: Each "coroutine subtask" further analyzes the content of the alarm rules and queries the indicator rule table for the specific indicator information to be collected based on the rules. For example, if there is an alarm rule that "issue an alarm when Tomcat thread saturation reaches 80%", although this is one rule, it requires two indicator data to calculate the result: one is the current number of Tomcat threads, and the other is the maximum number of threads set for Tomcat. These corresponding rules are recorded in the rule table. The subtask will obtain the required indicator data "current busy number of Tomcat threads" and "maximum number of Tomcat threads" from the indicator rule table, then query the data in ClickHouse for calculation, compare the calculated saturation with the threshold, filter out the data that meets the threshold, and then "pass on the result".

[0109] 5) Result Passing: The computation results of each subtask are passed to the "processing process" through a channel, and the processing process performs a final result merging and comparison.

[0110] 6) Result Merging and Comparison: The main task collects the results of all subtasks from the channel and determines whether to trigger an alarm based on the merging and comparison strategy. For example, if a logical AND merging strategy is used, all subtask results must meet the alarm conditions before proceeding to the next alarm triggering step; otherwise, the process ends.

[0111] 7) Alert Triggering: If the conditions of all alarm rules meet the merging comparison strategy, that is, multiple alarm indicators in combination reach the threshold at the same time, then an alarm notification is triggered.

[0112] 8) Alert Notification: After an alarm is triggered, the system will notify the administrator or relevant personnel through pre-set alarm notification channels (such as SMS, email, internal company app, etc.).

[0113] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0114] Based on the same inventive concept, this application also provides a data processing apparatus for implementing the data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data processing apparatus embodiments provided below can be found in the limitations of the data processing method described above, and will not be repeated here.

[0115] In one exemplary embodiment, such as Figure 3 As shown, a data processing device 300 is provided, including: a receiving module 301, a decoding module 302, a first acquisition module 303, and a second acquisition module 304, wherein:

[0116] The receiving module 301 is used to receive a data write request sent by the client, input the data to be written in the data write request to the data decompression component, and output the decompressed data to be written.

[0117] The decoding module 302 is used to decode the decompressed data to be written based on the data serialization protocol to obtain structured time-series data; wherein the data to be written in the time-series data is sorted according to the acquisition time.

[0118] The first acquisition module 303 is used to acquire a quadruple of any data to be written in the time series data; wherein, the quadruple includes the data to be written, the identifier of the indicator object to which the data to be written belongs, and the label and acquisition time of the data to be written.

[0119] The second acquisition module 304 is used to acquire the write statement and the cooperative program, and based on the write statement and the cooperative program, concurrently write all the corresponding quadruples of the time series data to the memory buffer of the writer, and store the data in the memory buffer to the columnar database.

[0120] In some embodiments, the writer is connected to the columnar database via a database connection pool; the second acquisition module 304 is further configured to write the data cached in the memory buffer to the columnar database via the database connection pool according to a preset period.

[0121] In some embodiments, the data processing device 300 is specifically configured to receive a data read request sent by the client, input the request data in the data read request to the data decompression component, and output the decompressed request data; decode the decompressed request data based on the data serialization protocol to obtain at least one data query request instruction, and construct a data query statement based on the data query request instruction; read the corresponding result data of the data read request from the columnar database based on the data query statement, and encode the result data based on the data serialization protocol to obtain encoded result data; input the encoded result data to the data compression component, output the compressed result data, and return the compressed result data to the client.

[0122] In some embodiments, the data processing device 300 is further configured to: determine, for the business attributes of the data queried by the data query statement, a data column in the columnar database storing business attribute data; based on the data column, filter out the data row containing the result data from the columnar database; and filter out the result data from the data row.

[0123] In some embodiments, the data processing device 300 is further configured to read the result data corresponding to the data reading request from the columnar database based on the data query statement, including: dividing the data stored in the columnar database into multiple uniform initial sampling groups based on a preset sampling key; determining a target sampling group from the multiple initial sampling groups according to a preset ratio; integrating the data in the target sampling group and scaling the integrated data based on the preset ratio; and determining the scaled data as the result data.

[0124] In some embodiments, the data processing device 300 is further configured to acquire a plurality of preset alarm rules and determine the triggering condition of any one of the preset alarm rules; wherein the triggering condition indicates that the indicator data of at least one indicator object in the business container meets the preset condition; acquire the indicator data of the indicator objects in the business container in real time; and issue an alarm based on the preset alarm rule when the indicator data of all indicator objects corresponding to the preset alarm rule meet the corresponding preset condition.

[0125] Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0126] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a data processing method.

[0127] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0128] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0129] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0130] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0131] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0132] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0133] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0134] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, The method includes: Receives a data write request sent by the client, inputs the data to be written in the data write request to the data decompression component, and outputs the decompressed data to be written. The decompressed data to be written is decoded based on a data serialization protocol to obtain structured time-series data; wherein the data to be written in the time-series data is sorted according to the acquisition time. Obtain a quadruple of any data to be written from the time series data; wherein, the quadruple includes the data to be written, the identifier of the indicator object to which the data to be written belongs, and the label and collection time of the data to be written; Obtain the write statement and the cooperative procedure. Based on the write statement and the cooperative procedure, concurrently write all the corresponding quadruples of the time series data to the memory buffer of the writer, and store the data in the memory buffer into the columnar database.

2. The method according to claim 1, characterized in that, The writer is connected to the columnar database via a database connection pool; the step of storing the data in the memory buffer into the columnar database includes: According to a preset period, the data cached in the memory buffer is written to the columnar database through the database connection pool.

3. The method according to claim 1, characterized in that, The method further includes: The system receives a data read request sent by the client, inputs the request data in the data read request into the data decompression component, and outputs the decompressed request data. The decompressed request data is decoded based on the data serialization protocol to obtain at least one data query request instruction, and a data query statement is constructed based on the data query request instruction. Based on the data query statement, the result data corresponding to the data read request is read from the columnar database, and the result data is encoded based on the data serialization protocol to obtain the encoded result data; The encoded result data is input into the data compression component, the compressed result data is output, and the compressed result data is returned to the client.

4. The method according to claim 3, characterized in that, The step of reading the result data corresponding to the data read request from the columnar database based on the data query statement includes: For the business attributes of the data queried by the data query statement, determine the data columns in the columnar database that store the business attribute data; Based on the data column, the data row containing the result data is filtered out from the columnar database; The result data is filtered from the data rows.

5. The method according to claim 3, characterized in that, The step of reading the result data corresponding to the data read request from the columnar database based on the data query statement includes: Based on a preset sampling key, the data stored in the columnar database is divided into multiple uniform initial sampling groups; The target sampling group is determined from multiple initial sampling groups according to a preset ratio; The data in the target sampling group are integrated, and the integrated data is scaled based on the preset ratio; The scaled data is determined as the resulting data.

6. The method according to claim 1, characterized in that, The method further includes: Multiple preset alarm rules are obtained, and the triggering condition of any one preset alarm rule is determined; wherein, the triggering condition indicates that the indicator data of at least one indicator object in the business container meets the preset condition; Real-time acquisition of metric data from the metric objects in the business container; For any given preset alarm rule, if the indicator data of all the corresponding indicator objects meet the corresponding preset conditions, an alarm is issued based on the preset alarm rule.

7. A data processing apparatus, characterized in that, The device includes: The receiving module is used to receive data write requests sent by the client, input the data to be written in the data write request to the data decompression component, and output the decompressed data to be written. The decoding module is used to decode the decompressed data to be written based on a data serialization protocol to obtain structured time-series data; wherein the data to be written in the time-series data is sorted according to the acquisition time. The first acquisition module is used to acquire a quadruple of any data to be written in the time series data; wherein, the quadruple includes the data to be written, the identifier of the indicator object to which the data to be written belongs, and the label and acquisition time of the data to be written; The second acquisition module is used to acquire the write statement and the cooperative program, and based on the write statement and the cooperative program, concurrently write all the corresponding quadruples of the time series data to the memory buffer of the writer, and store the data in the memory buffer to the columnar database.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images