Data processing method and system

Abstract

The embodiment of the invention provides a data processing method and system, and the method comprises the steps: determining a detection task of a target application, and obtaining to-be-detected data corresponding to the detection task; according to the method, the data identifier and the aggregation field are configured for the to-be-detected data, the target static data table is constructed based on the to-be-detected data, the data identifier and the aggregation field, the static data corresponding to each application attribute in the target static data table has a fixed storage position, and data sorting does not need to be performed again during subsequent data aggregation. The data processing overhead is reduced, and the data processing efficiency is improved. And constructing a data aggregation tree based on the detection items in the target static data table and the attribute data of the detection items, and executing a detection task based on the data aggregation tree. The data aggregation in the target static data table is realized through the data aggregation tree, the flexibility of data aggregation is improved, and hierarchical data aggregation is realized. Execution of subsequent detection tasks is guided, and the execution process of the detection tasks is optimized.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to data processing methods and systems. Background Technology

[0002] Monitoring and performance tracing are crucial during software development and application operation. Observable metrics generated during software development or application operation are typically quantifiable data, expressed numerically, and provide key information about system performance, health, and behavior. Aggregating this information can be used for application testing and performance monitoring of functional modules during software development.

[0003] In existing technologies, the open-source OpenTelemetry (OTel) SKD metric aggregation logic is commonly used for data aggregation and performance monitoring during software development and application operation. However, the core data structure of this data aggregation method is a hash table with attribute sets as keys. When generating the same hash for attribute sets with the same key-value pairs, it is inevitable to sort the attribute sets by key, increasing data processing costs and reducing efficiency. Furthermore, this metric aggregation method isolates different metrics, which is detrimental to subsequent metric aggregation. Therefore, a more efficient data processing method is urgently needed to address these problems. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising: Determine the detection task for the target application and obtain the detection data corresponding to the detection task; Configure a data identifier and an aggregation field for the data to be detected, and construct a target static data table based on the data to be detected, the data identifier, and the aggregation field; A data aggregation tree is constructed based on the detection items and attribute data of the detection items in the target static data table, and the detection task is executed based on the data aggregation tree.

[0006] According to a second aspect of the embodiments of this specification, a data processing system is provided, including a server and a client; The client is used to generate an application detection request based on the target application and send the application detection request to the server. The server is configured to determine the detection task of the target application and obtain the data to be detected corresponding to the detection task; configure data identifiers and aggregation fields for the data to be detected, and construct a target static data table based on the data to be detected, the data identifiers and the aggregation fields; construct a data aggregation tree based on the detection items and attribute data of the detection items in the target static data table, and execute the detection task based on the data aggregation tree; and send the task execution result to the client.

[0007] According to a third aspect of the embodiments of this specification, a data processing apparatus is provided, comprising: The determination module is configured to determine the detection task of the target application and obtain the detection data corresponding to the detection task; The configuration module is configured to configure a data identifier and an aggregation field for the data to be detected, and to construct a target static data table based on the data to be detected, the data identifier, and the aggregation field; The execution module is configured to construct a data aggregation tree based on the detection items in the target static data table and the attribute data of the detection items, and to execute the detection task based on the data aggregation tree.

[0008] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0009] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method described above.

[0010] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the data processing method described above.

[0011] One embodiment of this specification describes a method for detecting a target application. This involves determining the detection task for the target application and acquiring the corresponding data to be detected. Data identifiers and aggregation fields are configured for the data to be detected. A target static data table is constructed based on the data to be detected, the data identifiers, and the aggregation fields. Each application attribute in the target static data table has a fixed storage location for its static data, eliminating the need for re-sorting the data during subsequent data aggregation, thus reducing data processing overhead and improving efficiency. A data aggregation tree is constructed based on the detection items and their attribute data in the target static data table, and the detection task is executed based on this data aggregation tree. Using the data aggregation tree to aggregate data in the target static data table improves the flexibility of data aggregation and enables hierarchical data aggregation. This guides the execution of subsequent detection tasks and optimizes the execution process. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the processing procedure of a data processing method provided in one embodiment of this specification; Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification. Figure 4 This is a data aggregation diagram illustrating a data processing method provided in one embodiment of this specification; Figure 5 This is a global static representation of a data processing method provided in one embodiment of this specification; Figure 6 This is a schematic diagram of the structure of a data processing system provided in one embodiment of this specification; Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in one embodiment of this specification; Figure 8 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0013] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0014] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0016] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0017] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0018] Metrics: Data points used to quantify system performance or behavior, such as response time, number of requests, or error rate.

[0019] Attributes: Key-value pairs that provide additional context or detailed information when recording metrics, tracking links, etc., to enhance the interpretability and relevance of data.

[0020] Span: An important concept in distributed tracing, it generally represents the start and end of an operation, which can be a database query, an RPC call, or any operation with a defined start and end time.

[0021] Figure 1 This is a schematic diagram illustrating the processing procedure of a data processing method provided in one embodiment of this specification; as shown below. Figure 1As shown, when detecting a target application, the detection task for the target application is determined, and the corresponding detection data is obtained. The detection data contains at least one attribute data of the target application. A data identifier and aggregation field are configured for each attribute data in the detection data, and a target static data table is constructed based on the detection data, data identifier, and aggregation field. The static data corresponding to each application attribute in the target static data table has a fixed storage location, eliminating the need for re-sorting the data during subsequent data aggregation, reducing data processing overhead and improving data processing efficiency. A data aggregation tree is constructed based on the detection items and attribute data of the detection items in the target static data table, and the detection task is executed based on the data aggregation tree. The data aggregation tree is used to aggregate data in the target static data table, improving the flexibility of data aggregation and enabling hierarchical data aggregation. This guides the execution of subsequent detection tasks and optimizes the execution process of the detection tasks.

[0022] This specification provides a data processing method, and also relates to a data processing system, a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0023] See Figure 2 , Figure 2 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0024] Step 202: Determine the detection task of the target application and obtain the detection data corresponding to the detection task.

[0025] Specifically, the target application can be either application software in the development phase or application software in the running state. The detection task is used to perform performance testing on the target application through application metric aggregation during the development or operation of the target application. The data to be processed refers to the performance metrics associated with the detection task during the development or operation of the target application. The data to be processed can be a set of attributes of the target application's detection task, or it can be the target application's metric data or link tracing data. The detection task can be an application detection task corresponding to a target detection cycle during the development or operation of the target application, used to detect the development or operation status of the target application within the target detection cycle.

[0026] Based on this, during the development or operation of the target application, the detection tasks for the target application within a target detection cycle are determined. The data to be detected corresponding to the detection tasks is obtained, and subsequently, metrics can be aggregated based on this data before executing the detection tasks.

[0027] Furthermore, detection tasks can be stored in a task queue. By reading a detection task from the task queue, the detection task can be executed subsequently. The specific implementation is as follows: Determine the task queue corresponding to the target application; read the detection task of the target application from the task queue.

[0028] Specifically, the task queue can be a lock-free queue, and the task queue can act as a "message queue" to provide message queue functionality for traffic smoothing and decoupling.

[0029] Based on this, the task queue corresponding to the target application is determined. The task queue adopts a lock-free queue. When executing the detection task in the task queue, the detection task of the target application can be read from the task queue first, and then the detection task can be executed based on the data to be detected.

[0030] For example, the target application can be an application in the software development process. During application development, multiple detection tasks can be stored as events to be processed in a lock-free queue. Events are read from the lock-free queue to determine the corresponding detection task.

[0031] In summary, by storing the detection tasks to be executed in a task queue, the detection tasks can be read from the task queue when they are executed later. The task queue can smooth out traffic spikes and decouple traffic.

[0032] Step 204: Configure data identifiers and aggregation fields for the data to be detected, and construct a target static data table based on the data to be detected, the data identifiers, and the aggregation fields.

[0033] Specifically, after determining the detection task of the target application and obtaining the corresponding data to be detected, data identifiers and aggregation fields can be configured for the data to be detected. A target static data table is then constructed based on the data to be detected, data identifiers, and aggregation fields. The data identifier is used to identify the data to be detected; it can be an identifier or a sequence number, used to distinguish each piece of sub-data within the data to be detected. The aggregation field represents the aggregation pattern of the data to be detected, characterizing its divergence level (degree of divergence). The field value of the aggregation field can be a specified numerical value, the magnitude of which indicates the aggregation level of the data to be detected. The aggregation field can also indicate whether the data to be detected needs to participate in subsequent logical calculations. The target static data table can be a global static data table, storing the data to be detected, its data identifier, and aggregation fields. The target static data table can store the data to be detected as a one-dimensional array.

[0034] Based on this, after determining the detection task of the target application and obtaining the corresponding data to be detected, data identifiers and aggregation fields are configured for the data to be detected. A target static data table is then constructed based on the data to be detected, the data identifiers, and the aggregation fields. If the data to be detected contains multiple sub-data items, a data identifier and aggregation field are configured for each sub-data item. The target static data table is then constructed based on the sub-data items and their corresponding data identifiers and aggregation fields.

[0035] Furthermore, considering that the data to be detected exists in the form of key-value pairs, and that the data is stored as a two-dimensional attribute graph in this form, in order to reduce computational overhead, the attribute graph can be converted into a one-dimensional array to construct an initial static data table. The specific implementation is as follows: Configure the data identifier for the data to be detected, and construct an initial static data table based on the data to be detected and the data identifier; configure aggregation fields for the data records in the initial static data table to obtain the target static data table.

[0036] Specifically, the initial static data table is a one-dimensional array containing the data to be detected and data identifiers. In the initial static data table, the data to be detected is used as the attribute data of the target application. When there are multiple attribute data, different data identifiers are configured for each attribute data in turn to distinguish each attribute data.

[0037] Based on this, a data identifier is configured for the data to be detected. When the data to be detected contains multiple sub-data items, a different data identifier is configured for each sub-data item. An initial static data table is constructed based on the data to be detected and the data identifiers. Each data item in the initial static data table is treated as a data record, and an aggregation field is configured for each data record in the initial static data table to obtain the target static data table.

[0038] Continuing with the previous example, when the detection data is stored as a two-dimensional attribute graph, the detection data contains multiple sub-data entries: Key0 – attribute set 1, Key1 – attribute set 2, and Key2 – attribute set 3. By configuring a data identifier for each Key, an initial static data table consisting of 1 – Key0, 2 – Key1, and 3 – Key2 can be obtained. Then, an aggregation field is configured for each data record in the initial static data table: 1 – Key0 – aggregation field 0, 2 – Key1 – aggregation field 2, and 3 – Key2 – aggregation field 1, generating the target static data table.

[0039] In summary, we first configure data identifiers for the data to be detected and build an initial static data table. Then, we configure aggregation fields for the data records in the initial static data table to obtain the target static data table. This achieves dimensionality reduction of the data to be detected and reduces subsequent computational overhead.

[0040] Furthermore, when configuring aggregation fields for data records in the initial static data table, the aggregation mode of the data records needs to be considered. The specific implementation is as follows: Determine the aggregation pattern of the data records in the initial static data table; configure the aggregation field for the data records according to the aggregation pattern to obtain the target static data table.

[0041] Specifically, aggregation patterns can reflect the divergence level (degree of divergence) of the fields corresponding to data records. Aggregation patterns can be distinguished according to their levels. The higher the level of the aggregation pattern, the lower the degree of divergence of the data record corresponding to that aggregation pattern.

[0042] Based on this, data records are read from the initial static data table, the aggregation mode of the data records is determined according to the degree of divergence of the data records in the initial static data table, and the aggregation field is configured for the data records according to the aggregation level or aggregation hierarchy of the aggregation mode to obtain the target static data table.

[0043] Following the previous example, after obtaining the initial static data table consisting of 1--Key0, 2--Key1, and 3--Key2, determine the divergence level corresponding to each data record in the initial static data table, and configure the aggregation field for the data record according to the divergence level. The data record 1--Key0 has a high degree of divergence, so it is determined to be the first-level aggregation, with 0 representing the aggregation level, and 0 is added to the aggregation field corresponding to 1--Key0; the data record 2--Key1 has a low degree of divergence, so it is determined to be the third-level aggregation, with 2 representing the aggregation level, and 2 is added to the aggregation field corresponding to 2--Key1; the data record 3--Key2 has a medium degree of divergence, so it is determined to be the second-level aggregation, with 1 representing the aggregation level, and 1 is added to the aggregation field corresponding to 3--Key2.

[0044] In summary, by configuring aggregation fields for data records according to the aggregation level or aggregation hierarchy of the aggregation mode, the target static data table can be obtained, reducing the IO overhead of subsequent data aggregation.

[0045] Step 206: Construct a data aggregation tree based on the detection items and attribute data of the detection items in the target static data table, and execute the detection task based on the data aggregation tree.

[0046] Specifically, after configuring data identifiers and aggregation fields for the data to be detected, and constructing a target static data table based on the data to be detected, data identifiers, and aggregation fields, a data aggregation tree can be constructed based on the detection items and attribute data of the detection items in the target static data table. The detection task can then be executed based on the data aggregation tree. Here, a detection item refers to a row of data in the target static data table, i.e., a record. The attribute data of a detection item refers to its node level and node attribute data in the data aggregation tree; node attribute data can be the size of the node. A data aggregation tree is a tree diagram obtained by aggregating the detection items in the target static data table in a tree structure. Each level in the data aggregation tree corresponds to a detection item with a different degree of divergence. The higher the degree of divergence of a detection item, the higher its node level in the data aggregation tree.

[0047] Based on this, after configuring data identifiers and aggregation fields for the data to be detected, and constructing a target static data table based on the data to be detected, data identifiers, and aggregation fields, detection items are extracted from the target static data table. The number of attributes for each detection item is determined, the node level corresponding to the detection item is determined based on the attribute data, the node corresponding to the detection item is drawn, and a data aggregation tree is constructed. The detection task is then executed based on the data aggregation tree.

[0048] Furthermore, after determining the data to be tested, the test attribute data of the data to be tested can be determined through data callback, as specifically implemented as follows: The test attribute data associated with the data to be detected is determined through data callback; the detection guidance information of the data aggregation tree is generated based on the test attribute data.

[0049] Specifically, data callback refers to the data associated with the data to be tested, obtained using callback methods. Test attribute data refers to descriptive data indicating whether the data to be tested is divergent. Detection guidance information is used to guide the generation of the data aggregation tree; based on the detection guidance information, the detection items in the target static data table that need to be plotted as tree nodes can be determined.

[0050] Based on this, a data callback is performed on the data to be tested. The data callback determines the divergent fields associated with the data to be tested, and these divergent field data is used as test attribute data. Based on the test attribute data, detection guidance information for generating a data aggregation tree is generated to guide the generation of the data aggregation tree.

[0051] Continuing with the previous example, data callbacks can be used to examine divergent fields in the data to be tested. Based on these divergent fields, it can be determined whether other non-divergent attribute data needs to be obtained. Data callbacks can also be used to enrich the data to be tested. For example, if IP data has been collected using a probe, the callback method can be called to enrich the application information corresponding to the IP, such as the Kubernetes cluster and load balancer name.

[0052] In summary, the detection guidance information based on test attribute data to generate data aggregation trees is used to guide the generation of data aggregation trees and improve their usability.

[0053] Furthermore, when determining the attribute data of a detection item, the aggregated fields in the detection item can be referenced. The attribute data of the detection item can be determined based on the data in the aggregated fields, as specifically implemented as follows: Identify the detection items in the target static data table, and the aggregated field data contained in the detection items; determine the attribute data of the detection items based on the aggregated field data, and construct a data aggregation tree based on the detection items and the attribute data.

[0054] Specifically, the aggregated field data represents the aggregation level of the detection item, and can also reflect the node level of the tree nodes drawn when constructing the data aggregation tree later.

[0055] Based on this, the detection items in the target static data table are identified, along with the aggregate fields contained within each detection item. The aggregate field data of these aggregate fields is then extracted. The attribute data of the detection items is determined based on the values ​​of the aggregate field data; this attribute data represents the tree node level of the tree node. A data aggregation tree can then be constructed based on the detection items and the attribute data.

[0056] Continuing with the previous example, we determine the detection item 1 -- Key0 -- 0 in the target static data table. The aggregate field data is 0. Based on the aggregate field data 0, we can determine the higher tree node level corresponding to the attribute data of the detection item, that is, the first tree node level. We then proceed in this manner to determine the attribute data of each detection item in the target static data table and draw the data aggregation tree.

[0057] In summary, by determining the attribute data of the detection items based on the aggregated field data, the attribute data of the detection items can be accurately obtained, and then a data aggregation tree can be constructed.

[0058] Furthermore, when determining the attribute data of the detection items, the aggregation mode level corresponding to the aggregation field data can be determined first. Based on the aggregation mode level, the node hierarchy structure of the data aggregation tree can be determined. The specific implementation is as follows: Determine the aggregation mode level corresponding to the aggregation field data; determine the node structure level based on the aggregation mode level, and use the node structure level as the attribute data of the detection item.

[0059] Specifically, the aggregation mode level represents the divergence level (degree of divergence) of the detection item corresponding to the aggregation field. The aggregation mode level can correspond to the hierarchical structure of the data aggregation tree. The node hierarchy is the hierarchy of each tree level in the data aggregation tree.

[0060] Based on this, the aggregation mode level corresponding to the aggregation field data is determined, and the node structure level of the data aggregation tree to be drawn is determined according to the aggregation mode level. The node structure level is used as the attribute data of the detection item to guide the drawing of the data aggregation tree.

[0061] Continuing with the previous example, for detection item 1 (Key0-0), the aggregation field data is 0, so the aggregation level can be first-level aggregation. Therefore, the node structure level of detection item 1 (Key0-0) is a first-level tree node. For item 2 (Key1-2), the aggregation field data is 2, so the aggregation level can be third-level aggregation. Therefore, the node structure level of detection item 2 (Key1-2) is a third-level tree node. For item 3 (Key2-1), the aggregation field data is 1, so the aggregation level can be second-level aggregation. Therefore, the node structure level of detection item 3 (Key2-1) is a second-level tree node.

[0062] In summary, by determining the node structure hierarchy based on the aggregation mode level and using the node structure hierarchy as attribute data for the detection items, the node structure hierarchy can be referenced when drawing the data aggregation tree, thereby improving the drawing efficiency of the data aggregation tree.

[0063] Furthermore, after determining the tree nodes and node structure hierarchy, a data aggregation tree can be constructed based on the determined tree nodes and node structure hierarchy, as specifically implemented as follows: Extract detection data from the detection items and determine the tree nodes corresponding to the detection data; construct the data aggregation tree based on the tree nodes and the node structure hierarchy.

[0064] Specifically, the detection data can be application attribute data corresponding to the target application in the detection items, which is used to detect the target application.

[0065] Based on this, detection data is extracted from the detection items, and the corresponding tree nodes are determined. Each detection item corresponds to one tree node. A data aggregation tree is constructed based on the tree node corresponding to each detection item and the node structure hierarchy.

[0066] Using the previous example, for detection item 1--Key0--0, extract Key0 and its corresponding application attribute data as detection data.

[0067] For detection item -Key1--2, extract Key1 and its corresponding application attribute data as detection data. For detection item 3 -Key2--1, extract Key2 and its corresponding application attribute data as detection data. Draw tree nodes for each detection data, and combine the node structure hierarchy to draw a data aggregation tree.

[0068] In summary, a data aggregation tree is constructed based on tree nodes and node structure hierarchy to ensure that each detection item in the target static data table is drawn as a tree node in the data aggregation tree.

[0069] Furthermore, after drawing the data aggregation tree, the target application's metrics aggregation is complete. A detection task can then be performed based on the data aggregation tree to complete the detection of the target application. The specific implementation is as follows: Identify the target tree nodes contained in the data aggregation tree; read the node data of the target data nodes, and perform the detection task based on the node data.

[0070] Specifically, the target tree node can be any of the nodes in the data aggregation tree, or it can be a node within a single node level of the data aggregation tree. The node data is the data to be detected contained within the target data node, that is, the attribute data contained in the attribute data set of the target application.

[0071] Based on this, the target tree nodes contained in the data aggregation tree are determined. The node data of the target data nodes is read, and the detection task is performed based on the node data. When determining the target tree node, all tree nodes contained in the data aggregation tree can be used as the target tree node, or the target tree node can be determined layer by layer according to the node hierarchy of the data aggregation tree.

[0072] Following the previous example, we read the tree nodes of each level in the data aggregation tree, generate the node data corresponding to each level of the tree nodes, and send the node data to the server for processing to complete the detection of the target application.

[0073] In summary, by identifying target tree nodes in the data aggregation tree and extracting their node data, and then executing detection tasks based on this node data, the detection tasks can be performed at the node level of the data aggregation tree, thereby improving the efficiency of the detection tasks.

[0074] One embodiment of this specification describes a method for detecting a target application. This involves determining the detection task for the target application and acquiring the corresponding data to be detected. Data identifiers and aggregation fields are configured for the data to be detected. A target static data table is constructed based on the data to be detected, the data identifiers, and the aggregation fields. Each application attribute in the target static data table has a fixed storage location for its static data, eliminating the need for re-sorting the data during subsequent data aggregation, thus reducing data processing overhead and improving efficiency. A data aggregation tree is constructed based on the detection items and their attribute data in the target static data table, and the detection task is executed based on this data aggregation tree. Using the data aggregation tree to aggregate data in the target static data table improves the flexibility of data aggregation and enables hierarchical data aggregation. This guides the execution of subsequent detection tasks and optimizes the execution process.

[0075] The following is in conjunction with the appendix Figure 3 Taking the application of the data processing method provided in this specification in software development as an example, the data processing method will be further explained. Figure 3 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0076] Step 302: Read the pending events of the target software from the lock-free event queue and determine the data to be detected corresponding to the pending events.

[0077] In software development scenarios, monitoring the software development process and its operational status is becoming increasingly important. A high-performance method for aggregating and storing metrics is needed to improve monitoring efficiency.

[0078] like Figure 4 As shown, in terms of the solution architecture, a lock-free event queue is used as a "message queue" to smooth traffic spikes and decouple it. Several threads pull temporary, pending events from the kernel from the lock event queue and deliver them downstream. These events can be delivered to the span generation chain, the log chain, and the metrics chain. The metrics chain can selectively call callbacks (A callback, B callback) of the required aggregated attributes based on the configuration in the global static table. If only divergent fields need to be checked, it can be determined whether it is necessary to obtain other non-divergent attributes.

[0079] In practical applications, once the event to be processed (event 2) is read from the lock-free event queue, the data to be tested in the target software can be determined.

[0080] Step 304: Configure data identifiers for the data to be detected, and build an initial static data table based on the data to be detected and the data identifiers.

[0081] The aggregation of the data to be detected can be achieved using methods such as... Figure 5 This is achieved as shown.

[0082] The data to be tested is stored in key-value pairs as Key1-Attr1, Key2-Attr2, Key3-Attr3, Key4-Attr4, and Key5-Attr5. A global static data table (the initial static data table) can be constructed by configuring a data identifier for each specific piece of data. The global static data table stores 1-Key1, 2-Key2, 3-Key3, 4-Key4, and 5-Key5. Essentially, it encodes the attribute graph as an attribute array (Attr1, Attr2, Attr3, Attr4, Attr5).

[0083] Step 306: Determine the aggregation mode of the data records in the initial static data table, configure the aggregation fields for the data records according to the aggregation mode, and obtain the target static data table.

[0084] like Figure 4 As shown, based on the initial static data table, an agg field representing the aggregation mode is configured for each record in the table. Key1 is configured with -1, Key2 with 0, Key3 with 1, Key4 with 1, and Key5 with 2, thus obtaining the target static data table.

[0085] In this structure, -1 indicates that if it is known beforehand that a field will not diverge, it will not participate in any subsequent calculations. 0 represents the first level of aggregation (aggregation level 0), two fields with a value of 1 represent fields in the second level of aggregation (aggregation level 1), 2 represents the third level of aggregation (aggregation level 2), and so on. This data structure, on the one hand, allows for the reuse of duplicate attributes in nodes by placing fields with low divergence at lower levels in the aggregation tree. On the other hand, before reporting, the aggregated results can be reported to the server from a certain level by traversing the hierarchy based on the divergence of each level, reducing network I / O overhead.

[0086] Step 308: Determine the detection items in the target static data table, and the aggregated field data contained in the detection items.

[0087] Step 310: Determine the aggregation mode level corresponding to the aggregation field data, and determine the node structure level based on the aggregation mode level.

[0088] Step 312: Extract detection data from the detection items and determine the tree nodes corresponding to the detection data. Construct an aggregate tree based on the tree nodes and node structure hierarchy.

[0089] Construct an aggregation tree based on the aggregation hierarchy of the agg field in each record of the target static data table.

[0090] In practical applications, the measurement records corresponding to the aggregation tree can be: Total: 1; Status: 200; Total Errors: 0; Total Delays: 1.

[0091] Step 314: Determine the target tree node contained in the aggregation tree, read the node data of the target data node, and execute the pending events based on the node data.

[0092] Each node in the aggregation tree is used as a target tree node. The node data is extracted to form multiple messages and sent to the server.

[0093] In summary, the data processing method provided in one embodiment of this specification compresses the attribute set using a global static table. It uses a tree-structured multi-level Map as the core data structure for aggregation, supporting the configuration of aggregation information from either a global static or dynamic table. A lock-free buffer queue is used to asynchronously generate tracing links, metrics, and logs, bringing greater architectural flexibility to metric aggregation.

[0094] Corresponding to the above method embodiments, this specification also provides data processing system embodiments. Figure 6 A schematic diagram of the structure of a data processing system according to one embodiment of this specification is shown. Figure 6 As shown, the data processing system 600 includes a server 620 and a client 610. The client 610 is used to generate an application detection request based on the target application and send the application detection request to the server 620. The server 620 is used to determine the detection task of the target application and obtain the data to be detected corresponding to the detection task; configure data identifiers and aggregation fields for the data to be detected, and construct a target static data table based on the data to be detected, the data identifiers, and the aggregation fields; construct a data aggregation tree based on the detection items and attribute data of the detection items in the target static data table, and execute the detection task based on the data aggregation tree; and send the task execution result to the client 610.

[0095] In practical applications, when a client has a detection requirement for a target application, it can generate an application detection request based on the target application and send it to the server. The server then executes the corresponding detection task for the target application to perform the detection. When detecting the target application, the server determines the detection task and obtains the corresponding data to be detected. This data contains at least one attribute of the target application. A data identifier and aggregation field are configured for each attribute in the data to be detected, and a target static data table is constructed based on the data to be detected, the data identifier, and the aggregation field. The static data corresponding to each application attribute in the target static data table has a fixed storage location, eliminating the need for re-sorting the data during subsequent data aggregation, thus reducing data processing overhead and improving efficiency. A data aggregation tree is constructed based on the detection items and their attribute data in the target static data table, and the detection task is executed based on this tree. The task execution results are then sent to the client.

[0096] The data processing system provided in this manual, when detecting a target application, determines the detection task for the target application and obtains the corresponding data to be detected. It configures data identifiers and aggregation fields for the data to be detected, and constructs a target static data table based on the data to be detected, data identifiers, and aggregation fields. The static data corresponding to each application attribute in the target static data table has a fixed storage location, eliminating the need for re-sorting the data during subsequent data aggregation, thus reducing data processing overhead and improving data processing efficiency. It constructs a data aggregation tree based on the detection items and attribute data of the detection items in the target static data table, and executes the detection task based on the data aggregation tree. The data aggregation tree enables data aggregation in the target static data table, improving the flexibility of data aggregation and achieving hierarchical data aggregation. This guides the execution of subsequent detection tasks and optimizes the execution process of the detection tasks.

[0097] The above is an illustrative scheme of a data processing system according to this embodiment. It should be noted that the technical solution of this data processing system and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing system, please refer to the description of the technical solution of the data processing method described above.

[0098] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 7 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 7 As shown, the device includes: The determination module 702 is configured to determine the detection task of the target application and obtain the detection data corresponding to the detection task; Configuration module 704 is configured to configure a data identifier and an aggregation field for the data to be detected, and to construct a target static data table based on the data to be detected, the data identifier, and the aggregation field; The execution module 706 is configured to construct a data aggregation tree based on the detection items in the target static data table and the attribute data of the detection items, and to execute the detection task based on the data aggregation tree.

[0099] In an optional embodiment, the determining module 702 is further configured to: Determine the task queue corresponding to the target application; The detection task of the target application is read from the task queue.

[0100] In an optional embodiment, the execution module 706 is further configured to: The test attribute data associated with the data to be detected is determined through data callback; The detection guidance information for generating the data aggregation tree is generated based on the test attribute data.

[0101] In an optional embodiment, the configuration module 704 is further configured to: Configure the data identifier for the data to be detected, and construct an initial static data table based on the data to be detected and the data identifier; Configure aggregation fields for the data records in the initial static data table to obtain the target static data table.

[0102] In an optional embodiment, the configuration module 704 is further configured to: Determine the aggregation pattern of the data records in the initial static data table; Configure the aggregation field for the data record according to the aggregation mode to obtain the target static data table.

[0103] In an optional embodiment, the execution module 706 is further configured to: Identify the detection items in the target static data table, and the aggregated field data contained in the detection items; The attribute data of the detection item is determined based on the aggregated field data, and a data aggregation tree is constructed based on the detection item and the attribute data.

[0104] In an optional embodiment, the execution module 706 is further configured to: Determine the aggregation mode level corresponding to the aggregated field data; The node structure level is determined based on the aggregation mode level, and the node structure level is used as the attribute data of the detection item.

[0105] In an optional embodiment, the execution module 706 is further configured to: Extract detection data from the detection items and determine the tree node corresponding to the detection data; The data aggregation tree is constructed based on the tree nodes and the node structure hierarchy.

[0106] In an optional embodiment, the execution module 706 is further configured to: Determine the target tree nodes contained in the data aggregation tree; Read the node data of the target data node, and perform the detection task based on the node data.

[0107] The data processing system provided in this manual, when detecting a target application, determines the detection task for the target application and obtains the corresponding data to be detected. It configures data identifiers and aggregation fields for the data to be detected, and constructs a target static data table based on the data to be detected, data identifiers, and aggregation fields. The static data corresponding to each application attribute in the target static data table has a fixed storage location, eliminating the need for re-sorting the data during subsequent data aggregation, thus reducing data processing overhead and improving data processing efficiency. It constructs a data aggregation tree based on the detection items and attribute data of the detection items in the target static data table, and executes the detection task based on the data aggregation tree. The data aggregation tree enables data aggregation in the target static data table, improving the flexibility of data aggregation and achieving hierarchical data aggregation. This guides the execution of subsequent detection tasks and optimizes the execution process of the detection tasks.

[0108] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0109] Figure 8 A structural block diagram of a computing device 800 according to one embodiment of this specification is shown. The components of the computing device 800 include, but are not limited to, a memory 810 and a processor 820. The processor 820 is connected to the memory 810 via a bus 830, and a database 850 is used to store data.

[0110] The computing device 800 also includes an access device 840, which enables the computing device 800 to communicate via one or more networks 860. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 840 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0111] In one embodiment of this specification, the above-described components of the computing device 800 and Figure 8 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 8 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0112] The computing device 800 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 800 can also be a mobile or stationary server.

[0113] The processor 820 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data processing method.

[0114] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0115] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0116] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0117] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0118] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data processing method described above.

[0119] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0120] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0121] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0122] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0123] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method, comprising: Determine the detection task for the target application and obtain the detection data corresponding to the detection task; Configure a data identifier and an aggregation field for the data to be detected, and construct a target static data table based on the data to be detected, the data identifier, and the aggregation field; A data aggregation tree is constructed based on the detection items and attribute data of the detection items in the target static data table, and the detection task is executed based on the data aggregation tree.

2. The data processing method according to claim 1, wherein the detection task for determining the target application includes: Determine the task queue corresponding to the target application; The detection task of the target application is read from the task queue.

3. The data processing method according to claim 1, after obtaining the data to be detected corresponding to the detection task, further includes: The test attribute data associated with the data to be detected is determined through data callback; The detection guidance information for generating the data aggregation tree is generated based on the test attribute data.

4. The data processing method according to claim 1, wherein configuring a data identifier and an aggregation field for the data to be detected, and constructing a target static data table based on the data to be detected, the data identifier, and the aggregation field, comprises: Configure the data identifier for the data to be detected, and construct an initial static data table based on the data to be detected and the data identifier; Configure aggregation fields for the data records in the initial static data table to obtain the target static data table.

5. The data processing method according to claim 1, wherein configuring aggregation fields for data records in the initial static data table to obtain the target static data table includes: Determine the aggregation pattern of the data records in the initial static data table; Configure the aggregation field for the data record according to the aggregation mode to obtain the target static data table.

6. The data processing method according to claim 1, wherein constructing a data aggregation tree based on the detection items in the target static data table and the attribute data of the detection items comprises: Identify the detection items in the target static data table, and the aggregated field data contained in the detection items; The attribute data of the detection item is determined based on the aggregated field data, and a data aggregation tree is constructed based on the detection item and the attribute data.

7. The data processing method according to claim 1, wherein determining the attribute data of the detection item based on the aggregated field data includes: Determine the aggregation mode level corresponding to the aggregated field data; The node structure level is determined based on the aggregation mode level, and the node structure level is used as the attribute data of the detection item.

8. The data processing method according to claim 7, wherein constructing a data aggregation tree based on the detection item and the attribute data includes: Extract detection data from the detection items and determine the tree node corresponding to the detection data; The data aggregation tree is constructed based on the tree nodes and the node structure hierarchy.

9. The data processing method according to claim 1, wherein performing the detection task based on the data aggregation tree includes: Determine the target tree nodes contained in the data aggregation tree; Read the node data of the target data node, and perform the detection task based on the node data.

10. A data processing system, comprising a server and a client; The client is used to generate an application detection request based on the target application and send the application detection request to the server. The server is used to determine the detection task of the target application and obtain the data to be detected corresponding to the detection task; configure data identifiers and aggregation fields for the data to be detected, and construct a target static data table based on the data to be detected, the data identifiers and the aggregation fields; A data aggregation tree is constructed based on the detection items and attribute data of the detection items in the target static data table, and the detection task is executed based on the data aggregation tree; The task execution result is sent to the client.

11. A computing device, comprising: Memory and processor; The memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1-9.

12. A computer-readable storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-9.

13. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the steps of the method according to any one of claims 1-9.

Images