Data query method and device based on distributed link tracking, equipment and medium
By dynamically determining the storage location of tracing data in a distributed system and performing batch processing, the problem of low data processing efficiency in distributed computing architecture is solved, enabling efficient and real-time data querying.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2025-11-27
- Publication Date
- 2026-04-21
AI Technical Summary
In distributed computing architectures, existing data writing and querying schemes are inefficient when processing and responding to large-scale data, making it difficult to meet the requirements of real-time performance and accuracy, and making problem localization difficult.
By acquiring raw link tracing data from each service instance, the target topic is dynamically determined using data flow tables, and then processed and stored in the search engine through stream processing tasks. This optimizes the data storage structure and improves data processing efficiency and real-time performance.
It improves the efficiency and real-time performance of data processing, optimizes the data storage structure, effectively improves the efficiency of data query, and supports rapid location of system problems.
Smart Images
Figure CN121901334A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of distributed systems technology, and in particular to a data query method based on distributed link tracing, a data query device based on distributed link tracing, an electronic device, and a computer-readable storage medium. Background Technology
[0002] In the process of modern informatization, distributed computing architecture has become the standard in the fields of cloud computing and microservices. Under the distributed computing architecture, a single application is broken down into multiple independent services and modules, with a complex structure. Multiple services and modules communicate with each other to provide users with complete business capabilities.
[0003] However, distributed computing architecture also brings a significant problem: when a system encounters a problem, it becomes extremely difficult to locate the problem, because a single user request may involve hundreds to thousands of service instances, and when faced with massive amounts of real-time data, existing data writing and querying schemes are inefficient in processing and responding to large-scale data, making it difficult to meet the requirements of real-time performance and accuracy. Summary of the Invention
[0004] This application provides a data query method, apparatus, device, and medium based on distributed link tracing to solve or partially solve the problems of low real-time performance and low accuracy of data query in existing distributed systems.
[0005] This application discloses a data query method based on distributed link tracing, applied to a distributed system including multiple service instances. The method includes: Obtain the raw link tracing data for each of the service instances; Obtain the data flow table of the preset data topic, determine the target topic corresponding to the original link tracing data according to the data flow table, and send the original link tracing data to the target topic; The original link tracing data is processed by a stream processing task corresponding to the target topic to obtain the target link tracing data corresponding to the original link tracing data; The target link tracing data is stored in a preset search engine, which is used for data retrieval.
[0006] In some feasible implementations, the original link tracing data includes tracing information, and storing the target link tracing data in a preset search engine includes: The target link tracing data is stored in a preset search engine, and an index of the target link tracing data is constructed based on the tracing information.
[0007] In some feasible implementations, the method further includes: Receive user query requests for the search engine, the query requests including query information; Based on the query information and the index information, the link tracing data corresponding to the query request is retrieved from the search engine, and query results are generated based on the link tracing data.
[0008] In some feasible implementations, obtaining the raw link tracing data for each of the service instances includes: Embed a proxy service corresponding to each of the service instances in each of the service instances; The proxy service collects the original link tracing data corresponding to the business requests in the service instance.
[0009] In some feasible implementations, the proxy service is configured with a sampling strategy, and the collection of raw link tracing data corresponding to the service instance's business requests through the proxy service includes: The proxy service dynamically collects the original link tracing data corresponding to the business requests in the service instance according to the sampling strategy.
[0010] In some feasible implementations, the original link tracing data includes tracing information, the data flow table includes load information and routing rules for each data topic, and determining the target topic corresponding to the original link tracing data based on the data flow table includes: According to the routing rules, a first topic corresponding to the tracking information is selected from a plurality of data topics; Based on the load information, determine whether the first topic meets the preset conditions; If the first topic meets the preset conditions, the original link tracing data will be sent to the first topic.
[0011] In some feasible implementations, the method further includes: If the first topic does not meet the preset conditions, based on the load information, a second topic corresponding to the link tracing data is selected from the plurality of data topics, and the original link tracing data is sent to the second topic. or, Create a new data topic as a third topic corresponding to the link tracing data, and send the original link tracing data to the third topic.
[0012] In some feasible implementations, determining whether the first topic meets the preset conditions based on the load information includes: The load information of the first topic is compared with the load threshold corresponding to the first topic; If the load information is less than the load threshold, then the first topic is determined to meet the preset condition.
[0013] In some feasible implementations, the method of processing the original link tracing data through a stream processing task corresponding to the target topic to obtain the target link tracing data corresponding to the original link tracing data includes: The target topic corresponding to the stream processing task is monitored in real time through the stream processing task; In response to the detection of data updates in the target topic, the updated original link data in the target topic is retrieved and processed to obtain the target link tracing data corresponding to the original link tracing data.
[0014] In some feasible implementations, the distributed system further includes a link tracing gateway, and the method further includes: The raw link tracing data collected is sent to the link tracing gateway through the proxy service.
[0015] In some feasible implementations, the link tracing gateway stores a data flow table for the data topic. The step of determining the target topic corresponding to the original link tracing data based on the data flow table and sending the original link tracing data to the target topic includes: The link tracing gateway determines the target topic corresponding to the original link tracing data based on the data flow table, and sends the original link tracing data to the target topic.
[0016] In some feasible implementations, sending the collected raw link tracing data to the link tracing gateway through the proxy service includes: When the amount of raw link tracing data collected by the proxy service meets a preset threshold, the raw link tracing data is sent to the link tracing gateway.
[0017] This application also discloses a data query device based on distributed link tracing, applied to a distributed system, the distributed system including multiple service instances, the device comprising: The data acquisition module is used to acquire the original link tracing data of each of the service instances; The data distribution module is used to obtain a data flow table of a preset data topic, determine the target topic corresponding to the original link tracing data according to the data flow table, and send the original link tracing data to the target topic. The data processing module is used to process the original link tracing data through a stream processing task corresponding to the target topic to obtain the target link tracing data corresponding to the original link tracing data; The data writing module is used to store the target link tracing data to a preset search engine, which is used for data querying.
[0018] In some feasible implementations, the data writing module is specifically used for: The target link tracing data is stored in a preset search engine, and an index of the target link tracing data is constructed based on the tracing information.
[0019] In some feasible implementations, the apparatus further includes: The query module is used to receive a user's query request for the search engine, the query request including query information; based on the query information and the index information, it retrieves the link tracing data corresponding to the query request from the search engine, and generates query results based on the link tracing data.
[0020] In some feasible implementations, the data acquisition module includes: The deployment submodule is used to embed the proxy service corresponding to each of the service instances into each of the service instances; The data acquisition submodule is used to collect the original link tracing data corresponding to the business requests in the service instance through the proxy service.
[0021] In some feasible implementations, the proxy service is configured with a sampling strategy, and the data acquisition submodule is specifically used for: The proxy service dynamically collects the original link tracing data corresponding to the business requests in the service instance according to the sampling strategy.
[0022] In some feasible implementations, the raw link tracing data includes tracing information, the data flow table includes load information and routing rules for each data topic, and the data distribution module includes: The routing submodule is used to select a first topic corresponding to the tracking information from a plurality of data topics according to the routing rules; The judgment submodule is used to determine whether the first topic meets preset conditions based on the load information; The distribution submodule is used to send the original link tracing data to the first topic when the first topic meets preset conditions.
[0023] In some feasible implementations, the routing submodule is also used for: If the first topic does not meet the preset conditions, based on the load information, a second topic corresponding to the link tracing data is selected from the plurality of data topics, and the original link tracing data is sent to the second topic. or, Create a new data topic as a third topic corresponding to the link tracing data, and send the original link tracing data to the third topic.
[0024] In some feasible implementations, the judgment submodule is specifically used for: The load information of the first topic is compared with the load threshold corresponding to the first topic; If the load information is less than the load threshold, then the first topic is determined to meet the preset condition.
[0025] In some feasible implementations, the data processing module is specifically used for: The target topic corresponding to the stream processing task is monitored in real time through the stream processing task; In response to the detection of data updates in the target topic, the updated original link data in the target topic is retrieved and processed to obtain the target link tracing data corresponding to the original link tracing data.
[0026] In some feasible implementations, the distributed system further includes a link tracing gateway, and the data processing module is further used for: The raw link tracing data collected is sent to the link tracing gateway through the proxy service.
[0027] In some feasible implementations, the link tracing gateway stores a data flow table for the data topic, and the data processing module is further configured to: The link tracing gateway determines the target topic corresponding to the original link tracing data based on the data flow table, and sends the original link tracing data to the target topic.
[0028] In some feasible implementations, the data processing module is specifically used for: When the amount of raw link tracing data collected by the proxy service meets a preset threshold, the raw link tracing data is sent to the link tracing gateway.
[0029] This application also discloses an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method described in the embodiments of this application.
[0030] This application also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this application.
[0031] The embodiments of this application have the following advantages: In this embodiment, the method is applied to a distributed system comprising multiple service instances. The method involves acquiring raw link tracing data from each service instance; obtaining a data flow table for a preset data topic; determining a target topic corresponding to the raw link tracing data based on the data flow table; sending the raw link tracing data to the target topic; processing the raw link tracing data using a stream processing task corresponding to the target topic to obtain the target link tracing data; and storing the target link tracing data in a preset search engine for data querying. This embodiment dynamically determines the storage location of link tracing data through a data flow table and processes the data in batches using different stream processing tasks, improving data processing efficiency and real-time performance. Furthermore, storing the processed data in the corresponding search engine optimizes the data storage structure and effectively improves data query efficiency. Attached Figure Description
[0032] Figure 1 This is a flowchart illustrating the steps of a data query method based on distributed link tracing provided in this application embodiment; Figure 2 This is a schematic diagram illustrating the interaction scenario of distributed system components provided in the embodiments of this application; Figure 3 This is a schematic diagram of the link tracing data transmission process provided in the embodiments of this application; Figure 4 This is a schematic diagram of a data query scenario based on distributed link tracing provided in an embodiment of this application; Figure 5 This is a structural block diagram of a data query device based on distributed link tracing provided in the embodiments of this application; Figure 6 This is a block diagram of an electronic device provided in an embodiment of this application; Figure 7 This is a schematic diagram of a computer-readable storage medium provided in an embodiment of this application. Detailed Implementation
[0033] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0034] As an example, in the process of modern informatization, distributed computing architecture has become the standard in the fields of cloud computing and microservices. Under the distributed computing architecture, a single application is broken down into multiple independent services and modules, with a complex structure. Multiple services and modules communicate with each other to provide users with complete business capabilities.
[0035] However, distributed computing architecture also brings a significant problem: when a problem occurs in the system, it becomes extremely difficult to locate the problem, because a single user request may involve hundreds to thousands of service instances.
[0036] In related technologies, distributed tracing (DTLC) is used to record the flow path of user requests across multiple service instances, constructing a call chain to help trace the processing flow of each request. While existing DTLC technologies can accurately collect data flow information between services, the amount of DTLC data generated increases dramatically as the system expands. Existing data writing and querying schemes are inefficient when processing and responding to large-scale data, and cannot adequately meet the requirements for real-time performance and accuracy.
[0037] To address this, this application obtains raw link tracing data from each service instance; acquires a data flow table for a preset data topic, determines the target topic corresponding to the raw link tracing data based on the data flow table, and sends the raw link tracing data to the target topic; processes the raw link tracing data using a stream processing task corresponding to the target topic to obtain the target link tracing data corresponding to the raw link tracing data; and stores the target link tracing data in a preset search engine, which is used for data querying. This embodiment dynamically determines the storage location of link tracing data through a data flow table and processes the link tracing data in batches through different stream processing tasks, improving data processing efficiency and real-time performance. Furthermore, storing the processed data in the corresponding search engine optimizes the data storage structure and effectively improves data query efficiency. Reference Figure 1 The diagram illustrates a flowchart of a data query method based on distributed link tracing provided in an embodiment of this application, which may specifically include the following steps: Step 101: Obtain the original link tracing data for each of the service instances; In this embodiment, the application is applied to a distributed system comprising multiple service instances, where user business requests flow between these instances. First, raw link tracing data from each service instance is acquired. Specifically, raw link tracing data corresponding to business requests in each service instance is collected. This raw link tracing data can be understood as unprocessed link tracing information, which records the trajectory of the request within the distributed system. This embodiment lays the data foundation for subsequent data processing by automatically collecting massive amounts of raw link tracing data from the distributed system.
[0038] Step 102: Obtain the data flow table of the preset data topic, determine the target topic corresponding to the original link tracing data according to the data flow table, and send the original link tracing data to the target topic; In this embodiment, multiple data topics are pre-defined, such as multiple data topics in a Kafka cluster. A data topic can also be understood as a message queue, used to receive tracing data of a specific type or source and store the data through a partitioning mechanism. This application maintains a dynamic data flow table, which records the load status of each data topic in real time, such as partition utilization and data volume. Based on the data flow table, the most suitable target topic is dynamically determined for the original tracing data, and the original tracing data is sent to the corresponding data topic for storage. This application implements the distribution of tracing data based on the data flow table.
[0039] Step 103: Process the original link tracing data using a stream processing task corresponding to the target topic to obtain the target link tracing data corresponding to the original link tracing data; In this embodiment, each Kafka data topic is configured with a dedicated stream processing task, namely a Flink task. The Flink task subscribes to and consumes raw tracing data from the data topic in real time, performing tasks such as data cleaning and format conversion to obtain standardized tracing data. This embodiment utilizes multiple Flink tasks to process tracing data in real time, effectively improving data processing efficiency and real-time performance.
[0040] Step 104: Store the target link tracing data in a preset search engine, which is used for data querying.
[0041] In this embodiment, the processed link tracing data is stored in a search engine for data querying. The search engine is a distributed engine, such as Elasticsearch. The link tracing data stored in the search engine has a corresponding index, optimizing the data storage structure. The index allows for quick location of link tracing data, thereby effectively improving data query efficiency. This embodiment transforms massive amounts of link tracing data into a search engine for data querying with high efficiency, facilitating rapid identification and resolution of system problems.
[0042] In this embodiment, the method is applied to a distributed system comprising multiple service instances. The method involves acquiring raw link tracing data from each service instance; obtaining a data flow table for a preset data topic; determining a target topic corresponding to the raw link tracing data based on the data flow table; sending the raw link tracing data to the target topic; processing the raw link tracing data using a stream processing task corresponding to the target topic to obtain the target link tracing data; and storing the target link tracing data in a preset search engine for data querying. This embodiment dynamically determines the storage location of link tracing data through a data flow table and processes the data in batches using different stream processing tasks, improving data processing efficiency and real-time performance. Furthermore, storing the processed data in the corresponding search engine optimizes the data storage structure and effectively improves data query efficiency.
[0043] In some feasible implementations, the original link tracing data includes tracing information, and storing the target link tracing data in a preset search engine includes: The target link tracing data is stored in a preset search engine, and an index of the target link tracing data is constructed based on the tracing information.
[0044] In this embodiment, the link tracing data includes tracing information such as the identifier, service name, timestamp, and business attributes (such as order number). Therefore, when storing the processed link tracing data in the search engine, this application will construct a multi-dimensional index of the link tracing data based on the tracing information, thereby achieving unified and structured management of massive link tracing data through data indexing, which is beneficial for subsequent data querying.
[0045] In some feasible implementations, the method further includes: Receive user query requests for the search engine, the query requests including query information; Based on the query information and the index information, the link tracing data corresponding to the query request is retrieved from the search engine, and query results are generated based on the link tracing data.
[0046] In this embodiment, a user can send a query request to the query service through the user interface. The query request includes query information such as order number, link tracing data identifier, service name, and time range. Upon receiving the query request, the query service invokes the search engine and converts the query information into a query statement that the search engine can understand. The search engine performs a rapid retrieval based on the query statement and index information, locates several matching link tracing data points, and assembles the link tracing data to form a complete call chain as the query result, which is then returned to the user. This embodiment achieves rapid location of massive amounts of link tracing data through a pre-built index, effectively improving the efficiency of data querying.
[0047] In some feasible implementations, obtaining the raw link tracing data for each of the service instances includes: Embed a proxy service corresponding to each of the service instances in each of the service instances; The proxy service collects the original link tracing data corresponding to the business requests in the service instance.
[0048] In this embodiment, link tracing data of each service instance is collected through a proxy service. Specifically, an Agent (proxy service) is embedded in each service instance using OpenTelemetry (an open-source cloud infrastructure telemetry library). That is, the Agent and the service instance are deployed in the same runtime environment. The Agent then listens for and intercepts business requests within the service instance, automatically capturing link tracing data of these requests. This embodiment accurately collects link tracing data from the service instances through the Agent within each instance.
[0049] In some feasible implementations, the proxy service is configured with a sampling strategy, and the collection of raw link tracing data corresponding to the service instance's business requests through the proxy service includes: The proxy service dynamically collects the original link tracing data corresponding to the business requests in the service instance according to the sampling strategy.
[0050] In this embodiment, the proxy service is configured with a sampling strategy, which includes a fixed sampling rate or a dynamic sampling rate. For example, the sampling rate may be predefined by the user or dynamically adjusted based on the request volume of the service instance. As an example, the proxy service sampling strategy in service instance A is pre-set to collect link tracing data at a fixed 100% sampling rate. This means that all business requests involved in the service instance will have their corresponding link tracing data collected, ensuring that link data for all business requests is collected. As another example, the proxy service sampling strategy in service instance B is pre-set to dynamically adjust the sampling rate based on the request volume of service instance B. When the request volume is low, link tracing data is collected at a 100% sampling rate; when the request volume is high, link tracing data is collected at a 50% sampling rate. This means that some business requests require data collection, while others do not. This embodiment provides a dynamic sampling mechanism by setting a sampling strategy for the proxy service, improving the flexibility and practicality of data collection.
[0051] In some feasible implementations, the original link tracing data includes tracing information, the data flow table includes load information and routing rules for each data topic, and determining the target topic corresponding to the original link tracing data based on the data flow table includes: According to the routing rules, a first topic corresponding to the tracking information is selected from a plurality of data topics; Based on the load information, determine whether the first topic meets the preset conditions; If the first topic meets the preset conditions, the original link tracing data will be sent to the first topic.
[0052] In this embodiment, the link tracing data includes tracing information such as the identifier, service name, timestamp, and business attributes (e.g., order number). The data flow table not only records the load of each data topic but also includes pre-set routing rules. Therefore, in this embodiment, the most suitable, or optimal, data topic for the link tracing data is dynamically determined by combining the tracing information and the data flow table. Specifically, firstly, the first topic corresponding to the link tracing data is determined based on the routing rules and tracing information. For example, the link tracing data with order number A is pre-set to be stored in data topic 1. Further, it is determined whether to store the link tracing data in the first topic based on its load. If the load of the data topic meets the preset conditions, the link tracing data is stored in that data topic. This embodiment dynamically determines the storage location of the link tracing data by combining preset routing rules and real-time load status, ensuring the efficiency of data writing.
[0053] In some feasible implementations, the method further includes: If the first topic does not meet the preset conditions, based on the load information, a second topic corresponding to the link tracing data is selected from the plurality of data topics, and the original link tracing data is sent to the second topic. or, Create a new data topic as a third topic corresponding to the link tracing data, and send the original link tracing data to the third topic.
[0054] In this embodiment, if the data topic corresponding to the link tracing data determined according to the preset routing rules does not meet the preset conditions, a second topic for storing the link tracing data will be selected from other data topics based on the load of each data topic; alternatively, a new data topic will be created to store the link tracing data, thereby achieving dynamic data distribution and dynamic expansion of data topics; or, a new partition will be added to the first topic to store the link tracing data. This embodiment ensures optimal storage location for link tracing data by dynamically expanding data topics.
[0055] In some feasible implementations, determining whether the first topic meets the preset conditions based on the load information includes: The load information of the first topic is compared with the load threshold corresponding to the first topic; If the load information is less than the load threshold, then the first topic is determined to meet the preset condition.
[0056] In this embodiment, each data topic is pre-set with a corresponding load threshold. By comparing the real-time load of a data topic with the load threshold, it is determined whether the current data topic can receive link tracing data. It can be understood that if the current load of a data topic is less than the load threshold, it indicates that the current data topic is in a good load state and can still receive and store link tracing data; otherwise, it cannot, and dynamic scaling or data distribution to other data topics is required. This embodiment uses load thresholds to ensure that data is distributed to data topics with processing capabilities, avoiding data topic overload.
[0057] In some feasible implementations, the method of processing the original link tracing data through a stream processing task corresponding to the target topic to obtain the target link tracing data corresponding to the original link tracing data includes: The target topic corresponding to the stream processing task is monitored in real time through the stream processing task; In response to the detection of data updates in the target topic, the updated original link data in the target topic is retrieved and processed to obtain the target link tracing data corresponding to the original link tracing data.
[0058] In this embodiment, the stream processing task is essentially a consumer that subscribes to a corresponding data topic, thereby enabling it to perceive data updates to that topic in real time. Once new raw link tracing data is detected entering the data topic, it automatically pulls and consumes the updated raw link tracing data, transforming it into standardized target link tracing data. This embodiment achieves real-time processing of link tracing data through a stream processing task, improving data processing efficiency and real-time performance.
[0059] In some feasible implementations, the distributed system further includes a link tracing gateway, and the method further includes: The raw link tracing data collected is sent to the link tracing gateway through the proxy service.
[0060] In this embodiment, a trace-gateway is configured in the distributed system. The trace-gateway acts as an intermediary service between the agent service and the Kafka topic, centrally receiving trace-tracing data reported by each agent. By introducing a trace-gateway, this embodiment achieves centralized management of trace-tracing data, effectively improving the scalability and maintainability of the distributed system.
[0061] In some feasible implementations, the link tracing gateway stores a data flow table for the data topic. The step of determining the target topic corresponding to the original link tracing data based on the data flow table and sending the original link tracing data to the target topic includes: The link tracing gateway determines the target topic corresponding to the original link tracing data based on the data flow table, and sends the original link tracing data to the target topic.
[0062] In this embodiment, a data flow table is dynamically maintained by a link tracing gateway. When the link tracing gateway receives link tracing data reported by various service instances, it dynamically determines the data topic corresponding to the original link data based on the data flow table. This embodiment improves the efficiency and reliability of data distribution by centrally managing link tracing data through a link tracing gateway.
[0063] In some feasible implementations, sending the collected raw link tracing data to the link tracing gateway through the proxy service includes: When the amount of raw link tracing data collected by the proxy service meets a preset threshold, the raw link tracing data is sent to the link tracing gateway.
[0064] In this embodiment, each proxy service is pre-set with a threshold. When the amount of raw link data collected by the proxy service meets the preset threshold, the proxy service sends the collected raw link data to the link tracing gateway in batches, thereby reducing the number of interactions between the proxy service and the link tracing gateway and improving the performance of the distributed system.
[0065] In some feasible implementations, when writing target link tracing data to the Elasticsearch search engine, the Elasticsearch search engine determines the distribution of the target link tracing data according to the routing rules in the data flow table. That is, it dynamically adjusts the storage location of the link tracing data through the data flow table to ensure that specific link tracing data is written to the same shard in Elasticsearch.
[0066] As an example, when writing to Elasticsearch, specific fields (such as service names) in the link tracing data are extracted as routing keys according to the routing rules in the data flow table. The shard to which the data should be written is calculated using a hash algorithm, thereby ensuring that data with the same service name is written to the same shard.
[0067] In this embodiment, the original link tracing data of each service instance is obtained; a data flow table of a preset data topic is obtained, and a target topic corresponding to the original link tracing data is determined according to the data flow table, and the original link tracing data is sent to the target topic; the original link tracing data is processed by a stream processing task corresponding to the target topic to obtain the target link tracing data corresponding to the original link tracing data; the target link tracing data is stored in a preset search engine, which is used for data querying. This embodiment dynamically determines the storage location of the link tracing data through a data flow table and processes the link tracing data in batches through different stream processing tasks, improving the efficiency and real-time performance of data processing. Furthermore, storing the processed data in the corresponding search engine optimizes the data storage structure and effectively improves the efficiency of data querying.
[0068] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the following examples are provided for illustrative purposes: As an example, refer to Figure 2 , Figure 3 and Figure 4 This application provides a data query method based on distributed link tracing, which specifically includes the following steps: S1. In the distributed system, each service instance contains multiple applications and is embedded with an Agent (proxy service). The Agent is responsible for collecting the raw link tracing data of all applications on the service instance and sending the collected raw link tracing data to the Trace-Gateway (link tracing gateway). When S2 and Trace-Gateway receive the link tracing data reported by each Agent, they will dynamically determine the target topic (i.e., Topic) corresponding to the data based on the data flow table maintained internally for Kafka clusters (Kafka-Cluster), and send the data to that topic. When an S3 or Kafka cluster receives data, it partitions the data and stores it in the corresponding topic. In S4, multiple Flink tasks are deployed in the Flink cluster. Each Flink task subscribes to a corresponding Kafka topic. The Flink task will pull and consume the tracing data in the subscribed topic in real time, and write the consumed tracing data into the Elasticsearch search engine. When the S5 and Elasticsearch search engines receive data, they use a hash algorithm to calculate the data stream (i.e., DataStream) containing the tracing data based on the tracing information, and then create an index for querying. S6. Users initiate query requests to the query service (Logging-query) through the Starlink platform Web (i.e., the user interface of the distributed system); S7. When the query service receives a query request from the front end, it will call the Elasticsearch search engine to perform data query, obtain the link tracing data corresponding to the query request, generate query results, and return the query results to the user for display. S8. The distributed system listens to Flink tasks through an independent stream processing task (i.e., Logging-flow) to obtain metadata information (such as Flink task configuration, routing rules, etc.), and stores the metadata information in a database (MySQL). Based on changes in the metadata, the system dynamically synchronizes and maintains the data flow in Flink tasks and Elasticsearch.
[0069] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.
[0070] It should be noted that the embodiments of this application include, but are not limited to, the examples described above. It is understood that those skilled in the art can make further settings according to actual needs under the guidance of the ideas in the embodiments of this application, and this application does not impose any restrictions on this.
[0071] This application also provides a data query device 50 based on distributed link tracing, applied to a distributed system, which includes multiple service instances. Please refer to [reference needed]. Figure 5 The device includes: The data acquisition module 501 is used to acquire the original link tracing data of each of the service instances; The data distribution module 502 is used to obtain a data flow table of a preset data topic, determine the target topic corresponding to the original link tracing data according to the data flow table, and send the original link tracing data to the target topic. Data processing module 503 is used to process the original link tracing data through a stream processing task corresponding to the target topic to obtain target link tracing data corresponding to the original link tracing data; The data writing module 504 is used to store the target link tracing data to a preset search engine, which is used for data querying.
[0072] In some feasible implementations, the data writing module 504 is specifically used for: The target link tracing data is stored in a preset search engine, and an index of the target link tracing data is constructed based on the tracing information.
[0073] In some feasible implementations, the apparatus further includes: The query module is used to receive a user's query request for the search engine, the query request including query information; based on the query information and the index information, it retrieves the link tracing data corresponding to the query request from the search engine, and generates query results based on the link tracing data.
[0074] In some feasible implementations, the data acquisition module 501 includes: The deployment submodule is used to embed the proxy service corresponding to each of the service instances into each of the service instances; The data acquisition submodule is used to collect the original link tracing data corresponding to the business requests in the service instance through the proxy service.
[0075] In some feasible implementations, the proxy service is configured with a sampling strategy, and the data acquisition submodule is specifically used for: The proxy service dynamically collects the original link tracing data corresponding to the business requests in the service instance according to the sampling strategy.
[0076] In some feasible implementations, the raw link tracing data includes tracing information, the data flow table includes load information and routing rules for each data topic, and the data distribution module 502 includes: The routing submodule is used to select a first topic corresponding to the tracking information from a plurality of data topics according to the routing rules; The judgment submodule is used to determine whether the first topic meets preset conditions based on the load information; The distribution submodule is used to send the original link tracing data to the first topic when the first topic meets preset conditions.
[0077] In some feasible implementations, the routing submodule is also used for: If the first topic does not meet the preset conditions, based on the load information, a second topic corresponding to the link tracing data is selected from the plurality of data topics, and the original link tracing data is sent to the second topic. or, Create a new data topic as a third topic corresponding to the link tracing data, and send the original link tracing data to the third topic.
[0078] In some feasible implementations, the judgment submodule is specifically used for: The load information of the first topic is compared with the load threshold corresponding to the first topic; If the load information is less than the load threshold, then the first topic is determined to meet the preset condition.
[0079] In some feasible implementations, the data processing module 503 is specifically used for: The target topic corresponding to the stream processing task is monitored in real time through the stream processing task; In response to the detection of data updates in the target topic, the updated original link data in the target topic is retrieved and processed to obtain the target link tracing data corresponding to the original link tracing data.
[0080] In some feasible implementations, the distributed system further includes a link tracing gateway, and the data processing module 503 is further configured to: The raw link tracing data collected is sent to the link tracing gateway through the proxy service.
[0081] In some feasible implementations, the link tracing gateway stores a data flow table for the data topic, and the data processing module 503 is further configured to: The link tracing gateway determines the target topic corresponding to the original link tracing data based on the data flow table, and sends the original link tracing data to the target topic.
[0082] In some feasible implementations, the data processing module 503 is specifically used for: When the amount of raw link tracing data collected by the proxy service meets a preset threshold, the raw link tracing data is sent to the link tracing gateway.
[0083] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0084] In addition, embodiments of this application also provide an electronic device, such as... Figure 6 As shown, it includes a processor 601, a communication interface 602, a memory 603, and a communication bus 604, wherein the processor 601, the communication interface 602, and the memory 603 communicate with each other through the communication bus 604. Memory 603 is used to store computer programs; When the processor 601 executes the program stored in the memory 603, it implements the various processes of the above method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0085] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.
[0086] The communication interface is used for communication between the aforementioned terminal and other devices.
[0087] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.
[0088] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0089] like Figure 7 As shown, in another embodiment provided in this application, a computer-readable storage medium 701 is also provided, which stores instructions that, when executed by one or more processors, cause the processors to perform the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, these instructions will not be repeated here.
[0090] In some embodiments of this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute the data query method based on distributed link tracing in the above embodiments.
[0091] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0093] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
[0094] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0095] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0096] In the embodiments provided in this application, it should be understood that the methods and apparatus disclosed in this application can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.
[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0098] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0099] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.
[0100] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A data query method based on distributed link tracing, characterized in that, Applied to a distributed system, the distributed system comprising multiple service instances, the method includes: Obtain the raw link tracing data for each of the service instances; Obtain the data flow table of the preset data topic, determine the target topic corresponding to the original link tracing data according to the data flow table, and send the original link tracing data to the target topic; The original link tracing data is processed by a stream processing task corresponding to the target topic to obtain the target link tracing data corresponding to the original link tracing data; The target link tracing data is stored in a preset search engine, which is used for data retrieval.
2. The method according to claim 1, characterized in that, The original link tracing data includes tracing information, and storing the target link tracing data in a preset search engine includes: The target link tracing data is stored in a preset search engine, and an index of the target link tracing data is constructed based on the tracing information.
3. The method according to claim 2, characterized in that, The method further includes: Receive user query requests for the search engine, the query requests including query information; Based on the query information and the index information, the link tracing data corresponding to the query request is retrieved from the search engine, and query results are generated based on the link tracing data.
4. The method according to claim 1, characterized in that, The process of obtaining the raw link tracing data for each of the service instances includes: Embed a proxy service corresponding to each of the service instances in each of the service instances; The proxy service collects the original link tracing data corresponding to the business requests in the service instance.
5. The method according to claim 4, characterized in that, The proxy service is configured with a sampling strategy, and the collection of raw link tracing data corresponding to the service instance's business requests through the proxy service includes: The proxy service dynamically collects the original link tracing data corresponding to the business requests in the service instance according to the sampling strategy.
6. The method according to claim 1, characterized in that, The original link tracing data includes tracing information, and the data flow table includes load information and routing rules for each data topic. Determining the target topic corresponding to the original link tracing data based on the data flow table includes: According to the routing rules, a first topic corresponding to the tracking information is selected from a plurality of data topics; Based on the load information, determine whether the first topic meets the preset conditions; If the first topic meets the preset conditions, the original link tracing data will be sent to the first topic.
7. The method according to claim 6, characterized in that, The method further includes: If the first topic does not meet the preset conditions, based on the load information, a second topic corresponding to the link tracing data is selected from the plurality of data topics, and the original link tracing data is sent to the second topic. or, Create a new data topic as a third topic corresponding to the link tracing data, and send the original link tracing data to the third topic.
8. The method according to claim 6, characterized in that, The step of determining whether the first topic meets the preset conditions based on the load information includes: The load information of the first topic is compared with the load threshold corresponding to the first topic; If the load information is less than the load threshold, then the first topic is determined to meet the preset condition.
9. The method according to claim 1, characterized in that, The method of processing the original link tracing data through a stream processing task corresponding to the target topic to obtain the target link tracing data corresponding to the original link tracing data includes: The target topic corresponding to the stream processing task is monitored in real time through the stream processing task; In response to the detection of data updates in the target topic, the updated original link data in the target topic is retrieved and processed to obtain the target link tracing data corresponding to the original link tracing data.
10. The method according to claim 4, characterized in that, The distributed system further includes a link tracing gateway, and the method further includes: The raw link tracing data collected is sent to the link tracing gateway through the proxy service.
11. The method according to claim 10, wherein the link tracing gateway stores a data flow table for the data topic, and the step of determining the target topic corresponding to the original link tracing data according to the data flow table and sending the original link tracing data to the target topic includes: The link tracing gateway determines the target topic corresponding to the original link tracing data based on the data flow table, and sends the original link tracing data to the target topic.
12. The method according to claim 10, characterized in that, The step of sending the collected raw link tracing data to the link tracing gateway through the proxy service includes: When the amount of raw link tracing data collected by the proxy service meets a preset threshold, the raw link tracing data is sent to the link tracing gateway.
13. A data query device based on distributed link tracing, characterized in that, Applied to a distributed system, the distributed system including multiple service instances, the device includes: The data acquisition module is used to acquire the original link tracing data of each of the service instances; The data distribution module is used to obtain a data flow table of a preset data topic, determine the target topic corresponding to the original link tracing data according to the data flow table, and send the original link tracing data to the target topic. The data processing module is used to process the original link tracing data through a stream processing task corresponding to the target topic to obtain the target link tracing data corresponding to the original link tracing data; The data writing module is used to store the target link tracing data to a preset search engine, which is used for data querying.
14. An electronic device, characterized in that, Including processor and memory, among which Memory, used to store computer programs; A processor for executing a program stored in memory to implement the method described in any one of claims 1-12.
15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-12.