Link observation method, device and system and storage medium

Abstract

The embodiment of the invention provides a link observation method, device and system and a storage medium, the method is applied to an application observation system, each node in the system is configured with a node database for storing operation data of each application in the nodes, and the method comprises the following steps: receiving an application data query request sent by a user terminal; determining node information for storing target application data in response to the application data query request; sending a target application data query task to the target node based on the node information; and obtaining target application data fed back by the target node from the node database, and sending the target application data to the user terminal. By storing the data generated by operation of each application in each node of the application observation system in a scattered manner, the problem that the pressure of centralized storage on a single database is too large is solved, and thus the data storage pressure of the system can be relieved. During data query, the target application data is obtained by determining the storage node information, so that the data transmission pressure of the system can be relieved.

Description

Technical Field

[0001] This application relates to the technical field of data observation, and in particular, to a link observation method, device, system, and storage medium. Background Art

[0002] The full-link observation system can display the running status of each application and the upstream and downstream dependencies in the current service.

[0003] Generally, the observation system collects the status data of each application from the server or the application side and uniformly stores the collected data in a database, that is, adopts centralized data storage. Then, the observation system can query the data stored in the one database and perform visual display of the application status based on the corresponding data.

[0004] With the continuous increase in the types of application services, the data that needs to be stored in the database continues to climb, and the data storage pressure also rises accordingly. Various data storage problems may occur, affecting the operation of the observation system. Summary of the Invention

[0005] Embodiments of this application provide a link observation method, device, system, and storage medium. Through this method, at least the data storage pressure and data transmission pressure can be reduced.

[0006] In a first aspect, embodiments of this application provide a link observation method, which is applied to an application observation system. Each node in the system is configured with a node database for storing the running data of each application in the node, including: receiving an application data query request sent by a user terminal; in response to the application data query request, determining node information for storing target application data; based on the node information, sending a target application data query task to a target node; obtaining the target application data fed back by the target node from the node database, and sending the target application data to the user terminal. On the one hand, by dispersedly storing the data generated by each application operation in each node of the application observation system, the problem that the centralized storage puts too much pressure on a single database is solved, thereby alleviating the data storage pressure of the system. On the other hand, when the user queries application data, the node information for storing the target application data can be determined, so as to obtain the target application data fed back by the target node, solving the data transmission pressure on the single database caused by scheduling all data from a single database, thereby alleviating the data transmission pressure of the system. On the further hand, when a certain application generates a large amount of data in a short period of time (such as the data generated during application stress testing), if centralized storage is used, it will occupy the read and write resources of a single database for a long time, affecting the data storage of other applications and delaying the user's query operation. By dispersedly storing the data generated by application operation in each node of the application observation system, the possibility that a certain application occupies the data read and write resources for a long time can be effectively reduced, improving the user experience.

[0007] In a possible implementation manner, each node in the system is further configured with a collector for saving the operation data of each application in the same node to the node database, and the system further includes a cluster database; the method further includes: the collector of each node obtains the tracking identifier of the application operation data and the application identity information based on a preset period, and sends the tracking identifier of the application operation data and the application identity information to the cluster database for saving. By periodically reporting the tracking identifier of the application operation data and the application identity information, the periodic update of the database data tracking information can be realized, so that when the application data is requested to be queried, the node information storing the target application data can be quickly locked through the tracking identifier and the application identity information, thereby improving the system data query efficiency.

[0008] In a possible implementation manner, the system is further configured with a global configuration service; the collector of each node sending the tracking identifier of the application operation data and the application identity information to the cluster database for saving includes: the collector of each node sends the tracking identifier of the application operation data and the application identity information to the cluster database for saving through the global configuration service. On the one hand, by collecting the information reported by all nodes through the global configuration service, the reliability and efficiency of data collection are improved. On the other hand, by uniformly saving the tracking identifier of the application operation data and the application identity information reported by each node to the cluster database, when the user queries the application data, first querying the storage node of the target application data in the cluster database can improve the data query efficiency and the user experience.

[0009] In a possible implementation manner, the system is further configured with a query router and a global configuration service; determining the node information storing the target application data in response to the application data query request includes: the query router queries the target node information storing the target application data in the cluster database based on the application data query request through the global configuration service.

[0010] In a possible implementation manner, the sending the target application data query task to the target node based on the node information includes: the query router routes the target application data query task to the target node based on the target node information. By routing the query task to the relevant nodes through the query router, the response speed of the observation system data query task can be improved, thereby improving the query efficiency and the user experience.

[0011] In a possible implementation, the system is further configured with an observation service for interacting with the user terminal; the obtaining of the target application data fed back by the target node from the node database and sending the target application data to the user terminal includes: the query routing obtains and aggregates the target application data fed back by the target node from the node database, and sends the target application data to the user terminal through the observation service, so that the user terminal can visually display the target application data. The interaction between the system and the front end (such as the Web or App of the user terminal) is realized through the observation service. When the target node is multiple nodes, the query routing can also aggregate the obtained target application data and send it to the user terminal, making the data transmission of the system more reliable. On the other hand, after receiving the target application data sent by the back end of the observation system, the front end can also perform visual display, enabling the user to more intuitively understand the application data and improving the user experience.

[0012] In a second aspect, an embodiment of the present application further provides a link observation device, including: a request receiving module, configured to receive an application data query request sent by the user terminal; a node query module, configured to determine node information storing target application data in response to the application data query request; a task sending module, configured to send a target application data query task to a target node based on the node information; and a data transmission module, configured to obtain the target application data fed back by the target node from the node database and send the target application data to the user terminal.

[0013] In a third aspect, an embodiment of the present application further provides a link observation device, including a processor and a memory. The memory is configured to store at least one instruction, and when the instruction is loaded and executed by the processor, the link observation method provided in the first aspect is implemented. In one implementation, the link observation device may be a chip or a chip module, and the link observation method provided in the first aspect may be pre-burned into the chip or the chip module.

[0014] In a fourth aspect, an embodiment of the present application further provides a link observation system, which is deployed in a K8s cluster. The system is configured with a query routing, and each node in the system is configured with a node database for storing the operation data of each application in the node; after receiving an application data query request sent by the user terminal, the query routing determines node information storing target application data in response to the application data query request, and based on the node information, sends a target application data query task to a target node. After obtaining the target application data fed back by the target node from the node database, the target application data is sent to the user terminal. In one implementation, the system may include a processor, and the processor may be the link observation device provided in the third aspect.

[0015] Fifth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the link observation method provided in the first aspect is implemented.

[0016] Sixth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instruction, and when the computer program or instruction is executed by a processor, the link observation method provided in the first aspect is implemented. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 Schematic diagram of the K8s cluster architecture provided by an embodiment of the present application;

[0019] Figure 2 Architecture diagram of the link observation system provided by an embodiment of the present application;

[0020] Figure 3 Flow chart of the link observation method provided by an embodiment of the present application;

[0021] Figure 4 Front-end visualization display diagram provided by an embodiment of the present application;

[0022] Figure 5 Structure diagram of the link observation device provided by an embodiment of the present application;

[0023] Figure 6 Structure diagram of the link observation device provided by an embodiment of the present application. Detailed Embodiments

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0025] With the increase in application services, more and more data will be written into the database. The present application provides a link observation system to monitor and display the running status and upstream and downstream relationships of applications.

[0026] In some embodiments, the application to be observed by the link observation system provided by the embodiments of the present application can be deployed on a cluster. Exemplarily, the cluster can be a Kubernetes cluster (abbreviated as K8s cluster).

[0027] Figure 1 Schematic diagram of the K8s cluster architecture provided by an embodiment of the present application.

[0028] Referring to Figure 1 As shown, the K8s cluster can include a control plane and several nodes. Among them, the control plane can include:

[0029] API interface, used to manage and control the entire cluster.

[0030] cloud-controller-manager (C-C-M): used to integrate with the cloud platform, such as providing a cloud interface (Cloud Provider API).

[0031] controller-manager (C-M): responsible for managing various controllers

[0032] etcd: used to store the status data of the cluster.

[0033] scheduler: used to schedule Pods to run on appropriate nodes according to resource requirements and node load conditions.

[0034] A node is where the application programs actually run in the K8S cluster. Each node contains the following components:

[0035] kubelet: responsible for running Pods on the node and monitoring their status.

[0036] kube-proxy: responsible for routing network traffic within the cluster and managing the load balancing of services.

[0037] The embodiments of the present application provide a link observation method. Through this method, application data can be dispersed to each node for storage. Thus, when a user queries the link information of a target application (such as information including the application status and upstream and downstream relationship information), the link information of the target application can be obtained from the corresponding node and fed back to the user, thereby reducing the network load pressure of data transmission.

[0038] To implement the above link observation method, the embodiments of the present application provide a link observation system. The link observation system provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0039] Figure 2 This is an architecture diagram of the link observation system provided by an embodiment of the present application.

[0040] Refer to Figure 2 As shown, the system may include a front-end service and a back-end service.

[0041] In some embodiments, the front-end service may include a Web service and an App service. In one implementation, the App may be an Android App or an iOS App. Among them, users can view information such as the status of the application service through the front-end service.

[0042] In some embodiments, all back-end services are deployed in a K8s cluster, which may specifically include an observation part and an application part.

[0043] Refer to Figure 2 As shown, the observation part includes:

[0044] Observation service (observation-service), which provides services for the front-end of the link observation system. Exemplarily, it can respond to the link information query requests made by users through the front-end App service.

[0045] Observation operation (observation-operator), which is used to utilize the capabilities of the k8s controller and is responsible for creating a collector (Collector) and an OLAP database (OLAP Database) in the daemon set (Daemon Set) of each node in the cluster.

[0046] Global configuration service (global-confif-server), which is responsible for collecting the information reported by the collectors of all nodes in the cluster.

[0047] Query router (query-router), which is used to route query tasks to relevant nodes and summarize the queried data.

[0048] The time series analysis database (On-line Analytical Processing, OLAP-Database) is deployed in each node. On the one hand, it is used to store the information of each node, and on the other hand, it can also be used to store the data for routing decisions. In some embodiments, the time series analysis database (OLAP-Database) may include a cluster database and a node database. Among them, the node database is the time series analysis database (OLAP-Database) deployed in the node, which stores the running data of all business applications in the node where it is located. The cluster database can receive the data reported by each node and store the reported data in the cluster database.

[0049] Multiple nodes (Nodes) in the K8s cluster.

[0050] In some embodiments, each node (Node) is configured with a daemon set. Among them, the daemon set also includes a collector, which is responsible for collecting the running data of business applications and storing it in the time series analysis database (OLAP-Database) of the same node, that is, in the node database of the node where it is located. The relevant business applications run in pods in the K8s cluster. Among them, a pod is the smallest unit that can be created and managed in the k8s cluster, the smallest resource object model created or deployed by the user in the resource object model, and also the resource object for running containerized applications on k8s.

[0051] Based on Figure 2 On the basis of the link observation system architecture shown, the embodiments of the present application provide a link observation method. The link observation method provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0052] Figure 3 It is a schematic flowchart of the link observation method provided by an embodiment of the present application.

[0053] Referring to Figure 3 As shown, the method may include the following steps:

[0054] S301: Receive the query request for sending application data from the user terminal.

[0055] In some embodiments, the application data query request of the user terminal may include an application data query request initiated by the user through the front-end Web service or the front-end App service. In one implementation, the application data query request may include a query object (i.e., the target application data), and the query object may include the metrics of the target application or the link information corresponding to the user input ID. Exemplarily, the user initiates a query request for querying the metrics of Application A, or initiates a query for the link information with Trace ID (XXXX). In some embodiments, after the query request of the user terminal is sent out, the observation service in the observation part of the system may receive the query request.

[0056] S302: In response to the application data query request, determine the node information storing the target application data.

[0057] In some embodiments, when the observation service receives the query request initiated by the user terminal, it may send the query request to the query router, and the query router is responsible for querying the node information storing the target data.

[0058] It should be noted that, in some embodiments, the data generated by the operation of the applications in each node (Node) are all saved in the node database of the same node, that is, in the OLAP-Database (Online Analytical Processing Database) of this node. Specifically, in one implementation, all the data generated during the operation of all applications in each Pod will be sent to the collector of the same node, and then the collector will save the application operation-generated data obtained to the OLAP-Database of the same node. Exemplarily, referring to Figure 2 as shown, Node A contains 2 Pods, and the 2 Pods contain 5 running Apps. The collector of Node A saves all the data generated during the operation of the 5 Apps to the OLAP-Database of Node A.

[0059] In addition, it should be noted that in some embodiments, the collector in each node aggregates the data stored in the OLAP-Database of the same node during the current period based on a preset period, and filters out the Trace IDs in the aggregated data. Exemplarily, all Trace IDs of the aggregated data can be filtered out through a Bloom filter. Further, the collector can send the application identity information corresponding to the aggregated data and the Trace IDs of the data to the global-confif-server, and then the global-confif-server saves the obtained application identity information and the Trace IDs of the data to the cluster database, that is, to the OLAP-Database of the cluster. Exemplarily, referring to Figure 2 As shown, there are three nodes deployed in this cluster, namely Node A, Node B, and Node C. The collector of each node aggregates the data stored in the OLAP-Database of the same node during the current period (such as every minute), and uses a Bloom filter to filter out the Trace IDs of the aggregated data. Then, the application identity information corresponding to the aggregated data (such as application names App1, App2…) and the Trace IDs of the data can be sent to the global-confif-server, and then the global-confif-server saves the obtained application identity information and the Trace IDs of the data to the cluster database.

[0060] Based on the above operations, each node can periodically generate the application identity information of the running data and the Trace IDs of the data. When a user requests to query application data, the node location where the target application data is stored can be quickly matched in the cluster database, improving the data query efficiency. Moreover, by distributing the data generated by the application operation and storing it in multiple nodes, the data storage pressure and data transmission pressure can be reduced.

[0061] On the basis described above, the query-router can query the node information storing the target application data in the OLAP-Database of the cluster through the global-confif-server based on the application data query request. Exemplarily, the application data query request initiated by the user is to query the application metrics of application service A within half an hour. Further, the query-router can query, through the global-confif-server, in the OLAP-Database of the cluster the nodes storing the application metrics of application service A, including node A and node C.

[0062] S303: Based on the node information, send the target application data query task to the target node.

[0063] In some embodiments, after the query-router queries the node information storing the target application data, it may further send a target application data query task to the target node based on the node information. Exemplarily, if the node information includes Node A and Node B, the query-router may send the target application data query tasks to Node A and Node B respectively. After receiving the target application data query tasks, Node A and Node B may feedback the target application data stored in their node databases to the query-router. Specifically, when the application data query request initiated by the user is to query the link information with Trace ID 00xx1122 and the link information with Trace ID 00xx1156, based on the determination of the above-mentioned node information storing the target application data, it is obtained that the application data with Trace ID 00xx1122 is stored in Node A, and the application data with Trace ID 00xx1156 is stored in Node B. Furthermore, the query-router may send a query instruction for querying the link information with Trace ID 00xx1122 to Node A, and send a query instruction for querying the link information with Trace ID 00xx1156 to Node B. After receiving the query instruction, Node A feedbacks the link information with Trace ID 00xx1122 stored in the time series analysis database (OLAP-Database) in Node A to the query-router. After receiving the query instruction, Node B feedbacks the link information with Trace ID 00xx1156 stored in the time series analysis database (OLAP-Database) in Node B to the query-router. Further, the query-router may summarize the link information feedbacked by Node A and Node B, so that the query-router obtains the target application data feedbacked by the target node.

[0064] S304: Obtain the target application data feedbacked by the target node from the node database, and send the target application data to the user terminal.

[0065] In some embodiments, the query-router may send the obtained target application data to the front end through the observation-service, that is, send it to the Web or App of the user terminal.

[0066] In some embodiments, after the front end (Web or App) obtains the target application data, it may further perform visual display on the target application data, so that the user can more intuitively understand the situation of the target application data and improve the user experience.

[0067] Figure 4 Schematic diagram of front-end visual display provided for an embodiment of the present application.

[0068] Refer to Figure 4 As shown, in some embodiments, when a user wants to query application metric data of a target application, after the query router sends the obtained target application metrics to the front end through the observation service, the front end can visually display the metric data of the target application. Exemplarily, the application metrics may include: application latency in ms, requests per second, and Http status per second, etc. It should be noted that in other embodiments, the application metrics may also include other types of data, and the present application does not limit the types of application metrics.

[0069] Figure 5 Schematic diagram of the structure of a link observation device provided for an embodiment of the present application.

[0070] Refer to Figure 5 As shown, the device includes:

[0071] A request receiving module 501, configured to receive an application data query request sent by a user terminal;

[0072] A node query module 502, configured to determine node information storing target application data in response to the application data query request;

[0073] A task sending module 503, configured to send a target application data query task to the target node based on the node information;

[0074] A data transmission module 504, configured to obtain target application data fed back by the target node from the node database and send the target application data to the user terminal.

[0075] In some embodiments, the device further includes a data reporting module 505, configured to obtain a tracking identifier of application running data and application identity information based on a preset period, and send the tracking identifier of the application running data and the application identity information to the cluster database for storage.

[0076] Figure 6 Schematic diagram of the structure of a link observation device provided for an embodiment of the present application.

[0077] Refer to Figure 6As shown, the device may include a processor 601 and a memory 602. The memory 602 is used to store at least one instruction, and when the instruction is loaded and executed by the processor 601, it implements the link observation method provided in any embodiment of the present application. In one implementation, Figure 6 The link observation device shown may be a chip or a chip module, and the link observation method provided in the embodiments of the present application may be pre-burned into the chip or the chip module.

[0078] The embodiments of the present application further provide an electronic device, which may include Figure 6 The link observation device provided in the shown embodiment. In one implementation, the electronic device may be a server device.

[0079] The embodiments of the present application further provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the link observation method provided in any embodiment of the present application.

[0080] The embodiments of the present application further provide a computer program product, including a computer program or instruction. When the computer program or instruction is executed by a processor, it implements the link observation method provided in any embodiment of the present application.

[0081] It should be noted that the terminals involved in the embodiments of the present application may include, but are not limited to, personal computers (PCs), personal digital assistants (PDAs), wireless handheld devices, tablet computers, mobile phones, MP3 players, MP4 players, etc.

[0082] It can be understood that the application may be a native application installed on the terminal, or may also be a web application of a browser on the terminal. The embodiments of the present application do not limit this.

[0083] Those skilled in the art can clearly understand that for the convenience and conciseness of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0084] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections between each other can be through some interfaces. The indirect couplings or communication connections of devices or units can be in electrical, mechanical, or other forms.

[0085] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0086] In addition, in each embodiment of the present application, each functional unit can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0087] The above-mentioned integrated units implemented in the form of software functional units can be stored in a computer-readable storage medium. The above-mentioned software functional units stored in a storage medium include several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute some steps of the methods described in each embodiment of the present application. The foregoing storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs.

[0088] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included within the scope of protection of the present application.

[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A link observation method, characterized in that, Applied to an application observation system, where each node in the system is configured with a node database for storing the operation data of each application in the node. The method includes: Receiving an application data query request sent by a user terminal; In response to the application data query request, determining the node information storing the target application data; Based on the node information, sending a target application data query task to the target node; Obtaining the target application data fed back by the target node from the node database and sending the target application data to the user terminal.

2. The method according to claim 1, characterized in that, Each node in the system is further configured with a collector for saving the operation data of each application in the same node to the node database, and the system further includes a cluster database; The method further includes: The collector of each node obtains the tracking identifier and application identity information of the application operation data based on a preset period and sends the tracking identifier and application identity information of the application operation data to the cluster database for saving.

3. The method according to claim 2, wherein The system is further configured with a global configuration service; The collector of each node sending the tracking identifier and application identity information of the application operation data to the cluster database for saving includes: The collector of each node sends the tracking identifier and application identity information of the application operation data to the cluster database for saving through the global configuration service.

4. The method according to claim 2, wherein The system is further configured with a query router and a global configuration service; In response to the application data query request, determining the node information storing the target application data includes: Based on the application data query request, the query router queries the target node information storing the target application data in the cluster database through the global configuration service.

5. The method according to claim 4, characterized in that, Based on the node information, sending a target application data query task to the target node includes: Based on the target node information, the query router routes the target application data query task to the target node.

6. The method according to claim 4, characterized in that, The system is further configured with an observation service for interacting with the user terminal; Obtaining the target application data fed back by the target node from the node database and sending the target application data to the user terminal includes: The query router obtains and aggregates the target application data fed back by the target node from the node database and sends the target application data to the user terminal through the observation service, so that the user terminal can perform visual display on the target application data.

7. A link observation device, characterized in that, The device includes: A request receiving module for receiving an application data query request sent by a user terminal; A node query module for determining the node information storing the target application data in response to the application data query request; A task sending module for sending a target application data query task to the target node based on the node information; A data transmission module for obtaining the target application data fed back by the target node from the node database and sending the target application data to the user terminal.

8. A link observation device, characterized in that, The device includes: A processor and a memory, where the memory is used to store at least one instruction, and when the instruction is loaded and executed by the processor, it implements the link observation method according to any one of claims 1-6.

9. A link observation system, characterized in that, The system is deployed in a K8s cluster. The system is configured with query routing, and each node in the system is configured with a node database for storing the operation data of each application in the node; After receiving an application data query request sent by a user terminal, the query routing responds to the application data query request, determines the node information storing the target application data, and based on the node information, sends a target application data query task to the target node. After obtaining the target application data fed back by the target node from the node database, the target application data is sent to the user terminal.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the link observation method according to any one of claims 1-6.

Images