System for detecting service level of big data service application
By deploying a microservice gateway and a real-time computing engine on the container cloud platform, real-time collection and analysis of operation logs of big data service applications, the problem of inability to observe service levels in real time in the existing technology is solved, and real-time multi-dimensional service level indicators are realized Monitoring and observability have improved service guarantee capabilities.
Patent Information
- Application Number
- CN202411056069.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-08-02
AI Technical Summary
The existing technology cannot observe the service level indicators of big data service applications in real time, resulting in poor service guarantee capabilities and affecting the stable operation of the business.
设计了一种系统,包括容器云平台、微服务网关、消息队列集群、实时计算引擎和实时分析数据库,通过实时采集和分析微服务网关的操作日志,生成各大数据服务应用的服务水平指标, Such as response time, failure rate and success rate.
Real-time monitoring and observability of service level indicators for big data service applications has been realized, service guarantee capabilities have been improved, human analysis and monitoring costs have been reduced, and business support capabilities have been enhanced.
Smart Images

Figure CN118567964B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data service technology, and in particular to a system for detecting the service level of a big data service application. Background Art
[0002] With the gradual maturity of big data technology, many enterprises have completed the value mining of massive stock data through big data technology, achieved the digital transformation goals of business data, data assetization, and asset service, and effectively promoted the development of enterprise business. With the deep involvement of big data technology in enterprise business applications, enterprises have gradually accumulated a large number of big data service applications. These data service applications cover all aspects of the company's business and are usually provided to other business applications by different business departments in the form of data APIs. Since data services are provided by different departments and lack centralized and unified data service monitoring, a series of indicators such as the service level (SLA), service frequency, stability, reliability, and timeliness of data services are in a black box state. The observability of data services is almost 0, and the service guarantee capability is poor, which affects the stable operation of the business.
[0003] The monitoring and service level monitoring of existing big data application services are usually carried out by data demanders based on feedback from data service calls, such as web page data response freezes, data service timeouts, and failures. Then, operation and maintenance personnel from different business departments look for relevant data service logs, locate problems, analyze logs, and obtain indicators such as data service latency and reliability. Then, they reproduce the problems, modify the code, and re-release the data service application to solve the problems and improve service assurance capabilities. Another method is for service assurance personnel to regularly check the logs of different data applications and periodically inspect the stability and reliability of various services in the K8s (Kubernetes) container to achieve the purpose of predicting and troubleshooting potential problems in advance. At the same time, the log collection tool needs to be customized and developed to be bundled with specific service applications, and some service applications are used internally by the enterprise and cannot be directly accessed to collect service logs. Obviously, the above methods cannot solve the real-time observability requirements of service level indicators such as service level, stability, and reliability of big data service applications. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a system for detecting the service level of a big data service application, so as to solve the technical defect in the prior art that the service level indicators of the big data service application cannot be observed in real time.
[0005] In order to achieve the above-mentioned object, the first aspect of the present application provides a system for detecting the service level of a big data service application, the system comprising:
[0006] The container cloud platform is used to provide containers for the data collector and the microservice gateway. The microservice gateway is used to connect a variety of big data service applications. The various big data service applications at least include a client for initiating service calls and a server that provides data services. The microservice gateway is used to generate corresponding operation logs in real time when any client initiates a request or any server responds to a request. The data collector is connected to the microservice gateway to collect the operation logs in the microservice gateway in real time and forward the operation logs to the message queue cluster;
[0007] Message queue cluster, used to receive and cache operation logs forwarded by data collectors;
[0008] A real-time computing engine is used to perform real-time parsing and data fusion of the operation logs cached in the message queue cluster, and send the processed log data to the real-time analysis database, wherein real-time parsing and data fusion of the operation logs refers to parsing the operation logs into log data streams, and performing data fusion on the log data streams and the library data streams to obtain a mixed data stream for the operation logs;
[0009] The real-time analysis database includes multiple analysis models for real-time aggregation and statistical analysis of log data to obtain service level indicators of different dimensions for each big data service application, wherein the service level indicators include at least the average service response / request initiation time, the total number of services / requests, the service / request failure rate, and the service / request success rate of each big data service application.
[0010] In an embodiment of the present application, the container cloud platform is also used to: obtain the compiled microservice gateway compressed package; add the microservice gateway compressed package to the microservice gateway container to expose the access port of the microservice gateway container, so that each big data service application establishes a connection with the microservice gateway based on the access port.
[0011] In an embodiment of the present application, the container cloud platform is also used to: determine the image description file of the microservice gateway, wherein the image description file includes the working directory of the microservice gateway container and the development environment required for the operation of the microservice gateway; define the startup instructions for the microservice gateway container, and add the startup instructions to the image description file; when obtaining the production instructions for the first container image file for the microservice gateway, respond to the production instructions to build the first container image file; add an image warehouse tag to the first container image file to push the first container image file to the corresponding image warehouse.
[0012] In an embodiment of the present application, the container cloud platform is also used to: determine the first containerized deployment script file of the microservice gateway, wherein the first containerized deployment script includes the container namespace required for the operation of the microservice gateway and the number of copies of the microservice gateway container; determine the first container configuration file of the microservice gateway container, and add the first container configuration file to the container configuration item of the first containerized deployment script file, wherein the first container configuration file includes the first container image file of the microservice gateway, the container name of the microservice gateway, and the mount path of the log of the microservice gateway in the microservice gateway container; map the container directory of the microservice gateway container to a local directory; obtain the first configuration file of the microservice gateway, and add the first configuration file to the container configuration item of the first containerized deployment script file; when the first run instruction for the first containerized deployment script file is obtained, respond to the first run instruction to execute the first containerized deployment script file to deploy the microservice gateway to the container cloud platform.
[0013] In an embodiment of the present application, the container cloud platform is also used to: determine a second container configuration file for the data collector container, and add the second container configuration file to the container configuration item of the first containerized deployment script file, wherein the second container configuration file includes the second container image file of the data collector, the container name of the data collector, the mount directory of the data collector's operation log in the data collector container, and the location of the data collector's second configuration file in the directory of the data collector container; respond to the first run instruction to execute the first containerized deployment script file to deploy the data collector to the resource management component where the microservice gateway container is located in the container cloud platform.
[0014] In an embodiment of the present application, the container cloud platform is also used to: configure the configuration resource name for the data collector after the data collector is deployed to the container cloud platform; load the configuration resource name based on the variable configuration management of the container cloud platform to pull the second configuration file of the data collector, the second configuration file includes the log collection source, log collection strategy and log delivery address of the data collector; collect the operation logs in the microservice gateway based on the second configuration file, and forward the operation logs to the message queue cluster.
[0015] In an embodiment of the present application, the container cloud platform is also used to: determine a second containerized deployment script file for variable configuration management, wherein the second containerized deployment script file includes the resource type and resource name of the second containerized deployment script file; configure the configuration information of the second configuration file of the data collector, and add the configuration information to the second containerized deployment script file, wherein the configuration information includes input information and output information of the data collector, the input information includes the input type, whether the configuration is effective, the log path of the microservice gateway, and the strategic configuration required for collecting operation logs, and the output information includes the access address and service port of the service instance in the message queue cluster and the name of the cache topic corresponding to the forwarded operation log; when a second run instruction for the second containerized deployment script file is obtained, respond to the second run instruction to execute the second containerized deployment script file to deploy the variable configuration management to the container cloud platform.
[0016] In an embodiment of the present application, the real-time computing engine is also used to: perform real-time parsing of the operation logs cached in the message queue cluster to convert the operation logs into log data streams; obtain application data of the big data service application corresponding to the operation logs, and convert the application data into corresponding library data streams; fuse the log data stream with the library data stream to generate a hybrid data stream, in which each data in the hybrid data stream includes a data service call address, a service instance of the big data service application, a project code to which the data service belongs, environmental information, a service call method, a service response level, a service execution time, a service address, and a service call time.
[0017] In an embodiment of the present application, multiple analysis models include a data modeling model and a data update model, and the real-time analysis database is also used to: aggregate log data in terms of service instance, service call address dimensions, and service data dimensions based on the data modeling model, and perform data modeling on the aggregated log data based on the data update model to obtain service level indicators of different dimensions for each big data service application.
[0018] In an embodiment of the present application, the system also includes: a large-screen display module, which is used to generate a report for each big data service application based on the service level indicators of different dimensions of each big data service application, and display each report in real time.
[0019] The above technical solution provides a system for detecting the service level of big data service applications, including: a container cloud platform, which is used to provide containers for data collectors and microservice gateways, and the microservice gateway is used to connect multiple big data service applications. The multiple big data service applications at least include a client for initiating service calls and a server that provides data services. The microservice gateway is used to generate corresponding operation logs in real time when any client initiates a request or any server responds to a request. The data collector is connected to the microservice gateway to collect the operation logs in the microservice gateway in real time and forward them to a message queue cluster; the message queue cluster is used to receive and cache the operation logs; the real-time computing engine is used to The operation logs cached in the system are analyzed and fused in real time, and the processed log data is sent to the real-time analysis database, wherein the real-time analysis and data fusion of the operation logs refers to parsing the operation logs into log data streams, and fusing the log data streams and the library data streams to obtain a mixed data stream for the operation logs; the real-time analysis database includes multiple analysis models for real-time aggregation and statistical analysis of log data to obtain service level indicators of different dimensions for each big data service application, wherein the service level indicators include at least the average service response / request initiation time, the total number of services / requests, the service / request failure rate, and the service / request success rate of each big data service application. The above system realizes the real-time collection of big data service application operation logs through technological innovation, and realizes the real-time monitoring of service level indicators of big data service applications in multiple dimensions through real-time data analysis and statistics of different big data service call logs.
[0020] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:
[0022] Figure 1 A structural block diagram of a system for detecting the service level of a big data service application according to an embodiment of the present application is schematically shown;
[0023] Figure 2 A schematic diagram of the connection relationship between a microservice gateway and a big data service application according to an embodiment of the present application is schematically shown. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0025] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0026] Figure 1 The structure block diagram of a system for detecting the service level of a big data service application according to an embodiment of the present application is schematically shown. Figure 1 As shown, an embodiment of the present application provides a system for detecting the service level of a big data service application, the system comprising:
[0027] The container cloud platform 110 is used to provide containers for the data collector 112 and the microservice gateway 114. The microservice gateway 114 is used to connect to a variety of big data service applications. The various big data service applications at least include a client for initiating service calls and a server that provides data services. The microservice gateway 114 is used to generate a corresponding operation log when any client initiates a request or any server responds to a request. The data collector 112 is connected to the microservice gateway 114 to collect the operation logs in the microservice gateway 114 in real time and forward the operation logs to the message queue cluster 120;
[0028] In the embodiment of the present application, it should be noted that the container cloud platform may refer to kubernetes, referred to as K8s, K8s is used to manage containerized applications on multiple hosts in the cloud platform, and the goal of K8s is to make the deployment of containerized applications simple and efficient. K8s's containerization is based on docker, and K8s organizes docker containers through various resource definitions. The smallest unit managed in K8s is pod. K8s needs to manage containers, so a pod is abstracted as a management unit. Pod is the smallest resource control granularity in the K8s cluster. What runs in the pod is a docker container. A container can be understood as a group of service processes, and a group of service processes can include multiple service processes. Specifically, K8s can provide containers for data collectors and microservice gateways, thereby providing an operating environment for the operation of data collectors and microservice gateways. Among them, the microservice gateway can refer to a gateway, which is a unified entrance for all big data service applications. For example, some service applications are private internal applications, and all service applications cannot be accessed to each other, so a special "gate" is needed to intercept requests and decide whether to release them. These requests must first pass through the gateway, but the gateway does not process the business, but forwards the request to a service application according to certain rules. Figure 2 As shown, a schematic diagram of the connection relationship between a microservice gateway and a big data service application is provided. Big data service applications can be divided into a server for providing data services and a client for initiating service calls. All requests initiated by the client and all responses from the server will generate corresponding operation logs in the microservice gateway. Therefore, it is only necessary to collect the operation logs in the microservice gateway to achieve real-time monitoring of the behavior of all big data service applications.
[0029] It should be noted that the data collector can refer to filebeat, which can be regarded as a lightweight log collection, forwarding, and transmission tool. In the kubernetes or docker environment, filebeat is often used to collect container logs. Specifically, this technical solution selects the log collection and transmission tool filebeat and the microservice gateway to be bundled and deployed in the container cloud platform, realizing the real-time collection of operation logs of all big data service applications in a simple and convenient way. At the same time, the data collector filebeat will forward the operation logs collected from the microservice gateway to the message queue cluster in real time.
[0030] In an embodiment of the present application, the container cloud platform is also used to: obtain the compiled microservice gateway compressed package; add the microservice gateway compressed package to the microservice gateway container to expose the access port of the microservice gateway container, so that each big data service application establishes a connection with the microservice gateway based on the access port.
[0031] In this embodiment, it should be noted that the microservice gateway, as a service that is equally important as other big data service applications, plays a vital role in the entire microservice architecture. As the unified entrance of all big data service applications, it is responsible for forwarding the client's request to different server instances according to specific rules. In addition, the microservice gateway also has the functions of load balancing, authority control, current limiting, identity authentication and unified authentication and authorization. The most important thing is that the microservice gateway can centrally process the operation logs of all requests and responses. Therefore, in this technical solution, it is necessary to first establish a connection between the microservice gateway and all big data service applications. Specifically, the code of the microservice gateway is first compiled and packaged to obtain the compiled microservice gateway jar package, and then the instruction ADD gateway-v1.0-SNAPSHOT.jar is executed to copy the microservice gateway jar package to the microservice gateway container to expose the access port of the microservice gateway container as 8088 (EXPOSE 8088), so that each big data service application can access the microservice gateway in the form of IP+the access port 8088.
[0032] In an embodiment of the present application, the container cloud platform is also used to: determine the image description file of the microservice gateway, wherein the image description file includes the working directory of the microservice gateway container and the development environment required for the operation of the microservice gateway; define the startup instructions for the microservice gateway container, and add the startup instructions to the image description file; when obtaining the production instructions for the first container image file for the microservice gateway, respond to the production instructions to build the first container image file; add an image warehouse tag to the first container image file to push the first container image file to the corresponding image warehouse.
[0033] In this embodiment, it should be noted that this technical solution chooses to deploy the microservice gateway gateway and the data collector filebeat in the K8s container cloud platform, so that the convenience of K8s automated operation and maintenance can be fully utilized to ensure the high availability of the service. According to the service deployment specification of K8s cloud native technology, the container image files of the microservice gateway gateway and the data collector filebeat must first be made. Specifically, before making the container image file of the microservice gateway gateway, it is necessary to build the image description file dockerfile of the gateway, formulate the jdk environment for the gateway operation in the dockerfile file based on the instruction FROM openjdk:8-jre, and formulate the working directory of the microservice gateway container based on the instruction FROM openjdk:8-jre. The startup instructions for the microservice gateway container are defined as: CMD java-jargateway-v1.0-SNAPSHOT.jar, and the startup instruction is added to the dockerfile file. After adding the above startup instructions to the dockerfile file, execute the production instruction: docker build-f dockerfile -t bigdata / gateway:1.0.0. to produce the first container image file docker image of the gateway. After the image file of the microservice gateway is produced, add the image warehouse label to the image file, that is, add the address of the image warehouse to the front of the image file name, so that the image file of the microservice gateway can be stored in the private image warehouse based on the command docker push 10.39.52.xxx:8081 / bigdata / gateway:1.0.0.
[0034] In an embodiment of the present application, the container cloud platform is also used to: determine the first containerized deployment script file of the microservice gateway, wherein the first containerized deployment script includes the container namespace required for the operation of the microservice gateway and the number of copies of the microservice gateway container; determine the first container configuration file of the microservice gateway container, and add the first container configuration file to the container configuration item of the first containerized deployment script file, wherein the first container configuration file includes the first container image file of the microservice gateway, the container name of the microservice gateway, and the mount path of the log of the microservice gateway in the microservice gateway container; map the container directory of the microservice gateway container to a local directory; obtain the first configuration file of the microservice gateway, and add the first configuration file to the container configuration item of the first containerized deployment script file; when the first run instruction for the first containerized deployment script file is obtained, respond to the first run instruction to execute the first containerized deployment script file to deploy the microservice gateway to the container cloud platform.
[0035] In this embodiment, after the image file of the microservice gateway is built, the microservice gateway can be containerized and deployed. Specifically, the first containerized deployment script of the microservice gateway needs to be determined first. It should be noted that the first containerized deployment script can refer to gateway-deployment.yaml. After the image file is completed, the containerized deployment script gateway-deployment.yaml of the gateway is written. In this file, the namespace namespace of the microservice gateway container where the gateway service is running and the number of replicas of the microservice gateway container need to be specified. Among them, it should be noted that after the gateway service is started, it runs in K8s in the form of a pod. The pod is the smallest resource management granularity in the K8s cluster. The pod runs a docker container, and a container can be understood as a group of service processes. In this solution, two types of containers need to be deployed in the pod of the gateway service, one container is the gateway running service process, and the other container is the filebeat running service process. To achieve this goal, it is necessary to add container configuration files for the two containers, gateway and filebeat, under the container configuration item containers in the containerized deployment script file gateway-deployment.yaml of the microservice gateway. Among them, the first container configuration file of the microservice gateway container specifically includes the first container image file of the gateway, namely the docker image image: 10.39.52.xxx:8081 / bigdata / gateway:1.0.0, the container name of the microservice gateway: gateway, and the mount path of the gateway log in the microservice gateway container: - mountPath: / logs. Since the container shares storage, computing and other resources with the host where the container is located, the path in the container will eventually be mapped to a directory or network path in the host. In this solution, the container directory of the microservice gateway container can be mapped to a local directory through the configuration emptyDir:{} of the K8s container cloud platform volumes. In addition, you also need to specify some configuration files that the gateway itself depends on. The configuration files for the gateway service runtime are stored in the apollo configuration center. Here you only need to specify the access address of apollo, so that the microservice gateway container will automatically pull the corresponding gateway configuration information from apollo when it starts. Therefore, get the first configuration file of the gateway and add the first configuration file to the container configuration item containers in the first containerized deployment script file gateway-deployment.yaml.When the first running instruction kubectl apply-f gateway-deployment.yaml for the first containerized deployment script file gateway-deployment.yaml is obtained, the deployment of the microservice gateway container gateway pod can be completed in response to the first running instruction kubectl apply-f gateway-deployment.yaml, so as to deploy the microservice gateway gateway to the container cloud platform K8s.
[0036] In an embodiment of the present application, the container cloud platform is also used to: determine a second container configuration file for the data collector container, and add the second container configuration file to the container configuration item of the first containerized deployment script file, wherein the second container configuration file includes the second container image file of the data collector, the container name of the data collector, the mount directory of the data collector's operation log in the data collector container, and the location of the data collector's second configuration file in the directory of the data collector container; respond to the first run instruction to execute the first containerized deployment script file to deploy the data collector to the resource management component where the microservice gateway container is located in the container cloud platform.
[0037] In this embodiment, it should be noted that in this solution, two containers need to be deployed in the pod of the gateway service, one container is the gateway running service process, and the other container is the filebeat running service process. To achieve this goal, it is necessary to add container configuration files for the gateway and filebeat containers under the container configuration item containers in the containerized deployment script file gateway-deployment.yaml of the microservice gateway. Among them, the second container configuration file of the data collector container specifically includes the second container image file of filebeat, namely, the docker image image: 10.39.52.xxx:8081 / bigdata / filebeat-v1:7.7.0, the container name of the data collector name: filebeat, the mount directory of the filebeat running log in the data collector container - mountPath: / logs, the second configuration file of filebeat, and the corresponding host mapping directory configuration volumes (- emptyDir: {}. It should be noted that the container image production process of filebeat is similar to that of gateway. This step directly downloads the docker image of filebeat from the shared image warehouse of docker, and then re-labels the image warehouse label to be based on the instruction: docker push 10.39.52.xxx:8081 / bigdata / filebeat-v1:7.7.0 uploads the filebeat docker image to the private image repository. Add the second container configuration file of the constructed data collector container to the container configuration item containers in the containerized deployment script file gateway-deployment.yaml of the microservice gateway, respond to the first run instruction kubectl apply-f gateway-deployment.yaml to execute the first containerized deployment script file gateway-deployment.yaml, and the deployment of the gateway pod can be completed. After the deployment is completed, the containers corresponding to the two service processes of gateway and filebeat can be seen in the gateway pod, the resource management component of the gateway.
[0038] In an embodiment of the present application, the container cloud platform is also used to: configure the configuration resource name for the data collector after the data collector is deployed to the container cloud platform; load the configuration resource name based on the variable configuration management of the container cloud platform to pull the second configuration file of the data collector, the second configuration file includes the log collection source, log collection strategy and log delivery address of the data collector; collect the operation logs in the microservice gateway based on the second configuration file, and forward the operation logs to the message queue cluster.
[0039] In this embodiment, it should be noted that variable configuration management can refer to configmap, which is an abstract concept in kubernetes. It separates the configuration of services in containers from pods and components, which helps to maintain the portability of workloads and makes configurations easier to change and manage. Configmap is an API object used to save non-confidential data in key-value pairs. When used, pods can use it as an environment variable, command line parameter, or configuration file in a storage volume. In this technical solution, when the filebeat container is running, it is necessary to load the configuration file of filebeat from configmap, and then collect and forward the operation log in the gateway according to this configuration file. Specifically, after deploying filebeat to the container cloud platform, the configmap configuration resource name that needs to be loaded through the configmap configuration is filebeat-configmap, and the configuration resource name filebeat-configmap is loaded based on the container cloud platform, the second configuration file filebeat.yml of filebeat can be pulled, and the mounting of the second configuration file filebeat.yml of filebeat is completed. When the filebeat container is running, the relevant configuration of filebeat will be pulled from the configmap for operation log collection. Among them, the specific configuration information of the second configuration file filebeat.yml of filebeat includes the log collection source, log collection strategy and log delivery address of filebeat.
[0040] In an embodiment of the present application, the container cloud platform is also used to: determine a second containerized deployment script file for variable configuration management, wherein the second containerized deployment script file includes the resource type and resource name of the second containerized deployment script file; configure the configuration information of the second configuration file of the data collector, and add the configuration information to the second containerized deployment script file, wherein the configuration information includes input information and output information of the data collector, the input information includes the input type, whether the configuration is effective, the log path of the microservice gateway, and the strategic configuration required for collecting operation logs, and the output information includes the access address and service port of the service instance in the message queue cluster and the name of the cache topic corresponding to the forwarded operation log; when a second run instruction for the second containerized deployment script file is obtained, respond to the second run instruction to execute the second containerized deployment script file to deploy the variable configuration management to the container cloud platform.
[0041] In this embodiment, it should be noted that variable configuration management may refer to configmap. In this technical solution, when the filebeat container is running, it is necessary to load the filebeat configuration file from the configmap, and then collect and forward the operation logs in the gateway according to this configuration file. Therefore, the variable configuration management must first be deployed to the container cloud platform. Specifically, the second containerized deployment script file filebeat-configmap.yaml of the variable configuration management is determined, and filebeat-configmap.yaml is edited. In the filebeat-configmap.yaml, the configuration resource type kind of filebeat is first defined as configmap, in order to let the K8s cluster know that this is a configmap configuration resource, and configure the configuration resource name of filebeat: filebeat-configmap, and then configure the configuration information of the second configuration file filebeat.yml of filebeat under data. Among them, the configuration information includes the input information filebeatsinputs of filebeat and the output information filebeat outputs of filebeat. The following information is configured in filebeats inputs: input input type type: log, whether the configuration is effective enable: true, log path paths to be monitored: - / logs / *.log Here, the log path of the gateway is configured, as well as some strategic configurations in the process of collecting operation logs: recursive_glob.enabled: #Enable global recursive mode, for example, / logs / ** includes / logs, / logs / *, / logs / * / *; encoding: #Specify the encoding type of the monitored file, both plain and utf-8 can process Chinese logs; exclude_lines: ['^DBG'] #Does not include lines that match regular expressions; include_lines: ['^ERR', '^WARN'] #Includes lines that match regular expressions; harvester_buffer_size: 16384 #The byte size of the buffer used by each harvester when obtaining files; max_bytes: 10485760 #The maximum number of bytes that a single log message can have. All bytes after max_bytes are discarded and not sent.The default value is 10MB (10485760); exclude_files:['\.gz$']#A list of regular expressions for matching files that you want filebeat to ignore; ingore_older: 0 #The default is 0, which means disabled. You can configure 2h, 2m, etc. Note that ignore_older must be greater than the value of close_inactive, which means ignoring files that have not been updated for more than the set value or files that have never been collected by harvester, etc.
[0042] After configuring filebeat inputs, configure filebeat output. It should be noted that in this solution, the operation logs in the microservice gateway in the container need to be collected in real time and sent to the message queue cluster outside the K8s cluster for further analysis. Therefore, in filebeat output, the relevant information of the message queue cluster needs to be configured, mainly including: output.kafka: # Array of hosts to connect to. hosts:["ip1:port1","ip2:port2","ip3:port3"] topic: bigdata-gateway-topic. These configurations are mainly the access address IP and service port of all service instances in the message queue cluster and the name of the cache topic corresponding to the forwarded operation log. After the above configuration is completed, the second containerized deployment script file filebeat-configmap.yaml can be deployed in a container. When the second running instruction kubectl apply-f filebeat-configmap.yaml for the second containerized deployment script file filebeat-configmap.yaml is obtained, the variable configuration management configmap can be published to the K8s cluster in response to the second running instruction kubectlapply-f filebeat-configmap.yaml. The running filebeat container can load the configuration of filebeat.yml in filebeat-configmap from the K8s cluster, and then collect the operation logs of the microservice gateway service in real time according to the specific collection strategy and transmission address configuration information, and forward them to the message queue cluster outside the K8s cluster.
[0043] The message queue cluster 120 is used to receive and cache the operation logs forwarded by the data collector 112;
[0044] In the embodiment of the present application, it should be noted that the message queue cluster may refer to Kafka, which includes multiple customizable topics, each of which can be used to cache data. Therefore, after the operation log collected by filebeat from the gateway is forwarded to Kafka, the operation log can be cached to the topic in Kafka.
[0045] The real-time computing engine 130 is used to perform real-time parsing and data fusion on the operation logs cached in the message queue cluster 120, and send the processed log data to the real-time analysis database 140, wherein the real-time parsing and data fusion on the operation logs refers to parsing the operation logs into log data streams, and performing data fusion on the log data streams and the library data streams to obtain a mixed data stream for the operation logs;
[0046] In this embodiment, it should be noted that the real-time computing engine can refer to flink, and what is stored in the message queue cluster is the operation logs of the big data service application. These log data cannot obtain any useful information without processing and statistical analysis, so it is necessary to continue to perform real-time parsing and data fusion on these operation logs through flink. Specifically, flink parses the operation log into a log data stream, and performs data fusion on the log data stream and the library data stream to obtain a mixed data stream for the operation log. At the same time, after the real-time fusion processing of flink, the data elements of each log data are richer and more comprehensive, because flink is a real-time processing system, which is more suitable for real-time processing of limited data sets, and flink is powerless to perform statistical analysis on data sets with long spans. Therefore, in this solution, after flink performs fusion analysis on the log data, the processed log data still needs to be sent to the real-time analysis database in real time for real-time aggregation and statistical analysis.
[0047] In an embodiment of the present application, the real-time computing engine is also used to: perform real-time parsing of the operation logs cached in the message queue cluster to convert the operation logs into log data streams; obtain application data of the big data service application corresponding to the operation logs, and convert the application data into corresponding library data streams; fuse the log data stream with the library data stream to generate a hybrid data stream, in which each data in the hybrid data stream includes a data service call address, a service instance of the big data service application, a project code to which the data service belongs, environmental information, a service call method, a service response level, a service execution time, a service address, and a service call time.
[0048] In this embodiment, after the logs of the microservice gateway are collected into kafka through filebeat, due to the large amount of data on the server side and the high throughput, the client requests and calls the server side at all times, resulting in a large amount of log data. In order to improve the efficiency of real-time computing, a new generation of flink real-time computing engine is selected in this solution to process a large number of operation logs in the message queue kafka. Through programming, the real-time consumption of log data in the bigdata-gateway-topic in kafka by the flink engine is realized, and it is converted into the log data stream LogDataStream in flink. At the same time, through programming, flink is realized to read the service interface, service id, and application data related to the client recorded in the mysql business library in real time, and convert it into another data stream MySQLDataStream in flink. The data stream MySQLDataStream can refer to the library data stream. Because any server will register the relevant information of the service on the data service platform when publishing, any client will register on the data service platform when applying for the server, and apply for related big data services, and this information will be written to the mysql database in real time. Through flink programming, this information is also read into flink in real time to form a data stream. The purpose is to associate and fuse the two data streams MySQLDatastream and LogDataStream according to the interface address url of the big data service through specific operators such as join or union in flink. This makes the information of each log data more complete and rich, including all the data elements required for subsequent data analysis. In this solution, through flink computing programming, the fusion analysis of the two data streams is realized, and finally the two data streams are turned into a mixed data stream MixResultDataStream. Each data in the mixed data stream finally contains: data service call address url, service provider instance service-id, data service project id (project_id), environment information, service call method, server response level, server execution time, server ip, service pod name, service call date (year, month, day), client instance (business application), response code, response header and other data elements.
[0049] The real-time analysis database 140 includes multiple analysis models for real-time aggregation and statistical analysis of log data to obtain service level indicators of different dimensions for each big data service application, wherein the service level indicators include at least the average service response / request initiation time, the total number of services / requests, the service / request failure rate, and the service / request success rate of each big data service application.
[0050] In this embodiment, it should be noted that the real-time analysis database may refer to Dorisdb. In this solution, Dorisdb is selected as the storage medium for log data because Dorisdb has the functions of real-time aggregation and real-time update, so that real-time aggregation statistics can be performed based on the service instance or service address of the big data service application, or the project and other different dimensions, so that the latest call / request times, the latest service response / request initiation time, and the latest service level indicator data of different big data services can be obtained, and the average call response / request initiation time, the total service / request times, the service / request failure rate, the service / request success rate, and other indicators related to SLA can be periodically calculated for different big data services within a certain period of time (every 5 minutes, every hour, every day, every week, etc.).
[0051] In an embodiment of the present application, multiple analysis models include a data modeling model and a data update model, and the real-time analysis database is also used to: aggregate log data in terms of service instance, service call address dimensions, and service data dimensions based on the data modeling model, and perform data modeling on the aggregated log data based on the data update model to obtain service level indicators of different dimensions for each big data service application.
[0052] In this embodiment, it should be noted that the multiple analysis models of Dorisdb may include a data modeling model AGGREGATE_KEY and a data update model UPDATE_KEY. Specifically, the AGGREGATE_KEY data modeling model of Dorisdb is selected, and aggregation modeling is performed with dimensions such as server_id, url, and dt. At the same time, the UPDATE_KEY update model of Dorisdb is selected for data modeling. After the database table model is built, each log data in the mixed data stream MixResultDataStream fused by the real-time computing engine fink is written to Dorisdb in real time through flink programming in the flink computing engine. Dorisdb performs real-time aggregation, calculation, and update writing of data according to the corresponding data model, and then through SQL query, a series of service level indicators of big data service applications of different dimensions can be queried.
[0053] In an embodiment of the present application, the system also includes: a large-screen display module, which is used to generate a report for each big data service application based on the service level indicators of different dimensions of each big data service application, and display each report in real time.
[0054] In this embodiment, it should be noted that Dorisdb performs real-time data aggregation, calculation and update writing according to the corresponding data model. Through SQL query, a series of service level indicators of big data services in different dimensions can be queried. However, this manual viewing method cannot clearly reflect the service level of different data services, and the observability is still very low. In order to make the observability of big data service applications stronger, this solution selects a large screen tool to directly connect to Dorisdb. Through the development based on large screen display, a series of service level indicators such as the total number of calls, failure rate, success rate, average call time, top10 high-frequency big data services of different big data service applications are produced through reports. In a graphical form, the SLA indicators of big data services in Dorisdb are vividly and clearly presented in real time. For big data service applications with low SLA levels, they are presented in real time in warning colors to remind operation and maintenance personnel to pay attention to these big data service applications in real time and conduct problem troubleshooting in advance when necessary, which greatly improves the observability and visual monitoring and operation and maintenance capabilities of the SLA of massive big data analysis results, and effectively guarantees the service guarantee capabilities of enterprise business.
[0055] This technical solution realizes the real-time collection of big data service application operation logs through technological innovation, and realizes real-time monitoring of service level indicators in multiple dimensions of big data service applications through real-time data analysis and statistics of different big data service call logs.
[0056] Through the above innovative process design and technological innovation, the real-time observability of the service level of massive cloud-native big data service applications is achieved, the cost of manual analysis and monitoring is reduced, the monitoring efficiency and service guarantee level of big data service applications are greatly improved, and the service status, service response time, service failure, service frequency, success rate and other indicators of various data services can be perceived in real time, effectively guaranteeing the business support capabilities of business applications. In summary, the innovative points of this solution are concentrated in the following points:
[0057] 1) Successfully implemented the bundled deployment of the log collection tool filebeat and the gateway gateway in the K8s cluster.
[0058] 2) Innovatively choose to collect all data service operation logs on the gateway side. Compared with the traditional method of collecting operation logs on the data service side, it greatly simplifies the difficulty of log collection, saves resources and improves efficiency.
[0059] 3) In the container, the collection strategy of filebeat is configured through configmap, and the logs in the container are successfully collected and transmitted to the message queue kafka outside the K8s cluster in real time.
[0060] 4) By introducing the real-time computing engine flink and the real-time storage medium Dorisdb real-time OLAP (online analysis) database, the fusion analysis and real-time aggregation, calculation, and update of log data are realized, which greatly improves the computing efficiency, solves the problem of log data fusion analysis, and meets the needs of real-time statistical analysis.
[0061] 5) Through the visualization big screen technology, the visualization and real-time presentation of the SLA indicators of big data service applications is realized, and the service level monitoring and operation and maintenance are very convenient, simple and efficient. Through this solution, the black box of massive big data service application monitoring in the big data era is completely opened, which greatly improves the observability of data services and effectively promotes the realization of the digital transformation strategic goal of enterprise data assetization and asset service.
[0062] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0063] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A system for detecting the service level of a big data service application, It is characterized in that The system comprises: A container cloud platform is used to provide containers for a data collector and a microservice gateway, the microservice gateway is used to connect a variety of big data service applications, the various big data service applications at least include a client for initiating service calls and a server that provides data services, the microservice gateway is used to generate corresponding operation logs in real time when any client initiates a request or any server responds to a request, the data collector is connected to the microservice gateway, and is used to collect the operation logs in the microservice gateway in real time, and forward the operation logs to a message queue cluster; wherein the container cloud platform is also used to configure a configuration resource name for the data collector after the data collector is deployed to the container cloud platform; the configuration resource name is loaded based on the variable configuration management of the container cloud platform to pull a second configuration file of the data collector, the second configuration file includes the log collection source, log collection strategy and log transmission address of the data collector; based on the second configuration file, the operation logs in the microservice gateway are collected, and the operation logs are forwarded to the message queue cluster; The message queue cluster is used to receive and cache the operation log forwarded by the data collector; A real-time computing engine, used for real-time parsing and data fusion of the operation logs cached in the message queue cluster, and sending the processed log data to the real-time analysis database, wherein real-time parsing and data fusion of the operation logs refers to parsing the operation logs into log data streams, and data fusion of the log data streams and the library data streams to obtain a mixed data stream for the operation logs; The real-time analysis database includes multiple analysis models for real-time aggregation and statistical analysis of log data to obtain service level indicators of different dimensions for each big data service application, wherein the service level indicators include at least the average service response / request initiation time, the total number of services / requests, the service / request failure rate and the service / request success rate of each big data service application.
2. The system for detecting the service level of a big data service application according to claim 1, It is characterized in that The container cloud platform is also used for: Get the compiled microservice gateway compressed package; The microservice gateway compression package is added to the microservice gateway container to expose the access port of the microservice gateway container, so that each big data service application establishes a connection with the microservice gateway based on the access port.
3. The system for detecting the service level of a big data service application according to claim 1, It is characterized in that The container cloud platform is also used for: Determine the image description file of the microservice gateway, wherein the image description file includes the working directory of the microservice gateway container and the development environment required for the operation of the microservice gateway; Define a startup instruction for the microservice gateway container, and add the startup instruction to the image description file; When a production instruction for a first container image file for the microservice gateway is obtained, respond to the production instruction to build the first container image file; An image repository tag is added to the first container image file to push the first container image file to the corresponding image repository.
4. The system for detecting the service level of a big data service application according to claim 1, It is characterized in that The container cloud platform is also used for: Determine a first containerized deployment script file of the microservice gateway, wherein the first containerized deployment script includes a container namespace required for the operation of the microservice gateway and the number of copies of the microservice gateway container; Determine a first container configuration file of a microservice gateway container, and add the first container configuration file to the container configuration item of the first containerized deployment script file, wherein the first container configuration file includes a first container image file of the microservice gateway, a container name of the microservice gateway, and a mount path of the log of the microservice gateway in the microservice gateway container; Map the container directory of the microservice gateway container to the local directory; Obtain a first configuration file of the microservice gateway, and add the first configuration file to a container configuration item of the first containerized deployment script file; When a first running instruction for the first containerized deployment script file is obtained, the first running instruction is responded to to execute the first containerized deployment script file to deploy the microservice gateway to the container cloud platform.
5. The system for detecting the service level of a big data service application according to claim 4, It is characterized in that The container cloud platform is also used for: Determine a second container configuration file for the data collector container, and add the second container configuration file to the container configuration item of the first containerized deployment script file, wherein the second container configuration file includes the second container image file of the data collector, the container name of the data collector, the mount directory of the operation log of the data collector in the data collector container, and the location of the second configuration file of the data collector in the directory of the data collector container; Respond to the first run instruction to execute the first containerized deployment script file to deploy the data collector to the resource management component where the microservice gateway container is located in the container cloud platform.
6. The system for detecting the service level of a big data service application according to claim 1, It is characterized in that The container cloud platform is also used for: Determine a second containerized deployment script file for variable configuration management, wherein the second containerized deployment script file includes a resource type and a resource name of the second containerized deployment script file; Configure the configuration information of the second configuration file of the data collector, and add the configuration information to the second containerized deployment script file, wherein the configuration information includes the input information and output information of the data collector, the input information includes the input type, whether the configuration is effective, the log path of the microservice gateway, and the strategic configuration required for collecting operation logs, and the output information includes the access address and service port of the service instance in the message queue cluster and the name of the cache topic corresponding to the forwarded operation log; When a second running instruction for the second containerized deployment script file is obtained, the second running instruction is responded to to execute the second containerized deployment script file so as to deploy the variable configuration management to the container cloud platform.
7. The system for detecting the service level of a big data service application according to claim 1, It is characterized in that The real-time calculation engine is also used to: Performing real-time parsing on the operation logs cached in the message queue cluster to convert the operation logs into log data streams; Acquire application data of the big data service application corresponding to the operation log, and convert the application data into a corresponding library data stream; The log data stream is merged with the library data stream to generate a hybrid data stream, in which each data item includes a data service call address, a service instance of a big data service application, a project code to which the data service belongs, environmental information, a service call method, a service response level, a service execution time, a service address, and a service call time.
8. The system for detecting the service level of a big data service application according to claim 1, It is characterized in that The multiple analysis models include a data modeling model and a data update model, and the real-time analysis database is further used for: Based on the data modeling model, the log data is aggregated in terms of service instance, service call address dimension and service data dimension, and data modeling is performed on the aggregated log data based on the data update model to obtain service level indicators of different dimensions for each big data service application.
9. The system for detecting the service level of a big data service application according to claim 1, It is characterized in that The system further comprises: The large-screen display module is used to generate a report for each big data service application based on the service level indicators of different dimensions of each big data service application, and display each report in real time.
