Cloud service-based data processing method, apparatus, device, and medium

Abstract

Embodiments of the present application provide a data processing method and device based on cloud service, equipment and medium. The method comprises: receiving a data observation request sent by a terminal device; obtaining an observation strategy template corresponding to the data observation request, the observation strategy template comprising a to-be-observed data type, an observation index and an index analysis strategy; obtaining target observation data corresponding to the to-be-observed data type and a target data model; wherein the target observation data is data of the to-be-observed data type collected by a non-invasive implant probe and / or a log collector from an application in advance, and the target data model is metadata corresponding to the to-be-observed data type obtained in advance; performing data analysis according to the target observation data, the target data model, the observation index and the index analysis strategy to obtain an observation result; and returning the observation result to the terminal device. The method achieves the effects of improving the container environment adaptability of the probe and reducing the complexity of probe implantation.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a data processing method, apparatus, device, and medium based on cloud services. Background Technology

[0002] Application Performance Management (APM), as a key operational monitoring and management tool, focuses on real-time monitoring and analysis of various performance metrics of applications within a business system. Its core purpose is to promptly identify and effectively resolve potential performance bottlenecks and issues. In past technical practices, APM has primarily focused on application performance monitoring and analysis within three-tier application architecture systems.

[0003] However, with the rapid development and increasing popularity of cloud technology, more and more business systems are beginning to shift to cloud-native architecture, which directly leads to a large number of applications being deployed in containerized environments.

[0004] Current technologies for monitoring applications in cloud-native business systems mostly employ intrusive probe designs. This means that deploying probes requires significant modifications to the application code, and the implantation process is complex and cumbersome. Furthermore, this intrusive probe design may have a potential negative impact on application performance, resulting in poor adaptability of the probes to container environments and difficulty in meeting the performance monitoring needs of current cloud-native architectures. Summary of the Invention

[0005] This application provides a cloud service-based data processing method, device, and medium. When monitoring applications in business systems using cloud-native architectures, probes can be implanted non-intrusively to improve the container environment adaptability of probe technology and reduce the complexity of probe implantation.

[0006] In a first aspect, embodiments of this application provide a data processing method based on cloud services, including:

[0007] Receive data observation requests sent by terminal devices;

[0008] Obtain the observation strategy template corresponding to the data observation request. The observation strategy template includes the data type to be observed, the observation indicators, and the indicator analysis strategy.

[0009] Obtain target observation data and target data model corresponding to the data type to be observed; wherein, the target observation data is data of the type to be observed that is collected in advance by the application using a non-invasive implanted probe and / or log collector, and the target data model is metadata corresponding to the data type to be observed that is obtained in advance;

[0010] Based on the target observation data, the target data model, the observation indicators, and the indicator analysis strategy, data analysis is performed to obtain observation results.

[0011] The observation results are returned to the terminal device.

[0012] In one possible implementation, acquiring the target observation data and target data model corresponding to the data type to be observed includes:

[0013] Obtain the target observation data corresponding to the data type to be observed from the observable data database. The observable database stores data of multiple data types collected from multiple applications through non-intrusive probes and preset log collectors.

[0014] The target data model corresponding to the data type to be observed is obtained from the observable model library. The observable model library stores multiple predefined data models corresponding to different data types. The data model corresponding to each data type is used to indicate the metadata of the data of that data type.

[0015] In one possible implementation, obtaining the observation strategy template corresponding to the data observation request includes:

[0016] If the data observation request carries the template identifier selected by the user, then the observation strategy template corresponding to the template identifier is obtained from the template management library, which includes multiple predefined different strategy templates.

[0017] In one possible implementation, if the observation request does not carry a template identifier selected by the user, a policy template acquisition request is sent to the terminal device.

[0018] The terminal device receives the observation strategy template customized by the user through a query language.

[0019] In one possible implementation, before obtaining the observation strategy template corresponding to the data observation request, the method further includes:

[0020] Based on the obtained tracking status information and the basic configuration information of the application, a probe is implanted in the container corresponding to the application, and the container includes a mirror application of the application.

[0021] During application operation, initial observation data is collected based on the implanted probes and the preset log collector.

[0022] The initial observation data is preprocessed, and the processed observation data is stored in the observable database according to the data type.

[0023] In one possible implementation, the preprocessing of the initial observation data includes:

[0024] The log data in the initial observation data is parsed according to the preset data processing rules, and the parsed log data is cleaned to obtain cleaned log data.

[0025] According to the data processing rules, the non-log data in the observation data other than the log data is parsed, and the parsed data is converted by a protocol to obtain the converted non-log data;

[0026] The data processing rules include methods for parsing data from different data sources.

[0027] In one possible implementation, the step of implanting a probe in the container corresponding to the application based on the obtained tracking status information and the application's basic configuration information includes:

[0028] The type of probe is determined based on the technology stack type in the basic configuration information of the application. The probe type includes any of the following: Java probe, Go probe, or Node probe.

[0029] Based on the type of probe, the implantation method of the probe is determined, and the implantation method includes automatic implantation or manual implantation;

[0030] Based on the embedded point status information, the probe type, and the implantation method, the probe is implanted into the container corresponding to the application.

[0031] Secondly, embodiments of this application provide a cloud service-based data processing apparatus, comprising:

[0032] The request receiving module is used to receive data observation requests sent by terminal devices.

[0033] The first acquisition module is used to acquire an observation strategy template corresponding to the data observation request. The observation strategy template includes the data type to be observed, the observation index, and the index analysis strategy.

[0034] The second acquisition module is used to acquire target observation data and target data model corresponding to the data type to be observed; wherein, the target observation data is data of the type to be observed obtained in advance by using a non-invasive implanted probe and / or log collector to collect data from the application, and the target data model is metadata corresponding to the data type to be observed that is acquired in advance;

[0035] The result acquisition module is used to perform data analysis based on the target observation data, the target data model, the observation indicators, and the indicator analysis strategy to obtain the observation results.

[0036] The result return module is used to return the observation results to the terminal device.

[0037] In one possible implementation, the second acquisition module is specifically used for:

[0038] Obtain the target observation data corresponding to the data type to be observed from the observable data database. The observable database stores data of multiple data types collected from multiple applications through non-intrusive probes and preset log collectors.

[0039] The target data model corresponding to the data type to be observed is obtained from the observable model library. The observable model library stores multiple predefined data models corresponding to different data types. The data model corresponding to each data type is used to indicate the metadata of the data of that data type.

[0040] In one possible implementation, the first acquisition module is specifically used for:

[0041] If the data observation request carries the template identifier selected by the user, then the observation strategy template corresponding to the template identifier is obtained from the template management library, which includes multiple predefined different strategy templates.

[0042] In one possible implementation, the first acquisition module is specifically used for:

[0043] If the observation request does not carry the template identifier selected by the user, a policy template acquisition request is sent to the terminal device.

[0044] The terminal device receives the observation strategy template customized by the user through a query language.

[0045] In one possible implementation, this application provides a cloud service-based data processing apparatus, which further includes:

[0046] Probe implantation module: used to implant probes into the container corresponding to the application based on the obtained tracking status information and the basic configuration information of the application. The container includes a mirror application of the application.

[0047] Data acquisition module: Used to collect initial observation data during application operation based on the implanted probes and preset log collectors;

[0048] Data processing module: used to preprocess the initial observation data and store the processed observation data in the observable database according to the data type.

[0049] In one possible implementation, the data processing module is specifically used for:

[0050] The log data in the initial observation data is parsed according to the preset data processing rules, and the parsed log data is cleaned to obtain cleaned log data.

[0051] According to the data processing rules, the non-log data in the observation data other than the log data is parsed, and the parsed data is converted by a protocol to obtain the converted non-log data;

[0052] The data processing rules include methods for parsing data from different data sources.

[0053] In one possible implementation, the probe implantation module is specifically used for:

[0054] The type of probe is determined based on the technology stack type in the basic configuration information of the application. The probe type includes any of the following: Java probe, Go probe, or Node probe.

[0055] Based on the type of probe, the implantation method of the probe is determined, and the implantation method includes automatic implantation or manual implantation;

[0056] Based on the embedded point status information, the probe type, and the implantation method, the probe is implanted into the container corresponding to the application.

[0057] Thirdly, embodiments of this application provide a cloud service-based data processing device, including: a memory and a processor;

[0058] The memory stores computer-executed instructions;

[0059] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0060] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0061] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0062] The cloud service-based data processing method, apparatus, device, and medium provided in this application first receive a data observation request sent by a terminal device; then obtain an observation strategy template corresponding to the data observation request, containing the data type to be observed, observation indicators, and indicator analysis strategies; next, obtain the target observation data and the target data model corresponding to the data type to be observed in the template, which are pre-collected by the application using a non-intrusive probe and / or log collector; perform data analysis based on the target observation data, target data model, observation indicators, and indicator analysis strategies to obtain observation results; and finally, return the observation results to the terminal device. This method, designed with non-intrusive probes for applications in cloud-native architecture systems, achieves the effect of improving the container environment adaptability of probe technology and reducing the complexity of probe implantation. Attached Figure Description

[0063] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0064] Figure 1 A schematic diagram illustrating a scenario for the cloud service-based data processing method provided in this application;

[0065] Figure 2 A flowchart illustrating an embodiment of the cloud service-based data processing method provided in this application;

[0066] Figure 3 A schematic flowchart of Embodiment 2 of the cloud service-based data processing method provided in this application;

[0067] Figure 4 A detailed schematic diagram of the probe implantation process and data acquisition process in the cloud service-based data processing method provided in this application;

[0068] Figure 5 A schematic diagram illustrating a specific embodiment of the cloud service-based data processing method provided in this application;

[0069] Figure 6A schematic diagram of the structure of a cloud service-based data processing device according to Embodiment 1 provided in this application;

[0070] Figure 7 A schematic diagram of the structure of the cloud service-based data processing device according to Embodiment 2 provided in this application;

[0071] Figure 8 A schematic diagram of the structure of the cloud service-based data processing device provided in this application.

[0072] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0073] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0074] First, let me explain the terms used in this application:

[0075] Containerized applications refer to packaging an application and all its dependencies, runtime environment, and configuration into a lightweight, portable container. These containers can run in any container-enabled environment, thereby improving the portability and consistency of the application.

[0076] Container orchestration tools are tools used to automate the management, deployment, scaling, and operation of containerized applications. Commonly used container orchestration tools include Kubernetes, Docker Swarm, and Apache Mesos.

[0077] To clearly understand the technical solution of this application, the solutions of the prior art will be described in detail below.

[0078] Current APM technologies primarily focus on application performance monitoring and analysis within three-tier application architecture systems. However, with the gradual maturation and widespread adoption of cloud technology, business systems are increasingly adopting cloud-native architectures. This shift places higher demands on the monitoring and analysis of business services, requiring the attention to and analysis of richer information. Existing APM technologies have several shortcomings when dealing with cloud-native architectures:

[0079] First, existing probe technologies may intrude on applications during data collection and have poor adaptability to container environments. Deploying probes typically requires modifying application code, involves complex configuration processes, and results in incomplete data collection, making it difficult to construct a complete call topology. Furthermore, probe technologies face challenges in extending to customized attributes of specific business systems, and may lack sufficient information when tracing issues, affecting their scalability and effectiveness in changing technological environments.

[0080] Furthermore, existing technologies typically support only a limited set of protocols and rely on specific servers to parse probe data. This is insufficiently flexible in cloud-native architectures and struggles to adapt to the diverse probe technologies and dynamically expanding business attributes required. The lack of dynamic rule definitions in data parsing can lead to increased storage and processing costs for APM systems when handling the massive and complex data of applications in cloud environments. It also makes real-time performance monitoring difficult, potentially causing response delays when quickly identifying and resolving performance issues.

[0081] Meanwhile, existing technologies generally lack a unified data model in the field of system observability, making it difficult to centrally process logs, traces, metrics, and other custom observation data. This fragmented data state hinders the generation of a comprehensive monitoring view, thereby affecting the efficiency of analysis and diagnosis, and reducing the ability to quickly locate and handle problems in complex monitoring environments.

[0082] In addition, existing technology tools lack customization features in terms of user interface and experience, and cannot pre-define data monitoring and analysis strategy templates according to user needs, making it difficult for non-technical users to understand and use when querying and tracking issues.

[0083] Figure 1 A schematic diagram illustrating the scenario of the cloud service-based data processing method provided in this application, such as... Figure 1 As shown, the specific application scenarios of this application include: terminal device 101 and cloud server 102. The method provided in this application is applied to cloud server 102. The terminal device can be a computer, workstation, or other similar device; this application does not impose any restrictions on this.

[0084] Specifically, when performing performance monitoring and analysis on applications in a business system, the user-operated terminal device 101 sends an observation request to the cloud server. Based on the observation request, the cloud server 102 receives the request and obtains an observation strategy template corresponding to the request. This template includes the data type to be observed, the observation metrics, and the metric analysis strategy in response to the terminal device's observation request. Furthermore, the cloud server 102 also obtains the target observation data and target data model corresponding to the data type to be observed. The target observation data is data of the data type to be observed, pre-collected from the application using a non-intrusive probe and / or log collector; the target data model is pre-acquired metadata corresponding to the data type to be observed. Finally, the cloud server 102 performs data analysis based on the acquired target observation data, target data model, observation metrics, and the metric analysis strategy to obtain observation results, which are then returned to the terminal device 101. Users can monitor and analyze the business system through the observation results displayed on the terminal device. Furthermore, the cloud server 102 can also provide services such as probe customization and implantation, data collection, and processing.

[0085] Based on the above background and the application scenario of the cloud service-based processing method provided in this application, it can be seen that the cloud service-based data processing method provided in this application integrates probe technology with container orchestration tools, and uses non-intrusive probe technology to collect observation data of applications in cloud-native architecture systems. In response to changes in the containerized environment of cloud-native architecture, it improves the container environment adaptability of probes, reduces the complexity of probe implantation, and further reduces the impact of probe implantation on application performance.

[0086] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will be described below with reference to the accompanying drawings.

[0087] Figure 2 The flowchart of the cloud service-based data processing method provided in this application is shown in the following figure. Figure 2 As shown, the method includes:

[0088] S201, Receive data observation request sent by terminal device;

[0089] In this solution, when a user needs to monitor and analyze applications in a cloud-native business system, they send a data observation request to the cloud server via a terminal device. The cloud server then retrieves the observation strategy template based on the request, extracts the target observation data of the applications in the business system, and the corresponding model metadata, obtaining observation results that meet the user's needs. The observations are then displayed on the terminal device. It is worth noting that the target observation data in the method provided in this application is collected using a non-intrusive probe and log collector.

[0090] In this step, if the user needs to monitor and analyze applications in a cloud-native business system, the user can operate the terminal device to send data observation requests to the cloud server via a communication protocol.

[0091] In one specific implementation, the data observation request can carry a template identifier selected by the user. Specifically, after the user selects a template that meets the observation requirements on the terminal device, the terminal device sends a data observation request to the cloud server, which carries the template identifier selected by the user.

[0092] S202. Obtain the observation strategy template corresponding to the data observation request. The observation strategy template includes the data type to be observed, the observation indicators, and the indicator analysis strategy.

[0093] In this step, an observation strategy template corresponding to the observation request can be obtained from the template management library. This observation strategy template may include: the observation data type, the observation indicators, and the indicator analysis strategy.

[0094] In the specific implementation of this solution, it should be understood that an observation strategy template can contain one or more data types, and one data type can correspond to one or more observation metrics. Observation data types are used to indicate multi-dimensional monitoring sources, which may include log data, metric data, tracing data, etc. Observation metrics are used to indicate the performance, health status, or other key characteristics of the application that the user needs to observe. Specifically, observation metrics corresponding to log data can be: system security (such as login attempts, permission changes, abnormal behavior, etc.) and system stability (such as service crashes, restarts, error logs, etc.); observation metrics corresponding to metric data can be: response time, error rate, memory usage, network traffic, etc.; observation metrics corresponding to tracing data can be request paths, etc.

[0095] In the specific implementation of this solution, it should be understood that the indicator analysis strategy includes the methods and approaches for analyzing and interpreting the collected observation data. For example, the indicator analysis strategy includes: methods for analyzing observation data (such as trend analysis, anomaly detection, etc.), and methods for visualizing data (such as charts, dashboards, etc.).

[0096] In the specific implementation of this solution, it should be understood that the observation strategy template may also include data archiving strategies and notification strategies. The data archiving strategy defines how to store and manage historical performance data to facilitate long-term analysis of the observation data and compliance requirements, such as defining data storage location, data retention period, and data archiving frequency. The data analysis template defines at least one metric and its corresponding analysis method for evaluating application performance. The notification strategy template defines alarm conditions, alarm channels, and alarm formats. Alarm channels may include Lark, email, Webhook, etc.

[0097] In one specific implementation, if a user selects an observation strategy template that meets their observation needs on their terminal device, the data observation request will carry the template identifier selected by the user. The cloud server retrieves the observation strategy template corresponding to the template identifier from the template management library, which includes multiple predefined different strategy templates. Specifically, the observation data types, observation indicators, and indicator analysis strategies differ among the different strategy templates.

[0098] In the specific implementation of this solution, it should be understood that users can adjust the preset observation strategy template according to actual observation needs.

[0099] In one specific implementation, if the observation request does not carry the template identifier selected by the user, a strategy template acquisition request is sent to the terminal device; the observation strategy template customized by the user through the query language is received from the terminal device.

[0100] Specifically, if the user does not select any preset observation strategy template on the terminal device, an observation request is sent to the cloud server through other means. Since the observation request does not carry the template identifier selected by the user, the cloud server sends a strategy template retrieval request to the terminal device. The user then customizes the observation strategy template according to their observation needs using query languages ​​such as DSL, OTL, and OAL.

[0101] S203. Obtain the target observation data and target data model corresponding to the data type to be observed; wherein, the target observation data is data of the type to be observed that is collected in advance by the application using a non-invasive implanted probe and / or log collector, and the target data model is the metadata corresponding to the data type to be observed that is obtained in advance.

[0102] In one specific implementation, the target observation data and target data model corresponding to the data type to be observed can be obtained in the following way:

[0103] Step 1: Obtain the target observation data corresponding to the data type to be observed from the observable data database. The observable database stores data of multiple data types collected from multiple applications through non-intrusive probes and preset log collectors.

[0104] In this step, for each data type, the corresponding observation data is stored in the corresponding location in the observable data database.

[0105] In one specific implementation, the observation data is persistently stored in the search and analysis engine (Elasticsearch).

[0106] In the specific implementation of this solution, it should be understood that the probe provided in this application is integrated into the cloud service in a non-intrusive manner, capable of automatically discovering the application to be observed and collecting observation data of that application. This observation data may include user interactions, service calls, network requests, and database operations during application operation. Based on this observation data, observation metrics can be analyzed.

[0107] Step 2: Obtain the target data model corresponding to the data type to be observed from the observable model library. The observable model library stores multiple predefined data models corresponding to different data types. The data model corresponding to each data type is used to indicate the metadata of the data of that data type.

[0108] In this step, metadata defines the basic information indicating the data's data type, such as its name, meaning, type, data structure, and interaction method. The data model corresponding to metric data is the metric model, the data model corresponding to tracking data is the tracking model, and the data model corresponding to log data is the log model.

[0109] It should be understood that this step constructs a unified model for the observation data by defining metadata for data models corresponding to different data types. This ensures consistency and interoperability between data of different types, enabling the cloud server to integrate data of different types and facilitating the formation of a unified view in subsequent observation results. Furthermore, by defining metadata for data models corresponding to different data types and integrating the observation data into a unified model framework, it ensures that data of different types can be consistently expressed and processed across different services and components. It also facilitates centralized processing and comprehensive analysis of data of different types across service locations.

[0110] In the specific implementation of this solution, it should be understood that users can customize the dimensions of extended data types through model management definitions. By flexibly extending the data models corresponding to data types according to business needs, the method provided in this application can adapt to constantly changing business requirements and technical environments.

[0111] S204. Based on the target observation data, target data model, observation indicators, and indicator analysis strategies, conduct data analysis to obtain observation results;

[0112] In this step, the cloud server performs data analysis on the acquired target observation data based on the observation indicators and indicator analysis strategies in the template. Among them, the target data model metadata is used to unify target observation data of different data types, so as to centrally display different types of target observation data in the observation results.

[0113] In the specific implementation of this scheme, it should be understood that the observation results include the observation results of different observation indicators for different types of data. These observation results can be real-time indicator performance analysis or observation of historical trends.

[0114] S205. Return the observation results to the terminal equipment.

[0115] In this step, after the cloud server processes the data, it returns the observation results to the terminal device, which the user can then view.

[0116] In the specific implementation of this solution, it should be understood that users can monitor performance indicators and historical trends in real time through a graphical interface, and quickly respond to application performance bottlenecks or abnormal modes based on the observation results, thereby performing operations such as troubleshooting and repair, performance optimization, capacity planning and expansion of the business system, and realizing operation and maintenance monitoring and management of the business system.

[0117] For example, based on their observation needs for applications in the business system, users select an observation strategy template on their terminal devices. This template configures the data types to be observed, the common observation metrics for each data type (such as average response time and database response time), and the corresponding metric analysis strategies and alarm conditions. Based on the analysis results returned by the cloud server after data processing, users can promptly formulate performance optimization strategies or respond to alarms. For instance, when the observed data exceeds a preset data threshold, the cloud server triggers an alarm, and the Alert Manager component quickly notifies the alarm response team via Lark, email, or Webhook.

[0118] The cloud-based data processing method provided in this embodiment collects observation data from applications in a cloud-native architecture system using a non-intrusive probe technology. This improves the probe's adaptability to containerized environments and reduces the complexity of probe implantation, responding to changes in the containerized environment of cloud-native architectures. Furthermore, the cloud-based data processing method provides a standardized unified data model by pre-defining data model metadata corresponding to different data types, facilitating unified processing of different data types and improving the user's visualization experience of observation results. Additionally, the cloud-based data processing method provides a lower barrier to entry and enhances user experience by pre-storing observation strategy templates in a template management library.

[0119] Figure 3 The flowchart of Embodiment 2 of the cloud service-based data processing method provided in this application is as follows: Figure 3 As shown, based on the above embodiments, before obtaining the observation strategy template corresponding to the data observation request, the method further includes:

[0120] S301. Based on the obtained tracking status information and the basic configuration information of the application, a probe is implanted in the container corresponding to the application, and the container includes the mirror application of the application.

[0121] In this solution, in response to the cloud-native architecture of the business system, non-intrusive data collection from the application is achieved by containerizing the application and integrating probe technology with container orchestration tools.

[0122] In this step, the event tracking status information includes the event tracking location, event tracking type, and event tracking status. The application's basic configuration information includes the application's technology stack type, runtime environment, dependencies, and configuration files. Specifically, the event tracking location may be a button, page, or functional module; the event tracking type may be user behavior data (such as clicks, views, purchases), application performance data (such as page load time, call latency), or user attribute data (such as user ID, geographic location); and the event tracking status may indicate whether the event tracking is currently active.

[0123] Similarly, in this step, since the application and all its dependencies, runtime environment and configuration are packaged into a container, data collection from the original application can be achieved without being noticed by simply inserting a probe into the container corresponding to the application.

[0124] In one specific implementation, the probe type is determined based on the technology stack type in the application's basic configuration information. The probe type includes any of the following: Java probe, Go probe, or Node probe. The probe insertion method is determined based on the probe type, which includes automatic insertion or manual insertion. The probe is inserted into the application's corresponding container based on the tracking status information, the probe type, and the insertion method.

[0125] Specifically, the application's technology stack includes Java, Go, Node.js, Python, etc. Correspondingly, the probe types used are Java probes, Go probes, Node probes, Python probes, etc. For Java applications, due to the provision of Java Agent technology, in the sidecar mode of a container cluster, the cloud server can automatically embed probes in the container, achieving application-unobtrusive probe implantation. For other applications, manual reporting is required to embed probes in the container. This manual reporting method is implemented by the user modifying the application code.

[0126] The specific implementation of this solution refers to... Figure 4 The probe implantation process on the application side, such as Figure 4 As shown, the implantation process of the probe in application specifically includes:

[0127] Step 1: Probe initialization;

[0128] After the application starts, the probe can be automatically loaded using the Init APM Set method through the initialization interface.

[0129] Step 2: Create the service call binding;

[0130] Step 3: Invoking data reporting;

[0131] Steps 2 and 3 are required for applications other than Java applications. Since these applications do not provide proxy services, manual implementation is required for data tracking and reporting. Specifically, developers need to create the methods to be bound (i.e., data tracking points) based on the HTTP request binding interface and locally call the data reporting service. In this step, a New Trace Filter Function is used to create the service call binding, and the data reporting service used is LinkTrace Http Service.

[0132] For Java applications, after the probe is initialized, the cloud server can dynamically embed the probe in the container corresponding to the application, and automatically report data without calling the data reporting service.

[0133] In one specific implementation, it should be understood that the probe's observation metrics can be predefined, and third-party development teams can extend the metrics through interfaces.

[0134] Step 4: Report the event tracking status information and the application's basic configuration information to the cloud server's Observability Analysis Platform (OAP).

[0135] Step 5: Persistently store the event tracking status information and the application's basic configuration information in Elasticsearch. It should be understood that the reported data can be queried in OAP.

[0136] S302. During the application's operation, initial observation data is collected based on the implanted probes and the preset log collector.

[0137] In this step, during application runtime, the implanted probe collects the application's link data and alarm data; a log collector pre-configured in the container cluster is used to collect the application's log data.

[0138] In the specific implementation of this solution, please understand and continue to refer to... Figure 4 The data collection process on the user side, such as Figure 4 As shown, probe data acquisition includes:

[0139] Step 1: The user of the application initiates a service request;

[0140] For example, when a user uses application A in a business system, if the aforementioned probe implantation process includes a tracking point for the action of "the user opening interface B in application A", then when the user opens interface B in application A, a service request is initiated. This service request triggers the probe's data collection.

[0141] Step 2: Report the initial observation data acquired to OAP;

[0142] Specifically, whenever a user initiates a new request, the system automatically collects and reports the request access details in real time through the New Trace Filter Function.

[0143] It should be understood that developers can further customize the event tracking based on actual business needs. Similarly, when a user initiates a service request, the initial observation data corresponding to their customized event tracking needs to be reported to OAP.

[0144] Step 3: Persistently store the collected initial observation data in the search engine Elasticsearch, and query the reported data in OAP.

[0145] It should be understood that by persistently storing the data point status information, the application's basic configuration information, and the initial observation data and providing query services, the real-time and integrity of data acquisition are ensured, while also facilitating the later analysis and utilization of the data.

[0146] S303. Preprocess the initial observation data and store the processed observation data in the observable database according to the data type.

[0147] In this step, since the initial observation data obtained comes from different applications, and different applications may use different communication protocols, and the data quality of the initial observation data is poor, containing irrelevant and interfering information, it is necessary to preprocess the initial observation data and store the processed observation data in the observable database according to the data type.

[0148] In one specific implementation, log data in the initial observation data is parsed according to preset data processing rules, and the parsed log data is cleaned to obtain cleaned log data; non-log data other than log data in the observation data is parsed according to the data processing rules, and the parsed data is converted according to a protocol to obtain converted non-log data; wherein, the data processing rules include parsing methods for data from different data sources.

[0149] Specifically, the data processing rules include JSON parsing rules, delimiter parsing rules, and regular expression parsing rules. Non-log data may include link data, alarm data, etc. It should be understood that the initial observation data collected is parsed according to the data processing rules, and the parsed non-log data is adapted and transformed according to the protocol. The parsed log data is then subjected to in-depth log analysis, thereby classifying the data into standardized observation data of various data types, and storing the observation data in the observable database.

[0150] For example, the Kafka collector is used to clean, extract, and transform the initial log data collected.

[0151] In the specific implementation of this solution, it should be understood that the data processing rules also include an extensible rule base. Dynamic configuration is supported, allowing users to dynamically add, modify, and delete data parsing and processing rules through a user interface or application programming interface (API).

[0152] The cloud-based data processing method provided in this embodiment improves the applicability of probe technology to container environments by customizing probes and embedding them into the containers corresponding to the applications, and collecting initial observation data from different applications in real time. Furthermore, this method preprocesses the initial observation data based on scalable data processing rules, completing the standardization and data type classification of the observation data, facilitating subsequent observation and analysis of application performance by users.

[0153] Figure 5 This is a schematic diagram illustrating a specific embodiment of the cloud service-based data processing method provided in this application. Figure 5 As shown, the cloud service end-to-end observable data flow model provided in this application includes: custom probe data source collection and management, a data processing engine, and data analysis and monitoring alert strategy templates. Furthermore, probes can be customized according to the probe customization interface; the initial observation data collected is standardized according to the data rule engine, and the rules in the data rule engine are extensible; through the definition of model metadata, the tracking model, indicator model, and log model are integrated into a unified data model; users can analyze and monitor the observation indicators they are interested in based on the strategy templates pre-stored in the template management module, or templates created through the Observability Analysis Language (OAL), Object Template Library (OTL) query language, and Domain Specific Language (DSL) query language. Alerts can be sent via Lark, email, and SMS.

[0154] Figure 6 This is a schematic diagram of the structure of a cloud service-based data processing device according to Embodiment 1 of the present application, as shown below. Figure 4 As shown, the cloud service-based data processing device 40 provided in this embodiment includes:

[0155] The request receiving module 401 is used to receive data observation requests sent by the terminal device.

[0156] The first acquisition module 402 is used to acquire an observation strategy template corresponding to the data observation request. The observation strategy template includes the data type to be observed, the observation index, and the index analysis strategy.

[0157] The second acquisition module 403 is used to acquire target observation data and target data model corresponding to the data type to be observed; wherein, the target observation data is data of the type to be observed obtained in advance by using a non-invasive implanted probe and / or log collector to collect data from the application, and the target data model is metadata corresponding to the data type to be observed that is acquired in advance;

[0158] The result acquisition module 404 is used to perform data analysis based on the target observation data, the target data model, the observation indicators and the indicator analysis strategy to obtain the observation results.

[0159] The result return module 405 is used to return the observation results to the terminal device.

[0160] The cloud service-based data processing device provided in this embodiment can execute the methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0161] In one possible implementation, the second acquisition module 403 is specifically used for:

[0162] Obtain the target observation data corresponding to the data type to be observed from the observable data database. The observable database stores data of multiple data types collected from multiple applications through non-intrusive probes and preset log collectors.

[0163] The target data model corresponding to the data type to be observed is obtained from the observable model library. The observable model library stores multiple predefined data models corresponding to different data types. The data model corresponding to each data type is used to indicate the metadata of the data of that data type.

[0164] In one possible implementation, the first acquisition module 402 is specifically used for:

[0165] If the data observation request carries the template identifier selected by the user, then the observation strategy template corresponding to the template identifier is obtained from the template management library, which includes multiple predefined different strategy templates.

[0166] In one possible implementation, the first acquisition module 402 is specifically used for:

[0167] If the observation request does not carry the template identifier selected by the user, a policy template acquisition request is sent to the terminal device.

[0168] The terminal device receives the observation strategy template customized by the user through a query language.

[0169] The cloud service-based data processing device provided in this embodiment can execute the methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0170] Figure 7 This is a schematic diagram of the structure of Embodiment 2 of the cloud service-based data processing device provided in this application, as shown below. Figure 7 As shown, the cloud service-based data processing apparatus 40 provided in this application further includes:

[0171] Probe implantation module 406: used to implant probes into the container corresponding to the application based on the obtained embedding status information and the basic configuration information of the application, wherein the container includes a mirror application of the application.

[0172] Data acquisition module 407: used to collect initial observation data based on the implanted probe and the preset log collector during the application's operation;

[0173] Data processing module 408: used to preprocess the initial observation data and store the processed observation data in the observable database according to the data type.

[0174] The cloud service-based data processing device provided in this embodiment can execute the methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0175] In one possible implementation, the data processing module 408 is specifically used for:

[0176] The log data in the initial observation data is parsed according to the preset data processing rules, and the parsed log data is cleaned to obtain cleaned log data.

[0177] According to the data processing rules, the non-log data in the observation data other than the log data is parsed, and the parsed data is converted by a protocol to obtain the converted non-log data;

[0178] The data processing rules include methods for parsing data from different data sources.

[0179] In one possible implementation, the probe implantation module 406 is specifically used for:

[0180] The type of probe is determined based on the technology stack type in the basic configuration information of the application. The probe type includes any of the following: Java probe, Go probe, or Node probe.

[0181] Based on the type of probe, the implantation method of the probe is determined, and the implantation method includes automatic implantation or manual implantation;

[0182] Based on the embedded point status information, the probe type, and the implantation method, the probe is implanted into the container corresponding to the application.

[0183] The cloud service-based data processing device provided in this embodiment can execute the methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0184] Figure 8 A schematic diagram of the structure of the cloud service-based data processing device provided in this application. Figure 8 As shown, the cloud service-based data processing device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0185] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0186] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0187] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0188] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0189] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0190] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0191] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0192] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0193] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0194] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0195] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0197] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0198] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0199] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A data processing method based on cloud services, characterized in that, include: Receive data observation requests sent by terminal devices; Obtain the observation strategy template corresponding to the data observation request. The observation strategy template includes the data type to be observed, the observation indicators, and the indicator analysis strategy. Obtain target observation data and target data model corresponding to the data type to be observed; wherein, the target observation data is data of the type to be observed that is collected in advance by the application using a non-invasive implanted probe and / or log collector, and the target data model is metadata corresponding to the data type to be observed that is obtained in advance; Based on the target observation data, the target data model, the observation indicators, and the indicator analysis strategy, data analysis is performed to obtain observation results. The observation results are returned to the terminal device.

2. The method according to claim 1, characterized in that, The step of obtaining the target observation data and target data model corresponding to the data type to be observed includes: Obtain the target observation data corresponding to the data type to be observed from the observable data database. The observable database stores data of multiple data types collected from multiple applications through non-intrusive probes and preset log collectors. The target data model corresponding to the data type to be observed is obtained from the observable model library. The observable model library stores multiple predefined data models corresponding to different data types. The data model corresponding to each data type is used to indicate the metadata of the data of that data type.

3. The method according to claim 1, characterized in that, The step of obtaining the observation strategy template corresponding to the data observation request includes: If the data observation request carries the template identifier selected by the user, then the observation strategy template corresponding to the template identifier is obtained from the template management library, which includes multiple predefined different strategy templates.

4. The method according to claim 3, characterized in that, The method further includes: If the observation request does not carry the template identifier selected by the user, a policy template acquisition request is sent to the terminal device. The terminal device receives the observation strategy template customized by the user through a query language.

5. The method according to any one of claims 1 to 4, characterized in that, Before obtaining the observation strategy template corresponding to the data observation request, the method further includes: Based on the obtained tracking status information and the basic configuration information of the application, a probe is implanted in the container corresponding to the application, and the container includes a mirror application of the application. During application operation, initial observation data is collected based on the implanted probes and the preset log collector. The initial observation data is preprocessed, and the processed observation data is stored in the observable database according to the data type.

6. The method according to claim 5, characterized in that, The preprocessing of the initial observation data includes: The log data in the initial observation data is parsed according to the preset data processing rules, and the parsed log data is cleaned to obtain cleaned log data. According to the data processing rules, the non-log data in the observation data other than the log data is parsed, and the parsed data is converted by a protocol to obtain the converted non-log data; The data processing rules include methods for parsing data from different data sources.

7. The method according to claim 5, characterized in that, The step of implanting probes into the container corresponding to the application based on the obtained tracking status information and the application's basic configuration information includes: The type of probe is determined based on the technology stack type in the basic configuration information of the application. The probe type includes any of the following: Java probe, Go probe, or Node probe. Based on the type of probe, the implantation method of the probe is determined, and the implantation method includes automatic implantation or manual implantation; Based on the embedded point status information, the probe type, and the implantation method, the probe is implanted into the container corresponding to the application.

8. A data processing device based on cloud services, characterized in that, include: The request receiving module is used to receive data observation requests sent by terminal devices. The first acquisition module is used to acquire an observation strategy template corresponding to the data observation request. The observation strategy template includes the data type to be observed, the observation index, and the index analysis strategy. The second acquisition module acquires target observation data and target data model corresponding to the data type to be observed; wherein, the target observation data is data of the type to be observed obtained in advance by using a non-invasive implanted probe and / or log collector to collect data from the application, and the target data model is metadata corresponding to the data type to be observed that is acquired in advance; The result acquisition module performs data analysis based on the target observation data, the target data model, the observation indicators, and the indicator analysis strategy to obtain the observation results. The result return module returns the observation results to the terminal device.

9. A data processing device based on cloud services, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images